The structure of a semiconductor device with source/drain contact structures and via structures and a method of fabricating the semiconductor device are disclosed. A method for fabricating a semiconductor device includes forming a source/drain (S/D) region on a substrate, forming a S/D contact structure on the S/D region, and forming a via structure on the S/D contact structure. The forming of the via structure includes forming a via opening on the S/D contact structure, forming a non-metal passivation layer on sidewalls of the via opening, and depositing a via plug within the via opening in a bottom-up deposition process.