A semiconductor structure includes a semiconductor channel of a first conductivity type located between a first and second active regions having a doping of a second conductivity type that is opposite of the first conductivity type, a gate stack structure that overlies the semiconductor channel, and includes a gate dielectric and a gate electrode, a first metal-semiconductor alloy portion embedded in the first active region, and a first composite contact via structure in contact with the first active region and the first metal-semiconductor alloy portion, and contains a first tubular liner spacer including a first annular bottom surface, a first metallic nitride liner contacting an inner sidewall of the first tubular liner spacer and having a bottom surface that is located above a horizontal plane including bottom surface of the first tubular liner spacer, and a first metallic fill material portion embedded in the first metallic nitride liner.