A semiconductor-on-insulator (SOI) substrate includes a handle substrate, a charge-trapping layer located over the handle substrate and including nitrogen-doped polysilicon, an insulating layer located over the charge-trapping layer, and a semiconductor material layer located over the insulating layer. The nitrogen atoms in the charge-trapping layer suppress grain growth during anneal processes used to form the SOI substrate and during subsequent high temperature processes used to form semiconductor devices on the semiconductor material layer. Reduction in grain growth reduces distortion of the SOI substrate, and facilitates overlay of lithographic patterns during fabrication of the semiconductor devices. The charge-trapping layer suppresses formation of a parasitic surface conduction layer, and reduces capacitive coupling of the semiconductor devices with the handle substrate during high frequency operation such as operations in gigahertz range.