A waveguide optical gyroscope includes a multilayer waveguide rotation sensor fabricated on a substrate. The multilayer waveguide rotation sensor includes one or more overlaying non-intersecting, spiraling coils that are vertically separated to reduce or eliminate optical cross coupling. The waveguides are optically coupled by a vertical waveguide and are optically coupled to the other components of the optical gyroscope, including a light source and detector, which may be integrated or fabricated on the substrate. A lithium niobate phase modulator chip may be disposed on the substrate and optically coupled to the waveguides in the multilayer waveguide rotation sensor. The multilayer waveguide rotation sensor enables a small cross section for the guiding channels thereby achieving a high coil density in a small volume.