A ferroelectric memory device includes an alternating stack of insulating layers and electrically conductive layers, a memory opening vertically extending through the alternating stack, and a memory opening fill structure located in the memory opening and containing a vertical stack of memory elements and a vertical semiconductor channel. Each memory element within the vertical stack of memory elements includes a crystalline ferroelectric memory material portion and an epitaxial template portion.