A semiconductor memory device includes a substrate, an isolation layer, a trench, a semiconductor active structure, and a floating gate electrode. The isolation layer is disposed on the substrate. The trench penetrates through the isolation layer and exposes a part of the substrate. The semiconductor active structure is disposed in the trench, and the floating gate electrode is disposed on the semiconductor active structure. A manufacturing method of the semiconductor memory device includes the following steps. The isolation layer is formed on the substrate. The trench is formed penetrating through the isolation layer and exposing a part of the substrate. The semiconductor active structure is formed in the trench. The floating gate electrode is formed on the semiconductor active structure.