A method of manufacturing a semiconductor device is provided which includes providing a semiconductor layer having a first area and a second area separated from the first area by an isolation structure, forming a protection layer on the isolation structure, forming at least partly a memory device in and on the first area, removing the protection layer, and forming a field effect transistor (FET) in and over the second area after the removal of the protection layer.