A semiconductor layer is prepared in which a silicon substrate, a BOX layer and an SOI layer are laminated in this order. A silicon diode section used as an infrared detection portion is formed in the SOI layer. Further, an isolation portion is formed so as to extend from the SOI layer to a predetermined depth of the silicon substrate via the BOX layer. The isolation portion is formed so as to surround an area in which the silicon diode section is formed, and have the form of a circle or a regular polygon more than a regular pentagon in shape. A protective film is formed on the surface of the SOI layer. Thereafter, etching holes that penetrate the protective film, the SOI layer and the BOX layer are formed. The silicon substrate corresponding to each area surrounded by the isolation portion is etched using the etching holes.