A luminance signal and a color information signal having high S/N ratios are generated by combining a visible luminance image component and an infrared image component at an appropriate ratio according to brightness of a subject such as brightness on a sunny day or brightness in darkness. An image pickup element 3 picks up a ye image component Dye, an R image component DR, an infrared image component DBlk and a luminance image component DW. An evaluation value calculator 41 calculates an evaluation value eDW and an evaluation value eDBlk by adding pixel values of the luminance image component DW and pixel values of the infrared image component DBlk in a local area, and calculates an evaluation value e (=eDW−eDBlk). A factor determiner 51 determines weighting factors k, kw from the evaluation value e. A signal generator 44 weights the luminance image component by the weighting factor k, weights the visible luminance image component by the weighting factor kw, combines the luminance image component and the visible luminance image component to calculate a luminance signal dY, and also calculates color signals dR, dG and dB.