A system and method for generating high-resolution stereo images and depth map in multi-camera systems having multiple cameras with different resolutions and view angles. One method is to improve the lower resolution image and combining it with the higher resolution image, then the resulting image is processed by extensive algorithms to ensure utmost high quality. The system can also handle non-planar image contents. The process is to generate a crude depth map first and then divide the map into multiple layers. Each layer will be separately registered. The results from the registered layers will be merged to improve the depth map generation. The improved depth map could be repeatedly fed back to the beginning of the process to further improve the registration performance. The system and method can generate stereo images using uncalibrated cameras with different views and resolutions.