Embodiments relate to a multi-mode demosaicing circuit able to receive and demosaic image data in a different raw image formats, such as Bayer raw image format and Quad Bayer raw image format. The multi-mode demosaicing circuit comprises different circuitry for demosaicing different image formats that access a shared working memory. In addition, the multi-mode demosaicing circuit shares memory with a post-processing and scaling circuit configured to perform subsequent post-processing and/or scaling of the demosaiced image data, in which the operations of the post-processing and scaling circuit are modified based on the original raw image format of the demosaiced image data to use different amounts of the shared memory, to compensate for additional memory utilized by the multi-mode demosaicing circuit when demosaicing certain types of image data.