A video signal processing apparatus that reduces the degree of occurrences of false colors and of differing line concentration by performing knee processing properly, thereby providing desirable video signals. In the apparatus, at the occasion of generating luminance signals and color differences based on the result of knee processing, it is possible to properly generate a luminance signal which is derived from the addition result of pixels which are next to each other and generate a color difference signal which is derived from a difference of the pixels next to each other, achieved by performing the knee processing while keeping the signal level ratio at least between the pixels next to each other on video signals made up of color components arranged on a pixel by pixel basis. In this way, it is possible to keep a balance in hue of a color signal even after being subjected to knee processing.