The present disclosure provides a method and a system for determining a prediction mode for performing intra prediction of a plurality of frames of a digital video in H.264 video coding format. The method and system include a determination module to determine a 16×16 prediction mode from a first set of plurality of prediction modes for each of intra 16×16 macroblock utilizing a first pre-defined criteria and decide a 4×4 prediction mode from a second set of plurality of prediction modes for each of intra 4×4 sub-macroblock utilizing a second pre-defined criteria, a comparison module to compare computational cost of the 16×16 prediction mode and the 4×4 prediction mode and a declaration module to declare one of the 16×16 prediction mode and the 4×4 prediction mode as the prediction mode for each of the macroblock based on the compared computational cost.