A video encoding apparatus for encoding a plurality of image blocks in a video frame includes an intra-frame prediction module, a transformation module and a quantization module. The intra-frame prediction module performs intra-frame prediction on the image blocks to generate a plurality of residual blocks. The transformation module performs a transformation on a target residual block along a predetermined direction according to a transformation matrix to generate a transformation result. The transformation matrix is a product of an initial transformation matrix and a secondary transformation matrix. The initial transformation matrix corresponds to a one-dimensional initial transform performed along the predetermined direction in a two-dimensional initial transform. The secondary transformation matrix corresponds to a one-dimensional secondary transform performed along the predetermined direction in a two-dimensional secondary transform. The quantization module quantizes the transformation result.