A method and apparatus for encoding a video sequence comprising a plurality of pictures are disclosed. In video coding systems, the temporal redundancy is exploited using motion compensated prediction. The video sequence is often organized into multiple GOP (group of pictures) where different types of GOP may be used. In conventional coding systems, IPPP and IBBP GOP structure is often used. In H.264/AVC and the emerging High Efficiency Video Coding (HEVC), hierarchical GOP structure, including hierarchical P GOP structure and hierarchical B GOP structure, has been introduced to allow temporal scalability. Furthermore, low-delay IBBB GOP structure has been also introduced, for low-delay application. In the present invention, a low-delay hierarchical B GOP structure is disclosed. The new structure uses low-delay B-pictures only so as to minimize the processing delay while the hierarchical structure provides the temporal scalability. The low-delay hierarchical B GOP structure has been shown to result in substantial improvement in coding efficiency.