The present invention relates to a method and device (580) for encoding three-dimensional video data, the device comprising: a first encoder (505) arranged to encode multiple simultaneous views (501) of a scene from different viewpoints; a second encoder (510) arranged to encode depth information of the scene and a third encoder (515) arranged to encode additional information indicative of a relationship between the multiple views and the depth information and a combiner (520) arranged to combine the encoded information into a representation (513) of the three-dimensional video data. The additional information comprises a group classifier indicating whether or not the depth information and at least one of the multiple views correspond to the same source material for, when corresponding, using the depth information and the at least one of the multiple views to render further views of the scene from further different viewpoints. The invention further related to a method and device (585) for decoding three-dimensional video data, as well as a computer program product comprising program instructions for executing a method according to the present invention.