A three-dimensional (3D) display device for displaying a 3D image using at least one of a gaze direction of a user and a gravity direction includes a gaze direction measuring unit to measure the gaze direction, a data obtaining unit to obtain 3D image data for the 3D image, a viewpoint information obtaining unit to obtain information relating to a viewpoint of the 3D image, a data transform unit to transform the 3D image data, based on the gaze direction and the information relating to the viewpoint of the 3D image, and a display unit to display the 3D image, based on the transformed 3D image data.