The present invention relates to a method for hierarchically matching a building image. The method includes the steps of: matching a wall of a specific building in the building image inputted as a query with a wall(s) of a building(s) in at least one panoramic image by using a technology of matching a building's shape or repeated pattern; selecting a candidate panoramic image(s) which includes a building(s) recognized to have the same or similar wall to the specific building in the panoramic image(s) as a result of matching its wall with others; matching at least one local region, if containing a recognizable string or figure, in the specific building with local region(s) in the building(s) of the candidate panoramic image(s) by using a technology of recognizing a string or a figure; and determining top n panoramic image(s) as the result of matching the local region.