A luggage management system includes: a locking/unlocking device mounted in a vehicle and configured to lock and unlock the vehicle in response to a request from a terminal, the terminal being used by a user who delivers luggage; an imaging device configured to capture an image of an inside of a predetermined area located in the vehicle, the predetermined area being used as a delivery place of the luggage; a control device configured to cause the imaging device to capture the image in the predetermined area when the locking/unlocking device locks or unlocks the vehicle in response to the request, and store the image; and a determination unit configured to determine, based on a result of comparison between a reference image and a comparison image, whether the luggage has been carried away from the predetermined area.