A modular grip panel system for a firearm has a frame to support a pair of grip panels and a backstrap. The system includes a left and right grip panels having pockets, and a backstrap having a boss. The frame has a grip disposed at its rear end having a left and right side bosses to engage the pockets of the grip panels, and a rear surface pocket to receive the backstrap boss. The grip panels are disposed on the side surfaces of the pistol-style grip by insertion of the bosses into the pockets. The backstrap is disposed on the rear surface of grip by insertion of the backstrap boss into the rear surface pocket. When the backstrap is secured to the frame, the boss of grip panels are held in a forwardmost position. The backstrap prevents movement of the left and right grip panels towards the rear end of the frame.