A building block system includes building blocks which have elongated grooves on opposite faces. An elongated connector is received by the grooves of two adjacent building blocks and serves to hold the building blocks together. The connector is slightly wider than the grooves so it can be press fit into the grooves to ensure a tight fit. The connector is flexible so that it can be received by curved grooves, and the connector also has chamfered ends to facilitate insertion into the grooves. The system can be used to build structures containing a plurality of building blocks.