A toy building block set includes multiple polyhedron blocks of different lengths removably attachable together. Each block includes at least a top and four side walls. At least one connector extends from the top and includes opposing interior and exterior surfaces. A recessed area is provided at the bottom of each block. Engagement projections or depressions are provided on the connector and the remaining engagement structure are provided in the recessed area of each block where they might fully or partially engage or not engage with structures on the connector. The mechanical interference engagements and resulting engagement forces provided between any two of the blocks are less than three and even less than two times the number of engagements and resulting engagement forces provided by a mated pair of the smallest polyhedron blocks of the set.