Techniques are disclosed herein for determining whether there is a defect that occurred as a result of programming non-volatile storage elements. Example defects include: broken word lines, control gate to substrate shorts, word line to word line shorts, double writes, etc. The memory cells may be programmed such that there will be a substantially even distribution of the memory cells in different data states. After programming, the memory cells are sensed at one or more reference levels. Two sub-groups of memory cells are strategically formed based on the sensing to enable detection of defects in a simple and efficient manner. The sub-groups may have a certain degree of separation of the data states to avoid missing a defect. The number of memory cells in one sub-group is compared with the other. If there is a significant imbalance between the two sub-groups, then a defect is detected.