A monolithic three dimensional array of non-volatile storage elements is arranged in blocks. The non-volatile storage elements are connected to bit lines and word lines. The bit lines for each block are grouped into columns of bit lines. The columns of bit lines include top columns of bit lines that are connected to selection circuits on a top side of a respective block and bottom columns of bit lines that are connected to selection circuits on a bottom side of the respective block. Programming of data is pipelined between two or more columns of bit lines in order to increase programming speed. One embodiment of the programming process includes selectively connecting two columns of bit lines to a set of one or more selection circuits, using the one or more selection circuits to selectively connect one of the two columns of bit lines to one or more signal sources, programming non-volatile storage elements for the column of bit lines that is currently connected to the one or more signal sources, and changing one of the columns of bit lines connected to the set of one or more selection circuits while another column of bit lines is being programmed.