Described is a technology in a computing environment comprising a programming language for general purpose computation on a graphics processing unit (GPU), along with an associated compiler. A Bulk-Synchronous GPU Programming (BSGP) program is programmed to include barriers to describe parallel processing on GPUs. A BSGP compiler detects barriers corresponding to supersteps, converts BSGP programs to kernels based on the barriers, and combines them. During compilation, the compiler aligns barriers in the statements and bundles the corresponding supersteps together. A par construct is provided to allow the programmer to control aspects of bundling, e.g., by specifying a block independent statements. Thread manipulation emulation is provided to transparently emulate thread creation and destruction, with operations fork and kill. Also provided is remote variable access intrinsics for efficient communications between threads, and collective primitive operations.