An optimized computer architecture for training an neural network includes a system having multiple GPUs. The neural network may be divided into separate portions, and a different portion is assigned to each of the multiple GPUs. Within each GPU, its portion is further divided across multiple training worker threads in multiple processing cores, and each processing core has lock-free access to a local parameter memory. The local parameter memory of each GPU is separately, and individually, synchronized with a remote master parameter memory by lock memory access. Each GPU has a separate set of communication worker threads dedicated to data transfer between the GPU and the remote parameter memory so that the GPU's training worker threads are not involved with cross GPU communications.