Methods and systems for large scale data integration in distributed or massively parallel environments comprises a development phase wherein the results of a proposed jobflow can be viewed by the user during development, including the results of upstream units where the data sources and data targets can be any of a variety of different platforms, and further comprises the use of remote agents proximate to those data sources and data targets with direct communication between the associated agents under the direction of a topologically central controller to provide, among other things, improved security, reduced latency, reduced bandwidth requirements, and faster throughput.