A system and method for a secure and scalable data collection and ingestion data pipeline is discussed. The data pipeline provides secure ingestion of data into internet-of-things (IoT) platform, while providing high throughput, replication, and fault-tolerance, which makes it suitable for large scale systems where data can be received from millions or billions of devices, sensors, or clients. In addition, the data pipeline provides a platform for real time event generation and data analysis. Further the data pipeline reduces operational costs and platform-as-a-service (PAAS) vendor dependencies.