Various embodiments for implementing a system and method for managing and querying semi-structured, heterogeneously-typed sensor data in a distributed environment are disclosed. Example embodiments include: enabling data communications between a plurality of compute nodes and a plurality of physical data storage devices via an application programming interface (API) layer, a cache management layer, a server layer, and a storage layer; and receiving a data request from at least one of the plurality of compute nodes at the API layer, the data request including an identification of a topic of a dataset, the topic including a metadata file, a data file, and an index file, the index file including at least one pointer into the data file.