In general, in one aspect, the invention relates to a method for managing data. The method includes obtaining, by a local data manager, a first data confidence fabric (DCF) configuration file, where the first DCF configuration file is associated with a first DCF pipeline and a first workload. The method further includes registering the first DCF pipeline in a DCF pipeline registry, obtaining a data set, identifying the first DCF pipeline using the DCF pipeline registry, and processing the data set based on the first DCF pipeline to obtain first processed data.