In some aspects, an edge-based data collection system discovers, collects, processes, and forwards data in an observability pipeline system. In some implementations, an edge agent of the observability pipeline system runs on a computer node. The edge agent identifies processes running on the computer node; identifies files on the computer node that the processes have opened for writing; accesses log discovery parameters of the observability pipeline system; selects a plurality of files from the identified files according to the log discovery parameters; generates a list of discovered log files that includes a path and a name for each of the plurality of files; adds the list of discovered log files to a list of monitored log files to be monitored by the observability pipeline system; and then monitors the plurality of files to generate input for the observability pipeline system.