An apparatus comprises a processing device configured to obtain monitoring data for devices comprising a plurality of features, determining two or more rankings of the features using two or more filter-based feature selection algorithms, and selecting a subset of the features based at least in part on the two or more rankings. The processing device is also configured to generate a failure prediction for a given one of the devices using at least one classifier that takes as input the selected subset of features, and applying a conformal prediction framework to the generated failure prediction to obtain a confidence measure indicating a quality of the generated failure prediction and a credibility measure indicating a quality of the monitoring data. The processing device is further configured to initiate one or more remedial actions based at least in part on the generated failure prediction, the confidence measure, and the credibility measure.