Disclosed herein are examples embodiments of a progressive modeling scheme to enhance optical sensor transformation networks using both in-field sensor measurements and simulation data. In one aspect, a method includes receiving optical sensor measurements generated by one or more downhole optical sensors in a wellbore; determining synthetic data for fluid characterization using an adaptive model and the optical sensor measurements; and applying the synthetic data to determine one or more physical properties of a fluid in the wellbore for which the optical sensor measurements are received.