The present teaching relates to system, method, medium for in-situ perception in an autonomous driving vehicle. A plurality of types of sensor data are received, which are acquired by a plurality of types of sensors deployed on the vehicle to provide information about surrounding of the vehicle. Based on at least one model, one or more surrounding items are tracked from a first of the plurality of types of sensor data acquired by a first type sensors. At least some of the tracked items are automatically labeled via cross validation and are used to locally adapt, on-the-fly, the at least one model. Model update information is received which from a model update center, which is derived based on the labeled at least some items. The at least one model is updated using the model update information.