An image classification system is provided for determining a likely classification of an image using multiple machine learning models that share a base machine learning model. The image classification system may be a browser-based system on a user computing device that obtains multiple machine learning models over a network from a remote system once, stores the models locally in the image classification system, and uses the models multiple times without needing to subsequently request the machine learning models again from the remote system. The image classification system may therefore determine likely a classification associated with an image by running the machine learning models on a user computing device.