Methods, information handling systems and computer readable media are disclosed for a contextual image-based search. In one embodiment, a method includes receiving, at a computer implemented system for searching a knowledge base, an image submitted by a user of the system. The method continues with obtaining a classification label corresponding to the image, checking a context module database for the presence of a follow-up question associated with the classification label, and generating search term information for searching of the knowledge base, where the search term information comprises the classification label. If a follow-up question associated with the classification label is found, an embodiment further includes presenting the follow-up question to the user of the system, receiving from the user an answer to the follow-up question, modifying the search term information to include the answer, and checking the context module database for an additional follow-up question associated with the answer.