A hydrocracking system involves introducing a heavy oil feedstock and a colloidal or molecular catalyst, or a catalyst precursor capable of forming the colloidal or molecular catalyst, into a hydrocracking reactor. The colloidal or molecular catalyst is formed in situ within the heavy oil feedstock by 1) premixing the catalyst precursor with a hydrocarbon diluents to form a catalyst precursor mixture, 2) mixing the catalyst precursor mixture with the heavy oil feedstock, and 3) raising the temperature of the feedstock to above the decomposition temperature of the catalyst precursor to form the colloidal or molecular catalyst. The colloidal or molecular catalyst catalyzes upgrading reactions between the heavy oil feedstock and hydrogen and eliminates or reduces formation of coke precursors and sediment. The colloidal or molecular catalyst can be the sole or primary hydrocracking catalyst in a stand-alone hydrocracking reactor or it can be used together with a porous supported catalyst within a fixed bed or ebullated bed reactor.