Described are variational Expectation Maximization (EM) embodiments for learning a mixture model using component-dependent data partitions, where the E-step is sub-linear in sample size while the algorithm still maintains provable convergence guarantees. Component-dependent data partitions into blocks of data items are constructed according to a hierarchical data structure comprised of nodes, where each node corresponds to one of the blocks and stores statistics computed from the data items in the corresponding block. A modified variational EM algorithm computes the mixture model from initial component-dependent data partitions and a variational R-step updates the partitions. This process is repeated until convergence. Component membership probabilities computed in the E-step are constrained such that all data items belonging to a particular block in a particular component-dependent partition behave in the same way. The E-step can therefore consider the blocks or chunks of data items via their representative statistics, rather than considering individual data items.