We propose a neural network-based base caller that detects and accounts for stationary, kinetic, and mechanistic properties of the sequencing process, mapping what is observed at each sequence cycle in the assay data to the underlying sequence of nucleotides. The neural network-based base caller combines the tasks of feature engineering, dimension reduction, discretization, and kinetic modelling into a single end-to-end learning framework. In particular, the neural network-based base caller uses a combination of 3D convolutions, 1D convolutions, and pointwise convolutions to detect and account for assay biases such as phasing and prephasing effect, spatial crosstalk, emission overlap, and fading.