The invention provides an optical layer decomposition method based on a spatial transformation model, aiming to solve the technical problems of bad decomposition effect and low efficiency in a conventional non-supervision single mixed image decomposition method. The realization steps include reading in an original mixed image; fuzzing by means of a blind deconvolution algorithm; defining iterative variables and a transmission layer matrix iteration formula; creating a spatial transformation model and deriving the function expression of the transmission layer matrix and the reflection layer matrix; constructing target functions, adding constraint conditions by means of layer prior, performing regularization with L1 norms, and realizing the target function convex optimization; solving the optimal solution of convex target function by means of a Lagrangian multiplier method; calculating the transmission layer matrix and the reflection layer matrix by means of the function expression; and determining whether the iteration frequency is greater than the set threshold, if yes, the iteration is completed, otherwise, the convex target function is updated. The optical layer decomposition method improves the decomposition effect of the mixed image and improves the calculating efficiency.