The present invention proposes an ensemble deep learning method for identifying unsafe behaviors of operators in maritime working environment. Firstly, extract features of maritime images with the You Only Look Once (YOLO) V3 model, and then enhance a multi-scale detection capability by introducing a feature pyramid structure. Secondly, obtain instance-level features and time memory features of the operators and devices in the maritime working environment with the Joint Learning of Detection and Embedding (JDE) paradigm. Thirdly, transfer spatial-temporal interaction information into a feature memory pool, and update the time memory features with the asynchronous memory updating algorithm. Finally, identify the interaction between the operators, the devices, and unsafe behaviors with an asynchronous interaction aggregation network. The proposed invention can accurately determine the unsafe behaviors of the operators, and thus provide operation decisions for maritime management relevant activities.