An analysis system for unknown application layer protocols, which could automatically discover unknown applications existing in a network, and then obtain keywords, attribute values, status codes or type codes representing semantic meaning of each field in each type of unknown application as well as message formats, dialogue rules and status transfer relations of application layer protocols by using cluster analysis and optimal partitioning method based on hidden semi-Markov model. Unknown application analysis result could be used for flow management and safety protection of a network. The system has the following advantages: it avoids difficulties arising from manual discovery and analysis of unknown applications, and improves network management efficiency and responding speed against new types of network attacks.