Embodiments of the present invention extend protection of network traffic between different security realms based on security labeling. In particular, embodiments of the present invention label provide for implicit labeling of traffic shared between different security realms. The traffic may be shared using IPsec protocols. A gateway inspects the IPsec traffic and identifies security associations (SAs) of the IPsec traffic. The gateway then determines a security label of the SA. Various access control policies may then be applied to the traffic based on its security label.