A method and apparatus are disclosed for enabling nodes in a distributed system to share one or more memory portions. A home node makes a portion of its main memory available for sharing, and one or more sharer nodes mirrors that shared portion of the home node's main memory in its own main memory. To maintain memory coherency, a memory coherence protocol is implemented. Under this protocol, a special data value is used to indicate that data in a mirrored memory location is not valid. This enables a sharer node to know when to obtain valid data from a home node. With this protocol, valid data is obtained from the home node and updates are propagated to the home node. Thus, no “dirty” data is transferred between sharer nodes. Consequently, the failure of one node will not cause the failure of another node or the failure of the entire system.