A method of directly modifying ray tracing samples generated by a ray tracing renderer. Modifications to samples may be made after rendering and before rasterizing, in contrast to typical compositing workflows that manipulate pixels of a rasterized image. Modifications may be based on user input. Rasterization may be performed afterwards at any desired resolutions, for example to adapt to different displays. Samples may be tagged with object identities, facilitating object selection without the need for object masks. Pseudo-random ray patterns typically used by renderers may be supported directly. Many operations may be performed directly on samples, including color changes, object repositioning, and merging of samples from different scenes. Secure samples with scrambled ray directions may be modified directly. Modified samples may be compressed for storage or transmission, and may be streamed to receiving devices that rasterize the samples for display, possibly after decompressing or decrypting the received streams.