System, methods, and other embodiments described herein relate to conditionally generating custom images by sampling latent space of a generator. In one embodiment, a method includes, in response to receiving a request to generate a custom image, generating a component instruction by translating a description about requested characteristics for the object instance into a vector that identifies a portion of a latent space within a respective generator. The method includes computing the object instance by controlling the respective one of the generators according to the component instruction to produce the object instance. The respective one of the generators being configured to generate objects within a semantic object class. The method includes generating the custom image from at least the object instance to produce the custom image from the description as a photorealistic image approximating a real image corresponding to the description.