Computationally recreating vision evolution
A What-if Eye... Universe?
Evolution happened once. We build a video-game universe to replay evolution. This allows us to computationally study principles behind biological intelligence and create new forms of artificial intelligence.
Agents begin with a single light-sensing cell and evolve their visual intelligence by facing real physics, embodied constraints, and survival pressures.
The point is to let visual intelligence, the ability to sense (hardware), percieve, reason, and act (software) in an environment, emerge as opposed to being engineered by fixed datasets and human biases.
What if Darwin had an what-if evolution machine?* *Comic inspired by the xkcd comics and "What-if" books by Randall Munroe
What if questions about vision, answered by evolving AI agents
Each experiment is a hypothesis that we test in our What-If Machine. We pose a what-if question or counterfactual, evolve embodied agents inside video-game-like physics engines, and watch which eyes and behaviors emerge.
What if the goals of vision were different?
We initialize our agents with one light-sensing cell and a small brain and evolve their visual intelligence in a world with only two tasks:
- NAVIGATION where the goal is to move as fast as possible to the left while avoiding obstacles (walls of the maze).
- DETECTION where the goal is to detect the food and avoid the poison.
Navigation agents favor distributed, wide-coverage vision.
Detection favors high-acuity, forward-facing camera-like eyes.
What if brains stayed small throughout evolution?
When we systematically scale eyes and brain size at the same time, we uncover power-law scaling between neural capacity and task performance — but only when visual acuity scales too. If acuity is bottlenecked, scaling the “brain” alone stops buying better behavior.
Scaling laws in AI (Kaplan et al., 2020) relate test loss to compute, dataset size, and parameters, without accounting for an agent’s visual acuity. In embodied settings, we show that acuity matters because it compresses the representation the agent receives from the world.
In biology, we observe scaling between eyes and brains across animals. Figure from "The scaling of eye size in adult birds: Relationship to brain, head and body sizes" (Richard Burton). We show that this scaling behavior can be reproduced with our agents.
Scaling detection performance as eyes and brains grow together. Each blue line represents agents trained with the same visual acuity but larger number of parameters. Darker blue lines represent higher visual acuity.
What if eyes could bend light?
When we enable optical genes, evolution repeatedly discovers lens-like optics because they solve a brutal constraint:
- Without optics, systems hit a hard ceiling — pinhole strategies can sharpen images only by sacrificing light.
- With optics, lenses emerge as a solution to the fundamental tradeoff between light collection and spatial precision.
Optics evolve from open → cup → pinhole → unfocused lens → focused lens. Lenses preserve spatial precision while allowing larger pupils for more light.
The yellow frustrum indicates the light being collected by the eye. The pinhole eye collects less light but has higher reward than the open eye. Once the agents evolve lenses, they can collect more light and have higher acuity so they learn more robust behaviors as a result.
Unifying Genotype for Vision and Learning
We create a unifying genotype that includes components of the hardware (physical vision sensor) and the learning components (software). The encoding splits vision into three independently mutable gene clusters. Morphological genes set spatial sampling traits like eye placement and field of view. Optical genes govern light interaction (photoreceptor count, optical elements, pupil size). Neural genes specify learning capacity. This separation lets mutations explore realistic evolutionary pathways.
From left to right, top to bottom: agent vision when changing the number of eyes, photoreceptors, positions, and the pupil size.
Publications, talks, code, and exhibitions
Explore our publications, roadmaps, talks, open-source tools, and public exhibits.
Publications
Computationally Recreating Vision Evolution
Science Advances
The peer-reviewed publication detailing the full experimental setup, results, and evolutionary analysis.
A Roadmap for Generative Design of Visual Intelligence
MIT Press
Why should we generate and not hand design visual intelligence? We discuss why it's important, the applications of doing so, and how to get there.
Designing Imaging Systems with Reinforcement Learning
ICCV
We propose a new way to codesign imaging systems and task-specific perception models based on feedback from the environment.
Emergence of foveal image sampling from learning to attend in visual scene
NeurIPS
We show that learning to attend in visual scenes leads to foveal image sampling, a key visual system feature.
Designing neural network architectures using reinforcement learning
ICLR
We design neural network architectures using reinforcement learning to improve performance on visual tasks.
Upcoming work!
Coming soonWe are working on a lot of exciting things that we will be releasing soon! We are always looking for collaborators and partners to work with in expanding our work to new domains. Feel free to reach out to us to join our team!.
Press Coverage
A “scientific sandbox” lets researchers explore the evolution of vision systems
MIT scholars awarded a second round of seed grants for generative AI research
Why have so many different eyes evolved? Gamelike simulation could provide answers
Beyond computer vision: brains in jars and how they see.
Talks
Highlights from a talk at UC Berkeley's Redwood Center by Kushagra Tiwary. Covers the project vision and open research questions.
Tedx talk exploring the broader implications of evolving visual intelligence with artificial agents.
More talks and presentations are on their way! Stay tuned for upcoming speaking events and recordings.
Code
Simulator
Open SourceRun the evolutionary simulator, define new tasks, and evolve your own embodied agents.
Colab Notebook (coming soon)
NotebookInteractive notebooks for experimenting with Cambrian agents directly in the browser.
Exhibitions
Video Exhibition
Public EngagementHow can we interact with evolution? Our exhibitions let visitors experience evolving vision in a hands-on way.
A collaboration supported by the MIT GenAI Impacts of Generative AI Grant.