Researchers at the Lawrence Berkeley National Laboratory’s Molecular Foundry have integrated a large language model with advanced transmission electron microscopes, allowing scientists to operate complex imaging equipment using conversational plain language. The system, designated TEM Agent, bridges the gap between sophisticated nanoscale hardware and the shortage of operators trained to run it.
Connecting Scientific Hardware to Model Context Protocols
Transmission electron microscopes let researchers image atomic structures roughly half the size of a hydrogen atom. However, running these instruments requires specialized skills and generates massive data loads. A single detector can produce 700 gigabytes of data in 15 seconds, creating an operational bottleneck at facilities like the Molecular Foundry.
Traditional microscope automations resemble rigid factory lines. They execute only pre-programmed sequences and break down when faced with unexpected variables. To fix this, staff scientist Peter Ercius and his team rewrote existing automation functions into a Model Context Protocol format. This open standard provides the language model with a fixed set of specific, pre-written actions.
“With TEM Agent, I can just describe what I want in plain language, and it can string those steps together itself: even requests I never specifically programmed for,” Ercius said in a study published in npj Computational Materials.
Testing TEM Agent on Complex Electron Tomography
When the research team first tested the system, they assigned it standard tasks such as adjusting magnification, tilting samples, and refocusing beams. One major test involved electron tomography, a technique that builds a three-dimensional view of atomic structures from a series of tilted images over several hours.
The initial attempt faltered because the agent confused sample rotation with scanning direction rotation. The team corrected the behavior by providing a brief explanatory prompt about how to run the experiment. Following this adjustment, research scientist Stephanie Ribet pushed the system into more demanding territory, including ptychography, a computational imaging technique relying on diffraction patterns.
“While it is easy to connect LLMs to literature and data analysis pipelines, integrating an LLM with scientific hardware can be more challenging,” Ribet said.
Did you know? A single detector at the Molecular Foundry can generate roughly 700 gigabytes of data—equivalent to several smartphones’ worth of photos—in just 15 seconds.
TEM Agent Links to Supercomputing and Data Platforms
Beyond controlling the microscope, the team linked TEM Agent to the National Energy Research Scientific Computing Center and Crucible, a data platform developed at the Molecular Foundry. Morgan Wall, computer systems engineer and data scientist leading Crucible’s development, noted that the existing script infrastructure made integration straightforward.
“Instead of a user having to understand the semantics of five different scripts talking to five different computers, they can just use natural language to talk to one interface to control all of these systems,” Wall said. These connections allow raw data to be processed in seconds while storing experimental metadata for future retrieval.
Questions About Microscope Automation
What is TEM Agent?
TEM Agent is a virtual assistant powered by a large language model and built on the Model Context Protocol. It allows researchers to control advanced transmission electron microscopes using plain English commands rather than custom code.
How does the system handle massive data generation?
The system connects the microscope directly to the National Energy Research Scientific Computing Center and the Crucible data platform. This setup processes raw data streams in seconds and automatically organizes experimental metadata.
Can non-expert users operate the microscopes with this tool?
Yes. The system is designed to democratize access by allowing visiting scientists and non-experts to execute complex techniques like electron tomography and ptychography without writing custom code for every scenario.
The Molecular Foundry is collaborating with other nanoscale research centers to expand Crucible to new use cases and integrate it with platforms like the Advanced Light Source’s data catalog.