After Fire and Monsoons, DESI Resumes Cataloguing the Cosmos
Collaborators on the Dark Energy Spectroscopic Instrument breathe a sigh of relief as they successfully restart the state-of-the-art experiment
On June 11, lightning struck a remote ridge in the Baboquivari Mountain range outside of Tucson, Arizona. Within days, the Contreras Fire had traveled eight miles and climbed Kitt Peak, a 6,800-foot mountain dotted with white telescope domes. Within one was the Dark Energy Spectroscopic Instrument (DESI), the heart of a next-generation sky survey that is creating the largest 3D map of the universe.
Researchers use DESI to study dark energy, the mysterious force accelerating the expansion of our universe. It’s a clue into the fundamental workings of nature, how the universe evolved, and how it may end.
Collaborators who had spent years designing, building, and running the instrument watched the flames sweep over the observatory’s southern ridge on webcams – until the power went out. They switched to watching the curlicue paths of planes dropping fire retardant.
When the smoke had cleared, teams returned to find something astounding: All of the scientific equipment was intact. For several weeks, they carefully cleaned components and turned DESI’s systems on one by one. On Sept. 10, DESI began imaging the night sky once again.
“We’re relieved to return to our science with equipment that is performing almost as well as it was before the fire,” said Michael Levi, director of the international DESI collaboration and a scientist at the Department of Energy’s Lawrence Berkeley National Laboratory (Berkeley Lab), which manages the project. “I’m eternally grateful to the firefighters and the crews who secured the site, and their patience and ingenuity getting things running again.”
It’s not entirely business as usual yet, since the fire knocked out power lines and the high-speed network normally used to transmit data. The telescope is temporarily powered by a generator, and the information recorded each night has to take a more circuitous route to researchers around the globe. Each day, the data (roughly 80 gigabytes worth on a clear night, capturing about 150,000 celestial objects) is loaded onto an external hard drive and driven down the winding mountain road, past the recently charred mesquite and rebounding wild grasses, for processing in Tucson.
DESI owes much of the successful restart to quick actions by crews on the mountain who secured the precious equipment.
“We’ve performed tests during the restart and found little loss in performance despite the terrible conditions that the mountain experienced,” said Claire Poppett, one of DESI’s lead observers and a physicist at UC Berkeley’s Space Sciences Laboratory. “The work that the crew did to protect the instrument was phenomenal, and we wouldn’t be in the good shape we are in without it.”
Fire on the mountain
DESI is housed in the Nicholas U. Mayall 4-meter Telescope at Kitt Peak National Observatory. As the fire approached, non-essential staff evacuated. A small team stayed behind to secure the site as best they could. They rotated the telescope to face away from the oncoming smoke, powered off electronics, and covered the mirror and lenses that image galaxies billions of light-years away.
“The most important thing was the optics,” said Matthew Evatt, the mechanical engineering manager at NSF’s NOIRLab, which operates the Mayall Telescope with funding provided by DOE. “We scrounged around and found tarps and plastic left over from way earlier in DESI.”
Evatt and Bob Stupak, the electronics maintenance supervisor at NOIRLab, climbed ladders and secured the plastic sheets over the 4-meter-diameter mirror using bungee cords, ratchet straps, and electrical tape. They maneuvered the telescope and a scissor lift to access and cover DESI’s corrector barrel, which holds six glass lenses in alignment. Soon, they too evacuated, leaving behind only firefighters and two NOIRLab employees familiar with the site: Fred Wortman and Zade Arnold.
“This place is a second home to me,” said Arnold, the site’s environmental health and safety technician who grew up close to the observatory, which sits on Tohono O’odham land. Rising above the Sonoran Desert below, Kitt Peak (or Iolkam Du’ag) is considered a “Sky Island”: a remote mountaintop with a unique ecosystem, including some unexpected inhabitants, such as bears.
“It’s a little piece of paradise I go to every day, and I wanted to keep it safe,” Arnold said.
As hotshot crews cleared brush, controlled backburns, and put out spot fires, Wortman and Arnold supported their efforts, providing information on the site’s hydrant and water systems. When teams later cut power to avoid potential flare-ups, the hydrant system shut off, so the two rigged a gravity-fed water system for responders to drink and fill their trucks. “Fred and Zade’s efforts were vital,” Levi said.
In the early hours of June 17, the fire swept over the main observatory site, causing the white domes to glow red with reflected firelight.
The fire and smoke wrapped around the peak and continued north, burning a total of around 30,000 acres before being contained. At the observatory, four support buildings burned, but all the scientific equipment and telescopes survived.
Road to recovery
“The amount of work it takes to recover from something like this is always surprising,” said Stupak. “This facility is pretty much a small town up here. We’re pretty isolated. Everything from potable water to data is a huge effort by a lot of people.”
DESI collaborators took a methodical approach, starting up and quadruple-checking one system at a time. Experts looked for any smoke damage, changed out air filters, and cleaned the optical components with a special wash of carbon dioxide. They checked the 5,000 robotic positioners that rotate and lock onto galaxies, and placed the spectrographs (tools that measure the wavelength of light) under vacuum, removing all the air over several days. The last step was turning on sensitive image sensors known as CCDs, which turn light into data and operate in extreme cold. It all worked. When the monsoons finally cleared, DESI resumed cataloguing the cosmos.
In the coming months, crews will continue to repair the larger site and improve the instrument, doing additional cleaning on the optics to return them to pre-fire condition.
“It feels really great to be back on sky again,” said Poppett, who has worked on DESI for more than a decade. “The fact that the telescope and instrument is still there is all we need – and it just needs a small tune-up to be as good as before.”
DESI, including operations of the Mayall telescope, is supported by the DOE Office of Science and by the National Energy Research Scientific Computing Center, a DOE Office of Science user facility. Additional support for DESI is provided by the U.S. National Science Foundation, the Science and Technologies Facilities Council of the United Kingdom, the Gordon and Betty Moore Foundation, the Heising-Simons Foundation, the French Alternative Energies and Atomic Energy Commission (CEA), the National Council of Science and Technology of Mexico, the Ministry of Economy of Spain, and by the DESI member institutions.
Kitt Peak National Observatory (KPNO) is a program of the National Science Foundation’s NOIRLab.