UC Berkeley Launches Sky Computing Lab to Revolutionize the Cloud Industry

UC Berkeley formally launched this week a research lab aimed at building a new backbone for interconnected cloud computing, a milestone that would revolutionize the industry.

The Sky Computing Lab will establish a two-sided market mediated by services that identify and harness for users the best combination of compatible clouds for their needs. The market will catalyze innovation and competition, unleashing less costly and more secure computing power for scientists studying everything from self-driving car technology to human health.

“Today, more and more computation and more and more data is going to the cloud. There is no turning back from that,” said Ion Stoica, Databricks co-founder and the Berkeley computer science professor leading the lab. “Sky will knock out current barriers and accelerate the transition to the cloud, which will accelerate the progress across different fields.”

This lab is the latest in a series of five-year computer science labs at Berkeley focused on work from advancing basic research to software development. Efforts from these labs have resulted in influential research papers and groundbreaking start-ups like Databricks, Opaque, D2IQ, Alluxio and Ponder.

During the pandemic, cloud computing – providing services like data storage and video streaming over the internet – became increasingly important to all kinds of industries and organizations. And it’s expected to become more central to computing and society in the coming years. Sky Computing wants to shift the existing fragmented market, which can be costly and difficult for users who want to work across clouds, towards a utility market that is as connected and easy to use as the internet.

Some of the biggest names in cloud computing – including Google, IBM, Intel, Samsung SDS and VMware – are founding sponsors for the new lab. The lab’s team includes more than 60 Berkeley faculty, staff and students, including Natacha Crooks, Joseph Gonzalez, Raluca Ada Popa, Koushik Sen, Scott Shenker and Michael Jordan from the Department of Electrical Engineering and Computer Sciences. 

The Sky Computing Summer 2022 Retreat is being held May 25-27 in Incline Village, Nev., near Lake Tahoe.

Internet for the cloud

The mediator services, known as “intercloud brokers,” will be a linchpin of this new market format. Each will have a catalog holding information about the cloud’s services, compatibility, pricing and performance; a job in-take interface collecting descriptions of the user’s needs; and an optimizer generating a user-specific plan of what clouds to use when, and more.

Broker services will have tools to move or replicate data to the designated locations and to allocate resources and execute and troubleshoot the computational plan across clouds. They will also have a way to verify the identity of users and may have a billing component.

These brokers will operate on top of “compatibility sets,” which consist of services that are available across more than one cloud. In the future, Stoica expects cross-cloud agreements that allow for the free exchange of data to connect many of the disparate clouds.

“If you look at the internet, under the hood it's a collection of different networks and different technologies. But the internet ties them together. When you communicate, you do not know what different technologies and different networks are under the hood, right? You just send packets from A to B,” said Stoica. “Sky plans to do similarly for the cloud.”

This plan will expand the kinds and numbers of consumers interested in using cloud services, Stoica said. It will make it easier to use the hardware that’s right for user needs, regardless of which cloud originally held that application. Users will be able to aggregate resources across clouds to run large workloads that they couldn’t tackle on an individual cloud account. 

Users will also be able to find the most efficient and cost-effective approach for their computing task and to move their data to different cloud providers who best fit their current needs. This new technology will help companies more easily comply with data sovereignty rules in different countries and unlock new opportunities for multi-cloud security approaches to protecting their data. 

Smaller, specialized cloud providers could also more easily find customers and therefore find it easier to get off the ground and innovate. The growing numbers of new users moving to cloud computing because of the slew of new benefits and opportunities will benefit all cloud providers.

‘Nothing short of a revolution’

The Sky Computing Lab’s future is already becoming reality. With help from Berkeley artificial intelligence students, the team is testing an early version of an intercloud broker prototype for machine learning workloads. The team urged in a recent paper for other practitioners and researchers to join them in this all-encompassing effort.

“We should be clear that what we are proposing is nothing short of a revolution in an important technology ecosystem,” the team wrote in “The sky above the clouds,” a white paper recently posted on arXiv. “We don’t ask that readers accept Sky Computing as inevitable, merely as not impossible. And we invite interested readers to help us on this quest.”