October 9-11, 2019 Campus Power Outage: Power is Now Restored

10/11/19 -- 1:55 PM

We are pleased to report that the campus has re-connected to PG&E power as of 1:00 PM today, and the campus is back online.

Research activities may resume today and special events may take place later today and this evening as planned. All parking lots are now open. 

The campus is ready for full normal weekend operations for Saturday and Sunday.  Normal weekday operations will start on Monday. 

See full announcement at Berkeley News.

10/10/19 -- 6:30PM

PG&E power to the campus continues to be off and classes are cancelled for Friday, 10/11/19.  Please continue to work off-campus unless directed by your supervisor/manager.  See the latest campus update on Berkeley News.

10/10/19 - 3:45PM

PG&E power to the campus continues to be off.  Campus buildings continue to run on limited CoGen power to sustain life-safety and minimum critical infrastructure, but all employees and students are asked to continue working offsite to avoid overloading the plant's constrained capacity.  Many thanks for everyone's patience and good will.  Please contact your building manager if you have a time-sensitive critical need to access the building.

message was sent to all employees at 3:00 pm confirming the campus' intent to resume classes and operations as soon as feasible after PG&E power is turned on.  However, there is no fixed timeline for restoration at this time.  Please continue to monitor campus WarnMe/Nixle messages and refer to Berkeley News for the latest information.

10/10/19 -- 6:30AM

Classes are cancelled Thursday Oct. 10, 2019 due to power outage.  PG&E has shut down power to the campus.  

Some buildings are running on generator power as planned, and others are on limited backup power from the Berkeley Co-Generation plant in order to sustain power for life safety, animal care, and support of critical research infrastructure. CoGen plant is not able to power the campus at full occupancy, and will turn off automatically if power usage crosses a minimum threshold.

Employees and students are asked to work offsite until full PG&E power is restored to help ensure that minimum power remains on to our critical infrastructure.

10/9/19 -- 5:45PM

PG&E continues to warn of a campus power outage starting between 8-10PM tonight through 12:00 noon on Thursday, October 10 due to forecasted high winds overnight.

Campus leadership has determined that classes will resume tomorrow UNLESS PG&E power to the campus goes out.  Please watch for Nixle alerts and monitor Berkeley News for updates tonight and Thursday morning.

If power does go out, campus employees are asked to continue to work from home similar to today's arrangements; please check with your manager.

Many thanks to Facilities Services, Building Managers, OLAC and many other faculty and staff who have worked hard today to make sure sensitive research research infrastructure/facilities were safeguarded against an outage.

10/9/19 -- 3:00PM

PG&E now estimates that power will go out on campus at 8PM.  

The VC Research Office is working through deans and building managers for status updates on all campus buildings, and coordinating with Facilities Services where needed.  Please escalate issues through your building manager and contact vcr@berkeley.edu.

10/9/10 -- 12:14PM 

Though campus still has power, PG&E is now estimating that power will go out some time this afternoon.  High winds, expected overnight, are the concern.  Another update will be posted mid-day.  

10/8/19 -- 6:20PM  

PG&E has indicated that power to campus will almost certainly be turned off at or about 8 AM tomorrow morning Oct 9, and could be off for 48 hours.  The campus has made the decision to cancel classes, and close the campus on Wednesday, October 9. Campus leadership will make a decision mid day on Wednesday regarding Thursday and possibly Friday.  A campuswide message will be coming out shortly.  Please encourage all employees to sign up for nixle alerts and to watch the Berkeley campus news page for updates today and tomorrow.

The power outage affects buildings powered by campus power.  Other buildings fed with City of Berkeley power (4th street, Tang Center, some residence halls) are expected to stay on.  

We ask you to take the following steps (this information is being updated as we have it ).  We thank you in advance for your creativity and pulling together for this sudden alert.

Closing campus offices

Managers in on-campus buildings are asked to have staff work or stay at home on Wednesday unless critical to continuity of equipment or critical services, as buildings will be without power, and therefore without wifi, HVAC, etc..   The campus has determined that all employees will be paid for the day, and most will not be required to come into work.  Please see this update from the Office of the Assistant Vice Chancellor, Human Resources, with information for managers regarding their staff.  The Berkeley News web page and forthcoming Calmessages will include additional details about accessing buildings, UCPD coverage, etc.

Public Facing Units

Please plan to close your buildings and cancel events tomorrow if you are located on the campus.  We will not be able to accommodate the public in our on-campus facilities.

Research Continuity

If your unit has critical research or research infrastructure that requires power (freezers, critical servers) please contact your building manager to confirm whether the equipment will be served on building power.  Many deans are reaching out to faculty now to explore where there are gaps, and where there may be opportunities to accept samples, etc. from other buildings that lack sufficient power.  See also animal care support below.  Harry Stark (Stanley Hall, LKS Building Manager & VCRO Facilities) is working to identify potentially available resources at Stanley and LKS.

Where equipment can be powered down, please review plans for doing so, to help preserve power for critical needs in the building.

Identify essential staff/students to manage critical infrastructure.  Develop phone trees and ensure that critical staff/students have each others’ contact information. Note that access to the building may be limited if card readers lose battery power during an extended outage (potentially over 4-5 hours).

Please see the FAQ from EH&S with additional tips and important information on what to consider before shutdown, what to do while the power is out, re-occupancy after power is back on, and other emergency planning tips.  

Following are key points provided in the fact sheet:

  • Don’t go to in labs unless necessary
  • Don’t work with hazardous materials while the ventilation is down
  • Ensure caps are safely on chemicals and fume hood sashes are closed
  • Ensure experiments are stabilized

If you have any questions or concerns, please contact EH&S by emailing ehs@berkeley.edu or calling (510) 642-3073. 

IT / High Performance Compute - Savio Cluster

All campus enterprise systems will be up and running on backup power.  However if a building does not have power, wifi will not be on in the building.

Because Savio is not on UPS power, we will need to start shutting down Savio starting at 6am given that PG&E will be shutting off power to campus at 8am.

Savio’s login nodes and its data storage are on UPS and will remain up during a power outage; however, Savio’s compute nodes and the related cooling systems are all on non-UPS power and will go down. So while users can access their data, they will not be able to run jobs. Job reservations are in place so any jobs submitted between now and 6am will not start if they cannot be completed before the 6am shutdown.

Please reach Savio staff at brc-hpc-help@berkeley.edu if you have any further questions. Users can also check the status page at http://research-it.berkeley.edu/services/high-performance-computing/status-and-announcements and the campus website http://systemstatus.berkeley.edu/ for the latest updates.

If you have additional feedback or concerns please contact Ken Lutz directly, lutz@berkeley.edu

Gary Jung, Berkeley Research Computing HPC Manager 

Ken Lutz, Interim Director, Research IT

VPN Connections

The new GlobalProtect VPN service will remain up, but the old Cisco VPN will be down.   Watch the IST website for more details and other updates.

Animal Care

OLAC Emergency Update to Investigators, 10/8/19 -- 4:40PM 

OLAC staff will report as per usual to have all hands on deck in the event we may need to relocate some animals.  There is a high potential they may be released early after all animal assessments have been completed.  Expect OLAC staff to be on duty as per evening and weekend maintenance schedules.

All bottles, carboys and water pouches will be filled and made available on Tuesday evening; All cages will be washed and prepared for use.

All animals will be accessed (including PI Maintained), cages will only be opened/changed if excessive moisture is present, water needs to be replaced, or food added

Large machines: Tunnel washers, rack washers, autoclaves will be shut down tonight and only powered up if absolutely need be during this power outage. 

Report all animal issues to “3-VETS” (510) 643-8387

RAC - Sponsored Projects/OPHS/OACU

Based on current PG&E maps and information from PG&E representatives, we anticipate that RAC offices at 4th Street will be open for business.  

PIs with proposals due in the next 3-5 days should be in touch with their pre-award RAs in Berkeley Regional Services and CGOs at SPO to determine whether proposals can be submitted during the outage.

The campus data center can continue to operate for approximately 3 days on emergency generator power. Therefore, we expect that campus enterprise research systems like Phoebe and eProtocol will be available.

PIs should store local copies of proposal documents (loss of cloud services is unlikely, but internet connectivity may be unreliable depending on your location).

This is a stressful time for all of us.  Please be kind to each other and stay tuned to Nixle and Berkeley News.

Thank you in advance!