Three days after the largest computer system failure in recent memory at Yale, the root cause has yet to be fully determined.
Beginning Thursday evening at 6:30 p.m., the University’s data server crashed, causing all yale.edu websites and accounts to become inaccessible for hours. Over the next day, pages such as EliApps, Central Authentication Services and Yale’s Virtual Private Network were gradually restored, with Classes*v2 — the central portal for Yale courses — the last to be confirmed as fully functional at 10 p.m. on Friday. Although students interviewed said the technical problems created difficulty, Yale’s Chief Information Officer Len Peters said Saturday that he was committed to preventing an outage of this magnitude from occurring again.
“We are going to work on a full report this week to find the root cause,” Peters said. “We will not let the crisis go to waste and take our learnings from this and incorporate this into our systems.”
Peters said that last week’s server failure was particularly notable since it was a “hard” crash, meaning there was no way to smoothly shut down the system after the power went out. He said that while some sites could be restarted, others, such as Classes*v2, were entirely disabled and as a result had to be rebuilt rather than simply migrated to the alternate server on West Campus.
Yale’s primary server is located on 300 George St., and after both backup systems in place — an uninterrupted power supply and diesel generators — failed to kick in, ITS was forced to reroute servers to an alternate location on West Campus. However, as of Saturday morning, Peters said all functions had returned to the George Street location.
When asked whether George Street would remain the primary site of server function, Peters said West Campus would continue to serve as the back up. Still, he noted ITS will continue to consider alternatives to reduce the risk of similar glitches or errors from occurring.
“We are looking to increase the diversity of power supply into the data center,” he added.
The timing of the power failure was particularly inconvenient timing on Thursday night as students and faculty found themselves in the midst of midterm season. As a result of the data failure, students and faculty have begun to reevaluate the nature of their dependency on Yale network systems.
Of 14 students interviewed, 10 said the power outage interfered with their ability to complete work and the other four did not use Yale systems that evening.
Samuel Sussman ’16 said he had a problem set due the following morning and was unable to access Classes*v2 to complete his assignment. He said he was fortunately able to receive the instructions from a friend in the class, but feared that the outcome would have been different if he were taking a smaller class or was not able to reach anyone.
Zong Xuan ’18 said he faced a similar difficulty since he was unable to access a lab report posted on the shared platform. Like Sussman, he relied upon a friend to share a copy that had been locally saved prior to the system failure.
When asked how the incident may change his technology habits in the future, Xuan said he will look to download more assignments in advance.
Christian Probst ’16 said the computer failure almost caused him to miss a deadline for an essay.
“My computer broke yesterday, so I emailed some files to Yale email,” he said. “Luckily I was able to access some files on my phone, but otherwise I would not have been able to do it.”
Still, some students were not particularly troubled by the crashed websites.
April Wen ’17 said she was not affected by the power failure since the assignment she was working on did not require access to the central server. Amelia Nierenberg ’18 similarly was unharmed.
“It honestly didn’t have an impact because I wasn’t doing homework at the time,” Nierenberg said. “This is the first time this has really happened in my Yale experience and this sort of technology failures are really forgivable.”
Some departments even sought unconventional solutions, often reaching out directly to students when email was restored the following morning.
In an email to senior history majors, the department’s undergraduate registrar Essie Lucky-Barros wrote that the deadline for depositing research plans had been extended until Monday due to the site crash.
Peters said he was aware of communications within the Yale College Dean’s Office and among faculty regarding the impact it had on students. However, he said he did not have specific knowledge of actions taken within the departments.
Still, some students hoped for better communication from Yale administration and staff during the window of time in which servers were down.
“I think it would have been nice to get information directly from [ITS],” Artem Osherov ’17 said. “If they just sent out a mass blast saying they expected for it to take a certain amount of time, people could have gone to sleep or [gotten] other work done, rather than simply waiting.”
Osherov said one lesson the computer failures taught students was the importance of completing work in advance. He added that this type of computer problem is exactly what professors warn against when they urge students to not wait until the last minute.
When asked what students should do to prepare for the potential of a similar problem, Peters offered a similar solution.
“The best thing to do … is to ensure that you have a copy of your syllabus, any assignments with due dates you have in your own personal calendars, and [my] last bit of advice is to communicate with your faculty during those times and ask for assistance,” Peters said.
A recent unplanned network outage occurred on Aug. 22, lasting from 7:07 a.m. to 11:36 a.m.