A succession of crashes in one of the campus’ four central e-mail servers left thousands of Yale e-mail users without e-mail access for more than 10 hours yesterday.
After the server’s hard drive failed at 9:15 a.m., the backup system started the recovery process but crashed shortly afterward. The crashes shut down 4,000 campus e-mail accounts until 8 p.m.
“It’s ironic that the very thing that is suppose to avoid problems is the thing that failed,” Information Technology Services Director Philip Long said. “It’s one of those devices that is not supposed to fail.”
The second crash left ITS officials without an easy way to recover the server’s data.
Although ITS officials were concerned that the backup system’s crash might have corrupted the e-mail data on the server, ITS workers and a repairman from Sun Microsystems fixed the backup system and managed to save all of the data. The repair took a long time because ITS officials wanted to ensure no data was lost in the process.
A microcode error in the system caused the second crash, which Long said Sun Microsystems officials told him was rare.
“The tech says he sees it once a year across the New England region, but whether it’s rare or not we got nailed,” Long said. “It probably wouldn’t happen again. On the other hand you never have guarantees.”
Long added that in past server hard drive crashes, the backup system has automatically taken over without problems.
“This unit has successfully recovered from the same kind of [hard drive crash] that caused the problem today,” Long said.
The server hard drive and the backup system are both operational now. Each of the four main e-mail servers have two backup systems. The e-mail server that crashed served mainly undergraduates, but the crash also affected other members of the Yale community.
While the problem was an internal server problem, frustrated students kept residential college computing assistants busy with complaints through out the day.
“It’s the most calls I’ve gotten in a day,” said Morse College computing assistant Jerry Moon ’03, who estimated that he had received about 15.
Berkeley College computing assistant Joseph Tadros ’02 said he also received a high number of phone calls.
“I can’t recall one that was worse than that,” Tadros said of the crash.
When e-mail access finally returned late last night, students saw the day’s lost e-mails slowly trickle into their in-boxes.
“I have 50-some messages,” Karl Gunderson ’05 said. “I just checked my e-mail at about 9:15, and it worked.”