Callers coming into the SPAN office were worried. People at the other end of the phone were scared. Many of the calls came from network managers who took care of a piece of SPAN at a specific NASA site, such as the Marshall Space Flight Center. Some were panicking; others spoke in a sort of monotone, flattened by a morning of calls from 25 different hysterical system administrators. A manager could lose his job over something like this.
Most of the callers to the SPAN head office were starved for information. How did this rogue worm get into their computers? Was it malicious? Would it destroy all the scientific data it came into contact with? What could be done to kill it?
NASA stored a great deal of valuable information on its SPAN computers. None of it was supposed to be classified, but the data on those computers is extremely valuable. Millions of man-hours go into gathering and analysing it. So the crisis team which had formed in the NASA SPAN project office, was alarmed when reports of massive data destruction starting coming in. People were phoning to say that the worm was erasing files.
It was every computer manager's worst nightmare, and it looked as though the crisis team's darkest fears were about to be confirmed.
Yet the worm was behaving inconsistently. On some computers it would only send anonymous messages, some of them funny, some bizarre and a few quite rude or obscene. No sooner would a user login than a message would flash across his or her screen:
Remember, even if you win the rat race—you're still a rat.
Or perhaps they were graced with some bad humour:
Nothing is faster than the speed of light…
To prove this to yourself, try opening the refrigerator door before
the light comes on.
Other users were treated to anti-authoritarian observations of the paranoid:
The FBI is watching YOU.
or
Vote anarchist.
But the worm did not appear to be erasing files on these systems. Perhaps the seemingly random file-erasing trick was a portent of things to come—just a small taste of what might happen at a particular time, such as midnight. Perhaps an unusual keystroke by an unwitting computer user on those systems which seemed only mildly affected could trigger something in the worm. One keystroke might begin an irreversible chain of commands to erase everything on that system.
The NASA SPAN computer team were in a race with the worm. Each minute they spent trying to figure out what it did, the worm was pushing forward, ever deeper into NASA's computer network. Every hour NASA spent developing a cure, the worm spent searching, probing, breaking and entering. A day's delay in getting the cure out to all the systems could mean dozens of new worm invasions doing God knows what in vulnerable computers. The SPAN team had to dissect this thing completely, and they had to do it fast.
Some computer network managers were badly shaken. The SPAN office received a call from NASA's Jet Propulsion Laboratories in California, an important NASA centre with 6500 employees and close ties to California Institute of Technology (Caltech).
JPL was pulling itself off the network.
This worm was too much of a risk. The only safe option was to isolate their computers. There would be no SPAN DEC-based communications with the rest of NASA until the crisis was under control. This made things harder for the SPAN team; getting a worm exterminating program out to JPL, like other sites which had cut their connection to SPAN, was going to be that much tougher. Everything had to be done over the phone.
Worse, JPL was one of five routing centres for NASA's SPAN computer network. It was like the centre of a wheel, with a dozen spokes branching off—each leading to another SPAN site. All these places, known as tailsites, depended on the lab site for their connections into SPAN. When JPL pulled itself off the network, the tailsites went down too.
It was a serious problem for the people in the SPAN office back in Virginia. To Ron Tencati, head of security for NASA SPAN, taking a routing centre off-line was a major issue. But his hands were tied. The SPAN office exercised central authority over the wide area network, but it couldn't dictate how individual field centres dealt with the worm. That was each centre's own decision. The SPAN team could only give them advice and rush to develop a way to poison the worm.
The SPAN office called John McMahon again, this time with a more urgent request. Would he come over to help handle the crisis?
The SPAN centre was only 800 metres away from McMahon's office. His boss, Jerome Bennett, the DECNET protocol manager, gave the nod. McMahon would be on loan until the crisis was under control.
When he got to Building 26, home of the NASA SPAN project office, McMahon became part of a core NASA crisis team including Todd Butler, Ron Tencati and Pat Sisson. Other key NASA people jumped in when needed, such as Dave Peters and Dave Stern. Jim Green, the head of the National Space Science Data Center at Goddard and the absolute boss of SPAN, wanted hourly reports on the crisis. At first the core team seemed only to include NASA people and to be largely based at Goddard. But as the day wore on, new people from other parts of the US government would join the team.
The worm had spread outside NASA.
It had also attacked the US Department of Energy's worldwide High-Energy Physics' Network of computers. Known as HEPNET, it was another piece of the overall SPAN network, along with Euro-HEPNET and Euro-SPAN. The NASA and DOE computer networks of DEC computers crisscrossed at a number of places. A research laboratory might, for example, need to have access to computers from both HEPNET and NASA SPAN. For convenience, the lab might just connect the two networks. The effect as far as the worm was concerned was that NASA's SPAN and DOE's HEPNET were in fact just one giant computer network, all of which the worm could invade.
The Department of Energy keeps classified information on its computers. Very classified information. There are two groups in DOE: the people who do research on civilian energy projects and the people who make atomic bombs. So DOE takes security seriously, as in `threat to national security' seriously. Although HEPNET wasn't meant to be carrying any classified information across its wires, DOE responded with military efficiency when its computer managers discovered the invader. They grabbed the one guy who knew a lot about computer security on VMS systems and put him on the case: Kevin Oberman.
Like McMahon, Oberman wasn't formally part of the computer security staff. He had simply become interested in computer security and was known in-house as someone who knew about VMS systems and security. Officially, his job was network manager for the engineering department at the DOE-financed Lawrence Livermore National Laboratory, or LLNL, near San Francisco.
LLNL conducted mostly military research, much of it for the Strategic Defense Initiative. Many LLNL scientists spent their days designing nuclear arms and developing beam weapons for the Star Wars program.9 DOE already had a computer security group, known as CIAC, the Computer Incident Advisory Capability. But the CIAC team tended to be experts in security issues surrounding Unix rather than VMS-based computer systems and networks. `Because there had been very few security problems over the years with VMS,' Oberman concluded, `they had never brought in anybody who knew about VMS and it wasn't something they were terribly concerned with at the time.'
The worm shattered that peaceful confidence in VMS computers. Even as the WANK worm coursed through NASA, it was launching an aggressive attack on DOE's Fermi National Accelerator Laboratory, near Chicago. It had broken into a number of computer systems there and the Fermilab people were not happy. They called in CIAC, who contacted Oberman with an early morning phone call on 16 October. They wanted him to analyse the WANK worm. They wanted to know how dangerous it was. Most of all, they wanted to know what to do about it.
The DOE people traced their first contact with the worm back to 14 October. Further, they hypothesised, the worm had actually been launched the day before, on Friday the 13th. Such an inauspicious day would, in Oberman's opinion, have been in keeping with the type of humour exhibited by the creator or creators of the worm.
Oberman began his own analysis of the worm, oblivious to the fact that 3200 kilometres away, on the other side of the continent, his colleague and acquaintance John McMahon was doing exactly the same thing.
Every time McMahon answered a phone call from an irate NASA system or network manager, he tried to get a copy of the worm from the infected machine. He also asked for the logs from their computer systems. Which computer had the worm come from? Which systems was it attacking from the infected site? In theory, the logs would allow the NASA team to map the worm's trail. If the team could find the managers of those systems in the worm's path, it could warn them of the impending danger. It could also alert the people who ran recently infected systems which had become launchpads for new worm attacks.
This wasn't always possible. If the worm had taken over a computer and was still running on it, then the manager would only be able to trace the worm backward, not forward. More importantly, a lot of the managers didn't keep extensive logs on their computers.
McMahon had always felt it was important to gather lots of information about who was connecting to a computer. In his previous job, he had modified his machines so they collected as much security information as possible about their connections to other computers.
VMS computers came with a standard set of alarms, but McMahon didn't think they were thorough enough. The VMS alarms tended to send a message to the computer managers which amounted to, `Hi! You just got a network connection from here'. The modified alarm system said, `Hi! You just got a network connection from here. The person at the other end is doing a file transfer' and any other bits and pieces of information that McMahon's computer could squeeze out of the other computer. Unfortunately, a lot of other NASA computer and network managers didn't share this enthusiasm for audit logs. Many did not keep extensive records of who had been accessing their machines and when, which made the job of chasing the worm much tougher.
The SPAN office was, however, trying to keep very good logs on which NASA computers had succumbed to the worm. Every time a NASA manager called to report a worm disturbance, one of the team members wrote down the details with paper and pen. The list, outlining the addresses of the affected computers and detailed notations of the degree of infection, would also be recorded on a computer. But handwritten lists were a good safeguard. The worm couldn't delete sheets of paper.
When McMahon learned DOE was also under attack, he began checking in with them every three hours or so. The two groups swapped lists of infected computers by telephone because voice, like the handwritten word, was a worm-free medium. `It was a kind of archaic system, but on the other hand we didn't have to depend on the network being up,' McMahon said. `We needed to have some chain of communications which was not the same as the network being attacked.'
A number of the NASA SPAN team members had developed contacts within different parts of DEC through the company's users' society, DECUS. These contacts were to prove very helpful. It was easy to get lost in the bureaucracy of DEC, which employed more than 125000 people, posted a billion-dollar profit and declared revenues in excess of $12 billion in 1989.10 Such an enormous and prestigious company would not want to face a crisis such as the WANK worm, particularly in such a publicly visible organisation like NASA. Whether or not the worm's successful expedition could be blamed on DEC's software was a moot point. Such a crisis was, well, undesirable. It just didn't look good. And it mightn't look so good either if DEC just jumped into the fray. It might look like the company was in some way at fault.
Things were different, however, if someone already had a relationship with a technical expert inside the company. It wasn't like NASA manager cold-calling a DEC guy who sold a million dollars worth of machines to someone else in the agency six months ago. It was the NASA guy calling the DEC guy he sat next to at the conference last month. It was a colleague the NASA manager chatted with now and again.
