NAVAIR Lakehurst and SPAWAR ITC New Orleans activate IT System Emergency Response Plan

Archived Body

Prudent planning and the activation of a superb inter-agency Emergency Response Plan negated any Hurricane IVAN-related interruption of NAVAIR (PMA260) Support Equipment IT System support for Navy and Marine Corps operational commands.

NAVAIR Lakehurst’s Logistics Systems Analysis Department (Code 3.6) is the designated "Continuation Of Operations" (COOP) Site supporting the Support Equipment Management System (SEMS) for all Naval Aviation Support Equipment (SE). The production site for this computer system, that determines SE inventory allowancing and monitors SE worldwide positioning, is located at SPAWAR Information Technology Center (ITC) New Orleans. On Monday afternoon 13 September 2004, at approximately 3:30 p.m., NAWC Lakehurst team members received a phone call from NAVAIR (PMA260) and learned that New Orleans may be in the direct path of Hurricane IVAN. Joint Reserve Base (JRB) New Orleans determined it was necessary and prudent to secure the base. Consequently, NAVAIR (PMA260) (Shawn Durand, Bill Fennimore, Brenda Tominack, Lynn Huett, Bill Englehart and CDR Lisa Lamarre) and SPAWAR ITC New Orleans team members decided to shut down all of the computer servers to prevent system collapse, starting at 3:00 p.m. 13 September 2004.

Receiving notification of the JRB New Orleans closure, NAVAIR (PMA260) made a further decision to activate the NAWCAD Lakehurst COOP Site. This was deemed necessary to maintain system access and SE support for Navy and Marine Corps operational commands. Although all planned testing for COOP Site activation had not yet been accomplished; the Lakehurst team rose to the occasion. This team, managed by Fran Donovan, laid out a course of action to implement the COOP Site activation. Representatives from the Logistics Systems Analysis Department (Dominic DeRoma, Heidi Klein, and Erik Merk) and the SPAWAR ITC group (Joan Sharp, Richard Marx, Jay Palmisano, Candice South and Bill Smith) worked with Fran to develop the specifics of the plan. Both sites agreed upon critical elements for accomplishment and proceeded with their individual tasks. Full backups of SEMS were initiated by SPAWAR ITC and completed at approximately 10:00 p.m. The transfer of the 700,000 item management and inventory database began, with both teams working tirelessly until completion, at approximately 2:30 a.m. on 14 September 2004. Testing runs of the database were generated to ensure error free transfer. The New Orleans servers were finally brought down after SPAWAR IT and Lakehurst personnel ensured the latest data and forms were pushed to Lakehurst for their site activation on Tuesday morning. Both teams finally said good night at 3:20 a.m. Starting at 6:30 a.m. on 14 September 2004, Jeannette Fusco and James Gorman, from the Lakehurst Team, were providing help desk support coverage. When worldwide SEMS users arrived for work, they were greeted with several e-mails giving them a new IP Address to access the functional SEMS System. The database transfer had been accomplished with virtual invisibility to fleet users and SEMS was running with seamless functionality.

The transfer of operations back to SPAWAR ITC New Orleans was accomplished at noon on Friday, 17 September. The combined team notified the fleet of the transfer of operations and took access away from the customers to “freeze” the database; New Orleans brought their servers back up and the Lakehurst COOP Site pushed the data back to them, reversing the previous effort. New Orleans then proceeded to run normal weekly and monthly database maintenance jobs over the weekend and the system was back up and fully functional on Monday morning, 20 September. Although the SPAWAR IT New Orleans facility did not sustain any direct damage from Hurricane Ivan, since they were affected by the west side of the storm vice the east; there was an emergency evacuation of all personnel for three days. The absence of SEMS functionality could have caused database inventory location errors at an average of between 5,000 and 10,000 updates per day.

The extraordinary circumstances of this event are only seconded by the superb performance of the NAVAIR/NAWC/SPAWAR Team members involved. This seamless transfer of functionality and data is without precedence and ably demonstrates NAVAIR (PMA260) long-term investment strategies in support of the goals and vision of NAVAIR's Cost Wise Readiness initiatives.

Photo Caption: Naval Air Station Pensacola, Fla. (Sept. 19, 2004) - Students assigned to the Naval Air Technical Training Center (NATTC) on board Naval Air Station Pensacola, Fla., work together to clear debris left by Hurricane Ivan. Navy officials reported that nearly 90 percent of the buildings on the base suffered significant damage. Ivan made landfall at Gulf Shores, Ala., at approximately 3:15 a.m. EST Sept. 16, with winds of 130 MPH. U.S. Navy photo by Photographer's Mate 3rd Class Todd Frantom (RELEASED)