Total Online Solutions, Inc.



Article #109  Date Entered: 05-31-2008
All Windows hosting is currently offline due to a datacenter outage that started shortly after 6PM EST. The technical staff at the datacenter is working to resolve the problem and get all servers back online.

The outage is caused by a transformer in the datacenter caught fire. As per the fire department regulations, all generators had to be shut down. Once the fire is out and the fire department gives the 'ok', the generators can be started and the network will come back online while they replace the transformer and other related equipment.

Here is the latest report we received from the datacenter:
"This evening at 4:55pm CDT in our H1 data center,electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost. We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock."

8:00AM EST June 1, 2008 UPDATE
Here is the latest update from the datacenter: "The network engineering group worked overnight to prepare the network for the recovery of H1. We expect the reconvergence of the network to go smooth once H1 comes back online.
We are continuing to work through various issues this morning. We will have additional contractors on-site this morning starting at approx. 7 AM. Some will hand-off from contractors who worked overnight and others will start the recovery/installation of new electrical gear to power the data center.
We do not have an Estimated Time to Repair at present"

Update from datacenter as follows: "We expect to be able to provide initial power to parts of the H1 data center beginning at 5:00 p.m. CDT. At that time, we will begin testing and validating network and power systems, turning on air-conditioning systems and monitoring environmental conditions. We expect this testing to last approximately four hours. Following this testing, we will begin to power-on customer servers in phases. These are approximate times, and as we know more, we will keep you apprised of the situation."

2:50am EST June2 UPDATE
Update from the datacenter: "After the fire marshall inspected the H1 location, we were given the green light to bring power back to the facility. The generators have been turned on, and we are receiving power on the second floor. The generator power restoration is the first step in the full restoration of service to the data center. From here, we will begin the process of cooling the DC floor, which could take a few hours. As soon as the power integrity is confirmed and the DC floor is ready for operation, we will be restoring power and checking server hardware on a rack-by-rack basis."

5:20am EST June 2 UPDATE
Server9 is back online! This is the CWIPanel windows hosting server.

5:40am EST June 2 UPDATE
All servers are back online. Anyone who has any problems still should open a helpdesk ticket.

Return to Search

Back to Top e-commerce price conversion