[admin]

We've had a couple periods of unexpected downtime tonight. It appears our data center is doing some un-scheduled maintenance at this time and it is not working out as they expected. Hopefully the downtime is finished.

edit: Okay, seems to be resolved (finally).

Hello,

We are currently experiencing issues upstream from our Savvis IDC. Savvis was performing un-scheduled maintainence on a backbone devices which has caused a peering issue and sever or 100% packet loss for most hosts on the Savvis network.

We are working with Savvis now to get the issue resolved as quickly as possible and will update everyone with further details as they are found.

EDIT: 1:47 AM CST

I am still waiting on a call back from Savvis. Will be a few more mins before I get an update and ETA for resolution.

EDIT: 1:49 AM CST

Savvis has responded and is in the process of restoring previous configurations on the affected devices. The devices will be reloaded and brought back online in hopefully the next 15-20 mins.

EDIT: 1:53 AM CST

Some connectivity is responding. Still not 100% resolved. Will report as I learn more.

EDIT: 1:58 AM CST

Connectivity being restored. Heavy packet loss still occurring at this time.

EDIT: 2:02 AM CST

Packet Loss is starting to subside. We are still monitoring the situation and waiting on more details from Savvis as to the cause.

EDIT: 2:08 AM CST

Packet Loss is still occuring with intermittent blips of 100% loss. Still waiting on more details from Savvis as to the cause and resolution.

EDIT: 2:20 AM CST

Savvis reports they have and ongoing issue with 3 core backbone routers in the Dallas Fort Worth (DFW / DL1) region. The routers are back online now but going through further inspections to fix any outstanding issues that are found and being rebooted also to roll back previously made changes.

Current Savvis ETA: 45mins - 1 hour

I am waiting on a call back from Savvis NOC as more details are known and will update this post with them.

EDIT: 2:39 AM CST

Packet loss appears to be at 0% now. Repeated tests of 200-300+ pings show 0% loss in the past 15 mins.

— 209.67.208.178 ping statistics —
300 packets transmitted, 300 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.507/1.692/2.008/0.083 ms

Savvis has not closed out the issue or stated it is 100% resolved so some issues may re-appear until the issue is fully resolved.

EDIT: 2:44 AM CST

Savvis has reported 1 of the 3 routers has been fully resolved and is back online. They are now making changes to the remaining 2 that are showing issues. There will be brief lapses of 100% packet loss or heavy packet loss while the last 2 are being repaired.

I am now showing packet loss again from my remote location

Thanks,

Jeremy

Edited by admin, 25 August 2007 - 01:56 AM.
update