Entire day we had issues with connectivity and many people were involved helping us to resolve everything. I really appreciate all the help we got from our engineers.
Here is what was going on in the datacenter (behind the scenes)….
After a long and stressful night, agg1 is still decided not to cooperate. We are going to be replacing the device or migrating customers off the device.
UPDATE 11:32 AM: We are going to be swapping out the ram in the Sup2 module shortly.
UPDATE 2:26 PM: Ok we have swapped out a new sup into agg1 and it “should” be good and apparently seems alot faster than before according to some customers. If there is ANYONE having ANY issues at all please notify us immediately and we will work to resolve the issue.
UPDATE 4:11 PM : We are still working to resolve the instability issues..
UPDATE 4:24 PM: Everything seems to have stabilized we will monitor it closely and continue to watch for any problems. We are not classifying this issue as resolved however.
UPDATE 5:36 PM: We are ware of the current network situation.
UPDATE 6:48 PM: While we are configuring the replacement board (2nd board) we expect the agg to fail again, but it has been failing just about once an hour with a downtime of roughly 10min between down and coming backup. We have also replaced the RAM (again) in the current board.
UPDATE 8:05 PM: The new board has been inserted into the Agg and should be coming up in a min. This should be the final downtime of the evening.
UPDATE 8:40 PM: The board isn’t taking for some reason, we are investigating why, the old board is back in and the Agg is backup. The new board is back in the lab setup and the admins are working on modifying the configs to try again.
UPDATE 9:57 PM: We are in the home stretch, the new board is back in the agg and is undergoing final configs.
UPDATE 10:38 PM: Everything should be up now, there “might” be some additional downtime throughout the night while we iron out some more bugs but non lasting longer than 10min. Also more spares JIC are on the way in for 8am delivery.