Our apologies and explanation for the Aug 22-23 outages

posted by

Categories: Network Status, News

Comments: 0

Our apology to you!

Our mission and promise at Secom is to deliver fast and reliable services to our customers and provide exceptional, personal support when needed. They are equally important to us both as an organization and as the individuals who strive to make our mission a reality. We recently failed (quite miserably) on that first promise and for that we take full responsibility and apologize for the frustration and inconveniences we’ve caused you.

Growing a network and improving technology can be a tricky business. It takes careful planning, clear communication and precise execution. We thought we had made all the proper plans, we thought we had engaged all the right people, we thought we had made the necessary communication…and we were wrong. We’ve not yet fully uncovered all the mistakes that we made, but you can rest assured it is our top priority. At Secom we have a culture where we’re quick to own our mistakes, take accountability for them and then to plot a better way forward with all the lessons we have just learned.

What went wrong

We’ve been working over the past 9+ months on a long-term goal to make improvements and increase capacity to our transport equipment to better meet the needs of our customers both now and in the future. This is the equipment that serves as the backbone to get packets and calls from one of our communities to the next and ultimately to the internet, public phone network or elsewhere. It’s a critical element in allowing us to do what we do, and we’re very discerning when it comes to making changes because being anything less, can prove disastrous. With our well-made plan in hand, and enlisted help of professional services, we scheduled the necessary maintenance windows to make it happen. We had several successful upgrades in a row and were confident that everything was shaping up for the final, and largest upgrade to go according to plan as the others before it. Needless to say, it did not. There was not just a single failure that caused the problems, but, as in most disasters, a cascading series of failures of one critical component to another. After approximately 30 hours we were once again in full control of our network, and services have been performing as expected since. We anticipate it to remain this way, and will be holding off on any further changes we had planned.

The way forward

After all is said and done, we now have a laundry list of questions and problems for which we will need to find answers. None of them are easy, and it’s not something we will have solved in days, but rather in the weeks and months to come. We’re committed to coming up with the necessary solutions that will ensure future incidents like this can be avoided. We have a great team of folks who were instrumental in resolving the series of problems encountered during the outage and they will be presenting plans for what our future network will look like. We’ve got the full backing from our senior leadership that we will commit the necessary resources to make this a realization.

We thank you for choosing to remain a Secom customer, and we’ll do our best to win back your confidence and give you assurance that you made the right decision when you chose to become a part of the Secom family.

Sincerely,

Secom