Update 17:00
Hi guys,
The blog has just been updated. Please find the full article below:
We’ve been updating this afternoon on twitter, facebook and our communityabout the fact around 10% of our customers have been unable to make or receive calls, or use data. The good news? The cause of the fault was identified and fixed this afternoon.
The bad news? Due to high phone use during the “rush hour” early evening period customers may experience intermittent performance as full service comes back for everyone. We would like to reassure those customers still affected that we are working as hard as we can to restore normal service to everyone.
Update – 13 October 09:26
We’re still monitoring the situation, but everything is fixed. If you’re still having problems, try turning your phone on and off. We’re sorry for any inconvenience this has caused.
Update – 15 October 19:00
Everyone with a phone impacted by Friday’s network issue and knock-on disruption should now have fully restored service. We’ll be updating later this week and we’re sorry for the inconvenience.
Q & A
Q. What has gone wrong?
A. One of our network nodes which help connect and manage traffic, failed at lunchtime today. This meant that a number of our customers have had difficulty connecting to our network to make calls or use data. A fix was found by 3.30pm and since then we have been reconnecting customers systematically. We are working as quickly as we can to restore service to everyone.
Q. Why are only some people impacted?
A. The fault was on one of several nodes that help connect and manage traffic on the network, which is why only 10% of customers have been affected. To fully restore service a backlog of traffic has had to be processed following the fix.
Q. When will service be fully restored?
A. Some people had their service restored from 3.30pm this afternoon when we began systematically reconnecting affected customers to the network. We are working as quickly as we can to restore service to all our customers.
Q. Is this the same fault as the one in July?
A. No, the cause was different. However, it has had a similar impact on our customers, albeit fewer of them.
Q. Why did you not have the necessary resilience?
A. We have extensive continuity plans which we brought into effect to restore service as quickly as we could. In this instance, a fix was found in three hours and we were able to start reconnecting customers systematically from 3.30pm this afternoon. However, are having to address the backlog of traffic to fully restore service.
Q. Will you be offering compensation?
A. While the network outage on the afternoon of October 12th was highly regrettable we will not be offering compensation in this case.
We are genuinely sorry that a number of customers had problems with their service on Friday afternoon and evening. The problem is now fixed and full service has been restored. We can assure all our customers that we are continually working to improve the consistency of service they get from our network.
Q. Can you be confident this won’t happen again?
A. We operate to the highest industry standards and using leading industry infrastructure. We will continue to challenge and assess what further steps we and our infrastructure partners can take to further improve our network performance.
Best,
Leonard
Update 3:00
Hi guys,
Just another update for you:
Further to our previous update where some customers in some areas were unable to make or receive calls, send texts or use data, service has been fully restored for all remaining impacted customers.
Our engineers continue to monitor the situation through the night. Customers still experiencing issues should try to power the phone off and on. Once again we’re are sorry for any inconvenience this has caused.
That means that if you're still experiencing problems after switching off and on your device, it is most likely unrelated to this issue. Please check the status checker in this case or start a thread in the Pay Monthly and Pay & Go boards.
I would like to apologise for any inconvenience again. I can understand it's frustrating to have no network.
Best,
Leonard
Update 21:00
Hi guys,
We've just received a further update. Please find it below:
We’ve been updating this afternoon on twitter, facebook and our communityabout the fact around 10% of our customers have been unable to make or receive calls, or use data. The good news? The cause of the fault was identified and fixed this afternoon.
The bad news? Due to high phone use during the “rush hour” early evening period customers may experience intermittent performance as full service comes back for everyone. We would like to reassure those customers still affected that we are working as hard as we can to restore normal service to everyone.
Q & A
Q. What has gone wrong?
A. One of our network nodes which help connect and manage traffic, failed at lunchtime today. This meant that a number of our customers have had difficulty connecting to our network to make calls or use data. A fix was found by 3.30pm and since then we have been reconnecting customers systematically. We are working as quickly as we can to restore service to everyone.
Q. Why are only some people impacted?
A. The fault was on one of several nodes that help connect and manage traffic on the network, which is why only 10% of customers have been affected. To fully restore service a backlog of traffic has had to be processed following the fix.
Q. When will service be fully restored?
A. Some people had their service restored from 3.30pm this afternoon when we began systematically reconnecting affected customers to the network. We are working as quickly as we can to restore service to all our customers.
Q. Is this the same fault as the one in July?
A. No, the cause was different. However, it has had a similar impact on our customers, albeit fewer of them.
Q. Why did you not have the necessary resilience?
A. We have extensive continuity plans which we brought into effect to restore service as quickly as we could. In this instance, a fix was found in three hours and we were able to start reconnecting customers systematically from 3.30pm this afternoon. However, are having to address the backlog of traffic to fully restore service.
Update 20:00
Hi guys,
Just had the following update:
We’ve been updating this afternoon on twitter, facebook and Community about the fact around 10% of our customers, in some areas, have been unable to make or receive calls, or use data. The Good News? The cause of the fault was identified and fixed this afternoon.
The Bad News? Due to high phone use during the “rush hour” early evening period customers may also experience intermittent performance as full service comes back for everyone. We would like to reassure those customers still affected that we are working as hard as we can to restore normal service to everyone.
We’ll update further if there are further developments.
Update 18:50
Hi guys,
A fix has been made to the network and we're getting reports of phones coming back on the network. This is happening gradually, so if you're still experiencing difficulties, it shouldn't be taking much longer.
Many thanks for your patience
Update 16:00
Hi guys,
Around 10% of our customers are having problems when making calls or using data. Some are back up and running now and we expect a full fix this afternoon.
Again, I'm very sorry for any inconvenience this might have caused.
Update 14:00
Hi guys,
Please also use the status checker to keep up to date: http://status.o2.co.uk/
The latest update says:
"Making calls and using data
We've got some problems with our phone service right now. Some phone numbers might not be working. Those numbers are not location specific"
Original post:
Hi everyone,
We have received reports that some members are experiencing problems with their phones when trying to call or text. Data may also be affected and this is being investigated.
If you are experiencing problems, please post what you have tried to do to fix the problem (rebooting the phone, wiping your SIM card etc), even if it did not work. Please also have a look at the status checker: http://status.o2.co.uk/
Alternatively, if you have not been affected, that would be great to know too.
I shall update this thread with any news as I receive it, so please keep posted here and let us know if your situation changes.
Many thanks for your patience,
Toby
Following the network disruption that affected some customers last week (http://news.o2.co.uk/2012/10/12/o2-network-disruption-12-october), I wanted to share our plan to improve the stability of our network and rebuild confidence in O2.
First, let me say we are extremely disappointed to have let our customers down again. Two network faults in a short space of time is unsatisfactory. We took important steps with our supplier after the outage in July to prevent a similar fault happening again, and while this issue was not on the same scale, it did impact our customers. Despite industry-leading technology and major investment in our network, we recognise we need to take immediate action to address this specific problem:
1) We are removing the Central User Database provided by one of our suppliers, which has suffered two different faults in the last few months. We are not prepared to risk this happening to our customers for a third time and are implementing a proven alternative solution.
2) We will be committing an additional spend of £10 million on this change, and will also continue to invest £1.5m a day on building out and improving our network.
3) We are up-weighting and re-focussing our Service Experience Team to be solely dedicated to ensuring the highest level of customer network experience while we go through this period of unprecedented investment, culminating in the delivery of our 4G service. Their performance will now be measured on customers’ confidence in our network.
While we recognise that we have dented the confidence and trust of some of our customers, I hope this plan will demonstrate our commitment to rebuilding that trust. We will not rest until we have cemented the stability of our network and can deliver the level of service customers have come to expect of us over the last ten years.
At O2, the experience of our customers governs and determines everything we do. It is no accident that we have established ourselves as the benchmark for customer satisfaction with the fewest complaints of any network as determined by the regulator, Ofcom. We are determined to live up to that in the eyes of our customers.
As such, the continual improvement of our network and the strive for excellence in customer experience is our focus, and will be a topic that I and my colleagues will be continuing to talk about with customers over the coming weeks.