[Resolved] Call and text difficulties issue 12/10/2012

[Toby]

Update 17/10: 13:00

Hi everyone, 

We've just had a blog post go live from our COO, Derek McManus on our plans to improve the stability of our network and rebuild confidence in O2.

---

Rebuilding your trust in our leading network

by Derek McManus 

Following the network disruption that affected some customers last week (http://news.o2.co.uk/2012/10/12/o2-network-disruption-12-october), I wanted to share our plan to improve the stability of our network and rebuild confidence in O2.

First, let me say we are extremely disappointed to have let our customers down again. Two network faults in a short space of time is unsatisfactory. We took important steps with our supplier after the outage in July to prevent a similar fault happening again, and while this issue was not on the same scale, it did impact our customers. Despite industry-leading technology and major investment in our network, we recognise we need to take immediate action to address this specific problem:

1) We are removing the Central User Database provided by one of our suppliers, which has suffered two different faults in the last few months. We are not prepared to risk this happening to our customers for a third time and are implementing a proven alternative solution.

2) We will be committing an additional spend of £10 million on this change, and will also continue to invest £1.5m a day on building out and improving our network.

3) We are up-weighting and re-focussing our Service Experience Team to be solely dedicated to ensuring the highest level of customer network experience while we go through this period of unprecedented investment, culminating in the delivery of our 4G service.  Their performance will now be measured on customers’ confidence in our network.

While we recognise that we have dented the confidence and trust of some of our customers, I hope this plan will demonstrate our commitment to rebuilding that trust. We will not rest until we have cemented the stability of our network and can deliver the level of service customers have come to expect of us over the last ten years.

At O2, the experience of our customers governs and determines everything we do. It is no accident that we have established ourselves as the benchmark for customer satisfaction with the fewest complaints of any network as determined by the regulator, Ofcom. We are determined to live up to that in the eyes of our customers.

As such, the continual improvement of our network and the strive for excellence in customer experience is our focus, and will be a topic that I and my colleagues will be continuing to talk about with customers over the coming weeks.

---

Best, 

Leonard

Please click the spoiler to see previous updates

More info

Update 17:00

Hi guys,

The blog has just been updated. Please find the full article below:

---

We’ve been updating this afternoon on twitter, facebook and our communityabout the fact around 10% of our customers have been unable to make or receive calls, or use data. The good news? The cause of the fault was identified and fixed this afternoon.

The bad news? Due to high phone use during the “rush hour” early evening period customers may experience intermittent performance as full service comes back for everyone. We would like to reassure those customers still affected that we are working as hard as we can to restore normal service to everyone.

Update – 13 October 09:26
We’re still monitoring the situation, but everything is fixed. If you’re still having problems, try turning your phone on and off. We’re sorry for any inconvenience this has caused.

Update – 15 October 19:00
Everyone with a phone impacted by Friday’s network issue and knock-on disruption should now have fully restored service. We’ll be updating later this week and we’re sorry for the inconvenience.

Q & A 

Q. What has gone wrong?
A. One of our network nodes which help connect and manage traffic, failed at lunchtime today. This meant that a number of our customers have had difficulty connecting to our network to make calls or use data. A fix was found by 3.30pm and since then we have been reconnecting customers systematically. We are working as quickly as we can to restore service to everyone.

Q. Why are only some people impacted?
A. The fault was on one of several nodes that help connect and manage traffic on the network, which is why only 10% of customers have been affected. To fully restore service a backlog of traffic has had to be processed following the fix.

Q. When will service be fully restored?
A. Some people had their service restored from 3.30pm this afternoon when we began systematically reconnecting affected customers to the network. We are working as quickly as we can to restore service to all our customers.

Q. Is this the same fault as the one in July?
A. No, the cause was different. However, it has had a similar impact on our customers, albeit fewer of them.

Q. Why did you not have the necessary resilience?
A. We have extensive continuity plans which we brought into effect to restore service as quickly as we could. In this instance, a fix was found in three hours and we were able to start reconnecting customers systematically from 3.30pm this afternoon. However, are having to address the backlog of traffic to fully restore service.

Q. Will you be offering compensation?
A. While the network outage on the afternoon of October 12th was highly regrettable we will not be offering compensation in this case.

We are genuinely sorry that a number of customers had problems with their service on Friday afternoon and evening. The problem is now fixed and full service has been restored.  We can assure all our customers that we are continually working to improve the consistency of service they get from our network.

Q. Can you be confident this won’t happen again?
A. We operate to the highest industry standards and using leading industry infrastructure. We will continue to challenge and assess what further steps we and our infrastructure partners can take to further improve our network performance.

---

Best,

Leonard

Update 3:00

Hi guys,

Just another update for you:

Further to our previous update where some customers in some areas were unable to make or receive calls, send texts or use data, service has been fully restored for all remaining impacted customers. 

Our engineers continue to monitor the situation through the night. Customers still experiencing issues should try to power the phone off and on. Once again we’re are sorry for any inconvenience this has caused.

That means that if you're still experiencing problems after switching off and on your device, it is most likely unrelated to this issue. Please check the status checker in this case or start a thread in the Pay Monthly and Pay & Go boards.

I would like to apologise for any inconvenience again. I can understand it's frustrating to have no network.

Best,

Leonard

Update 21:00

Hi guys,

We've just received a further update. Please find it below:

We’ve been updating this afternoon on twitter, facebook and our communityabout the fact around 10% of our customers have been unable to make or receive calls, or use data. The good news? The cause of the fault was identified and fixed this afternoon.

The bad news? Due to high phone use during the “rush hour” early evening period customers may experience intermittent performance as full service comes back for everyone. We would like to reassure those customers still affected that we are working as hard as we can to restore normal service to everyone.

Q & A 

Q. What has gone wrong?

A. One of our network nodes which help connect and manage traffic, failed at lunchtime today. This meant that a number of our customers have had difficulty connecting to our network to make calls or use data. A fix was found by 3.30pm and since then we have been reconnecting customers systematically. We are working as quickly as we can to restore service to everyone.

Q. Why are only some people impacted?
A. The fault was on one of several nodes that help connect and manage traffic on the network, which is why only 10% of customers have been affected. To fully restore service a backlog of traffic has had to be processed following the fix.

Q. When will service be fully restored?
A. Some people had their service restored from 3.30pm this afternoon when we began systematically reconnecting affected customers to the network. We are working as quickly as we can to restore service to all our customers.

Q. Is this the same fault as the one in July?
A. No, the cause was different. However, it has had a similar impact on our customers, albeit fewer of them.

Q. Why did you not have the necessary resilience?
A. We have extensive continuity plans which we brought into effect to restore service as quickly as we could. In this instance, a fix was found in three hours and we were able to start reconnecting customers systematically from 3.30pm this afternoon. However, are having to address the backlog of traffic to fully restore service.

Update 20:00

Hi guys,

Just had the following update:

We’ve been updating this afternoon on twitter, facebook and Community about the fact around 10% of our customers, in some areas, have been unable to make or receive calls, or use data. The Good News? The cause of the fault was identified and fixed this afternoon.

The Bad News? Due to high phone use during the “rush hour” early evening period customers may also experience intermittent performance as full service comes back for everyone. We would like to reassure those customers still affected that we are working as hard as we can to restore normal service to everyone.

We’ll update further if there are further developments.

Update 18:50

Hi guys,

A fix has been made to the network and we're getting reports of phones coming back on the network. This is happening gradually, so if you're still experiencing difficulties, it shouldn't be taking much longer.

Many thanks for your patience

Update 16:00

Hi guys,

Around 10% of our customers are having problems when making calls or using data. Some are back up and running now and we expect a full fix this afternoon. 

Again, I'm very sorry for any inconvenience this might have caused.

Update 14:00

Hi guys,

Please also use the status checker to keep up to date: http://status.o2.co.uk/

The latest update says:

"Making calls and using data

We've got some problems with our phone service right now. Some phone numbers might not be working. Those numbers are not location specific"

Original post:

Hi everyone,

We have received reports that some members are experiencing problems with their phones when trying to call or text. Data may also be affected and this is being investigated. 

If you are experiencing problems, please post what you have tried to do to fix the problem (rebooting the phone, wiping your SIM card etc), even if it did not work. Please also have a look at the status checker: http://status.o2.co.uk/

Alternatively, if you have not been affected, that would be great to know too.

I shall update this thread with any news as I receive it, so please keep posted here and let us know if your situation changes.

Many thanks for your patience,

Toby

[Liquid]

I won't lie I completely agree with your stance.


2.2 The Service isn’t fault-free; a range of different geographic, atmospheric or other conditions or circumstances beyond our control can impair it. For instance, coverage is affected by things like the thickness of the walls of the building you’re in. It might also depend on how many people near you are trying to use the Service at the same time. For more information about the things that affect coverage, please look at the dedicated Network pages of our Website. You’re entitled to the quality of service generally given by a competent mobile telecommunications service provider, using its reasonable skill and care.

It's built into the T&Cs.

Sorry

[Anonymous]

I will be in contact with the Office of Fair Trading on Monday morning.  I doubt that you have many legs to stand on, from a professional viewpoint.  I think you might find that paying customers have more rights than you imply.  But I thank you for your prompt response.

[Liquid]

I'm not an O2 rep sorry if I have that impression.

If your successful please let us know I'm always open to making a difference:)

[Anonymous]

You certainly did give that impression.  Thank you for clarifying.  Am relieved that you are not as had I received that sort of response from a real O2 employee ......

Are there any actual O2 people on here?

[jonsie]

---

@Anonymous wrote:

I will be in contact with the Office of Fair Trading on Monday morning.  I doubt that you have many legs to stand on, from a professional viewpoint.  I think you might find that paying customers have more rights than you imply.  But I thank you for your prompt response.

---

I do truly wish you every success with that.

Please do keep us posted, I would like to see the official response to the question.:smileyhappy:

[Liquid]

There's a few around:) they're rather helpful on the whole:) much better than CS.

The way my T&Cs are written are to stop people from backing out of a contract unless service can be proven to be below average. Only reason I quoted the above extract.

The way it's worded I believe would be difficult to circumvent.

Again I apologise I should try and not sound like a rep haha:(

[jonsie]

There is no official O2 participation on here. There are one or two O2 staff who post on a personal basis but I do know that with problems such as network outages, O2 do monitor the forums.

Indeed on the last outage Ronan Dunne himself responded to a couple of points.

[Anonymous]

Apology accepted - no worries.  Below average service ?  Nice one, will use that when I speak to whoever.  I hope that O2 are indeed monitoring this, it would be quite pleasant to see a response from the Company.  I will indeed report back on Monday, as I feel strongly that this is not a 'fit for purpose' contract any more.  This , aside from a phone which half way into the contract appears to be failing ( i.e. also not fit for the duration of the contract) (not the first time this has happened BTW) makes me think that there is something to work with.  If we as paying customers do not make ourselves heard outside forums, nothing will ever improve.

[Anonymous]

I found the forum was crashing on Friday as well when I was trying to look for information, in the end I got signal back sometime between 5:00 and 7:00 on Friday. My reason for posting is that I am reliant on my mobile as a way to be contacted due to elderly parents in ill health. After the network problems had developed on Friday, my father's nursing home were trying to urgently contact me. The problems encountered coincided with our landlines being down at work, so I was uncontactable. Following on from the incident in July (when I also lost mobile service), I find that I have little confidence in reliability of service.

[macblank]

Hi Carrieh888

I see u can't sleep either. LoL

I do appreciate the problem u had and time be unlucky twice ain't good.

All I can say is in all the yrs (19) I've had a mobile this is the only occasion I have known a network to have known a network to have 2 bad outages in 1 yr. However cos I don't know doesn't mean it not happened. TBH the only reason I knew of this (the 2nd) was from browsing on the community.

I think most could.put up with what happened if the communication was better and a "likely" timescale was put in place.

All I know is that NO network is fallible and since 1993, mobile networks say not to rely on a network or handset in an emergency! Being disabled myself I do rely on my mobile network to be there when I need it (and I have a few times esp for asthma attacks).

I just hope next time I need my handset in an emergency I have no problems.