O2/EE not routing email data traffic properly on 4G, issue stops at Lumens.

[Marcuspd]

Hi, I have loads of customers now who can't use email when connecting to the Internet via O2 and EE. This is across various devices from iPhones to PC's running Outlook. All work fine when using a different ISP/WiFi and has been happening since around 20th July. 

I've tried changing the connection from 4G to 3G, even though the signal is very good in almost all cases and this made a little difference. 

The email host is Giacom and they're brushing the issue off as this only affects customers connecting via O2 or EE mobile networks. All of these devices email works fine and connects to their Exchange servers when on WiFi and not O2/EE.

The Tracerts from affected devices stops at an IP belonging to a company called Lumens who O2 and EE use to route some of their traffic.

The tracerts involved are..

For Exchange 2016 use the below:

tracert -h 30 autodiscover.giacomcp.com > "tracert-Giacomcp.txt"

For Exchange 2013 use the below:

tracert -h 30 autodiscover.cloudplatform1.com > "tracert-Cloudplatform1.txt"

For Exchange 2010 use the below:

tracert -h 30 autodiscover.messageexchange.com > "tracert-Messageexchange.txt"

Who is technical enough at O2 to understand the level of this problem at Lumens? 

Cheers, Marcus

[madasaf1sh]

@Marcuspd 

They could run the test against one of your mailboxes and provide the traces, not sure why the are reluctant to provide a PCAP which shows no identifiable information, maybe the HEX and Binary Values. 

I don't think anyone is saying they are capable but seem out of their depth completely, as this is more of a network issue than an Exchange one, and they need to be reaching out to their suppliers to get it resolved.. 

One Month in, and they are no closer.. 

[Marcuspd]

Thanks for your help, but we're banging our heads against a brick wall with this lot. 

They still refuse to acknowledge they have an issue, are passing the buck and are just hoping someone magically fixes it for them. 

Here's the latest.

"

Good afternoon,

We have concluded further investigations of our infrastructure and end-to-end packet captures, the conclusion of both is that the issue falls outside of our platform and thus our control. In addition we have used a 3rd party to complete an independent investigation to which the conclusion was the same as ours.

As the issue only occurs with users of EE and/or O2 this indicates the fault is within their platform or something they share in common. As this is the case the further findings including the definitive packet captures have been submitted to both ISPs for them to investigate further and explain.

In the meantime, please see a reminder of the workarounds we currently know to work below:

·         Change ISP from EE/O2 to an alternative such as Vodafone

o   We appreciate this may not be viable for all due to contracts, etc. however this is the only known permanent solution

·         Where possible connect to a WiFi connection that is not supported by an EE/O2 internet connection

·         Use a “Global” VPN to reroute the traffic away from the affected ISPs infrastructure

o   Not all customers will have a VPN ready to go, however we have had very positive feedback on the use of free VPNs from the Android/IOS app stores

o   We have also seen that a proxy service has resolved the issue, we do have some success with a proxy from internal testing

·         For some users using the Outlook app instead of the default mail apps on handsets has resolved the issue

o   This however has not been proven in all cases and we have only been able to replicate this success on one device on O2

·         Dropping the use of 4G/5G and using “3G Only”

o   This again however is not proven in all cases and we have not been able to replicate the results internally, some of our partners however have reported success

Long-term we are still pushing EE/O2 for them to continue their investigations and own the issue, as we do this, we ask that you continue with raising/pushing your cases to EE/O2 for investigation and for a permanent resolution.

Please continue to provide us with traceroutes and PCAP logs as they will still assist with our investigations and push back.

We will continue to provide updates as we learn more. "

[Marcuspd]

Oh well, after swearing blind that the issue was out of GIacom's hands, it appears they found a "work around" and implemented in on to THEIR datacentres. We'll never know what the issues were, but at least they finally listened to people and accepted that the issue was at their end. Latest below. Thanks for the help folks. The pointers from here probably helped steer them in the right direction.

"Our infrastructure team along with a third-party have discovered a workaround that has now been tested and rolled out to all of our data centres.
Any affected users should now be able to connect to their Exchange Mailbox via both EE and O2.

We are still monitoring this and will continue to do so until we are confident all cases have been accounted for with the workaround."

[madasaf1sh]

@Marcuspd 

Apologies meant to reply yesterday, but unfortunately the day job got in the way.

Glad they have found the root cause and implemented a fix into their datacentres... 

I have a sneaky suspicion it is something to do with either an ISP or a network infrastructure fix...   

Let us know if you have any luck with their RCA and try and move away for you and your customers