O2/EE not routing email data traffic properly on 4G, issue stops at Lumens.

[Marcuspd]

Hi, I have loads of customers now who can't use email when connecting to the Internet via O2 and EE. This is across various devices from iPhones to PC's running Outlook. All work fine when using a different ISP/WiFi and has been happening since around 20th July. 

I've tried changing the connection from 4G to 3G, even though the signal is very good in almost all cases and this made a little difference. 

The email host is Giacom and they're brushing the issue off as this only affects customers connecting via O2 or EE mobile networks. All of these devices email works fine and connects to their Exchange servers when on WiFi and not O2/EE.

The Tracerts from affected devices stops at an IP belonging to a company called Lumens who O2 and EE use to route some of their traffic.

The tracerts involved are..

For Exchange 2016 use the below:

tracert -h 30 autodiscover.giacomcp.com > "tracert-Giacomcp.txt"

For Exchange 2013 use the below:

tracert -h 30 autodiscover.cloudplatform1.com > "tracert-Cloudplatform1.txt"

For Exchange 2010 use the below:

tracert -h 30 autodiscover.messageexchange.com > "tracert-Messageexchange.txt"

Who is technical enough at O2 to understand the level of this problem at Lumens? 

Cheers, Marcus

[madasaf1sh]

@Marcuspd 

This isnt customer services. but I so administer a 10K M365 environment and a 2 MDM solutions so let try and understand and if you can provide more information.. 

I take it you work for Giacom,  who is your ISP into the DC? and can you switch ISP to see if that is the issue, or do you have only one ISP 

You say loads, can you quantify that?

1. What is the IP Details (run ipconfig /all on a windows machine) of the device trying to connect over o2 - IP, Subnet, Gateway and DNS please? 

2. Can you provide the traceroutes? along with the hops or try a PathPing from Windows? 

3. I assume this is hosted Exchange?

5. When you do the connectivity tests using o2 / ee what is the response https://testconnectivity.microsoft.com/tests/o365 

6. Are your services behind a firewall? are the nat'd? do you use CA at all? 

If the traffic is stopping at Lumen aka Level 3 then your host has a problem with peering, from L3 (and lots of ISP's and websites use L3) 

[madasaf1sh]

Just to add 2010 is out of Support and Extended support ( 2021)

2013 is now in Extended Support (April 2023 for all updates) 

2016  is now in Extended Support... 

Do you get the same with Exchange 2019? or any other hosted email services> 

[MI5]

@Marcuspd 

I'll ask @O2Lisa if she can put you in touch with someone at O2 to look into this for you.

[O2Lisa]

Thanks for the tag @MI5.

@Marcuspd I’ll message you privately so I can look in to this for you. 

[Marcuspd]

Thanks for the suggestions folks, but I'm way beyond that. 

I just need to talk to someone at O2, who can contact Lumens to sort this issue out. So far, O2 has not given me a way of contacting anyone who even understands what I'm trying to say. The chat assumes I'm talking about one handset with a problem, not the entire routing of data from O2 to Giacom. Even users abroad get this issue. 

I run an IT support company who uses Giacom for Hosted Exchange. I have countless users (At least 50), across various locations, companies and devices, from Outlook on a PC to Mail on an IOS device, to Android, who have connection problems to the Giacom servers when connecting via O2 or EE, either via 3G/4G dongles or just the O2/EE service from a sim or phone.

As soon as they connect via Wireless, using a different ISP to O2/EE (Loads are OK, Virgin, BT, etc), the email works fine again. Giacom blame O2, as the issue only affects O2 and EE connections, but they have seen from various tracert results, that the journey to them stops at Lumens, who used to be Leven 3 Communications, a company based in Manchester, who route some of O2 and EE's data traffic. The month long case is here, if you can see it. Status Page for Giacom powered by StatusCast . The affected url's/server addresses are pasted below, along with some tracert commands, so O2 connected people can see what I mean . Any chance you could escalate this to whoever deals with Lumens and make them aware of the issue at their end ?They won't talk directly to me or Giacom as we're not direct customers of theirs? Thanks, Marcus

Exchange 2013:
Name: autodiscover.cloudplatform1.com
Addresses: 51.140.253.155
51.140.78.157

Exchange 2016:
Name: autodiscover.giacomcp.com
Addresses: 51.140.3.125
51.140.227.60

Exchange 2016	tracert -h 30 webmail.giacomcp.com > "tracert-Giacomcp.txt"
Exchange 2013	tracert -h 30 cas.cloudplatform1.com > "tracert-Cloudplatform1.txt"

[madasaf1sh]

@Marcuspd 

I cant even replicate what you are seeing from an o2 connection, as if I try and hit the Ex2013 CAS, then the traffic gets into KCOM's network at 195.66.224.119 from 195.66.227.169 which is Linx and into Giacom's MPLS (i assume) at 213.249.133.1 and then hops to 213.249.133.2 

I don't get any hops going into Lumens across both and Enterprise o2 sim or a Consumer Sim, not to say it doesn't happen. 

I can get in to both URLS via OWA using EE, o2 and Virgin Media sims... 

I have a feeling that KCOM has a peering issue with Lumens, do Giacom have a backup ISP, that can test the connection with.. 

Giacom need to raise it with KCOM / Other ISP, as nothing o2 can do to force the traffic for 50 people to use a different peering network, unless the companies want to pay for a private APN. 

Another suggestion, from one Email Admin to Another, get your customers off Ex2013 its a dead product and MS are killing off a lot of features that still uses.. and your customers would be better in the cloud than 2013  

What errors are presented by the clients or the Exchange Connectivity tools. 

o2

10 49 ms 47 ms 39 ms 195.66.227.169
11 59 ms 66 ms 78 ms ge-2-0-2.er10.the.ipc.kcom.com [195.66.224.119]
12 166 ms 67 ms 42 ms 213.249.133.1
13 171 ms 65 ms 55 ms 213.249.133.2
14 * * * Request timed out.    --- UDP Packet Block

For Comparison VM Gig1 

 8 25 ms 29 ms 23 ms be2350.ccr42.lon13.atlas.cogentco.com [130.117.51.137]
9 24 ms 25 ms 29 ms be2871.ccr21.lon01.atlas.cogentco.com [154.54.58.186]
10 37 ms 45 ms 52 ms 149.6.148.210
11 33 ms 49 ms 56 ms 213.249.133.2

o2 cant help you with this, and as I said Giacom need to go through there ISP's to contact Lumens and check peering, also ask them to check there edge FW's as well to see it they are dropping the IP's due to a misconfigured ACL

[madasaf1sh]

Just noticed something... 

These are Exchange hosted in Azure?  

As the AutoDiscover records all point into Azure ASN's 

What is the traffic flow, and this may turn into PS from me, so have a bacs payment ready  

[Marcuspd]

Thanks, very useful. I've passed that info on to Giacom to push them to progress this as they're too keen to pass the buck to O2. 

As for Giacom's Exchange servers being hosted in Azure, I don't think so, as frequently the Autodiscover when setting up in a client's Outlook, takes forever or times out as Outlook looks for the account on the MS 365 Cloud platform. Sometimes I have to add reg keys to get outlook to look outside of 365 to find them, such as DWORD - ‘ExcludeExplicitO365Endpoint‘ =1

I know eventually I should move them all to the MS Cloud, but with 300 mailboxes, it's a massive task as all mailbox date will need transferring via Outlook on a PC for each mailbox. 

[madasaf1sh]

@Marcuspd 

Let us know what they say, as curious to find out if KCOM is the fault (they are not the best of ISP's but it is Hull lol )

Also out of curiosity, have you done any packet analysis on the traffic to and from the host? As this will show failures at a TCP level, and will identify where the issue is.