O2/EE not routing email data traffic properly on 4G, issue stops at Lumens.

[Marcuspd]

Hi, I have loads of customers now who can't use email when connecting to the Internet via O2 and EE. This is across various devices from iPhones to PC's running Outlook. All work fine when using a different ISP/WiFi and has been happening since around 20th July. 

I've tried changing the connection from 4G to 3G, even though the signal is very good in almost all cases and this made a little difference. 

The email host is Giacom and they're brushing the issue off as this only affects customers connecting via O2 or EE mobile networks. All of these devices email works fine and connects to their Exchange servers when on WiFi and not O2/EE.

The Tracerts from affected devices stops at an IP belonging to a company called Lumens who O2 and EE use to route some of their traffic.

The tracerts involved are..

For Exchange 2016 use the below:

tracert -h 30 autodiscover.giacomcp.com > "tracert-Giacomcp.txt"

For Exchange 2013 use the below:

tracert -h 30 autodiscover.cloudplatform1.com > "tracert-Cloudplatform1.txt"

For Exchange 2010 use the below:

tracert -h 30 autodiscover.messageexchange.com > "tracert-Messageexchange.txt"

Who is technical enough at O2 to understand the level of this problem at Lumens? 

Cheers, Marcus

[Marcuspd]

I've heard nothing more back, other than the info was passed to a higher level of support. 

Their latest update is still blaming others. It's a shame resellers like me are having to try to gather knowledge from others, to pass on to them. You'd like to think they would have tried this forum at least. 

"

We have concluded further investigations of our infrastructure and end-to-end packet captures, the conclusion of both is that the issue falls outside of our platform and thus our control. In addition we have used a 3rd party to complete an independent investigation to which the conclusion was the same as ours.

As the issue only occurs with users of EE and/or O2 this indicates the fault is within their platform or something they share in common. As this is the case the further findings including the definitive packet captures have been submitted to both ISPs for them to investigate further and explain."

[Marcuspd]

It's a bit of an odd one. DNS is working and returning the correct IP. It just times out pinging when on O2, but not on my Virgin WiFi.

If I try to tracert to the IP direct, nothing much returns other than a few hops to MSN.

Pinging autodiscover.cloudplatform1.com [51.140.253.155] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 51.140.253.155:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Users\Marcus>tracert 51.140.253.155

Tracing route to 51.140.253.155 over a maximum of 30 hops

1 3 ms 4 ms 3 ms 172.20.10.1
2 * * * Request timed out.
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.

7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 69 ms 31 ms 27 ms 195.66.227.169
11 54 ms 24 ms 27 ms 195.66.224.112
12 25 ms 34 ms 31 ms ae22-0.icr02.lon22.ntwk.msn.net [104.44.238.215]
13 70 ms 27 ms * be-122-0.ibr02.lon22.ntwk.msn.net [104.44.21.97]
14 70 ms 39 ms 34 ms be-5-0.ibr02.cwl20.ntwk.msn.net [104.44.18.99]
15 67 ms 43 ms 37 ms ae122-0.icr02.cwl20.ntwk.msn.net [104.44.20.180]
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

The 2016 platform is slightly different but still not contactable via O2. Giacom say the routing is being reset every time you try.

Tracing route to webmail.giacomcp.com [46.175.54.173]
over a maximum of 30 hops:

1 4 ms 4 ms 4 ms 172.20.10.1
2 * * * Request timed out.
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 98 ms 41 ms 26 ms 195.66.227.169
11 62 ms * * linx1.orbital.net [195.66.225.93]
12 82 ms 29 ms 48 ms no-dns-yet-assigned.orbital.net [80.88.221.233]
13 72 ms 48 ms 58 ms core-mst1.quickline.co.uk [85.95.39.148]
14 44 ms 40 ms 33 ms 91.214.231.6
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

[madasaf1sh]

@Marcuspd 

None of those tracerts are routing into Lumens (L3) from cursory glance...

Can you do a packet trace with WireShark at all, as would be interesting to see what the TCP Packet Teardown is..

This needs to be done when Outlook is open and trying to connect to the server. 

[Marcuspd]

Hi, I've installed Wireshark, connected to O2, left Outlook open trying to send an email using one of the Giacom problem accounts and captured this.

I'm no expert on this, but it looks to me as if Giacom are closing the connection (46.175.55.170)

I can't seem to add a photo, so have liked to the Wireshark capture file here..

https://drive.google.com/file/d/1AFfsgC7dwJR8rDMU3_Y0Qzm3p9BKZzBZ/view?usp=sharing

and a screenshot of it here..

https://i.ibb.co/k11G6rm/Wireshark-Giacom.jpg

[madasaf1sh]

So, 

Its a Giacom owned IP, maintained by KCOM 😞

Having a look at the PCAP, I can see the following

The thing that concerns me more in the PCAP is the "SACK_PERM=1" as this is TCP requesting the lost packets and is usually seen where you get high packet loss, and is were it rerequests the missing packets. 

I have a feeling something on a CASB/CAS server is blocking the connections as 46.175.55.170 resolves to outlook.giacomcp.com as the client sends its hello, then their is a TCP handshake then it looks like GIACOM drop packets and then the local machine is sending its re- requests.. 

This is my take on things from the information given, and too many years experience, but think Giacom are passing the buck as they dont know how to fix the issue. 

[Marcuspd]

I also get the impression they don't know how to trace the fault and are hoping one of their customers can find the answer for them. From talking to them, I suspect you know far more than their support staff do.

They claim the same thing happens, but in reverse if they try it from their end. 

"Regarding the reset of the connection you've provided, this is exactly happens when we attempt to communicate out. From the Server perspective it appears the client is resetting the connection and the client see's the server doing it. This is what we are currently working on with EE+O2. All we know is somewhere in the middle there is an issue with the TLS handshake that results in the reset connection"

[madasaf1sh]

@Marcuspd 

The fact that it's the TCP handshake and not TLS, tells me Giacom are out of their depth..

I would love to know how they are getting the same going out, as the connection will be being reset on there side, from their Infrastructure, so it wont reach the client... 

I manage a 10K Email Platform, and Email Gateways, so understand Exchange and M365  

I hope they start to understand the problem...  I can always sent them my day rate to help them fix it  

[Marcuspd]

They claim 2 data centres and 2 ISP's as below. I've asked to see a log file of the reverse of the Wireshark packet capture file I sent them, showing their claim of it showing the client end reset, not their end. 

This is their latest. Surely their front end setup will be identical at each site though too, so if this is happening at both sites, it only points more towards their setup?

"As mentioned we do have another ISP. We have two data centres and each one's primary ISP is different. The issue remains the same on both.

I will pass the information along to those involved but I suspect this is something we are aware of. As I said previously, we get the same issue when we communicate out from our side as when someone communicates in, and this is when it's passed out past our ISP."

[madasaf1sh]

@Marcuspd 

The Internal networks should be stretched across both Data Centres, and the external connectivity should be provided by MPLS into each DC, which can be different ISP's which load balances the traffic across both and they should be able to switch to the redundant ISP.. 

I have noted they are being very vague, and not actually providing any evidence of the fault. 

Where are they connecting to / from withing their network, as Exchange is a Client Initiated service

If the packets are not getting out beyond a point, then either the TCP request is timing out, its a malformed request, or the Handshake is not completing, which they are responsible for...

[Marcuspd]

Yep, as expected, they wont show me the proof that their claims of the packet capture shows the reverse from their end and blames the client side. I passed on your comments via a copy and paste (not sharing this forum or post), wondering if they'd offer to pay you to do a bit of consulting got them. A month on and they still can't offer anything better than use a VPN, change mobile provider or use WiFi, but they're experts apparently, even though this issue only seems to affect Giacom customers on O2 or EE. 

"Whilst we appreciate the offer, we're not in a position to be able to provide the trace files from tests done against customer mailboxes.

As said previously, we've investigated it and we continue to do so. I am not sure what is meant by "The fact that it's the TCP handshake and not TLS, tells me Giacom are out of their depth.." However if it helps we've ran the Hosted exchange platform for over 10 years and we've had in excess off 300k mailboxes on here. Our engineers are capable of running the service.

When we have more information on this issue we will provide it to you.