RIPE 93
.4 o'clock.
.
.
RIPE 93,
Main hall
27 October 2026
MAT Working Group session.
MASSIMO CANDELA: Hello everybody. I had to run around with the bell, sorry.
So, welcome everybody to this new session of, honestly I have no idea. I knew that I shouldn't have asked Stephen to upload the slides.
We are on the right Working Group. So, multistream analysis and tools so, MAT.
Okay, so, I am Massimo Candela, I work at the global IP network NTT and chairing with me we have Stephen Strowes from Fastly, and we have a quite packed agenda but before we go there, so the MAT Working Group is a session where we present research and excerpts both from the academia and the industry and we show this to the operators and we ask them for feedback and the operators provide feedback and they guide the future of this research, this dataset tools that they are presented.
And we feel that it's important to create this connection.
Just a few reminders: When you want to ask questions, which we really encourage you to do so, please do ask questions, just stand up, go to the microphone, state your name and affiliation and do ask questions, preferably questions, not speeches, but ‑‑ and remember that you will have to rate the presentation at the end. Also, we have a mailing list for the MAT Working Group mailing list.
They reminded me just before coming that there is an extremely important message from the RIPE NCC that I would like you to have attention.
Please...
.
The voting period for the PC election has just opened. It will close Thursday, 29 October at 5 p.m. There will be a session on diversity, equity, inclusion in tech today in the main room at 6 p.m. Buses for tonight's networking event will depart from the entrance of the hotel at 8:20 and 8:30. Drop‑off will be a short walk from the venue. So this basically says you could also walk.
Return buses will operate every 30 minutes between 22:45 and 1 a.m. This was the message.
Now, let's go straight to the agenda, so it's again a packed agenda. And I am quite proud, just out of pure, let's say, content in this agenda we have all peer reviewed material, so quite good. And we can go with the first presenter, Vaishvavi.
She will present the Internet PathMTU, how big can you go. Here is the list of the talks.
Then we will have Eric presenting LightScope
Then we will have Danesh presenting the State of ECN.
After that we will be back with the closing remarks. Let's start immediately with the first presenter. Vaishnavi is a Ph.D. student. Her talk is on Internet measurement on MTU. She is also open to interesting opportunities. Keep that in mind.
VAISHNAVI RAGHAVAJOSYULA: Thank you very much. This presentation is joint work by the lovely people on the screen from Zaland university, TU V and M T I. Path MTUs, we all know what they are but I am still going to give a small introduction do it of what they are. It's the smallest MTU that can travel on a path, it's the largest MTU that can travel end‑to‑end on a path without fragmentation. So in this case it's the Orange one on the screen that you see here. But the problem is, especially in IPv6, PathMTU discovery often fails. So, you don't really know what the PathMTU is, but operators and application engineers had a work around to it. If you don't know the PathMTU you just choose the safest path. In the case of an IPv6 packet where you have a 40‑byte header the safest packet size would be 1240 bytes. But is the smallest packet size the most efficient packet size? No. But what is the most efficient yet safe packet size? We try to answer this question through our work where we understand, where we basically needed to understand how these path MTUs look in the wild on the Internet and what does this say about the existing assumption that is we have of path MTUs. And our work provides data to do so. We did this by using RIPE Atlas to measure across Internet paths and with the sums that RIPE Atlas anchors are placed closer to the Internet's core near ISPs and IXPs, we split our measurements into three regions. So we have core paths where we are between anchors to anchors, we are access networks where we are measuring from a probe to an anchor and then we are also measuring towards the web servers and we call that web paths. We did this in a traceroute based method that is PathMTU discovery independent, that is TCP MS independent and we basically manually iteratively searched the search space for the MTU MSS bit.
If there was a network hint like a timeout or a path, packet too big message, we used it. But if it didn't exist, we didn't rely on it at all.
With this, we measured 76,000 paths across the Internet and clearly 1,500 bytes prevailed over 80% of the time. And the next highest numbers we saw was 1492 bytes, but that's a very big difference.
We don't really see anything else on this graph so I am going to switch to a log scale on the Y axis. Now we have 97 distinct PMTU values that we saw out of which outliers that we call are values less than 1,200 bytes or greater than 1,500 bytes. Those were less than 0.15 percent of the paths. To these paths we analysed where the sub‑1,500s are occurring, where are the non‑standard ones occurring. And we saw that for of the access network paths they occurred very early on. This graph plots the hop at which the PathMTU defining break is happening, and for the access network paths going towards the core, they break within the first or second hop itself. But for the paths that are going back towards the assess network from the core, they happen at the penultimate destination hop, or the hop before. So ‑‑ but on the other hand, for core network paths or network paths we see that the MTU defining break happens in transit and 75 percent of the time it happens in the AS before the destination AS.
We also looked into the difference ‑‑ we saw that there was differences between the PathMTU forward ‑‑ the forward PathMTU versus the reverse MTU between two end points. We saw there was a difference. In the majority of the time that the reverse PathMTU that was higher than the forward PathMTU, this means that you are receiving higher packet size than you are sending. And one thing that I wanted to point out here is that the difference of 160 to 220 was all on IPv6 paths, which shows that the receiving side could receive 1,500 bytes but it was only sending 1208 bytes. But we also looked into why these sub‑1,500 bytes are occurring, so we did a little bit of analysis based off of publicly accessible information like network operator documentations or RIPE Atlas tags. We also looked into where these PathMTU defining breaks occur based off of the hop at which they occur, as well as the location, the AS they are occurring in, and I am going to present two funky ones here in no particular order of importance. The first one, which is 1492, which was the second biggest one we saw, which is an 8 byte difference from 1,500 bytes which likely corresponds to PPPoE. And we confirm that it did confirm with PPoE in the ASes that stated that they used it for a customer authentication, like DTAG, like one and one. And we also looked at the RIPE Atlas probes that had a PathMTU break of 1492 very early on and we saw that they had tags associated with PPoE, fiber and DSL associated with them.
The next one is 1460 bytes. We saw this most commonly in IPv4 paths over IPv6 paths, we we also saw it in all of the Google Cloud path ASes in our data. That's because Google clouds's default virtual network MTU is 1460. So, it's not that there was a tunnelling mechanism or something, it was their default MTU. And we also saw it in all of the Japanese ASes in or network, and this is because we found out that the Japanese ISPs provide access to their customers through NTTs, IPv6‑only backbone. So for other funky numbers common as well as outliers, we would like you to refer to the paper we did a great analysis on this.
The main motivation again for this work is to provide a data driven view of path MTUs for protocol engineers or operators to understand what the most efficient yet still safe packet size is, and at a glance we can clearly see that the very low conservative 1280 is not the minimum. The minimum can be higher. We see in the core paths especially that it supports up to 1,500 bytes in IPv6 never mind IPv4. But compared to the access network doesn't support so high of a PathMTU.
So, we suggest that you ‑‑ we can now shift from using conserve low packet size to maybe path specific or protocol specific packet sizing in order to optimise network transmission.
So to summarise: Through this work we provided data driven insights into PMTU values. We quantified them, we saw where they occurred and we saw 1,500 bytes dominates whilst still a 98% of the path support a minimum of 1400 petabytes. We also characterised the PathMTU bottle necks based off the locations and their causes. And we finally suggested to move beyond very conservative packet sizing into optimistic probing while using using or data driven PathMTU list as a fallback mechanism.
All right, that's it from me, now I am open to any questions.
STEPHEN STROWES: Thank you.
AUDIENCE SPEAKER: Thanks for the presentation. Very nice work. I wonder, I mean you did measurements in the traceroutes from anchors to anchors, did you before the DNS did the research work on what would be like a maximum of query response size for DNS, did you look into DNS work that they did in the past about that and what did you find?
VAISHNAVI RAGHAVAJOSYULA: There has been work previously that stated that the most efficient ‑‑ no, the most effective packet size for DNS should be 1232, if I'm not wrong, for considering EDNS. And we suggest ‑‑ I mean we looked into that, yes, but we didn't do our measurements based off of ‑‑ prior work has used DNS and using DNS queries in responses to understand packet sizes. We acknowledge that but the limitation of that work was they basically all queried from multiple vantage points to one server, and hence why we tried to more ‑‑ distribute it more, and we tried to measure from various anchors to anchors. I don't know what you want to ‑‑
AUDIENCE SPEAKER: I just wanted to know if you compared with the DNS work, and you did.
STEPHEN STROWES: If I can throw a bonus question. PMTU is notoriously hard because packet too big messages often get lost on the way back to their origin. How much data do you have on the reliability of packet too big in the data that you have collected?
VAISHNAVI RAGHAVAJOSYULA: Thank you for your interesting question.
So, we saw that only 30% of the paths supplied a PMTU for us, how of which we saw the least amount of the web and core path we saw less than 5 percent. So, the most amount of PMTU messages we could get were from the access paths, 5% to 7% from the web and core. So very few.
STEPHEN STROWES: Okay, interesting, thank you. Do we have any more questions? No.
.
(Applause)
.
Now, our next speaker is joining us remotely, Erik, he is on Meetecho, and we will be using a recording of his talk, he will be online for Q&A at the end. Erik is a PhD candidate at the University of Southern California. His research is to include LightScope.
ERIK KAPITANSKI: My name is Erik Kapitanski and this is LightScope. So, in a nutshell, LightScope takes unused ports on production servers and it turns them into honeypots or telescopes. Think it as a port based honeypot or a port based telescope system that we are deploying directly on to production machines. We have a couple of reasons for doing this. The first is that dark space which we normally need to deploy standalone honeypots or network telescopes for IPv4 is shrinking and the second is that researchers sort of hinted at the fact that attackers and scamers interact differently with our dark space monitoring tools than they do with the production machines. So LightScope gives us this visibility and LightScope actually shows us that not only do attackers and scanners interact differently with dark space monitoring tools and production networks but even individual production machines on the same production domain are targeted differently. I have different examples of that in the paper and I'll show a bit of that here.
From the research community perspective, this is why LightScope is exciting and this is what it shows.
From the adopter perspective, it's exciting because you get a lot of good threat intelligence. You have a very low risk way to gather information about how you are being targeted, what people are trying to do on your systems, things like that. That's sort of the push for adoption.
So the idea is this is not meant, again from a research community perspective, this is not meant to replace honeypots and telescopes, the dark pace monitoring tools. The idea is you distill is deploy those and LightScope will be deployed on your end use addresses. I wanted to announce that LightScope won the community contribution award this year IMC, so I a huge thank you to everyone who deployed Tor helped the project in some way. So that was very exciting.
So, as I mentioned, LightScope is a port based telescope or a port based honeypot. So in this scenario, this is your production server that has deployed the LightScope application. You'll notice you have legitimate users that are interacting with your open ports and your traffic. We're not interested in that at all. So legitimate users that are interacting with your services, this is not something that we look at for anonymisation reasons, and we don't want to, you know, interfere with anything that's happening on the production side. What we are interested in is your unused ports. So LightScope has two modes of operation. The first is this passive telescope mode. So if a scanner sends a TCP SYN to port 23 for instance, we are going to capture that TCP header, we are going to anonymise the source IP address of the server that's running LightScope and we'll collect it for analysis. This is telescope mode, this is passive.
Separately we can choose a number of ports, usually it's between 10 and 26, that will turn into honeypot ports. These will fully accept attacker connections and they will transparently tunnel that traffic to a honeypot that we have running here. So there is no honeypot running on the production machine, it's just a lightweight think of it like forwarding software. So, the resource utilisation for this is very low, it's 7% of a single core on average and less than 200 megabits of RAM. For an attacker perspective, this makes for some very interesting interactions because attackers can interact with port 80, you know, if that's one of your open services, they could see okay this is a live server, this doesn't appear to be a honeypot, but it looks like they have something open here and we'll try to connect. This is another reason we don't open all of the ports for honeypot, for instance, because we don't want it to be obvious that this is instrumented.
So, there are some limitations. In order to keep it lightweight, we're not looking at UDP or a backscatter for instance, and this did pass IRB, and it's already deployed.
So from a network operator perspective, if you deploy it this is some of the threat intelligence that you'll see. So you get a dashboard like this, and you can see this this IP for instance, 64.202 attempted on port 2222 to SSH. So they connected and you can see the user names and passwords they tried. This allows you to go through your infrastructure and make sure that none of these user name and passwords are actually in use. Could you rotate them. You could also see the commands that the attackers were attempting to run. Again thinking that they were interacting with your production machine when really they persistent. And this will allow to do further analysis on things you may want to clean up in your environment, for instance.
So, this is the optional honeypot mode. You don't have to do this if you don't want to. Some of our deployments just do passive telescope mode. That's fine, if you do this, passive telescope mode usually people will run both, honeypot and telescope, if you just do telescope, you can also see who is scanning your system. This is time, these are port numbers and you can see these actors are doing vertical scans on your machine. Certain actors are only interested in individual ports, for instance, things like this, which can still be useful for block lists, and stuff like that.
So, LightScope is very much alive and well. This is our deployment graph right around here we were deployed on NRP which was huge, we're still there now. And I'm working to, you know, hopefully this graph will go up more after this presentation.
So, we're very US and very education network heavy. So please consider deploying this on some of your end points, and even if you don't feel comfortable deploying it on a production machine, just put up a small VM and use it there, it's useful.
This leads to the natural next questions I get. Is this safe to deploy? You are asking us to put this on a production machine? So we have been running on 200 NRP nodes for eight months without any issues, and on some AWS end points we have been there for over two years as you can see here.
So the source code is all open. It's on GitHub, you can run it through your favour LLM to find any vulnerabilities. I know I have done that certainly, and if you find anything, let me know. I am very active in patching, I will update the code and I am just generally responsive. Again this is just LightScope, the application is just a lightweight. Think of it as a tunnelling application. If you feel you can donate 7% of a core maybe, less than 200 megabits of RAM and your unused ports to help research and to help threat intelligence, then that would be excellent, that's what would hope for.
Now let's see what do we see from a research perspective? We have seen some threat intelligence. Now look at from a research perspective. So this is a time series of basically a single scanner. So the X axis you have individual end points. The Y axis you have port numbers. It looks like the scanner is only scanning a single port. We feel like we know the scanner well, but it turns out the scanner has a secondary behaviour which is completely invisible to any of the dark space monitoring vantage points that we have. I looked at seven of them for instance. So even though everyone saw this scanner and they saw one behaviour, there's a targeted secondary behaviour that's completely absent from any of our existing vantage points using dark space monitoring and the interesting thing to point out, spokey for instance, is a reactive telescope so that means it's accepting TCP connections so the attacker knows that there are machines here that are reactive but it's choosing not to target them. Same thing thing on AWS, it chose not to target any of these AWS end points and then the really interesting thing is even on the same production network, so like the UCSD network for instance, there are certain end points that the scanner chose not to use this behaviour on and certain ones that it did.
So, with LightScope we have the granularity to look at individual production machines and this allows us to learn a lot more about scanners, even if they are widely visible. Just because you have seen them in an existing dataset it doesn't mean that you know everything about them. That's something that's cool about LightScope, one of our contributions.
So, in the study period for the IMC paper, we got 738,000 scanners basically that were doing TCP SYN scans of us, these were widely visible in the dark space monitoring tools. MERIT also saw 74% of these. However, the cool thing about the honeypot ports is, it allows us to ask scanners a question which is how many of you would like to become attackers. So you can see of those who sent us TCP SYNs, how many actually went on to connect, how many tried to log in and how many ran commands and the answer is 2%. About 13 thousand of the 785 thousand that were sending us SYNs actually connected and ran commands. At this point I consider them attackers. And the thing to note, these are much less visible than the initial scanners. So, like I said, merit saw 74% of our scanners but only 18 percent of our attackers, for instance.
.
The other thing that's really cool is even with 235 end points LightScope is able to contribute meaningfully to visibility. So for instance, we saw 64,000 IPs that persistent seen in any of the other dataset and usually when you do a paper like this maybe you compare against one or two, I compared against 7 different dataset with 10.8 million different vantage points and we were still able to meaningfully contribute, which again is pretty cool.
Now, the next question is okay, well what don't you see? Well this is why it's complementary to dark space monitoring tools. It turns out there is quite a lot that we still don't see. So for instance in all of the scanners that merit saw we only see 6 percent of those. All the scanners that CAIDA saw we only see 3% of those. You still need your dark space monitoring tools to give you this broad visibility. Like the giant telescopes are very useful. But LightScope helps to contribute by showing some of these exclusive highly targeted behaviour and the granularity level to look at individual production machines.
It also helps to contribute information, even when compared with commercial threat intelligence platforms. So GreyNoise for instance, of the 13,000 IPs that were, that we classify as attackers who actually ran commands. GreyNoise saw almost all of them but only classified 74% as malicious. That means either they have a different criteria for what institutes classifying someone as diminish or that some of these attackers are interacting differently with theGreyNoise standalone honeypots than they are with the LightScope production machines. Same story for abuse IP DP, only 58 percent are recommended to be on a block list, whereas, you know, LightScope saw these logging in and running commands on end points. So, they should probably be blocked, in my opinion, if you want to be conservative.
In conclusion, you should deploy LightScope. Even on a single VM you want to install on your network it would be helpful. This is a new research platform for the community, it's OpenSource, you can download the data, you can e‑mail me, I appreciate your time and now I'll switch to the live version of myself to take your questions.
MASSIMO CANDELA: Thank you very much, Erik, for the presentation. Any questions?
AUDIENCE SPEAKER: Hi, Yasper, LN labs, you briefly said you do not support backscatter. Is there any way where you could actually support backscatter or is it just too expensive?
ERIK KAPITANSKI: So the trade‑off is on production machines I want to be very respectful of your resources. So initially I did support backscatter it's just the computational requirement was kind of, I don't know, it was a bit higher, so, to answer your question, I am looking for a more efficient way to do this, and this is something that I may push out in another update, especially if it's interesting to the community. Anything that will help deployment, if you tell me I'll put this on my machine if you support backscatter, okay I'll do that.
AUDIENCE SPEAKER: Do you have any clue about what backscatter would give you, like CAIDA relies heavy on backscatter. Is that the difference between you and CAIDA?
ERIK KAPITANSKI: So my initial thought is that it wouldn't be quite as interesting as CAIDA, because you know usually what you seeing is DDOS attacks, like they happen to spoof one the IP addresses and CAIDA is like massive IP range and so you are getting traffic back. LightScope, because it's only right now on 235 endpoints, I am not sure what it would see. I guess it would depend if people are spoofing the production endpoint machines that LightScope is running on, then we would see something back, I don't know, maybe there is some interesting attacks or scan methods, it's a good question, I just don't have the answer.
AUDIENCE SPEAKER: Thank you, we will deploy one I guess.
MASSIMO CANDELA: Thank you very much. Any other questions. Thank you very much Erik.
Now our next talk.
DENESH ZEYNALI: He is going to present the state of ECN adoption in the Internet. Thanks for the introduction, today I am presenting on behalf of my collaborators here.
There is a lot of hype around the ECN in recent years, probably you have heard some discussions in the IETF in the last couple of ITFs. That was one of the main reasons we did this study, but let me give you why exactly why we want to revisit it.
First of all ECN is pretty old standard, it has been there since 2001 and there were a lot of studies on it. But what has changed since then is that we have L force is standing for low learns low loss and scalable to report. It's actually standing on the shoulders of the ECN, it's using the same technology and using the same beats in the TCP and IP header but it's not, it's a bit different technology, it has double key to all pie square queues and the underlying senders and receivers are different.
.
But L4S is not just a standard, and we say that no one is using it actually. We see that the Cloud gaming services in in the G force now is using ECN now. And also in the ecosystem of the Apple macOS and Facetime and the service that is you are using in the Apple you can enable the ECN and L4S and use it. Last but not least, Comcast. This is a snapshot of the IETF in 122, just a couple of years ago, and it showed that Comcast is migrating 2 million homes to L4S enabled CPEs and that they were playing to migrate 5 million or so. It means that they are really into using L4S for the future of their network.
It's not just this, since the last study of the L4S ‑‑ sorry, ECN in 2017,s IPv6 deployment has increased a lot, so this is another reason that we believe that it's necessary to take a look at ECN again. In addition to this, is that now it's not just about the cubic and new in the Internet. PBR is also coming into the picture and a lot of traffic after Google is on the BBR version 3 which was recently introduced is also planning to use ECN. It's not standardised yet, the RFC is not there but it's planned to be, it's planned to be, which is similar to the L4S. So without these reasons, we believe that it's very necessary to do another study on the ECN option in the Internet. But just to refresh your memory, maybe you forgot what was the ECN.
When senders are congesting the networks, for example we have a bottleneck in the middle and senders are sending to saturate the network, the queue will build up and eventually that queue will overflow and packets get lost and that's a signal for the congestion, but the change is that in here we have a threshold and the packets that are passing that threshold we put a flag on them. That flag goes to the receiver side. The receiver regards the flag back to the sender, the sender can slow down, this is actually easy. Those flags that you see are the ECN flags. But what's the benefit of this why people care about having ECN flags?
.
The point is that by informing the sender earlier before the condition reaching to the point that packets get lost, we can avoid the retransmissions, and also if you push that threshold, the threshold that is there in the queue to the very shallow queue, we can remove the queue delay in the network. So that's the solution of the L4S, that's why it's called low latency.
So this is the main reason why ECN is there in the Internet. If you don't remember, these are the bits of the ECN, it's both in IP and TCP header. In the TCP header it's mostly used for the handshake and informing the sender about the congestion. And in the IP header it's showing if a flow is supporting ECN and also flagging it as a congestion.
So, the point is in the TCP header, it's the end hosts should negotiate the ECN before having a flow that is ECN support and those two bits are for that and for the IP header the point is the routers in the route should not remove the ECN, or ECN bleaching should not happen in short.
Our goal is to assess ECN option by the end hosts and also the routers about the reaching of the ECN bits.
What do we measure? First of all, about the endpoints. We care about the servers not the clients. So this story is not about the clients at all in the endpoint support part. For the network we are checking the bleaching and the routers, not if the queue is smart queue or, we are just checking, we are just checking if the ECN bits are being erased by the routers or the media boxes. The longitudinal view is checking all the noise but the main idea is showing the trend of the ECN usage over the time.
The first are active measurements and the last one is passive measurement. Probably I won't go into the details of the last one but let's take a look at the endpoint support of the ECN which is active measurement we did.
For this we need some endpoints first. For endpoints we are taking Trust Achor owe top 1 million list and also university websites, and we just resolve the IPs using different resolvers and DNS to make sure as many IPs as we can find as many IPs as we can for the services and the servers. Also with that IPv6 hit list, we know that's not all of the IPv6 hit lists servers are actually having ‑‑ not most of them are servers, but actually we can enhance our dataset to have more IP addresses.
And second thing is of course vantage points. We are using 7 vantage points from different continents, two of them are the US and the rest of them are in the other continents. We need a scanning tool, we are using path spider. It's doing some kind of A B test checking without without ECN and then with ECN to see if it is failing, why it is failing. I actually know why it's failing, it's checking if it is failing at all.
This is the first result I am showing. In the X axis you see the number of vantage points and then Y is the number of IPs. For example, the bar that is 7 is showing that all 7 vantage points agree that those IP addresses are supporting ECN, even one is enough for us but we check with the 7 vantage points. The same for the IPv6 and you see that above 92, 93% of the IP addresses are supporting ECN but what is more important than? Supporting is not supporting ECN. Here, the the bar with the zero is showing all 7 vantage points agree that IP addresses are not supporting ECN, meaning that the handshake with ECN failed with them. And you see about 2% and 3.5 percent of the IP addresses in both IPv4 and v6 are failing to negotiates. Will
.
But, who are these people that are not supporting? It here, I am showing a number of observed IPs and the number of face of the ECN unaware of the IPs in our measurement. And each dot in this plot is showing one AS, meaning that if you go to the right side of the plot, number of IPs in that AS is increasing, so if you go down in the plot, it means it's better, they are supporting more ECN. There are some dots at the top and in the middle that are showing that these are pretty big ASes, they have a lot of IP addresses, but they are not supporting ECN in their servers.
.
Looking into more detail, some of them are very long, for example one of the ASes of Google is 100% IPs that we observed is 100 ECN unaware.
So,s it means that most of the hyperscalers are supporting ECN but there are minor ones which are some hyperscalers that are not supporting at all or partially supporting, which is more interesting.
This was about the endpoints. There are much more details in the paper as well, you can take a look. How about networks? Maybe endpoints you say it's my AS I decide to not support it. What about networks? If one AS is putting the flag there but the next AS is removing the flag, that would be a question.
So to take a look at the network support, we take the same dataset and we enhance it by taking one random IP from BGP prefixes /64 for the IPv6 and a /24 for the IPv4, and we use the same vantage point but this time we are using a scanning tool, Scamper, and this is doing ICMP to see if it is being erased somewhere in the network. This one, I am showing only IPv4 result, but the IPv6 results are also similar, you can take a look in the paper.
.
As we see, the majority of the passes, on the A axis you will see the vantage points we had, for the majority, about 90% of the passes the service bits are not touched by the routers in the middle, but for 10% of them, it was happening.
So, ECN bleaching is happening, but what is happening there is it's not just flipping one of the bits, in most of the cases the ECN is being completely erased. So if it's 01, it's going to 00 no matter what kind of signal we are using. If your flow is supporting ECN, it could be somewhere in the middle somewhere is erasing it go and that flow is not there anymore.
This was about the network support. I am not going to the longitudinal view or the passive dataset that we have, but in short, the ECN is increasing in the passive dataset that we looked at.
Endpoints are mostly supporting ECN but there are some hyperscalers that are not supporting at all.
For the network we see that 90% of the pass I was are support ECN but there are some of them that are erasing it, and when the erasing is happening it's completely erasing the bits, it's just flipping from L4S to classic or classic to L4S, It's completely removing of the bits.
And the question maybe for the people is what do you think in your network, should you support ECN? Do you believe in L4S that is coming and then using that your network should support it? And if you want to see more details about the papers, scan this QR code. Thanks.
(Applause)
STEPHEN STROWES: Do we have any questions? You showed some numbers for IPv4 and IPv6 bleaching, I am kind of interested in the transit network situation. Like do the same v6 transit networks perform bleaching as the v4 networks that you see in your data?
DENESH ZEYNALI: No, actually for the bleaching of the ECN bits in the IPv6, the percentage was less than the IPv4, meaning that probably because of the infrastructure is newer for the IPv6 and then the deployment is newer, probably configurations are correct. I can't be sure about that, but...
STEPHEN STROWES: I have a question on is that related to using the IPv6 hit list versus the IPv4 targets?
DENESH ZEYNALI: Number of targets were more, but what we made sure that it's not because of the service that is behind the IPv4 or IPv6. Yeah.
STEPHEN STROWES: Right, anyone else? Okay, thank you.
Our next speaker now.
Pascal is going to be presenting on BigTime characterising large time service providers.
PASCAL HUPPERT: My research is generally about time and time syncronisation on the Internet. This research in specific is called BigTime, because we were basically looking at big tech time servers.
Now, the first question you should have is why should you even care about time? And I agree with you that users may not immediately notice if time is incorrect, and especially when it's just a few seconds, or maybe even a minute, but there are many technical implementations that demanded on time. Your users will not notice but certificates will, and security cameras will show a time that is off, and logs and caches will be affected, key roleovers and packet mechanisms.
Now, the silver lining is that for most of these use cases, rough accuracy is sufficient. Which is good.
.
Now, how does time syncronisation work on the Internet? Well, it's pretty simple. A client goes and asks a server what's the time and the server then use the network time protocol to respond. Where does the server itself obtain the time from? It has a source, either that can be a clock, it can be satellites, global navigation satellites are a very common source of very precise time, or such a server can also obtain its time from another ever is, which can then recursively have its time source.
Now ‑‑ this sounds very simple, but of course a few things can go wrong here.
First you need to make sure that the server is even available. Then you need to make sure that the server's time is correct. Thirdly, you need to make sure that the client is receiving exactly what the server has sent. And then, somewhat optionally, you also want to minimise the delay between client and server because this will lead to more accurate time and generally also to less packet loss.
Now, how can we work towards these goals?
Well minimising a delay is mostly a question of having somewhat local servers that are close to the client and mapping the client to those exact servers. Making sure that clients receive what the server has sent is a question of mostly network time security which improves on MTP because NTP runs via UDP and is itself not secured in any way. Network time security changes this. Also RPKI and DNSSEC play a role in securing time syncronisation. And in making sure that the time server's time is correct and making sure that the server is available is, in large part, a question of redundancy.
Now, which providers to we look at? We thought about big tech, and then we found seven companies with public NTP services that are responsible for a large amount of clients.
Now, I don't have to tell you what kind of devices Apple produces, but all of the clients are synchronised with Apple time servers. Maybe I can challenge you a little bit, if you have an iPhone, try changing the time server on your phone, you will not be able to do it. If you are, please let me know. But most of the others of course allow changing time servers. But still, many devices depend on them, and we're talking about OSes, mobile devices, lots of different IoT devices and servers and other clients.
Now, the exact time services that we measured in our study were these, and you can see that there are billions, billions of devices dependent on them. And while Meta has multiple different domain names for their servers, all of the others, certainly do not serve this many devices with just a single server. So how do they do it?
.
Let me introduce to a few different ways to hide multiple different servers behind one domain name, you might know them, or maybe not, I'll just run through them pretty quickly.
Up at the top, we have the DNS. If you have one domain name, you can let it resolve to multiple different IP addresses. You can also use IP Anycast so that every IP address can then have multiple different sites and your clients are automatically routed by BGP, which we have already heard a lot about in the last two days. BGP will automatically route those clients to a nearby site if you use Anycast. And as a last layer you can also use local load balancing use a reverse proxy, what it's often called, so that at one side you have multiple servers and each client is routed to one of them based on how much traffic you are getting.
Now, how do we measure which one of these the providers use?
.
Well, the DNS measurements we performed using RIPE Atlas. Why? Because from one vantage point, you might only get a subset of the IP addresses. I told you, you want NTP servers close to the client, and the providers may use geographical DNS meaning clients in Germany, for example, are routed ‑‑ are returned different IP addresses than clients in Bulgaria.
For the Anycast replication, we looked at publicly available Anycast consensus data because those measurements exist. And for local load balancing, we hypothesised that it's not in use, but we could not devise a measurement to completely rule this out so I will not be talking about this today.
.
Now the results: Basically our providers fall into three different groups. You can see Amazon and Apple in the first group using lots of IP addresses and no Anycast. So these are completely DNS based. You can see CloudFlare, Google and Meta using few IP addresses and then lots of Anycast sites. So these are completely Anycast based. And then the third group, Microsoft and Ubuntu also do not use Anycast, and sadly they also have a very small number of IP addresses.
Now, a few things are worthy of your attention here.
You can see all of the providers have a bunch of IP addresses generally. But not all of them return multiple IP addresses in their DNS responses. And this is a problem because if you are a client and you want really reliable time, you will not just ask one time server because one time server might be incorrect. But if you ask multiple of them, you can detect which time servers are incorrect. However, if your time service provider only gives you one IP address, then you are completely dependent on one time server, and if the server's time is incorrect then yours will likely be incorrect too. Now, we see that Microsoft, while they have 12 IPv4 addresses, they do not have a single IPv6 address. I have seen a few IPv6 fans at the meeting, and I'm just as outraged as you are because this means that if you were to configure a Windows machine with IPv6‑only, it would not be able to obtain correct time, and that's a showstopper for everything basically, you cannot connect to the Internet even.
Now, we also measured the time quality of these services, and I'm not going to talk about this graph too much because it has a few caveats and you have to factor in network latency and a lot. But, one thing is striking again; you can see that during one of our measurements, these are measurements from four vantage points. During one of our measurements, Microsoft servers were a little off from the correct time. You can see on the left axis of the spider man plot that we are talking about tens of milliseconds here. What caused this for Microsoft? It was just three servers in western Europe, and they were off by the correct time by more than the network latency could explain. But we were just talking about 40 to 50 milliseconds at most. Do these off sets matter? Probably not, unless you are really dependent on extremely accurate time, then you should probably be using a different protocol than NTP. But it shows that Microsoft was subpar.
Now, we also looked at the features of the time services. Once again a few striking things here. Microsoft is the only one still offering an NTP version of 3, which is from 1992, while NTP version 4 was standardised in 2010.
All of the servers are supposed to communicate their references. So, where they obtained their time from in the NTP responses. Most of the servers do not so we're mostly in the dark about where the time is from, which is a problem if you want reliability and do not want to depend completely on GPS for example.
Network time security, I already talked about, is a very important improvement on the NTP protocol for ensuring authenticity so that your NTP packets cannot be manipulated while they are sent over the network. Will you can think of it as a little bit of time over TLS. Only CloudFlare and Ubuntu support it so far.
We also looked at RPKI in DNSSEC. Well, RPKI is mostly implemented. DNSSEC is not. So, this, as well is a little lacking.
So, to conclude: We have analysed seven different time server providers which billions of devices depend on. We found that they basically fall into three groups: Microsoft and Ubuntu only have very few servers and coming with that also, insufficient geographic distribution.
There is very little transparency about time sources. Support for NTS is not yet wide separate, just like DNSSEC. There are very simple improvements possible. I told you just sending more IP addresses back using your DNS implementation and including your references, including your time sources in the NTP response are two very easy points of improvements.
We also found that Microsoft servers sadly lags behind in more than one way.
Now you want to know more about this, here are QR codes for you to find our published paper which will be published at CEO next this year, you can also find me at the conference at any time. Any questions?
AUDIENCE SPEAKER: Jen Linkova. Yes, it's a shame that Microsoft still doesn't have v6, but you can still synchronise the time. I just successfully did it on my Mac, which has v6 only, because it will just go through CLAT. So don't scare people.
PASCAL HUPPERT: With Microsoft time servers? Oh, okay.
MASSIMO CANDELA: Let's find this out later. Okay, thank you very much for your presentation.
Now to the next talk. Katherine is online.
Katherine's presentation is mostly a star for discussions, I hope this conversation can go ahead also later in the mailing list. Thank you very much.
KATHERINE IZHIKEVICH: Thank you so much for the introduction, let's do it.
All right, so today I am presenting our work rooting out incorrect RIPE Atlas probe geolocations. So we have monitoring probes at UCSD for a very long time, and I wanted to tell a story about a probe that is located in Germany. So we have been looking at this probe since around 2019, and we saw it was reporting round‑trip times to servers in Germany that were as low as 20 milliseconds, and this makes quite a bit of sense, this is expected for infrastructure that's all located in Germany. Then the probe went offline or a little bit which happens with RIPE Atlas probes, but the strangest thing occurred right after, It started reporting round trip times to Singapore that could be as low as 30 milliseconds. And let me take you through why that doesn't quite make sense.
Germany and Singapore are separated by over 10,000 kilometers in distance, and the speed of Internet constraint tells us light will propagate through optical fibre at around 200 kilometers a millisecond. So if we do some math there that rate times the one way, the one way time, which is 30 milliseconds divided by two, would mean that the probe should be within 3,000 kilometres of Singapore, and Germany evidently is not. So, what's going on here? Well, it turns out that the operator forgot to mention that they moved to Brunei. And Brunei is of course within 3 thousand kilometres of Singapore. So what that means is that if someone had used this probe between 2022 and early 2023 when it was up they could have mistakenly thought the probe was in Germany when it was actually in Brunei. Given that information, we were curious. At scale how often does this occur? How often are RIPE Atlas probe locations impossible and why?
.
We were going to figure this out. We were going to use the built inmeasurements on the RIPE Atlas platform. It's already querying DNS route servers every few minutes, that's step one. Step two, we're going to use host name dot BIND which identifies which root instance responded. For example I got the command and it says that an instance in Singapore responded ‑‑ this is for the L roots. The third step is we're going to calculate the minimum possible round‑trip time using a manipulation of the formula I showed earlier. We're going to calculate the minimum possible round‑trip time so in our Germany Brunei example, this is 100 millisecond, then we're going to track with the measured round‑trip time was and if it's too low, like I said we saw 30 millisecond round trip times, we say this probe can't be where it says it is, therefore it's violating. You can check out the paper for the methodologies.
So, 731 probes have violated the speed of Internet for the past seven years. You can see they are distributed all over the world. And they can be pretty far off. And so we saw that 25% were between 100 and 330 kilometres away from where they said they were and we also see that about 50% are over 700 kilometres away from where they are ‑‑ where the operator said that they were.
We wanted to figure out what are potential explanations for this. One is the example that I have been showing throughout the talk. 41% of these violating probes violate the speed of Internet due to a stale location, eventually they update. 23 .5 percent will have violate the speed of Internet from day one because they were set up incorrectly with the wrong location, eventually the location is fixed. Finally there were 35 .5 percent where they just are still violating the speed of Internet. We wanted to see can we do a campaign to contact operators.
So, we worked with Robert the next speaker in the special to report probes to RIPE Atlas. 93 operators updated their location and so I have a map in the top right that shows how substantial these location updates could be. But ten of them denied our report, and this was very surprising to us. So, some of them graciously left us a comment why they were denying our report. One said I look at the configuration of my VPS and nothing changed according to the hosting service I UCITS still located in Paris. Another said "It appears the VM was migrated by is make to Toronto no it's back to" what seems like Kuwait. That begs the question neither of the operator is operating where the probe is located. It begs the question of whether we should be using VMs for RIPE Atlas probes if they are being migrated without our knowledge.
So, on that note, if you use RIPE Atlas in your research, there is one thing I want to you take away from my talk. We have made it extremely easy for you to check whether a probe that you are using might be in the wrong place. All you have to do is scan this QR code. You can download the latest list and you can look it the GitHub repository and you can manually look up probes and it will tell you whether they are violating the Internet to make your research as smooth as possible.
Thank you very much.
(Applause)
STEPHEN STROWES: Thank you. I for one is guilty as charged for having a hardware probe which has moved around and taken me a minute to update the geolocation. It's currently reasonably accurate. You may have given the answer already, but are there particular types of probes that are more likely to be incorrectly geolocated?
KATHERINE IZHIKEVICH: What a surprising question. I have a backup slide for this. So, software probes are about three times more likely to violate the speed of Internet than hardware probes, this makes sense, they are easier to move. Violating probes also tend to be younger. So the older probes tend to be in the same place. There is a couple of other examples here as well.
STEPHEN STROWES: All right. We have two questions in the mic line. I am going to take them, we have time, so...
AUDIENCE SPEAKER: This is not a question but ‑‑ Joe Abley from CloudFlare. I thought I might offer to translate from the Canadian, KW in southwest Ontorio means kitchen of Waterloo so that's probably where it migrated to Toronto from.
KATHERINE IZHIKEVICH: That makes a lot more sense.
AUDIENCE SPEAKER: Hi, Martin, from the RIPE NCC. You mentioned you tested the probes against whichever instance locations. Did you verify that all the route server instances are in the right location?
KATHERINE IZHIKEVICH: Excellent question. So, that is, that was definitely a tricky part of this. First thing that we did was that we used CAIDA Ark probes, which also could have their own issues to probe all of the DNS instances. And we found only one that seemed to be in the wrong place. And then we looked at all the results from the RIPE Atlas built in measurements and we found that there was that same one was flagging a bunch of probes, and it was the only route server instance that was flagging a bunch of probes, and that one is a root server located ‑‑ it says it's located in Conception, but it's actually probably located in Santiago. But we also spoke with DNS route operators, so Ray Balins, for example, is one of the co‑authors in the paper, and by word of mouth I guess, DNS route server operators really do believe that their servers are in the right place. And so that was a surprising result that we found one that seems to be in the wrong place.
AUDIENCE SPEAKER: Thank you.
STEPHEN STROWES: All right, very quickly.
AUDIENCE SPEAKER: Giovane, Thanks for the presentation. If I got it right you can have a this is a lower bound limit of the probes report, right. What about the rest?
KATHERINE IZHIKEVICH: That's a great question. So, there are so many things that could affect the results. So for example, we are measuring the direct path. So that is the shortest possible path. We are using speed of light. We are looking for probes that are a egregiously wrong, such that you really should not use them. But for the other ones maybe if they are a little bit wrong and are still in the same country, maybe it doesn't matter for the research community. We didn't want to make assumptions or ‑‑ we didn't want to introduce any uncertainty. With these probes the networking should not be that fast.
STEPHEN STROWES: All right. Thank you all. Thank you Katherine.
(Applause)
.
And by way of response, our next and final speaker is Robert.
ROBERT KISTELECKI: I will not speak for the previous speaker, we can continue on the mailing list. There is the SLAs side of the story. I am standing here in front of you to be the messenger again for the three measurements services, RIS, RIPEstat and SLAs that the NCC does giving you an update of what we have been doing and what we plan to do.
First, RIS.
There was a huge migration of hardware for the RIS machinery from OnPrem to rent a new hardware. So now we are in a stable situation with that again which also triggered us to do more monitoring and the metrics for to us know if the system is working. You may have seen many status updates lately that we produced when something is not correct so that you know that we have an issue and we are fixing it.
There were two talks on Monday morning, no, afternoon, sorry, one about the RIS Whois new service, it has been renewed, that talk was by Sam. If you are interested in that look up that presentation. But also, Teez presented the RIS peering strategy and possibly the way forward and asked for feedback on that one. If you do have feedback on that, talk to Teez or us in general.
Earlier on we produced, started to producing parka files, that's a different format of the data that RIS is collecting intended to be used in big data pipelines. This seems to be useful, so we are doing it continuously.
I will not talk for this, but I have the 27 plans for all of the services. Look at the RIPE NCC agenda for the activity plan for next year, this is the RIS part, so you will know what we plan to do.
RIPEstat. Again monitoring, other things improving. The team is making sure that the system is more stable than it was before. It's quite a complex system. Behind it there is a lot of databases, a lot of machinery, this is more challenging than you would city. There is a new feature in the LookingGlass, we have know exposed communities, BGP communities so you can take a look that.
Significant energy was spent on replacing the previous generation of what we call widgets that are powering RIPEstat, so this also helps us to improve security behind the site and we will carry on with this.
Again changes to the routing history. This is an area we would love to see your feedback. That piece of information that widget is more useful than it used to be, we believe tech be more useful but you should tell us what you need from it. Please do.
Something similar, upstream visibility, it's a lot faster than it used to be. There were some bottle necks. It was resolved.
'27 plans, if you are interested in the service please take a look at the activity plan.
Finally, RIPE Atlas.
RIPE Atlas has a sizable user interface, all of the pages in the last year of two have been replaced by client site code. Everything is API based now it's running in your browser. It's lovely. On top of this it's basically much faster than it used to be because we could now optimise how it works.
We lost count of how many features have been implemented but we do have updates on the UI, in the documentation look up all the details if you are interested in that one. We have quick start. We notice that had a lot of newcomers don't exactly know how to use the system, what they can do with it, so now there is a single page that can tell you how to use T guides you through creating new measurements and so on.
Rank self‑management. A bunch of you run anchors it has been pageful to re‑number or move them. It's self service. You can update those details on your page and we will make the back‑end changes to make them visible.
We have been working on use cases. These are common uses of the system where we saw that people are using it the same way and we really wanted to make the system easier to use. I will just quickly run through them and you can go to the website or to this presentation and click them to see what they are and how they work.
Domain morning.
You may know DNSMON T monitors some ccTLDs and other TLDs in the root zone is. You can go to the website and monitor our own DNS zone if you want to. You set up the measurements, you are in charge. You don't have to go and set up the measurement. The tool will do it for you. It is almost kind of a bug in plane, just click and it will do the thing.
Reachability testing or the other name, quick look. The only thing you need to do is enter your target, press the button and a MAT will come up with green dots and red dots and all of those things so you can check if the thing works or not. Multipath visualisation, also known as keep you, it's a tool where you have a target where you trace out towards from as many probes as you want. This tool will give you a nice visualisation, identify choke points, you can filter in all kinds of way if you want to, to get the gist of what the system sees.
Region meshes. You may have heard about that the IXP country... we modernised that. This is the next generation of it. The intention is to know if probes in the same country or in the same region, when they talk to each other they try to see and traceroute each other, does that path stay local or not? This is a very difficult question, but we are trying to address that, and as we improve our infrastructure geolocation, this is expected to be more useful over time. If you are a member of the RIPE NCC, you may already have probes in your network, you may not know yet that you do. You may already be measured. You may not know, but this tool will tell you. So if you log in, The system obviously knows who you are because it's the RIPE SSO, so it can tell you how many probes are in your system, where they are, what kinds of measurements are running against you. Path diversions, it's the opposite if you will, it's not how the world looks into a particular thing but how that particular probe looks outside. So you can see the path divergence, some of the things go left and some right and where it splits and joins.
Probe connectivity. There are many cases where correlated events, countries sometimes have a problem, there is a power outage and the probes behave correlated. There is a visualisation that can help you explore that. For example if you zoom in on the power outage in Portugal, space, you will see a huge dip and then as you would expect.
Anchor mesh status, they are measuring each other, sometimes there is disturbance in the force, this will let you spot those instances especially if you think that there was some kind of network suburban ance, you can zoom? On that, the tool will confirm or deny in this case.
Up thing which is not out yet, I will come out soon, is we call it visibility monitor or maybe BGP leak he detector. In some cases you want your prefix to be seen in a particular point of the world or maybe on a map, maybe geographically, but we have data on the data plane, so we can measure if your assertions are true. You can have, I imagine you will have an approved list and a deny list and we can tell you if real life actually disagrees with what you think should happen.
Moving on. API changes, we will not go into the all of of those details. Lots and lots of changes. All probes now have latched on to tomorrow anchors they are trying to be stable in the Internet and see if those he anchors are still reachable. This is where the name anchor comes from.
Software probes, anti farming measures are in two places, we renewed the manuals, the reference. There is a new tool it has been pointed out that Atlas sometimes is biased towards places we have too many probes. We try to fight that with a dissimilar all right tool. You can ask the system to give you probes which are unbiased so to speak. A feature which is I think it's unused by it's highly useful if you set up a measurement and let it run for how many years, eventually probes will die on those measurements. So there is a feature where you can just say can you please fill it in. So the system will find the best replacement it can and it will keep your measurement topped up with probes.
I will not even go into the DNS look up.
Finally on the firm wayside we released 5130, the anchors are upgraded. But the soft we need your help as well because only you can upgrade them. He encourage you to do so. It's a good time to set up upgrades as well. It's good security. Do that and the next time we will not nag you. The graph shows where we released it. You can see that clearly. This is where we released it and this is where we started nudging people who did not have ‑‑ it does have the effect, so people are reacting. Thank you for that.
.
The team is relatively small, so, we only have so much energy to support all kinds of platforms and yet we are asked to support more and more and more, so we ourselves support Debian and Raspberry OS and RedHat, We have binary releases for this, and there is really good community support that. Thank you for that for those of you who do it.
Finally, we do believe that the next step is going to be not more hardware probes that we can give you for reasons, but instead, we are working on a concept that we call bring your own device. So the idea is that if you happen to have a Raspberry lying around somewhere you can flash it with the image that we can supply you and it becomes a hardware probe. So it is kind of circumventing the problem of supply chain problems and so on.
Again '27 plans, we have plans. Will please take a look and let I say know what you think of them. It's a draft so things can change. That was it. Thank you very much.
STEPHEN STROWES: Thank you Robert.
AUDIENCE SPEAKER: Gert Döring: Just I want to say thank you on the probe refill thing. This is something I have been asking for I think for ten years, and now I have it, thank you.
ROBERT KISTELECKI: We had it a little bit before, but now I have the chance to tell you.
STEPHEN STROWES: Any further questions? All right, thank you Robert once again.
Thank you all. That's our MAT Working Group session for RIPE 93. We have had a bit of a cavalcade through ECN and NTP and everything else. We hope you have enjoyed the talks, please remember to rate the talks. There is actually genuinely useful for us. You can rant at us if you like at the MAT Working Group Chairs mailing list, you can submit your future presentations for RIPE 94, we will welcome them gladly. If there is anything you would like to take the to list, please do so. The list is generally pretty quiet but we will welcome additional stuff. Other reminder for you, that the NCC handed to us earlier, the DEI and tech session is in here at 6 p.m. and the buses for the social this evening will leave at the front of the hotel at 8:20 and 8:30 which seems oddly specific to me. But those are the rules. And that's all we have got. Thank you all. Thank you to our presenters.
(Applause)
.
Coffee break.