RIPE 93
Main hall
28 October 2026
11 a.m.
Routing Working Group session.
BEN CARTWRIGHT‑COX: We are going to get started. Welcome to the Routing Working Group, or Routing Working Group, depending on if you are British or not. We have me, Sebastian, Ignas is not here today. It's a different Working Group Chair this time. We have got three talks and an election. One of the chairs is up for election. So, I am going to prepare you ahead of time and say you should open your in box and try and find the e‑mail that RIPE sent you on Monday with your Meetecho link. It is really hard to inbound find the link otherwise, so please find it now because you will need it in three times 25 minutes. And so we will run a Meetecho poll for the election. So that's really important.
SEBASTIAN BECKER: You need to have the link to vote. Otherwise your vote cannot be cast.
BEN CARTWRIGHT‑COX: We'd like you to vote.
Also, a different portal of course, please rate your talks as you go. This does require you to log into your RIPE account, which requires to you do the two factor authentication shuffle. So, do that ahead of time. Last time we had six people vote for talks, which is pretty good, but, you know, it's almost statistically relevant. So, we would like to get the N number up a bit. Rate the talks. Rate the talks as you go, if you like a talk, let us know. If you don't like a talk, also let us know.
Okay. There is something I have been told to read out. I have a message from the RIPE people.
The voting period for the Programme Committee election is open until Thursday 29th October, 17:00 That's some. You have got until tomorrow at 5 to vote for the PC election. Please vote for the PC election.
And with that, I think we can invite our first speaker.
SHADI MOTAALI: Hi everyone. Let me start with a question. If something happened on your networks, how long would at that take you to notice as soon as possible, a few seconds, a few minutes or when the customer calls you? This question shaped my Ph.D., I am Shadi Motaali, RIPE fellow, and Ph.D. student at the University in Madrid. Before my Ph.D. I worked in ISP in Iran, so I have seen many routing incidents from provider sides and I suffered a lot from customer complaints. I tried to answer these questions and tried to enhance the detection method. Today I will explain my pipelines and I will ask you how can you help me to improve it.
I think this community these days has achieved something remarkable. Almost 70% of the ROAs now are RPKI valid in the Internet routings and also try to valid AS path. The prevention nowadays is in the better place than before, but detection plays a different role I think.
It's better to detect anomaly as soon as it happens, because as soon as it happens you can make reactions, you can try to help the customers. However the roll based monitoring tools works fine, but when they know what's exactly happened, they just check your prefix. But an anomaly will happen in different ways in different shapes and much learning try to amount these unusual things, but building a machine means detector faces some challenges. When you want to train machine learning detecters for what is the normal traffic, you need to extract features because machine learning works on metadata so you need to extract feature from raw public datasets and this extraction will cause you much noise. So you should select which feature is relevant. Also when you want to train an algorithms, you need to know what is normal and what is not normal. So you need a label, and this labelling it's a bit hard because we don't have enough label dataset in BGP sites. And also when you build machine learning detecters, you want to publish it for all ASes, for all topology, you want to generalise this creation, but you cannot, it needs some requirement to enhance it. So we try to solve these challenges. You can see our pipelines here, it includes four phases, each one tries to answer one question. And each answer shaped the next phases. For our input we collected data from RIPE RIS public dataset, mainly RFC 4 and RFC 5. So we just worked on a control plane, we don't have any accessibility to any road telemetry. We have no SNMP configuration, we have no log, we don't have any traffic, it just works on control plane dataset.
Let me start with Phase 1, feature extraction.
When we collect update packets from RIPE RIS, we can extract feature from this, because as I told you, we need metadata to train machine learning algorithms. We can extract thousands of features, but most of them are noise, so we try to work and focus on the signature on each event.
At the first step we collected 48 possible statistical features from update file in RFC 4, but maybe there is some redundancy information, there is some relationship between features, so less features means faster response. Will we tried to find out those redundant features and reduce them, but not only by make assumptions, we checked with six different algorithms making votes and the feature remained just only if all the algorithms agreed on it.
Luckily, all features which remained, 25 features which remained, are the ones that we use mainly when we want to troubleshoot. I mean when something happened, usually we check AS path changes. So it remains by our steps as well, and these frameworks not only reduce the features but give you a hint for explainability.
I mean then when a alarm arises, you will know which is the cause, it will generate the explainability as well.
It's the case we reduce the feature to almost a half, 25 features, but it does it cost? As you can see here based on these figures, no, in fact it helps. The horizontal on the axis is the detection time. Further left means faster. And the vertical is F 1 score, that's related to the detection, and the higher is better. As you can see before an after, full and hybrid one, I mean 48 features and 25 features, both are arrow ups, it means less features means faster response. So for live network it matters, as soon as you can collect data, it's better to everything so fast.
So, now we know what is the features, what is the signals. But to train machine learning what is the normal traffic and what is not normal, we need a label, we need examples. We have the normal data but we don't have enough abnormal data. 5% of the traffic is anomaly. We need the balanced data, otherwise machine learning will ignore all the traffic for anomaly.
For that one we tried to generate the traffic by ourselves, not just over sampling method, not by mixing each feature and creating new ones, but my starting from scratch we generate BGP traffic from scratch by ScanAPley, it contains open message, people live notification and update file and also regenerate 3D friends, update flooding and prefix hijacking, and as far as it generates traffic, we have label on it. So the labelling process and the balanced dataset problem solves there, and one normal and simple algorithm can detect these differences between normal traffic an not normal traffic.
But the thing is really the traffic are similar to real one, and we need for criteria for creating synthetic traffic for generating synthetic traffic. All existing works means at least one of them. So we tried to generate from are scratch, but as a proof of concept we generate 46,000 update messages in 11 ASes. But there is one limitation that we don't consider the road converges. The ScanAPIey cannot capture the update fail in one peer can affect on the other peers. So the next question is if the synthetic traffic is similar to real traffic and if we train at one synthetic traffic based on one route collector can we get a anomaly on our route collector. It shapes our next phase we tried to compared in different generators, so for this step we need real data.
We collect data from RIPE RIS RFC 4, for training, and for testing we use RFC 5. We collected date data from update file, we had 25 features, and also we faced the limitation of anomaly, we tried to collect 30 documented event based on the published time stamps so we collect data and we label it as anomaly. We have balanced dataset so we tried to regenerate synthetic traffic by 13 different generators, AI based general generators, over sampling and ScanAPIey. And over sampling works fine when we train and test on one route collector. When we tried to generate synthetic traffic on RRC 4, it works fine and it can detect anomaly. Also it can detect our real traffic as well.
When we move to cross collectors, when we train on RRC 4 and we want to test in RRC 5, it fails reductions on the scores for the detection steps scores.
Here is the 13 different generators as you can see, all of them fell down in fidelity. Fidelity is how is the synthetic traffic similar to the real traffic. Is the score between zero to one hundred. And all of them fall down. You see a couple of them worked better. Why? Because it can generate features based on relationships between each other. It captures structures, not just copy and pasting, or mixing the value. It will focus topology. So it gave us a hint.
We worked on topology in our last step. Topology is the thing we face everyday because when we want to troubleshoot, we will work on our neighbours so it matters for us. Here, because we need topology, more than the update file we consider the dump file because we need the graph based features.
I use RIPE RIS for update file and down file also from RIPE RIS but for other features for graph size I use Peering DB and CAIDA dataset. But thanks to Sam from RIPE RIS, I don't know if he is here or not ‑‑ yeah, thanks to Sam, Monday I understood that we can capture everything that I want it in RIPE RIS update files so maybe next time I can capture everything from RIPE RIS dataset.
.
They mix both features, the statistical view and the graphic view in our detecters but first we checked if these two views have repetitive things or every features carries new information. Nothing was repetitive everything was new thing so We considered them and fed them to the detecters. These detecters, we have assumptions for it. Every features will mix. But we will consider the topology. So, we should know which AS collected to the other AS or not, and also the topology matters here.
I mean one AS with two peering, it will have different behaviour anomaly with one AS with 100 transit ports. So we considered topology in the detector side and if it's a sparse topology or flat topology or a star shaped topology, we will choose the flat class fires. We will choose simple classifier because just update the statistics matters for detection. But on the other hand, if there is a dense and messy AS draft, it takes a bit longer times to consider it because you need to calculate feature relationships, but it will work better and can detect anomalies better.
Here, when we detect anomaly, we don't just raise alarms, we don't fire alarms, we will give you a hint for the explanation, we will tell you what feature matters. But this pipeline. Does it work? We tested in two different events.
One of them was from Telefonica point of view. Telefonica has a dense topology so we assign a graph based class fires and can detect anomalies, but not the power outage but just focus on BGP events.
As you can see at 10:30 we have outage, but it affected the BGP at 4. And there is some false alarms after the events because it wants to be recovered.
During our development for this hypothesis that each detector should be considered a topology, we worked on two AS, one of them sparse from Iran and one Telefonica, which was dense from Spain. For the test of these assumptions, we did one blind test, we tested our pipeline. It's educational AS in Spain, we collected data from RRC 18, which is located in Barcelona, has a better view for registries, we collected data and we tried to get the features. Because we don't know the fellow of the traffic and there isn't any label, we labelled the data ourselves by five consequence based he algorithms, and then we feed it to the detector site. Because based on the structure, we should assign classifiers.
The structure of registered AS bit between simple and dense and more simpler, so a flat classifier random for X can detect anomaly and doesn't need many resource. We collect data for April 2026 and we test our detector on May 2026. So it's also a blind test because the anomaly will happen in future without knowing anything. We have some alarms and we share it with the registry guys. Here is the tool that we designed for the visualisation of this pipeline. As you can see, you can see the features at the right, and you can see the neighbour of the AS here, it's a star shaped, so it's fine that we select a flat classifier, because the peering doesn't have any relationship, so nothing will happen to them.
And also what we understand from the registries. As you can see here, we have selected six alarms which 1st and 3rd of May are something repetitive and with the same pattern. So it's not related to AS, it's related to route collector, we just share the alarms in the second of the cluster with four alarms and here you can see the features that affected those alarms and the RIS guys checked their network and they said to us, yeah, there is something happened with the first feature that we shared with them, and there is a bunch of withdrawal which arose with their networks from upstreams, maybe road converges. It doesn't affect the traffic flow but they have seen these incidents. So this can select events not only the outage.
Here, till yet we worked based on the red RIS public dataset. When we train a data, went we train the detector based on one structure, we can detect anomaly with 90% of accuracy with a similar or same structure. But if we want to fight it to a different structure which will not fit to this topology, it will be dropped to 70% of accuracy. So mainly anomaly will be lost.
It makes sense for us because one upstreams with two peers will behave different on the different situations with the other transit port with 100 peers.
.
For the next step we want to automate this choosing of steps, I mean we will collect the topology, we will generate synthetic traffic and then we choose the detector based on automation, not based on the selection and check how the AR.
We need your feedback. Maybe if you are welcome, we can do and run these blind tests on your AS. And if we see any alarms, we can share with you, you can check, you will confirm or reject it, it's fine. And also if you have any anomalies on your network with time stamp, AS an IP prefix, you can tell us so we will have a stronger dataset in the training phase and also testing phase and you can tell us which events matters matter for you. Which times of anomaly it more happens to your networks, so we will focus on those signature.
So far everything I explain is based on label dataset. But what will happen if you don't have any label? As you can see, as you understand, I passed many phases to just label the traffic. Will if you don't want to use labelling, we can use unsupervised algorithms on machine learnings, but you can see this presentation at the other session. Thank you.
(Applause)
SEBASTIAN BECKER: Thank you very much. Any questions from the room? We do have some time, actually. No one? Good. Then thank you very much.
Now, our next talk. The Anycast 101 load balancing for fun and sometimes profit from Tom Strickx from CloudFlare.
TOM STRICKX: Good morning. I am Tom Strickx. I am at CloudFlare. And it's been a while since there's been an Anycast talk at RIPE. Normally for about five to ten years, there is always been at least one Anycast talk at RIPE meetings. The same with the RPKI talks, it's mandatory you need to have at least one. I see this year we have decided to do multiple sessions on RPKI so that's awesome. I'm going to talk about not Anycast. Specifically some lessons learned over the years that we have deployed a global Anycast network.
And how we have decided to do load balancing and lessons learned and the mistakes that we made.
Very brief primary on Anycast. I hope that for the majority of the people here, I don't need to explain this. But this is mostly for, you know, my own sake, more than anything else.
So Anycast, the main goal with Anycast is you advertise a single IP address from multiple locations in the world and you terminate that across multiple places in the world. The important thing is that termination component, because otherwise it's just sparkling multihoming.
The really cool thing with Anycast, is you don't need to do anything for load balancing yourself. That's a really neat part is that everyone else is doing the hard part for you. They are doing the routing, the picking of which location do I need to go, which location ingresses in, or egresses into the CloudFlare network at which location. That's really cool but that's also really sucks because that means we don't have the control that we want.
Will but the really important thing so kind of account for here as well is that when we talk, when CloudFlare talks about the Anycast network, we're actually talking about two things all at the same time. We're talking about the Anycast network, which is the load distribution between the CloudFlare POPs. And then we have the ECMP component which is the load balancing we do within the POP and that's something that we don't really talk about all that much. Primarily because for ECMP, which stands for equal cost multipath. It's all CRC 16. Everything in the network is CRC 16. Is it Layer3 ECMP you are doing with BGP it's CRC 16. If you are doing you are doing M C LAG, it's CRC 16. It's all fine and dandy. The problem with that is after a while, things start kind of hashing towards the same thing. It's CRC 16 is not a great hashing function, It's slightly better than 4. But we're there and that's an unfortunate consequence of things.
One of the things that we decided to do was instead of having a single IP address or a single prefix that we have that we advertise internally and ECMPing that internally, we started doing something that I think everyone in this industry is now started doing, which is disaggregation. What we do is instead of advertising a single /24, we still advertise that, but we start disaggregating that internally within our POPs so that he can start advertising more prefix to say our router, because bear in mind at CloudFlare, we started in 2012, 2013, the ECMP size of a router was 32, maybe 64, if you got a very spicy router. That's not anywhere near enough for the amount of compute that we were even at that time deploying in most of our locations so the easiest way you can fix the feedback that I have deployed 200 machines and it can only receive network or receive traffic on 64 of them, is you just split that /24 into loads of /25s or /26s, specifically the computation that we did at CloudFlare was /20 to /25s, that is a lot of prefixes. In the way that that kind of works, that's similar to a splitting a /24 all the way up into/31s. That's meant that we could deploy 300, 400, 500 compute nodes all at the same time. Disaggregate that entire space up into /25s. These are documentation prefixes that's why we're doing/31s.
Split that all up, then you have this massively disaggregated prefix plane that is just forwarding traffic and everything kind of works, because at that point, even if your hashing functions aren't that great they get blasted across the compute.
That works well. The problem with that is specifically here is if you do this for a/20 as I mentioned earlier, you have 32 prefix that is you disaggregate to. Those are 32 prefix that is you can allocate to 64 ECMP buckets or even 128 ECMP buckets these days. That is a shit ton of prefixes. If you add to that the fact that CloudFlare also started doing B Y R B, we allow our customers to onboard their own prefixes , you go from 200 advertised in 2019 to over 5 thousand advertised in 2,026. Bear in mind yet again the delightful disaggregation story, if we are going to do that for the 5,000, 20,000 prefixes that a customer is onboarding. If we do that 32 level disaggregation, we're storing is millions of prefixes in our routers, and as you can expect the routers don't like that. Specifically, when you on board a single customer that on boards ten thousand to 15,000 prefixes all at once. Then your router does that. Usually when you see that kind of flat line, that's when your routing daemon is out of memory and apparently the reboot here took about two or three hours before it came back.
Not great. Right.
So, what's the thing that we started doing is,s we built a bunch of other tools and a other of other systems that allow us to load‑balance traffic a whole lot better, because at the end of day the reason we were doing the ECMP with all of these more specifics is the mere nature of I don't know what your TCP packet is going to induce in CPU load. So the only way you fix that is you get a very giant bucket and then you have five buckets of those and you throw everything in there and you hope that it balances out. That works to a point.
So, what we started building was a forwarding system or a Layer4 load balancer, we named that Unimog internally, we blogged about that internally. Realistically what it is is a EBPF XDP programme that will allow us to encapsulate packets coming coming in and then forward them to a different machine where there is more available memory, depending on what our classification of load or demand is.
Once we have built that, that also meant that we no longer actually needed to do that super harding of ECMP and getting all of those packets across as many machines as possible. So once that was deployed, with he could actually start reducing or removing those more specifics. Once we did that, you can see this here, we more than halved the amount of prefix that is we were receiving globally from all of our computers. So this is a total sum of every single BGP session we have with every single compute node summing the total received prefix count. So we went from I think it's 145 million down to about 85 million. That's still a silly amount of prefixes but that's sharded across a very large amount of POPs.
Now, the really cool thing with more specifics is, you don't want to leak the more specifics. By default or transit providers and our peers won't accept anything smaller than a /24 anyway. But we want to be the, a good citizen of the Internet and don't blast the entire internet with /24 through the entire IP space, because that would blow up routers, that would be unfortunate. So, we do prefix aggregation, and for those of you familiar with it it's just the aggregate stands on basically every single operating system. What's really cool with the aggregate stanza is it has a BGP attribute which is transitive, which is the aggregator ID and AS. That's where leak where are you comes into play. What you can see here is very cold BGP routing table output specifically because you can see prepends and we have not had prepends in seven years. What you can see here, aggregated by 1335 and then you get an IP address. That's present in basically every single route collector globally, we can get that anywhere from anything. That means that once we get alerted about a route leak, the only thing we need to is look at any route collector and we look for the aggregated by identifier because that IP address, I just look it up internally and I gives me the router that as aggregated that specific prefix which in turn basically tells me it's coming from from that location. What that tells me is that its leak is originating from that specific POP. For this, for example, that might be Brussels, this might be Hong Kong, and this might be Sydney. That gives me all of these capabilities. That means that I can now go forward and any route collector and figure out where the leak is coming from, shut down the session or shut down the Internet Exchange if it's going through the route servers or some other form of thing. And that works, right. That's really cool.
And that comes through the aggregator. So if you look at the RFC, it says, it's kind an optional transitive attribute may be included. I think basically every single routing daemon that is used on the Internet today uses it. The thing with that is, like I said, we can basically map these things to a code so we get the leak.
The problem with TLA is we stopped doing more specifics. The more specifics were blowing up my routing table, bad routers go bad. So I no longer need to do router aggregation, that means I no longer have the aggregator ID. And that's important for us to figure out does Anycast leaks. RPKI has change that had massively and improved our life wonderfully but the leaks do still happen and in those locations and cases I do want to figure out what's going on.
The way we did was that we patched BIRD. This is a BIRD.
So, we did exactly that, right. It's we added about 190 lines worth of code that allows us to set the aggregator ID and the aggregator ASN in BIRD config. Yes, we're technically not aggregating prefixes, but they are/32s in lieu back, so they are aggregated. So we patched that, that allows us to actually still do the delightful thing of no more specifics without router aggregation but we still got the aggregator ID. This is a bit of a call to action I guess for operators of route collectors and route servers is that please maintain the aggregator ID and make it somewhat exposable. It is helpful for us to figure out what's happening, who is receiving what and where.
So that's basically it. Overall overview is ECMP good? Anycast good? More specifics bad. Please keep using the aggregator ID and please keep transmitting that, so that I can make my life a whole lot easier.
That's basically it. Any questions?
SEBASTIAN BECKER: Thank you Tom. Again, please look for the email from Monday for the Meetecho link that you can cast your vote for the selection of the next routing Chair.
.
Now the next talk. On the RIPE NCC RPKI update. Tim is on the way.
TIM BRUIJNZEELS: Thank you. I'll do the mandatory RPKI talk.
I'll talk about what about we have been doing, what we're doing right now, what we'd like to get into next, I hope to focus on that mostly because I think that's the most interesting part of it.
But, you know, without further words, let's get into it. What have we been up to?
.
Lots of work on compliance, it's a recurring thing, it will come back later as well.
We have implemented the policy to remove non‑functional delegated CAs this was kind of going on on the side. Last week I sent an e‑mail to the Working Group, we have now actually implemented BGPsec router key certificates.
SOC 2 audit. Well I kind of said it just now, it's a recurring thing for us. We did the first Type 1 audit two years ago. We did a Type II audit where you have to prove opportunity. Last year, we'll do another one now, it's not completely finished but it's going to be finished really soon.
Also, because I don't know if you have experience with doing this work, but I can tell you engineers and auditors, they don't speak the same language. It's a challenge. But we now have a dedicated role for this and we hope that that really help us to improve for all parties involved and allow us to automate more and choose the right kind of controls for the auditing.
Now, this one. Policy:
Removal of non‑functional delegated CAs. Delegated CAs are RPKI CAs where somebody decided to not use our host platform but run their own softwares. And the issue is that sometimes these delegated CAs, they stop working or, you know, and this has issues, well it causes a lot of noise in RPKI validators. So a policy was made to instruct us to remove non‑functional delegated CAs that have been non‑functional for three months, which we translated to 90 days, because it's a more constant central value, and, that's not been implemented since June 9 we have started monitoring for this. In September, we removed 35 non‑functional delegated CAs. We actually, if you look very carefully, and some people did, we also have internally we still have the remnants of some CAs that you don't see in your validator because they don't have any resources. So they don't show up in RPKI validators at all. But some of that content is still in the publication as a service that we provide. So, it does cause additional da data to be downloaded by relying parties and we're still in the process of cleaning that one up.
We improved the RPKI Dashboard, Because previously, the interface for people who used delegated CAs was really simple, it was just essentially this is how you set it up we have now added a bit more on diagnostic data that we tell you we see that your most recent manifest that was valid that was found that date. And we see that you you are actively talking to us, for example.
Now, going forward, we will continue to monitor this. We have just now implemented the automation of the sending out warning e‑mails. Because the policy says we have to remove them after 90 days, but the spirit of the policy is also that we actually remove people way before that. So, we will only actively remove a CA ourselves after 90 days, but we will warn people much earlier, so, from seven days, 30 days, 60 days is what we have in mind now, we can treat these numbers depending on experience.
When people get an e‑mail, essentially they are directed to the dashboard where the diagnostic information exists and we ask them, maybe you want to fix it, or if you don't want to use your CAs, you can remove it yourself. We you don't have to wait 90 days they do it.
We just send e‑mails yesterday as a follow‑up to ‑‑ three CAs, non‑functional for seven days and ten that were non‑functional for 30 days. To be continued.
BGPsec router key certificates, well even though I was told I have time, I don't think I have time to explain exactly how BGPsec works. But in a nutshell, it allows the RPKI to be used to sign a key associated with an AS number, that then can be used in routers to sign updates and in other routers to validate updates. And what it does is, it actually ‑‑ so it helps to detect path spoofing. So if there was any lies in what the path looks like, then that will stand out.
It is complementary to origin validation, it's also complementary to ASPA, because it doesn't care whether the path was a good path. It only cares about what is the actual path.
There are some challenges with deployment of BGPsec that are outside of our control essentially, because it's mostly support in routers that still needs to happen, and it's not easy because it requires a lot of resources to do all this stuff. Yet again, on the signing side, things are not that difficult. So, from our part, we have now made this available in the API only in production, so we don't provide a UI yet, we can do that if and when there is a high demand for this. But we hope that having this will actually help people to test this more in the wild and hopefully improve all this so it becomes more deployable.
This should have been after the next slide because it's kind of... well the good news is it's done today, and there is a white lie on this slide, I'll get to it.
If you recall last year, I haven't seen Sascha around this time, maybe she is online, Sascha reported the security issue at the last RIPE meeting. There was an issue with logins and they shared cookie that allowed cross site scripting attacks. When that was reported to us we mitigated the issue promptly in the RPKI Dashboard. But, to do the second part we really needed to implement OIDC. We have done that now, it's live since this morning. The white lie here is the cookie is actually still there until the end of today, because otherwise people who were logged in before would be logged out and we want to make that transition more smooth. The good news is it's there, happy that we got this done.
My next slide, it's there, because what are we doing right now, OIDC. Just finished. API keys, and more audits.
API keys:
.
As you may know, we have an API, and the current setup uses essentially a shared secret with no expiry, it is set up in the LIR portal, so it's available to LIRs. Will it is not available to end users, so people with PI or legacy space and a sponsoring LIR, they don't get to use the API. We want to migrate this to a new architecture associated with SSO accounts, similarly to have things on the database, we want to have a one year lifetime limit on the keys, which is a due diligence requirement that we have.
People can renew these keys obviously, and we're also looking into warning people before their keys are, you know would expire.
This will also, as an additional benefit, it will allow us to support this for end users as well. So, they can then finally use the API as well.
ISO, we do both SOC 2 and ISO for RPKI. The scope of ISO is much broader, it includes in very small print, many other services. We did an internal audit for ISO earlier this year and we're now about to kick off the external certification in November. So that's coming.
Now, onto the future.
API keys will keep us busy for the next couple of months. So will ISO. But if we look beyond that, there are some other big projects on the way. I want to talk mostly about the BGP preview plus plus as I have called it here, but we also have a big infrastructure prong going on. There will be more on that today in the Services Working Group. And after the next RIPE meeting, more or less, it will have another SOC 2 Type II audit.
That's all in the activity plan. So this is mostly what I have just said. You can have a look, you know, on the slide or on the actual activity plan.
What I mean about BGP preview plus plus, that's the new thing, the more functional thing, if you will, I really would like to get into is that we believe we should improve the RPKI service by providing clearer ROA and ASPA sessions because if we look at the data there, there is work to be done, and let's look at the data.
So, as you may know, or probably do know, the RPKI Dashboard gives users information about BGP announcements that we see in RIS, and it compares them to the ROAs and then it really guides people into creating ROAs for the announcements that they intend to do.
.
This has been mentioned in other talks this week as well. The uptake of ROAs is pretty high these days, so globally around 73% of all prefixes ‑‑ this is just counting prefixes, IPv4 and IPv6 together to get one number. ‑‑ so especially in our region up take is quite high. Not just in our region.
If we look at the quality, if you will, so, how well ‑‑ out of all the announced prefixes that are covered, how many of those are ROV valid and we get to nearly 100%, and there are several reasons for this. People are dropping, that is the biggest reason, there is a quick feedback loop. If you start doing an announcement and it's RoV invalid you find out very quickly and you fix it.
Note on limitation. It maybe with all these statistics, that ROV invalid announcements are happening but they simply don't make it to our route collectors and, you know, we don't show the data. That is also possible, but I don't think that's a huge fraction. And essentially it's ROV doing its work.
.
The outlier here ‑‑ so, I don't want to call out them specifically but they have a relatively low uptake, if you look at ratios and things, stuff fluctuates. This will come back a bit in the next slides.
Because, we do have an issue, and it was mentioned earlier today by James, I believe. If we take this from the other angle and we actually look at okay how many ROAs we have and how do they match up with announcements, then we find that globally about 37% of the AS prefix definitions in ROAs are not actually seen in BGP will, and a lot of those are likely still. So a lot of those are probably ‑‑ well some of them might be backup routes that people create, but we don't see at the moment. But others will be cases where, you know, there is a strong incentive to create your ROA when you start doing an announcement there is much less of a strong incentive to remove it when you stop doing the announcement. But they can be problematic because essentially they do enable hijacks of, if somebody doesn't take that ROA and spoof the path, do the announcement for that AS, then it looks ROV valid and, you know, that's not great.
.
So, we really want to do something about this. It is visible in the interface at the moment but you have to look quite hard. So, we want to improve the tooling around this to make this much more obvious.
Also as you may know we have this opt‑in feature that you can say alert me about ROV invalid or ROV not found announcements. I think well we would want to extend that to cover some of these cases as well.
The action I think would be well remove these things if you don't need them anymore. If it really is a backup route, then we should give you the tools to tell us that it is a backup route so we stop bugging you about it.
Another issue also mentioned from time to time, is ROAs that are actually too permissive. ROAs have a max length, which is optional. So, if you leave it out, what a ROA says is this specific prefix can be announced by in AS number. But optionally you can say, well for this IPv4 /22 for example, I will want to allow all the 24s. And the max length originally was intended as a macro so you wouldn't have to create all these things individually, you could just make one statement that covers it all.
The issue with that, is that overuse of this also leads to more specific announcements being ROV invalid and if you don't tee see them in BGP, then this too enables somebody else to just do that announcement, spoof the path, and well they would win because they are more specific. So this is also an issue. We see about 7% globally. So one take on this would be to say you shouldn't use max length at all. The take that I used in this analysis is like okay, if you use max length but you are really announcing all the more specifics, then that's a valid use of max length, that's what it was intended for. So, those I don't really care about.
What I would want to do though is warn our users about cases where this might be dangerous. In the end of course you get to decide if you want to take this risk or not. So that is a recurring theme in all of this. Ultimately of course it's the responsibility of the network operators to ensure that their ROAs and ASPA reflect their operational reality. What we can do is do as much as we can to show to people what the impact is and where problems may be, so that's our part.
Are these all the cases? Well, no, there are some more but they are probably less dangerous, if you will. There is also cases where you might have a ROA or an AS, a prefix, where we don't see it at all. That might be still. But if, we can see perhaps that it is only disallowing certain things. So it's a bit of a special case, if this is intentional, then perhaps what you should be doing is create an AS zero ROA instead. So we need to think a bit about how to message this, but the point is that, yeah, this may not do what you want it to do, because just like with a still ROA, that actual ‑‑ an announcement can be spoofed that reflects it and it would be considered valid.
ASO ROAs, I don't think we have that many, I don't know how many we have. But for AS zero ROAs, we would typically see or possibly see announcements that are invalid. Now, our current interface is very much geared around telling you this thing it invalid, maybe you want to make it valid. In this case in particular, that is by design, right. So we should probably stop encouraging you to make those things valid that you just told the world about that should be invalid.
Related, if you are going to do AS zero ROAs and you have an additional ROA for the same prefix or a covering prefix and the real ASN, then that AS zero ROA does nothing. So it's redundant. We should also flag that. That's less of an issue possibly, but, you know, it can help with, to explain to users what's going on. So those we likely want to discourage.
Now, going on, moving on switching gears to ASPA. I will not go into how ASPA worked and all that. I have done that in the past meetings and you can have a look there. We also had a tutorial on Monday. Just focusing a bit on the data quality of it and availability. We see that the uptake and signing is still going up quite steeply. Only just recently, if you have keen eyesight you can make out the dark orange line for LACNIC, they have now also enabled this as the fourth RIR in their UI since September, and since then the amount of ASPA objects have doubled for their region already.
Now, this is.‑‑ talk to me afterwards if you want to know details about the methodology ‑‑ but I have some RIS code on GitHub, I have used RIS paraquat files that were mentioned as well this week, validation data taken from Routinator, and done some munching.
So, what I have looked at is all the unique paths leading up, the shortest ‑‑ so starting from the announcement side, I found all the unique paths leading up to the first network that has made a statement, has issued an ASPA object that says AS zero, so they are transit free. And the assumption here, so I'm very interested to see if other people have done similar analysis and have similar results, so the assumption here is that paths leading up to that point should all be customer to provider.
And then I have done the analysis on those paths. And we find that, well there is a number of invalids, so, there are cases where you find a next hop that is not provider. There are some, a lower value of paths that are fully covered and fully valid. What I call here partially valid is well in the algorithm something that is not covered at all or something that is partially covered is both unknown, but I wanted to get a feel for, you know, how many of these paths have at least some protection given by ASPA. This is a case where you have at least one other AS in the path that that issues with the object, and that one is then also valid. Because if it would have been been invalid it would have shown up in the red line. We're now at around around 12.5% partial coverage. I think I have a percentage number on the previous slide, but that's around 5 percent of registered ASNs in our region has an ASPA object. It's not surprising that this number would be higher because you know all it takes it only one AS in the path to have an ASPA object.
.
I wonder how common are invalids relatively? I have shown this slide at a previous meeting as well.
What I see here ‑‑ so what I have plotted here is the fraction of paths that are invalid relative to paths that have ASPA at all. And then we find a very constant line that's around, 8, 9 percent of paths that appear ASPA invalid. This is actually a similar level to what we had with ROAs before people started dropping invalids, because that would provide a quick feedback loop, but I am hoping that with ASPA, we can prevent some issues and catch them earlier before ASPA validation becomes prevalent and, you know, people have to scurry and go into the interface quickly and fix it.
So, we want to provide data information here as well.
Now, I don't know, the algorithm doesn't know it either, are these cases where we have actual leaks or are these cases where people have forgotten a provider. It's a choice of the user, you should know your who transit providers are but it can be easy to forget some especially if you have many, so we will likely do something where we want to take this information and maybe other approaches to tell people about, you know, these ASNs may be part of your ‑‑ maybe you should include them as part of your provider AS‑Set on your ASPA object.
This work is likely to start after the API migration and after the ISO audit that is planned for the end of this year. So hopefully we can start early in Q1. We haven't done the design yet. We are keen to get started with it, so what I wanted to say here is if you have ‑‑ if you are interested in this work, if you want to help us be part of the user test group or, you know, somebody we can interview on what you believe would be useful here, then please reach out to me, my e‑mail is on the last slide, or Antonella, who has been doing user testing with many people this week. So, find us and we'll be happy to get back to you when this work starts.
.
In terms of deployment, we will likely do it incrementally, so whenever things are ready, we'd like to deploy it, so I don't think we need to do a big bang. It depends a bit on how we shape all this. So coming.
And I think that already brings me to the end? Yes, it does.
SEBASTIAN BECKER: Thank you very much Tim. I think Gert was first.
GERT DÖRING: Thanks for the presentation. Thanks for all the good work. I have ‑‑ I think we have time, and this is good because I have three different things on my plate that I want to bring up.
One is the BGPsec router keys, which is a nice thing to have because I have been be in discussion with somebody about what does BGPsec gain us compared to our ROV and ROAs, and I didn't have the answer. I got the answer I wanted and that is basically you can have signed originated prefixes. So even if you don't have the whole path validated you can actually improve the RoV by an actual crypto validation. But I think I am still fuzzy on the details. So this is the ball I want to throw at the chairs.
If we could have, like, a talk about what you can to with BGPsec with an incremental deployment today to improve the protection gap that is left open by ROV, ASPA and ROAs, which is basically spoofed origins, spoofed AS origins, but I am happy to really see progress here, so that's definitely cool.
The second thing is also more of an observation. You said you have ASPA invalids, and I received an e‑mail from a Chinese researcher like three, four weeks ago, that looked at our ASPA object and the BGP path they could see from their vantage point and said, we see a couple of ASes that we think you have forgotten in your ASPA. I looked at that, and half of them were Cloud providers, or DDOS entities. So I never heard back from them on why they came to the conclusion that these are my providers. But there seems to be some measurement artifacts in here. Like, how do you come to the conclusion that something is ASPA invalid, or maybe the actual definitions on how the validation has to be done is not that clear. I remember a talk from Maria about the standard being hard to read. So, that is just another data point that ‑‑ I know my ASPA objects are correct because I checked, but still people see them as invalid. So, there is something of a mismatch here.
TIM BRUIJNZEELS: So, yeah, first of all, there may be a flaw in my approach and if there is, I am really happy to learn about that, that's why I said I am curious if other people are doing a similar analysis, if you have a route server or you are a big transit provider, does this match with you? I'd be very interested in that data.
But this kind of was what I could do with available public data. So there is that assumption in there. Also, with ASPA, well, it could be actual leaks that they see, it could also be path spoofing, even, that is then flagged by ASPA as something that should not happen. It's hard to tell.
GERT DÖRING: I wanted to bring up that we still see ‑‑ let's call it artifacts. So there is a bit of uncertainty on the RIS. I am not saying that your measurements are wrong, I am just saying we have this other data source which says there is something and it's not me. And I don't think it's CloudFlare, because they know what they are doing generally. Supposedly. We can have a look into this.
A very short point. I mentioned this on the mailing list when you brought up the new regime for the RPKI keys, I am still not happy. Because if you are on the client side and you want to automate something with basically LIR account, you cannot tie this to persons. Persons leave their RPKI keys go invalid. Just stating this again, I have said it before.
AUDIENCE SPEAKER: Silvan. I have one question to your CA reminder e‑mail. From which vantage point is this actually? Because it might be beneficial to everyone to also get that linked into RIPE Atlas to know that our CA is invisible from parts of the world.
TIM BRUIJNZEELS: The vantage point is essentially our network, we do constant RPKI validation, in this case with RPKI client because we use an API, well the CCR that, implemented in RPKI client. We hope to extend this also with Routinator when they have support for it as well. So we can have multiple inputs. Multiple vantage points might also help, but it's a long time. So I don't think it's likely that we just don't see somebody for 90 days but the rest of the world does. But, yeah... did I misinterpret what you said or...? In any case ‑‑
AUDIENCE SPEAKER: The idea was the reverse, so you see us but not the rest, or some parts of the world would not see us.
TIM BRUIJNZEELS: Right. I think that requires a different approach. That would be interesting work to do as well. But that's not what we have done in the context of this.
AUDIENCE SPEAKER: Ben. Ignore my Working Group co‑Chair on this. To Gert, on your question about the researchers suggesting providers that don't exist. Part of the reason why ASPA is here is because detecting providers is really difficult. There is a lot of academic work in in universe that has been spent many many hundreds of hours, probably thousands, trying to automatically figure out the providers. My assumption is they have run one of these well known algorithms and it has come up with the wrong answers. It's a really tricky thing to do. Trust me from me, this is my day job. ASPA is fantastic.
To Tim, thank you so much for ‑‑ and your team thank you so much for doing BGPsec certificates, I created one, testing one, admittedly the private key was in fact full, and researcher e‑mail me this morning alerting me to the fact that the private key was full. It's good to know that the time for reporting for this has shrunk significantly. So thank you, thank you for the API, it's lovely to see like momentum on the RPKI stuff for RIPE. I look forward to whatever RIPE is working on next. I think that was all I have to say. Thank you.
AUDIENCE SPEAKER: Peter Hessler. I want to make two comments on the ROAs that you mentioned earlier with AS zero and with the more specific.
So a lot of networks will create the more specifics, or will allow a longer prefix than they normally original for for DDOS mitigation purposes. And the reason why we keep it in the database that way or keep it published that way is because the reaction time between I have to activate this for DDOS mitigation purposes and people receive the new ROA that allows those prefixes, those specific sizes. So, I think it's quite unfortunate that we might have to keep that for effectively forever.
TIM BRUIJNZEELS: Yeah, I think it's a choice. Also, I think ASPA will help to make spoofing more difficult, so then maybe this problem becomes less of a problem but we're not there yet. Today, it's your choice. There was a presentation earlier this week I think the median time for a propagation of a new ROA is about 20 minutes with a long tail. That might be too long for your DDOS mitigation, so if you feel that is not something you can live with and you are looking at the risk of somebody doing a hijack of a more specific that you don't announce, yeah, maybe your conclusion is that you actually want to have the more specific, either by having a specific, more specific ROA if that sentence still makes sense, or by using max length, whichever way you want to do it.
PETER HESSLER: Our DDOS mitigation reacts ‑‑ the reaction time is measured in seconds and not minutes or hours. So 20 minutes is effectively too long for our scenario. Then the AS zero scenario, again reason why you may want to have max prefix length of foo, and then also inside that this specific sub‑net within that far carrier block must not be announced that that one has a specific AS zero for it. Like to prevent a hole punch. If you have authoritative name server for example or whatever.
TIM BRUIJNZEELS: It depends on your ‑‑ yeah, it gets complicated. It depends on your max length, whether that would already be forbidden by the other ROA.
PETER HESSLER: Again I assume this is not the problem you are referring to, but we have a container block /18 that has an AS zero, and then because sob blocks are signed to the customers and those have their specific ROAs. But I am assuming that's not what you are referring to. You are referring to the other way.
TIM BRUIJNZEELS: So, the redundancy of AS ROAs I meant essentially if you have an AS zero ROA that is not effective, is it covering ROA already? That would... yeah, essentially disallow other announcements already. The AS zero ROA doesn't add anything more at that point.
PETER HESSLER: And alerting on those where we think it may work but it actually doesn't that would be useful and which I believe what you said you are doing. Thank you.
AUDIENCE SPEAKER: Tom. Thank you very much for that presentation. I am just going to echo what Ben said, excellent work and it's been exciting to see where you are going. I want to echo what Peter said. The big problem that we're seeing with a lot of the RPKI side of things isn't just a publishing delay, it's also the verification delay and that's harder to measure. So, even if RIPE were to get like commitments that we're doing publishing within five minutes and you have 95% reachability within five minutes for the entire database, that would be awesome. But then you still have networks that will do validation every 30 minutes, every hour, every six hours or even everyday. Which at that point means that your latency is 24 hours in the worst case and then being operators I think everyone here will agree that it's usually going to be that network that you want it to do the thing immediately is going to be the one that's not doing the thing immediately. So, I think we need to acknowledge as a community that RPKI, the RPKI as a database is not for realtime purposes and that we're going to see situations like you are describing where we're allocating things in the database for once upon a time, and I think that's just going to be a thing that we'll need to accept.
TIM BRUIJNZEELS: Yeah... in the talk earlier this week it was also mentioned that some increase that's becoming effective in routing can be achieved by validating more frequently. I think the default Routinator implementation does every ten minutes and the many default RPKI clients every hour. But it should be feasible to reduce that and I will have ‑‑ it won't get you all the way there, it won't help with you this case, it won't be second but it will help a bit.
SEBASTIAN BECKER: Thank you very much Tim.
(Applause)
.
We will now do the selection of the next Routing Working Group Chair.
BEN CARTWRIGHT‑COX: The way this is going to work, as I have mentioned before, is we're going to have a Meetecho poll, but before that, before we open the Meetecho poll, we're going to hear a short like introduction from both of the candidates. So, could Mick and Tom come up.
MICK O'DONOVAN: Mick O'Donovan is my name. I am part Asiera, the research and education network in Ireland. As per my message in the mailing list, I have been coming to RIPE meetings since 2013, mostly as an eyeball network really and a consumer of content, although in recent times, I have started to kind to try and give back in my day job, and here as well I'd like to contribute as co‑chair. And despite what it might look like, I am actually friends with Tom, and there is no ulterior motive, we are aware of each other, and I would be happy to see either of us co‑Chair this Working Group. So thank you.
STENOGRAPHER: And most important of all I know Mick and he is cool!
TOM STRICKX: I have been part of the RIPE community for over eight years, I think my first talk in RIPE was in RIPE 77 in Amsterdam, in 2018, and I want to give back to the community, right. A lot of the reasons why I am where I am has been in large part due to the RIPE community and in the presentations and the content that's been provided by RIPE. So I want to be contributing back to that and make sure that we have a follow‑up generation that can yell at a microphone and say stupid shit. So that's the important bit, right, and I want to enable that, right?
BEN CARTWRIGHT‑COX: Is there anything I am forgetting? I think we are probably good to open the poll. In which case I am going to scurry over to my laptop and push the buttons and then we'll have an awkward five minutes where we're not sure what to do.
I hope everyone has found the Meetecho link for that, and ‑‑
BEN CARTWRIGHT‑COX: The poll is running for five minutes. I think last time I read out you a bunch of very bad puns, but I forgot to update my stash of very bad but acceptable jokes.
SEBASTIAN BECKER: Maybe I will just read three little notes that we have for the programme. Again please note that the voting period for the PC election is open until tomorrow, 5 p.m. And the deadline for RIPE NCC members to register and attend the General Meeting is today at 2 p.m. And you can pick up your sticker for attending the General Meeting at the registration desk until 4 p.m.
BEN CARTWRIGHT‑COX: I don't think we have anything to say at the moment, I guess we will let everybody figure out Meetecho. Everybody is yearning for lunch I am sure, Ten away everyone.
AUDIENCE SPEAKER: So, I am coming to the mic because I can spare you some time and you don't have to fill in the three minutes that are left. I was thinking about doing a lightning talk at connect, then I left my slot to Remco because he is a much better speaker than I am. And I was wondering, it's like a routing matter actually, I was thinking ‑‑ I was told by Claudia from open BGP that the way we are doing inter‑connection between IXPs will not work correctly with the current state of ASPA, so he did come up with a suggestion to give us a way to do that correctly. So ‑‑ and that's what I wanted to mention, I think we have got a way to go and fix that and I think I could bring that suggestion to the writing Working Group on the mailing list to have feedback on that to see if I am not understanding the things in a wrong way, so that was my suggestion. So I can take some of your ‑‑ that was one minute. Sorry I couldn't do better.
BEN CARTWRIGHT‑COX: That's genuinely very useful. I wasn't aware of that suggestion. So that is definitely appropriate for the Routing Working Group mailing list. I am sure there is a lot of extra eyes on how that fits within the sort of legalese of ASPA.
AUDIENCE SPEAKER: That was my thing and you have got like one minute and 30 seconds left.
BEN CARTWRIGHT‑COX: Thank you very much.
STENOGRAPHER: What did the zero say to the 8?
.
Nice belt!
.
Just checking everybody is reading! And I am not redundant!
.
Thank you. No AI so.
BEN CARTWRIGHT‑COX: I can announce the results. We have 34 votes is Tom Strickx, and Mick O'Donovan is 39 votes. So congratulations to Mick. (Grand, another Irish man!)
BEN CARTWRIGHT‑COX: I would also like to say thank you to Ignas, who has been with the Routing Working Group for many years, isn't here today, but I think we both appreciate his long term standing here, and I think that's it. There any other ‑‑ we have some closing slides but I don't know how detailed they're.
SEBASTIAN BECKER: The closing slide is just closing, so any other business? If there is anything you just have for the Routing Working Group, please come to the mic. Otherwise... you have six minutes early to the lunch.
BEN CARTWRIGHT‑COX: This sound a bit incompetent. Do we have to say magic things like the minutes are approved or something? I can't remember.
SEBASTIAN BECKER: Yes, maybe. These things are complicated, last time I nearly closed the entire Working Group permanently by saying the wrong words. You have got to be careful with these things.
Okay, in which case, I think we're done.
SEBASTIAN BECKER: People are moving so...
BEN CARTWRIGHT‑COX: Yes, so thank you very much everybody. Congratulations to our new Working Group Chair, and thank you. See you next time in Vilnius.
(Lunch)