RIPE 93
Monday, 26 October 2016.
Main Hall.
4pm.
Plenary.
MATT PARKER: OK. Good afternoon everybody. Welcome to the second plenary session of the day, my name is Matt and I am going to be co‑chairing this session together with Clara and we have two 30 minutes talks scheduled with Maria and Petya followed by three lightning talks from Sam, Ties and Tobias. Before we get started, we have a couple of brief announcements, nominations for the PC close tomorrow at 3.30 local time. If you are interested in joining the PC, please take a look at the website and submit your nomination, you can self nominate if you like.
And secondly, there are a couple of BoFs today starting at 6 pm. One in the main room and one in the side room, the main room is key topics at the IETF and the side room is layer 3 internet exchanges, if you are interested in those topics, make yourselves available there and without further ado, let's get start with Maria who is going to talk about embracing NETCONF.
(APPLAUSE.)
MARIA MATEJKA: Hello everybody, nice to be here again. Nice, it's quite not many people like at NETCONF, that's nice.
That's very nice. Yeah, Tobias was and Franziska were speaking about things like well, you should be not always right. Well we were wrong the whole time and we are probably wrong again. Let's speak about how we actually want to get with NETCONF and what we run into because one of the best topics are where we failed or where somebody else failed so let's go into it.
But well that's a cat. Users were looking at us like this for a really long time, very stubbornly refusing to have any reasona ble API because in BIRD you have the CLI and there are things like well BIRD... on the screen there is a bird watcher which is some birds which actually look into the CLI map and convert it into JSON or other things. Well that's not anything standard, let's go for something standardised. So let go for NETCONF and NETCONF is the standardised thinking for 20 years, I expect that most of you are using NETCONF for everything here now after 20 years?
Is there anybody not using NETCONF at all? Oh no, there are some people not using NETCONF, what has gone wrong.
This is, I am having some quotes from the RFC because these are quite necessary for this and NETCONF brings us things like you have, you don't have to think again about how to send things to the clients, you just implement standard thing as it was written and there was a problem with SNMP, you can't extend it in any way, well in the NETCONF, you can add your things, it's self‑contained and also you can read it by your eyes because it's XML. And XML, well 20 years ago XML was the only thing. There was nobody thinking about JSON even. And nobody thinking about anything else and XML was the saviour so yeah, we have this.
And then we have some funny things like yeah, you open the connection and then you expect the connection just lifts and lift and so on and so forth and you just like open the dock CLI which is put onto the NETCONF and you write a command and it stays as it was intended, it's just, it just stays on.
And you can lock part of the device so that nobody else actually looks into that place, nobody else writes into that place and that lock is shut down whenever your connection goes down, which is nice if your connection is totally flakey and if it crashes every two seconds.
But yeah, it unlocks every single time when your session crashed.
But yeah, come on, let install NETCONF, it specified, let's implement it and then you come to the theme and that theme asks you a stupid question. What about the web interface, what about the browser. The browser is not going to keep any session long left, the browser is having no sessions at all, it just fetches the request and then closes the session because it wants to spare some sockets. Also, you have the question do you want to rewrite your CLI to be on top of the net link or you were lucky if it separates, well these are questions which are asked by the teams and are you really sure that you know the right answer. I don't. I don't know what to do with it.
We have some decisions. But the IETF found another thing which should be actually done and that's RESTCONF, that's another version of NETCONF but this time it doesn't have the sessions. It has no sessions at all and it's intended to be used from browsers. Well nice, we have a second interface, let's implement both. But then we have RESTCONF and NETCONF on the same router and somebody opens NETCONF and starts editing things and starts logging things and then somebody else goes in and invokes RESTCONF and by RESTCONF, well they can't access the thing which somebody else has logged. And voila, some of your scripts has now failed, spuriously, and nobody knows why because that engineer who accidentally locked something by some CLI command knows nothing about even logs because that CLI command was doing something complicated and nobody knows that your tooling fires at the very exact point in the time so, well, it can go worse because you may start editing your new configuration because well you need to reconfigure some sessions and you are in the middle of the reconfiguration and somebody comes into, some tooling comes in and the tooling says well, let's change this one specific thing, it can be your ansible, whatever, it can be some random script, you even don't know that runs RESTCONF on this specific router you are just editing the configuration of, well nice.
Now your work is committed to the running configuration just in the middle of reconfiguration because somebody else did something.
And we have to two standards. And these two standards are exchanging the same data about the same thing well over different connections. And with a little different semantics and with a little different appearance.
Well what can we do.
Well I forgot something. There is the YANG thing. Because well, I was speaking about what's happening on the protocol but what's the data actually inside the connections. Well the data is modelled by a specification by files called YANG and these YANG files say how the data should look like in the XML because it was created for the NETCONF which is specified to run with XML, well no problem. Other protocols and encodings are possible but are out of scope, well let's specify more.
Indeed other encodings may introduce alternative presentations, in XML everything is text, even a single number is text, it's just a deck Dick number, deck Dick representation of the number. And so we have another RFC which says well now we have YANG defined data, how to encode that node in XML but in the J so I son, nice and in JSON, we do have numbers. Let's use numbers foreign coding numbers but not all numbers.
Because most JSON browsers use floats, uses doubles, for interger storage and therefore you can't use the integer type in JSON for 46 bit numbers because somebody else somewhere like 20 years or 15 years ago forgot that are there are... numbers and now, one has to think is that number ever capable of being over four billion, well billion or million, which English are we using, is it ever going to have more than 32 bits? By specification and if yes, you have to use the string even if for a single number, even for a zero.
Well in this situation we looked at it, what do we want. Do we want NETCONF, do we want RESTCONF, RESTCONF actually also allows sending XML besides JSON, I don't know whether anybody implements it but it's technically allowed.
But you want JSON for the website because it's easier to parse, it's more expected than XML. OK.
So we looked at it, maybe even both, because well there are some customers who want NETCONF, there are some who want RESTCONF, there are some customers who don't want both so this is how it's going to look like and we have SSH transport with NETCONF and HTTP transport with RESTCONF which both connect to young YANG data store which is specified in the RFC. And then there are the native data structures because the YANG data store is not exactly what we need, we don't need the IPv6 address as a bunch of text, we need the 16 bytes. But are those 16 bytes the source of truth? Or is the text the source of truth. What is the actual meaning, what is the major primary version of the data.
And the RFC is very specific, all data is the canonical representation by the the XML, welcome to hell. And then I pulled some data, I did some dump of routes from BIRD and ransom Linux perf on it and did some numbers and I found out that the most, not most but a lot of time is spent by formatting the correct numbers. Or formatting IP addresses or formatting times, writing existing strings from the memory.
I don't say that we can make this faster so much. But still it's quite significant.
So back to binary maybe or as Italians say... (ITALIAN TEXT)
There's a third one, CORECONF, hello. And this came out from a different word from IoT where people said JSON is too big, we don't need things like sending five bytes of data but we want some standardised things so well there is CBOR which is a binary thing, kind of resembling JSON but it's in binary and there is CoAP which is resembling HTTP but very much reduced to what you actually need for machine to machine application so that it's like an ideal thing for the API and the best thing on this is that I actually could take several of the RFCs and implement thousand lines of code into in two afternoons without ever having to include library. So this is the same CBOR, it's so concise, so small, one can write it themselves.
The CoAP is actually intended to be over UDP which has some bandwidth problems so there's a TCP version, this is just, this is just to be efficient enough because 16 megabytes per second is quite not enough for our fire hose of data.
And then there is CBOR, the CBOR, the difference between between all those SNMP data and CBOR, you can be called CBOR even if you don't understand it, you don't have to have the schema, you just take the data and decoded it.
You get the structure; it's self‑contained.
And you already have the lip CBOR in your laptop probably if you are any using any cryptographic keys, if you are touching that key on the side of your laptop, well the key is communicating with your laptop via CBOR, that's already deployed. That's no experiment al technology and here's some examples, you may look at the hex stamps, the nice thing is that no string, there is no escaping in the strings. There is no, nothing like parsing the numbers, you just say well if you look at the string, the first byte is saying how long it is and then you just copy the memory.
If you look at the number, the first byte is saying how long the number is in bytes and then you copy the number which is a network order and you are done.
With the date and time you may actually say instead of parsing this, you will have to write this out to your user anyway, probably in a different format, why bother having this in a JSON. You can have load the binary thing into your, directly into your software and the same is with the IP address. It's just there in binary. With some header.
The CORECONF is not finished, there's the problem and Vojtech is doing a lot with that, it's my colleague working on the API things the most. There are some drafts about actually encoding the IP addresses and dates with the end data, there are some other works and other problems and there are, there's the obvious idea of politics where people want things to be included and excluded and things to be put into different draft which obviously means they want it delayed by a year at least. So we have some disagreements with people who already have implemented things in some way and we want it another way because it doesn't make sense in high performance and deployment, like the usual stuff.
But we have now basically three standards, what a nice world. Well is this success or succession? This is probably more succession. And what should we use?
And then there is the problem of composition. Let's assume that I am building a Linux based router and I want to put a NETCONF interface on it. But I am not putting there a single routing stack, I have system D, I have network manager and somebody has written, I don't know whether system D does RESTCONF but somebody may rate it, whether the network manager does NETCONF but some of those like JSON, some of those like XML, why not put the appropriate interface over it. And then there's the come 'Tore which will have to compose all the trees into one end point and again, what's going to be exposed to the customers, is it going to be the NETCONF, the CORECONF, the RESTCONF, all those three, are they going to be in parallel allowed, are we going to forbid using CORECONF when CORECONF and RESTCONF when NETCONF is in use? What's the right approach?
I have no idea.
But the thing is we have to have some API. Also this is totally not specified at all, nobody knows what to do with it.
So yeah, there is a draft which is expired again, because I had no time updating it. And yeah, we are trying to find out what to do when you try to proxy NETCONF into RESTCONF or whatever else you want.
There is also a project by Vojtech UI TC, which wants, which aims to create code parses code and encoder code based on the YANG, this project itself may help some proxy slide in future because you need to know the, the model, how the data looks like, you need to know how it looks like because if you get something which looks like an IP address, is it actually an IP address? Can you convert it into binary IP address or should you treat it like a string. You have to check it.
You have to know it. Because if you don't know, you may end up with something which was intended to be a text, but looked like an IP address being converted and it doesn't work.
And that's why we need to put work into it and that's why we still don't have an API but we are working on it.
It may look like this. This is a boat on a rail bridge over a road. Let's hope it's not going to look so much strange.
And then there are like three last mandatory slides, the credits, the contacts, you may download it from the slides and the QED.
And now questions, explanations and discussion. Thank you.
(APPLAUSE.)
MATT PARKER: OK. Thank you, Maria. Please?
AUDIENCE SPEAKER: Ivan Garnizov: So the question, I am really struggling to understand because with the JSON and his now, you have also the scheme, the meta data that defines what the data means as far as I understand, with CBOR you don't have that so I am not sure what the question was because how do you interpret the data after the label.
MARIA MATEJKA: In CBOR you have the same structure as in XML and JSON so you can parse, you can basically parse the CBOR into a similar tree like structure as with JSON and XML and the same way as you need the schema for the JSON to make like the full sense, you need the schema for the CBOR as well. The difference is that the CBOR is way smaller, and I don't have to divide it by ten, which means using something in the CPU which is quite slow to divide by ten every single time I write the correct number which may be if I write a dump of routing table which is basically bunch of numbers then I would prefer to not divide it by ten every single byte.
AUDIENCE SPEAKER: How do we interpret the data? Because if you have a schema that that says this data is this, this question is kind of...
MARIA MATEJKA: I said if you don't have the schema, you can't have a proxy.
AUDIENCE SPEAKER: Yes with JSON and XML, you have to it.
MARIA MATEJKA: With XML and Jason, you can't make the proxy as well.
MATT PARKER: Ruediger?
RUDIGER VOLK: Hi. Well. Kind of you are fighting with various presentation layers of the same data, well OK, minor thing, there may be some small restrictions in one of your ‑‑ one or several of your presentations. The thing for doing proper system and programme design is to have an abstract view of what you are modelling and kind of the YANG is the thing that kind of is supposed to offer you a way of defining and talking about it. And all the free and 40,000 other ways of presenting it should have their own automatic ways for actually doing the presentation layer mapping. Very simple. So, kind of talking about XML schema, talking about JSON schema, talking about CDDL, the CBOR schema language is kind of out of the way, the question is, is there a good mapping for the YANG common schema definition to the various presentation layer oriented schemas. And kind of, I think we have to check whether there are any gaps for having those mapings fine and having well done YANG models that actually allow us to define what we need in the devices, then we are done. Kind of on the other hand, trying to do the lower level problems directly at the lower levels is not helping a lot and yes I can see that network engineers typically are thinking in the presentations they are used to. But kind of we will actually make the whole industry work much better if we get to the abstract layer for dealing with stuff and using the tools for doing the mapping. Thanks.
MARIA MATEJKA: Yeah, we are basically on the same page. The only note on this is well, the YANG itself is specified to basically describe the XML first, that's what the RFCs say and we are basically fighting last ten years with the problems we did 20 years ago.
MATT PARKER: OK. Thanks again, Maria.
(APPLAUSE.)
Next up is Petya who will be talking about path changes with measured impacts in R and E networks.
PETYA VASILEVA: Hello everyone. I am here today to share with you a system we are currently building, it's not done yet, it's still a work in progress, and today I will share with you some of the important and interesting parts of it, as well as some examples of the findings of this system.
So, we have built a system which is based on person R measurements that are done on the worldwide LHC computing grid. Let me first start with a bit of context about why this work is important. So large hydron collider at CERN in Geneva is the largest machine ever built, it smashes sub atomic particles at close to the speed of light and those smashes those particles, they produce huge amounts of data, so whenever the collider is running, on average we get about three Beta bytes of data, so this is three big data and this data needs to be spread across the world at different data centres in order for the scientific community to process this data, to replicate it, to run simulations and other things on the same data sets.
So all this is is done on the worldwide LCG computer grid, or WLCG. This covers about 170 sites and it has the network is, so we have tier one, tier zero which is CERN, sites which operate on the LHCOPN, the optical private network, and the rest of the sites are running on LHC one which is the overlay for the wider community on the shared R and E network. About 20% of the computing resources, storage resources are at CERN. Locally. The rest is spread around the globe.
And here is where perfSONAR comes in and it's important for because we would like to know the global state of the network, the global picture of how the network behaves and whenever there is a complaint for example of a slow data transfer, we want to be able to say what is going on, whether it's the network or it's the application and what clues we can find in our data.
So we are running active measurements of standard network tests, we have leans which are divided into two data sets, we have the through puts and the traceroutes which are standard traceroutes, so currently we receive about 4 million tests daily.
Those tests are produced by thousands of source destinations pairs or corridors and you can imagine how hard it is to get, to distinguish the signal from the noise in such a global infrastructure.
So we face many challenges, some of which are incomplete tests, different test rate as you can see here each of the test ‑‑ folks the traceroutes are once every ten minutes, the latencies are quite light, we run them every minute, while the throughput test it is, they are once every six to 24 hours.
So we need to have a meaningful way to align those events and to extract the knowledge from this wealth of data.
Regarding the traceroutes, everyone in this room probably tried at some point to debug the network of your traceroute and you know how messy those measurements could be and how varying the paths could be, we can see there for example load balancing, we can see silent hops, some cases in which the TTLs jump, all these are small changes but most of the time they are meaningless.
So, we tried to approach this, all these challenges and to find a solution for them and the way we deal with them is by creating a graph from the measurements which takes into account both theSpath and the IP level path and we create this exact version of the network keeping all the statistical information intact and we tried to overcome the challenge which, in which we have unevenly tested pairs, for example some pairs may be ideally tested six time per hour but others which may be part of different matches can have 20 tests instead.
So anyone who tried to load this kind of, this type of data to a machine learning method knows this is not favourable for machine learning. So when we tried, what happened was that the less tested pairs were immediately flagged as anomalies and we needed to find a solution for this without necessarily injecting synthetic data. So this was our method. We created this biased random walks on your compact version of the network and we create about three hundred walks from each source of measurement and then we feed those paths into a WordVec model. The result of it is an nought representation which answers the question whether two nodes were seen very often close to each other. So if that's the case, those nodes will receive very similar embeddings.
OK. So those are the node embeddings. Each node is each ASN or IP gets a list of 32 values, a vector of values which define its position in the embedding space. And then what we do is we create an aggregation for each of the paths and in addition to that, we have structural dimensions, this is the embedding dimension and we have the structural dimension to keep as much information for the raw data as possible, and we keep the information about the IP repetition, whether we have loops on the paths or not. We keep track of the TTL increased gaps as well as the path length ratio. So each of those values are taken from the baseline for each of the pairs.
So the idea here is that the structural dimension answers the question was this path healthy, did it look healthy while the embedding really flags the paths that are truly different, truly novel.
In the end we combined those two dimensions into one composite score. Which is between zero and one. And it's defined for each of the paths and we are able to say whether the paths were normal or anomalous. This is one of the lanes for our triage system. So processing the paths was done the way I just explained and then we have the performance values which are all latency tests, all throughput test it is, here the flags are straightforward, they are defined from our operational experience. And at the end we temporarily aligned those events in the same data set and each of the corridors we are monitoring is placed in one of the currently seven bins.
So for example the red ones are probably the most interesting ones. Those are the corridors in which, for which we saw rerouting and performance degradation in some way. Then we have for example the blue ones, those are, there are things stay exactly the same, the usual routes but suddenly there is some performance degradation and so on.
So this is an overview of our triage heat map. Which the rose are sources of measurements, the columns are the destinations and this was a live catch from our dashboard, it was an incident in May this year where there was the cable fault between Amsterdam and London and although you see many colourful boxes here, a lot was going on at this point on the network, the detection was quite clear because if we look at the UK, one of the UK sites here and this column here is Netherlands, this is exactly the place where we would expect to see this cable fault IP fault impact.
If we zoom in on the same example, we can see the four main plots from our dashboard, you can see that the routing score jumped directly to the maximum score which is one and there are a couple of interesting things here, for example we were able to detect a routing shift hours before we saw the ticket, this ticket came from Geant. So the other think is if we look at the one way delay, it looks like the delay improved and you will see why in a minute.
The packet loss is a little bit increased but only a few events cause the threshold while the throughput was the one that was really suffering in this case and why this particular corridor was flagged as rerouting for slow performance. So the same pair will look at the baseline paths on the left and we see that usually for this pair UK to Netherlands, the traffic was going from Janet through CERN and then Netherlands, while now the new path is going from R A L UK to through Geant and to Netherlands. And here it's interesting because the path is geographically shorter although there are more hops, the distance is really shorter than previously.
But previously the optical network was used so the capacity was much larger than what it is on the general RNE network, on the LMC 1.
OK, so this is the part with the new features still not deployed in the live application so we tried to visualise also how the paths, diverged so this is usually how those corridors were running their traffic through CERN and now the impacted corridors are going Geant here, we tried to local lies where the degradation happens, in this case we are trying to attribute the low throughput to a specific segment or the network, so we are basically running topography and trying to see on all corridors experiencing the same problem, what are the common paths or parts.
So in this case we can see that there are a couple of hops, the darker the colour the more used those hops were. So in this case, there are a couple of Janet interfaces which are 100% of the time on the corridors having low throughput.
And the final example I thought this will be interesting for this audience because this is something that is not resolved, this is ongoing six months. And maybe we were the first to notice it because it's interesting that we see some honey packet loss, again it's on the UK corridors, 240 important, again the RAL institute, and those columns here in red are the ones that we are discussing right now and on top you can see the packet loss so the interesting part here is that this loss is not constant, it's intermittent so we see some bursts from time to time up to about 5% but once those values are aggregated, the mean loss becomes 0.5 in most cases so it's right at the threshold to be declared a problem.
So when we look at the localisation method and try to local ice what was the common for those corridors experiencing this loss, we saw that those two hops are probably at fault and in this case, 92% of the corridors which experience loss were crossing those two hops.
So our idea is to, we cannot know for sure whether this is true or not because not all aspects of the network are visible for for the perfSONAR's test but we can at least give the network operators some region in which to focus to try to find the fault.
So yes, all this work was labelled free, we don't have any tranche of data, it's completely unsupervised and in the past few months, it was extremely helpful when we were receiving the Geant tickets that were confirming what we were seeing and detecting with our triage and our methods. So, if you are part of an NREN or network and you have a way to give us some access to your ticketing system or to be able to subscribe for your incident emails, that would be really grateful for our work because this way we can validate our findings.
So feel free to check out our dashboard, keep in mind that it's not done, it's still a work in progress.
And finally thanks to the national science foundation for supporting this work. And I am ready for your questions if you have if you have any.
(APPLAUSE.)
AUDIENCE SPEAKER: Mick O'Donovan, from Asiera, one of our R and E colleagues. Great work. I am not sure if Rob is in the room from Geant, aforementioned, I am sure you could probably contact him and reach out about the packet loss during the week. He is usually here. He is here tomorrow.
PETYA VASILEVA: Thank you, we contacted them at some point but it was maybe a month ago and we haven't heard from them, they verified and said they see the same behaviour in their plugs, their monitoring tools, but we didn't hear the end of the story. Thank you.
MATT PARKER: Over here?
AUDIENCE SPEAKER: I just wanted to give a comment.
MATT PARKER: Your name and affiliation.
AUDIENCE SPEAKER: Ivan Garnizov of perfSONAR. So, I could give a hypothesis here on the RAL issue, it appears to be an SLA that's tried to be satisfied but at some point in time it's so close that it's, it makes mistakes, drops packets, so the packet loss and when there's a packet loss then the transfer or the throughput falls down very fast. And yeah, and that's also the explanation for the that I saw there earlier, you know, where at some point in time it made a switch and there was this tradition and then to another, to a shorter path but they were, there was a packet loss at the throughput fell down.
PETYA VASILEVA: Here?
SPEAKER: Yes. That's a pattern that I really like it, so this is actually an anomaly detection system which allows us to observe a huge map and be able to concentrate on focus on really the problems to, to find the problems earlier than the others.
PETYA VASILEVA: Yes. This is the idea. Thank you.
SPEAKER: I had a question about this whole project. So as I currently understand, this is a triage idea or find out where current problems lay? What are the ideas beyond this, so in some of the slides I have seen an alternate path, are there any ideas to proactively use that or...? That's the basic question.
PETYA VASILEVA: Can you rephrase it, I am not sure I understand
AUDIENCE SPEAKER: I see alternative path here, right, is there any idea beyond this current project to use this information proactively to send, if at all possible, data to through another route?
PETYA VASILEVA: You mean to use our implementation or other use cases?
AUDIENCE SPEAKER: No. Do I understand this is a tried tool right now? OK, at some point you see that data is not going over a current link very well. Is there any idea beyond this current project to use that information to proactively send data another route?
PETYA VASILEVA: No but we can discuss it after the session. Sounds interesting.
AUDIENCE SPEAKER: Definitely.
AUDIENCE SPEAKER: This is observation, this is monitoring, so we are observing the area, we are not providing, giving solutions or decisions to to someone, this is just an observation, you know, so that's the point but we can yes discuss it.
MATT PARKER: OK. Then thank you very much, Petya.
(APPLAUSE.)
MATT PARKER: So we are now on to the lightning talks and first up we have Sam van Kampen from the RIPE NCC, he is going to talk about RISwhois past present and future.
SAM VAN KAMPEN: Yes, thank you.
Yeah, I think I know how the clicker works, OK, thanks, this evening I am here to talk about RISwhois, it's past, present and future. The slide tells you. A little bit about me, I'm Sam, I have been working at the RIPE NCC now for about a year. And yeah, my main job is in the internet measurements team on RIPE Stat, occasionally it's nice to do something else so in this case RISwhois and this is my first RIPE meeting so yeah, I am excited to meet you all and if you want to chat about RIPE Stat or RISwhois, feel free to come up to me.
OK, so what is RISwhois, I assume a lot of people already know but the idea is essentially we have this collection of route collectors around the world and we would like to give sort of a Whois view that's based on the data that we collect so you can see on the screenshot here, you can query for prefix like one that we announce with any /21 and you get an RPSL object telling you there's a route, it's announced by this ASN, and you get some visibility information so it's seen by this many peers and this is when it was last seen, and this is the simplest form, it can get more complicated, you can do more or less specifics or do inverse look‑ups based on ASN and there's more stuff. So yeah, it's a really nice tool to get a quick look at what the state of routing looks like from your terminal or end scripts, etc.
So a little bit about its past. It's been around for actually a really long time. It was recently released in 2003, that's 23 years ago at this point and as you would expect from technology to the time. It was written in C, about 4,000 lines and it's contemporary technology, there was no multi‑threading because multi‑core systems were of course just getting started, the server was a forking server and unfortunately for us there's really no test suite to speak of. And that means it was kind of, yeah, tough to extend sometimes and yeah, it's really been languishing for a long time kind of until now.
So into the present. You may have noticed this if you are really up to date on the RISwhois news because there have been some user visibility changes, but we have been working over the last couple of months on a drop in replacement. RISwhois plus, yeah, it's written in Rust, from the ground up but it's intended to be fully compatible, you shouldn't notice many changes if any at all. And the goal is essentially improve the speed, maintainability and extensibility and so to to that end we added multi‑threading all over the place, the ingest is optimised, the server component is optimised and doesn't do any forking server shenannigans any more, it's sort of state‑of‑the‑art async IO server, many worker threads and so on.
On the back‑end behind side, it's now got extensive test suites so we're much more confident making changes to it and it has built in metrics now, from an up servability perspective, it's really nice improvement.
Then fun fact: This is the first production Rust service that we have launched at the NCC ever, I thought it would be fun to say a little bit about the developer experience and I like to joke that using Rust allows me to be dumb, maybe that's a bit excessive but at least it conserves brain power, right, so this old code base is a bunch of C, it's not even that many line but consider adding multi‑threading to that and other stuff in 2026, yeah, it makes sense to switch to a language that prevents a lot of food guns for you, and so, on the language side it's a nice improvement and on the tooling and library ecosystem sides. So we make use for parsing MR 2 files, it's wonderful to use and nice to use what the community has created there. And prefix tree for of course if you need optimise data structures to do this kind of thing fast and yeah, it's been, it's really nice to have that kind of tooling and library ecosystem as well.
Yeah, and in general just notice that using Rust, it has a low barrier to doing it right, whether it's a concurrency or stuff like the built‑in testing framework. It's simple but it pushes you to do testing right from the get‑go so, yeah, honestly been a really nice experience.
Then on the improvements side, this you may have noticed in the last few weeks, we have increased the ingest frequency, we are updating the data hourly instead of every eight hours. In addition, we are experimenting with parquet dumps, we have noticed when you are doing data analysis, it can be nice to use sequel, you can access the parquet dumps, the information about it in the documentation, it can be efficiently accessed, it has range requests and everything, it's quite nice.
Then other than that, we are adding support for giving RPKI route origin foundation status, you can see on the screenshot if you pass the ‑‑ R flag you will get extra attributes and RPKI attributes for any relevant VRPs. It works in the dumps and you get extra fields there, we added behind the flag because we noticed it broke our own parsers, we thought it might break other people's as well.
Then lastly of course, really nice has been the improved performance and I have some benchmarks.
So first we are looking at ingest performance, that's essentially we have these 22 correctors that we load and you see at the bottom the orange one is line X, 840 seconds, that's 14 minutes, that's how long it takes classic RISwhois to load these dumps and just switching to the new implementation gets you have a 5 X speed up and skills very well up to eight cores. And you can get essentially 48 speed up, it stops improving thereafter because of the units of work, we split it into B views and the biggest takes about 19 second it's so ingest performance is much improved.
Then of course the other component is actually serving requests. So what we did was we replayed query logs and so the units are a bit weird here because essentially they are in connection logs per second, so these are not individual queries but connections as a whole.
Essentially because it's a bit unrealistic to replay as many queries as possible, that's a realistic workload. But ‑‑ it's not a realistic workload, you can see have have time to get into the whole thing, you can get a couple of how X speed improvement as compared to classic RISwhois and a lot of that will just have to do with the fact that classic RISwhois is forking so that turns out to be extremely slow.
Yeah, so that's sort of the benchmark stuff, OK, maybe you need to take that with a grain of salt but we have some evidence that the performance has improved in the real world. So you see this graph here, this is from one of our grave dash boards, it's the title is a bit weird but it essentially measures latency and you can see when we deployed the new version, latency to do a desk query goes from 100 milliseconds to ten and this is a nice message I got from someone on the registry services team going oh so that's why some of your scripts have become five times faster so that was really wonderful to see.
And on the back‑end side also, we used to run RISwhois on 22 different nodes and that's down to four now and we could go down even lower but four is nice for the redundancy and you have the two data centres and so on. So that's where we are keeping four but if you look at the CPU usage graph, it used to be 22 nodes that are hovering around 6 0% and now each node is like 4% CPU or whatever so yeah.
So that's the present. Then the future. At present we don't really have that many plans for what we want to do with RISwhois in the future but we have some ideas, of course we did hourly ingest so we are thinking maybe we can do even more frequent ingest, something based on update, but we are not quite sure about that yet.
And the main question is what are does the community want from RISwhois so using as a RIPE Stat developer I think it's quite often useful to find weird prefixes and I use it quite a bit for that, it's unclear to me what the situation is from the community so yeah, I would be interested to hear from you guys.
I have a question slide but I don't think I have time.
MATT PARKER: Actually you are in luck, I am pleased to say we are running a little ahead, we have a couple of minutes, if anyone has any questions for Sam, please come to the mic.
Please keep them brief though.
AUDIENCE SPEAKER: Do you plan to do a time select thing so I could say I want this for logs of 23rd January 2024?
SAM VAN KAMPEN: No.
AUDIENCE SPEAKER: Are you planning to do a bulk thing so I can just send a whole list of prefixes?
SAM VAN KAMPEN: You can do that already, you can open a persistent connection and send 100 prefixes in a row and it will send you responses for all of them.
AUDIENCE SPEAKER: Not in JSON but just in...
SAM VAN KAMPEN: It will give the RPSL. Alternatively you could download the dump.
MATT PARKER: OK. Over here?
AUDIENCE SPEAKER: So I was wondering are you able to provide history, historical...
SAM VAN KAMPEN: So the RISwhois data set is based on the B views that we have that can be downloaded from data RISwhois.RIPE.net, we don't offer sort of the derived data set essentially but you can get all the source data so yeah and no.
MATT PARKER: OK. Thanks
ANJA GENGO: Sam.
(APPLAUSE.)
Next we have Ties from the RIPE NCC.
TIES DE KOCK: I am in luck because I don't have to do an introduction for RIPE RIS but I do want to highlight this is a project I have been running for a long time, it's our RIPE route collector project, it has been running since 1999 and the important thing to realise here is while RIPE NCC runs the infrastructure, we run this together with the community because you guys provide the BGP feeds, we cannot do this alone. It's a three year collaboration.
We have a set of 23 route collectors worldwide. We have a cluster in Europe at IXPs, we have collectors worldwide and we have also a few collectors that are multihop, that's where we accept a BGP sessions over the internet and that's where the blue dots on the map come from, those are multihop BGP feeds, those are the peers that are not where the BGP peer is not at the collector.
The main data project we produce are these multi‑trail routing tool kit dumps. We have textual views, binary views and the BGP updates. We group into files per five minutes so you can follow those and replay those yourself in your own tool.
Since last year we publish parquet files that are easy for data analysis, on which I still need to do a blog post, we used to see extensively in RIPE Stat, you can follow the RIPE data via RIS live, also there are a lot of third party tools that consume this data.
One thing I want to show is from RIPE Stat that we have improved over the last year which is upstream visibility Widge it, I want to take a look at a small routing incident, this is a /24 prefix, we look at zero from the origin, we look at the origin and we see there's a visibility changed over time, so on the 29th, we see the visibility popped up and went way way and came back up, it was visible to all collectors and there's a dance repeated and if we look at the first hop from the origin, you see that in one case the upstream is this ping purple AS, when it's fully visible you see a mix and it repeats again and what happened here was that there was a force origin sub‑prefix hijack where the purple pink period is where the hyjacker announced a prefix and where you see the mix, it's a legitimate holder in this prefix and if you have recognised it, this is a recent hijack of /24 prefix by Hessler. So this is the data. Let's go on to the data collection problem I am facing because we have been running this over a long time and adding route collectors over time. 2015 we started to actively expand the number of collectors and BGP sessions we had, if you look at what you collect, you see the volume really increased over time, if you look at the updates, the volume increased with with the number of peers that we had and the size of the internet and the size of the rib domains, it exploded over time and luckily we were saved by the implementation change improved the compression, we noticed we have a data size problem and that's where we now get back to that.
Now, if you dive into the data itself, and here I am looking at a full model of BGP updates from RIPE RIS and I am looking how much data each peer sends and how many prefixes I see, you can see two clusters, one for IPv6, you see that the amount of data that's sent basically grows to the size of the table and as a cluster below 5 gigabytes for a full table, and for IPv4 you see a similar cluster around 1.2 million prefixes, and you also see some peers that are chattier and it's hard to see the distribution from the previous plot, when you look at the distribution function, you see send peers that send a partial table and you have full tables which grow in size and at the end this is a log arithmetic I can plot, we have some peers that sent us above ten, up to 100 gigabytes of data a month which is more expensive to store than 5.
And it's really important here not to blame the peer that sends the data because what we capture is really happening on the internet. So what we observe is really happening on the internet and that's what we want to capture and when we contacted the peer that's sending this data because we temporarily depeered them, they often do investigation and tell us what happened so sometimes it's internal to the network and sometimes it's BGP churn from their upstreams. So these are valid observations, we see that peers sent, yeah, all send different data, all sorts of distribution and it's hard for me to determine what data is relevant for the users. For example you can look at a number of unique BGP communities, you see, again over a month where you see that you have some peers that sent a varied set of communities and in our table size, if you look at the aJason sees, at the AS paths without pro pence and you look at how many ASNs we see next to each other, similar shape with some peers sending a very high volume of very high, very large numbers of unique relationships, then a stable for full table and it tapers off the table size. But what you want to do is select the relevant information both for operators and researchers and that's why we need your feedback.
We historically had an open peering policy where we then started to actively add additional peers and that caused the data volume to grow. We switched to selective peering policy and that was a response to the big data volume, but what we should realise here is selective outreach and selective peering resulted in some of our most unique feeds when it comes to what we capture. On the other hand, we also know that what we used to select the best peers for us was very hard to apply, hard for us to make a choice when we got a peer request, so this criteria we had, we are were not perfect for us to do.
What did we learn over the time? We tried to get sessions from IXPs and remotely and we knew this was hard and we felt it's really hard to get a session with route server from across the internet, it's not possible basically.
And on the other hand, it's a lot of extra work and you want to avoid that because it's expensive to do operationally. Now what we learned about peer selection is that countries don't necessarily... a lot about BGP topology is available there and what you want to capture is varied BGP topology and not necessarily these things locations, geography has a huge influence for example the biggest number of prefixes including some deaggregates is from Uzbekistan but different location doesn't guarantee a unique view. For us the question is not where the peer is but what the unit contribution of a peer is
That's part of our optimisation problem where we want to really capture that's happen on the internet worldwide and where we have to balance the amount of information that we get, what we store and the long‑term storage cost of storing that data because hitting one of the two extreme is very easy but our goal is maximise the useful visibility of what we capture while keeping the permanent archive size and operational costs sustainable an also make it sustainable for research to use the data because if we capture a lot of data and researchers use two out of three RFC and papers, it means that 90% is not being used.
On the other hand we can't not just accept what we get and what's easiest for us to get because we need to cover the represented visions and cover the whole world fairly, so we also need to consider adding collectors to increase visibility and we should start thinking about what a new collector starts to add, you hit the core issue because the hard question is how do you know what it will add before you add a session or what the collector adds before you add it.
So we want your feedback on, we prefer full tables, where we tried to get different views, we also start accepting multiple sessions from an AS if they have different topology in different regions and where we focus on the IXP route service sections where possible. We can be easier when adding a peer at a route collector at X because it's less loaded but if it's a multihop session we can be more picky with what we add and the controversial thing is that I want to propose that we measure the data first, so we accept a session first, and evaluate it before deciding if if it goes into the long‑term archive because the storage problem isn't in storing data for two weeks, it's in storing it forever. So what I want to do is to propose to first establish the session and reserve the data and decide on archive al at the end and I want your feedback what data we should include in this archive and how we can expose the data that we only have for a short period.
So yeah, I need your feedback on what to select and what data should enter the permanent archive and I need to know whether it's acceptable that once the peer does the work, you set up the session, we evaluate it and it doesn't make the archive, if that's acceptable for you, of course we will try to use all the data we get in our tools that use short‑term data but we cannot accept everything and store it forever, if we capture the same thing twice, it doesn't really add something.
That's my presentation, I am totally out of time. Please talk to me this week if you have feedback.
MATT PARKER: Yeah, we have two minutes for questions if there's anybody who wants to come to the mic, it is two minutes only. Otherwise for a lengthier discussion, I am sure he will be available.
No? OK. Thank you. OK the final talk for this session today is Tobias fee big and he is going to talk about TTL jumps.
TOBIAS FIEBIG: Hello. It's me again. So TTLs, fun thing, mostly important in traceroutes. And I think we all still kind of remember how traceroutes work, we do have a path and we start with sending a packet with a TTL of one and we get from our first router, well we get an sample back and a destination unreefable, TTL expired, we do that and we send a TTL of two, it goes one hop further and we get our responses, it's a nice tool, you can use it to monitor networks and you can use it in measurements for example, and this year is measurements for a paper where we were looking at liberty global and their fragment Robin they fixed last year but to be able to do that, we instrumented RIPE Atlas to demonstrate we had inbound passes for different measurements set ups via liberty global or not via liberty global because we wanted to compare the two and I set up RIPE Atlas and I plod the data and I found something odd especially in IPv6 which is roughly 12.5% you will see in grey, which are saying I do have a direct adjacency to the source AS of the RIPE Atlas probe which included small ASes like orange and AT&T and I am very, very certain I never configured any session with those two so I was a bit puzzle why like 12 somethingìsh % of IPv6 measurements like traceroutes, two hosts in my network, would actually show such adjacency and interestingly like the label says number of times AS 211, 286 is in the parse equals one which is kind of well something that cannot really happen because well I have more than one router in between my border and the measurement set up so it's odd.
Well here's what happened. We did do the TTL thing like we saw in the previous slides but there was a router that was like well you know what, the packet you have been sending with a nice TTL, it was a nice TTL of three and arrive at me it was a TTL of two, let's just put a TTL of 255 in there and send it along.
Which of course means that it just reached the destination, we never saw another destination unreachable TTL expired and we see in the traceroute that seemingly direct adjacency which is kind of odd and annoying.
So what we then first did is grabbed like basically all atlas probes under certain conditions that was like 11.3k IPv4 and 5.7k IPv6 probes. By the way if you are running AS and don't have a probe there yet, please deploy one. If you have a probe and that is IPv4 only, please also add IPv6.
What we found is that there's total of 63 ASes where we did find some form of rewrite, it's a bit more frequent in the number of probes for IPv6 while it's a bit more frequent for the number of ASes in IPv4.
And those rewrites happened in various locations, either in the source AS, we would only have basically two ASes in our traceroute like the destination and one or two hops in the source AS or even somewhere on paths to so it would happen to transit traffic, we would see this in large tier ones like at at that time and orange and more carriers and some frequency of cloud networks.
Then we thought like well maybe this is not a new thing and we actually did have access to some RIPE Atlas traces, we did this for V had and V6, the slides only say V4, we find like 1.8 million traces there that result in some ICMP error and for those 86 Xìsh contain in the response a TTL of more than one. And when we tried to clock all these traces we see that there's of course a nice one which do not have any rewrites and then there's a couple of ones that contain a TTL larger than one, that can happen for various good reasons.
But then there's also roundabout 23,000 traces which most likely have seen a rewrite and you already see like the clustering of these is pretty close around 64, 128 and 255.
Well. There's one additional point of noise which is a liveness check, RIPE Atlas does not get a response at some point, they are like hey does this thing actually live and they send a TTL of 255, those of course we had to filter and they also exist in our data.
We looked at this over time and it appears that traces that are doing this become kind of more and more frequent over different ASes. And especially for IPv6. Like the more and more frequent but there's a couple of ASes where this disappeared again, there's a couple of ASes that started again but usually it is somewhat persistent. Well we have no clue. We do have a couple of hypothesis why somebody would do something like that. We have seen it in different environments so we reached out to a couple of ASes where we saw it relatively reliably. We had one axis prider, it was a known bug in the next generation network operating system in the larger operators we also saw this happening mostly at the aggregation level of the access layer, we saw it in an operator with an MPLS core and saw it with a Juniper MPLS core, none of the people intentionally rewrote TTLs and like one conversation we had pointed towards Broadcom Asix having a single stage pipeline to push from the outer to the inner when a packet is dual encapsulated, it's a single stage pipeline, loses that additional meta data and if you don't know what to do, you better put in 255 working hypothesis at the moment, if somebody knows something about that, please share.
Thank you.
MATT PARKER: Thank you. We have five minutes for questions.
JEN LINKOVA:. Thank you for bringing back good memories, 20 years ago when I was trying to put IPv6 in MPLS I discovered a certain vendor was sitting 255 was sending in the outer and when I asked why, because it was happy and currently for V6, the answer was because nobody asked us to do anything else. However, that problem was fixed and at least that vendor was copping it back, so I suspect here there are implementations, nobody is using V6 so nobody noticed, I wonder if it might be somehow related to a service 6 or something, maybe that's the reason you see, I don't know, right. So yeah, it looks like or maybe it's configuration because I believe you can hide your infrastructure, right, you can say I don't want to expose my struck but in this case it should be consistent between ingress and egress, thank you, that's exciting.
TOBIAS FIEBIG: It might be a thing where it goes towards what we saw debugging with one of the providers there they observed it that if they had packets through one of their upstreams and they ensured it didn't leave the Metro, that the TTL wouldn't get rewritten if but if they left the Metro area, double encapsulation thing. Like...
AUDIENCE SPEAKER: Believe it nor not, not a RIPE Atlas question. This is fascinating because the primary use for lowering the TTL is for routing loops and what you use is a side effect of this mechanism, right, so the question is, if this remight go is happening and there is a routing loop, are there eternal packets in the internet, and if so would we ever know?
TOBIAS FIEBIG: Theoretically I think you could build a model where you calculate at which point that accumulating background radiation of undying packets congests all links, so I am pretty sure they are not undying packets.
AUDIENCE SPEAKER: Maybe they also killed the router.
TOBIAS FIEBIG: I am also doing security research, I thought how I could maybe make such packets.
AUDIENCE SPEAKER: Please don't.
TOBIAS FIEBIG: As the rewriting is usually happening like at the axis layer, it's relatively hard because you basically just Dos your own axis, so that is a bit difficult. But this was very helpful so I found this really fun thing if you have a routing loop you can use RIPE Atlas, or could use, because I imported stuff and to do like 10 to 100 gigabits of link congestion, for that it's also useful because you can get a rewrite closer to the source and get basically 250 times the impact.
AUDIENCE SPEAKER: Thank you.
MATT PARKER: OK. Thank you Tobias.
(APPLAUSE.)
So that brings us to the end of this session. Just to remind you that at 6pm we have a BoF here in the main room about the IETF and a BoF in the side room about Layer 3 internet exchanges. Thank you very much everybody.
(APPLAUSE.)
Coffee break.