RIPE 93.
Main hall. Tuesday, 27th October 2016.
9am.
Plenary.
BABUK FARROKHI: Good morning. We are about to start in 30 seconds. So please have your seat.
There are a couple of reminders or announcements I have to make, PC nominations are until half past 3pm today, so if you want to nominate yourself or nominate someone else, you will have until half past three today. And RIPE NCC members to attend the general meeting, please make sure you collect your badges before 5pm today from the reception.
With that I'd like to invite Thomas Holterbach to present expanding routing visibility at IXPs with BMP.
THOMAS HOLTERBACH: OK. Good morning everybody and thank you for the introduction and so yes I am going to talk about how we can expand routing visibility at IXPs with BMP. OK. Looks like the clicker doesn't work, maybe, OK, now it works.
OK now it works. OK, thank you.
OK, so yeah, let's talk about the basics, so what is an IXP, an IXP operates layer 2 mid work which is called the exchange fabric fabric fabric fabric, it's a place where next can connect the router and then they can establish BGP sessions between one another, so these sessions are called bilateral sessions or direct BGP sessions.
And you can also connect to the route server which is operated by the IXP and then once all these BGP sessions are set up, the data traffic can flow through the fabric.
OK there are more than a thousand IXPs scattered everywhere on planet so they are very large IXPs that many many hubs, there are also smaller IXPs are a few dozens of participants so today this is going to be the outline of this presentation, I will show you why routing visibility at IXPs matters, then what are the existing approach that enables routing visibility and why BMP can help, and then what's the road map towards BMP based monitoring at IXPs and finally we introduce BMP routes DDosier a platform I am being with my colleague which enables BMP based monitoring and I will also conclude with the frequently asked questions.
OK, so why routing visibility at IXPs matters, first it matters because IXP nowadays are really particular parts of the internet infrastructure, I think all of the networks rely on IXPs for connectivity and MANRS which is a group of network operators that agree on some norms for routing security that actually ask IXP operators to provide monitoring and debugging tools for better routing visibility, BGP LookingGlasses.
So routing visibility in fact helps IXP operators to better secure routing at the IXP, for instance we can use a visibility to better detect and mitigate routing incidents like BGP hijacks, route leaks, also we can better identify bug announcements and misbehave participants but routing visibility also helps network operators in their operations so for instance if we assume we are a network operator and we are already connected to a few IXPs here in red and now we want to put in a new router somewhere and we have three possible candidate and we want to connect to an IXP, another question is where you should connect your next router so you have three possible candidates in green and you need to decide and to decide routing visibility can help because it can tell you which prefixes are available where and then you can better optimise your decisions.
Aside from that, routing visibility can help you to decide whether or not to connect with the bilateral sessions to other participants or just connect to the route server, instead and finally it can also troubleshoot reachable issues, here for instance as an illustration we can take prefixPand focus on AS 2 here and imagine we want to reach prefixPat the top and we know that we should use this BGP pass that goes through the route server and to reach that prefix, if we realise maybe during the night we this transient path was used instead, we need to troubleshoot because maybe this transient path is more expensive so we need to troubleshoot to prevent this problem from happening again and here routing visibility can also help.
OK, so now what are the existing approaches. So a simple approach is just to connect a monitoring platform on the right here to the route server which is a BGP session configured with add path and we can get this data, we store it into a database and we can provide some tools to analyse this data.
In addition to this, we can also connect to some of the routers of the participants with the BGP session and get data from there as well. So then we have even more data.
And actually there is a platform that exists and that does this, it's PCH, so PCHconnects to about three hundred IXPs and then they have sometimes BGP sessions with some of the participants or connect to the route server and the data is declared valuable, they have a public Looking Glass, they maintain a public archive, we as operators, as researchers we can use this data to make analysis to study for instance BGP hijacks so this is very useful.
The problem, though, is when we use BGP sessions to collect the BGP data, visibility is going to remain limited.
Why? Because the paths that are filtered will not be announced over the BGP sessions, we will not see the peer state information, so we will not know when a peer goes down or up, we'll not know this information.
So if we take our previous example here, we have this transient path for instance and now we have this data coming from BGP sessions and we want to troubleshoot, because the visibility will be limited, it will be hard to precisely understand what happens here.
Particularly we cannot answer questions such as did for instance AS 4 change its export policy or did the session between AS 4 and the route server go down. Or even maybe AS 3 announce a shorter pass during this time, these questions are harder to ensure with BGP data because we have limited visibility.
OK, so the good news is that there is an alternative approach which is to actually get the data from the route server software directly so we can get this data particularly and the good news is here is that it's like if you were in the CLI of the route server more or less, we see the packet paths, we see the routes that are filtered and we see other peer state information so these very useful, we can store it into a database and we can provide some tools to analyse the data
There is a very good example which is the Alice Looking Glass, it's used by many IXPs including DE‑CIX, I.S.M. IX and Linux and it pull willing the data every five minutes,s in configureable, there is an API if you want to access it automatically
This is good. The problem, though, is here we get only snapshots of what happens, we don't get a stream of data, so this means if you want to troubleshoot this problem, again we will have limited visibility, and typically we have limited historical context so this means, this kind of questions such as when did the change occur or which sequence of updates triggered that path change or also did activity on the bilateral sessions contribute to, it's hard to answer these questions.
OK so now I am going to tell you why BMP can help. So BMP stands for the BGP monitoring protocol and it's very nice because it can export routes at multiple stages of the BGP processing pipeline, and a side of that it also can report peer states information. So let's take an example. So we have this BGP router which is connected to three peers, AS 1, AS 2 and AS 3 and they announce BGP updates, these updates arrive to the router, they might be filtered by infound filterers, they go through the decision process and the RIB is allocated and finally the best paths announced to the peers if they are not filtered by the outbound filters, this is the usual pipeline in a BGP router, what we can see here is there are five stages, they are two stages on the left in the Adj‑RIB part of the pipeline, before and after the inbound filters, there is another stage here in the RIB, these are the best paths, it's called Loc‑RIB and finally there are two more stages in the Adj‑RIB out parts of the pipeline, one before and one after the outbound filters. So now if we connect this router to BMP collector such as Rotonda or open BMP, then we can really instruct that router to export routes at one or more of these stages.
So now what about route server, because route server works almost as a normal BGP router but there are differences, the difference of the goal of the route server is emulate the full mesh of BGP sessions and because of this, the internal pipeline is a little bit different compared to traditional BGP router, it actually depends on the implementation, but the good news is that Adj‑RIB in and Adj‑RIB out, these stages remain the same, they are identical, they are inbound and outbound filters and we can export data at this stage as with a normal BGP router so the good news is that with routeservers we can definitely use BMP.
So how would this look like in practice, well it's pretty simple, we have the monitoring platform on the right and we just connect it to the route server, with the BMP session this time we can instruct the route server for instance to export data for the Adj‑RIB in and the Adj‑RIB out stages.
So now if we do this and we want to troubleshoot this issue, it becomes much more easier. Why, because again we have to the stream of data an all the data so all the back up routes and the routes that filtered and all the peer state information, for instance we can know if the session between AS had and the route server went down, we know this information. We also know the routes are filtered, for instance we can inspect pre‑policy, post policy and back up routes and we really have the raw stream of updates so we can really identify which sequence of updates triggered that path change.
And this is really useful to prevent this problem from happening again.
OK, so of course we might not also be able to connect to the route server with BMP so the good news is we can also connect to the routers of the participants. Here principles we can connect to the router in AS 2 with a BMP sessions and we can configure it to export data for Adj‑RIB in, only but if we just do this, this is already quite a lot of data, in fact it would export all the routes coming from AS 1 and AS 3 through the bilateral sessions, all the routes coming from the route server, this is likely more data compared to if we had just one BGP session with the route server for instance.
OK, so now what's the road map towards BMP based monitoring at IXPs?
So there are multiple stakeholders involved, the first is route server software maintainers, their goal is to make sure the BMP is well supported and all the RFC are supported as well. Why? Because BMP was introduced ten years ago and then it was gradually extended over time so first we had Adj‑RIB in and then Adj‑RIB out and then Loc‑RIB, they support all the RFC now, it's getting better and better, the current state is FRR supported Adj‑RIB in and Loc‑RIB and BIRD supports Adj‑RIB in only, there are still some work to do to support more RFCs but there is great, it's enough to get quite a lot of data.
Now for the BMP data collection platforms, they are good their goal I believe is to have efficient scalable storage schemes and efficient also data access models because BMP exports more data compared to just a BGP sessions, this means we need to be equipped with tools to effectively store this data and get this data quickly as well.
The current state is we are still very often relying on MRT which is not the right way to store BMP data, it analyses too slow, instead what I think and what we did with BGP routes, we tend to rely on APIs, we provide access to with an API.
OK, IXP operators, their goal is deploy BMP security on route server and for IXP participants, their goal is exports BMP data to data collection platform which makes the data available to to benefit the entire community, researchers and operators. The current status here is there are IXP operators and participants already exporting BMP data, but we need more, we need more data so this is also why I am here today to try to convince as many of you to connect with the if possible to do this which makes the data typically available.
Now I am going to introduce you to the platform we are developing and which is BGP is monitoring in practice. It's super simple, we have BMP routes on the right side and connect to the route server and to the routers of the participants with BMP sessions.
Then we get the data, we store it into a database and provide tools to analyse the data. So BGP routes.iois built to support network operators and researchers, everybody can contribute, if you are an IXP, you are very welcome to connect with BMP actually everything is automated, go on the website and fill out a form and boom, the session is enabled. Super simple.
Then all the data is publicly available so we really provide the raw data publicly and we have a data sharing policy which is essentially almost identical to the one from RIPE RIS for instance.
In terms of the data access, we have different ways to access the data, you can access it with public dashboards or API and we try to maintain an MRT archive. That's the few ways to access the data.
There are already three IXPs that have connected their route server so BGP routes with BMP, these are smaller IXPs, it's a proof of concept that it works, one actually uses FRR routing, the two others use BIRD and it's just, it works fine and by the way I want to thank Alarig for configuring these BMP sessions, I think he is in the room so thank you for that.
Aside from that, it also collects data from IXP participants through 60 BMP sessions and from these feeds we get visibility into about two hundred BGP sessions between participants and route server and more actually now than 1,000 bilateral sessions between IXPparticipants and between the participants and this is quite a lot of data, if you want you can already analyse.
OK, so we also like did a new dashboard recently about a month ago, we released it, it's called the IXP explorer and from there you can look up for an IXP, for instance here the Amsterdam exchange point and you will see on the left the two route server that AMS‑IX are operating, and you will see they are not connected with BMP, but below you see all the participants connected to AMS‑IX and you will see many of them are also connected to BGP routes with BMP. Actually see par tech is a large network in French but it spans many countries in Europe, and see par tech has a router in Amsterdam, which is connected to AMS‑IX and which is also connected to BGP routers with BMP and I want to thank you the operators of see par tech for having configured the session and all the other operators who are connected to the BMP with us, I think there are many of them in this room, so thank you to all of you, if you click on this button here, you can really explore the data so you can see the two sessions that the router here in Amsterdam has with the two route server, there are two sessions and we can see that these two sessions basically send about a little bit more than one every thousand routes to the router at see par tech and you can also see the bilateral sessions, there are more than 150, you can see all of them, you can see the number of routes the router is receiving and there are piece buttons here on the right side, you can click on these buttons and then you will open another dashboard where you can really see the raw data the raw updates and the RIB at any point in time and explore the data and do some analysis.
If you are interested you can access to this data programme atically, we have an API here, it's simple to use, there are just three end points, vantage points updates and RIB, there are parameters, so we have documentation available on the website if you are interested.
And I just want to quickly show you how it actually works. So the vantage points end points let's you identify a set of vantage points from which you then want to get the updates or the RIB.
So basically what it means, it means if we have this router that we were using before as an example, we can say for instance I want all the vantage points connected to that router so that means all the peers connected to that router in this case.
Once you have selected a set of vantage points with this end point, you can call the updates end point, you can give us input, the vantage points you have selected and you can also specify routing stages and then you will get all the updates up served by these vantage points at these routing stages. And the second goes for the RIB end point. So if you provide us input a set of vantage points, if you provide as input the routing stages from which you want the data and a time, then it will return the RIB studies at this point in time for this vantage point and at this routing stage.
So again if we take our previous example, if we query the RIB end points with a parameter vantage point, is it going to peer 2, instead it's Adj‑RIB in post, then we'll see the RIB starter at this part of the pipeline between the inbound filters and the decision process. That's what you will get.
And actually there are many more parameters that are supported, including for instance filters based on the RPKI, ROV and ASPA statuses. You can ask to IP agents to return RIB entries for which the ASPA status is downstream invalid. OK, this is useful if you want to know if the route server has accepted routes that are marked at ASPA downstream invalid and then this is useful for instance to detect route leaks that might propagate through your IXP for instance.
If you want to learn more about how we implemented ASPA, we have a blog post where we explain everything, we follow the ASPA internet draft, we let the user pick the one they want and finally the validations when the route is received by a platform, not when the user queries the data, so it makes the thing a little bit more accurate as well.
And to conclude I just want to talk a little bit about the frequently asked questions because I presented it already a month ago at a conference and then I got some interesting questions which I think are nice to discuss here.
So the first question is can operators choose which routing data is published with BMP. So yes, here there is some fixability over what you want to export, it's particularly when you configure BMP in router you can specify for which BGP sessions you want to export the data. Here for instance this is an example with Cisco, we are just exporting the BGP data for two BGP sessions, that's it. If you cannot do this on the router, you can do it on the platform, when you configure the session, you can specify which peers, for which peer we should not start the data, that's the way to proceed.
So then what about legal restrictions and agreement with participants. Of course you need to make sure the participants are willing to share the data or are willing the IXP is sharing the data. But here what I want to emphasise is that I don't think there is any difference in terms of legal restrictions compared to if you were using like BGP sessions for instance like RIPE RIS or if you use Alice Looking Glass, in the end it's the same data that can be exported with BGP, it's just BMP is less subject to filters and packet routes, there's less probability about which router you will see, with BGP you will see things but you will never really know what you will see, it's probabliability depends but in terms of legal restrictions in my opinion it's the same thing.
So it's not tied to what I just presented.
Is an IXP that shares routing information via BMP better or worse? The first thing is if in my opinion greater visibility presents problems or limitations, it's a feature, not a bug. Now that said, if you are an IXP I wouldn't just share BMP data, you need to also share like other information like port capacity, latency, reliability and with all this information, then users for instance can better troubleshoot and they can make better decisions about whether to connect to the IXP or not.
Can sharing routing data help malicious actors? This is a question that I got, the answer is yes, of course richer routing visibility can help attackers better understand route policies for instance and also better assess whether the attack is successful or not, the thing though is that in the meantime it helps the defenders because it gives more visibility so we can better detect hijacks and this kind of things, from my experience I think actually the benefit is much bigger for defenders. Why, because one of the typical issues that we have today when we want to detect hijacks, we lack visibility, attackers can make hijacks that evade collectors because of this, if we have more data, it's going to be harder for an attacker to actually evade route collectors. That's it. I want to say we have a Youtube series, we have a video that explains this explorer, you can scan the code. If you like the project, you are welcome to connect and share data with us, thank you.
(APPLAUSE.)
MAX STUCCI: I see we have a couple of questions now.
AUDIENCE SPEAKER: If operators are concerned about the data they are exporting in BMP and there's some kind of filtering needed that is certainly something we could look at, I would prefer to not do because the point of BMP is export the data as unfiltered as possible but I would like to know that as the developers, we don't necessarily understand what prevents people from wanting to export the data there, like what is actually sensitive, what is it that the people want to not share so if you need something like that, please come talk to us so we can understand what the actual requirements are.
THOMAS HOLTERBACH: I am not sure what are the requirements, I can just expect in fact for some prefixes and for some peers the data is maybe more critical, for now unfortunately yeah, I also need feedback from operators. Of course I am also happy to get feed backs on that side as well.
MARIA MATEJKA: I have one note: Please don't speak with ASPA downstream in route servers in IXPs, even upstream should be valid or unknow, if you ask for downstream you are thinking about something being a buff, like a provider of the route server which is a nonsense, so please, in IXPs, it's only upstream, which is relevant, which is the stricter one.
The second thing, I am not sure about the performance of monitoring adjacent RIB out simply because if you have like 1500 BGP connections and one of the BGP peers exports, woman imports like 50,000 routes, you multiply this event to this number of routes every single time this use flaps so I would maybe recommend reconsidering what you are actually monitoring because the data you are trying to monitor may be like expanded the most brutal way at the worst point and there you are trying to measure it.
THOMAS HOLTERBACH: Yes, thank you for the feedback. Yeah, probably you are right about your ASPA recommend here, so what I presented is specific to IXP but this API can be used also for sessions that are not relayed to an IXP in which case I think it makes sense to have these two upstream and downstream validation but it's true for the context of an IXP, you are probably right. For the scaleability and maybe the Adj‑RIB out part and when you have sessions that are like exporting quite a lot of data, well probably I am happy to discuss more afterwards but definitely like scaleability is going to be a challenge, especially with the BMP, this is why actually most of our efforts, they go into developing these schemes that I was talking about, this is really, really challenging and this is probably the hardest part in all the pipeline we have designed, it making things scalable, it's not just scalable, it's also making sure you find the right way between scalable, fast at answering queries and ‑‑ yeah, fast and well that's a different trade off. Basically it's always a matter of trade off, that's basically the challenge and this is what you are really working hard on.
Yeah.
MAX STUCCI: OK, thank you Thomas, we have ran out of time. Thank you very much.
(APPLAUSE.)
Next we have Giovane Moura, we move from route to root and we visualise DNS in this case.
GIOVANE MOURA: You have the microphone, right. I think it would be nice to have it. So... yeah, good morning everybody. I am going to get the mic here.
While I get this thing set up, I present this thing before, this presentation, and it's an update version of OARC last meeting, let's see if you can hear me.
I think it's fine? Can you hear me I don't know, oh whatever.
Is it working? Yeah. Yeah so I presented this before at RIPE meeting, sorry DNS work meeting, I apologise to you folks who have seen it, it's an updated version, it should be all right. My name is Giovane with TUDelft, let's get down to this presentation.
How many folks here have seen this website, can you raise your hands please? Some of you. So we are running a dashboard called RootVIZ, that's the goal of the presentation here today. Yeah. And for folks who work on DNS these are the 13 root server operators, here's the list of them. They run their infrastructure, they are independent and they can decide pretty much how to run that, how to run that and that has implication as you are going to see now on the reachability of the servers and why people need to monitor their root servers, without root DNS, there's no TLDs or noting pretty much no internet, if it breaks the entire ecosystem suffers, so the thing is like you can measure using passive data but you know, it's much better to have active data that comes from vantage points like RIPE Atlas which is the largest measurement platform out there. So we use pretty much all the Atlas measurements they monitor with servers, the dashboard I did is just gets the data from RIPE, RIPE NCC is running, RIPE does all the heavy lifting, and we just visualise in an easy way. So every root server is being measured every four minutes, at least since 2010, and there are at least 14,000 of them and they do different types of measurement and ten years ago I we did this paper anycast versus DDoS, a root DNS attack, a DDoS attack at the roots and we did this graph, you see the dips it was exactly where the DDoS against the roots and this is the RIPE Atlas visibility of the roots during the time on the Y axis you see how many probes could reach each individual root server ladder so we thought why not do this thing we did ten years ago, dashboard, the data is there, RIPE is running this thing. They have DNSMON is with a subset of the probes, but why not all of them. So the things like DNSMON that a lot of operators use, they use the anchors and these are clean works, they sit in data centres, they are highly reliable but they are fewer and kind of misses the mass internet of the eyeballs and if you get all the probes, like 14,000, you can have a lot of issues but you see kind of more closer user experience so that's the idea.
So if you use other probes, you get like 14,000 more, 14 times nor vantage points and more everything, we have better coverage pretty much.
So you have a wider view of reachability and our approach is that since all the probes are not very reliable individually, each of them, we can't source that, if you see stuff in your rest rooms at airports, there's a lot of people reporting the same status of the rest rooms, they are going to send somebody to clean it which means one individual report is not really relevant but if you crowd source and get a lot of people saying the same stuff, maybe there's something there.
We kind of light up the data that RIPE produces anyway, the realtime monitoring crowd source reachability on that. So the pipeline is RIPE generates lots of data and that's a challenge not us for them but for for us, we download it and we have to filter some of the data because there's pro hijacks, all that kind of stuff and after that we kind of maybe we reduce the data by 99%, because we do aggregation and then we throw it in a Prometheus database and dashboard, my colleagues help me a lot with that. So that's what we do, we get data, aggregate, filter and throw it on the dashboard, that's pretty much it and I am going to OpenSource this code, I have to clean up the code but it's going to be there.
If you are a root operator, I think there are some in the room here, with the roots talking to them, what they really care is reachability. They want to ‑‑ latency is kind of secondary. All the root servers are very much replicated. You see here on the graph, the number of anycast instances that each of the different roots they have, F root has more, 89 and I think Cloudflare runs part of their infrastructure. So they are very dissimilar. So if you are a resolver you can choose from 13 roots and there are 1,500 sites for to get a response from. Just look at the reachability first.
So we also wanted to identify the root causes of time outs, it's kind of difficult because if you see at the spikes and the data, it can be in three different places. The client side, which is the RIPE Atlas, in the network like anywhere between client and server and then you have to find where issues happen, you can look at traceroutes and complement the data, you can look the traceroutes and look the banks and try identify, it's not very easy due to that.
So dashboards: This is the main dashboard. You see like a combined view of the number of probes, the percentage of probes up there. They get like, the time‑out rates here and here we have for V6 I think here, in each I have made a special dashboard for each operator here, they have their own thing, you can check it out later. For example, here you can see, I think this is A root, a spike here of 2,000 probes, so we are looking trying to figure out what's going on, there's a different baseline of timeouts for each root server letter and those how this dashboard helps, it allows you to identify eyeball issues that otherwise would be hard to identify.
And so in we look at how many ASes are failing to reach root servers, L root has a very large deployment and a larger root than other roots, this is kind of detect Tore, not really the root cause. Anyway the data was there, RootVIZ raises many more questions. Folks why they have different baseline time out levels and let's do deeper analysis. I presented this at OARC and since then I have got some group of students look into individually, some of these issues and I have a master student at TU Delft, so what if you find. As I said there's a different baseline, the root server letter, the more politically correct term letter now for the root server letters, they have a different baseline time outs so like 13 roots, different networks, different operators but RIPE sees them differently, let's put it this way. And why? Well we need to do research on that.
One other thing we found a lot of these spikes we see on timeout, we reach out to B‑Root and K‑Root and others and often we get a response that's very, it is a network, it's not the RIPE Atlas clients, it's not the server, the server is fine, it's something something happened the path between client and server and the root operators themselves they have no control over that and that's kind of frustrating for them, researchers as well, you can't pinpoint who is to blame.
And I think I also learned from that the root operators when there are time outs, people automatically blaming them, whereas in reality, the fault is somewhere often somewhere in the network.
So they take a lot of heat which they don't deserve, that's what I found under this research.
I also talked to folks at K‑Root, RIPE NCC. There is an event in April 26th, you see like on V4, 20% of probes do not reach the time here, K‑Root but not for V6, it was a transit provider issue, there was like yeah, it was upstream from us, it's where we couldn't do much about it and it faded away very quickly as well so. This one is more interesting. I had a student look into this and this one, this is like days here, you can see there's kind of a view up pattern, the percentage of probes that reach C‑Root run by Cogent goes up and down like a wave and that was the only root that was doing that, very periodic and when I look at the data, I was like I have no explanation for that but I'd like to know what's going on.
So only C‑Root, C‑Root shows that and the others not so I didn't guesses, anybody have an idea why this is happened for C‑Root?
So the reason is like my student Jess see did a deep dive on this, he did a lot of different analysis on this, I am going to simple here what happens let's look at the different points of the data to see what's going on, I gotten minutes of data here from RIPE Atlas before these time outs occur and if you compare them, you have an idea of what was going on, you have got a 6am in the morning and 4pm and a look at what happened.
So what we found is that like the bunch of probes, their only time out at 16 but they were fine before that, so 684 probes were timing out only in the afternoon.
But they were fine in the morning, so what it's very strange, let's look at the probes.
Most of them, 96 of them come from one particular AS, I am not AS bashing here, just reporting the data!
And some of the others, they connect via this Deutsche Telekom to C‑Root, so these are the locations of the Deutsche Telekom probes failing to reach C‑Root at a particular time here.
It's 16 o'clock, but not at 6 o'clock. So there was very strange and then when it was going on, Jess see found this stuff out and why it was going on and we suspect this, what's going on here.
So yeah. It's not a technical layer, it's a management layer, the policy layer, apparently Cogent has a beef with Deutsche Telekom, it has been going on for years and they sued Deutsche Telekom and I think they lost but what you can read on the news is that they haven't solved this and what we can infer from the data without knowing C‑Root is operated by Cogent, Cogent has a peering dispute with Deutsch Telekom, it seems like when the traffic, when we see like when German folks go to bed and the traffic drops, things are fine, when they start to wake up, more traffic, maybe the pipe gets filled, their policy, I don't know, and then see where traffic starts to get dropped and I think like for Deutsch Telekom, maybe DNS traffic is not really relevant, it's just a little, maybe it's collateral damage and we reported it to Deutsch Telekom during the OARC meeting and again I am not taking sides but I see the point I want to make here if this is the case of policy issue, I think you should be aware if you are troubleshooting stuff like this, be aware the policy may be the cause root here, I don't know, I suspect but we cannot prove it, that's what we have inferred.
There was another event with L root, thanks to the L root team, they helped a lot with this event, we had online meetings and exchange data, it was really nice working with them. Eyeballing stuff on RootVIZ, we saw this massive spike here for days. For RIPE Atlas probes time out for L‑Root, there was a baseline here around 1,000 would not reach over IPv4 and there was 3,000 for days.
And so again let's do a deep‑dive what was going on here.
So we look at this reach out for L‑Root and it was only affecting IPv4, IPv6 here in yellow, things were just fine. Other operators had no issues, we can we can rule out issues with Atlas so there's some there, something there what was going on.
So probes were not off‑line, we rule out everything else and start to look at the data, for example this is K‑Root at the same time, there no similar spikes or the probes could still reach K‑Root. So again let's take ten minutes of data before the event and during the event and process it and compare it but that is kind of manual labour we are doing now and we saw that 1700 probes time out only during these spikes but not before and that's strange and they are all over the world here, mostly in Germany and US. And you can see here the distribution here.
And 97 countries so look at the different, some of them could not reach at all, some of these systems, the 23 probes for example, this one here, 3292, 123 probes of Atlas does not reach L‑Root any more, some of the others could personally reach. This is, I need to do a look over here. We are looking at traceroutes that Atlas also provides, we went there, looked like a ten minutes before the event where the traffic was going to and looking at during the event and what we saw for this particular probe here on Deutsche Telekom now, this is a measurement ID and it could not somehow go to the nexthop AS up and I have no idea why. There's a bit of frustration as researcher, you see that why it stops but you cannot exactly identify what happened there.
So we did a lot of other similar probes, similar issues, different autonomous systems, sometimes the event route is completely AS from the original baseline, it's not something very easy to automate, I talked to Robert a bit about that yesterday too and we tried to automate more stuff here but it's not that easy.
And we thought maybe let's look at where these probes are time out, which catchment they had, which sites they were hitting during the event and a lot of them are going to Czechia, where they were going before time out. And I made a, there's a lot of information in this graph but on the Y axis is any anycast right of L‑Root and this shows the sites were running at the time, none of them went down, so other probes could reach those sites, so L‑Root was fine. So if you see this time out here, let's blame L‑Root, let's blame I can here but turns out to be somewhere in transit and the operate operators get a lot of heat because of that, so there's something in the path was broken, not the server, the server was fine and what it would do, we can do deeper dive figuring out what is going on, we need to have some automation but it's not always easy, not always the root operators fault and usually it's the network's.
So what have we learned so far, I think this dashboard helps a lot, the data that that provides is amazing, a lot of data but it's nice to have the visualisations to quickly eyeball stuff.
One thing I learned by working on the root operators, they are usually very friendly and responsive and open and I feel sorry for them because they take the blame sometimes they don't deserve.
Reachability issues: Root causes are hard to pinpoint. And the management layer, the policy layer place a role. And if you are a student here make some dashboard, its easy, get some data, plot stuff, eyeballing stuff and maybe that will help, that's how we found with RootVIZ.
So to recap a little bit, RootVIZ for, great for eyeballing, a realtime view across all that probes, we confirm root operators, we can look, now we can haves look at the more deeper dives and identify what's going on, like this spike, we identify issues but the other part here, look at which probes have failed, any Anycast size, look at the BGP information, transit, data plan, this is all man manual. There is a group that did OpenSource code that can do detection on the data and visualisation that the link is right here, it turns out it's much more difficult than anticipated. This was a softer project, not a research project, I had to tell them hey, let's figure out normal detection is now on the softer part, that's what you are going to get graded for. I had a second group doing MCC thesis on, on the 13 root servers and the PDFs are here, because of a bug, these things are not working in the data repository, so if you are interested in these reports for an operator, download them if you want to and I am going to have students look into this thing and try to automate the root causes of this reachability. I would like to thank the students that worked on that individually, suffered project like, I have the names here on screen and individual types and individual letters, and Josh with the others here and two of them have been to RIPE meetings before and. Road map now, the dream thing is like to have fall automation of event analysis, and provide public data sets and OpenSource, I have to clean up my code and I will publish on Github, the code is not outstanding, I am just processing the data RIPE collect and that's it, it's from dash measurements here, that's the goal, questions.
(APPLAUSE.)
MASSIMILIANO STUCCI: Thank you for a very interesting talk. So are there any questions from the audience in the room? .
AUDIENCE SPEAKER: If nobody else does, I will. Hans Petter Holen, CEO of RIPE NCC and we are the operator of K‑Root, really great research and presentation. This topic is discussed in root ops and ICANN RSAC so I will share this information with several operators so we can have a discussion on this, I think there is another dimension here that is discussed and I haven't seen any resource in and that is why does this matter. Because you know when I query the internet, there are so many other factors and it's very seldom we create a root server system, I haven't seen any research on why does this matter and intuitively, it does matter but what's the order of magnitude, I think this is great foundation for that work but having someone with a big resolvers may be sharing something about what happens from the resolver point of view and so on, that could be an interesting follow up research here. So thank you for doing this.
GIOVANE MOURA: Thank you.
AUDIENCE SPEAKER: Yes, out of curiosity, you said it perhaps, I thought there are three root servers, A, J andH that are not in the study. What is the reason?
GIOVANE MOURA: Yeah, the study, three of them are not validated by the students. I had five students, I could only give two letters to each of them, we look at the data, A and J were boring, all flat, can he couldn't see any issues, let's see the ones that are more interesting for you guys, I didn't have enough capacity, that's it.
AUDIENCE SPEAKER: Sebastian from Deutsche Telekom. Let me Finnish! We are happy to help you find out where the root causes is, find me later or I will find you. We want to resolve that.
GIOVANE MOURA: Thanks, great.
AUDIENCE SPEAKER: Two things. One, if a root server transit provider fucks off, that's the root server's problem, they need to fix it. The second component is operating the network that reaches the root server so it is very much their problem, so comment to that.
Second thing, one question I have around this the vast majority of eyeballs don't do full scope route resolution or root resolution or DNS resolution on their CPE, they will delegate that to a resolver. How relevant are the majority of the majority of the RIPE probes? Even if 80% of the RIPE probes can't reach specific root that's likely to not have a massive impact on the DNS resolution if they are using a recursive resolver in the ISP, it's how are you going to link a tonne of RIPE probes having outages with genuine reachability issues.
GIOVANE MOURA: That's a very good question and thanks for the comment. Nobody has a perfect view on the internet, how things work. So what are we doing here is the best effort. I agree with what you said, we are using Atlas as they would be sort of resolvers but most users are using other resolvers somewhere else, in this sense it's not really representative but on the other hand, it is because if they would be resolvers of those networks, that would be the behaviour and we are getting 40,000 vantage points, I forget the number of prefixes in our system, it's a subset of the internet and we are doing the best we can with that. I am not sure if resolver on the same AS would behave the same way.
AUDIENCE SPEAKER: I would recommend you add a validation probe lead that probes something else such as probing Cloudflare or quad 1 or quad 8 or whatever at the same time from the same probes because if those probes start having issues at the same time you are seeing shoes are roots, you can likely rule out last mile shoe issues a problem.
GIOVANE MOURA: Most of them are completely different and Cloudflare runs part of the root, so yeah, you are right. And about the comment you said about I agree that the root operators are sort of co‑responsible to upstreams but sometimes not even in their upstreams, it's between the upstream and the rest of the internet so it's, that's what I'm, it's complicated, yeah, thanks.
HANS PETTER HOLEN: Hans Petter Holen RIPE NCC again. I want to respond to the upstream thing. You are right right but also a call to action from anybody in the community, we provide hosted K‑Root notes, please contact us if you have a feeling that the root server system isn't reachable in your area at your internet exchange or whatever and I know one of my colleagues will present at the DNS working group, reach out to him to see if we should deploy more K‑Root instances there, there may be other root servers that have similar approaches to distributing this so there is really no upstream for those nodes, it's about bearing relationships and how the local topology is in the country so there are multiple place to ways to solve this thing so I just wanted to shout out for that. Thanks.
AUDIENCE SPEAKER: I don't work with L‑Root. I just wanted to say that an outrage of a single root letter has nothing to do with the DNS resolvability. The roof server system has not been down a single second the last 50 years, so resolving will always work at least at the root level, and so the question of the reachability of single letters is for the whole system not so important but obviously we can't let it deteriorate and so the reachability isn't important question for the whole system, to keep up the no downtime.
GIOVANE MOURA: Yeah, the root is replicated, that's why we never had a downtime on that but my contact with the video operators, they really want to see what's going on with the networks, they like that, yeah.
AUDIENCE SPEAKER: Last question. Beside you saw that in one case it was working correct for fail, in general, did you see a big difference between IPv4 and IPv6 as they have different rooting let's say?
GIOVANE MOURA: So, from eyeballing the data and without deep dives, they fail completely in the panelling and I would have to look at the data carefully, and I think it's per root operator. If you go to the dashboard, you see B‑Root, there's spikes on this sometimes and I talked to them and again upstream, so it's very operator specific, they have very different networks so the bottom line it's very hard to generalise.
Thank you.
MAX STUCCI: Now we are going back from roots to routes and I would like to invite Shyam to present detecting and characterising exposed BGP routers.
SHYAM KRISHNA KHADKA: Good morning everybody, I am Sam, I am a final year PhD student in the university of Twente, I work with BGP security and DDoS mitigation, today I am going to present our work on exposed BGP routers, in this work, we ask the question how many routers on the internet that run border gateway protocol are publicly exposed.
We all know BGP is the defactor routing protocol on the internet, it runs on TCP port 179, realising the importance of criticality and BGP routers, RFC 7454 recommends to restrict TCP port 179 to configured peers only.
However, there are routers on the internet which accept TCP connection from any host on the internet, they do not only accept the TCP connections they accept even the BGP open messages from those hosts for their information, BGP open message is the first message that's sent after the connection is established.
Then in response to those requests, those routers respond back with their open messages, followed by notification messages.
If it is an open message, it contains many information such as AS number configured on those routers, BGP ID which is also known as router ID. Along with parameters configured in that router, in the case of notification message, they simply send that error message to the requester to terminate the connection.
In our history, we define an exposed BGP router or those routers with response to unsolicited BGP open messages.
Now to detect those routers, we developed a tool and named it as Scan 179. Here I am showing the five line of our methodology. So the first is scanning TCP 179 using that using Z map, the objective of doing so is put in BGP routers on the internet. It is because there is no comprehensive list of BGP routers on the internet.
With ZMap scan, we get potential routers which we put as TCP 179 responders. The next step is sending them BGP open messages, which we term as custom BGP speaker. In this case we create a custom open message that response RFC compliant parameters such as BGP version 4, AS number of our institute, BGP ID configured with oblique address of our institute and other optional capabilities of a router.
Next step is we collect the responses from those 17 the responders, which store the full packet capture and analyse them to extract many useful information such as router ASNs and their characteristics as well which he will explain later.
And for information, it was a one shot internet wide IPv4 measurement we conducted in February 2016.
Here I saw the high‑level summary of our result. Not result, it's a preliminary result that was obtained from responses to Scan 179. With the ZMap scan, we found 6.25 million IP addresses that responded to 179, TCP Scan 179. Then we categorised those responses into three, the first one is BGP responders; it means those IP addresses send messages which are compliant with BGP RFC 4271. It means 5% of those responded IPs sent either open messages to our probes or notification only messages to our probes.
We found a higher number of open responders, which is 51% compared to notification only responders. And there are other two categories we named as non‑BGP and pre‑BGP category, in the case of non‑BGP category, it would be established TCP connection to them but upon sending BGP open messages, they are either don't respond at all or they sent TCP sent messages to to our probes which was 86% of 179 responders. And the part category is BGP categories which means they don't allow TCP connection to our probes.
They sent TCP recent or we opt in to connection timeouts from those targets. So ideally for a secure BGP routers, the last two categories are the recommended categories.
In our analysis, open responders formed the basis of our subsequent analysis, it is because open responders contains much information about the IP.
Then the next step is associating those responders with ASes. Particularly doing so is to map IPs to routers and then know which ASes are operating those routers.
For that purpose, we extract two types of ASes. First one is interface ASN and ASN, and the second one is open ASN. For interface ASN, we map those responders IP to ASN using IP info data and in the case of open ASN, we extract the ASN number configured in those open BGP open masses. We had two types of BGP responders, one is open and second is notification. In the case of open responders, we check whether the open ASN is public or not. If it is public, then it is straightforward to map that IP to that ASN, it means that ASN is operating with the router. If it's not public, it means it could be private or reserve ASN and those ASNs could be used for internal purposes by ASes and in that case we extract interface ASN using IP info data and that is the associated ASN for such cases. And in the case of notification only responders, we say it in 4 ASN because notification message doesn't contain AS number as in the case of open responder.
The next step is mapping those open responders to routers and ASes. So in a practical scenario, a router has a large number of IP addresses corresponding to a large number of interfaces. It means those responder IPs might belong to the same router. For that purpose we use BGP ID which is unique per router in an ASN. Then first we take whether the ASN is public. If so, we use the combination of BGP ID and open ASN to group to a router. And if it is non‑public, it means the same ASN could be used by multiple other ASes on the internet, and in that case we use the combination of BGP ID, open ASN and interface ASN to group those IP addresses to a router.
Since we already mapped IPs to ASes, we can easily map IPs to routers and then to ASes.
We furthermore characterised those routers as interim and border. Internal routers are those routers which have non‑public open ASN and those ASNs could be used for internal purposes also known as AS confederations.
And there are routers which we level as border, it means they are participating in eBGP peering with another router on a different ASN.
To find, to identify the border router we first take whether open ASN is public or not. If it is public then we check whether it's interface IP belongs to another ASN. This is based on a heuristic that a point of when connection between two routers that self the IP addresses that come from, that seven the same software that come from one of those two participating ASes. And we furthermore verified our finding with other external data set that CAIDA uses for AS relationship to verify our border routers and the third category is unclassified, it means it couldn't confidently classify the router at border or internal, it means they had public ASN in their open messages but all the interface IPs mapped to the same ASN.
So it could be an iBGP or the host AS which is providing the IP address to the router.
Here I present our findings on IP addresses, routers and ASes.
So we found around 141,000 exposed IP addresses which correspond to 20,000 unique routers and we found a very high number of internal routers compared to border routers. And there were some unclassified routers which is 3,181 and all those exposed routers are managed by 4,194 ASes on the internet.
We measured the criticality of those border and internal routers. It is because it seems that those routers are violating the recommended security practices so we measured how critical those routers are. In the case of border router, we measured the criticality in terms of connected ASes, how many ASes those border routers are connected.
As you see in the graph, we found that 75 percentage of border routers connect to only one AS. There was a border router which connects to 21 other ASes. So you can imagine the criticality of that router and that router would effort that 21 ASes unless they have some redundancy or other backup options.
Similarly to measure the criticality of internal routers we used the number of IP, number of IP addresses configured on them as an indicator to measure the criticality, it is because the number of IP addresses correspond to number of subnets in that router and the stability of that router would affect that number of subnets of that AS. We found that 60% have only one IP address and around 90% have fewer than ten IP addresses configured on those routers.
And 10% of internal routers have up to 1,208 IP addresses.
And we did another analysis to find the vendor of those routers. In our case we were not able to find the vendors of all those 20,000 routers, that's I am I am seeing it as a partial view and for the purpose of finding diversity, we used Shodan's API which is based on the SNMP version 3 responses of those open responders. We were able to identify around 43% of the exposed routers, as you can see here Juniper seems to be the dominant vendor, followed by tell data, in the figure you see some vendors with numbers, it is because those vendors didn't use the private enterprise number that is provided by IANA, that's why I couldn't find the name of those vendors to summarise our findings, we found a number of exposed BGP routers which are around 20,000 routers, 4,194 ASes and there are many other interesting findings on our paper such as vendor diversity, router types within ASes, also like distribution of ASes by router type, etc.
And also after our this research, we reached out to a few network operators and only very few of them responded because we wanted to know the reasons of such exposure and one of the vendors thought that BGP's default protocol is enough because by default the router said notification matters to the requester and they terminate the connection and he thinks that that protection is good and we don't need extra protection for that. And another vendor was happy to know about exposure and fixed their routers so the operators here, I want to know about the more reasons of such exposures and why you are not restricting such exposure and maybe a simple way to would be using NMap or ZMap from a host outside of your network and the other ways are configuring and TCP authentication option and many more things and you can read more in my paper and since I am finishing my PhD, you can check my profile if you are looking for someone like me. Thank you very much for your attention.
(APPLAUSE.)
MAX STUCCI: I see questions.
AUDIENCE SPEAKER: . Thank you, I think anyone bringing to the attention RFC 7454 has my applaud, especially when you bring it to the network operators community, a word of warning about the way that it introduces a sense to the exposure IP addresses. Especially on border routers or if you have customer providers relationships. There are two ASN involved and one provides the IP address the link. Now you are not the first one to do this research and to scan the networks, I remember that the DE cert is providing a scan or at least what I know, the German registered IP addresses we get notifications of them and what we see is is that in 99% of the cases that OpenSource, port 179 is that it is our customer router, but they use our IP addresses so they send the messages to us and then we are supposed to notify our customers saying you know, you are doing a bad thing, you should fix this.
And we do that.
However, what I also see is that a lot of network operators start to defend being accused by not publicly routing their infrastructure IP addresses, therefore limiting you to scan those IPs because they are not routed on the internet but they stay exposed on these large networks.
I think that's something that we should avoid so try to see who is on the other side of the connection in our measurement, I don't know how to do that but what I see is like I said in 99% of the cases, it's the downstream network that is not implementing it, not the upstream networks that delivers the IPs.
SHYAM KRISHNA KHADKA: Thank you very much for your feedback, I agree.
AUDIENCE SPEAKER: First of all, if you are affected by this or you haven't properly firewalled your BGP sessions, please, for the love of anything, go do that. We are getting security reports like everybody else. All the other vendors are too. This is unnecessary exposure and if you are caught by something with this, it's at least halfway your fault for not following 7454. I do however have a question. You had a part about pre‑TCP in your differentiation of things that happened, do you mean the TCP connection was established and then closed immediately or do you mean something else?
SHYAM KRISHNA KHADKA: We were not able to establish the TCP connection
AUDIENCE SPEAKER: Did you record cases where the TCP connection was established and closed without any messages and how much was that?
SHYAM KRISHNA KHADKA: Sorry, your point is that first second whether they are allowed TCP connection and then how many of them. We did that thing but I do not exactly remember that number.
AUDIENCE SPEAKER: OK. So that is the default behaviour that you will get for a lot of BGP stacks if there's no configured session and that's in my opinion also worth recording, it's not as much of an issue but still it's worth including in data like this.
SHYAM KRISHNA KHADKA: OK, thanks.
AUDIENCE SPEAKER: Gert Doering: Thanks for doing the work, the result is I find the result scary, I thought this is a niche problem but it seems to be a fairly big problem, I am surprised there's so many vendors that cannot get their can he defaults right it seems because end users don't know how to harm their stuff, they expect things to connect to the internet and they sort of, a router should be able to well do the right thing connected to the internet. I know some vendors do, some vendors don't, I had discussion with vendor about SNMP being reachable to the world and no way to ACL that off, which is another pet peeve of mine, thanks for bringing that one up, good work, continue please.
SHYAM KRISHNA KHADKA: Thank you very much.
MAX STUCCI: We have exhausted all the questions so thank you very much.
(APPLAUSE.)
So before we go for the break, we have a couple of small announcements. The PC nominations run until today at 3.30, you can still submit your nomination if you are interested in joining us. Then I would say please rate the talks, let us know how much you liked these three talks or even the ones yesterday, and then last is, but you should have received an email about it, you can collect your general meeting badge or sticker at the Registration Desk, please do it as soon as possible and then we see you at 11. Thank you very much.
(Coffee break).