RIPE 93. Main hall.
11 a.m.
Closing Plenary
30 October 2026
BRIAN NISBET: Good morning, and welcome to the closing Plenary session of RIPE 93.
We hope we'll have some interesting, we have lots of interesting couldn't but we have a discussion and a presentation and then some of the more regular closing Plenary matters with which you are familiar.
But before we start with our first bit of content, just to let you know the PC elections again thank you to everyone who volunteers for this, whether they are skeletons or not, but I'd let you to Antonio Prado and Babak were relected to the PC.
So, first up, we are going to, this is the wonderful ability to kind of work on the agenda as the week goes on, we are going to revisit the lightning talk we had on Monday because I think it was obvious when we were discussing sovereignty and the Internet and the concerns, we didn't have enough time to discuss things on Monday in that lightning talk session. So Timo has agreed to give us a short reintroduction to the topic and lead the discussion for a few minutes this morning. So...
TIMO HILBRINK: Good morning. So, here I am again. On Monday we had a brief talk about what the implications are when you are.sanctioned by the US. I am not going to do the whole presentation again, I am going to do a quick recap of what happened and then we'll hopefully have sometime to discuss this further.
What happened? On August 26, the US Office for Foreign Asset Control declared an Italian service provider a specially designated global terrorist. On the 28th August, the domain name was removed from the DNS by PIR, the registry for the dot org TLD. On that same day PayPal suspended their account, the account they used to receive donations for their services. On the 2nd December, Banca Etica, Italian bank, suspended their bank account because they said okay, well if we're going to keep servicing you, then we'll be cut off from Mastercard and Visa, and we can't afford that, so that account got closed as well.
On the 6th September, Autistici/Inventati, shut down their operations in order to protect the volunteers that worked for them, and their users. And October 13, just recently, the Dutch police seized a server in an Amsterdam data centre that belonged to this organisation. And they did so on the request from US authorities.
That is scary.
So, why should be worried about all this?
.
Well, the three main, mainly used TLDs, gTLDs,.org,.net and dotcom are operated by registries that are head quarterd in in the US. Almost all Internet governance organisations are headquartered in the US. Cloud services are headquartered in the US. You could say the same about SSL certificates, it's not that easy to get an SSL certificate from a CA that is not in the US. Our banks depend on Visa, Mastercard.
As you probably saw in the news, there is more sanctions on the way. Sanctions, economic out cast against Iran, sanctions against the international criminal court in Amsterdam because they do things that the American government doesn't like. Who knows what comes next. I don't know. Which other organisation could be sanctioned, it could be anyone, if you do something that is not in agreement with the US administration, you could be sanctioned. It would be extremely stupid if they would sanction some organisation like the RIPE NCC, but stupid decisions have been made recently that we never thought would happen, right.
So, are we prepared for this?
AUDIENCE SPEAKER: Question to you: Do you think that the RIPE is prepared for this?
TIMO HILBRINK: At the moment no.
AUDIENCE SPEAKER: What do you think we should do?
TIMO HILBRINK: I think we should have some contingency plan in place to decide what we are going to do if this happens. And what steps we can do to prepare ourselves ahead of such a situation. But I mean for the moment, obviously the domain name is a dot net domain name, the SSL certificates are from Letsencrypt. The Cloud services are being used in the NCC. I don't think that there is any ‑‑ I don't know, I mean I don't work for the NCC, so maybe someone else could clarify that. But I think we are not prepared for such a situation at the moment. And what we should do, I think we should start thinking about what we can do to prepare ourselves, what sort of measures can we take.
AUDIENCE SPEAKER: Benedikt Stockebrand. It's not just the US we might ‑‑ with the right shift we see here in parts of Europe, we might have a similar problem. I'd say we have a problem with national governments or possibly even multinational organisations like the EU doing some stupid stuff. It is something we should prepare for and what needs to be done, in my opinion, is first figure out where do we have dependances and then decide how we can possibly sort them out. But I consider this a serious problem and not just with the US but generally speaking.
TIMO HILBRINK: I think you are right.
AUDIENCE SPEAKER: Babak, with Quad9. What does even preparation mean in this context? And it's not even about DNS part, you are talking about banks, you are talking about servers being seized, and does moving to .nl, is it a solution? What if they go after SIDN, I don't know, I'm not a lawyer, and does this mean that now we have to accept cryptocurrency and move to .onion, I am being serious, like maybe that's a solution.
RANDY BUSH: IIJ of course I am speaking for myself but you just know who pays me.
I'd like to take this down a couple of layers. We haven't done a good job of analysing the dependancies of our infrastructure, just as an example. The RPKI certificate structure relies on the DNS. The DNS relies on the RPKI. Oops!
AUDIENCE SPEAKER: Andre. Of course speaking for myself, like everyone else. I would just like to focus on one of many things here, because with DNS, yeah, I feel like of course the root zone is also under control of the US about the TLS certificates which are now completely vital and you cannot do anything about that. We have Letsencrypt which is in the US we are our outage switches which are in worse regions like China and so on. The thing is Letsencrypt has done all the heavy lifting, all the software to our the router is OpenSource, it should be had in theory quite easy to take the code from GitHub and spawn a new Letsencrypt clone in Europe. The only problem is it will cost money and somebody has to pay. I wonder if there is an idea who would be willing to pay this money if the alternative is free for everyone?
TIMO HILBRINK: I think I saw there is a similar service to Letsencrypt with the same basic, API basically. They are based in Austria, but they were recently acquired by an American investment fund. Oh well...
But, yeah, but I think this is things that we need to look at.
AUDIENCE SPEAKER: Actually, you have an Italian which is open which is named Actlis.
TIMO HILBRINK: Their UI is not very good.
AUDIENCE SPEAKER: It's not US.
TIMO HILBRINK: And there is no automation.
REMCO VAN MOOK: We are royally screwed, and not ‑‑ I mean, down to the hardware levels. I don't think that we're, that we are actually fully understanding how deep this goes. This goes into your internal CPU. There is so much to unpack in terms of dependances that just going, getting worried about Cloud logins and certificates, I mean there is a fantastic Indian CA provider that has a most graciously offered the entire Internet the use of their material, so now everyone can have very secure certificates.
I think the rabbit hole goes a lot, lot deeper than is described here and I think that if you really want to get out of this and go into sovereignty mode, you are going to have to start by soldering stuff.
AUDIENCE SPEAKER: I would just like to say that it's also recursive, because in your presentation you mentioned that the Italian bank was fine but they got pressured into doing that. So it's not just about our providers but their providers as well and what if they cut them off?
TIMO HILBRINK: Secondary sanctions.
AUDIENCE SPEAKER: The other thing is also whether we are ready, whether we're ready to accept certain things because for example Switzerland is a country that chooses to remain neutral and the country wanted to use Alibaba Cloud for certain services rather than just Amazon or Google Cloud. Some people did not find it easy to accept that this is may be part of the solution or this may be what neutrality looks like as well.
TIMO HILBRINK: As an example of those secondary sanctions you can see for example the prosecutors from the International Criminal Court they lost their health insurance because their insurance provider was ‑‑ had an American owner so they lost their health insurance. And of course, their bank accounts, so employees are getting pay for a year now so at least for the next year their wages can be paid. So, yeah, it goes very wide wide.
DANIEL KARRENBERG: What I hear from many of the comments is despair. You know, we can't solve this, it's too big, we have to go solder, it goes all the way down, all that kind of stuff. The one thing I want to say is if we despair, our enemies win. So, what we have to do is we have to attack this wherever we can personally, locally, piece by piece and not despair. That's the very important thing. So, we also can do this personally, we can look at our own use of the Internet and I think as a foreign natural person, we can some steps. Of course we need a bank account and all that, but that shouldn't keep us from doing the basic things: Get our personal life, i.e. our e‑mail, you know, out of the hyper‑scalers. Let's all do the things that we can do where we have reach, where we have power and I know that in the NCC, there is a lot of people are doing things piece by piece to improve the situation and the important thing that we do it and we do not ring back and say ah, we can't solve it. Thank you.
(Applause)
AUDIENCE SPEAKER: Joe Abley. Lots of people seem to be worried about the idea of how to we build a wall and surround a small sub‑net of services that have no external dependances, but this is not the Internet that we have built in fact, we built an Internet that was supposed to cross borders, span the entire world and rely on cooperation. We didn't build something that looks like a private Internet for Europe. It's intended to be global. So, any instinct to try and cut this off is the opposite of what we were trying to building and we should probably remember that.
A lot of examples around dependencies which have fallen victims to things like sanction enforcement can be solved from the edge, in an Internet way from the edge by saying don't depend on a single bank, on a single currency, on a single data centre, don't depend on a single TLD or DNS, you can do these things from the edge, you don't have to do it in the core. The Internet was all about pushing these things to the edge, not to centralise them. Be careful we're not trying to build the opposite of what we want.
TIMO HILBRINK: Thank you. But I also think, that I mean I agree fully with you that the Internet was not to be an island, needs to be decentralised etc., but the danger is that the organisations that should maintain this are all in America, right?
TOBIAS FIEBIG: I would like to repoint to the issue of capabilities like personal capabilities of people doing things, because there is clarification and centralisation that we have seen has led to the loss of those because it got too complex to run your e‑mail, too complex to keep a service running when you are constantly being dossed and we really need to ground ourselves in this. Daniel said we really need to get those capabilities back so we can solve those issues from the edge. The Internet is not about everyone having Gmail. The Internet is about everyone being able to run a mail server. And as I usually like to end these statements, lacking IPv6 adoption is something that is actually preventing that quite a lot because it changes the balance of ability to host services to those who have money to buy addresses. Thank you.
AUDIENCE SPEAKER: Just a question for the lawyers. How safe are the IP address allocations of the RIPE NCC? Is there a scenario where IANA could be requested to deallocate the sub‑nets that have been allocated to other RIRs.
DANIEL KARRENBERG: I was part of the negotiation that is set up this ICANN thing. At the time there was lots of questions about authority being top‑down. You know, from someone to ICANN down to the RIPE NCC and down to the local RIRs down to the ISPs. And at the time my standard talk was to say authority goes both ways. It goes top down, but it also goes bottom up, and in the Internet, it has been alluded to before, it's more bottom‑up than top‑down. You can look at this as top‑down if you are a diplomat or a military person, you can look at this as bottom‑up if you are an Internet person. So, how could whoever, ICANN, let's take ICANN but I'm not going to hit on them, it's just because you also alluded to that, enforce deallocation? It could only be enforced if all the ISPs agreed that that was a good thing to do because the ISPs are doing the routing. So I think ICANN can say all along it's deallocated as long as the ISPs say oh, we don't really care, it has no effect. That's my answer to this.
If you as a community do not want this top‑down actions, we can bottom‑up, basically say we have organised this RIPE NCC, we have organised these RIRs, and we trust them. And I would hope, that's what I said 25 years ago, would I hope that that would happen if abuse like that would come to pass. Does that answer your questions?
AUDIENCE SPEAKER: I think the ISPs, the US ISPs could also be first ‑‑ that would be a break of the Internet in two.
DANIEL KARRENBERG: We can argue about this at the bar, but I think ISPs and everybody on the Internet makes money by universal connectivity. If that promise goes away the Internet disintegrates, not disintegrates, splits itself into different zones.
MASSIMILLIANO STUCCHI: Thank you for your comment, but let's move over because we have ‑‑ we are running out of time but we still have a queue here.
PETER HESSLER: The world has changed quite significantly in the last 25 years, and a lot of the actions that we used to be able to do were personal, either your own behaviour or how you spent your own money. Society has changed. Politics have changed. Either use your vote for both political, make sure that you vote for the political parties that support the things that you value and not just things that will give you a bit more money now. And also, vote with your employer, only work for employers that are ethical and match your personal standards. A lot of people in technology, not necessarily this room, but in the greater technology sphere will work for unethical employers doing unethical things because they make a lot of money.
You do need to make money because we live in a society but you can make enough money to liver comfortably without violating your own ethics.
MARIA MATEJKA: I would like to know that while it's inevitable that we are going to depend on things from all over the world, it's still scary to find out that there is in the end one piece of software on which you are going to depend no matter what. And that's why I like that our customers from IXPs are actually financing creation of another software for IXPs, because if everybody in the IXPs depended on in BIRD, we are going to get into trouble.
And the same problem gets with everything else and that's why we are trying to get as little dependencies as possible and even with the development, even in the situations where somebody tries to only develop BIRD, we want to be able to do it without much of dependencies because as soon as these dependencies start to grow and we all right hit this, you are getting problems like some of your dependencies upgraded mid‑time and you suddenly can't do a release because some part of your network needs to be upgraded and that part has just broken.
So, yes, we are interconnected, we are inter dependent but please don't is mistake for dependence in all parts of the world because that doesn't scale.
AUDIENCE SPEAKER: I would like to make an invitation as about how we are a community of makers like our peers in APNIC, in LACNIC, in AFRINIC, in ARIN, and makes the Internet what it is today through parties as IETF etc. I think we need to double down and look twice, thrice any technology that we make today that it cannot be held hostage against any of us. And if there are technologies that are currently prone to being held hostage, get them out of that position.
SANDER STEFFANN: I just want to comment on one of the previous remarks about fixing this in the spirit of the Internet. I think one of the core things of the Internet is avoiding single points of failure and I think we are just identifying one so I think what we are doing is definitely in the spirit of the Internet.
MAX STUCCHI: I think we have run out. Do we have a question online.
BRIAN NISBET: So, Sasha Romijn. I think with he need to be cautious not to make the perfect enemy of good. The RIPE NCC should not try to manufacture its own hardware. It can do without hardware supply for a while. It cannot do without RIPE.net for weeks. What the US Government could industry to make SIDN to drop right .nl it's harder than convincing the US organisation to drop RIPE.net. Of course they could drop NL from the root zone that could cause serious other impacts. I imagine RIPE NCC staff is also thinking about this and perhaps they could publish more of their risk analysis. More fundamentally instead of thinking about all the things we can't do, or that are really hard, let's think about things that we can do and how we can harden instead of making it perfect.
TIMO HILBRINK: Fantastic comment.
MASSIMILLIANO STUCCHI: Thank you, we have run out of time so thank you.
(Applause)
.
Now our next talk from Tobias.
TOBIAS FIEBIG: Good morning everyone. As we see, I am usually doing with the debugging funny things and this is the wrong slide set.
That's what we wanted to talk about. As usual, I can promise you this is the last time to see me during this meeting and now we are going to talk about me debugging stupid things or I like to call it if it was only DNS for once.
What is this talk about? It's about operators being stupid, vendors being stupid me being stupid, I like to do stupid things and I'm fundamentally convinced that we turn off the Internet we'd all be better off.
We go over to really nice little errors I encountered in the wild, and the thought process in debugging it. And of course some cases are partially anonymised, chronically overworked and constantly tired people who give their best everyday but sometimes just slip.
So, the Internet routing system, the people are represented by two separate yet equally important groups. The operators who deploy the systems and the vendors who build these systems.
These are their stories.
Ping and Packets. Internet routing systems. Keep in mind this is all of course purely hypothetical.
So, what are stupid things? Other examples which we are not going over. I spend a major part of RIPE 92 figures out what e‑mails from customers seemed to disappear. It turned out Microsoft was considering ED 25519 DKIM signatures as a syntax or, there was unvalid signature, if you have an e‑mail with an in invalid signature, it's obviously spam, it doesn't a matter if it has valid ones. Me leaking ROAs advertisements in the peering LAN, That makes you very friendly with the local NOG of IX, I didn't have that configured but my routing daemon was configured if I do RFC 8950 I'd definitely do unnumbered and need router advertisements, so it just send the them. Someone getting solar panels making me found an open BGP because the father got solar panels could measure power consumption and randomly turn things off which led to a pattern of connects and reconnects which triggered another pointer issue is.
But, let's get into the cases. The case 1. Digital courage dot social is like an independent freedom people, social media thing. And they were complaining about one specific ISP rate limiting them. And found that very not nice.
Specifically, Deutsche Telecom and they complained about it, there was a lot of bus like telecoms rate limiting us, I kind of was enticed by the problem. Because I do like unsolvable problems because it actually it worked. It worked for all other ISPs. Just not for us, it worked from another virtual machine it had next door on. It was odd.
So that's a problem statement. Various users behind 3320 have low download speeds, others are fine. People are unhappy. Bugs work and I want to delve into that.
The first thing I do is reproduce it from my machine at home. I am not behind 3320 but I got the issue. Which was really, really good, because it meant I had a place where I could start debugging. The next step I did was trying the same thing from elsewhere in the same AS. So, different box, in this case my personal shelf server and that worked out fine immediately. Which is kind of funny because it's still the same AS.
Well, that's when you start drawing PCAPs. So, I found the path, the packets were taking it out of my AS, ran TCP dump, wanted to see what is actually happening and make sure nothing it wrong on my side.
We are seeing like things on screen, destination unreachable, need to frack, speaking to the on path router it's lightly a little bit too many of them because usually you get one of those packets and then you get fragmentation which should make the problem go away. We have our request, something goes wrong. Especially the MSS is definitely wrong, because we do happen to have this MTU break on path between my machine, like the one I tried from the first time, and the service, but the local interface of my home machine also has an MTU of 1,500 so it cannot determine the right MS which is not nice, so the response by the server we are getting for requests is too big, then we get the packet too big, which we are sending to them, right, because the MTU break is ours. That gets ignored, we still get retransmissions and so the PathMTU kicks in and we get our slow slow slow slow transfer speeds.
Of course the first question is: Who needs ICMP? In this specific case somebody decided nobody, well except for echo reply, which is useful for debugging, but not ICMP type 3 which would be packet too big so that is not allowed, which led to a small excursion, like why no PTB? Because the Mastodon documentation back then said you do not need packet too big messages, so you just firewall them in like the default firewall recommendations. That got patched, fixed on the server so everything should work, right? No.
Which of course, I mean ‑‑ it would be fun if things would just work.
So, back to the server, and let's take a look at the MTU which is recorded for that host.
IP, minus D to get debug information, route get for my IP address and it actually has discovered the right MTU, 1420. So, is apparently the nice way of telling the host that we do have an MTU break doesn't work. I decided to do something forceful, which is a locked MTU on the more specific route all packets will be sent out, do not fragment bit in the IPv4 case, which should not make a difference because we had the right MTU in the first case yada yada. So I tried again and it worked. Which did not make any sense because the host already had the right MTU. If I tell it to have the right MTU without do not fragment, it works, but it does like what is this?
So, what does one do in such a situation? I always like to say, change what you can, accept what you can't.
Of course you don't want to interrupt any other part of the configuration so you just set more specific routes you lock the MTU 1320 which should you plenty of head room so everything should be good and then you let it go.
Except, I am very bad at that. So, I decided to reproduce it, like I set up a virtual machine, everything the same. Well while doing that I set it up, I couldn't reproduce it. I had completely the same software, I pinged the same packet. Everything. But it couldn't reproduce. And I would love, like out of a weird reason, I asked them for an LSP CI. Which like on a virtual machine, it's kind of worthless. Anyway I got this. Did somebody see something?
Fast ethernet? Does that ring some bells somewhere in the room, like, you know, PTSD, 1,000 yard stare back in the day, because it looked like this. Right. I can switch back. Like this is on their server which doesn't work. This is mine. Yeah, real tech 8139, a very classic NIC, even more classic firmware box and apparently there is a virtual version of that because every crappy Windows 95 machine you have to virtualise with these drivers so if you have to virtualise a legacy system, you have to use that. It kind of makes sense.
That led to a little bit of a code chase, specifically it led to this here from 2006, which implemented a fixed MTU of 1,500. So, since 2006 the MTU in the virtualised 8139 had been fixed to 1,500 which was already noticed in 2016. So then years after it appeared, six years after it ate away at my tame, it was patched like there was a patch submitted which was never merged. Ultimately that issue was being recorded as reported by two people, including the one person who did it ten years earlier.
So, summary:
MTU pinning, like it's 8139. We had that 16 year old bug. It is in the ‑‑ well it was in the TCP off loading, so technically if we would have disabled TCP off loading to the NIC, it should also have gone away and like the smart choice is to just replace the virtualised 8139 with virt IO, that's also what they did in 2016 and the bug was gone so they didn't care anymore.
What did he we learn? One, legacy comes to byte. There is all these things which we have in place to like handle these special cases which we still haven't around haven't gotten around to fix. There is we have those code paths. If we keep that around if we keep that around, at some point somebody will use it and it will ruin our evening.
Next one when you reproduce something really make sure that it's one to one. Like, there might be this little oversight, like the type of network interface which shouldn't impact anything, which you are like that, no. Actually do it one to one.
Also the less something is used, because I think most people in this room do not feel the agenda need to have an 8139 virtual adapter machine, the less bugs get found. PCAP didn't happen. You actually have to have the ability to look on the wire, what is happening with your packets. Kind of obvious one PathMTU discovery is kind of exhausting. So, kind of sadly effect.
And you also need to be able to look on path. Because the packet too big message were actually originating from the border router in my AS. If I just P capped on my machine, I wouldn't have seen.
Next one. And disclaimer, when I initially submitted this talk, I promise that had we wouldn't be talking about DNS. I sadly cannot keep that promise, but I can promise you that it's not the root cause.
What was the problem? I was not able to send mails. We did have this discussion about sending e‑mails running your own mail system, I am stupid enough to do that. Apparently I like that pain and at some point I was unable to receive and send some e‑mails, occasionally, it would be nice if this is a permanent issue but occasionally. And in fact dig dig dig, I found that my server was unable to do DNS resolution and for some reason mail servers get unhappy if they can to the do DNS. So I had an intrinsic motivation to make this problem go away.
So, what do you do if DNS doesn't work? I logged into my mail server, dug at the DNS server, and it said permission denied, which well, it shouldn't, it's allow listed, the firewall is allowed to send things. So I did what I said you should do, I went for trying it first with another host. It worked so I know it's not the server. And then with the PCAP.
Now, this one is a bit interesting. Because what we see here is our host doing requests like, in this case name servers for dot, the roots, and then R server getting the answers for that. So, this is the one who says permission denied. And actually, the one who says permission denied in this picture is us. So, I do have my mail server, I use dig there, I sent a query, the remote answers, but for some reason the IP stack on my machine is like please go away, UDP port unreachable, I didn't ask for this.
So, then I looked a bit further, because well we learned sometimes ICMP messages which you did not expect. I found this. Me sending the requests, and then somebody else sending out no, no, no, no, you are not going to do that. This is administratively prohibited. Which is odd, because this is on the Internet, like I am not running any firewall there, I have some packets filtering on the local machines, but those packets should be allowed.
So, I am confused. The client obviously ignores the responses because it received a destination unreachable and just tore down the sockets. The ICMP 6 destination unreachable must contain the original packet to cause that. Because you cannot just spoof things and make things go away. On path I only have an MX 150 doing so the routing and no firewalling for the revenue ports because you don't want to do that because that's not funny.
Well, so, a little excursion. You might have noticed there is some people around here, like Remco and me, who are really into something called RFC 8950 or IPv4 with an IPv6 next hop. So, the Juniper router has the 36 on that interface. I am also doing VRRP, that's why there is two ones in there and each host gets a /56 on his own interface and IPv4 can be done like I had another talk about that.
So, two hosts would always communicate via the router because for the router they are in the same 36 and the hosts are in their own 56s so they sent perfect to the neighbours through the router I thought that was a nicer setup because I can easily migrate it to for example having dedicated Layer2 segments for each other.
So, if the MX should be anything, should be sending anything, it should be an ICMP redirect because from its perspective both hosts are on the same Layer2. They get in the packet and they are like no, you can reach that one directly which the host capital because it does have another prefix on the interface. It's not the smartest configuration, but it works and I like it.
So, it should send an ICMP redirect. How do you do that? Well, if you are, say data plane, you get a packet, which has to get out of the same interface. So, the first thing is you forward the packet. You also want to let the other one know you don't have to bother me, you can send it directly. So you have to create that ICMP 6 redirect. But we can't do that because we are just a data plane. So, what we do is we forward that packet through the router engine. So it can do that ICMP 6 packet. Funnily enough for that packet is being evaluated against the firewall route for the control plane, because reasons:
It doesn't happen for TTLX expired, or echo reply but it apparently happens for redirects. Our data plane is like ‑‑ our control plane is like that packet isn't allowed. So, it cannot really generate that redirect. And indeed it is like yeah, but it's administratively prohibit, right? So, so just have that, right, and forward it to the source.
That is then how we ended up it a very simple resolution. Which finally made my mails work again, and me relatively frustrated with realities of networking.
So, in summary, hosts were unable to communicate with the network that was routed to a host on the same segment as them, as a the host initiating the connection would reply with a port unreachable to the response. Being unable to do so, sending administratively prohibited stat because it was evaluating packets against the firewall.
First lesson learned, sometimes the packets you are looking for are actually not the ones you need. So, remember the first slide with the interaction between like we are saying no to something we ask for.
Most importantly thing, I think this is basically the thing you need to take away are from this talk, PCAP didn't happen f you don't have PCAPs, it didn't happen. And if you build things around corner cases you will experience corner cases, so ‑‑ but that is a me saying, I have somehow have to generate content for funny talks.
So, overall takeaway of this. This often break in unexpected ways. It's helpful to look at actual packets while debugging. PCAP didn't happen and this ties in with a discussion we had a while ago in this community. How many personal ASNs should there be? Well, this one may be I could have resolved without my own ASN, but the previous one, definitely not. So, having the ability to have a network which you can break, where you don't care that much actually helps in debugging things you might not want to debug in that way in a production environment or you go for the two more specific routes with a locked MTU because it actually works then.
Each case which we debug is unique but some very simple steps remain, reproduce, collect data, reduces valuables, only change one thing at a time. I am Very bad at that because I am chronically impatient, but if we are debugging we really have to hold ourselves to that and whatever then remains brings us further, more grey hair. Well networking.
And you are more likely to find non‑standard issues in non‑standard environments does not mean that you won't find them in standard environments.
.
So thank you.
(Applause)
BRIAN NISBET: Thank you very much.
AUDIENCE SPEAKER: Jen Linkova, Google. Thank you very much I love those stories, can we have with their Working Group so we can have dedicated sessions just for those? Thank you very much for iterating PCAP didn't happen. I normally refuse to look at anything until it's PCAP attached. I think it's very important to talk about the proper troubleshooting techniques. It's probably an idea for the next talk. Thanks again for bringing this.
TOBIAS FIEBIG: Thank you for suggesting ‑‑
JEN LINKOVA: I forgot to ask, it was only for v6 right, v4 reacted okay I presume.
TOBIAS FIEBIG: I didn't try because it was v4. Thank you for trying to chair that BoF next time.
JEN LINKOVA: We need to run it in parallel for the Plenary for five days I guess, right, we have enough stories.
TOBIAS FIEBIG: You mean a five day whiskey BoF.
BRIAN NISBET: Are you telling us that the Internet is broken is not a sufficient problem statement to troubleshoot? My god!
PETER HESSLER: A serios question about these stupid things. In your first example, you suggested that this was only affecting one or a small number of networks versus all the global networks. If it was a host firewall problem, how did that happen? Why ‑‑
TOBIAS FIEBIG: It was not a host firewall problem. It was a problem in the off loading of the virtualised 8139, and if you look at for example like all the docks X based network they have an MTU 1,500. Deutsche Telekom takes away 8 byte for PPoE. So you have 1492. Of course there was other networks that are being hit by this, because 1492.
PETER HESSLER: Okay. Thank you.
AUDIENCE SPEAKER: Andrei, RIPE meeting tech team. I wish we could move one thing on the time but we have these issues, for example, with wi‑fi that only manifest when this room is full of people. As soon as people go to coffee break the issues are gone and unlike good debugging things like, you know, let's try one thing and see ‑‑ like the meeting only takes one week, it's really hard to make changes, one can hope for the best but you will hear in the tech report about this. I wanted to also say you said the more non‑configuration you have the more weird bugs you can expect. What make me a bit sad is that yeah. Running IPv6 is something that is considered non‑constant because running by IPv6 networks you see bugs that are there for everyone but nobody cares about them. That's sort of like sets off 20 years off IPv6 deployment.
TOBIAS FIEBIG: I think we can just fix it by turning off IPv4, it would make so many things so much easier.
ONDREJ CALETKA: We are trying hard in this network, yes.
BRIAN NISBET: Okay, I see no other questions, nothing online. So thank you very much.
Your homework for the next RIPE meeting is to all have some weird debugging thing that we can talk about during this trauma BoF. Okay. Next up we have the report. So that ends the kind of the submission content. So now we are moving into the formal pieces of the closing Plenary and first up Sebastian is going to give us its report from the Code of Conduct team.
SEBASTIAN BECKER: Welcome to the RIPE Code of Conduct team report for this meeting.
Since our last presentation, we received one report. So, last presentation on RIPE 92. We can say that the report was solved and we concluded it was a violation and we issued a warning about that.
In RIPE 93, we have got two reports so far, because the meeting is not ended. We have two open cases. For both, we formed an assessment group, we looked for any conflicts of interest in the team and if necessary, the member is not part of the assessment team. And we contacted in both cases both parties to go forward with the cases.
.
Very important, how to report: So if you feel uncomfortable, if there is something you think that's not okay, not even by yourself but you think also another party might feel uncomfortable, very much you have the option to report that. You can do it by form, you can do it write to the mailbox, and you can report to the individual members of the team. So either approach us here at the meeting, or you can write especially to one member, it's first name, last name as you see@ripe.net.
And anymore information you find in the link here for the Code of Conduct as well.
Team updates:
The team didn't change since last time. We did some outreach. We presented at the NOG in September. We were present at the NOG lunch at this meeting. We are still working on our FAQ and we are still working on publishing the transparency report, that's more due to we want to have things right and don't reveal any private information or confidential information with that report.
These are the members right now. But please feel free to join us if you think you want to be a part of that. Still members are needed. Not every time every member can participate on the meeting. So, having also new members are very welcome.
And that's it. Do you have any questions?
MASSIMILLIANO STUCCHI: I don't see anyone going to the mic.
Okay, thank you.
(Applause)
Now, we have the moment you are all waiting for to understand how the meeting went from Menno, the tech report.
MENNO SCHEPERS: Good morning everyone. This is the tech team, and the event team, the people you have been seeing walk around all week. We have been helping out with all the technical aspect of the meeting. So the network you are using, but also helping out in the smaller meeting rooms with tech support, where needed. We have in this team a new joiner, that is Wulfert, the tallest guy in the group, we're very thankful having him around, and Olivia and Kajal from the events team, we have been helping them during the event.
The meeting infrastructure that we have is... well, quite interesting. What you see on the picture is five floors and if you would open the PDF on the meeting website, you can zoom in on it, but this shows all the switches and access points that we have rolled out through the building. Often, we have one floor or two floors for the conference, but in this case, we were on five floors which was interesting, but it all worked very well. This hotel is quite new, all the infrastructure is working. The internal cabling was all working as it should.
Let's have a look at this picture here. This is the network we're using. It didn't change from the last meeting. It's a picture showing the uplink and it also knows a NAT64, I'll show you later how that works. And all the internal components.
These are the software we're using at the meeting. It's OpenSource technology, most of it, and the routers, they run RPKI and ASPA, using BIRD and other ‑‑ one more new thing is the short‑lived certificate, we didn't use that before in the previous meeting, so, we started using that as well in as many places as we could. There are still some left, but I'll show you that later as well.
The wi‑fi:
We use IPv6 mostly. For a few RIPE meetings already, it works really well around 80 plus percent ‑‑ 86 percent of the clients support this. So, they have, they only get the IPv6 address and they use NAT64 for translation of the v4 traffic. And that works very well. The there is a lot more information on be this available online, also from our team member Ondrej Caletka who has an article which is a bit old already but still interesting to read but also tomorrow nice presentations on this, you can find and they are very interesting to look at so I recommend having a look there. If you want more information on this, and then maybe you could try it out on your office network or something like that. Really cool stuff.
Then also, on the wi‑fi, like last two RIPE meetings, we had a few of these compass APs, you see them there, there is one there and the back three in total. They run with 25 gigahertz radio, so that would mean when we are using them, all three, there is six 5 gigahertz in general in use in this room, but they gave issues, the same issues as last time, or similar issues. They are still dropping a solicited node multicast, and we had a look into that, we opened a ticket with Unify. They claim to have fixed the issues, but we were still seeing issues also in the office in Amsterdam where we also have similar access points, not this model but same generation. But yeah, we tried to work around these issues. There is a nice RFC from Jen about things that you can do to fix this and Ondrej created NDP fixer which helped with the solicited node multicast being dropped by creating the still entries for addresses in our firewall. We were running this on the firewall host. But yeah, if you search for NDP fixer on the GitHub, you can also have a look at that.
Then, this is the wi‑fi map. It shows where we have put access points. Actually in the main room, you also see, if you look closely, you see six access points extra on top of those three, so those six access points, when we noticed the issues were still here, we decided to turn off the radios on these campus APs and enable the access points and those six access point you see in this picture. Just to make sure that in the main room we still had proper functioning wi‑fi, at least most of the time. There were still some little issues. I'll show you on another slide more about that.
And you also see the side room and the coffee break area where we also used these new access points in the side room and in the coffee break why are, but there, we didn't see the same issues as here in the main room. So, that's interesting. But also, yeah, we don't know if everyone reported the issues when we saw them. Some of you did and then we can look at it but if you don't report the issues, it's also fine, but we would like to know still.
About shows short‑lived certificates. It's six days certificates with Letsencrypt. There is a list here showing that. So it's automated of course because otherwise we will run into trouble. So, we used the certificates for the web certificates but also the DNS. And then the presentation system that we are using, nothing changed there really from last time, except that we had some issues last time with our, we got HDMI over cat5 cables, so we also got a couple of these HDMI over fiber devices devices that help us, here the AV people also had HDMI over wi‑fi, 5 gigahertz and 2.4 gigahertz, but we asked them not to use it because we didn't want to add an extra thing to check if there were any wi‑fi issues.
But these HDMI over fiber devices work very well, so we're happy with those. What we did notice when we turned on the meeting kit, one of them in this room I think it was the MAC‑mini, one of the MAC‑minis didn't boot anymore. Fortunately we bring a third kit as a spare, and we put that into use. When we get back to Amsterdam, we will have to see what's happening with that MAC‑mini, but probably will replace it.
This is also on the PDF, may be interesting to look at how we do this, the presentation setup. To make sure that when someone is presenting with the other MAC‑mini, the other presentation is already being put ready to, for the next presenter. And this is how that setup works if you are interested in those details. You can also have a look at the PDF.
Then some technical issues. There were in the beginning some Kahoot issues, and that was unfortunate for Alistair, but he really winged it, he was great at dealing with the issues, and it also didn't help that in the Monday, in the first session, we experienced some issues with the presentation freezing every so often, because of one of the video mixers being used gave issues and that was solved after, but yeah, that wasn't nice for Alistair and also for Mirjam to deal with.
There is a short summary of those issues. I think I mentioned most of these. There was also an access point somewhere in that corner that had issues, we replaced it, still issues, and we managed to see it was an issue with the wiring, and then that was also fixed.
So, not that many exciting issues. A bit boring even, I must say, compared to RIPE meetings where we sometimes have water leaking in the server room, having to use umbrellas to fix it, and so hopefully the next RIPE meeting will be more eventful. No, I don't know.
Some statistics. Also, just some graphs from Grafana that show it was rather busy on the Tuesday, and I think on the Wednesday, you see some more activity in the evening, that might have to do with a certain BoF happening in the Da Vinci room, but also in the other rooms.
So the NAT64 traffic, is also interesting to have a look at. This is the traffic that was flowing through there. The translated v4 to v6 traffic.
Then, also an important statistic is the amount of coffee people have been drinking compared to the previous meetings. So, the number you see 1,000 is not entirely the correct number probably but we asked the barista if they have the number ready, they said that this is what they expect it is, but on Monday they will have a better number to share. But that will be too late for us. But this is what they think is rounds up to..
Then, some thanks, thanks to the stenographers for their hard work.
STENOGRAPHER: You're welcome. A long week! We had a few strange things coming up!!
MENNO SCHEPERS: We'd like to thank them for all the jokes they sometimes put in there. It's really funny. Also sometimes these little mistakes are also really cool, like General Linkova. Thanks for making us laugh.
STENOGRAPHER: They are not intentional!
MENNO SCHEPERS: Also a special thanks to Meetecho, we have a hybrid meeting. The meeting is ‑‑ there is lots of people joining us online, and that's all thanks to Meetecho. Thank you.
.
(Applause)
.
And also, a big special thanks to BIX, Bulgaria Internet Exchange, they helped us before the meeting, all the months before and also during the meeting, their support was amazing. So, thanks a lot BIX.
(Applause)
That was it from me. Any questions or comments?
BRIAN NISBET: We have enough time for a couple of brief questions.
AUDIENCE SPEAKER: Mick O'Donovan, Asiera, it was mentioned at the GM I think, boring is good, especially for the meeting network. It's fantastic. Just a practical question about the Jool statistics. Did you measure it last time and how much of a reduction of that has there been this meeting?
MENNO SCHEPERS: We have might have, but I didn't compare it yet.
JEN LINKOVA: Regarding your multicast stuff, do you do multicast to Unicast conversion on APs? Have you tried to turn it off?
MENNO SCHEPERS: We played with the feature, yeah.
JEN LINKOVA: Because the first thing you might want to do if you see issues, it's not a proper long‑term solution but for troubleshooting.
AUDIENCE SPEAKER: Could you scroll back to the graphs.
BRIAN NISBET: We really don't have a lot of time here.
MENNO SCHEPERS: This one you mean?
AUDIENCE SPEAKER: Don't you think they look too symmetrical.
MENNO SCHEPERS: Yeah.
AUDIENCE SPEAKER: Any possible explanation?
MENNO SCHEPERS: That they are not symmetrical.
AUDIENCE SPEAKER: They are.
MENNO SCHEPERS: These?
AUDIENCE SPEAKER: They shouldn't be.
MENNO SCHEPERS: They are slightly different.
AUDIENCE SPEAKER: If this is in and out they shouldn't be symmetrical. But they are completely.
MENNO SCHEPERS: We'll have a look at that. It's from the Grafana graphs today, yeah...
AUDIENCE SPEAKER: The red and blue line shouldn't mirror each other.
BRIAN NISBET: They are going to look into it.
AUDIENCE SPEAKER: Thank you for finding these issues with multicast and IPv6. Jen and I doing some IETF stuff about this, please bring them to our attention to we can get them maybe fixed somewhere between the IETF and the next meeting.
MENNO SCHEPERS: Thank you.
BRIAN NISBET: Thank you very, very much you and all the team for the amazing work you do.
(Big applause)
.
Okay. So, that's the end the Programme Committee bit of the meeting. Now we will pass back to the RIPE Chair to close it. Please, Mirjam.
MIRJAM KUHNE: Thank you, Brian. A few more people to thank I think during this session. This closes our RIPE 93 meeting here in Sofia. Here are some more statistics. The people who have attended, participated during the week. We had over 600 people checked in both, you can see the numbers there online and here on site. The we had 88 newcomers on site, which is great and then another 40 online. And this you also had the number of countries. 54 countries represented, which is great, even here on site.
Thanks for hosting us, Sofia. The weather wasn't quite what it shows on the picture but I still hope you enjoyed being in Sofia and had a bit of time to explore the city. This was the first time we were here, we might be back, we'll see what the future holds.
Also, thanks to, I want to highlight these local hubs. It's always a great addition. They are not visible during the week, but they are there and they participating and they are listening. We had three local hubs this time from Poland and, it's recurring, they have been a local hub and participating for a few years now and we had one at Vilnius, Lithuania, in kind of preparation for the upcoming meeting. Also one in Amsterdam, hosted there.
Thanks again to our RIPE Code of Conduct team, you already saw the report. It's really important to have the team here and to keep us safe during the meeting and to help provide a safe environment. Maybe give them another round of applause.
(Applause)
Here you see another slide with a lot of pictures on it. These are the current RIPE Working Group chairs, you have seen them during the week. They have also done a fantastic job running the meetings, even some helped each other out because chairs be there so others stepped in and helped which I really like this collaboration between the Working Group Chairs. Will maybe also give them a round of applause.
(Applause)
We had two co‑chairs, two Working Group Chairs stepped down during this meeting and they had announced that before, they are not here so we can't give them a gift. But Achilleas has been co‑Chair of the cooperation for a long time and Ignas as has been a co‑chair of the Routing Working Group so thank you for their long service in these two Working Groups.
(Applause)
And we'll have a few more new cases on that big chart of Working Group Chairs. So Mick O'Donovan, you just heard on the microphone, is joining the Routing Working Group co‑Chair team. Bastiaan and Peter are joining the cooperation team. I added Peter here because he joined technically last time, but it was kind of after the meeting, so he is now fully incorporated into the team and Janos and Christian and Sascha, they are continuing in their respective Working Group as a co‑Chair moving forward. So, it's great to have a complete team there.
Then again thanks to the Programme Committee who had I think, you know, provided a really good programme, including the BoFs and the tutorials and also good discussions this morning and Monday and Tuesday, so thank you the Programme Committee for their work.
(Applause)
.
Two Programme Committee members terms ended. These were Babak and Antonio, you have heard this that they are also ‑‑ nothing much changed on this slide, outgoing, incoming, both for these two to continue. So congratulations to you too.
(Applause)
And then of course last but not least the amazing meeting organising team, already mentioned, the tech team, the stenographers, and the Meetecho team. You see a picture here of the kind of core meeting team, some are standing over there, thank you very much, especially Kajal as the main meeting organiser this time.
(Applause)
And all the other colleagues from the RIPE NCC, you can see all the RIPE NCC staff who were here at this meeting,s supporting you in one or other functions, either with your tickets, with audit, requests with training, testing, you know, user testing, or just basically providing the services here at the meeting. So thank you all, thank you very much to the RIPE NCC for organising this one.
(Applause)
.
Then of course we have the sponsors. They were already mentioned. I would like to might a few, give some, handsome out to some of the sponsors, I don't know where my little gift bags are. So a special thanks to BIX BG, the Bulgaria Internet Exchange who have been mosting us here in Sofia and they have put a lot of work into preparing. I don't know if there is somebody here from BIX, who can receive a gift of appreciation.
(Applause)
Mila has been the main organiser also behind the social events and the dinner.
SPEAKER: I want to thank you the RIPE NCC, I want to thank all the RIPE community and BIX BG for the responsibility to be part of RIPE 93. I want to thank you all of you dear friends for coming and supporting the event. I hope you had a great time in Sofia, and have a safe trip home and hope to see you soon. Thank you.
(Applause)
MIRJAM KUHNE: Thanks Mila. We also would like to explicitly thank for platinum sponsor, AWS, is someone here, is Clara here or someone else? They have left already. So, we will see how we get that to you as usual.
Thanks anyway for the platinum, long term platinum sponsor, AWS.
(Applause)
And last but not least, we also have IPXO the gold sponsor here who will also be the host for the next time.
(Applause)
Who doesn't like it. Exactly.
Right. And that brings me almost to the end. Please don't forget to rate the talks, it would be really helpful for both the working group chairs and the Programme Committee, and also as the last slide here, I would like to keep this on, it's really important to give us general feedback about the meeting, the venue, anything you'd like to bring back to us, also the content, any topics you'd like to see, so I'll leave that up here for a little while, you can scan the code, fill it in right now.
Last time we had very little feedback, I'd look to see more this time because otherwise it's a bit hard to see the trend and see if we are getting better or worse. So please, provide us some feedback.
And then last but not least, I hope you will join us again next time. I'd like to remind you here at this point that as we have kind of started this tradition to open registration right at the end of this meeting, I'm going to go back so you can still scan that code if you want.
Usually we open registration at this point for the next meeting. However, we are in the process of transitioning from our old home grown registration system into a new shiny system, and we are still working on this so this will be ready in January, when we'll inform you about this. However, if you are already know you are planning to go, you can already go to the website, RIPE 94.RIPE.net and pick your accommodation, for instance, so I would like to advise you to look at that because there is not that much available. It's going to also be a busy time in Vilnius in May next year so please prepare early, and we will inform you about this of course also on the mailing list, and also on the website, that's correct, I am not lying.
That's it. That brings me to an end. Thank you very much for joining us here at RIPE 93. I hope you had a good time, and that's it from me.
(Applause)
.
.
.
PAULIUS JUDICKAS: I have the easiest job to invite everyone to the next one, and BIX BG we have big shoes to fill for next year. But we hope to see you all in Vilnius next year, May 24‑28. It's in the Radisson Blu, an amazing place of the city, I promise you'll love it.
So, what can you expect from Vilnius? First of all, our old town, our history, it's been in UNESCO for what, almost 20, 30 years by now. I'll take a chance to do a quick Lithuanian lesson because I know none of you know a single word in Lithuanian. At least you will know how to say hello whenever you land near the airport. It's labas. So, all together. One, two, three, Labas!
So, apart from the old town obviously and our meeting and our talks, the hallway talks and socials, there is a huge night life in Vilnius as well. This is what we are famous for and Vilnius is considered to be an overlooked capital of Europe. So, I can promise you a really lively town, an amazing old town, amazing modern districts, live music. So if you have up for evening walks, contact us, we're happy to share what you can see, what you can do in Vilnius. And well, see you in a couple of months. Thank you all.
(Applause)
.