RIPE 93. Main hall. 2pm
RIPE NCC services.
ROB EVANS: Hello, good afternoon and welcome to your favourite working group. So, welcome to the RIPE NCC Services Working Group, my name is Rob Evans and I am co‑chair along with Steffan Wahl and Janos Zkako who cannot be here at this meeting. I will run through the agenda and kick things off, we are trying to learn from what happened in Edinburgh, I have a much more compact agenda, I will kick things off with a bit of a welcome and a few issues, co‑chair selection and Hans Petter is going to do a session on the draft activity plan and budget and Sjoerd is going to talk about the RIPE NCC technical infrastructure, renewing the technical infrastructure, those are going to take 35 minutes and 50 minutes, as you can see the meeting is already set up for the GM, we don't necessarily need to be, to beat such a hasty departure as we did in Edinburgh, we have planned that we can maybe flex a little bit but hopefully won't be taking too much coffee time.
So first of all thanks to the steno, Tina, at the front here, and whoever is taking the minutes, Adonis, for taking the minutes, thank you very much, appreciate that.
We have sent the minutes of the last meeting to the list, and we haven't received any comments so I shall declare those final. And now it's the co‑chair selection so it's Janos's turn to step down, we sent the call for people who are interested four weeks ago, it closed two weeks ago. We had a couple of people who showed a vague interest but it was only Janos that put himself forward again. There was support and no objections on the list so I would like to congratulate Janos for continuing to assists us. Thank you Janos, I don't think he is joining us remotely but we give him a round of applause away.
(APPLAUSE.)
A quick check, when I sent the agenda to the list I mentioned there was some homework which was to make sure that we are ready for active discussion session and read the draft activity plan and budget, hands up who read the draft activity plan and budget? OK. A few. OK. We'll see what happens after doing Hans Petter's talk. A few announcement before we go on, the voting period for the PC election is open until tomorrow at 5pm, so log on to the meeting website and vote.
There is no official social event this evening, you are free to explore the city, have dinner with friends, or you can join the board games BoF which is in the Newton Hall from 6 until 8.30, so Newton is opposite the side room just, if I get my bearings, down the corridor down there.
So we will go straight from here to the general meeting that will start the 4pm, only RIPE NCC members who have registered may attend and I believe the deadline expired a few minutes ago, so if you haven't got your GM sticker, pick it up before 4pm or you will not be allowed back in the room for the GM. The GM voting will open towards the end of the GM and you can vote until 9am on Friday morning.
So with that, I shall hand over to Hans Petter and kick things off. Thank you Hans Petter.
HANS PETTER HOLEN: So thank you very much. I am Hans Petter Holen, managing director of the RIPE NCC and I am going to take you through the draft activity plan and budget for next year. I have split the presentation in pieces so that we will stop and ask for questions after each part so you don't have to wait all the way until the end to start, you will see how this works, if you have a lot of questions to the registry, then we will never hear the rest of me so that's of course an option you have to get this over with soon.
So, we now entering a new strategic cycle, I presented this strategy at the last RIPE meeting and the board approved that in turn. And we know going to a period where we have more focus on resilience, security and independence, and a big part will be the topic of the next presentation which is renewing our IT infrastructure while continuing to modernise services and reduce technical debt. And as you know, once you have done IT or software development for some years, you tend to accumulate things that you so wish you had time to clean up years ago and that is now one of the things that we are going to have focus ongoing forward.
So, since we obviously have to spend a lot of money on new IT infrastructure, we need to keep a tight ship on other costs, we need to keep focusing on automation and efficiencies so we can partly absorb this cost but not only where possible. We are also preparing for evolving governance, regulatory and geopolitical environments, including the new IR RIR government document. So alongside infrastructure work we have stayed ten years in our office and the contract runs for another ten years, we looked at the market and decided not to move as I told you a year back. But this means that we will have to do some work on the offices to make them fit for purpose going forward. That's the context that we are in. The new strategy focuses on registry accuracy which is not now, internet resilience, scaleability and routing security, RPQI, ASPA, scaleability, IPv6 and internet resilience in general which is assisting you to make sure that the internet works because we only pay a tiny, tiny part of that. We have measurement services that many of you appreciate and use. And turning that vast amount of raw data into insight is going to be a strategic focus area. Community partnership and trust, the RIPE NCC it trust is a big cause of our community, we need to continue to work closely with the community to remain a trusted partner for you, for our members and for governments and others.
Governance is increasingly important, we have a lot of people outside this room, governments, others, looking at us so making sure that we have a robust governance going forward as important if not even more important now.
And then none of this is possible without our excellent staff and maintaining and even improving the agility and execution culture is the last of the bullet points here.
So budget.. and that's always what everybody wants to talk about. Strengthening the resilience of the services required significant investment in our IT infrastructure and to fund this, we are not only adding costs we are also looking at where we can save costs across the organisation. You voted to increase the LIR account fee to 1894 euro, an increase of 94 euro. In May, there is no increase now, but that's the income that was set already then. Normally I would try to stay within those boundaries but unfortunately with the plans we have for next year, the expenses that we budgeting with is 44.2 million, leading to a negative result of 1.7 million. Luckily the RIPE NCC is in the excellent position that we have large reserves, this is not a critical thing, and the board allows me to put forward a budget like this because we have the reserves of the and because of the importance of improving the infrastructure.
This is a draft budget, it's still under discussion, you can give input and the board will approve the final version in December.
Alongside this, we are adding some people to the organisation in the IT area. CAPEX is budgeted to 10.5 million mainly for infrastructure and renewal and office repairs and I will come back to that. The number, you have seen this every year, the vast majority of our income comes from existing membership fees, independent resource fees. ASNs and member sign up fees and other income is kind of visible here and then you have smaller amounts for new member services fee, RIPE meetings, 300K, legacy resource fees and reopening fees. When you look at the split of the budget, you can see that technology and organisational sustainability are the biggest parts but the supporting the registry and external engagement and community.
Now, looking at the changes of the budget, we created this graph and it's a bit difficult to read so if you zoom in on it, you can see a bit more where the changes are. So personnel costs because we have budgeted with increasing salaries according to the market development in the Netherlands, we have project costs for the new infrastructure project of 1.8 million, but we are also budgeting with saving a million euro on all the infrastructure support contracts and other things that ends. So it's not just adding, we are also saving there.
Since we are investing, the depreciation of IT infrastructure, exactly how much that depreciation will be next year depends on when we finish the project because the investment is written off over the lifetime of the equipment so if we make all the purchases in January, it will be the full 12 months, if we make it in the middle of the year, it will be the half of that.
The office investment is much smaller when you look at the depreciations and we have some general budget reductions, which then brings it down another 300K and that ends up with the 44 million, that's the sort of big lines in the changes in the budget.
If you look at this per division, the only big change here is really in technology, the registry is flat, well there is a 15K increase, it's kind of almost like we should have just kept it at zero to make the story easy. External engagement and community are running on the same budget third year in a row, so that is going to be a challenge with the increasing fees. Organisational sustainability, that's also no change. Depreciation of course increases due to the investments and the total budget of the RIPE NCC then increases with 3 million and 50,000 euro, which is a 7% increase.
So the 14% increase in technology turns into 15%, 7% increase on the top line because the depreciation also adds up to this.
So, main changes. Registry broadly stable, we need to continue optimisation to absorb the increased salary costs that we get every year and of course you as members don't like to that we have to pay staff more, of course staff is not thrilled of not getting any pay raise and yes you are all employees of some companies, you know exactly how that works out, right, but we want to make sure that we keep the same pay policy going forward, we do not want to disrupt that because the RIPE NCC is nothing without the staff.
Technology, I talked a bit about that already, the largest increases are in IT, external engagement and community, we need to shift some of the resources towards events to keep the same level there. And on organisational sustainability, the shift between the activities, facilities and people and culture will increase, and will lower the budget slightly in finance, legal and information security risk and compliance so the reduction in information and risk and compliance is something I don't like but it's really not touching the budget, it's taking out of the reserve that we had there because I always kept enough money for unforeseen things there.
And we have in this budget also split out project delivery and agility out of the office of the managing directors, that's a separate budget line.
Technology, CAPEX, so the big CAPEX part is 6.6 million euro, we'll talk more about this in the GM, there's also other CAPEX replacing some of the hardware that's come to the end of the life in some of the K‑root and AuthDNS instances, RIS servers need some replacement and then we have the regular upgrades of laptops and network equipment for the RIPE meetings.
And then in the organisational sustainability, there's almost 2.2 million budgeted for office renovations.
So if you are not familiar with the organisation, you have a map here to see which parts fits in which of the divisions, going to the registry which is led by Gabor de Wit, I don't know Gabor are you here, yes, he is sitting there, if you don't want to ask questions here about the registry, you can ask questions of him afterwards.
As I said, the budget is broadly stable. The focus areas for next year is maintain an accurate and up to date registration data by reaching the target of 2,800 assisted registry checks and introducing automated monitoring. We want to introduce self service registry checks to increase verification without increasing manual workload, we want to extend automation for repetitive manual tasks, who wants to do them these days and increase verification using without increasing manual workload.
So stream lining and automating is kind of the main theme of what we are going to do here. And unfortunately we are also seeing an increase in fraud so strengthening fraud detection is something we are doing and as you also may be very aware, EU has sort of upped their game on sanctions, so even improving the sanctions screening is something that we need to increase or focus on.
And then part of what we also are trying to do is to make it easier for you so as you may know already, we have a user study, a user experience study out here on various aspects of the LIR portal, if you are interested in that, feel free to contact my colleagues to give input on this.
So, if you look at the difference between the different areas in the registry, not really that much changed, savings, automations and self service to absorb inflation and support increased registry monitoring activity. So that's the registry, are there any questions to the registry part? Such as are we focusing on the right things? Are we allocating our resources in the right places? What could we do differently?
Nobody. Well Gabor it seems like your plans are perfect!
Want me to go on? OK. Technology. Of course that's what everybody is waiting for but the real meat of this actually comes in the next presentation, I am going to give the same overview of technology so that I did for the registry if you want a technical content, you have to wait for Sjoerd to come on stage after me.
Budget as I already said, 14% budget increase.
Renew and modernise technical infrastructure, already said that several times now. Moving away from cloud architectures and reducing single points of failure.
Years back we thought that cloud would solve all our problems, some of you were very vocal against that and yes, guess what, you were right. There are still some services we will use in the cloud but having the main production on solid ground above sea level in multiple countries in Europe in our service region is going to set the direction going forward.
Now building the new infrastructure a huge project, migrating existing services is another huge project. And we want to do this while also improving security availability and maintainability. Improve ROAs and ASPA guidance and RPKI and maintain SOC2 assurance, we want to be on the bleeding edge of development of RPKI and connected protocols, we are active there in the IETF and collaborating with our other RIRs, we don't only want to follow that, we want to be at the front line of this.
Improving the RIPE database security and available include DDoS protection, deployment, modernisation and containerisation, replace outdated K‑root and AuthDNS hardware and AS 112 infrastructure in key locations. This is regular lifecycle replacements. Enhance RIPEstat usability and integration, the RIS data set stability and RIPE Atlas anchor self service while exploring better data storage and new measurement options. And on the measurements area, I think it's really important that you go to the MAT Working Group and give input whenever we post our roadmaps that you come up with your ideas and give ideas to our product manager in that area so that we can drive development and innovation in that area.
Yeah, already mentioned that the real budget changes are in IT, you can hardly see it, it's really light pink to the right here, there's also an increase in business application where we already increased the team with one person.
Questions.
BRIAN NISBET: So this is obviously, as you said, a huge project that you are embarking on, and I guess my, and I think it's the right thing to do, my concern is the potential delays, the potential, you know, we have just been through something similar which took a lot longer than we were expecting to, so what kind of ‑‑ I guess what type of contingencies do you have in place there and also from the point of view of internal staffing and balancing the operations piece against having enough staff to work on the migration project, because the two of those often interact in bad ways.
HANS PETTER HOLEN: No, and that is definitely the challenge and it's to some extent more of a question from Sjoerd to answer and maybe you want to save it until after the presentation. The thing that I have done in the budget is to allow for increasing FDs, so we are not only going to do it with consultants, we are going to do it with staff and hire staff with the right skills to maintain this afterwards in areas where we don't have it so that's, you know, there.
Does it take shorter or longer time, yeah you know if you had asked me in my impatient mood, I said we should have been done already, but that's not realistic. Is it urgent? Well some equipment comes to end of life and in some cases that's critical, in some cases having hardware that out of maintenance may not be a big problem if you have spare parts available. But if don't get security patches any more, you are in a very different position, right. So that's part of what's driving us forward, and Sjoerd will talk more about the timeline there, but it is a big risk of course.
BRIAN NISBET: I am asking you from the budget point of view on that, to be clear. Thank you.
HANS PETTER HOLEN: If we don't do it the risk is higher, so that's the thing, right.
ROB EVANS: I will remind online participants that you can type into the Q&A box or also raise your hand there and ask for mic permissions, but... carry on.
HANS PETTER HOLEN: So, external engagement and community. Well, that's basically the department that makes sure that you can come here, right. That's the most visible part of what they are doing. It's the big RIPE meeting. We are keeping that budget the same. And of course we will try to keep the activity level at the same level, and that's going to be the real change here. So maintain the core portfolio of the RIPE meetings, regional meetings and local hub with simpler delivery and targeted outreach. We have picked the locations and signed contracts for next year, so there is only so much we can do with those. We will continue to support NOGs, RAci, fellowship, mentoring and newcomers programmes while reviewing RAci and RIPE Atlas ambassadors. We don't want want to do what we do, we want to review it and see if we can do better for RAci and Atlas ambassadors. Deliver online engagement and fewer more targeted in person training courses, so that's an area where it's easy to save, more online, less on prem, or in person, increasingly through hosted courses, that's been a successor so far rather than renting places, we find a host that can help us host the training course.
Release in person courses on BGP fundamentals and internet measurements and a new e‑learning course for law enforcement agencies. And the more educated law enforcement agencies are, the less time we spend supporting them with their requests, this is also important balance to keep the pressure on the registry and legal down.
Focus on public policy engagement on higher impact work in the service region, supported by targeted research and insights.
And we have been through a year now with a lot of international governance going on, renewal of the mandate for IDF, making that permanent, this planning part is next month, maybe next year will be a bit more quiet on that front but we will see.
So shifting the budget here, there is an increase in the community building from 3.6 to 4 million an that's basically to be able to maintain the cost of organising activities because it is getting increasingly expensive to travel and rent venues like this.
Membership engagement goes down a bit. Technical training and development goes down a bit. Curriculum development goes up a bit. And then public policy and internet governance goes down a bit and then research goes down a bit. So basically it's a shift from almost everywhere else community building.
Questions.
ROB EVANS: We have a question that came in. It might be more towards the last session but we will take it.
KURT: If you want to end the political risk, you need more than two locations, yes it's expensive, but...
HANS PETTER HOLEN: Yes, let's build the two first two and we can discuss the third or the seventh or how further we go.
I challenged my people internally saying I wanted a sign that scales to seven locations, they thought I was insane, thank you for giving that input.
PETER KOCH: I think everything that you put up in the budget is useful for this part. I am just wondering you explained the budget and also the engagement and the direction of internet governance is being reduced, I understand that the planning part is a big thing this year and we had high focus issues the previous two years, it's great if it doesn't need so much budget but I am worried about the future co‑operation and co‑ordination with the rest of the technical community beyond the number space and if that's budget neutral, that's fine but I would appreciate if that would appear on the priorities a bit higher.
HANS PETTER HOLEN: Yes and thanks for that, Petter. And if you see here the reduction in the budget on public policy and internet governance, it's 50,000, that's mostly the travel budget, right, and with all these big events this year, we have sent a lot people there in order to be very visible an engaged so we are more scaling back to the more natural level of having people present but not as many.
SPEAKER: Thank you, it's a tiny item in the overall thing, I am more worried about the priorities in that co‑ordination part, even if if that's budget neutral and maybe you can think about that. Thank you.
HANS PETTER HOLEN: So, you want to double it?
SPEAKER: No, I said it's like the 50,000, I don't know, the point is like really the continued co‑ordination.
HANS PETTER HOLEN: Thank you.
Then it's to remain, organisation, people and culture, information security risk and compliance, legal, finance, facilities, office of the managing director that covers my costs and the costs of the board, project delivery and agility which is a new area and funding of the RIPE chair.
So this is then led by the nice people that you can see the pictures of here and I think you will know them all.
The budget on the top is no change, that's an easy thing to defend. The biggest effort here which you don't see in the operational expenses but see in investments is the refurbishment of the office? Amsterdam for necessary repairs and updates. Introducing a new performance management system and implement personnel continuity plans. Support adoption of the RIR governance document and update the legal framework, there will be a presentation in the community plenary tomorrow on the status of that legal framework and if there is one big risk in the budget, what's it going to cost to implement that new RIR governance document.
Maintain SOC2 type 2 assurance and ISO certificates, we are in the process of doing the external audit on am IS 0 and we had an internal audit that was only and there were no major deviations, I think we are in a very good position and the reason we are doing all this is to implement the legislation, none of the legislations says that we have to have an ISO certificate but all of the things that are in the legislations is in the information security management system and is now checked and audited so that we can document that.
Continued cost control and improving financial insights and consult members on outstanding charging scheme task force recommendations. So there were things in the charging scheme task force we didn't implement and we'll look into what more of that we can implement next year. So new elements of charging rather than increasing the fee for members, for LIRs, that will be in the May GM, and a vote on what direction the members want to take that.
And then yeah establishing a project delivery and agility team to improve prioritisation and co‑ordination across the organisation, we have had good results with that this year and now we want to make it a permanent part of the organisation.
So as you can see here there is very little movement between the different areas as I already mentioned info sec is taking out some of the reserves, it is not cutting anything, legal we found a new legal partner that actually is better and invoices us less so that was a great win‑win and then yeah facilities, there are some increased costs around the lease and also insurances tend to increase and then office the managing director, we have taken out the bit but we have added project delivery and agility as a separate part.
And the RIPE chair budget increases with 10K to offset the increase to travel costs.
Questions to that.
It's very quiet: I still have seven and a half minutes left.
ROB EVANS: I am wondering whether to put you on the spot or maybe not so much you on the spot, there was a topic that caused some discussion at the last RIPE meeting. Now we saw the NCC adapt the response according to community input, I am wondering was there any follow up afterwards from that, we saw the updated statement into ITU‑T, was there anything subsequent to that
HANS PETTER HOLEN: Which topic are you talking about?
ROB EVANS: ENUM.
HANS PETTER HOLEN: Oh my favourite topic, so it's a rather small registry, we haven't described it specifically in the activity plan, maybe we should add it as an activity there. Cleaning it up, the ITU‑T study group something has now agreed on their internal procedures on what to do if an operator of a delegated registry do not respond to us, or do not keep their service up to date. And unfortunately it's kind of like 50/50 of the delegations that are in working, full working conditions. So there is clean up to be done there. So yeah, we can report more on the result of that clean up going forward, but yeah, I don't really have an in‑depth presentation on that this time.
ROB EVANS: It was more just kind of a follow up really from RIPE 92 just to see what happened as a result of that statement.
HANS PETTER HOLEN: Yes. I see Jim running to the microphone.
JIM REID: Thank you. I have actually got two comments, one is about this ENUM situation and the second one is more general, nothing to do with ENUM at all, taking the ENUM situation, this is still unfortunately a live issue inside the ITU‑T, it's not quite done and dusted yet and unfortunately one or two people that are got neglect better to do with the time has been using this as an excuse to make work for them, it's going to be a problem for the NCC, I think you are going to have to continue to participate and try and get that bear to go back to sleep. The agency woke it up, it's time to put it back to sleep again and it's not quite there yet, this is going to take an ongoing commitment from the NCC to make sure this prevails. Some members will be supportive, let's put this thing and make it sleep but there will be others loo looking for an excuse to make work for themselves and engineer to try and persuade those people not to create work for themselves and create more needless hassle, that's an ongoing problem we are going to have another year or so at least.
The second question I wanted to talk about is actually insurance, surprisingly. Two aspects of that is back in my time the board we had some discussions around professional indemnity insurance for the board members. And I wonder if that's ever been reviewed because I am not sure if the premiums are in place for the level of liability has been reviewed in light of what's been happening on the internet over the last however many years it's been.
The second consideration, personal interest here is on the insurance that happens at RIPE events, some of you might remember I had an unscheduled swim in Lisbon. Now, I wasn't particularly hurt, and I had my professional travel insurance cover the cost of one or two things I was out of pocket. But it got me thinking what happens, perish the thought, if there was a serious incident at a RIPE social event, where is the exposure for liability there, and is RIPE or RIPE NCC adequately covered for that? I don't know if anything has been done about it.
HANS PETTER HOLEN: So for the board liability part, that is in place and the coverage has been increased, and all new board members tend to ask that question and review the policy. The first time we brought it up it was a bit difficult to find an insurance company that wanted to increase it, but we were able to resolve that with some work.
So I think it's basically for the board members to tell me whether they think that coverage is good enough or not but that is in place.
JIM REID: I think this is one of these issues where it's almost like a file and forget thing. You set up the policy and then you just renew the premium every year. But whether there's a review to see is the level of coverage adequate.
HANS PETTER HOLEN: There's been a thorough review of the policy, it's not just continuing the policy.
JIM REID: OK.
HANS PETTER HOLEN: And I think some board members have even used external legal advice also to make sure that they are properly covered so, yeah.
On events, I have seen that the terms and conditions I have to sign going to an ICANN meeting is many more pages than going to a RIPE meeting, so if you want to protect the RIPE NCC in a better way, we way want to copy some of their terms and conditions because that's completely avoiding all liability whatsoever of the organiser.
I think as a starting point everybody is responsible for their own travel insurance and their own things but of course there will be situations where the RIPE NCC may be liable as well this is something we are keeping a close eye on and if there is interest from the community to share more about that, I am sure we can do that but I can't speak to the details on that.
JIM REID: Thanks, to my mind it's perfectly reasonable to say you enter these meetings at your own risk but having more clarity around that would be very helpful.
HANS PETTER HOLEN: Thank you.
ROB EVANS: Any other questions, anything online? No. OK. Is that it?
HANS PETTER HOLEN: That was it from me, I still have one and a half minutes left.
I will leave you!
ROB EVANS: Feel free, thank you, Hans Petter.
(APPLAUSE.)
And now Sjoerd to talk about the RIPE NCC technical infrastructure.
SJOERD WOLTERS: Thank you very much, I am happy with a little extra time, because maybe we need it, so I am planning to present maximum 30 minutes, and then we will have 20 minutes for questions which I think will be here. Am I loud and clear like this? I think so.
So my name is Sjoerd, I recently started as the new CTO for RIPE NCC, happy to be involved in the company and as well as the community. And of course major focus for me in this role will be the infrastructure renewal that Hans Petter talked about in a previous RIPE meeting and also in his presentation that you just seen.
So part of it will be repeating, which you already know, so I will go quickly through that areas and I will dive into the area you have not seen before because for instance the presentation has been uploaded today, so I think that will be new for most of you.
So let me start with why now. So I think the first thing is that the international environment is changing, we as an organisation started looking at cloud I think for many years ago, and also defined a cloud strategy, Hans Petter said it today that's not a strategy any more so looking at our strategy for the coming years, we decided to move away from the cloud and I think it should be clarified we don't have a big cloud footprint at the moment.
So it's not like moving everything back from the cloud, so our main footprint we have AWS services and we have Google cloud services, but it's mainly for internet measurement and of course there's more public data and it is also for identity and exit management which is more sensitive, that's the first priority there to create a fall back in that area. But in the financial terms, moving away from the cloud does not mean moving away from cloud expenses to unpremised expenses because their footprint is relatively limited so it's not moving towards the cloud any more but having control over our own services on premises.
That I would like to clarify so now many reasons of course to move away from the cloud are the changing geopolitical circumstances, which you probably all would recommend, also increasing compliance pressures and you can name various ones but yeah, the standards that we have to live up to are getting more and more, yeah, stretching so that also makes us reconsider what kind of infrastructure needs that strategy, that's the external part. Internally we have an infrastructure that we have built upon for many, many years it also is technical debts and past year it's been explained by Hans Petter, in a previous presentation we had two incidents and then we found out that when we have an incident in one data centre, it can also affect the other data centre. So there we don't have full independence of these data centres. In case these incidents we were able to solve it relatively quickly, and we can't guarantee for instance there was a big data centre fire a few months ago in Norsee, we are situated in Norsee, luckily it was a different location, but for us it was an additional warning sign that we should be sure that we should become independent of specific data centres so removing this independence and have a different set up in place. These are major reasons to look at our infrastructure which I with like to share a little bit more about.
So I think one of the good things with our current infrastructure is that we have quite an efficient set up, I have a picture of our infrastructure that will explain a little bit further about in the next slide, but in general you could say that we have two data centres each having roughly speaking a single equipment for the different functions, we have a network that stretches over two data centres so yeah, it acts as virtually one network and it is of course an efficient set up but has some down sides. It has been designed for two data centres being close to each other. It is an active set up that's immediately fail over to another data centre. We were located in two Amsterdam data centres until more than a year, ago. And there because we have flooding risk, we would like to change the set up so we moved one data centre to Nieuwveen, a big project so at least the flooding risk was mitigated because Nieuwveen is above sea level, one of the rare areas in the Netherlands you can have that so that was I think a successful change.
But the downside of that is that our storage environment that we use was not able to cope with the latency of the distance that was, yeah, that we face, it was about 40 kilometers distance and you saw the storage performance was degrading and we were still migrating from an more business storage environment to you could say a consumer grade environment to save costs and in doing so, we faced storing set up did not perform well enough for our services so also creating unavailability.
So we had to do something so our current situation is that we implemented these two storage environments in one single data centre, with a back up in the other data centre so we have geographical redundancy. But it does mean if we would have failure in the primary data centre for the storage, that we have to recover from the back up environment from the other data centre. And of course that will take quite a significant time because the primary equipment will not be there then, because it's a one Joe graphical location, so it's not acceptable risk, it was one of the reasons to start thinking about this programme and also the performance of this you could say consumer grade environment was in general also not up to the standard that we were used to working with another environment in the past.
So that was a major reason to look at changing our set up. And the last one is our virtualisation platform so in order to not have a lot of maintenance, we had our computer and storage platforms set up by a supplier like pre‑installed environment on the hardware where software is also managed by the supplier so that we have a VM ware based environment that we can just use. But in trying to extend that environment, we saw that the costs had risen, you can say exponentially, and we didn't have alternative because it was all in the same environment and still relying on VM wear, it was can much more expensive, they are doing this project, we thought we have to change this set up as well, there we would like to have another virtualisation platform to get out of this vendor locking, I am trying to find the right distance to the microphone, story.
So then, now visualising and what I just told you, you see the equipment that we have of our architecture you see that the two main blackbox, these are the two data centres, we have single equipment in each of the data centres. And the network is virtual layered that combines this equipment in one total set up, so they can work together well. And level one you can see is the Amsterdam area and the right one is the new behind area but they are still too close to each other, one of the things I would like to do, I can explain more about that in the next slide is we would like to spread our locations more. But that does not seem to fit this architecture, it's based on what network with active synchronisation and when you would like to build long distances, you will probably have too will I go latency issues with the equipment that we will use.
So goals that we like to achieve, we would like to remove the data centre interdependence that we have, improve our redundancy. We would like to spread our geographical footprint so no dependence on regional issues and as little dependence as possible on internal providers.
Doing so we would like to strengthen our security capabilities both in for instance our firewalls and inspection that we do, the threat intelligence as well as in creating smaller zones because the set up that we have now is a micro zone network set up, once you are in, you can have a lot of access within the network so there we would like to have smaller compartments in the network if somebody enters one part of the network, the radius is more limited and the remaining services are safe still.
Then on maintainability, yeah, we are working on this infrastructure for 20 years, there are quite some exotic settings that we have created of course this delivering all kinds of changes in the past and that that requires a lot of maintenance and a lot of manual work. So there we would like to standardise more in a set up that we get use from out of one architecture.
And also doing that we would like to use infrastructure as code as a basis, I don't think for everything, so it should be possible to do lot but we will look at what is beneficial, so activities that you are hardly ever do of course can still be manual and do not have to be automated so depending on the use case. The frequent activities or activities that have to be traced in audit for instance, we should be able to do with infrastructure as code. We are planning to do a lap environment we can experiment with the different suppliers and see what works out best. One of the main reasons to have this meeting is also to get your input on it, because we are in the high‑level design phase at the moment but we still have a lot of questions probably you do as well, also we do, we would like to align with you what works best for you, what things have not worked so we should not fall in the same pitfalls as you maybe have done, we would be happy to learn more about that, could be in the questions but I think afterwards would be the best way to do it in the hallway or something.
I think that's mainly this slide, the last point on infrastructure as code, engineers in the network team and infrastructure team an also the application areas because now they have to rely on people helping them out and doing a lot of activities so the aim for the application teams is that they can just click things together themselves within their knowledge area using self service because infrastructure as code is available. These are the main goals that we would like to achieve.
Then it's good of course if we keep a working on premise that we keep also the knowledge in place we have at the moment, I think we have quite some strong engineering capability internally. We are able to retain ownership of our systems with the strategy and I think also as well to have our costs under control, unfortunately our costs are significant so yeah we see the prices increasing of our vendor and equipment that we would like to buy but yeah, signing this off for quite some years will give some guarantee that costs will be not within the current estimates that we will have. So this is one of the reasons to do this project and make sure we are in control in that area as well.
So what will the new infrastructure look like.
First of all, we would like to spread our architecture but also standardise it, when Hans Petter Holen talked about, there were five data centres when he asked me to step in this programme and it's quite a little bit more than what we have now. But one main reason to ask for such a number or big numbers is that we have a standard architecture because and I feel it in the programme as well as soon as we now focus on having two full data centres in this programme, you feel that also the architecture you would like to chose changes quickly and that is not what we want, we want to have an architecture that is also usable for the third data centre and the fourth and the fifth, yeah, yeah, how many in the future would be required. It's not in scope at the moment so our scope is two full data centres to be delivered.
And there we are focusing on the western Europe, we have an Amsterdam data centre, probably we will move to another data centre mainly because we would like to have a new gauge, we would like to have a different set up there but the other data centre we are looking for a location probably in Germany or in France, of course with great connectivity, great peering possibilities, but also to do not stretch the location too far at the moment, also for our engineering team that then is able to travel and also for the connectivity costs that can come when the distance is too far away.
Now these data centres have to be independent, I think that is stated already, we will then are a redundant accept up within the data centre, having double equipment in each data centre, of course that contributes to the costs because then you have two data centres with double equipment instead of two data centres with single equipment. So that's a contributing factor, the resilience you would like to have and the availability when it also faces us comes with a cost indeed.
Doing this migration we would like to also apply high security standards, I think we already strive for it in the current architecture, we are limited by firewalls that we were using, but also for the legacy, that's been implemented, we can start anew and set things up the way we would like to. And last it will be probably already ever understood that it will be a greenfield approach for implementing this architecture.
So next to our current data centres the network we are planning to build two new data centres and migrate all our services from the old to the new.
Then adding to the data centre architecture, I think I already mentioned we will have an active set up within data centres, we will have an active stand by set up across data centres that helps to create a simple architecture with independence between the data centres so failing over immediately is done within the data centre of course not talking about disaster case, equipment or rec failure or those kind of examples. We will set up encrypted circuits that support not interconnections that would create between the data centres. And we will have maximum physical security in place so at this moment we have cages that can still be approached by climbing over fences, we don't think that's secure enough, we will have floor to ‑‑ floor to top fences to ensure the cages are not enterable with the strictest exit management that goes with it.
Then the network and this picture is built up, this is not the full picture yet. This is the part where I show the two full data centres where of course you recognise the redundant equipment that's in place in each data centre, but also the separate network for data centre so it's not a set up any more with a single network supporting the full two data centres model. We have an application to segmentation, moving away from the markers around creating... eah zones not for full applications for but for part of applications. So each application is expected to have separate VLANs for front‑end middle wear and for back‑end, so that we split this applications up and also in the interconnection with other applications, there will be firewalls between for instance something we don't have now between applications. I think this is yeah a big challenge in the project because that asks from us we know exactly what the interconnections are which can sometimes be challenging to investigate, we spent a lot of time in that so we could do the right sizing, I think we have enough new, it's not fully accurate but gives the impression of what sizing we would need.
Struck as code, I already mentioned, sorry, and IPv6 support I think is goes without saying but of course it's top of mind in everything we select in this programme.
Then the features, we plan to have a next generation firewall, our firewalls already have a default deny set up, this will be of course a different form having these next generation firewalls that can help better inspect but also better interact and engage with threats that come about, is it also possible to hear me like this.
I am struggling with the mic, sorry. I am moving back and forth a bit. So that will help us to inspect traffic better and immediately intervene, the current situation if there is suspicious traffic within our firewall, we have to manually do activities in a network after the detecting it separately, and that of course takes time and it's not allowing us to directly block things that we would not like to have in our network.
Then on load balancing, we will buy equipment that supports global service load balancing. And there are several models that can support that with the equipment that we have. So we have not decided the final four, we have a high‑level design phase now ready to as an important topic, we need to balance out the traffic between the two data centres with a large geographical spread and the global surge load balancing solution should be able to cover that.
So that's an important element in our architecture.
And for some areas we also will use flexible licensing models that would be for instance in the load balancing area where we can decide later on if we for instance have to supports quantum readiness or AI features or you name it, but also load balancing is one example for that, and application firewalls is an example for that. We now have an external search for that, I think it's quite usual, it's not the first priority to move over to an unprems application firewall, but it will be part of our plans after the base infrastructure implementation has been done.
And the then equipment that we are planning to buy that will allow us to.
Then compute and storage, maybe you have seen this I mentioned a lot of solutions yet, that's probably because we did not buy anything yet, of course they were waiting for budget approval but also for the final selection of the equipment that we would like to buy. I think this area is more sure because we plan to use Prox‑mox as the main solution for computer storage. This will be in a hyper converged compute and storage set up. And yeah, we will have high availability there also supporting complete rack failure, it also will be possible in such situation easily as well.
These will be data centre independent clusters, both for virtualisation and for storage. We will have the possibility to useS3 objects storage as an alternative way of storing our data.
And we have the flexibility to expand with whatever brand of equipment we would like. Which was not the case, as I explained, in both our storage and computer environment. So I think this will give far more flexibility and give the possibility to negotiate the cost levels that you will face.
Then on the solution, this is the final part, the back up, I think it's also the least clear for us yet what we completely will have as a solution but in general, we are aiming for a three two one one zero back up strategy, so having three copies of all our data related to our core activities, two of them in separate technology, one of them being off site and one of them being immutable, and yeah of course being tested heavily, so we are already with all our teams are spending quite some time on resilience and recovery strategies and testing them, and of course with back up it's very important that they will actually work in case of failure.
So that will be important part of the strategy.
And you see here this picture is not in one of the two data centres but it's in the separate area so we will not have a third full data centre, but we will have a witness location that also has the back up in it so that you have separate area where back ups are stored. And that that witness location also decides the priorities between the two data centres, so if there are issues in data centres, we have the third location deciding which data centre will be leading.
And specifically for solutions that require a short recovery point of objective, restoring data with loss like within an hour's time, we are planning to have storage in place that can do that quickly in the same data centre. So for solutions with long RPLs there we can just synchronise with the other data centres but for short RPLs, we will have set up that is also in the same data centre, so that you can have immediate synchronisation there without any latency or delays.
I think that's the main thing for the back up scenario. Then what does it look like in total. Here you can already guess because the pictures were already do, I do not see this as a full and accurate picture but as an impression, if you try to find a logic there will be some things not completely logic in it but it gives the overview, for instance one thing you can miss here is our metal service, so the non‑virtual areas, we do have them and they are situated in our data centres because this is not the scope of our project, it's not visualised here, so no we are not going to have everything in a virtual environment, but it's just the scope of the programme that does not express them in this picture as an example
I think all elements here are addressed. Maybe to add to it, we aim to support availability levels of 99.99 with this set up. It should be able to do more, but at least this would be the base that could be covered with data centre set up that we are putting in place.
Then how will we get there and that's of course also addresses the question asked before, we already spent quite time assessing our current situation, building on this for 20 years does come with the fact that you don't know everything any more so it required a lot of investigating, we do have a better picture now of our legacy, and that helped us define the requirements that we would like to set, so this has been done in the first few months of this year. After the summer period time we are spending time on the design, so we are now in the high‑level design phase. We intend to be done with the low level design for before the end of the year, it does still require a lot of work, a lot of people are involved in the programme within the NCC organisation, approximately about 50 people, a lot of them are from the application teams because of course it has quite some impacts, changing the architecture on the services that we run.
And in the meantime we will select the equipment that we would like to buy and procure, the right equipment hopefully we have still some costs reductions because prices are quite significant and they are even increasing at the moment with the vendors that we have had mind, but hopefully we can still reduce a little bit there because we will need it in this expensive programme.
And then after having the equipment delivered, it can take months to have delivery but also the data centres with the cages to be delivered we plan to build up the new infrastructure in approximately 22, during the next RIPE meeting we will be quite busy in working on the two data centres and also if the base is ready for production, we will migrate services towards that new data centre and we aim to be done at the end of next year, or 2028, let me put it correctly. It will take a significant time, this programme, to be sure that all of our services are migrated and that our current data centres are decommissioned.
But time lines are being done at the end of 2028. Yeah, and there is indeed time pressure because we have equipment that's becoming end of life and in the end it will also become end of support and different between end 27 or in 2028, so there is time pressure to migrate over to new solutions, new equipment with the core services that we run.
So yeah, what does this mean for our services. Of course it will mean that our resilience improves, I think I explained already a little bit about that in a few examples. We will be better preventing the impact of disasters if we would be faced with them.
Also be less dependent on suppliers. We will have higher security standards that we can implement with the set up we have chosen. Our maintainability will improve because we have more standarisation and more infrastructure as code and it will help us do more self service to have audibility in place in a better way just doing your work and have audit trails in place. So yeah, I think this programme will contribute to a lot of factors.
I think that is the main story line. So, we have some additional time for questions.
ROB EVANS: Thank you.
(APPLAUSE.)
I am going to stand this side, I realised I was in front of the steno screen over there. There has been quite a discussion there from some of the let's say more experienced members of the community on some of the choices and use of NFC and the relative green credentials of power generated in Germany, France and Sweden, which leads to the first remote question.
SPEAKER: One question from Shane, please consider the carbon footprint of your new deployments, for example, France is mostly nuclear for electricity and Germany uses mostly coal.
SJOERD WOLTERS: That's a good point indeed. I recently addressed in the environmental impact, that's something we should take into account, completely agree.
Maybe to add to it, in the front row, they are quite more involved in this programme for many years, or at least in the engineering areas, so part of the questions will be answered by them. Indeed wave your hands, because they have more knowledge on the topic than I.
SPEAKER: So one more question from Robert, do you consider the effective legal jurisdiction of the data centre location with regard to geopolitical risk, depending on the corporate structure, data centre, unfortunately end up following instructions from countrywide.
SJOERD WOLTERS: I missed part of the question, looking at who was talking. Can you please repeat the question?
SPEAKER: OK, I can read it again.
HANS PETTER HOLEN: Yes, we have understood that, that if a data centre is owned by a different jurisdiction, there can be instructions, so we are looking for vendors on all levels that we can live with. But as you all know, that is difficult because there is some multinationals that tend to buy the independent ones all over the world, so it is on their radar definitely.
SPEAKER: Jen Linkova. Two questions related, I am very happy to see that you explicitly mentioned IPv6, so I would like to confirm, when could we expect that Meetecho link we get in our email will be dual stack and not IPv4 only ‑‑ will be dual stack and not IPv4 only, I would just like to get some estimation.
SJOERD WOLTERS: I think we were looking at the Meetecho set up we currently use, we have plans to renew this, it must be IPv6 supported, if it's there next meeting it's the ambition, I hope to be able to do so.
JEN LINKOVA: We'll check next time. Second question, so you mention all these data centres set up and I am wondering if you go into doing it dual stack or may be considered doing like more advanced things like V6 only with stateless translation.
SJOERD WOLTERS: I don't think dual stack for us with small scale that we have, I talked about maintainability, of course it comes with an additional effort and expertise so it's not at the moment our focus area but for resilience it would be the better set up, I agree with you on that. Yes.
SPEAKER: Will. That set up looks a lot like what we did do with my former company, that's really good, I am really curious if you would be able to share some of the scale, the amount of volume of data and expected interaction between the data centre that you would have because that will definitely affect the amounts of machine and resources that you need so if we can have some of ‑‑ we can do that in the corridor or the hallway later on, I would be really curious to know about the volume of data that we are discussing here.
Speaker: I think the a fair question and I don't know if men know could say a little bit about that, I have some numbers prepared so we could go to to the corridor, I think.
SPEAKER: Yeah, so for the links between the data centres we are probably going to do 40 gig circuits, the problem is you have if you have a data centre in the Netherlands and one in France, you have to rent a circuit, the bigger it is the more it costs, so we have to balance it out. And for storage yeah I have a bunch of numbers here, I can show them to you over coffee. You know who I am, you can find me.
SPEAKER: Tom. I apologise if you have mentioned this briefly, I couldn't see it in the slides at all, have you looked at separating out the devices that you are going to use to administrate the equipment, the new environment, separating them out from the devices you use for privilege management administration versus corporate email access and various other things, have you gone down to that route of having a different environment for infrastructure management versus corporate use.
SJOERD WOLTERS: I don't think it's the main focus of this project, of course it is a debate internally what we will use in these areas, also like privilege access management is a separate project we are planning to spend a lot of time on, we defining at the moment the plans for that together with the information and security department. This is about the data centre infrastructure so it's not about for instance our mailing environment or that's not the main scope of this programme if that's what you mean.
SPEAKER: I had lumped it into the security part.
SPEAKER: It's very much a case of making sure if someone clicks on an email with a dodgy attachment it doesn't end up in the routers or your VM ware infrastructure or something similar.
ROB EVANS: Fair point, I think Tom and I are working in an environment where regulations coming and they mandate access work stations which do not have internet access and only have management access.
SPEAKER: Hello. My name is Valentine, maybe I am speaking more like a Bulgarian, but taking a look at the budget and one thing that maybe got me off is 10 millions there, 10 millions on another thing, and you are considering hyper converge compute and maybe events are 10 million and technology in 15 million, or something like that, as a technological company isn't this, disproportional and we are looking at small scale, this topologies that you shared seem like more small business customer wise or compromise on the storage, they are professional storage solutions which are far less expensive than other things really that RIPE is the most in my career it's not much 15 years but the RIPE NCC is more, the most respected organisation of mine and I was thinking that technology, we are far ahead, so this is my question, is the budget you as leader of technology, it fair for you.
SJOERD WOLTERS: Do you mean fair as being too little or too much.
SPEAKER: Maybe too little, not that I didn't like the lunch but I didn't like the hyper converged thing.
SJOERD WOLTERS: Yes so in our budget indeed the storage was one of the elements we looked at can we safe costs there or do we need more but our, sorry OK, but yeah, we have experience with a cheaper storage solution, mainly because we would like to save costs, a more expensive solution in the past and there we faced issues we are now still struggling with, we tried to kind of get the benefits of the experience that we have with the more expensive storage but choosing hopefully to not have the same price tag but it's still expensive, yeah.
SPEAKER: In this scale, it's more expensive and when you go hyper converge, it usually is, but you have two things in the budget have like five events or the five things in the budget, two of them are not changed over two years, when I seen the budget that things are not changing there, it either means that the perfect exactly the same for two years or they are not optimised, if it doesn't change for two years you could optimise that and go more for technology.
SJOERD WOLTERS: Fair point and make we can offline discuss what element you are talking about and dive into that and do more, yeah.
SPEAKER: The budget, they know, there are five, two of them not changed this year.
SJOERD WOLTERS: OK, thanks for your comments.
SPEAKER: If this is out of scope, push me back, the dane records for the email infrastructure, do you have an update on that, is there anything happening on this, they seem to be not there. Is this being worked on?
SJOERD WOLTERS: This may be a bit of a sidetrack, but I can give a short update on that; we discussed recently again that because we communicated in the mailing list that we would investigate after the summer period of time. No, we had something that took quite some attention being the presentation and infrastructure programme, but we did discuss to set the priority straight because we have a lot of things that we can improve on, this is one of them and to determine when can you expect this to be picked up.
SPEAKER: Just on the to do list
SJOERD WOLTERS: It's not forgotten. Thank you.
SPEAKER: I think it was you then Jim.
SPEAKER: Two questions actually. As an engineer I really liked the presentation and I think a lot of people here in this room they are quite enthusiastic about this, I hope. The second question that I have, it was related to this presentation in general, was it like on purpose that you omitted the word "sovereignty"? I haven't seen it a lot in the presentation, and I think it might be a very interesting topic in, well just choosing various hardware for this project.
SJOERD WOLTERS: I think it's a fair point, probably Hans Petter would like to to react on this topic.
HANS PETTER HOLEN: Managing director of the RIPE NCC. I try to avoid using the word sovereignty, because if you look up in the dictionary, it's something that requires a sovereign state, and this serving 72 states, so what do we mean by sovereignty. If it's us governing everything, sure, but then we have to build data centres and proper plans and everything, we can't do that. So we are located in the Netherlands and we rely on the good relationship with the Dutch government and they have served us well for many years. But there aren't that many hardware providers and network providers and so on, so who can we trust. Back in my day as a network engineer, we bought Cisco because we could trust Cisco, that's part of the problem now, right, you don't know who allies are any more. The thought behind this that we need to have control, that's definitely there, right, but yeah.
SPEAKER: Thanks.
JIM REID: I just wanted to say that it's good that you discussed in detail your plans about this infrastructure project and what's going on here. We have to also be reminded this is a very diverse bunch of people, probably got 100 and something people in this room at the moment who all have different perceptions and opinions how the money should be spent and how it should be configure and which data centre you use and which kit you buy and all the rest of it ignore that advice, make your best decisions you are supposed to be trusted to get on with that, we respect your opinions, you and your teams do an amazing job, there can be discussions around is this too expensive, too cheap, the way to make the points is primarily through the general meeting, that's how members accounts express opinions how money should be spent. This can be a sounding board for people to express their opinions, if you wanted to make concrete changes to these things, the general meeting is the way to go. I think you guys are doing a great job. I don't know if you are spending too much or if you are spending too little, I don't have an opinion on that, and it's good you bring these points of view here but it's the point of view of transparency, it's useful to know what's going being done, is this the best use of our money, it's something we can all make our own decisions on, if you have got concerns about that, please raise them at the general meeting, not in this working group.
(APPLAUSE.) Thanks a lot.
SPEAKER: Maybe to answer that, yeah, I am keen on having your opinion said, I understand your advice, and probably I cannot serve them all, that indeed in this case.
ROB EVANS: At the end of the day, it's you guys have to look after it as well. Is there anything else remote? I think it's really useful, really interesting, thank you very much Sjoerd, so if there are no other questions. Thank you. (APPLAUSE.)
SJOERD WOLTERS: Thanks a lot.
SPEAKER: So thanks very much for that. So fortunately we have, even though we had reserved extra time, we are now early, so I think we as the working group chairs would like to know whether you felt this session worked good, bad, indifferent, whether there were things we missed, whether there are things we spent too much time on, so feel free to email us and let us know but if that's, if there's nothing, no other comments or any other, I will say thank you for joining us and participating and we'll see you in Vilnius.
Thank you.
(APPLAUSE.)
Coffee break.
And we all need to leave the room and then reenter for the general meeting. So make sure you have got your sticker if you are coming in for the general meeting.
Coffee break.