CERN moves thousands of accelerator control computers to Debian

CERN is moving several thousand computers that control its particle accelerator hardware from Red Hat-family Linux distros to Debian. CERN – the place that runs the Large Hadron Collider – generates a lot of data, and that means it uses a lot of computers. Most of them run Linux, and for over 20 years, it used distros in the Red Hat family. Now, that's beginning to change: it’s moving what it calls the “front end” computers, the ones underground that control the LHC, over to Debian Linux. At the end of last month, the MiniDebConf Winterthur 2026 was held in Switzerland, and two CERN engineers, Federico Vaga and Nikos Tsipinakis, gave a talk about Controlling CERN’s Accelerators with Debian. In a dense 40 minutes, they talked about the size and complexity of CERN’s computer systems and the many problems with moving some of them from the Red Hat family over to Debian. The European Organization for Nuclear Research - whose CERN acronym comes from the French name of its predecessor, Conseil Européen pour la Recherche Nucléaire - is one of the world's most high-tech research organizations. It's where the World Wide Web was invented, and CERN has also long been a major user of Red Hat-family Linux. For many years, CERN and Fermilab – its equivalent in the USA – jointly maintained their own RHELative, called Scientific Linux. This started in 2004, and its first mention on The Register was in 2007. For some years, Scientific Linux was one of the leading free RHELatives… until 2014, when Red Hat brought CentOS in-house. CERN’s job is doing physics, not maintaining a Linux distro, so once CentOS became official, CERN switched to it – starting with its own build of CentOS 7. You can probably already guess where this story is going. With hindsight, that wasn’t a great move. There was a lot of detail in Vaga and Tsipinakis’s talk, and we only have space for some of the highlights. Alongside some paid copies of RHEL 7 and RHEL 8, CERN had a lot of CentOS Linux 7 installed. It was planning a migration to CentOS 8, but in 2020, Red Hat cut eight years off CentOS 8’s support lifespan. Version 8 was meant to go out of support at the end of 2029, but Red Hat cut it to the end of 2021. With so much hardware and such very expensive machinery to turn on and off, CERN has to plan as much as 15 years ahead. Obviously, it evaluated CentOS Stream 9 as a candidate, with a view to moving to CentOS Stream 10 as soon as possible. But there was a big problem. RHEL 9 went into beta in late 2021, and as we mentioned in 2022, RHEL 9, and thus CentOS Stream 9, has a minimum CPU requirement of x86-64-v2. This would eliminate 47 percent of CERN’s embedded control computers. In 2025, RHEL 10 moved that to x86-64-v3, which would eliminate another 17 percent of the survivors. That means that tracking future Red Hat releases would mean replacing some 65 percent of what CERN calls its “front end” computers – the ones that live underground, connected to the accelerators. Some of these are specialized hardware: some are VMEbus single-board computers, others use purpose designed and built CERN monitoring hardware, much of which is on parallel PCI cards. The CERN engineers estimated that to replace all these, they would need to redesign 11 custom boards, which would mean hiring two electrical engineers, two software engineers, and two technicians. Also, installing new racks and new cabling in some places would mean digging new tunnels, 100 metres (330 feet) underground. They estimated the project would cost at least CHF 5.4 million ($6.7M or £5M), and it only had a 20 percent chance of success. (All the custom hardware designs are online and openly available – they just aren’t much use to anyone else.) It started evaluating other distros, placing them along two axes. One is control and maintenance: how much control each distro gives and how much maintenance it takes. The other is the level of integration with CERN’s platform, which strongly favors the Red Hat family – so, RHEL, CentOS Stream, Rocky Linux, and AlmaLinux. It looks to us like CERN did some in-depth evaluation, including some less-common distros such as the Civil Infrastructure Platform. Debian and Ubuntu both scored well as relatively low-maintenance options, but their integration with CERN systems would be poor. The answer was, as Vaga put it, “a software solution for a software problem.” If their kit couldn’t run more modern RHELatives, then they would not use RHELatives. They would switch the accelerator computers to Debian instead. He pointed out that this is just one of their three main tiers of computers. The other two are the “consoles”, desktops and laptops which the physicists use to set up their experiments. These currently run AlmaLinux 10 with the Xfce desktop, on an isolated network which is not connected to the Internet. We reported on that choice back in 2022. The second tier systems are called “datacenter” machines, which log, validate, and analyse the commands from the consoles before they’re transmitted, and are also used for modeling and other experiments. These formerly ran CentOS Linux 7, currently run RHEL 9 and will probably move to AlmaLinux 10 in future. It’s the third tier of machines that are currently being moved to Debian. The second part of the talk went into some technical detail about how. The custom hardware needs kernel drivers, which must be built for each specific build and version of the kernel – and CERN builds its own kernels, its own initramfs images, and configures its own bootloader. All these computers are network-booted, and mount their root filesystems over NFS. These images also need to be versioned, but this does make updating them easier. Deploying kernel drivers is much harder than user-space applications. They can’t be built locally on the machine the usual way with DKMS – it’s much too demanding for embedded systems. A key problem is that Debian repositories do not handle multiple different versions of packages well, so the team had to split files between separately versioned repositories instead. Packaging these things is hard, too. Tools such as Debhelper and Debusine, or openSUSE’s OBS, don’t help. In the end, support for .deb packages was “hammered into” the Koji build service CERN already used, and it used ELBE, which “turns the build servers into Debian environments.” For now, the other systems are staying in the greater Red Hat family. At the end, one question asked if AlmaLinux’s greater platform support (including for x86-64-v2 and x86-64-v3) would have helped, but at the time that CERN had to make tough choices, the AlmaLinux project wasn’t able to commit to supporting this. It doesn’t only mean building binaries, but also a lot of testing and validation work. It’s easier to go with an OS that natively supports these things, and both speakers were full of praise for Debian’s self-description of “the universal operating system.” This is an interesting study in distro choice. With Red Hat’s sudden changes of plan and direction in recent years, we’re sure that many other organizations and companies have faced similar problems. The difference here is that CERN is a publicly funded, non-profit hi-tech institution, meaning that it is both willing and able to talk about this in public. We wouldn’t be surprised if CERN chose to move more systems in the direction of Debian in future, but it may not be soon: the current project is already looking towards 2033. For a timeline like this, CERN bought support and training from Freexian and will be using the Freexian-backed Debian ELTS versions of Debian 13, and it’s starting to plan for Debian 15. We cannot help but think that Red Hat made some very bad decisions in almost every aspect of its recent product transitions, although as we have said before, the real original mistake here was adopting CentOS and making it official in 2014. IBM’s subsidiary has alienated a lot of non-paying users, but just like CERN, many users of its free distros were also using the paid ones – however, we are confident this is just one of many such large-scale migrations. It’s just unusually public. ®