Can We Go All-In on ARM?
At mx51, Linux is the backbone of our software delivery and runtime platforms. It is reliable and fast, and comes with the support of an enourmous global community of users and contributors.
When AWS, our primary provider of cloud-based Linux systems, announced the availability of their new low cost ARM-based Graviton processors in late 2018, the world was quick to adopt the new chips, which offered higher performance at lower prices compared to traditional x86 based machines. AWS has since rolled out a number of updates, including the release of their second- and third-generation processors: Graviton2 and Graviton3. As more and more AWS products announced support for ARM-based instances, including RDS, Aurora, Lambda, EKS, Fargate and others, we knew we needed to explore how we could leverage the benefits of ARM-powered machines at mx51.
Initially, the goal was to work out how to utilise Amazon’s Graviton instances in parts of our continuous delivery platform. Long term, we thought that going all-in on ARM-based processors across all our Linux systems, from build automation agents to Kubernetes cluster nodes, could help us significantly reduce our infrastructure costs without sacrificing performance. So we decided to experiment.
Before we get to ARM-based servers, we need to have ARM-based containers. Along with Linux, containers are a core technology at mx51, powering our customer-facing runtime platforms, as well as performing functions in every step of our software delivery pipeline. We have containers for compiling, testing, analysing, scanning, packaging and deploying every piece of software produced by our engineering teams. And at the time we started this project, every single container we created was built to run on the x86 architecture.
The question of whether we could move to ARM-based servers came down to Docker and its extensions for building multi-platform images.
The goal of the experiment was simple: can we cross-compile all our Docker images? Specifically, can we compile both x86 and ARM images simultaneously and independently of the underlying host architecture?
This approach avoids any sort of sudden cut over from one architecture to another. It also avoids the cost and complexity of provisioning and maintaining parallel delivery pipelines, one for each architecture.
With the flexibility of cross-compilation, we can plan a careful migration of Linux systems, one function at a time.
Bonus goal: can we write our automation software to be compatible on either architecture? One code base to work on both. This, too, would be needed for a smooth migration from x86 to ARM.
The experiment got off to a rough start in early 2021, a sign that cross-platform tools on both architectures were in their early stages in terms of their support for the Docker toolset.
Our first roadblock came in the form of intermittent segfault errors when compiling Go applications within x86 Alpine-based images on ARM machines.
At the time, I used Go’s example GitHub project to reproduce the issue, documenting my observations as I went.
While the issue hinted that there might be something wrong with the
qemu-x86_64 emulator used by the Docker BuildKit to compile x86 images, I was unable to find, at the time, a solid clue as to the cause.
In addition, when the emulator did work, it would often do so very slowly. We were looking at 1 minute compile-times jumping to 10 minutes or more. And this was not limited to cross-compilation on ARM-based machines. Cross-compilation on x86 machines showed similar performance issues.
In mid 2021, we decided to put the project on hold, knowing that the open-source community was hard at work at enhancing the toolset, and that we were sure to see improvements in the weeks and months to come.
Stability and Improvements
Returning to this experiment in early 2022, we saw that a number of the components used in the Docker toolset had received updates. The segfault issue had disappeared, and cross-compilation build times had been reduced (although still well above the 1-minute compile times we were used to).
As of April 2022, the Docker toolset is showing signs it might be able to do what we need. And early tests on a small subset of our projects have shown that images built using the emulators are indistinguishable from their natively-built counterparts, in terms of their reliability and performance at runtime.
The next phase of this experiment hopes to answer some of the following questions:
- What techniques can we use to improve the cross-compile build times?
- How do cross-compiled containers perform under load compared to their natively-compiled counterparts?
- Are there any containers that we manage that simply can’t be built and/or run on ARM?
- What level of ARM support is out there for all our third-party stacks and toolsets?
In the next post, I will do a deeper dive on the methods we use to build and test our cross-compiled images. I will compare the architectures in how well they perform this function, and discuss some of the outstanding issues we still face.