Why Android Teams Are Moving CI Builds to Dedicated Hardware

Avatar

Editorial Note: Talk Android may contain affiliate links on some articles. If you make a purchase through these links, we will earn a commission at no extra cost to you. Learn more.

Why Android Teams Are Moving CI Builds to Dedicated Hardware 2
Image Source

Every build on a hosted CI runner is the first build. The machine is created, it does the work, and it is destroyed, which is the difference GitHub's documentation is pointing at when it says self-hosted runners “don't need to have a clean instance for every job execution.” A fresh virtual machine arrives with no Gradle daemon holding compiled state, an empty dependency cache and an empty local build cache. A project a developer rebuilds in seconds on a warm laptop can take twelve minutes in CI on identical code, and for a growing Android team that gap becomes queued pull requests and idle reviewers.

That is the case for putting a steady Android pipeline on a machine the team keeps. It has a price attached, and it gets weaker the moment the repository is public.

Start with what the runner throws away

An Android CI job runs a long sequence of stages. Gradle resolves dependencies, configures the module graph, compiles Kotlin and Java, processes resources, runs unit tests, and packages an APK or an Android App Bundle, often across several build variants in one run.

Each stage leans on a different resource. Gradle runs independent subprojects in parallel, so a modularized project can use every core it is given and then wait for the ones it is not. The Gradle daemon keeps project state in memory between runs, and a heap too small for the project converts build time into garbage collection time. Compilation and packaging write and delete thousands of intermediate files, which the storage layer has to absorb.

The standard GitHub-hosted Linux runner is a two-core machine. That is the constraint underneath most complaints about slow Android CI: the pipeline is not starved of a faster processor so much as of a second, third and fourth core, a daemon that survives to the next run, and a cache directory that was already there when the job started.

Doubling the cores doubles the invoice

Hosted CI is billed by the minute, so the sum is minutes per build, times builds per day, times working days, times the per-minute rate. Take forty Linux builds on a working day at twelve minutes each: 480 minutes a day, 10,560 across a twenty-two day month.

Two things decide what that costs. 

The first is whether the repository is private, because GitHub Actions usage is free for public repositories on standard runners, which settles the question outright for open source projects. 

The second is runner size. GitHub's pricing reference puts a standard Linux two-core runner at $0.006 a minute, and private repositories draw on an included monthly allowance of 2,000 minutes on Free and 3,000 on Pro and Team. The example workload therefore bills at $45.36 a month on a Team plan once the allowance is applied.

Then the team does the obvious thing about the two cores. A Linux four-core runner is $0.012 a minute, and the same reference states that included minutes cannot be used for larger runners at all. The identical workload now costs $126.72 a month with nothing set against it. The fix for the slow build is what makes the hosted bill real.

Run for a month, that comes to roughly six times what a cheap dedicated server costs over the same period: the KS-3 in Kimsufi's KS range lists at US$21.10 excluding VAT a month for subsequent months, with a four-core, eight-thread Xeon E3-1245 v5, 32GB of DDR4 ECC memory and two 480GB SSDs in soft RAID, and 64GB of memory for another US$8.90. That price buys hardware. The Xeon E3-1245 v5 is 2015 silicon, so the core count flatters the per-core comparison against a current hosted runner, and everything the hosted service was doing invisibly now belongs to somebody on the team.

Read the other way round, a four-core hosted runner passes the KS-3's monthly price after about 1,760 billable minutes, which is fewer than seven twelve-minute builds a working day.

Behind the runner label sits a machine somebody owns

Registering the box is the short part. A dedicated Linux machine is enrolled as a self-hosted runner from repository or organization settings, the runner application is installed and started as a service, and jobs are directed to it with labels in the workflow file. GitLab Runner follows the same pattern, with tags in place of labels and merge requests in place of pull requests.

What follows registration belongs to the team. The Android toolchain has to be pinned and maintained on the box: a fixed JDK, the SDK command line tools, the build tools and platform versions the project targets, and emulator images if instrumentation tests run there. Signing keys and API credentials have to sit somewhere the runner can read and nobody else can. 

GitHub is direct about where the boundary falls: self-hosted runners “are free to use with GitHub Actions, but you are responsible for the cost of maintaining your runner machines,” and “you are responsible for updating the operating system and all other software.”

Before any of it, benchmark. Run the same commit clean and warm on both, several times, and record the wall clock. Faster hardware does not produce a faster pipeline on its own, and a team that has never measured its own build has no idea whether the twelve minutes are compilation, download or queue.

GitHub says almost never

The honest limit here is written into GitHub's own security hardening guidance, and it is sharper than the usual note about maintenance overhead. Self-hosted runners “should almost never be used for public repositories,” because “any user can open pull requests against the repository and compromise the environment.” The warning extends to private and internal repositories, where anyone who can fork and open a pull request can reach the runner, its secrets and the GITHUB_TOKEN.

The reason is the property that makes the machine fast. Hosted runners “execute code within ephemeral and clean isolated virtual machines, meaning there is no way to persistently compromise this environment,” while self-hosted runners “do not have guarantees around running in ephemeral clean virtual machines, and can be persistently compromised by untrusted code in a workflow.” The warm daemon and the populated cache that cut the build time are the same state an attacker gets to write into.

Hosted runners hold their ground on low or spiky volume too, where the included allowance covers most of the month and a fixed box would idle through it, and for any team that would rather buy the maintenance than do it.

So the comparison a team is actually making is not $126 against $21. It is $126 against $21 plus the patching, the key handling, the disk that fills at 2 a.m. and the person who owns all three. The arithmetic is the easy half, and it is the only half that fits on an invoice.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
La Femme secrète

Why This Danish Thriller Is Taking Over Netflix—and Reminds Everyone of Harlan Coben