Deploying

GitHub Actions runners

stackd can run your GitHub Actions jobs on your own servers. Settings → Runners keeps self-hosted runners there, each used for one job in a fresh container and then replaced. Jobs on them cost no GitHub Actions minutes.

Quick start: run a project's CI on stackd

  1. Connect a GitHub account in Settings → Builds, if none is. It registers the runners, so it needs admin on the repositories (or, for an organization, its runners); see Permissions.

  2. Make a runner pool in Settings → Runners → Add runner pool (admins only):

    • Serves: your repositories (pick them; up to 20), or an organization (all its repositories).
    • Servers to run on, and Runners per server: three servers with one each run three jobs at once. Keep them off the server that runs the panel if you can.
    • CPUs each and Memory each: the most a job may use. A build takes every core it gets; leave room for what else runs on the server (on a 4 CPU, 8 GB server, 2 CPUs and 4 GB is a fair start).
    • Docker in jobs if the project's workflows use services:, container:, Docker actions or docker build. Then click Install Sysbox on each server the pool shows without it (Docker in jobs; it doesn't restart anything).
  3. Point the workflow at it. In each job of .github/workflows/*.yml, replace runs-on: ubuntu-latest (or ubuntu-24.04) with:

    runs-on: [self-hosted, stackd]

    With Docker in jobs, ask for it by label where a job needs it: runs-on: [self-hosted, stackd, docker].

  4. Push. The pool's rows show each runner: Free (waiting for a job to start one, for a pool of several repositories), Starting, Waiting for a job, Running a job. GitHub lists them under the repository's Settings → Actions → Runners while they exist.

Adding another project

  • A pool of repositories: Edit the pool, add the repository under Serves, save. Its runners start when its jobs wait; the pool's slots are shared by all its repositories.
  • An organization's pool: nothing to do for a repository in it, unless the organization's default runner group leaves it out (GitHub: the organization's Settings → Actions → Runner groups; public repositories are left out by default).

Then change the project's runs-on as in step 3.

Labels

Every runner has self-hosted, Linux, X64 or ARM64 (its server's CPU), stackd, docker with Docker in jobs, and the pool's extra labels. A job runs on a runner that has every label its runs-on lists:

runs-on Runs on
[self-hosted, stackd] any of the pool's runners
[self-hosted, stackd, docker] runners with Docker in jobs
[self-hosted, stackd, ARM64] runners on ARM servers only
[self-hosted, stackd, big] a pool with the extra label big
ubuntu-latest GitHub's own runners, never these

Two pools with different sizes or servers can serve the same repository: give one an extra label (big) and ask for it where a job needs it.

Examples

A Node project:

jobs:
  test:
    runs-on: [self-hosted, stackd]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npm test

Tests against a database, and an image built and pushed (Docker in jobs):

jobs:
  test:
    runs-on: [self-hosted, stackd, docker]
    services:
      postgres:
        image: postgres:17
        env: { POSTGRES_PASSWORD: test }
        ports: ["5432:5432"]
        options: --health-cmd pg_isready --health-interval 2s --health-retries 30
    steps:
      - uses: actions/checkout@v5
      - run: ./run-tests.sh   # reaches the database at localhost:5432
        env: { DATABASE_URL: "postgres://postgres:test@localhost:5432/postgres" }

  image:
    runs-on: [self-hosted, stackd, docker]
    permissions: { contents: read, packages: write }
    steps:
      - uses: actions/checkout@v5
      - uses: docker/login-action@v3
        with: { registry: ghcr.io, username: "${{ github.actor }}", password: "${{ secrets.GITHUB_TOKEN }}" }
      - uses: docker/build-push-action@v6
        with: { push: true, tags: "ghcr.io/${{ github.repository }}:latest" }

A Rust project: install the toolchain (nothing is preinstalled but a C toolchain) and keep the compiled dependencies in GitHub's cache, which the runners don't keep between jobs:

jobs:
  test:
    runs-on: [self-hosted, stackd]
    steps:
      - uses: actions/checkout@v5
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2
      - run: cargo test --locked

To switch a project between these runners and GitHub's without editing every job, read runs-on from a repository variable (GitHub: Settings → Secrets and variables → Actions → Variables):

    # RUNS_ON = ["self-hosted","stackd"]; unset: GitHub's runners
    runs-on: ${{ fromJSON(vars.RUNS_ON || '"ubuntu-24.04"') }}

How they differ from GitHub's runners

GitHub's ubuntu-24.04 stackd's runners
Cost Actions minutes none (your servers)
CPU x86_64 (or arm64 runners) your servers' (X64 or ARM64)
Size 2–4 CPUs, 7–16 GB the pool's limits
Jobs at once many the pool's slots; others wait
Software hundreds of tools preinstalled Ubuntu 24.04 with git, curl, jq, a C toolchain, Python 3 and venv, zstd, xz, Docker's CLI; add the rest with setup-* actions or sudo apt-get install
Docker always with Docker in jobs
Network the internet the internet only: not your stackd apps, databases or private network
Between jobs fresh machine fresh container (caches through actions/cache)
Start seconds seconds; with several repositories up to ~15 s more

Things that follow from these:

  • Architecture. On ARM servers, a job building or downloading x86_64 binaries needs them for arm64, or cross-compiling. Most setup-* actions pick the right one by themselves.
  • No private network. A job can't reach your apps or databases by their private names or the mesh; deploy or test against public addresses, or have the job call stackd's API (stackd CLI with an API token as a secret).
  • Nothing persists. Each job's disk is new: no build cache, no Docker image cache, no files from the last job.

Troubleshooting

A job waits with "Waiting for a runner to pick up this job".

  • Its runs-on labels must all be on one runner: ubuntu-latest never matches, docker needs Docker in jobs, X64 needs an x86_64 server.
  • The repository must be in the pool (or the organization's runner group).
  • Look at the pool in Settings → Runners: Paused, Server offline, Needs Sysbox or Retrying with its error. All slots busy with other jobs means it waits its turn.

A slot shows Retrying. Its error says why the last runner failed; it's tried again after 30 seconds, doubling up to 10 minutes. The pool's Last error shows problems with GitHub itself: "github answered 403" or "404" means the git account lost admin on the repository (or the organization's runners), or its token expired: connect it again in Settings → Builds.

cc: not found, ensurepip is not available. The server built its runner image before stackd 0.13.48: update stackd (Settings → System).

A step is killed (exit code 137) or the job ends with "lost communication with the server". It ran out of the pool's memory: raise Memory each, or use fewer parallel jobs in the build (CARGO_BUILD_JOBS, make -j2).

docker: command not found or "Cannot connect to the Docker daemon". Turn on Docker in jobs, and ask for docker in runs-on.

A connection to a private address times out. By design: jobs reach the internet only (see What a job gets).

What a job gets

  • A fresh, single-use runner. Each runner takes one job, then is replaced by a new container. Nothing a job leaves on disk reaches the next one, so there's no local cache: use actions/cache or Swatinem/rust-cache as on GitHub's runners.
  • The image: stackd/actions-runner:2 by default (a pool's Image left empty): GitHub's runner image, ghcr.io/actions/actions-runner:latest (Ubuntu 24.04 with git, curl, jq, sudo and the Docker CLI, as the user runner), plus a C toolchain, pkg-config, Python's venv, zstd and xz; and for Docker in jobs, dockerd's iptables and Compose. It's rebuilt every week with a current runner. Jobs that need more install it (sudo apt-get install -y …), or the pool names an image built FROM GitHub's with it installed; such an image must keep run.sh in its working directory.
  • No Docker daemon, unless the pool has Docker in jobs (below). The server's Docker socket is never given to jobs.
  • Limits: CPUs and memory per runner, from the pool. 0 is no limit, and a build uses every core it gets, so limits keep room for whatever else the server runs.
  • Network: the internet and DNS only, whatever the host firewall says. A job can't reach the server itself (any of its addresses, published ports included), your stackd apps or databases, private, shared or link-local addresses (a cloud's metadata service), nor another runner; nothing can connect to it. A server without nftables starts no runner.
  • Time limits. A runner that doesn't connect to GitHub within 10 minutes, or that exits with an error, is replaced and its slot retried after 30 seconds, doubling up to 10 minutes. A job still running after 12 hours is stopped.
  • Changing a pool. Pausing it, removing a server from it or lowering the runners per server removes the idle runners at once; a runner in a job finishes it first. Deleting a pool does the same for all of its runners. Runners are named stackd-<pool>-<server>-<slot>-<random> on GitHub.

Docker in jobs

With Docker in jobs on, each job gets a Docker daemon of its own, as on GitHub's runners: container:, services:, Docker container actions, docker build, docker run and Compose all work. Its runners also have the label docker, for runs-on: [self-hosted, stackd, docker].

  • Sysbox runs those runners: a container runtime whose containers can run Docker unprivileged, in a user namespace. The job's Docker can't see the server's Docker or its containers, and what it starts shares the runner's network, so it reaches the internet only, like the runner.
  • Installing it: the pool shows each server's Sysbox, with Install Sysbox (POST /api/servers/{id}/sysbox). It works on Ubuntu or Debian, x86_64 or arm64, and needs kernel 5.15 on Ubuntu 22.04, 6.8 on 24.04, 5.5 elsewhere. Running containers aren't stopped: stackd pins Docker's current addresses in /etc/docker/daemon.json first (keeping the old file as daemon.json.stackd-before-sysbox), so Docker only reloads. A dockerd started with --bip or --default-address-pool isn't touched: move those to daemon.json first.
  • AppArmor: on recent Ubuntu, stackd allows the FUSE mount Sysbox needs in /etc/apparmor.d/local/fusermount3. If a runner never starts with "fusermount3: mount failed: Permission denied", that file was changed or removed.
  • The image: the default one has dockerd, iptables and Compose. A pool's own image must have dockerd and iptables.
  • No cache between jobs: each job's Docker starts empty, so it pulls its images every time, as on GitHub's runners.
  • A pool with Docker in jobs runs only on its servers with Sysbox ready; the others' slots show Needs Sysbox.

Who can use it, and what it risks

Only admins see or change runner pools. The git account must be able to manage the runners, which making a pool checks first:

  • a repository: admin on it. A classic token or OAuth connection with repo, or a fine-grained token with Administration: read and write.
  • an organization: a classic token or OAuth connection with manage_runners:org (accounts connected through OAuth before v0.13.44 have to be connected again to grant it), or a fine-grained token with the organization's Self-hosted runners: read and write.

Finding waiting jobs uses the git account's GitHub API rate limit: two requests a repository every 15 seconds, plus one a workflow run in progress. Unchanged answers don't count, so quiet repositories cost nothing.

A job runs the repository's code, and its dependencies' build scripts, on the server. Use pools for your own private repositories, where only people you trust push. GitHub advises against self-hosted runners on public repositories, where a fork's pull request could run code on them. Keep them off the server that runs the panel and off servers with data you can't risk.