niks3

S3-backed Nix binary cache with garbage collection

The idea is to have all reads be handled by the s3 cache (which itself can be high-available) and have a gc server that tracks all uploads to the cache and runs periodic garbage collection on s3 cache. Since writes to a binary cache are often not as critical as reads, we can vastly simplify the operational complexity of the GC server, i.e. only running one instance next to the CI infrastructure.

Architecture

flowchart LR
    subgraph Clients
        niks3cli[niks3 CLI]
        nix[Nix Client]
    end

    subgraph Infrastructure
        s3[(S3 Bucket<br/>NAR files, narinfo,<br/>logs, realisations)]
        niks3[niks3 Server]
        db[(PostgreSQL<br/>closure tracking)]
    end

    niks3cli -->|1. request upload| niks3
    niks3 -->|2. signed S3 URLs| niks3cli
    niks3cli -->|3. upload NAR/narinfo| s3
    niks3 -->|track references| db
    nix -->|4. read NAR/narinfo| s3

Write path: The niks3 CLI requests an upload from the server, which returns pre-signed S3 URLs. The client uploads NAR files and narinfo directly to S3. The server tracks references in PostgreSQL for garbage collection.

Read path: Nix clients read directly from S3 (or a CDN in front of it) without going through niks3. This allows the read path to scale independently and remain highly available.

Features

Binary Cache Protocol Support

niks3 implements the Nix binary cache specification with the following features:

• NAR files (nar/): Compressed with zstd, stored in S3
• Narinfo files (.narinfo): Metadata with cryptographic signatures
  • StorePath, URL, Compression, NarHash, NarSize
  • FileHash, FileSize (for compressed NAR)
  • References, Deriver
  • Signatures (Sig fields)
  • CA field for content-addressed derivations
• Build logs (log/): Compressed build output storage
• Realisation files (realisations/*.doi): For content-addressed derivations
• Cache info (nix-cache-info): Automatic generation with WantMassQuery, Priority

Advanced Features

• Cryptographic signing: NAR signatures using Ed25519 keys (compatible with nix key generate-secret)
• Content-addressed derivations: Full CA support with realisation info
• Multipart uploads: Efficient handling of large NARs (>100MB)
• Transactional uploads: Atomic closure uploads with rollback on failure
• Garbage collection: Reference-tracking GC with configurable retention
• Parallel uploads: Client parallelizes NAR and metadata uploads

Operational Features

• Authentication via API tokens (Bearer auth)
• OIDC authentication for CI/CD systems (GitHub Actions, GitLab CI)

Choosing an S3 Provider

niks3 works with any S3-compatible storage provider. We recommend Cloudflare R2 for most users due to zero egress fees and excellent performance.

For detailed pricing comparison and alternative providers, see the S3 Provider Comparison wiki page.

Setup

For complete setup instructions, see the Setup Guide in the wiki.

OIDC Authentication (CI/CD)

niks3 supports OIDC authentication for CI/CD systems. See the wiki for details:

• OIDC Configuration
• GitHub Actions
• GitLab CI

Development

For development setup, database migrations, benchmarks, and contribution guidelines, see CONTRIBUTING.md.

Real-World Deployments

• Clan infra: Configuration | Instance
• Numtide: Instance
• TUM-DSE: Configuration | Instance

Need commercial support or customization?

For commercial support, please contact Mic92 at [email protected] or reach out to Numtide.