Controlled Linux Patch Management: From Supply Chain Risk to Fleet-Isolated Package Delivery

TL;DR: Every Linux server pulling packages directly from the public internet is a supply chain risk — a compromised upstream repo means malware on your fleet. Add AI-powered malware targeting Linux infrastructure, zero visibility into which CVEs affect your hosts, no rollback when a patch breaks something, and an auto-reboot that took production down. Rather than fix just one symptom, I designed a complete patch management system: disable public repo access, serve everything through a fleet-isolated Aptly mirror with immutable snapshots, enforce bake times via SQL gates, track every CVE in PostgreSQL, and visualize it all in Grafana. Automated via Ansible + AWX. MIT licensed, lab runs in Docker with one script.

Let me tell you about the morning that changed how I think about patching forever.

"Is the site down?"

It was 10:32 AM. I was mid-sip of coffee.

"Site not loading" "Getting 502 errors" "Is anyone else seeing this?"

I checked the server. Uptime: 5 minutes. The machine had rebooted 300 seconds ago without telling anyone.

/var/log/apt/history.log told the story:

Start-Date: 10:28:38
Commandline: /usr/bin/unattended-upgrades
Install: linux-image-6.8.0-48-generic (6.8.0-48.48, automatic)
End-Date: 10:29:12

A kernel security update auto-installed. The system auto-rebooted. Our app went down at 10:30 AM on a Tuesday — right when our users were most active.

unattended-upgrades doesn't reboot by default — unless Unattended-Upgrade::AutomaticReboot "true" is set. It was. Config drift put it there. No one noticed until the 502 errors started rolling in.

Two weeks later, it happened again. Different server, same result.

The auto-reboot was frustrating, but it wasn't the real problem. It was a symptom of a much deeper set of issues I'd been ignoring:

The unattended-upgrades reboot was just the wake-up call that made me stop tolerating all of these at once.

The philosophy: address the root causes, not just the symptom

I didn't want to just disable auto-reboots. I wanted a system that addresses every item in that table — treating package management the same way we treat code deployments:

1. Disable unattended-upgrades — nothing auto-installs, nothing auto-reboots
2. Isolate the fleet from upstream repos — servers only talk to the internal mirror, never to the public internet
3. Self-host via Aptly — mirror Ubuntu repos internally, freeze them into immutable GPG-signed snapshots, serve them locally
4. Monitor exposure — a PostgreSQL governance DB tracks every CVE per host, visualized in Grafana
5. Automate on your terms — Ansible + AWX orchestrates the full pipeline on a schedule you control

None of this is exotic. It's the same shape as patch governance at any serious org — just rebuilt as something any team can run themselves instead of buying a six-figure license for.

How it works: isolating package distribution with Aptly

Why not just a caching proxy?

An apt-cacher-ng or squid-deb-proxy caches packages but still proxies requests upstream. That means:

• Your servers are still reachable to the internet (even if just outbound)
• A compromised upstream repo can still serve you malicious packages
• You have no version pinning — whatever the upstream serves today is what your fleet gets

I wanted immutability: a point-in-time snapshot that's signed, frozen, and cannot change.

Enter Aptly

Aptly is a Debian repository management tool. It mirrors upstream repos, creates timestamped snapshots, and publishes them as local HTTP repositories. Think of it as git tag for APT packages.

Upstream repos (Ubuntu, Docker, NodeSource, Nginx)
    │
    ▼
aptly mirror create → daily sync → snapshot → publish
                                            │
                                            ▼
                                    Clients fetch from
                                    internal nginx endpoint

The mirror setup

Creating a mirror of Ubuntu's security repo, syncing it, and freezing it into a snapshot:

# Create a mirror of Ubuntu's jammy-security repo
aptly mirror create \
  -architectures="amd64" \
  jammy-security \
  http://security.ubuntu.com/ubuntu \
  jammy-security main restricted universe multiverse
 
# Sync (takes a while the first time)
aptly mirror update jammy-security
 
# Freeze into a timestamped snapshot
SNAPSHOT="patch-$(date +%Y%m%d-%H%M)"
aptly snapshot create "$SNAPSHOT" from mirror jammy-security
 
# Publish as a local repo
aptly publish switch \
  -component=main,restricted,universe,multiverse \
  jammy \
  "$SNAPSHOT"

The key insight: snapshots are immutable. Once created, they never change. If you need to update packages, you create a new snapshot. The old one still exists for rollback:

# Rollback is just pointing the publish at an old snapshot
aptly publish switch jammy "patch-20260724-0100"

Serving it internally

Aptly ships with its own HTTP server for development, but for production it runs behind nginx:

server {
    listen 80;
    root /opt/aptly/public;
 
    location / {
        autoindex on;
        # Only internal network can reach this
        allow 10.0.0.0/8;
        deny all;
    }
}

Clients are configured to point only at this internal endpoint:

# /etc/apt/sources.list.d/internal.list
deb [signed-by=/etc/apt/trusted.gpg.d/internal.gpg] http://aptly.internal/repos jammy main restricted universe multiverse

There is no http://archive.ubuntu.com in their sources. They literally cannot reach upstream.

The snapshot promotion pipeline

A single snapshot isn't enough — you need a pipeline. One snapshot is created per day and promoted through environments:

Daily:  mirror sync → snapshot → dev publish
24h+:   promote to staging (SQL gate: "has it been in dev for 24h?")
48h+:   promote to prod    (SQL gate: "has it been in staging for 48h?")

Each promotion is an Ansible playbook that:

1. Checks the governance DB for bake time compliance
2. Switches the Aptly publish to the target snapshot
3. Runs apt update on the target fleet
4. Records the promotion in patch_history

Bake-time gates enforced in SQL

The bake-time gates are the secret sauce. A patch that looks fine in dev can still break staging under real traffic. The SQL gate means it physically cannot reach prod until it's sat somewhere long enough to prove itself — or fail.

The gate check as SQL:

-- This is not a suggestion. This is a wall.
SELECT CASE
  WHEN EXISTS (
    SELECT 1 FROM patch_history
    WHERE environment = 'staging'
      AND applied_at <= NOW() - INTERVAL '48 hours'
  ) THEN 'promote_ok'
  ELSE 'bake_time_not_met'
END;

And the Ansible enforcement that calls it:

- name: "Enforce staging bake time (48h)"
  ansible.builtin.fail:
    msg: >
      Bake time not met. Current snapshot has only been in staging
      for {{ staging_hours }} hours. Minimum required: 48 hours.
  when: staging_hours < 48

Where staging_hours is queried from PostgreSQL:

SELECT COALESCE(
  EXTRACT(EPOCH FROM NOW() - MAX(applied_at)) / 3600,
  0
) FROM patch_history
WHERE environment = 'staging'
  AND snapshot = current_snapshot();

No one can "accidentally" skip the gate. The playbook fails with a clear message. No one has to "remember" to wait — the automation literally refuses.

Rollback that actually works

Here's what most "rollback" solutions do: they re-point the repo to an old snapshot. That only prevents new hosts from getting bad packages. Already-patched machines stay broken.

PatchOps does a two-phase rollback:

1. Repo switch — point the Aptly publish back to the last-known-good snapshot
2. Client downgrade — apt install --allow-downgrade actually reverts the packages

Full audit trail in PostgreSQL. Every promotion and rollback is timestamped and recorded.

The stack

┌──────────────────────────────────────────────────────────┐
│                    PatchOps                              │
├──────────────────────────────────────────────────────────┤
│  Aptly       → Package mirroring + snapshot              │
│  Ansible     → Automation + playbooks                    │
│  AWX         → Orchestration (14 templates, 7 schedules) │
│  PostgreSQL  → Governance database (patchgov)            │
│  Grafana     → 3 dashboards (fleet, CVEs, history)       │
│  Docker      → Lab environment + AWX hosting             │
└──────────────────────────────────────────────────────────┘

14 job templates in AWX. 7 cron schedules. SQL-enforced gates. Email notifications for every scan and patch run — success or failure, with real error messages.

The entire AWX configuration is declared in a playbook. Destroy it and recreate it with one command. For the full details on running AWX on Docker Compose — including the custom image build, bootstrap scripts, and AWX-as-code approach — see the companion post: Running AWX on Docker Compose: A Stand-Alone Alternative to Kubernetes.

The governance database

Without data, you're guessing. The governance database (patchgov in PostgreSQL) answers three questions:

1. What packages need updating? — per host, per package, with CVE severity
2. Has it baked long enough? — promotion gate enforcement
3. What changed? — audit trail of every patch operation

The schema

Four tables, one view:

-- What we found during the last scan
CREATE TABLE scan_results (
    id SERIAL PRIMARY KEY,
    host VARCHAR(255) NOT NULL,
    scan_date TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    security_packages INT DEFAULT 0,
    regular_packages INT DEFAULT 0,
    reboot_required BOOLEAN DEFAULT FALSE,
    risk_score DECIMAL(3,1) DEFAULT 0.0,
    risk_tier VARCHAR(20) DEFAULT 'Low'
);
 
-- Individual package details per scan
CREATE TABLE scan_package_details (
    id SERIAL PRIMARY KEY,
    scan_id INT REFERENCES scan_results(id),
    package_name VARCHAR(255) NOT NULL,
    installed_version VARCHAR(100),
    candidate_version VARCHAR(100),
    is_security BOOLEAN DEFAULT FALSE,
    cve_ids TEXT[],
    cvss_scores DECIMAL(3,1)[]
);
 
-- Audit trail of every patch operation
CREATE TABLE patch_history (
    id SERIAL PRIMARY KEY,
    environment VARCHAR(50) NOT NULL,
    operation VARCHAR(50) NOT NULL,  -- 'promote', 'rollback', 'patch'
    snapshot VARCHAR(100),
    applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    success BOOLEAN NOT NULL,
    error_message TEXT
);
 
-- Snapshot lifecycle tracking
CREATE TABLE snapshots (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL UNIQUE,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    current_environment VARCHAR(50) DEFAULT 'dev',
    promoted_at TIMESTAMPTZ
);
 
-- Which hosts need a reboot
CREATE VIEW reboot_debt AS
SELECT host, scan_date, risk_score
FROM scan_results
WHERE reboot_required = TRUE
  AND scan_date > NOW() - INTERVAL '24 hours';

Risk scoring

Each host gets a score from 0–10:

risk_score = MIN(
  (security_packages × 3 + regular_packages × 1)
  + CASE WHEN reboot_required THEN 15 ELSE 0 END,
  10
)

Mapped to tiers:

CVE enrichment pipeline

The security scan doesn't just count packages — it extracts the actual CVEs from Debian changelogs:

apt changelog <package>          # Fetch changelog
    → extract_cves.py            # Parse CVE IDs from text
        → Ubuntu Security Tracker API  # Get CVSS scores
            → enrich_package_rows.py   # Build DB insert rows

This is the part most teams find most valuable. When you look at a host in the dashboard, you don't just see "5 security updates pending" — you see the actual CVEs with their CVSS scores:

No more wondering whether a critical CVE affects your fleet. The data is there.

What the Grafana dashboards show

The monitoring stack consists of three auto-provisioned dashboards, each serving a different audience:

1. Fleet & Snapshots Dashboard — for the operations team. Shows every host's risk tier (Low/Medium/High/Critical), which hosts need reboots, and the current snapshot position in the promotion pipeline. If a snapshot is stuck in dev, it's visible immediately.

2. Package Upgrades Dashboard — for the security team. Lists every pending package across the fleet with CVE IDs, CVSS scores, and drill-down per host. Filter by CVE severity, package name, or host group.

3. Patch History Dashboard — for compliance and audits. A timeline of every promotion, patch, and rollback operation with success/failure status. Mean time between patches, rollback frequency, and environment-level breakdowns.

Grafana connects to PostgreSQL via a provisioned datasource:

# grafana/provisioning/datasources/patchgov.yml
apiVersion: 1
datasources:
  - name: patchgov
    type: postgres
    url: govdb:5432
    database: patchgov
    user: grafana
    secureJsonData:
      password: ${GRAFANA_DB_PASSWORD}

Every dashboard is JSON, committed to Git, and deployed as code. No click-ops dashboard creation.

Email notifications for every run

Every automated operation — scan, patch, promote, rollback — sends an email via SMTP. Success or failure, you get notified. A scan report looks like:

Subject: [PatchOps] Fleet Scan Complete — 3 Critical hosts
Total hosts scanned: 12
Hosts needing reboot: 2
Security packages pending: 47
Highest risk host: web-03 (8.3 - Critical)
CVEs detected: 12

No silence means no surprises. If the email doesn't arrive, something's wrong with the automation itself.

Putting it all together

The full data flow on a typical week:

[Daily 01:00]  Mirror sync → Aptly snapshot → dev publish
[Daily 02:00]  Patch scan → PostgreSQL → CVE enrichment → Grafana
[Daily 02:30]  Security patches applied to dev
[Weekly Mon]   Promote to staging (24h gate check)
[Weekly Mon]   Patches applied to staging
[Weekly Tue]   Promote to prod (48h gate check)
[Weekly Tue]   Patches applied to prod (serial=25%, max_fail=25%)

All orchestrated by AWX. All tracked in PostgreSQL. All visible in Grafana.

Lessons for your own setup

If you're building something similar, here's what I'd do differently knowing what I know now:

1. Start with the governance DB first — the schema is cheap to change early and expensive to change later
2. SQL gates aren't optional — if bake times aren't enforced in code, they'll be skipped under pressure
3. AWX on Docker Compose works, but has limits — for more than ~50 hosts, the official K8s deployment is worth fighting for
4. CVE enrichment is slow — parsing changelogs across 50 packages × 12 hosts takes time. Cache aggressively
5. Email notifications get ignored — I'm adding Slack/webhook integration next

What I learned

Try it yourself

The entire lab environment — 5 Ubuntu clients, an Aptly server, the governance database, Grafana, and the full AWX stack — comes up in Docker with one script:

git clone https://github.com/2SSK/patchops
cd patchops
./scripts/start.sh

MIT licensed. Build it, break it, make it yours.

For the full deep-dive on running AWX on Docker Compose — custom image build, launch scripts, AWX-as-code configuration, and credential injection — see the companion post: Running AWX on Docker Compose: A Stand-Alone Alternative to Kubernetes.

I write about infrastructure automation, PostgreSQL, and the systems that keep production running. If this resonates — how do you handle supply chain risk for your Linux fleet? Do you trust public repos directly? Use a mirror? Something else entirely? I'd love to hear.