Charlie/project-charlie

Fork 0

Files

T

amacocian 47af17df37 Switch deploy-stack to SSH-into-host model; add charlie-ssh composite actions

2026-05-05 16:18:41 +02:00

32 KiB

Raw Blame History

Host onboarding

End-to-end recipe for bringing a new host into the fleet. Migrated from the wiki so changes go through PRs.

Note on step-ca migration. Charlie/cert-issuer is deployed; its SSH user CA pubkey now lives at /mnt/nas/ca/ssh/ssh_user_ca.pub and is distributed by sync-ca.sh exactly as before. The X.509 root at /mnt/nas/ca/ca.crt (LDAP TLS) is still the original — not yet migrated. The pending rewrite of sync-ca.sh to fetch directly from step-ca's HTTPS endpoint is captured in Step-ca migration impact. Until then, this section stays accurate.

Admin age key (one-time)

Before onboarding any host, generate the admin age key on the admin laptop. This key is listed as a recipient on every .sops.yaml in Charlie/* so secrets stay decryptable / re-encryptable from off-host. Loss = re-key every repo, so keep a backup. Design: secrets.md → Admin key custody.

# On the admin laptop (one-time per fleet, NOT per host).
#   Arch:   sudo pacman -S age sops
#   macOS:  brew install age sops
#   Ubuntu/Debian:  apt has age but NOT sops. Install age via apt and sops
#                   from the upstream .deb (see Host age key section below for
#                   the exact recipe).

# Generate the keypair. Default location is what sops looks at.
mkdir -p ~/.config/sops/age
test ! -f ~/.config/sops/age/keys.txt || { echo "admin key already exists"; exit 1; }
age-keygen -o ~/.config/sops/age/keys.txt
chmod 600 ~/.config/sops/age/keys.txt

# Print the pubkey — goes into docs/hosts.md and into every .sops.yaml.
grep '^# public key:' ~/.config/sops/age/keys.txt

Example output: # public key: age1adminadminadminadminadminadminadminadminadminadminad

Back up to vaultwarden

The full file is small (~190 bytes). Store the entire contents as a Secure Note:

Name: Charlie SOPS admin age key
Notes: paste the file content (one # created: line, one # public key: line, one AGE-SECRET-KEY-1... line).

Recover by reversing: paste the note contents back into ~/.config/sops/age/keys.txt, chmod 600.

Every step except (4) and (9) is required. (4) is required if the host runs containers. (9) is required if humans will SSH into it. (10) is required if Gitea Actions deploys land here.

One-time tasks not tied to a specific host:

Admin age key (one-time) — done once, on the admin laptop, before any host gets onboarded.

Hostname

Ubuntu

sudo hostnamectl set-hostname <HOSTNAME>

Mac

sudo scutil --set HostName    <HOSTNAME>
sudo scutil --set ComputerName <HOSTNAME>
sudo scutil --set LocalHostName <HOSTNAME>

# Add the new hostname to /etc/hosts.
sudo nano /etc/hosts

NAS mount

NAS is miquella.lan:/volume1/ubuntu (NFSv4), mounted at /mnt/nas on every host.

Ubuntu

Make sure mDNS works (Bonjour ships on macOS):

sudo apt install avahi-daemon avahi-utils
sudo systemctl enable --now avahi-daemon
avahi-browse -at   # sanity check

Install NFS client utils:

sudo apt install nfs-common

Whitelist the host on the NAS: DSM → Control Panel → Shared Folder → ubuntu → NFS Permissions.

Add to /etc/fstab:

miquella.lan:/volume1/ubuntu  /mnt/nas  nfs  vers=4.1,_netdev,nofail,x-systemd.automount,x-systemd.requires=network-online.target  0  0

Mac

Redirect /mnt/ to /mnt/System/Volumes/Data/mnt/ via synthetic.conf (System Integrity Protection wants it that way):

sudo bash -c 'printf "mnt\tSystem/Volumes/Data/mnt\n" > /etc/synthetic.conf'
sudo chown root:wheel /etc/synthetic.conf
sudo chmod 644 /etc/synthetic.conf
sudo reboot

Edit /etc/auto_master, append at the bottom:

/-    auto_nfs    -nobrowse,nosuid

Create /etc/auto_nfs:

/System/Volumes/Data/mnt/nas  -fstype=nfs,vers=4,resvport  miquella.lan:/volume1/ubuntu

Reload:

sudo automount -vc

Docker

Ubuntu

If docker is installed via snap, uninstall it and reinstall via apt. Snap docker doesn't play nicely with systemd ordering against the NAS automount.

Wire docker to wait for the NAS:

sudo systemctl edit docker

Add:

[Unit]
Wants=network-online.target
After=network-online.target

RequiresMountsFor=/mnt/nas/docker
After=mnt-nas.automount
Requires=mnt-nas.automount

Mac

brew install docker docker-compose docker-machine colima

Set colima to start at boot:

sudo cp /mnt/nas/home/plist/dev.colima.start.plist /Library/LaunchDaemons/dev.colima.start.plist
sudo chown root:wheel /Library/LaunchDaemons/dev.colima.start.plist
sudo chmod 644 /Library/LaunchDaemons/dev.colima.start.plist
sudo launchctl load -w /Library/LaunchDaemons/dev.colima.start.plist

home/plist/ is Tier 5 backlog — see plist. Path will move once that repo is migrated.

Stacks that pull images from gitea.alexandru.macocian.me (e.g. Charlie/runners -> amacocian/gitea-runner, Charlie/ldap -> sidecars, etc.) need authenticated docker pulls. Log in once as root, the credentials persist in /root/.docker/config.json:

sudo docker login gitea.alexandru.macocian.me
# Username: sys-host-<hostname>
# Password: <PAT with read:package scope>
#
# PAT belongs to the host's LDAP system account (sys-host-morgott,
# sys-host-melina, etc.) and lives in vaultwarden as
# "Charlie sys-host-<hostname> Gitea Docker PAT".

⚠️ Make sure you ran with sudo — docker login writes to the running user's ~/.docker/config.json. If you log in as your normal user, root (which is what runs sudo docker compose pull) won't find the credentials.

Verify:

sudo grep -A2 gitea /root/.docker/config.json
sudo docker pull gitea.alexandru.macocian.me/amacocian/gitea-runner:0.6.1-5

Portainer agent

Bring the agent up locally so the host shows up in portainer:

cd /mnt/nas/stacks/portainer-agent
sudo docker compose up -d

If the host needs a different layout (e.g. DSM), the deploy workflow's per-host file resolution picks the right compose. See strategy.md → Shared stacks.

Then go to https://portainer.alexandru.macocian.me and add a new environment for this host.

Metrics (otelcollector)

Ubuntu

cd /mnt/nas/stacks/otelcollector
sudo docker compose up -d

Mac

The mac path doesn't run otelcollector in docker (NFS / colima friction). Install the binary directly:

cd ~

# Check latest release, swap the version below if newer.
curl -s https://api.github.com/repos/open-telemetry/opentelemetry-collector-releases/releases/latest | grep tag_name

curl -LO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.144.0/otelcol-contrib_0.144.0_darwin_amd64.tar.gz
tar -xzf otelcol-contrib_0.144.0_darwin_amd64.tar.gz
sudo mv otelcol-contrib /usr/local/bin/otelcol-contrib
sudo chmod +x /usr/local/bin/otelcol-contrib
rm otelcol-contrib_0.144.0_darwin_amd64.tar.gz

otelcol-contrib --version

Config + LaunchDaemon:

sudo mkdir -p /usr/local/etc/otelcol-contrib
sudo cp /mnt/nas/stacks/otelcollector/otelcol.mac.yaml /usr/local/etc/otelcol-contrib/config.yaml

sudo cp /mnt/nas/stacks/otelcollector/com.otelcol.contrib.plist /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo chown root:wheel /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo chmod 644 /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo launchctl load /Library/LaunchDaemons/com.otelcol.contrib.plist

Verify:

ps aux | grep otelcol
tail -f /var/log/otelcol-contrib.log
tail -f /var/log/otelcol-contrib.err

Service management:

# Stop
sudo launchctl unload /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo pkill -9 otelcol-contrib

# Start
sudo launchctl load /Library/LaunchDaemons/com.otelcol.contrib.plist

# Restart
sudo launchctl unload /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo launchctl load /Library/LaunchDaemons/com.otelcol.contrib.plist

Tracked: re-onboard malenia (mac) for otelcollector — currently deferred (NFS unreliable from malenia). See roadmap.md → Phase 1.

DNS

Intranet DNS is served by Marika (OpenWrt router). DHCP from Marika hands out Marika as the resolver, so a new host doesn't need any client-side DNS config — just a working DHCP lease.

On Marika (one-time per new host)

Add a static lease + hostname so the host gets a stable IP and an <hostname>.lan A record:

LuCI → Network → DHCP and DNS → Static Leases → add MAC + hostname + IP.

Or /etc/config/dhcp:

config host
    option name '<hostname>'
    option mac  '<mac>'
    option ip   '<ip>'

then /etc/init.d/dnsmasq restart on the router.

On the new host

# Force a lease refresh so the host picks up Marika as the resolver
# (only needed if the host is already up and currently using a stale config).
sudo dhclient -r && sudo dhclient

Verify:

resolvectl status | grep -A1 'Current DNS Server'   # expect: Marika's IP
getent hosts <some-other-host>.lan                  # should resolve

Historically there was a docker-dnsmasq stack on miquella; that's gone. Marika's built-in dnsmasq is the only resolver now.

Trusted CA

Partially migrated. The SSH user CA pubkey at ssh/ssh_user_ca.pub is now step-ca's (rolled May 4, 2026). The X.509 root at ca.crt is still the original. Both are still distributed by sync-ca.sh from /mnt/nas/ca/. Pending rewrite of the script to fetch directly from step-ca: see Step-ca migration impact.

Source of truth: /mnt/nas/ca/

ca.crt — internal X.509 root (LDAP TLS, internal services). Original.
ssh/ssh_user_ca.pub — SSH user-cert CA pubkey (sshd TrustedUserCAKeys). Now contains step-ca's user CA pubkey from Charlie/cert-issuer.

Each host runs sync-ca.sh periodically to copy these into local trust stores and bounce sssd / sshd if anything changed.

Ubuntu

/usr/local/sbin/sync-ca.sh (mode 0755, root:root):

#!/usr/bin/env bash
set -euo pipefail

SRC="/mnt/nas/ca/ca.crt"
SSSD_DST="/etc/sssd/certs/ca.crt"
SYS_DST="/usr/local/share/ca-certificates/macocian-ca.crt"
SSH_USER_CA_SRC="/mnt/nas/ca/ssh/ssh_user_ca.pub"
SSH_USER_CA_DST="/etc/ssh/trusted_user_ca.pub"

install -d -m 0755 /etc/sssd/certs
install -d -m 0755 /usr/local/share/ca-certificates
install -d -m 0755 /etc/ssh

if [[ ! -f "$SRC" ]]; then
  echo "sync-ca: source $SRC not found; skipping"
  exit 0
fi

changed_trust=0
changed_sshd=0

copy_if_changed () {
  local src="$1" dst="$2" flagvar="$3"
  if [[ ! -f "$src" ]]; then
    echo "sync-ca: source $src not found; skipping its target $dst"
    return 0
  fi
  local tmp
  tmp="$(mktemp "${dst}.XXXX")"
  cp -f "$src" "$tmp"
  chmod 0644 "$tmp"
  if [[ ! -f "$dst" ]] || ! cmp -s "$tmp" "$dst"; then
    mv -f "$tmp" "$dst"
    chown root:root "$dst" || true
    echo "sync-ca: updated $dst"
    if [[ "$flagvar" == "changed_trust" ]]; then
      changed_trust=1
    elif [[ "$flagvar" == "changed_sshd" ]]; then
      changed_sshd=1
    fi
  else
    rm -f "$tmp"
  fi
}

copy_if_changed "$SRC" "$SSSD_DST" changed_trust
copy_if_changed "$SRC" "$SYS_DST"  changed_trust
copy_if_changed "$SSH_USER_CA_SRC" "$SSH_USER_CA_DST" changed_sshd

if [[ "$changed_trust" -eq 1 ]]; then
  update-ca-certificates >/dev/null 2>&1 || true
  systemctl try-reload-or-restart sssd || true
fi

if [[ "$changed_sshd" -eq 1 ]]; then
  systemctl try-reload-or-restart sshd || systemctl try-reload-or-restart ssh || true
fi

Systemd units:

/etc/systemd/system/sync-ca.service:

[Unit]
Description=Sync private CA from NAS into local trust
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/mnt/nas/ca

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/sync-ca.sh

/etc/systemd/system/sync-ca.timer:

[Unit]
Description=Run sync-ca hourly and at boot

[Timer]
OnBootSec=2min
OnUnitActiveSec=1h
Unit=sync-ca.service
Persistent=true

[Install]
WantedBy=timers.target

Permissions + enable:

sudo chmod 755 /usr/local/sbin/sync-ca.sh
sudo chown root:root /usr/local/sbin/sync-ca.sh

sudo systemctl daemon-reload
sudo systemctl enable --now sync-ca.timer
sudo systemctl start sync-ca.service

Mac

/usr/local/sbin/sync-ca.sh (mode 0755):

#!/usr/bin/env bash
set -euo pipefail

SRC="/mnt/nas/ca/ca.crt"
# macOS equivalent for LDAP CLI trust:
SSSD_DST="/etc/openldap/cacerts/ca.crt"
SYS_DST="/usr/local/share/ca-certificates/macocian-ca.crt"
SSH_USER_CA_SRC="/mnt/nas/ca/ssh/ssh_user_ca.pub"
SSH_USER_CA_DST="/etc/ssh/trusted_user_ca.pub"

install -d -m 0755 /etc/openldap/cacerts
install -d -m 0755 /usr/local/share/ca-certificates
install -d -m 0755 /etc/ssh

if [[ ! -f "$SRC" ]]; then
  echo "sync-ca: source $SRC not found; skipping"
  exit 0
fi

changed_trust=0
changed_sshd=0

copy_if_changed () {
  local src="$1" dst="$2" flagvar="$3"
  if [[ ! -f "$src" ]]; then
    echo "sync-ca: source $src not found; skipping its target $dst"
    return 0
  fi
  local tmp
  tmp="$(mktemp "${dst}.XXXX")"
  cp -f "$src" "$tmp"
  chmod 0644 "$tmp"
  if [[ ! -f "$dst" ]] || ! cmp -s "$tmp" "$dst"; then
    mv -f "$tmp" "$dst"
    chown root:wheel "$dst" 2>/dev/null || chown root:root "$dst" || true
    echo "sync-ca: updated $dst"
    if [[ "$flagvar" == "changed_trust" ]]; then
      changed_trust=1
    elif [[ "$flagvar" == "changed_sshd" ]]; then
      changed_sshd=1
    fi
  else
    rm -f "$tmp"
  fi
}

copy_if_changed "$SRC" "$SSSD_DST" changed_trust
copy_if_changed "$SRC" "$SYS_DST"  changed_trust
copy_if_changed "$SSH_USER_CA_SRC" "$SSH_USER_CA_DST" changed_sshd

if [[ "$changed_trust" -eq 1 ]]; then
  /usr/bin/security add-trusted-cert -d -r trustRoot \
    -k /Library/Keychains/System.keychain "$SSSD_DST" || true

  if [[ -f /etc/openldap/ldap.conf ]]; then
    if ! grep -qE '^TLS_CACERT[[:space:]]+/etc/openldap/cacerts/ca\.crt$' /etc/openldap/ldap.conf; then
      sudo sed -i '' -e '/^TLS_CACERT/d' /etc/openldap/ldap.conf 2>/dev/null || true
      echo "TLS_CACERT /etc/openldap/cacerts/ca.crt" | sudo tee -a /etc/openldap/ldap.conf >/dev/null
      echo "sync-ca: updated /etc/openldap/ldap.conf"
    fi
  else
    echo "TLS_CACERT /etc/openldap/cacerts/ca.crt" | sudo tee /etc/openldap/ldap.conf >/dev/null
    echo "sync-ca: created /etc/openldap/ldap.conf"
  fi
fi

if [[ "$changed_sshd" -eq 1 ]]; then
  /bin/launchctl kickstart -k system/com.openssh.sshd || true
fi

LaunchDaemon at /Library/LaunchDaemons/com.macocian.sync-ca.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.macocian.sync-ca</string>

  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/sbin/sync-ca.sh</string>
  </array>

  <key>RunAtLoad</key>
  <true/>

  <key>StartInterval</key>
  <integer>3600</integer>

  <key>StandardOutPath</key>
  <string>/var/log/sync-ca-macos.log</string>
  <key>StandardErrorPath</key>
  <string>/var/log/sync-ca-macos.log</string>
</dict>
</plist>

Permissions + load:

sudo chown root:wheel /Library/LaunchDaemons/com.macocian.sync-ca.plist
sudo chmod 0644 /Library/LaunchDaemons/com.macocian.sync-ca.plist
sudo launchctl load -w /Library/LaunchDaemons/com.macocian.sync-ca.plist

Append to /etc/ssh/sshd_config:

TrustedUserCAKeys /etc/ssh/trusted_user_ca.pub
PubkeyAuthentication yes

sudo launchctl kickstart -k system/com.openssh.sshd

Host age key (SOPS)

Each host gets one age keypair, used to decrypt SOPS-encrypted secrets at deploy time. Public key is non-secret and gets registered in docs/hosts.md. Private key never leaves the host. Design: secrets.md.

Ubuntu / Mac

# Install age. (Apt on Ubuntu; brew on macOS.)
sudo apt install age            # or: brew install age
age --version

# Install sops. Ubuntu/Debian apt has no sops package; pull the upstream .deb.
# macOS: `brew install sops` instead.
# Bump VER deliberately. Releases: https://github.com/getsops/sops/releases
VER=3.12.2
curl -fLO "https://github.com/getsops/sops/releases/download/v${VER}/sops_${VER}_amd64.deb"
sudo dpkg -i "sops_${VER}_amd64.deb"
rm "sops_${VER}_amd64.deb"
sops --version

# Generate the key. Refuse to clobber an existing one.
sudo install -d -m 0700 -o root -g root /etc/charlie
test ! -f /etc/charlie/age.key || { echo "age.key already exists; refusing to overwrite"; exit 1; }
sudo sh -c 'age-keygen -o /etc/charlie/age.key 2>&1'
# That prints the public key on stderr. Capture it for the next step.

sudo chmod 0600 /etc/charlie/age.key
sudo chown root:root /etc/charlie/age.key

# Print the pubkey one more time for safety (the comment line at the top of
# the key file). This goes into docs/hosts.md.
sudo grep '^# public key:' /etc/charlie/age.key

Example output:

# public key: age1xyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyz

Register the public key

Add a row to the Age public keys (SOPS) table in docs/hosts.md. Pubkeys are public; commit + push is fine.

Make the host actually able to decrypt

SOPS looks for the age key at $SOPS_AGE_KEY_FILE. The deploy automation sets this to /etc/charlie/age.key and runs as sys-gitea-runner over SSH — which has read access via the charlie-deploy group (see Charlie deploy group). Until that section runs, the key is root-only.

# Sanity-check that root can read the key (initially the only reader).
sudo test -r /etc/charlie/age.key && echo OK

Adding this host to a stack's recipients

When this host needs to decrypt a SOPS-encrypted file in a Charlie/<stack> repo, add its pubkey to that repo's .sops.yaml and re-encrypt:

# On the admin laptop, in the stack repo:
$EDITOR .sops.yaml      # add the new host's pubkey to the recipients list
sops updatekeys secrets/env.sops.yaml
git commit -am "Grant <host> SOPS access"
git push

LDAP / SSSD

Identity: SSSD-backed against ldaps://ldap.lan (the ldap stack on morgott).

Ubuntu

Install SSSD:

sudo apt update
sudo apt install sssd-ldap libpam-sss libnss-sss oddjob-mkhomedir oddjob sssd-tools

/etc/sssd/sssd.conf:

[sssd]
services = nss, pam
config_file_version = 2
domains = LDAP

[domain/LDAP]
debug_level = 9
id_provider = ldap
auth_provider = ldap

ldap_uri = ldaps://ldap.lan
ldap_search_base = dc=alexandru,dc=macocian,dc=me

ldap_default_bind_dn = cn=admin,dc=alexandru,dc=macocian,dc=me
ldap_default_authtok = __PASSWORD__

ldap_id_use_start_tls = false
ldap_tls_reqcert = demand
ldap_tls_cacert = /etc/sssd/certs/ca.crt

# posixAccount + posixGroup/memberUid
ldap_schema = rfc2307
ldap_user_object_class     = posixAccount
ldap_user_name             = uid
ldap_user_uid_number       = uidNumber
ldap_user_gid_number       = gidNumber
ldap_user_home_directory   = homeDirectory
ldap_user_shell            = loginShell

ldap_group_object_class    = posixGroup
ldap_group_name            = cn
ldap_group_gid_number      = gidNumber
ldap_group_member          = memberUid

Permissions + restart:

sudo chmod 600 /etc/sssd/sssd.conf
sudo systemctl restart sssd

/etc/ssh/sshd_config:

PubkeyAuthentication yes
TrustedUserCAKeys /etc/ssh/trusted_user_ca.pub

Append to both /etc/pam.d/common-session and /etc/pam.d/common-session-noninteractive:

session required        pam_mkhomedir.so skel=/etc/skel/ umask=0077

Grant admins sudo:

echo '%admins ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/90-ldap-admins
sudo visudo -cf /etc/sudoers.d/90-ldap-admins

Mac

Copy the LDAP plist:

sudo cp /mnt/nas/home/plist/ldap.lan.plist /Library/Preferences/OpenDirectory/Configurations/LDAPv3/ldap.lan.plist

Bind password may need refreshing in Directory Utility after copy.

Grant SSH to users:

sudo dseditgroup -o create com.apple.access_ssh
sudo dscl . -append /Groups/com.apple.access_ssh NestedGroups "users@/LDAPv3/ldap.lan"

sudo dseditgroup -o create -n /Local/Default com.apple.access_loginwindow
sudo dseditgroup -o edit -a users -t group -n /Local/Default com.apple.access_loginwindow

sudo dscl . -append /Groups/com.apple.access_loginwindow NestedGroups "users@/LDAPv3/ldap.lan"
sudo dseditgroup -o edit -a "users@/LDAPv3/ldap.lan" -t group com.apple.access_loginwindow

Grant sudo to admins:

sudo dseditgroup -o create -n /Local/Default admins
sudo dscl . -append /Groups/admins NestedGroups "admins@/LDAPv3/ldap.lan"

echo '%admins ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/90-ldap-admins >/dev/null
sudo chmod 440 /etc/sudoers.d/90-ldap-admins
sudo visudo -cf /etc/sudoers.d/90-ldap-admins

Charlie deploy group

The charlie-deploy LDAP group is the single identity that owns deployment on every host. Members of this group can:

Read/write /mnt/nas/stacks/<stack>/ (working trees the deploy workflow operates on)
Talk to the host's docker daemon (via /var/run/docker.sock group ownership)
Decrypt SOPS-encrypted secrets (read /etc/charlie/age.key)

This setup means the deploy workflow runs without sudo as sys-gitea-runner (or any future deploy identity). One LDAP group gates everything; no scoped sudoers to maintain.

Pre-req: charlie-deploy exists in LDAP, sys-gitea-runner is a member. GID is whatever LDAP picked; SSSD resolves it consistently across hosts.

Ubuntu

# 1. Tell dockerd to use charlie-deploy as the socket-owning group, instead
#    of the default local `docker` group. Means LDAP membership in
#    charlie-deploy alone grants docker access. No local group juggling.
sudo install -d -m 0755 /etc/docker
sudo tee /etc/docker/daemon.json >/dev/null <<'JSON'
{
  "group": "charlie-deploy"
}
JSON

# 2. Override docker.socket to use charlie-deploy as the socket group.
#    On Ubuntu, dockerd is socket-activated: docker.socket creates
#    /var/run/docker.sock with SocketGroup=docker hardcoded BEFORE dockerd
#    starts, then dockerd inherits the FD via `fd://`. The `group` setting
#    in daemon.json only applies when dockerd creates the socket itself,
#    not when systemd hands it one. So override the socket unit too.
sudo install -d -m 0755 /etc/systemd/system/docker.socket.d
sudo tee /etc/systemd/system/docker.socket.d/group.conf >/dev/null <<'INI'
[Socket]
SocketGroup=charlie-deploy
INI

# 3. Make sure dockerd starts after SSSD is up, so the charlie-deploy group
#    is resolvable when docker.service starts. Without this, dockerd may
#    fall back to the default group on first boot.
sudo install -d -m 0755 /etc/systemd/system/docker.service.d
sudo tee /etc/systemd/system/docker.service.d/wait-for-sssd.conf >/dev/null <<'INI'
[Unit]
After=sssd.service
Wants=sssd.service
INI

sudo systemctl daemon-reload
sudo systemctl restart docker.socket docker.service

# 4. Verify the docker socket is owned by charlie-deploy.
ls -l /var/run/docker.sock
# Expect: srw-rw---- 1 root charlie-deploy ...

# 5. Verify a charlie-deploy member can talk to docker without sudo.
sudo -u sys-gitea-runner -H docker ps
# Should list containers (or "no containers"), no permission error.

# 5. Stack working trees: sys-gitea-runner owns them, charlie-deploy is the
#    group, group-writable, with setgid on directories so new files (e.g.
#    created by `git fetch`) inherit the charlie-deploy group automatically.
#
#    User ownership matters because git refuses to operate on a working tree
#    owned by another user ("dubious ownership"). Setting the user to
#    sys-gitea-runner avoids needing per-host safe.directory config files.
sudo chown -R sys-gitea-runner:charlie-deploy /mnt/nas/stacks
sudo chmod -R g+w /mnt/nas/stacks
sudo find /mnt/nas/stacks -type d -exec chmod g+s {} \;

# Verify a charlie-deploy member can do a write against the working tree.
# Pick any stack you have on this host (e.g. homarr).
sudo -u sys-gitea-runner -H -- bash -c '
  cd /mnt/nas/stacks/homarr &&
  git status
'

# 6. SOPS host key: charlie-deploy readable. The directory must also be
#    traversable by the group, otherwise sys-gitea-runner can read the file
#    permission but not actually open it.
sudo chgrp charlie-deploy /etc/charlie
sudo chmod 0750 /etc/charlie
sudo chgrp charlie-deploy /etc/charlie/age.key
sudo chmod 0640 /etc/charlie/age.key

ls -ld /etc/charlie /etc/charlie/age.key
sudo -u sys-gitea-runner -H test -r /etc/charlie/age.key && echo OK

# 7. Pre-create sys-gitea-runner's home directory and seed it with a Gitea
#    PAT so `git fetch/clone/push` against Charlie/* repos works without
#    interactive prompts.
#
#    LDAP defines the home as /Users/sys-gitea-runner (mac-style; ready for
#    a future shared-home rollout). pam_mkhomedir only creates the home on
#    interactive login; `sudo -u ... -H` doesn't trigger it, so we make it
#    ourselves on each new host.
#
#    The PAT is the same one used by the runner image's `RUNNER_REPO_PAT`
#    org-level secret. It belongs to sys-gitea-runner and is read-only on
#    the Charlie org. Source of truth: vaultwarden secure note
#    `Charlie sys-gitea-runner Gitea Repos PAT`.
SR_HOME="/Users/sys-gitea-runner"

sudo install -d -m 0755 /Users
sudo install -d -m 0700 -o sys-gitea-runner -g charlie-deploy "$SR_HOME"

PAT="<paste-from-vaultwarden>"

sudo tee "$SR_HOME/.git-credentials" >/dev/null <<EOF
https://sys-gitea-runner:$PAT@gitea.alexandru.macocian.me
EOF
sudo chown sys-gitea-runner:charlie-deploy "$SR_HOME/.git-credentials"
sudo chmod 600 "$SR_HOME/.git-credentials"

sudo tee "$SR_HOME/.gitconfig" >/dev/null <<'EOF'
[credential]
	helper = store
EOF
sudo chown sys-gitea-runner:charlie-deploy "$SR_HOME/.gitconfig"
sudo chmod 644 "$SR_HOME/.gitconfig"

# Verify: should list refs without prompting for a password.
sudo -u sys-gitea-runner -H git ls-remote https://gitea.alexandru.macocian.me/Charlie/runners | head -2

When NFS-shared homes land (planned), /Users/ becomes the bind mount. The per-host .gitconfig/.git-credentials above become superfluous (the shared home brings them along). Migration: delete the local copies after the bind takes over.

# 8. Set up root's git credentials, using THIS host's system account
#    (sys-host-<hostname>). Used for any sudo git operations done by humans
#    or by other root-level automation. The runner uses sys-gitea-runner
#    via the credential file in step 7; `root` here is for everything else.
#
#    Source of truth: vaultwarden secure note
#    `Charlie sys-host-<hostname> Gitea Repos PAT`.
HOST_PAT="<paste-from-vaultwarden>"

sudo tee /root/.git-credentials >/dev/null <<EOF
https://sys-host-${HOST}:$HOST_PAT@gitea.alexandru.macocian.me
EOF
sudo chmod 600 /root/.git-credentials

sudo tee /root/.gitconfig >/dev/null <<'EOF'
[credential]
	helper = store
EOF
sudo chmod 644 /root/.gitconfig

# Verify
sudo git ls-remote https://gitea.alexandru.macocian.me/Charlie/runners | head -2

PAT identities reference

Three Gitea identities are used per host. Keep them straight:

Identity	Where the PAT lives	Used for	Scopes
`sys-gitea-runner`	`~sys-gitea-runner/.git-credentials` (host) + `RUNNER_REPO_PAT` org secret + runner image env	Runner-side git operations: `act` cross-repo `uses:` clones, runner deploys via SSH-to-host.	`read:repository`
`sys-host-<host>`	`/root/.git-credentials` (host)	Root-level git operations on the host: humans doing `sudo git ...`, manual `git fetch && git reset --hard` for fallback deploys.	`read:repository`
`sys-host-<host>` (separate PAT)	`/root/.docker/config.json` (host)	Root-level docker registry pulls: `sudo docker compose pull`.	`read:package`

Having three separate PATs is deliberate: independent rotation, scope-limited compromise. All three live in vaultwarden under predictable names (Charlie sys-<account> Gitea <Purpose> PAT).

Verify end-to-end

sys-gitea-runner should now be able to do the entire deploy by hand, without sudo:

# Become the deploy identity. -H sets HOME so credential helpers work.
sudo -u sys-gitea-runner -H bash

# Inside that shell — no sudo for any of these:
cd /mnt/nas/stacks/<some-stack>
git fetch --prune origin && git reset --hard origin/main
SOPS_AGE_KEY_FILE=/etc/charlie/age.key sops --decrypt --output .env secrets/env.sops.yaml
docker compose pull
docker compose up -d
rm -f .env

If any step fails with permission errors, fix the corresponding chgrp/chmod above. The deploy automation calls these exact commands over SSH as sys-gitea-runner; if it works manually, it works in CI.

NFS GID mapping caveat

/mnt/nas/stacks/ lives on the NAS (DSM/miquella). DSM doesn't know about LDAP; when you chgrp charlie-deploy <path> on a host, NFS sends the numeric GID to DSM, which stores it as-is. Other hosts then read it back and SSSD resolves the GID to charlie-deploy. Works because the kernel cares about GIDs, not names.

Cosmetic side effect: from a host where SSSD is down, ls -l shows the numeric GID instead of the group name. Functionally fine.

If DSM's NFS export is configured with all_squash or any UID/GID mapping, this approach breaks. Verify by chgrp from one host, ls from another — both should resolve to charlie-deploy.

SSH defaults

Global readline config (advanced command-line editing):

sudo cp /mnt/nas/home/shell/inputrc /etc/inputrc

home/shell/ is Tier 5 backlog. Path will move once shell is migrated.

Color + sane defaults in /etc/profile:

sudo tee -a /etc/profile >/dev/null <<'EOF'

# === Global color and terminal setup ===
export TERM=${TERM:-xterm-256color}
export CLICOLOR=1
export LSCOLORS=ExFxBxDxCxegedabagacad

# Make sure /etc/inputrc is used
export INPUTRC=/etc/inputrc

# Colored prompt for interactive shells
if [ "$PS1" ]; then
    PS1='\[\e[1;32m\]\u@\h\[\e[0m\]:\[\e[1;34m\]\w\[\e[0m\]\$ '
fi
EOF

Step-ca migration impact

Charlie/cert-issuer is deployed. Migration so far:

✅ SSH user CA at /mnt/nas/ca/ssh/ssh_user_ca.pub swapped to step-ca's pubkey. sync-ca.sh distributes it unchanged.
✅ Charlie/gssh rewritten as a step-ca client. Old CA key no longer mounted into Gssh.

Pending. When Charlie/cert-issuer takes over from /mnt/nas/ca/ entirely, this is what changes for onboarding:

Today	After step-ca takes over
`sync-ca.sh` reads `/mnt/nas/ca/ca.crt`	Pulls from `https://cert-issuer.lan:9000/roots.pem` (or the bootstrapped `~/.step/certs/root_ca.crt`).
`sync-ca.sh` reads `/mnt/nas/ca/ssh/ssh_user_ca.pub`	Pulls via `step ssh config --roots`.
Trust source = NFS share readable by every host	Trust source = HTTPS endpoint pinned by root fingerprint at first contact.
SSH host trust = TOFU (`known_hosts` accumulates)	sshd presents an SSH host cert from step-ca (`HostCertificate`); clients trust via `@cert-authority *.lan ...`.
Adding a host = NFS whitelist + `sync-ca.timer`	Adding a host = NFS whitelist + one-time fetch of root + register host with step-ca for its host cert.

What does not change:

LDAP / SSSD section. SSSD still wants ldap_tls_cacert = /etc/sssd/certs/ca.crt. step-ca populates that file via the same sync-ca.sh rewrite.
The hourly refresh model. Just changes its source, not its behaviour.
Everything outside the Trusted CA section.

What goes away once the rewrite ships:

/mnt/nas/ca/ as a NAS dependency. After migration nothing reads it; archive then delete.
The sync-ca.timer hard dependency on RequiresMountsFor=/mnt/nas/ca (since it'll fetch over HTTPS, not NFS).

The actual rewrite of sync-ca.sh (and its Mac equivalent) lives with Charlie/cert-issuer. Until then this section stays accurate and authoritative.

32 KiB Raw Blame History

Host onboarding

Admin age key (one-time)

Back up to vaultwarden

Order of operations

Hostname

Ubuntu

Mac

NAS mount

Ubuntu

Mac

Docker

Ubuntu

Mac

Charlie registry login

Portainer agent

Metrics (otelcollector)

Ubuntu

Mac

DNS

On Marika (one-time per new host)

On the new host

Trusted CA

Ubuntu

Mac

Host age key (SOPS)

Ubuntu / Mac

Register the public key

Make the host actually able to decrypt

Adding this host to a stack's recipients

LDAP / SSSD

Ubuntu

Mac

Charlie deploy group

Ubuntu

PAT identities reference

Verify end-to-end

NFS GID mapping caveat

SSH defaults

Step-ca migration impact

32 KiB

Raw Blame History