Self-hosting this TIL site: Forgejo on the LAN, static VPS¶

A runbook for building, publishing and backing up this site. Git hosting and the build stay inside the LAN. The public VPS serves static files only.

The machines¶

How it connects¶

graph TB
    subgraph daily["3 - Daily machines"]
        laptop["Mac / Linux laptop"]
        browser["Browser<br/>Forgejo web editor"]
    end

    subgraph lan["2 - Intranet server (LAN only)"]
        forgejo["Forgejo<br/>:3000"]
        recv["webhook receiver<br/>127.0.0.1:9000"]
        build["build.sh<br/>mkdocs build --strict"]
        intsite["internal site<br/>edit button"]
    end

    subgraph vps["1 - VPS (public)"]
        deploysh["deploy-til wrapper"]
        web["nginx + TLS<br/>/var/www"]
    end

    remote["4 - GitHub / Codeberg"]
    cold["5 - Cold bundles"]
    visitor["Visitors"]

    laptop -- "git push over SSH" --> forgejo
    browser -- "edit page" --> forgejo
    forgejo -- "webhook, HMAC signed" --> recv
    recv --> build
    build -- "internal build" --> intsite
    build -- "public build, tar over SSH" --> deploysh --> web
    build -- "read-only deploy key" --> forgejo
    forgejo -- "push mirror" --> remote
    forgejo -- "bundle" --> cold
    browser -- "read/write" --> intsite
    visitor -- "HTTPS" --> web

Every connection is initiated from inside the LAN. The VPS holds no credentials and no route to any other machine.

1. Prepare the intranet server¶

sudo apt update
sudo apt install git python3 rsync nginx
curl -LsSf https://astral.sh/uv/install.sh | sh    # uv, for the build venv

2. Install Forgejo¶

Follow the official binary installation exactly. Version 12.0 or later is required for the CMS in step 11.

curl -sSLO https://codeberg.org/forgejo/forgejo/releases/download/v<VERSION>/forgejo-<VERSION>-linux-amd64
sudo install -m 755 forgejo-<VERSION>-linux-amd64 /usr/local/bin/forgejo

sudo adduser --system --shell /bin/bash --gecos 'Git Version Control' \
  --group --disabled-password --home /home/git git

sudo mkdir /var/lib/forgejo && sudo chown git:git /var/lib/forgejo && sudo chmod 750 /var/lib/forgejo
sudo mkdir /etc/forgejo && sudo chown root:git /etc/forgejo && sudo chmod 770 /etc/forgejo

sudo wget -O /etc/systemd/system/forgejo.service \
  https://codeberg.org/forgejo/forgejo/raw/branch/forgejo/contrib/systemd/forgejo.service
sudo systemctl daemon-reload
sudo systemctl enable forgejo

Write the config before first start, with all four secrets supplied and INSTALL_LOCK set. Generate each with forgejo generate secret <SECRET_KEY|INTERNAL_TOKEN|JWT_SECRET|LFS_JWT_SECRET>.

APP_NAME = <NAME>
RUN_USER = git
RUN_MODE = prod
WORK_PATH = /var/lib/forgejo

[server]
PROTOCOL         = http
DOMAIN           = <LAN_ADDRESS>
ROOT_URL         = http://<LAN_ADDRESS>:3000/
HTTP_ADDR        = <LAN_OR_VPN_ADDRESS>
HTTP_PORT        = 3000
APP_DATA_PATH    = /var/lib/forgejo/data
DISABLE_SSH      = false
START_SSH_SERVER = false
SSH_PORT         = 22
SSH_DOMAIN       = <LAN_ADDRESS>
LFS_START_SERVER = true
LFS_JWT_SECRET   = <GENERATED>
OFFLINE_MODE     = true

[database]
DB_TYPE = sqlite3
PATH    = /var/lib/forgejo/data/forgejo.db

[repository]
ROOT = /var/lib/forgejo/data/forgejo-repositories

[security]
INSTALL_LOCK   = true
SECRET_KEY     = <GENERATED>
INTERNAL_TOKEN = <GENERATED>

[oauth2]
JWT_SECRET = <GENERATED>

[service]
DISABLE_REGISTRATION = true

[webhook]
# Required: the receiver in step 8 listens on loopback, which Forgejo
# blocks by default as SSRF protection.
ALLOWED_HOST_LIST = loopback

[log]
MODE      = console
LEVEL     = info
ROOT_PATH = /var/lib/forgejo/log

[mailer]
ENABLED = false

Write it as a file and install it rather than pasting a heredoc into a shell:

sudo install -o root -g git -m 640 app.ini /etc/forgejo/app.ini
sudo chmod 750 /etc/forgejo
sudo systemctl start forgejo

Create the admin account, then enable 2FA in the web UI:

sudo -u git /usr/local/bin/forgejo admin user create --admin \
  --username <USER> --email <EMAIL> --random-password \
  -w /var/lib/forgejo -c /etc/forgejo/app.ini

No nginx or TLS is needed for Forgejo: it is reachable only from the LAN.

3. Reach it from outside the LAN¶

Install Tailscale (or WireGuard) on the intranet server and every daily machine, and bind Forgejo's HTTP_ADDR to the VPN address.

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

Editing from a phone or while travelling requires the device on the VPN.

4. Add SSH keys¶

On each daily machine:

ssh-keygen -t ed25519 -C "<MACHINE_NAME>"

Add each public key in Forgejo under Settings → SSH / GPG Keys, then:

ssh -T git@<LAN_ADDRESS>

5. Create the repository¶

Create an empty repository in Forgejo, then from a daily machine:

git remote add origin git@<LAN_ADDRESS>:<USER>/til.git
git push -u origin main

6. Two builds: public and internal¶

The edit button must not appear on the public site. Rather than hiding it with scripting, the site is built twice.

mkdocs.yml (public) carries no repo_url and no edit_uri, so Material renders no edit button and no repository link.

mkdocs.internal.yml adds them, taking the host from the environment so the internal address is never committed:

INHERIT: ./mkdocs.yml

repo_url: !ENV [FORGEJO_REPO_URL, "http://localhost:3000/<USER>/til"]
edit_uri: "_edit/main/docs/"

mkdocs build --strict                             # public  -> VPS
FORGEJO_REPO_URL=http://<LAN_ADDRESS>:3000/<USER>/til \
  mkdocs build --strict -f mkdocs.internal.yml    # internal -> LAN

7. The hosting project¶

Static hosting lives in its own directory, separate from Forgejo:

~/web_services/til.anandas.in/
├── til.anandas.in.conf      # remote, branch, ports, deploy target
├── justfile                 # just build / serve / stop / status
├── scripts/
│   ├── build.sh             # sync, build both, verify, promote
│   ├── serve.sh             # local nginx containers
│   └── webhook-receiver.py
├── .ssh/deploy_key          # read-only, repo-scoped
├── src/                     # working clone
└── static_hosting/{public,internal}

Generate a read-only deploy key for the build and add it in Forgejo under Repository → Settings → Deploy Keys, leaving write access unchecked:

ssh-keygen -t ed25519 -f ~/web_services/til.anandas.in/.ssh/deploy_key -N ""

build.sh fetches with that key, builds both variants into temporary directories, and only promotes them if two checks pass:

• the public output contains no Edit this page
• the public output contains no occurrence of the Forgejo host

Promote with rsync -a --delete into the existing directories. Do not replace them — see the gotchas below.

8. Auto-build on push¶

A small receiver runs as the user that owns the web root, so no step in the publish path needs root.

body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
sent = self.headers.get("X-Forgejo-Signature", "")
expected = hmac.new(SECRET, body, hashlib.sha256).hexdigest()
if not hmac.compare_digest(sent, expected):      # constant time
    self.send_response(403); return

It must verify the signature, return immediately (Forgejo times out in seconds while a build takes longer), serialise builds with a lock, and ignore refs other than main.

Run it as a systemd user service, bound to 127.0.0.1 only:

systemctl --user enable --now til-webhook

Then in Forgejo, Repository → Settings → Webhooks → Forgejo:

Use Test Delivery to confirm. This requires ALLOWED_HOST_LIST = loopback from step 2.

9. Serve the internal site on the LAN¶

Bind it to the LAN/VPN address only. This is the copy with edit buttons; use it when writing, and the public URL when reading.

server {
    listen <LAN_OR_VPN_ADDRESS>:80;
    server_name <LAN_HOSTNAME_SITE>;
    root /home/<USER>/web_services/til.anandas.in/static_hosting/internal;
    index index.html;
    location / { try_files $uri $uri/ =404; }
}

10. Publish the public site to the VPS¶

The VPS needs nginx only. No git, no Python, no secrets. In this setup it is a shared host already serving unrelated sites, so everything below is additive — check what is there before installing (nginx -T, certbot certificates).

sudo adduser --disabled-password --gecos "" deploy
sudo install -d -o deploy -g deploy -m 700 /home/deploy/.ssh
sudo install -d -o deploy -g deploy /var/www/til.anandas.in

Authorise the publishing key — generated on the intranet build machine, so the private half never leaves it:

ssh-ed25519 AAAA... til-publish@<BUILD_HOST>

Optionally restrict it to one directory with rrsync, which ships with rsync and needs no custom script:

command="rrsync /var/www/til.anandas.in",no-pty,no-agent-forwarding,no-port-forwarding ssh-ed25519 AAAA... til-publish@<BUILD_HOST>

nginx, then certbot. Order matters: certbot validates over HTTP from the public internet, so the DNS record must already point here or issuance fails.

server {
    listen 80;
    server_name til.anandas.in;
    root /var/www/til.anandas.in;
    index index.html;
    location / { try_files $uri $uri/ $uri.html =404; }
    error_page 404 /404.html;
    location ~* \.(css|js|png|jpg|jpeg|gif|svg|woff2?|ico)$ {
        expires 30d;
        add_header Cache-Control "public";
    }
    gzip on;
    gzip_types text/css application/javascript application/json image/svg+xml;
    gzip_min_length 1024;
}

On a multi-site host do not remove sites-enabled/default without checking what it is — it may be a real vhost rather than the stock placeholder.

sudo nginx -t && sudo systemctl reload nginx
sudo certbot --nginx -d til.anandas.in
sudo certbot renew --dry-run

Publishing is manual, by choice¶

The build machine pushes when told to, not on every commit. Edits reach the LAN copy immediately; the public site moves forward on one command, once a batch of work is worth showing.

just release     # build, then publish
just publish     # publish an existing build

scripts/publish.sh is a plain rsync. An earlier version used dated release directories and an atomic symlink swap — that is machinery for services where a moment of inconsistency costs something, and was removed as overkill for a static notes site.

One guard was kept, and it is the important one:

pages="$(find "$pub" -name '*.html' | wc -l)"
if [ ! -f "$pub/index.html" ] || [ "$pages" -lt 50 ]; then
    echo "REFUSING: $pub looks wrong ($pages pages)." >&2
    exit 1
fi

rsync --delete faithfully mirrors an empty source, which would erase the live site. The check runs before anything leaves the machine.

The script then verifies over HTTPS rather than trusting rsync's exit code — see the gotcha about builds that succeed while publishing nothing.

Certbot on a shared host: check the authenticator¶

Certs issued with authenticator = standalone need port 80 completely free. On a host where nginx runs permanently that can never succeed — such a cert works at issuance and then fails every renewal, silently, until it is close enough to expiry to warn.

sudo grep -E 'authenticator' /etc/letsencrypt/renewal/*.conf

Anything reporting standalone on a host with a permanent web server is already broken. Convert it to renew through nginx:

sudo certbot certonly --nginx --cert-name <NAME> -d <DOMAIN> \
     --deploy-hook "systemctl reload nginx"

--deploy-hook is required with certonly: without an installer nothing reloads nginx, so a renewed cert sits on disk while nginx serves the expired one from memory. Check the full domain list with certbot certificates first — certbot replaces the SAN set with whatever -d flags are passed, so omitting a name silently drops it.

11. Editing: why there is no CMS¶

Considered and rejected. Sveltia CMS is compatible on paper — Forgejo 12.0+ exposes the Gitea backend, and the CMS uses PKCE, so no client secret and no auth broker are needed.

It was dropped because Forgejo's own web UI already covers both jobs:

• Editing — the edit this page button on the internal site opens the file directly.
• Creating — Add File → New File in Forgejo, typing the full path. The nav picks the page up on its own (section 12), so no config edit is needed.

A CMS would add an OAuth application, a /admin route that must stay off the public site, and a config file duplicating the site structure — for a create-page form. Not worth the moving parts at this scale.

12. Navigation without editing mkdocs.yml¶

New pages must appear without hand-editing mkdocs.yml, or the --strict build fails and nothing publishes.

mkdocs-awesome-nav      # in requirements.in

plugins:
  - awesome-nav

Set titles and ordering per directory with .nav.yml:

title: linux-infra
nav:
  - Nginx.md
  - certbot.md
  - "*"

"*" matches everything not listed, so a new file appears with no config change.

13. Mirror to GitHub / Codeberg¶

An off-site copy that costs nothing to maintain. Forgejo pushes to it automatically; the remote is never edited by hand.

On GitHub — Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new:

• Repository access: Only select repositories → the mirror repository
• Permissions: Contents → Read and write (nothing else is needed)
• Store it: pass insert til/github-token

In Forgejo — Repository → Settings → Mirror Settings → Add Push Mirror:

Then Synchronize Now.

Verify it properly rather than trusting the UI: push a commit to Forgejo only, wait, and confirm the remote moved on its own.

git push origin main                 # Forgejo only
sleep 30
git fetch github && git rev-parse --short main github/main   # must match

git remote set-url --push github no_push://mirror-is-read-only

13a. Secrets¶

Four secrets exist in this setup. None belong in a repository.

pass is the source of record for the shared secrets. hook-install.sh reads til/webhook-secret from it, generating one on first run:

pass insert til/webhook-secret     # or let hook-install.sh create it
pass show til/webhook-secret       # to paste into Forgejo's webhook

The secret is read once at install time and written into the systemd unit (mode 600), because a service starting at boot cannot prompt to unlock a GPG key. pass is therefore the durable record, not the runtime source.

What pass is actually for here. The secret still reaches the systemd unit in plaintext, so on-box protection is the same as a mode-600 file. What it adds is a record that is encrypted at rest and survives the machine — which is the failure this setup already hit once, when a stray rsync --delete-excluded destroyed the only copy of the webhook secret and the deploy key. Treat pass as the backup of record, not as runtime hardening.

The SSH deploy key stays a file. A mode-600 private key is already the conventional secure form; putting it in pass would mean writing it back to disk at build time, which is worse. Keep a copy in pass only if you would rather restore it than re-register a new one.

Do not back up the Forgejo instance dump routinely. It carries the four app.ini secrets and the mirror's access token in cleartext, duplicating values that pass already holds — and those values never change, so the copy buys nothing. What is unique to the dump (one user, two repos, a deploy key, a webhook) is faster to recreate from this page than to restore. Back up ~/.password-store instead: it is GPG-encrypted at rest, so it is safe to hold on an unencrypted drive. If you do keep a dump, encrypt it first.

Rotate, don't recover. Any of these can be regenerated in under a minute, and the correct response to a lost or exposed secret is a new one plus revocation of the old — never an attempt to retrieve the original.

Gitignore them, and verify before the first commit of any hosting project:

git ls-files | grep -E '\.ssh/|secret|token' && echo "STOP: secret staged"

14. Cold backups¶

Forgejo lives on the intranet server, so its disk cannot be the only copy. A second machine pulls both repositories — the content repo and the hosting project, which holds the build script and the webhook receiver — and writes dated bundles.

Put the bare mirrors and the bundles on two different physical disks, so a single drive failure loses at most one of them. On the laptop here the mirrors sit on the NVMe and the bundles on a second internal SATA disk.

Be clear about what this leg does and does not cover:

Bundles on a second internal disk are not an off-site copy. The mirror in step 13 is the off-site leg; this one adds point-in-time history that a force-push cannot destroy.

Keep it all in one place — script, mirrors and bundles together:

<BACKUP_ROOT>/
├── backup-til.sh
├── mirrors/            bare mirrors, incrementally fetched (working state)
│   ├── til.git
│   └── host_til.git
└── bundles/            dated snapshots of full history (the actual backup)
    ├── til-<date>.bundle
    └── .til.last-bundled

Compare against what was last bundled, not against the current run's fetch. Detecting change from the fetch alone means any git remote update run outside the script silently advances the mirror, and the new commits then appear in no bundle at all:

marker="$BUNDLES/.$name.last-bundled"
current="$(git -C "$repo" rev-parse --all | sha256sum | cut -d' ' -f1)"
[ -f "$marker" ] && [ "$current" = "$(cat "$marker")" ] && continue
# ... bundle, verify, then:
printf '%s' "$current" > "$marker"

Bundle only when the refs have changed:

before=$(git -C "$REPO" rev-parse --all | sha256sum)
git -C "$REPO" remote update            # deliberately no --prune
after=$(git -C "$REPO" rev-parse --all | sha256sum)
[ "$before" = "$after" ] || { git -C "$REPO" bundle create "$f" --all; git bundle verify "$f"; }

Run it from a persistent timer so a missed run happens at next boot:

[Timer]
OnCalendar=daily
Persistent=true

A user timer only runs while that user has a session. To have it fire on a machine nobody is logged into:

sudo loginctl enable-linger <USER>

Test the restore, once. A backup that has never been restored is a hypothesis. Clone from a bundle and build the site from it:

git clone <COLD_STORAGE>/til-<DATE>.bundle /tmp/restore-test
cd /tmp/restore-test
uv venv .venv && uv pip install --python .venv/bin/python -r requirements.in
.venv/bin/mkdocs build --strict

14a. The instance dump¶

The bundles hold the repositories and nothing else. Everything needed to rebuild the instance around them — accounts, SSH and deploy keys, the webhook and its secret, the push mirror and its token, and app.ini — lives in Forgejo's database. Dump it separately.

Skip the repositories: they are already in the bundles, and including them would multiply the size for no gain.

sudo install -d -o git -g git -m 750 /var/lib/forgejo/tmp

sudo -u git forgejo dump -c /etc/forgejo/app.ini -w /var/lib/forgejo \
  -t /var/lib/forgejo/tmp \
  --skip-repository --skip-log --skip-index \
  -f /var/lib/forgejo/tmp/forgejo-$(date +%F).zip

The temp directory must exist and be writable by git, or the dump runs to completion and then fails at the final step.

Move the result somewhere the backup machine can read, then automate it with a system timer — a user unit would need lingering, and this needs root regardless:

[Timer]
OnCalendar=weekly
Persistent=true

The backup machine pulls the dumps alongside the bundles, so the server still needs no route to it.

Verify a dump rather than assuming it:

unzip -l <dump>.zip                          # app.ini, forgejo-db.sql, data/
unzip -l <dump>.zip | grep -c repositories/  # expect 0
unzip -p <dump>.zip forgejo-db.sql | grep -c "INSERT INTO \`user\`"

Gotchas¶

Things that cost time on the first build of this setup.

The service hangs on systemctl start. The upstream unit is Type=notify. If INSTALL_LOCK is not set, Forgejo parks on the web installer and never signals readiness, so systemd waits the full 90 s timeout and kills it. Pre-seed app.ini instead of using the wizard.

Fatal permission denied writing app.ini. Forgejo generates JWT_SECRET and LFS_JWT_SECRET on first start and persists them into the config. With app.ini as root:git 640 that write fails and the service crash-loops. Supply all four secrets up front, or make the file writable by git. Supplying them keeps the config read-only and reproducible.

Webhooks to localhost are refused. Forgejo blocks loopback and private targets as SSRF protection, failing with deny '127.0.0.1'. Set ALLOWED_HOST_LIST = loopback — the narrowest value that works. Avoid *.

Prefer the system sshd over the built-in SSH server. With START_SSH_SERVER = true, a correctly registered ed25519 key was rejected with no log line at any level. START_SSH_SERVER = false with SSH_PORT = 22 uses OpenSSH and /home/git/.ssh/authorized_keys, which works. If sshd_config has an AllowUsers line, git must be added to it.

A post-receive hook is the wrong tool here. Under the standard install the repositories are owned by git with mode 750, so installing a custom hook needs root, and the hook then runs as git — which cannot write build output into another user's home. Use a webhook and a receiver running as the user that owns the web root.

Do not replace a bind-mounted directory. Docker pins a bind mount to the inode it saw at container start. Promoting a build with rm -rf + mv orphans the mount, and the container serves the old, unlinked copy while the build reports success. Use rsync -a --delete into the existing directory.

Use a read-only deploy key for the build. The machine running the build needs its own key. A per-repo deploy key without write access is enough, and is preferable to reusing a personal account key.

A systemd user service dies when the last session closes. Without lingering, systemctl --user units stop when the user logs out — so the webhook receiver was alive only while an SSH session happened to be open. Forgejo's delivery then failed with connection refused, and Forgejo does not retry: the push stayed unpublished, with a green push and a silent site. Enable lingering on any machine running a --user unit unattended:

sudo loginctl enable-linger <USER>
loginctl show-user <USER> -p Linger      # expect Linger=yes

Choosing --user units to avoid needing root is a real trade: it buys a service that stops when you log out. For anything in the publish path, either enable lingering or use a system unit running as that user.

systemctl enable --now does not restart a running service. After rotating the webhook secret, the unit file, pass and the Forgejo webhook all held the new value while the live process still ran with the old one from an hour earlier — so every delivery failed with bad signature, and it looked like a pass problem. is-active reported active throughout and the unit file read correctly; the only way to see it was the running process's own environment:

tr '\0' '\n' < /proc/$(systemctl --user show <unit> -p MainPID --value)/environ \
  | grep <VAR>

Use systemctl --user restart <unit> after changing a unit, not enable --now. Config files describe intent; a long-running process holds whatever it started with.

rsync --delete-excluded deletes excluded files from the destination. It is not "--delete, but safer about excludes" — it is the opposite. Used against a project directory whose excludes were .ssh/, .webhook-secret, src/ and static_hosting/, it destroyed the deploy key, the webhook secret and the served output in one command. Use plain --delete to prune files absent from the source, and --dry-run first whenever the destination holds anything not reproducible.

Check that a file exists, not just that git ignores it. A verification step that reports "ok: not tracked" is equally happy about a correctly ignored file and a file that has been deleted. When the check protects something irreplaceable, assert existence and permissions too.

Never load third-party scripts you do not control. This site loaded polyfill.io on every page for years, inherited from an old MathJax example. That domain changed hands in 2024 and began serving malware. Audit extra_javascript and extra_css for external hosts, and vendor what you can.

Security setup¶

Restore¶

Losing the VPS costs nothing but uptime. Rebuild a host, repeat step 10, then push any commit to trigger a fresh deploy.

Losing the intranet server:

1. Reinstall Forgejo (steps 1-2).

2. Restore the repository from the newest source — the mirror (step 13) or a bundle (step 14):

   git clone <COLD_STORAGE>/til-<DATE>.bundle til
   

3. Push it to the new Forgejo, re-add the deploy key and webhook.

4. Restore accounts from the forgejo dump archive, or recreate them by following this page.

Verify the chain once a year on a spare machine:

git clone <COLD_STORAGE>/til-<DATE>.bundle /tmp/restore-test
cd /tmp/restore-test
uv venv .venv && uv pip install --python .venv/bin/python -r requirements.in
.venv/bin/mkdocs build --strict