eden is the Proxmox server that runs my personal infrastructure: DNS filtering, remote access over Tailscale, a reverse proxy, notifications and a few apps. In June 2026 I replaced the shell scripts that built its containers with OpenTofu and Ansible. Then I destroyed the production containers and recreated them from the repository, and the first recreate came up broken in a way the existing system had never shown.
Why I rebuilt it from declarative code
A container configured by hand is a pet. Its configuration lives only inside it, and if it dies I lose the password, the block lists, the client groups and every adjustment I made. My first answer was a create.sh and a provision.sh per container. That was versioned and repeatable, but it was not declarative.
Now the repository is the source of truth. Code is reviewed before anything touches production, and any change made by hand over SSH goes into a log file and then back into the code.
How it is put together
Everything runs in LXC containers on one node with ZFS storage. Two tools own two layers, and they never own the same state:
- OpenTofu, with the
bpg/proxmoxprovider, owns resources: that a container exists, its CPU, memory and disk, its network and the SSH key injected at creation. - Ansible owns everything at operating-system level, inside the guests and on the host. There are 10 roles.
The whole rebuild is two commands:
cd tofu && tofu apply
cd ../ansible && ansible-playbook -i inventory/hosts.yml site.ymlOne exception shaped the boundary. Proxmox refuses device passthrough through an API token (HTTP 403, root only), so the /dev/net/tun device that Tailscale needs is set by an Ansible role on the host. The alternative was putting the root password in the provider.
A change is tried first on throwaway containers in a separate test workspace, and a role is done when a second run reports changed=0.
What checks a change before it reaches the server
make gates runs format and validation for OpenTofu, tflint, ansible-lint, gitleaks and zizmor. A pre-commit hook runs it locally and CI runs it again. Renovate opens dependency updates with a minimum release age, and I merge them by hand.
Secrets are in an Ansible Vault file that is committed encrypted. Before the first push I ran gitleaks and trufflehog over all 37 commits and the working tree, and everything they flagged was a placeholder or an example string.
During an incident I skip the gates on purpose, apply from my laptop and record it afterwards, because a gate that blocks an incident gets disabled for good after the first bad night.
What breaks, and how I find out
A backup runs on the host every night. It dumps every container to a local ZFS dataset with vzdump, then sends the dumps offsite with restic, which keeps daily, weekly and monthly snapshots. It runs on the host so that it does not depend on any guest being alive.
Two signals come out of each run. The first is a push notification. The second is a ping to an external service that expects one every day and emails me when it does not arrive. Nothing inside the server can report that the server is dead. If the disk, the power or the uplink goes, the missing ping is the only signal left.
I turned down one monitor. An external HTTP probe aimed at a hostname that sits behind a login page served at the edge never reaches the server. It gets the login page, and it stays green with the server dead.
What destroying it taught me
When I adopted the running containers into the code, the role that sets up the tun device was a no-op against them, because the configuration lines were already there. Then I destroyed the production containers and recreated them from code. Tailscale was down in every container that runs it.
OpenTofu creates a container already running, without the tun device. Ansible wrote the device lines into the container’s configuration, but those lines only apply when the container next starts, and nothing restarted it. On the old containers the lines were already live, which is why the same role had looked correct.
The fix is in the host role. It checks each container for a live /dev/net/tun, reboots the ones that lack it, and runs before the play that installs Tailscale.
The project has been in maintenance since August 2026. If your own infrastructure code has only ever been applied to machines that already existed, its create path has never run. Destroy something small and recreate it from the code before you need to.