“Back in my day we called our servers names”

When I started out, companies had 3 or 4 servers, maybe 5 if they were a bank. Everything was small and bespoke. I worked for a consulting firm, getting sent out to install a piece of software on a server, configure it, and leave. I’d do this 20 times at 20 different customers, same thing each time. Boring, but that was the job - no internet for remote management, and if you were lucky, a modem for emergencies instead of driving in at 2am.

Then servers multiplied. Big midrange boxes fell out of fashion, and suddenly people had tens of servers, with teams running down to the computer room to feed a CD-ROM into a machine and shiver while the app installed. Same process, just faster and more local. Numbers were still manageable, if annoying. I kept notes on what I’d installed and how, because things were inconsistent and installing anything usually meant a handful of strange steps only the vendor understood.

Servers got faster, and along came VMware with the promise you could shift the same OS image around hosts with vMotion. Servers you’d literally forgotten how to rebuild could live forever, perpetuating the beauty of bespoke. Meanwhile Linux was growing fast, especially at ISPs and internet-focused companies, and things started to change - turns out mailing a CD-ROM halfway around the world is a pain, and so is installing Apache on 20 machines by hand and hoping you got it right, remembering which values to swap in your stash of text file templates. “I manage those servers and know them by name” was starting to get old, fast.

Then an online bookstore started selling excess compute over the internet, in your browser. Wild idea, and intoxicating for companies watching powerpoint after powerpoint of shiny web interfaces launching servers with a click, trashing them when done, promising no more operations. What the sales pitch left out was that sizing, managing, configuring, and securing these machines was still impossible to do the old way. Turns out clouds aren’t datacenters either: if a machine dies, your VM dies with it, and you’re expected to just spin up another one. No more infinite-lifetime VM image. Add autoscaling and the growth that followed, and suddenly managing your fleet, whatever size it happened to be that hour, was critical. The tools that had started appearing as Linux matured grew up fast, and the whole paradigm shifted: ideally you never log into a host. It appears, gets configured, does its job until it’s no longer needed, and disappears.

When teams went all in on configuration management, the fleet stopped feeling like an unfathomable pile of hosts and started feeling like “storage,” “cache,” “web,” “app” - how many there were mattered mostly to whoever reconciled the AWS bill. Infrastructure was “described,” not configured, and it felt like nirvana. Not perfect, but definitely improving. The catch was that these systems tended to go one of two ways: incredibly complex, requiring people who could compile manifests against host facts in their heads because exceptions are the rule, or incomplete, managing “some” configuration but not all, leaving you squinting at a config wondering if it’s actually in there. Done well though, by some of the genuinely talented engineers I’ve had the privilege to learn from, a fully integrated provisioning, configuration, and deprovisioning system is a beautiful thing to see.

Thanks for letting me reminisce - I’ll get to the point. Configuration management at its most complete is designed to be rigid: things have to be just so, and defined constructs and frameworks are how you get there. I’ve tried rolling my own, always thinking “I don’t need all that complexity,” and a week later I’d remember why it’s needed. The barrier to entry was steep too - walking into a company with a Puppet repo that felt bigger than the Linux source code was daunting, and the skill required to engage safely had shot up. The benefits were clear, but so was the risk of a manifest resolving in ways you didn’t expect. For intermediate engineers it was a steep climb. A few different approaches emerged over the years, some simpler at first but more limiting as complexity grew, others you just had to have used before to get comfortable with.

Rigid frameworks - you could write flexible code within them, but the whole idea was distilling what needed to happen to one machine out of millions down to metadata, not a per-box list. Makes sense: why maintain a list per box? Add a user, and now you’re updating every list. Doesn’t scale.

And then AI happened. I tried it for a bunch of things, including seeing if it could take some of the mental gymnastics out of the old ways. As House MD used to say, everybody lies, especially AI. House meant omissions, leaving something out through shame. AI lies differently: it’ll do what you ask until it hits the slightest issue, then go into a frenzy trying to find any way to complete the task, no matter how unhinged. Give an agent a transient permission error writing a file, and don’t be surprised if it starts hacking together a C program to write bytes directly to the drive’s firmware interface. Getting it to write code for review, by itself or a human, was a solid use case. Letting it run commands on machines unsupervised was not.

Things are changing though. We’re learning to tame the toddler and getting real value from this wild thing. A big part of that, for me, is that the people who spent years thinking about configuration management back when Linux was still a “science experiment” to serious companies have found fun ways to blinker an AI agent into doing the right thing, every time, the same way. Give it a strongly typed, durable framework with audit and state, all the things agents crave, and there’s almost no guesswork left, which changes its behaviour completely. I’ve spent the last couple of months experimenting with the framework from the crew at swamp-club.com. Instead of the agent running commands on judgement, it writes automation code inside the framework, which then runs consistently like any traditional automation. You get the structured, audited change-making with clear success/failure feedback that production systems demand, plus the speed of an agent writing the code.

Funny thing is, in my early experiments with Swamp I kept modeling the classic ideas: “playbook,” “groups,” “inventory,” “manifests,” “consistency.” All the hallmarks of the systems built to centralise management for humans. Easy path to stay on, and it works, but it’s boring. AI is exciting, and experimenting is what made us the engineers we are - things might work, might not, but it’s worth the thought experiment. I read this piece on extension overlap this morning, plus a chat in the swamp Discord, both about the new economics of using existing code that doesn’t quite fit versus just cutting the code you actually need. Not here to litigate that decision, but it got me thinking again, which is always dangerous.

What if we ignore the technical and staff-scaling issues that drove us to config management tools in the first place, and come at this from the same angle as 30-odd years ago? What if the engineer who installed something on a server had a perfect memory of exactly what they did, and of everything done to that server since it booted? Perfect enough that if the server broke, they could pull out that list, run to a new box, and reapply it identically, checking each step along the way. Sure, that’s easy with one host - but what if every engineer in this huge elastic basement office was insanely good at keeping lists, and kept detailed notes on every app they deployed, including multi-host bootstrap steps? It’s the model of the past, back when servers were named after hobbits and had one PSU each. Madness to scale, clearly?

The swamp framework has the concept of “models,” similar to objects or classes. Models can have arbitrary methods and store data, and that data is versioned, with every update tracked. Method calls are logged and always available from the model, and success or failure is clear and programmatic, in a way agents can’t help but respect. Workflows tie multiple models together, using data and methods across them. Once written, all of this is just code - not AI guessing what to do, but reliable, dependable, thoroughly logged and versioned code and state, built into the framework so you don’t have to invent it yourself. The framework and its skills are structured well enough that prompting an agent with “add a method to this model to do X, expect Y back” will likely work first time, with tests, end-to-end validation, and framework standards met. It’s eye opening to watch. You get the agent’s flexibility where it’s valuable, but the commands that actually run are logged, stateful code. People are using it for all sorts of things, and it’s a lot of fun to experiment with.

Back to configuration. Here’s the mad thought I’ll leave you with. What if you could have a flow that was this, and just this? Swap “team” for agents or humans as you like - the fun part is that with the framework above, the story is almost interchangeable.

A request comes in, maybe from capacity planning, for more resource for service “data.” The infra team picks it up, checks the approvals, confirms there’s capacity in the right place, and runs the cloud API to do the work. They record every instance’s details in a magic document (data in Swamp) that lives with that host for its lifecycle. They mark the ticket done and pass it to the platform team with links to the instance data.

The platform team gets the ticket, checks which service and roles the capacity is for, and pulls up the document describing how to install and configure that service. They dispatch an engineer per new instance with a copy of the doc. Each engineer works through it, confirms every step and overall success, then marks their instance complete and updates the ticket. Along the way, they’ve recorded exactly what happened to every instance.

A new application gets onboarded. An engineer installs it in a sandbox, works out how it needs to be configured, same work you’d do writing a manifest. They iterate with outside review until it’s signed off, and it joins the stack of instructions for apps. When a request comes in to deploy that service, those instructions go back out to engineers, who use them on the right machines based on the request and host roles.

What I like about this naive picture is that it describes what happens naturally, but in the human world, things get out of hand fast as they grow, and once they do, there’s no going back. In the flow above, everything is durable, tracked, and documented. Nothing relies on inference to apply policy, it’s all explicit. We don’t need to feel bad about “someone” picking X servers out of the fleet and making a list, because that selection is part of the automation too, versioned like everything else.

I have no idea how this would actually hold up in production. I’ve run plenty of config management systems, but I don’t write them. What I do like is being able to think about this stuff in ways that would have seemed utterly bonkers a year ago.

I like bonkers.