Managing Dozens of Linux Machines

10 years ago I only had a couple Linux machines. A network management machine at home and one I setup at my last job. Admittedly, I did a poor job managing them. I would occasionally install updates when I’d think of it but mostly they would only be installed when I was installing a new package/program.

As I detailed in my Server 2.0 post, I decided to virtualize my Linux servers to make them portable and hardware agnostic as well as enabling easy complete backups. Backing up and maintaining even a few Linux machines grew tiresome and I began by automating the snapshots and backups in Virtualbox. Last year as I pulled some old Raspberry Pis out of retirement I quickly doubled the systems I was using. This number then began to grow exponentially as I started buying single board computers (SBCs) to use as OOB console servers.

Seemingly overnight maintaining these systems became unmanageable. I had begun using Ansible for network automation last summer and it occurred to me I had sort of done that backwards…I could have more easily learned Ansible by using it to automate Linux server management. Fortunately that meant it was fairly trivial to adapt its use.

I started with a playbook to perform apt updates:

Next up were a couple playbooks to automate snapshot creation and deletion done in vSphere before and after apt updates respectively:

I have since used Ansible to automate various other changes. This final example lowers the delay I previously added to the ser2net systemctl service script to lower the delay from 30 seconds to 10. It then reloads the systemctl daemon so that the changes are added when it reloads the ser2net service at the end of the playbook.