4 October 2026
So much news to cover!
The biggest change is that I’ve moved to Melbourne. I’ve been here for two months now, and living in vibrant Fitzroy. The other big change is that Bliptown has migrated to NetBSD. But more on the in a minute.
Not long after I arrived here I learnt that one of my favourite pieces of music, The Disintegration Loops, was to be staged in its entirety overnight at the beautiful Melbourne Museum. The show was sold out, which broke my heart a little because seeing this had been on my bucket list for years. But then a few days before the event I opened my email on a tram ride and saw a message offering a second round ticket, but I had to act now or the ticket would be offered to someone else. I did act, and shakily entered my bank card details to secure a ticket. The show was of course awe-inspiring, and I am now one checked bucket list item closer to death.
I was lucky enough to get a house sharing with only one other person, mere steps from the Fitzroy high street, and with enough room for a desk. I should be making better use of this, but I have spent most of my time at this desk working on Bliptown.
There is a theory called the Lindy Effect that posits the longer something has been around, the more likely it will stick around. The name comes from Lindy’s delicatessen in New York, which is ironically now closed. This effect surely applies to software. Until just recently Bliptown was running on OpenBSD, an operating system first released in 1996. This week I migrated servers, to an OS and infrastructure that has been around even longer; Bliptown is now running on NetBSD, which was first released in 1993, the same year as Jurassic Park. (OpenBSD is a descendent of NetBSD, and both are Unix systems.) Bliptown is written in Perl (est. 1987), and now the back-end file operations are handled by the “internet super server” inetd (first released in BSD 4.3 in 1986).
Migrating Bliptown from OpenBSD to NetBSD
During its first year, Bliptown has only suffered three instances of prolonged downtime (i.e. over a few mins), the most recent on Friday last week. All three times were due to the same piece of OpenBSD software — the reverse proxy relayd. OpenBSD has a reputation for unmatched code quality, so having the same part of the OS fail three times is concerning. I had also seen someone say that the code for relayd and httpd was languishing due to developer disinterest. I won’t go into furrther detail but it was enough to make me question OpenBSD’s reliability as a web server.
Something I liked about OpenBSD is its high level of integration — Perl, httpd, relayd, the pf firewall, acme-client and more are all part of the base system. This level of integration really makes system more than the sum of its parts. It’s reassuring to know that the same level of care and attention, by the same people, that went into making the OS a battle-ready tank, also went into say the reverse proxy. It makes it an easy decision to build your stack on components “in base.”
The problem for me was that when something in that base system failed, I had to understand that the system is really a combination of different sub-projects, each with their own level of care and attention, and some lagging. I guess what I lost was the idea of OpenBSD.
I’ve always liked NetBSD. It’s officially called “The NetBSD Project” as it operates primarily as a research project. The community are very nice and friendly, and as for longevity, SDF.org (est. 1987) have been running NetBSD since the turn of the century, and serves as a test bed for the project.
NetBSD doesn’t try to include everything in the base system. Sure it has Lua and Fortran, and NPF in the kernel, and a bare-bones HTTP server, but if you want a reverse proxy you install nginx, if you want an ACME client you install uacme. Same with your preferred sudo replacement (I’m using priv, est. 1997).
So by Friday night I had created a NetBSD server and had it running the Bliptown staging site. The Bliptown production site had migrated completely by Monday.
inetd + bliptown_helper
An important note about Bliptown is that it is a flat-file CMS, all web pages are files underneath, all users are regular system users, which means that all files are owned by different users. This is very different to how most CMS’s work. What this means is that in order for Bliptown to do file operations there are two options: operate on the files as root and then change ownership to the user, or (the safer option) forks a process that drops privileges to the user and then performs the file operations. This is the way Bliptown works.
Previously I had done this by running bliptown_helper daemon as root, then forking and dropping privileges for every file operation. With the server migration I thought it might be time to reasses my hand-rolled pre-forking daemon code and maybe hand the responsibility instead on software that has been battle tested for the last 40 years.
The idea to use inetd actually arose from my efforts to block malicious requests — commonly called “script kiddies.” The logs of any server connected to the internet are going to be flooded with requests to known vulnerable endpoints (usually of misconfigured WordPress sites). I wanted to immediately block any IP requesting one of these known endpoints.
I set an nginx location match to forward the IP of any such request to port 9666, where inetd would spawn a quick Perl script to add the IP to NPF’s block table.
The idea of running an “internet super-server” just to block script kiddies seemed like overkill, so I got the idea that maybe I could use it to also run bliptown_helper...
It turned out this was not only possible, but would mean I could jettison much of my code (goodbye maintenance burden) and no longer need to run a daemon as root.
Now I have inetd listening on port 9000 for a request from the Bliptown server, which then spawns bliptown_helper (now a simple script) and hands the connection over to it. It takes input from STDIN, drops privileges, performs the file operations, sends its output back to STDOUT, then dies.
If you’re thinking this sounds a little like CGI then you’d be right! It’s pretty close. However, the pitfall of CGI was that is required a single process per request, whereas Bliptown only needs a single process per user file operation.
That didn’t mean I wasn’t concerned about implementing a CGI-adjacent process model. It just meant a good time to improve the file operation code. Previously the helper would fork one process for one file operation. 99 filename changes? 99 processes. The pre-forking worker pool of 4 was pointless for such a (not uncommon) scenario. Now operations on multiple files are handled by one process.
Other News
Sick of GPG I wrote my own POSIX shell wrappers for OpenSSL public/private key encryption/decryption (and an associated Emacs mode for transparent file operation) called enc.
I’ve also revived my POSIX shell todo list manger from ten years ago, t_done. It now have an Emacs mode too.
Now I begin my mornings with a cup of coffee, my journal.md.enc in one Emacs window, and my t-done-mode todo.md in the other. It’s a good way to start the day.