An obstinate port-forwarding router


I had reason to port forward through my home router’s NAT to our bhyve box this weekend. There are some updates to Minecraft and Plex, and I wanted to do my Sunday maintenance from a coffee shop over SSH, like a gentleman.

Before I left, I opened the requisite port and enabled the port forward on the router. I tested it from an external IP and… nothing. I rebooted it to confirm the setting was correct and had been committed… still nothing. OpenSSH dutifully timed out each time.

I was in a hurry and couldn’t be bothered doing a port scan or any further troubleshooting, so I opened a remote SSH tunnel to my external jump box and left.

You know the saying that the cobbler’s son walks barefoot? Well we use the crappy home router our ISP gave us when we moved; albeit one I keep regularly patched. Its Wi-Fi range is more than sufficient for our tiny apartment, and I haven’t ever been bothered to replace it because if it ain’t broke, don’t fix it. But the fact I couldn’t get a console up or even do basic troubleshooting in a pinch may be enough to convince me otherwise.

In the old days I ran a router directly on my homelab server, and relegated IPoE duties of our Australian NBN connection to an external modem. I’m tempted to do this again, or get some MikroTik kit 🇱🇻. I like the idea of an all-in-one box for a modem, router, switch, and access point, especially given our limited space. Something small, quiet, and easy to configure sounds fantastic.

I just wish there was a single box that did everything and 2/5G Ethernet. Well, 10G Ethernet too, but that’s also only existed for two decades at this stage and is still too expensive for me to prioritise.

New-ish theme for 2022


A few eagle-eyed readers among you noticed my new blog theme. Truth be told it’s the same theme I wrote and used up to 2015, albeit with backported fixes for modern Hugo (cough)!

I realised that for all my talk about how I missed depth and textures in desktop interfaces, my site was also entirely flat. I think the border around each section makes them much clearer.

The “paper” methaphor extends to posts, the About Me section, and archives. Some of you think it should only be around the posts themselves, but I didn’t want large blocks of text floating in the background.

Rubi is also taking a break. Having an anime mascot annoys so many people on link aggregator sites which can only be a good thing, but Clara wants to redraw her in her current style.

Update: A special thanks to James for helping to troubleshoot a few issues with the dark mode version of the theme, much appreciated! You should see this reflected if you force a refresh.

Baristas are councillors


I don’t appreciate just how much baristas act like councillors. I wonder if they’re vetted by their managers for additional soft skills like this? They must be.

Every morning I go to my favourite local coffee shop, and the baristas are always talking with customers about their day, the weather, what problems they’re currently facing, how the long weekend wasn’t long enough. Some lend a sympathetic ear, others are actively involved in mediation or working out solutions in between running the coffee grinders, the espresso machines, the filter drips, and asking if they wanted almond milk instead of skim lactose free.

My cognitive ability limits my multitasking to unpacking dishwashers while tailing server logs and listening to a podcast, and those require no external human input! Imagine if I was trying to figure out how Alice can talk with her estranged son, or if I had to offer inspiring words of encouragement to Bob for his job interview.

I’m enough of a regular customer that they even remember conversations we had yesterday, and can tell when I’ve had a bad night. Sometimes that does more for my outlook and mood than the beverage they brewed for me.

I’ve talked about how much I respect retail workers, and the abuse they cop from terrible people. Baristas are up there too.

One memory from Clara’s and my last trip to New York was visiting Battery Park City and seeing all the bankers lining up inside a Starbucks. The sullen faces and grumbling voices could have easily brought the place down, had it not been for the cheerful baristas. I wonder how much of the modern world would collapse if not for their dedication?

Thinking aloud about web engagement


Last Wednesday I talked about the growing trend of superficial Linux distro reviews, both on YouTube and in thousands of cookie-cutter websites. Michael Dexter has lamented the fact that site wrapping software announcement with ads places higher in search results than the announcements themselves.

I have intimate experience with this. Software and writing I once published under my (now retired) alias would routinely get picked up and disseminated, usually without attribution. My primary blog here is now big and old enough that its harder to get away with this, but I still find people wrapping my words wholesale so they can get cents of ad revenue. I still continue to publish full articles in my RSS feeds, but I’m starting to understand why others only want to include summaries.

This is a fractal of what’s going on at a much larger scale, and more people are starting to notice. Sandy Maguire explores this in Reasonably Polymorphic, emphasis added:

Why would someone go through the hassle of making a website about something they weren’t interested in? In 2022, we have a resoundingly sad answer to that question: advertising. The primary purpose of the web today is “engagement,” which is Silicon Valley jargon for “how many ads can we push through someone’s optical nerve?” Under the purview of engagement, it makes sense to publish webpages on every topic imaginable, regardless of whether or not you know what you’re talking about.

As I said in that Linux desktop review post, I don’t think everyone is guilty of this. But it does go part of the way to explain why we’re seeing so many more of these mass-farmed videos and blogs, all saying broadly the same thing. Substance has been replaced with SEO (an abbreviation I’ve long thought a red herring), quality with quantity, and search engines like Google are, at best, enablers. There’s a reason everyone thinks search results aren’t as good now as they used to be.

Doc Searls touched on a similar point in mid-July when he used the evolution of television to discuss subscriptions:

Economically speaking, all that built-in smartness is about two things. One is facilitating subscriptions, and the other is spying on you for the advertising business.

It is interesting to think about. Sandy is correct that we’ll need to technically architect better search engines to separate the wheat from the chaff, but we’ll also have to address the incentives that lead people down this advertising, tracking, engagement stuff in the first place. People will continue to act this way as long as it makes financial sense. I keep going back to former Australian Prime Minister Paul Keating’s comment that markets move behaviour.

The web seems to be cleaving in two directions: rubbish, and paywalls. I’d guess there are just as many people sharing knowledge, experience, and ideas as ever before, but they’re being drowned out by an increasing tide of churnalism, theft, and low-effort spam. Sandy demonstrates as much when doing some basic geographic and health searches in the first linked post, some of which has already cost lives.

The necessity (or not) of paywalls and subscriptions is another massive topic, and one I may even be more sympathetic to than I first thought. I’d prefer to pay for a newspaper than have it track me, but that accepts the premise that those are the only two models in which the web (and society) can work, which I’m not yet willing to concede. It also has the pernicious side effect of locking out important information while spreading lies and misinformation.

The people I read are such a welcome reprieve from all this, and I love when I find someone new to add. That’s the other takeaway I want people to get from posts like this: if you’re reading my blog here and aren’t writing yourself, I think you should! Let’s tip the balance back towards good stuff again.

People who end comments with an ellipsis…


I’ve noticed comments that end with an ellipsis disproportionately skew towards trolls and condescending people. They often accompany a “you do know that” as well.

Errrrrr, you do realise that not everyone who does this is a troll…

I just added ...$ (three full stops) and …$ (an ellipsis) to my social media and chat clients that support regex filters. Instant improvement!

Links for week 31, 2022


We haven’t had a listpost for a while. Here’s some stuff I’ve read this week:

Should I Use a Carousel?

The answer may surprise you!

Stories from the Cavelab: Making planter boxes for Mona

It’s been so much fun watching Thomas’ latest house additions come together. 🌷

Amy Castor and David Gerard: Bitcoin mining in the crypto crash — the mining companies’ creative accounting

Rather than selling their bitcoins and tanking the market further, miners have been using them as collateral for loans. It’s going as well as you’d expect.

SoraNews24: JAL system makes air travel easier and lets you keep laptops, liquids in bag for security check

Most airport security is still theatre, but I’m all for making the process easier. Most flights from Australia to Tokyo go to Narita; here’s hoping the system is extended from Haneda when we’re (finally!) allowed to return.

Rubenerd: Links for the 31st week of 2022

Meta! Damn it, Facebook even ruined that word.

Nikkei Asia: Mahathir launches Malay political alliance with election in sights

Mahathir is Malaysia’s proverbial whack-a-mole. Without the support of Anwar Ibrahim's progressive coalition this time though, I don’t like his chances. My friends I made over there are more bemused than anything else.

Tom’s Hardware: Intel Arc Board Partner Ceasing Production

The discrete GPU market needs Intel’s competition, but production delays, reported driver issues, and now news like this makes me think it won’t be a substantial force anytime soon.

Feedback from @Tubsta, @Crosse3, Paul Traylor


I’m terrible at thanking people and sharing comments; maybe because I spend too much time on jerks! Here are some from the last few days which I appreciated. More coming soon.

Paul Traylor, on my social capital post:

I like your wording of this! I’ve often called myself a “social introvert”. I’m fine with giving presentations and talking to people, but I need my alone time to recharge. Social capital is another neat way to think about it.

This is perfect. It also helps to clarify the difference between socially anxious and introverted. I’m a bit of the former too, but I also enjoy talking with people, presenting, and hanging out… provided I also know there’s an exit and solitude on the other side. Paul’s blog has made the list :).

Jason Tubnor, whom I’ve had the pleasure of hanging out with at BSD conference and hope to again soon:

Well written article by @Rubenerd about the pitfalls of the modern web as we know it today. It was actually more resilient in the past but some network operators are doing their bit to keep it neutral and how it was intended to be.

Thanks for the kind words. I have to give a shout out to @phessler for inspiring me to run our own AS. Having full control of our hosts and network means we can do funky things with BGP to reduce the impact of DDoS without a big vendor. OpenBSD for the win there too!

Thanks! I was originally going to break out the ideas into their own posts, but I think the broad message of decentralisation, independence, and resilience were applicable to all of it. I also had comments about the increasing complexity and brittleness of modern Linux tooling too (and how the BSDs demonstrate it’s unnecessary), but decided to break that out elsewhere.

And finally Seth Wright:

I had/have cause to look at the guts of RSS feeds for work today, and I couldn’t think of a better example than to take a look at your blog’s feed. So nice and clean!

That means a lot! I try hard to generate clean pages, feeds, and headers, even though I know almost nobody notices or cares. Messy source code has always existed, but I do miss the day when people took as much pride in how their stuff was presented under the covers as above. Markup today is mostly generated by intermediate tools, CMSs, and JavaScript, then swept under the rug. It works the same, but the art is gone.

Superficial Linux distribution reviews


Google’s search results for software projects, technical announcements, and questions have steadily been reduced to a mountain of spam and duplicate content wrapped with ads. The remaining original material also tends to be hastily produced, with only a superficial exploration and grasp of the topic they discuss before moving onto the next thing.

This was inevitable, given how online incentives are structured. It’s also slowly creeping into other places, as nixCraft commented:

Most Linux distro reviews on YouTube are really only desktop environment reviews.

It’s true. Search for popular Linux distributions like Ubuntu or Fedora, and you’ll be shown around a desktop running in a virtual machine.

They rarely mention:

  • how they’re booted (UEFI, BIOS, laptops, desktops, etc)
  • how they’re installed, beyond mounting an ISO
  • physical hardware compatibility (does suspend/resume work?)
  • release schedules (rolling, snapshots, based on RHEL…)
  • package managers, or alternative processes
  • drivers (open source and binary blobs)
  • toolchains
  • documentation (official, wikis)
  • the community (where they are, friendly to newcomers?)
  • practical licencing considerations
  • does it run popular application XYZ
  • what the upgrade process looks like
  • and so on.

These define the ethos and practical applications of a distribution, and get to the interesting questions and points of difference people care about.

I don’t mean to criticise everyone here. For every churn factory producing low-quality clickbait (the Linux equivalent of those 5 Minute Craft lifehack videos), there are others who are breaking into the space and wanting to share their journey. We should encourage this! I just think with only minor tweaks and a few additional ideas, this content could be way more useful.

We need more authentic voices if we’re ever going to be heard over spammers.

When CDNs and DDoS vendors go offline


Everyone notices when a content delivery network or distributed denial of service protection vendor goes offline, because they take half the modern web with them. Much of the world’s Internet traffic is transmitted and delivered by just a handful of these vendors.

For a global network originally designed by the US military for resiliency, our current situation seems ridiculous. Why would everyone put their trust into a just a few players like this? Is it ignorance? Penny pinching? Bad design?

Content distribution

Prior to large streaming platforms, I’d argue BitTorrent was the most widespread, reliable, and cost effective way for most people to get video and audio online. The protocol meant that no one system shouldered the burden or responsibility of distributing content to every user; provided a full copy could be assembled, the network was resilient to outages. It’s an elegant, robust solution that worked for millions of people.

Media companies don’t like this. They want to retain:

  • ultimate control of the source files, such that local copies can’t (as easily) be pirated or redistributed, and can be injected with ads; and

  • how people access them, including the ability to limit and revoke if they need or want to.

Whether required contractually for licencing deals, or because they were spooked by the rampant piracy BitTorrent facilitated, streaming gave them a solution that satisfied those two criteria.

It’s a classic example of a meatspace limitation being imposed on digital architectures. Without the distributed advantage of protocols like BitTorrent, the client-server streaming model relies on massive servers and pipes to work, which few providers can deliver or maintain at that scale. The ultimate irony from an architecture perspective is that these media companies now routinely sneakernet drives to large ISPs in different countries to help them locally cache and deliver content.

CDNs apply that principle to every site, but for economic reasons. You don’t need a physical point-of-presence on every continent to deliver your assets. Performance is a key metric people use to validate your service, and modern web users have a low tolerance for latency and lag. Large data centre providers and cloud platforms will then “peer” with these CDNs, such that traffic can operate directly between them. It’s a reality that if you operate outside a CDN (as I do for all my stuff), the perception of your site’s performance will likely suffer depending on where they are.

The turnkey, low maintenance, and relatively cost effective solution then locks platforms in, which means they’ll probably continue using them even when they’re at the scale and size that they probably could roll their own global distribution system. If you’re a beancounter allocating resources between reinventing the wheel or adding features, which would you choose?

The Internet was designed to be a robust network of peers that can route around damage. This model doesn’t break OSI, but its lopsided nature introduces brittleness, which is all too often on public display.

What about DDoS attack vendors?

More of the general public now know names like Cloudflare and Fastly from their proxied forwarding pages (“this site is protected by XYZ”), and from when tired NOC engineers go to social media to explain that their site is offline because of such services.

Unfortunately, the architecture of the modern web makes their use all but necessary.

The Internet was designed with an implicit level of trust between nodes. It assumed people wouldn’t spoof their IP headers, read your cleartext communications, or perform too much mischief. Amplification attacks and rentable botnets now make DDoS attacks a regular fact of modern sysadmin life, and depressingly easy and affordable to perform.

There are no effective mitigations. You can’t feasibly block the source of a distributed botnet attack. Paying protection money, or giving into ransom demands by DDoS attackers only emboldens them, and may be illegal. In layman’s terms, the only way to survive is to hope you have a bigger pipe than the attacker can saturate. And again, there aren’t many choices at that scale.

Like CDNs, the concern is that sysadmins and their managers may have become complacent in their use. It’s tempting to throw a DDoS protection provider in front of your server and call it a day, but you’ve arguably substituted one problem (attacks) with another (potential brittleness).

Where we go from here

That’s the open question!

I’d love to see a reversal of the consolidation we’ve seen, which will only happen if people appreciate placing eggs in fewer baskets is a problem. I work at a small provider that’s trying to do this, with local “clouds” and kit in people’s own facilities rather than centralised. People like Jason Tubnor maintain and operate a fleet of bespoke servers without the need for any cloud services; check out his awesome blog and BSD talks. I know other people who have feet in both cloud and bare metal to spread risk while still having burstable capacity.

We need diversity again, and there’s big money behind convincing people that’s either undesirable, expensive, or impractical.

The legislative cat is out of the bag for DDoS attacks; cryptocurrency has made anonymous threats and payments feasible while they boil the planet for their shitcoins (pardon the French). The only way I see we’ll bring them under control, at least with our current protocol suite, is doubling down on endpoint security. It was entirely preventable and completely ridiculous that the so-called Internet of Things weren’t designed with security in mind, and we’re now paying the price.

It’s a controversial position, but I’m coming around to thinking ISPs should be more involved in scanning for vulnerable devices and notifying customers. Diffusion of responsibility is a real problem that will require the collective effort of everyone on the Internet.

The Internet and WWW survived far longer than even its designers expected, but I think we’re at a junction now. I hope there are enough of us who still care about this stuff.

The limits of personal social capital


I like to consider myself a high-functioning introvert. I do a day job and volunteer some of my evenings, much of which involves talking to people in the real world. Many of these conversations I enjoy, especially when I get to hear how people build systems and processes, and what advise and documentation I can give or build.

The corollary of “if it’s not documented, it doesn’t exist” means I get to bring things into existence! But I digress.

Despite this, there’s a finite amount of social capital I can spend on such things a day before, as Clara puts it, I retreat into my shell.

We all have this to an extent, even those who identify as extroverts. The most outgoing person might feign being exited about joining two dozen calls a day, but there’s a reason why corporate retreats and onsens exist.

Realising one has limited personal social capital does help to frame priorities. There’s a real opportunity cost to every nonsense conversation or person, especially those who don’t act in good faith. I want to spend this time on people I care about.