Rubenerd

Last week I wrote about the latest flareup about shingled magnetic recording. SMR is useful for making more efficient use of platter space and therefore reducing costs, but at a performance penalty for random writes. This was exacerbated by drive manufacturers not labelling SMR drives as such, leading to the drives being used in environments for which SMR was not suited like RAIDs and ZFS pools. And by deliberately obfuscating the use of SMR in drive reporting, even file systems or RAID cards that could handle the inconsistent and slow performance of SMR wouldn’t be aware to do so.

I was reading the Synology knowledge base about SMR, and they had an interesting point:

… when the data on a [Shingled Magnetic Recording] drive is edited or overwritten, the write head will not overwrite the data onto the existing magnetic track. Instead, the new data will be written onto an empty area of the disk, while the original track with the old data will temporarily remain. When the HDD becomes idle, it will enter a reorganization mode where the old bits of data on the original track will be erased and made fully available for future use.

This reorganization mode must occur to completely delete tracks, making the idle time essential for an SMR drive. If an SMR drive is being used heavily for read and writes, it will not have enough time to reorganize the magnetic tracks, causing the tracks with the old data to remain temporarily. As a result, the SMR drive may need to write new data and reorganize the old track at the same time, resulting in a negative impact on the overall read/write performance. To combat this, SMR drive manufacturers have developed firmware that optimizes the read/write performance while data is being overwritten.

My first thought was to what extent the tech would affect durability, when rumblings about hidden SMR came out in 2018. Without hard numbers all I could do was speculate, but it stands to reason that a drive that constantly has to perform data gymnastics as described above would have more physical wear than an equivalent capacity drive using conventional recording. At best this would only be partly negated with added intelligence to perform this reorganisation more efficiently, and that assumes the drive has sufficient idle time to do so.

Even if somehow these drives existed in a parallel serial universe where these additional writes physically came for free, the point Synology raised above leads to a larger and more scary question about data integrity. File systems and RAIDs depend on drives truthfully reporting when they’ve written data. As soon as a cache is introduced, or another layer of abstraction, the drive’s firmware is lying to the OS or RAID controller about the current state of the drive. If you lose power between when the firmware reports data is written and when it actually has, it’s gone.

(Perhaps good SMRs have a persistent cache like hybrid drives, or they depend on a constant source of power to be safely deployed. Which, again, makes hiding the fact they’re SMRs all that more dangerous).

Trustworthy copy-on-write file systems like ZFS are better able to handle this; as are RAID controllers and drivers designed to be SMR-aware. But then we get back to that original issue of drives obfuscating the fact they’re SMR, and people buying them assuming they’re not.

The only reasonable way to regain trust and confidence in these drives is for them to report they’re SMR: in marketing, in technical specifications, and from the firmware itself. Lists on a blog post that may or may not be maintained, and that people may still otherwise be unaware of, are grossly insufficient.

I’ve heard it said the cobbler’s children walk barefoot. While posessing the qualities of a famed financial investment strategy, it speaks to how we generally put more effort into things for others than ourselves; at least in business.

The HP Microserver I share with Clara is a modest affair compared to what we run at work. It has six spinning rust drives and two SSDs which are ZFS-mirrored; not even in a RAID 10 equivalent. This is underlaid with GELI for encryption, and served to our Macs with Netatalk over gigabit Ethernet with jumbo frames.

I tried switching to Samba once, but Netatalk is marginally faster for what we do, and is way more compatible with the software we use, even with the recent additions to Samba 4 and the fact AFS is depracated. I also run NFS to serve my FreeBSD Panasonic laptop, because they basically only need to backup text. But I digress.

So imagine my surprise and irritation when our AFP shares suddenly went to absolute crap last night. The Mac Finder struggles to list directories on NASs at the best of times, but I could barely load anything. Worse, the Finder would hang when loading folders with more than a dozen items, which required a killall on the client side. I do still like macOS after all these years, but the fact a network share can bring down everything else with it reminds me of the dark days before process isolation.

This sudden drop in performance has only happened for me in the home environment a handful of times, but was usually an immediate sign one of the drives in a ZFS array was about to kark it. But a zpool status returned optimal for the pool in question:

 state: ONLINE
  scan: scrub repaired 0 in 1 days 02:26:01 with 0 errors on...
config:
	NAME              STATE     READ WRITE CKSUM
	zroot             ONLINE       0     0     0
	  mirror-0        ONLINE       0     0     0
	    gpt/ssd0.eli  ONLINE       0     0     0
	    gpt/ssd1.eli  ONLINE       0     0     0

Yes I’m running these SSDs over the USB3 headers. Yes I know that’s bad, and that I should feel bad. Cobblers aside, they also say necessity is the mother of invention.

So my next check was logs. And the issue became orders of magnitude clearer:

# ls alh /var/log
==> -rw-r--r-- 1 root wheel 500G May 7 22:00 afpd.log

Alamak! I suppose this is a credit and testament to the performance of ZFS’s inline lz4 compression: this is a 120 GiB boot SSD, and it’d been written to continuously. I wonder how much I shortened the lifespan of these devices?

I cleared the log, and the performance immediately returned. Next step is to figure out how that log got so massive in the first place, and set a quota for the log dataset.

Podcast: Play in new window | Download

29:37 – Join Ruben for a late-night wander around the abandoned Sydney suburb of Chatswood during these lockdown adventures. Topics include 100 days of COVID-19, stairwall-derived exercise, Australia and New Zealand’s potential new travel bubble, finding coffee shops, spooky silhouettes in the night, a Liverpool FC barber, JoAnne Hook paintings, active voice, saying “you too!” to retail staff, and the uncanny valley as it pertains to… sheep‽ Recorded Monday 4th May 2020.

Recorded in Sydney, Australia. Licence for this track: Creative Commons Attribution 3.0. Attribution: Ruben Schade.

Released May 2020 on The Overnightscape Underground, an Internet talk radio channel focusing on a freeform monologue style, with diverse and fascinating hosts; this one notwithstanding.

Subscribe with iTunes, Pocket Casts, Overcast or add this feed to your podcast client.

Is this the first time I’ve ever shared an AKB48 song here, in fifteen years? Am I publishing a Music Monday on a Tuesday again? The answers to both questions may surprise you!

The DJ’s introduction to this unashamedly bubbly song from 2013 seems oddly prophetic and well-suited to our current times. I hope we’re all dancing around in our places of work and in the street again soon, and that I’m even 10% as cool as that cab driver when I’m his age :)

I should have titled this with mostly-hidden so I could have had three consecutive hyphens.

Patrick Kennedy has weighed in on the hidden shingled recording debate over on STH. It was an assertive but balanced summary of what’s going on. His conclusion hits the nail on the head, emphasis added:

For those who think it is OK to obfuscate vital product information to a lower-cost segment of customers, I offer this: just because someone or some business does not have the opportunity or means to purchase a higher-end drive, does not mean we should deprive them of an opportunity to make informed purchasing decisions to protect their livelihoods and memories.

Michael Dexter commented at AsiaBSDCon 2019 that all it takes is for a file system or drive to shred your data once, and your trust in it permanently gone. Those are the stakes, whether it’s OpenZFS on a new OS architecture, or a class of hard drives.

This is what I found most disturbing about this obfuscation. People are trusting their critical data to these drives, from sysadmins in data centres to home users with their NASs. Arguing that people get what they pay for, as mentioned in these companies’ PR announcements, assumes consumers were in possession of all the facts when deciding on the cheap option. This clearly wasn’t the case.

Others in the comments suggested further stratification of the storage market is the only real endgame here. I agree; the only reasonable outcome would be drive manufacturers offering cheaper SMR drives, and charging a premium for ones guaranteed not to use it.

I do empathise with their position. We’re reaching the physical limits of storage miniaturisation with this tech. SSDs have kicked them out of most performance applications, and are nipping at their feet at lower capacities. As Patrick says:

Let us be clear, in 2020, there are a few reasons one buys a 2TB to 6TB hard drive. The biggest, by far, is cost. There are consumers and businesses who have tight budgets and cannot afford higher capacity hard drives for their storage. Storage density is a big deal and there are significant costs to just connect a hard drive to a NAS or PC. Also, fewer drives mean less vibration, noise, power consumption, and heat.

There are significant benefits to building arrays with more drives if you’re hosting high and variable IO workloads like virtualisation, but for NASs and bulk storage Patrick hits the nail on the head. Though I suspect eventually SSDs will take that away too, once the cost per gigabyte makes spinning drives untenable.

There are some creative and fun ways people secure SSH, or any other service running on an open port. At a previous company they sent a specially-crafted UDP packet containing a nonce to knock on a server, which would temporarily add their IP address to the firewall and permit them access.

My favourite way, in part because it’s so easy, is to lock down access to specific hard-coded IPs or ranges. I have static IPs at home and work, so this is easy. Otherwise you could enable it just for your ISP’s range, such as with ipfw.

The primary risk of this approach is you can lock yourself out if one or more of your IP addresses change. If you use a good cloud provider like an OrionVM reseller (nudge cough) you can use the out-of-band console to connect and add another IP. For other clouds, you’d need to implement something more like above.

Locking a firewall port to specific ranges or addresses for SSH isn’t foolproof, but it’s one more layer of security which is always welcome. Above all else, it drastically cuts down on drive-by auth spam.

I do try and see the good in things, or what we can do to make the best of a bad situation; it helps me get through it all. Maybe for some of you, reading it might also help.

Here are some things I hope come out of this, in no order of importance except for the first:

• We acknowledge that free, universal, and accessible healthcare isn’t just ethically obvious, it’s required for biosecurity and the viability of economies.

• We realise tax cuts aren’t an automatic good. Taxation isn’t theft: sickness, illiteracy, and lack of social mobility are: they steal people’s futures and opportunities, regardless of merit or any other metric we use to superficially judge people’s worthiness.

• We continue to see fewer people leaving public bathrooms without washing their hands!

• We stop demonising the unemployed as lazy and entitled for being on social security, while ignoring billions of dollars of tax loopholes under the reasoning that the latter are within the spirit letter of the law, but the former aren’t. Governments are scrambling to fund people and companies through these lockdowns, stating that it’s tough right now. It was tough for others before as well, and the system is still needlessly punitive.

• I get a higher-capacity coffee machine.

• Delivery and supermarket retail workers, many of whom are casually or part-time employed, are recognised alongside medical staff for their essential and valiant efforts when all this is over.

• Companies and city planners realise that people are effective working from where they need to, and the notion of peak hour, packed trains, and traffic will be seen for the communal silliness it is. We map this curve-flattening mental model to transport, to make more efficient use of what we have.

• Telecommunications infrastructure will finally be seen as essential as roads, plumbing, and the power grid. This would include right to access, net neutrality, and either government intervention or operation in the case of natural monopolies.

• Domestically in Australia, the far-right acknowledge that Wayne Swan and Kevin Rudd were right about stimulus during the global financial crisis, and that Malcolm Turnbull and Tony Abbott were unequivocally wrong about the necessity and capacity of the National Broadband Network. Pigs will also be acknowledged as having taken flight.

• Domestically in Singapore, foreign workers are given access to better accommodation and working conditions. If it takes legislation, so be it.

• And finally, we continue to reduce the overexposure and tedious obsession with celebrities and celebrity culture, and spend more time making and watching/listening to independent media.

I’m nothing if not a cautious optimist. I’d be overjoyed if we just got two things off that list.

The Guardian have a quiz to test your international flag knowledge if you’re after something fun to try this weekend. I got full marks, even despite being thrown by all the incorrect aspect ratios in their lead image grrrr!

I’ve always loved reading about flags and their history. I’d always bring one back from countries we’d travelled to; at one point I had them pinned to my bedroom ceiling which was delightfully colourful.

It’s tough picking my absolute favourite, but I have a few candidates below. See if you can guess them before clicking through any of them! It’s interesting that all my favourites thus far are sub-national entities.

There was a Twitter thread about FreeBSD earlier this year where someone was shocked to realise a FreeBSD user had a Mac laptop, and other ran Windows at work. The Twitterer and new FreeBSD user suggested that FreeBSD people should only ever use FreeBSD. I’m paraphrasing, but the responses were that gatekeeping isn’t helpful to the project.

I’ve had one foot in the Apple camp since the mid-1990s. Over the years I made the switch from Windows to Linux and the BSDs, but I’ve always had a Mac of some description as well. Despite Apple’s best efforts of late, there’s still no better desktop environment for most of my work and personal projects. Same for the iPhone.

What stopped me jumping all in with Apple:

• Overwhelmingly the number one reason: Apple can’t deliver a trustworthy file system, even with Core Data and APFS. Metadata verification is simply insufficient for any data you care about, especially family photos. Ditto for backups, which will rot in ways you don’t expect until it’s too late.

• They’ve abandoned the server space in any practical sense. Which is a shame; especially in the PowerPC days they were one of the few remaining suppliers that offered a supported UNIX on something other than x86. Damn I miss Sun something fierce.

• I like computers, and playing with a range of them. And there was always the nagging feeling in the back of my mind not to become dependent on a single vendor; thus even most of the software I run on Macs is cross-platform and, more importantly, saved to open file formats.

• Their GPUs suck. Which is bizarre, because their laptop displays still leave everyone else so far in the dust it’s not even a contest. It blows my mind how awful most PC displays still are: 1.5x HiDPI if you’re lucky, are you kidding me? I’ll stop now!

• They’ve given up on ultraportable laptops after retiring the 11-inch MacBook Air and the 12-inch MacBook. The new MacBook Airs weigh more than a kilo once more, and they’re made thin by stretching them out too wide for my small shoulder bags.

FreeBSD fills all these needs nicely:

• FreeBSD with netatalk4, ZFS, and GELI makes a reliable, robust, fast, and trustworthy file server stack. My experience has been that it consistently outperforms SMB and NFS to Macs, and marginally more with AFS on Debian, even though Apple lists AFS as deprecated; go figure. I squirm thinking about all the storage hosted on third-party NAS systems that have rudimentary integrity checks, if any at all.

• A small home FreeBSD server is affordable, fast, and easy to administer for sysadmins. And if you’re not one, there’s FreeNAS. I say affordable not free, because you really should donate to the Foundation if you find the OS useful.

• My FreeBSD workstation with a decent NVIDIA card can be dual-booted into Windows 10 to renew my disgust for that client OS, and for running the precious few games I care about.

• FreeBSD on my tiny Japanese Panasonic Let’s Note makes for a fantastic ultraportable with a great HiDPI screen, better keyboard that Apple ships, and all-day battery. On the go I mostly only need an SSH prompt, a good text editor, my blog static site generator, a PDF viewer to do presentations, and a browser I can run the current conference software flavour of the month in. My MacBook Pro is there if I need something beefier, though it rarely leaves my desk now.

I only feel comfortable using Macs now because of FreeBSD. And a little NetBSD still too, but that’s for another post.