For your consideration below, a sample of songs from my iTunes playlist that begin with “Here Comes The…”. I had no idea there were this many.
In moves that should bemuse or terrify us (I’m undecided which), Australians have been subjugated to empty justifications, misunderstandings and general denial over this government’s plan for mandatory data retention.
Earlier this week, intrepid reporter and awesome guy Josh Taylor quoted the Attorney General as saying:
“The question of data retention is under active consideration by the government. I might point out to you as recently as yesterday, the House of Commons passed a new data retention statute. This is very much the way in which western nations are going,” he said.
And I “might point out” that if your friends all jumped off a cliff..?
At the risk of preaching to this rather attractive choir that have assembled to read this post, there are serious ethical and technical concerns here.
Mass metadata surveillance would require an inordinate amount of storage and processing power, as well as the ability to archive and mine it in a meaningful way. The steep financial and performance penalties involved would be borne by the users of services, rendering Australia’s IT sector even less competitive.
Such a system represents a huge security and privacy risk, both by blackhats gaining access to this wealth of delicious data, as well as internal staff abusing it.
Ethically and legally, the idea of storing everyone’s information, regardless of whether they’re under suspicion, is a breathtaking overreach of government authority. If someone is to be investigated, a warrant should be served. It’s almost as if we figured this out before.
And finally, the activity would almost certainly backfire. Those who perform illegal activity, if they’re not already doing so, would merely cover their tracks with VPNs, vegetable flavoured routers and a ton of other tools. I assert it would also be trivial to pollute the pool of data, as well as plausibly deny or explain away connections. I have an open WiFi hotspot, no password on my computers and do AirBnB behind a NAT, how can you be sure it was me?
My favourite though would have to be the pathetic excuse that we shouldn’t worry about metadata collection, because it doesn’t invade our privacy and doesn’t reveal much about us. If so, why collect it?
There are far more technical and ethical issues at play here, but I’m just too tired. Maybe its time to do some more keysmash or random dictionary GET requests to random servers and search engines again.
An email from earlier today:
We would like you to be the first to know of an exciting update to Ohloh.net. This week, Ohloh will be changing its name to the Black Duck Open Hub.
What is it with F/OSS communities and branding?
With a little spare time, my girlfriend Clara and I decided to see why my venerable Antec A300-encased Core 2 Duo Xen tower had trouble rebooting. In the process, we managed to clean her up, reorganise her internals, tie down cables and generally have a fun and terribly nerdy Saturday evening!
"Her" referring to the computer, not Clara.
Since our last move, my secondary tower has had trouble with rebooting. I could almost hear my Mac Pro chucking to herself each time I had to reach to the back of this DIY box to unplug her. In this case it wasn't a three finger salute, as much as it was a scraped knee and soft muttering.
Originally designed as an outlet for DIY fun since going to the Mac, this tower has gradually assumed more responsibilities once I started getting into KVM and, more recently, Xen. Like all good open source projects, this tower had no plan whatsoever, but evolved organically as I sourced cheap parts to fit exactly what I needed.
In keeping with my anime girl naming scheme, she became Yuki. An understated machine with more power than her outward appearance would suggest. 2008 Ruben thought that was clever.
Until last night, she would shut down just fine, but soft or hard reboots would result in a blank screen, no reassuring whirring of her hard drives and misleading LED indicators. The only viable remediation was flicking the physical switch on the PSU, waiting ten seconds, then booting her again.
That should have sounded warning bells that the PSU was shot, but like any good self diagnosing gentleman who assumes they have some horrible condition when they're manfluing, I assumed I had a shot motherboard or worse. I reseated the soft power switch cable in its pins, tested the RAM in other machines, even went out and bought a replacement graphics card.
(To be fair, the old one was noisy, and I was able to source a fanless one. For a server I’d even just serial console into if I could, this is fine).
So over the course of the evening, Clara and I decided to pull apart some other half working, accumulated desktops into one fully functioning unit. Turns out our mothball collecting, CD ripping tower had a far newer PSU, so on a hunch we decided to try it out.
One transplant later, and Yuki is completely back in business. She can shutdown, soft reboot, hard reboot and sleep like the best of them. I was also able to show Clara some of the finer points of DIY computer hardware, gleaned from years of introverted teenage tinkering that often resulted in serious damage and wasted money.
Moral of the story, if your machine has power troubles, shock of horrors, it may be the power supply. And if so, use the opportunity to upgrade the machine and generally have some nerdy fun :).
For a few months now, Clara’s and my VPS has been running nginx. Our sites are by no means high traffic, but we’ve benefited greatly from nginx’s reduced memory requirements compared to Apache. Some of the configuration still confuses me, and I doubt I’ll be as competent at it as I am with Apache for a while, but I’m learning.
To state your character set in your HTTP headers, use this in your relevant http, server or location section of your nginx.conf:
charset UTF-8;
And to stop advertising your nginx version:
server_token off;
And if broad IE compatibility is still important, you can save yourself some invalid HTML5 meta attributes by adding this header:
add_header X-UA-Compatible 'IE=Edge';
In an article for the Australian Business IT site, Dave Stevenson details some Dropbox tricks. Most of it is the usual stuff about its operation, how you’re able to increase your account space, and sharing documents between users.
Under security, we get the first tales of caution:
Dropbox is terrifically convenient for sharing personal documents and professional projects – but is it safe for sharing sensitive documents?
He goes on to discuss Dropbox’s previous security breaches, their use of 256-bit AES SSL, and their new (in 2012) two factor authentication, which he explains as:
a new code is created for you every time you log in. The idea is that even if someone gets hold of your password, they won’t be able to access your Dropbox without the code.
Smooth sailing so far. But then we get into the specifics of account access, versus file access.
If you’re running a Dropbox for a client, you should still tread carefully. By default, Dropbox’s employees can’t access the contents of your files, but they can read file metadata (such as filenames, file sizes and the EXIF data attached to images).
Unfortunately, that’s not true. While there may be some business logic or rules preventing employees from accessing your files, there’s no technical reason why others in the company couldn’t. In transit, your files are protected with TLS (SSL), but after this they’re stored in the clear. The only way you can be sure people can’t access your files is by performing client–side encryption, either on individual files or disk images.
(It’s not enough to do server-site encyrption, as the server would necessarily have to keep the key. This means it could be handed over to anyone, thereby rendering your encryption moot).
I admit, I’ve been letting pedants have it of late, and now I’m being picky about a journalists phrasing. Still, this is a critically important distinction for user’s privacy that I’m surprised doesn’t get much coverage. It has real world implications for people.
I use Dropbox for notes, draft blog posts, and other material I wouldn’t mind people seeing. Its utility more than pays for the sacrifice I make for privacy. But be under no illusion that your data can’t be viewed.
(Since writing this, much of the syntax in Xen 4.4 is different. Debian Wheezy still ships with the older versions, but Ubuntu users will likely need to refactor some of this).
Since first running Connectix Virtual PC on my childhood blueberry iMac DV, I've been unhealthily obsessed with virtualisation. Long time readers of Rubenerd would have seen me post about the first beta releases of Parallels Desktop in 2006, VMware's consumer and enterprise fare, the versatile QEMU and the nostalgic DOSBox and ScummVM.
Now that you've cleared the pointless introduction, this post will be exploring Xen, the system that powers much of the world's cloud infrastructure, and the more discerning Linux and NetBSD users' personal virtual machine collections.
Xen is a bare—metal hypervisor that runs virtual machines. Once you've provided it with a configuration file, the system spins up your virtual machine which (by default) you can access with a serial console, VNC or a few other remote access protocols. Of course, once you've enabled an SSH daemon and networking in your guests, you can access through that too.
Broadly speaking, your Xen host is referred to as dom0, for domain 0. Guests are referred to as domU, and can be started either fully virtualisaed with HVM, or paravirtualised (PV) if the guest OS supports it. What's the friggen difference? More on that shortly.
Once you've decided to try Xen, the next step is finding a hypervisor-compatible OS. Those who know me wouldn't be surprised to know I first tried Xen on NetBSD; of all the (albeit limited) non-Linux options, NetBSD's dom0 Xen support is supurb. For most of you though, I'm assuming you'll want to use Linux. Debian is what we use at the office and where I have most Xen experience, so that's what we'll be looking at here.
At a bare minimum, you'll also need a system with Intel VT-x or AMD-v support. Most decent "modern" systems have these, but this website is a great resource for checking what your CPU supports. For full hardware assisted virtualisation (HVM), you can check Intel VMX or AMD XVM support by searching this:
egrep '(vmx|svm)' /proc/cpuinfo
Once you have your OS of choice installed, these are roughly the steps to get started quickly:
As a tinkerer myself, I appreciate the urge to try something quickly. This is arguably the bare minimum you'll need to do to get started; you'll want to tune your system after to get an optimum setup.
To install Xen on Debian, grab the following:
# apt-get install xen-linux-system
Next, define a network bridge in /etc/network/interfaces for your domUs to access. In this case, I'm defining an unremarkable xenbr0 on my eth0 interface:
# Ruben's Xen bridge
auto xenbr0
iface xenbr inet dhcp
bridge_ports eth0
As with most hypervisors, you have the choice to use a partition or disk image for domU storage. Using GPT and logical volumes is beyond the scope of this post (aka, stay tuned), but seems to be the accepted standard.
For now, we can create a disk image for our domUs with the following.
$ qemu-img create 5G guest.img
Here's where we have to make a decision about how to run our VM. In Xen, we can run using paravirtualisation (PV) or HVM. Briefly:
PV uses some of the dom0's resources directly, including drivers and drive volumes. The benefit is far greater performance under some circumstances, though the domU needs kernel support. xen-tools can automate the installation of some PV domUs, but for others it can be quite a bit of work.
HVM virtualises the entire hardware stack, meaning most OSs can run in it without modification. This is more like what you may be used to in other contemporary hypervisors on the desktop and otherwise. Recent OSs (such as FreeBSD 10) include so–called PVHVM drivers which will give it PV-like drivers for use in HVM, thereby giving you the best of both worlds.
With that in mind, let’s make an HVM. In this basic example, I'm creating a FreeBSD 10 domU config. You may need to adjust the Xen paths for your system.
## Ruben's freebsd.cfg file for FreeBSD HVM
kernel = "/usr/lib/xen-4.1/boot/hvmloader"
device_model = "/usr/lib/xen-4.1/bin/qemu-dm"
builder = "hvm"
memory = "256"
name = "freebsd"
## Enable VNC access
vnc = 1
vnclisten = "0.0.0.0"
## Virtual file devices
## Attempt to boot from "c" (hard drive) first,
## then boot "d" (cdrom). Same flags as QEMU
boot = 'cd'
disk = [ 'file:/var/vm/freebsd.img,hda,w',
'file:/var/vm/freebsd-10.iso,hdc:cdrom,r' ]
## Virtual network interface
vif = [ 'bridge=xenbr0' ]
Now we can launch our new VM! Depending on your local install, you'll either want to use xl or the older xm:
# xl launch freebsd.cfg # xm launch freebsd.cfg
You can confirm the machine is running
# xm list => Name ID Mem VCPUs [..] Domain-0 0 8096 2 [..] freebsd 0 256 1 [..]
If you have an X server locally running, you can now preview with:
$ vncviewer :0
To access VNC from another machine, one option is to use an SSH tunnel:
$ ssh -X <your Xen machine IP> $ vncviewer :0
And there you have it, your own Xen machine! Ideally, your next steps will be to install your domU guest as normal, then configure console access and/or SSH so you can access remotely without VNC. These will be discussed in future posts, and linked back to here.
Initially configuring Xen can be time consuming, but it's a lot of fun and you'll be rewarded with a high performance platform to run your workloads.
If you need to download a series of images, such as nineteen project slides that were uploaded individually (argh!), you can get this done easily with curl:
% curl -O "http://example.com/[01-10].jpg"
Sometimes though, you’ll end up with empty files. If people publish their files through a CDN, the URL may simply be a redirect. To follow these and download the original:
% curl -OL "http://example.com/[01-10].jpg"
This is just scratching the surface, but it’s a substantial part of what I use curl for. Now I just wish my hair didn’t in humid weather.
From Wikimedia Commons:
Kaiser shipyards, Richmond, Calif. Miss Eastine Cowner, a former waitress, is helping in her job as a scaler to construct the Liberty Ship SS George Washington Carver launched on May 7, 1943
Sometimes, we need to remind sexist pricks that woman can be developers, scientists, engineers, scalars building gigantic wartime ships, whatever the fuck they want to be.
DISCLAIMER: Your Linux box won’t normally prevent you unmounting something without a reason. Run these commands at your own risk… to data integrity.
If you’re attempting to unmount a volume (such as /mnt), and it refuses…
# umount /mnt ==> umount: /mnt: device is busy. ==> (In some cases useful info about processes that use ==> the device is found by lsof(8) or fuser(1))
…there’s a chance a process is still accessing it. You can use lsof to see:
bash 14115 root cwd DIR 202,16 4096 2 /mnt
Great, in this case the only thing using it is our shell, and we’re not even in the directory. In that case, time to force:
# umount -f /mnt ==> umount2: Device or resource busy ==> umount: /mnt: device is busy. ==> (In some cases useful info about processes that use ==> the device is found by lsof(8) or fuser(1)) ==> umount2: Device or resource busy
For more persistent errors, its time to kill the process using it with fsuer.
# fuser -km /mnt ==> /mnt: 14115c # umount /mnt
Then the volume should unmount. Now would also be a good time to re–read the disclaimer at the beginning of this post.
