Backups and Linux's Bullshit

Actually, I think I do like to blog.  So here's a new wall of text for you to read:

After linking to the awesome web server I wrote in Perl from my previous blog post, I accidentally deleted it.

I'd noticed in the access log that occasionally a random IP comes along and requests a bunch of files from /cgi-bin/ (which doesn't exist) presumably looking for exploits.  Just for shits, I decided to create the folder and put empty files in there.

Then I looked at the directory listing, but all of the empty files looked unsatisfying.  So I typed this command:

for i in *; do dd if=/dev/urandom of=$i bs=4096 count=1; done

Then I thought "what if I toss enough random binary data in there that, by luck, whatever script they're using to parse the output crashes?" Seemed worth an attempt.  So I edited the command to change count=1 to count=4096 and went about executing it again. 

However, somehow (and I'm still not sure how, as I had to type cd .. twice to get there) I executed the command within the directory that contained the web server itself rather than the folder which contained the bogus files.  Thus, I accidentally deleted it while doing something that, even if it had happened the way it was supposed to, would still have been entirely pointless.

Honestly, the command like is a dangerous place. 

I fuck shit up all of the time by pressing the up arrow to re-execute a previous command, but accidentally pressing enter after returning to the wrong one.  Far too often I find myself in a loop like this:

edit code.c
compile code.c && ./code
edit code.c
compile code.c && ./code
edit code.c
compile code.c && ./code

Like any sane person, I don't re-type the commands, I just press "up up enter" each time I find myself at the command line.  Then I decide I want to make major changes to code.c, but I'm not sure I want to keep them, so I decide to make those changes to test.c instead.  So I type this...

cp code.c test.c
edit test.c

Then I spend about an hour making those changes to test.c.  Finally, ready to test out my changes, I exit the editor and, like I've been doing in all of the previous instances, I press "up up enter" to re-execute the compile command.  ...but of course, that just re-executes the cp command, wiping out the last hour of work.  Fuck!

Indeed, every time I find myself removing a bunch of files, and typing a command like this...

rm -r /this/that/whatever

I become ultra-paranoid as I'm typing it.  Pressing enter accidentally is too fucking easy.

Computer filesystems are simply too unforgiving of mistakes.  They aren't designed for use by humans.  Undo is one of the most important features of almost any software.

Anyway, since I still haven't written my awesome backup software, my last "backup" was a few months ago, and the web server is newer than that.  So no more awesome little web server.

Since what prompts me to make "backups" is the realization that I can lose all of my data, I now set about making new "backups" by once again copying my home directory to external media.

In copying to one disk, I accidentally nudged (and just barely) the USB connector, which prompted the OS to give up and assumed the drive was gone forever, even though it became accessible again almost immediately. 

I'll never understand why the people implementing USB haven't considered the possibility of intermittent outages.  Rather than just corrupt the file system, the OS's response to unplugging of removable media that hasn't been dismounted should be a pop-up window that says "plug that thing back in, I ain't done with it," at which point it merely continues to use it as if it hadn't been unplugged at all.

After copying my files to the second drive (the one I didn't accidentally unplug) I decided to run a filesystem check.  First thing I noted was that it had journal items to replay, which is interesting since that happens at every mount, so those items must have been left over from when I'd just cleanly unmounted the drive a moment ago.  How the fuck that happened I have no idea.

Linux does make it confusing as fuck to unmount media.  In Windows, you click unmount, then it tells you when it is safe.  Apparently in Linux they decided that was an unnecessary notification since in many cases it appears immediately.  So at first they disabled that notification, but sometimes media doesn't unmount instantly, and so you disconnect it too soon.  So then they made a pop-up that appears if the unmount is taking more than a few seconds, so then you click unmount, and since you rarely see anything telling you to remove the media, you wait a second, then remove it, and then a split second later a window pops up to tell you that the unmount is taking longer than usual.  Their latest incarnation of incompetence is to give you no notification at all, but let you see when the unmount is complete by keeping the icon for the external media in the file browser until the unmount is finished, then the icon disappears.  So now, since you don't know this, you unmount, wait 20 seconds (to make sure that the old behavior, where a window pops up to tell you it's taking longer than usual, doesn't occur), then disconnect the drive, at which point a window pops up to say "you shouldn't have done that." So you plug it back in to fix the filesystem, but then it says "nope, since you used NTFS (since FAT can't do > 2 GB files and you need to access the media from Windows as well) you need to plug your media into a Windows computer, repair the filesystem, unmount it, mount it and check it again, and only then can you access it from Linux." (Seriously.) Dumb fucks.

So anyway, I sit there for a while waiting for the icon to disappear, but it never does.  So I type "sync" in a console, and a second later it's done, but the icon is still there.  So I try the umount command manually, but it's all like "it ain't mounted, bitch." So I try typing "sync" a few more times, but I'm left to assume that the dumb fucks only implemented the behavior of the icon disappearing for flash media, under the assumption that hard disks aren't ever actually removed from a system.  So I just unplug the drive. 

So I don't know, perhaps it wasn't cleanly unmounted.  It's hard to tell when apparently anyone with a stupid idea for how to change the behavior of a GUI used by millions of people can do so without anyone stepping in to say "Jesus fucking Christ, it works the way it is and no one has a problem with it but you.  If you don't like the fact that it notifies you that you can remove the media immediately after you unmount it, then just use slower media and write more data to it so that you have to wait."

Anyway, the file system check then went on to discover a bunch of errors, which it told me I'd have to use --rebuild-tree to fix.  ...and --rebuild-tree warns that I should back up the data first.

Yes, I needed to back up my backups.  Deciding I just didn't care, as I assumed it was unlikely to lose the data and it wasn't my only backup (except for some really old backups, from before I obtained the second external drive), I didn't bother to make a backup of the drive first.

After that ten hour process, somehow the drive now contained a basic Linux filesystem (busybox + things like /proc/ and /etc/) and a couple of the folders which were on the drive before, but the vast majority of stuff was now missing.  Thinking that perhaps that stuff was in a sub-folder now, I looked around, but it wasn't.

So I decided "fuck it" and created a new filesystem, using ext4 rather than reiserfs as it had been before.

While doing that, I noticed that the output of the rebuilding process, which was still visible in the command window, indicated that it had put enough files and folders into /lost+found/ that everything I had on the drive was likely in there.  However, that folder wasn't on the drive when I'd examined it.  Being the whole purpose of that folder, I certainly would have looked there for my files.

After making the new filesystem, I mounted it, to reveal a totally empty hard disk.  This seemed a tad surprising at first, but then I remembered "oh yeah, I just reformatted it."

Then, in a command window, I made my way into the directory where it was mounted, as in formatting it I'd chosen a specific volume label and wanted to see that it was mounted under that name rather than a random UUID.  Indeed it was.  However, I have this habit of typing ls even when I don't care to look at what files are present, and I noticed that it didn't indicate an empty drive.  It told me that there was single folder, /lost+found/, on the drive.

I looked up at the folder window, which was still empty.

So I needlessly deleted all of my backups because the motherfucking file browser in Linux hides the lost+found folder!

Undoubtedly this is because ext4, unlike reiserfs, re-creates that folder at every mount whether it has anything to put in it or not, and the authors of the file browser were simply sick of looking at it, so they made it hidden.  However, since I was using reiserfs on all of my removable media, I'd never noticed that the folder wasn't being shown since it usually isn't there and so I'd never expected to see it before.  Why would it be hidden, when the convention in Linux is that only file names which begin with a period are hidden?

Well, stupid me for assuming that the file browser is actually going to show me what files are on my disk.

I guess I really need to write my awesome backup software, for seemingly countless fucking reasons.  Apparently everything about how my computer works is conspiring against me to cause me to lose data, and so having it written to uneraseable media is the only way to be safe.