A New Kind of Failure Point
It all started with a simple email from a client with the subject line “Broken Link”; looked like one of my client’s sites was down. Since this was the one that used that lovely of loveliest of programs, Dolphin CMS, I didn’t really thing much of it. Remember, Dolphin CMS sucks and, yes, it’s been known to just crumble on occasion (fucking Boonex…). So, yeah, initially, I didn’t think much of this. But then again, I have been pretty happy lately and the universe really does hate us all. I should have known this was gonna be bad…
A New Kind of Failure Point
Looking into the problem it was immediately obvious what was going on; my server was gone. I couldn’t access any site, much less my client’s site who alerted me to the issue, using any protocol or tool (ssh, ftp, cpanel or whm at least). It was like it didn’t exist…
Immediately, I contacted the server provider, HostGator, where I had a dedicated server requisitioned a few years ago. Thinking this was at worst a network issue I contact HostGator and submitted a support ticket; that’s the only thing to do since I couldn’t personally deal with the issue. That’s the trade off for leasing a dedicated server; on the other hand HostGator will fix anything that goes wrong so it’s worth it in my opinion.
HostGator, to their credit, got back to me within 20 minutes to inform me that the server account was closed. My first thought:
WHAT THE FUCK??!!??
Looking into just what the hell happened I found out that the client who was paying the bill (I had worked out a “deal” for him where I built and maintained his sites and he would pay the hosting bill) just decided that he was done with the whole Internet thing. Seriously. Apparently, he decided to just stop paying the bill a couple months ago forcing HostGator to cancel the account because he, in his words, “Hadn’t made shit from this crap”.
Anyway, this all highlighted a failure point in my backup strategy. Yes, I had a backup strategy and I even had backups locally. No, my problem was that I didn’t have access to any backup newer than 2 weeks old. Here’s how my backup strategy worked:
- Daily backups stored to a NAS on the rack.
- Weekly backups FTPed to another HostGator account
- Bi-weekly backups were being downloaded to my local network (home) every 2 weeks.
See the problem point? Yeah, keeping too much of an interval on the network instead of within reach and not having access to my recent backups. That. Was. My. Bad.
Two weeks was just too much time to go without a physical backup. It’s an infinity on the Internet; too much stuff can happen in that time and ironically my client with the nasty, nasty, Dolphin CMS is proof of that. Unfortunately, two weeks ago her site had no traffic and no real use to anyone. Come two weeks later and she had gotten the membership up to a couple hundred users who were actually using the site. That’s the part that really sucked; I had failed my client.
This whole ordeal prompted a number of changes to my overall routine. Here’s the new setup.
- For starters I’m no longer leasing a server; instead I purchased my first server and have it hosted at a data center where I have personal 24/7 access to it and the rack it’s on.
- Daily backups are being stored to a backup drive in my server and being mirrored to an external NAS on the rack.
- Weekly, I will go to the data center and swap out the backup drive on the server with another keeping the drive at my house.
- Bi-weekly backups will be downloaded to my laptop and kept close at hand.
Overkill? Probably, but considering that the majority of this is automated I don’t see too much of an issue with it. In fact, the only thing I have to actually do is go to the data center once a week and switch out the drive so it’s really not too bad.
Hopefully, this’ll work out better than the last strategy.
Email
Twitter
I think the “deal” was really the failure point. I mean WTF?!?!?!