A New Kind of Failure Point

Looking into the problem it was immediately obvious what was going on; my server was gone. I couldn’t access any site, much less my client’s site who alerted me to the issue, using any protocol or tool (ssh, ftp, cpanel or whm at least). It was like it didn’t exist…

Immediately, I contacted the server provider, HostGator, where I had a dedicated server requisitioned a few years ago. Thinking this was at worst a network issue I contact HostGator and submitted a support ticket; that’s the only thing to do since I couldn’t personally deal with the issue. That’s the trade off for leasing a dedicated server; on the other hand HostGator will fix anything that goes wrong so it’s worth it in my opinion.

HostGator, to their credit, got back to me within 20 minutes to inform me that the server account was closed. My first thought:

WHAT THE FUCK??!!??

Looking into just what the hell happened I found out that the client who was paying the bill (I had worked out a “deal” for him where I built and maintained his sites and he would pay the hosting bill) just decided that he was done with the whole Internet thing. Seriously. Apparently, he decided to just stop paying the bill a couple months ago forcing HostGator to cancel the account because he, in his words, “Hadn’t made shit from this crap”.

Anyway, this all highlighted a failure point in my backup strategy. Yes, I had a  backup strategy and I even had backups locally. No, my problem was that I didn’t have access to any backup newer than 2 weeks old. Here’s how my backup strategy worked:

1. Daily backups stored to a NAS on the rack.
2. Weekly backups FTPed to another HostGator account
3. Bi-weekly backups were being downloaded to my local network (home) every 2 weeks.

See the problem point? Yeah, keeping too much of an interval on the network instead of within reach and not having access to my recent backups. That. Was. My. Bad.

Two weeks was just too much time to go without a physical backup. It’s an infinity on the Internet; too much stuff can happen in that time and ironically my client with the nasty, nasty, Dolphin CMS is proof of that. Unfortunately, two weeks ago her site had no traffic and no real use to anyone. Come two weeks later and she had gotten the membership up to a couple hundred users who were actually using the site. That’s the part that really sucked; I had failed my client.

This whole ordeal prompted a number of changes to my overall routine. Here’s the new setup.

1. For starters I’m no longer leasing a server; instead I purchased my first server and have it hosted at a data center where I have personal 24/7 access to it and the rack it’s on.
2. Daily backups are being stored to a backup drive in my server and being mirrored to an external NAS on the rack.
3. Weekly, I will go to the data center and swap out the backup drive on the server with another keeping the drive at my house.
4. Bi-weekly backups will be downloaded to my laptop and kept close at hand.

Overkill? Probably, but considering that the majority of this is automated I don’t see too much of an issue with it. In fact, the only thing I have to actually do is go to the data center once a week and switch out the drive so it’s really not too bad.

Hopefully, this’ll work out better than the last strategy.

The Horrors of C99.php

A Little History

See, back in the day some php developers were pretty stupid. (Admit it; you were stupid once too.) What other explanation could there be for writing code that allowed the injection of arbitrary routines into a program. Trivially easy too.

To be fair, PHP was to blame a little for this as well. Given PHP’s high adoption, and design, by, and for, newbie programmers allowing such a technique by default was just ill conceived, and maybe even a little negligent. I understand the desire, and sometime need, for a technique that could be dangerous but to enable the feature by default…. damn man…

So, the risk was known, yet code was still being written (like the below example) that allowed remote file inclusion to be possible. Mostly because of the aforementioned default setting.

<?php
$color = 'blue';
if (isset( $_GET['COLOR'] ) )
{
	$color = $_GET['COLOR'];
}
require( $color . '.php' );
?>

BTW, if you currently write code that does anything like the above, frankly, you’re an idiot. You aren’t nearly as smart and clever as you think you are. I promise you this will bite you. Bad too.

About C99.php

So, using a technique like the above opens you up to learning first hand about c99.php. Finding information about the program itself is a little tricky but there are a couple examples that highlight just how devastating it can be.

When malicious intruders compromise a web server, there’s an excellent chance a famous Russian PHP script, r57shell, will follow. The r57shell PHP script gives the intruder a number of capabilities, including, but not limited to: downloading files, uploading files, creating backdoors, setting up a spam relay, forging email, bouncing a connection to decrease the risk of being caught, and even taking control of SQL databases. All these functions become readily available through an easy to use web interface, but now you can fight back.

Using the above explanation, which I agree with, c99.php acts as an interface to control your server. Once it’s on your server an attacker has easy access to view all the files and their contents, make changes to the system, upload new files, manipulate the database(s) and more.

Quite the nasty little script but pretty elegant in how it’s implemented. c99 is a completely standalone script; even the images are embedded inside using base64!

Until a month ago I would have thought the risk of encountering c99.php in the wild would have been small these days. Then, SMACK!!, a client had a site get hacked (quick CYA; that I didn’t’ work on ) using c99. So be warned. It’s out there and if you’re not smart, or if you’re a lazy, lazy, coder, c99 will get you.

Parse Apache Log Files With PHP

This always felt like a good idea at the time the need came up. These days, for some ungodly reason, parsing Apache logs seems to come up a little too frequently to keep this up. In the spirit of making my life a hell of a lot easier for tomorrow I’ve taken a shot at writing an Apache log parser written in PHP.

One thing I decided to implement is a filtering system so you can filter out based on a provided regex. Might not be too useful to everyone but it should be trivial to remove the functionality.

Anyway, I hope someone finds this useful (even to learn from and, of course, use)

Here’s the main class:

<?php
/**
 * Apache Log Parser
 * Parses an Apache log file and runs the strings through filters to find what you're looking for.
 * @author Eric Lamb
 *
 */
class apache_log_parser
{
	/**
	 * The path to the log file
	 * @var string
	 */
	private $file = FALSE;
 
	/**
	 * What filters to apply. Should be in the format of array('KEY_TO_SEARCH' => array('regex' => 'YOUR_REGEX'))
	 * @var array
	 */
	public $filters = FALSE;
 
	/**
	 * Duh.
	 * @param string $file
	 * @return void
	 */
	public function __construct($file)
	{
		if(!is_readable($file))
		{
			return 	FALSE;
		}
 
		$this->file = $file;
	}
 
	/**
	 * Executes the supplied filter to the string
	 * @param $filer
	 * @param $status
	 * @return string
	 */
	private function applyFilters($str)
	{
		if(!$this->filters || !is_array($this->filters))
		{
			return $str;
		}
 
		foreach($this->filters AS $area => $filter)
		{
			if(preg_match($filter['regex'], $str[$area], $matches, PREG_OFFSET_CAPTURE))
			{
				return $str;
			}
		}
	}
 
	/**
	 * Returns an array of all the filtered lines 
	 * @param $limit
	 * @return array
	 */
	public function getData($limit = FALSE)
	{
		$handle = fopen($this->file, 'rb');
		if ($handle) {
			$count = 1;
			$lines = array();
		    while (!feof($handle)) {
		        $buffer = fgets($handle);
		        $data = $this->applyFilters($this->format_line($buffer));
		        if($data)
		        {
		        	$lines[] = $data;
		        }
 
		        if($limit && $count == $limit)
		        {
		        	break;
		        }
		        $count++;
		    }
		    fclose($handle);
		    return $lines;
		}		
	}
 
	/**
	 * Regex to parse the log file line
	 * @param string $line
	 * @return array
	 */
	function format_log_line($line)
	{
		preg_match("/^(\S+) (\S+) (\S+) \[([^:]+):(\d+:\d+:\d+) ([^\]]+)\] \"(\S+) (.*?) (\S+)\" (\S+) (\S+) (\".*?\") (\".*?\")$/", $line, $matches); // pattern to format the line
		return $matches;
	}
 
	/**
	 * Takes the format_log_line array and makes it usable to us stupid humans
	 * @param $line
	 * @return array
	 */
	function format_line($line)
	{
		$logs = $this->format_log_line($line); // format the line
 
		if (isset($logs[0])) // check that it formated OK
		{
			$formated_log = array(); // make an array to store the lin info in
			$formated_log['ip'] = $logs[1];
			$formated_log['identity'] = $logs[2];
			$formated_log['user'] = $logs[2];
			$formated_log['date'] = $logs[4];
			$formated_log['time'] = $logs[5];
			$formated_log['timezone'] = $logs[6];
			$formated_log['method'] = $logs[7];
			$formated_log['path'] = $logs[8];
			$formated_log['protocal'] = $logs[9];
			$formated_log['status'] = $logs[10];
			$formated_log['bytes'] = $logs[11];
			$formated_log['referer'] = $logs[12];
			$formated_log['agent'] = $logs[13];
			return $formated_log; // return the array of info
		}
		else
		{
			$this->badRows++; // if the row is not in the right format add it to the bad rows
			return false;
		}
	}
}
?>

And here’s an example of how to use it:

<?php
$data = new apache_log_parser($d->path.'/'.$entry); // Create an apache log parser
$data->filters = array(
	'path' => array('regex' => '/^.*\.(FLV|flv)$/') //pull only flv files
);
 
$data = $data->getData();
?>

A couple things to note about this script though:

1. The regex and parsing was pretty stolen from the Apache Log Parser on PHPClasses.org.
2. Without filters the script is pretty memory intensive. My needs don’t require anything client facing but heed my adivice; Don’t use this on a public web server.

Decibel meter

Normally, the signal strength meter is a series of bars like the below image. Hold “Alt” and press “N”, “M”, “L”, “L”. To undo just redo

Normal Signal Strength Indicator

Decibal Meter

View Source Code

This one every web developer should know. In the Browser, hold “Alt” and press “R”, “B”, “V”, “S”

View Source Code

Help Me! menu

If you ever call up for support chances are you’ve been told to enter this one. It shows a bunch of info in one place but most of it will be greek to most people.  Hold “Alt” and (left) “shift”, then press “H”.

Help Me! Menu

Display IMEI Code

As far as I know this is really only useful for unlocking a device. Type “*”, “#”, “0″, “6″, “#” on the Home screen.

Display IMEI Code

Event Log

The Event Log is probably only going to be useful if you’re a developer or similar. It displays all the last Java goodies that have happened since starting the device. Hold “Alt” and press “L”, “G”, “L”, “G”.

Event Log

Show SIM Information

This code will display the details about a SIM card. You have to enter this code from the SIM Options page and press “M”, “E”, “P”, “D”

Show SIM Information

View Refld and ServiceUserId

From inside a message hold “Alt” and press “V”, “I”, “E”, “W”

View Refld and ServiceUserId

View Data Usage & Voice Usage

This one is really only good if you have a new, unused, Blackberry. It maxes out at the below but if you buy a “new” Blackberry enter this code and verify it. From Status Options page press “B”, “U”, “Y”, “R”

View Data Usage & Voice Usage

Hard Reset

Doing a Hard Reset is like taking the battery out and inserting it again. Handy if you’re not in the mood to fuddle with the case. Hold ALT + Right Shift + Del

Validate Contact List

I’m honestly not really sure what these last two do but I’m listing them anyway for completeness. Hold “Alt” and press “V”, “A”, “L”, “D” in address book list

Rebuild Contact List

Hold “Alt” and press “V”, “A”, “L”, “D” in address book list.

These are all the “secret codes” I could find but that doesn’t mean there aren’t any more. Also, these codes were only verified on the Blackberry Bold 9000 so they may not work with all devices. If anyone has any new ones please leave a comment and I’ll update a list.

How To Unlock AT&T Blackberry

Initially, I was a little nervous about the process of  unlocking the Blackberry. I’d never personally tried anything like this and my direct experience with the process was that a 3rd party company would have to be brought into the mix. I didn’t relish the idea of paying someone to handle what seemed to be a simple exercise when I’m capable of pushing buttons on a freaking phone.

Turns out it’s actually extremely easy to handle this yourself though it does require a phone call to AT&T support. Simply call them up and they’ll ask you for your IMEI code; it’s under the battery on the serial number label. They’ll send you an email, like the below, that’ll walk you through the process.

Your device unlock request was received and processed, see below for details:

IMEI: YOUR_IMEI_NUMBER

Unlock Code: YOUR_UNLOCK_CODE

Caution: If this process is unsuccessful ten times in a row, the phone will be permanently locked to the at&t network. Do not attempt to enter the code more than one (1) time total.  Instructions below will assist you in unlocking your device, if these steps are unsuccessful please contact us at 1-800-331-0500 or (916) 843-4685 from overseas.

Blackberry 8310

Follow these steps to unlock device:

1. Turn off the radio! VERY IMPORTANT

2. Go to “Options”

3. Scroll to and select “Advanced Options”

4. Click on “SIM Card”

5. Type “MEPD” (You will not see on display what is being typed. To obtain a “P” double tap “OP” key)

6. Type “MEP2” (To obtain a “P” double tap “OP” key. Press “ALT “key to obtain a “2″)

7. Enter the unlock code

8. Press enter

9. Reboot device. Device is now unlocked.

Note: To verify the IMEI, dial *#06# on device’s keypad, 15 digit # IMEI will display on the screen.  If this sequence does not work, pull the back/battery off the phone and the IMEI will be listed on the back of the phone.

All told the total time invested was only about 5 minutes. Seriously, 5 minutes. So why would I want to pay someone for this again?

The first thing you want to do is backup all your installed CPAN modules. These are basically the library of functionality your system relies on to work. Pretty important stuff in there. It’s a pretty simple command though:

perl -MCPAN -e 'autobundle'

The above will create a “bundle” of all the CPAN modules you have installed to be used after the upgrade. Make a note of the bundle location though; you’ll need it later.

The instructions to do the actual installation were included in the email and is geared specifically towards cPanel servers:

cd /root
wget http://layer1.cpanel.net/perl588installer.tar.gz
tar -zxf perl588installer.tar.gz
cd perl588installer
./install -optimize-memory

This parts pretty simple, but time consuming. The whole process is automated and should take about an hour or three. When you come back you’re going to want to install the autobundle you created earlier. To do that you’ll need this command:

perl -MCPAN -e 'install Bundle::NAME_OF_BUNDLE_YOU_CREATED_EARLER'

This is where things started getting difficult. This process is going to require all sorts of input from you. It’ll, thankfully, give you default options but the install will sit there until you enter something. All told, this process took me about 4 hours; just sitting there watching the process.

Definitely a pain but not upgrading PERL would have been worse. If you have to do it be sure to set aside an afternoon though.

Windows 7

Because the promotion was for a version of Windows 7 that was the same flavor as the installed Windows Vista, I was given a 64bit Windows 7 Home Premium copy. Personally, I would have preferred Business or Ultimate but ok, fine Home Premium it is. At least this meant that I wouldn’t have to do a clean install so I could keep all files and programs where they were (still backing up the data of course).

The upgrade package came with 2 discs; the Windows 7 disc and a driver upgrade disc. The instructions said to insert the upgrade disc first and I’ll be prompted to enter disc 2 when required. Doing so started an upgrade program that inspected my system and warned me about deauthorizing my iTunes account which immediately made me feel good about the experience.

All told the install took around 3.5 hours and was like watching water boiling. I did it super late at night but I was still up and every time I would check on it I swear the progress rarely looked like it was making any progress. Still, it is Windows, so I was used to this; just wait and be patient, it’ll finish. And eventually it did.

Upon first booting up there were a couple issues. One was my fault. The others… not so much. Probably the worst offense was that I had no Internet connectivity. I checked both my wired NIC and the wifi and both were working  fine; I could connect to my router through both and I could find my Xbox and PS3 on the network. I just couldn’t get online. I eventually found that this was caused by a conflict between Esets firewall (which I had disabled in Vista) and the native Windows firewall. Uninstalling Eset and reinstalling it solved the issue.

I also had an issue with my local Apache webserver working. After checking the logs it turned out to be a soft link I had created under Vista to link the conf directory (makes editing the files from my working directory that much easier). Creating a new link solved the issue nicely.

The last issue is with the touchpad; and I haven’t really fixed it yet. At first, Windows thought the touchpad was a PS/2 mouse. This wouldn’t be an issue except I like the scrolling functions on the touchpad and the PS/2 drivers don’t support it. This seemed like a cut and dry driver issue except I installed all the latest drivers for my laptop that Gateway offered and it still doesn’t work all that well. Yes, it’s there but not in any real functional way. It’s jerky and hesitant when it works (around 30% of the time). Not enough to get me to downgrade but still a pain.

I’ve never really had an issue with Vista but I was still excited about Windows 7. Even though there were a couple hiccups during the install, and that my touchpad isn’t operating 100%, I’m still really happy with the experience.

IMAP on Blackberry: The Poor Mans BES

When I had the day job I had access to a nice little network with Exchange 2003 and the latest Blackberry Enterprise Server (BES). It never really occurred to me how spoiled this would make me but, now that it’s gone, it’s pretty obvious how much I relied on it for my email management.

Why? Syncing. Loved the whole syncing aspect; open an email on your Blackberry and the email was marked as open in Exchange. Send an email from your Blackberry and it appeared in your sent folder in Outlook. Even better if you sent an email in Outlook, through Exchange, it was put in your sent folder on your Blackberry. Simple. This was great for things like filing and responding to emails. Put simply, using the Exchange/BES combo allowed you to keep all your communication in sync.

Then I get unplugged from the BES and have to setup a new solution. First, I went to POP3 (as I was most familiar with the protocol) but this was a cluster fuck of a bad idea. Yes, email was put on the Blackberry but it’s a one way method and nothing was kept in sync. I’d do something on the device and when I would go to Gmail nothing would be there; this couldn’t do.

I also tried the native Gmail Blackberry app but this too wasn’t ideal; I hated how it wasn’t truly a “native” app in that other apps would still open the default Blackberry message program to do anything email related. For example, if I clicked on an email on a web page while on the Blackberry it wouldn’t open the Gmail app but the native messaging one instead. I was surprised how much this happened and how much it bothered me but it did.

Then I learned that Gmail has IMAP support and have been using it ever since. So far, IMAP is the closest approximation of the BES experience I’ve found. It’s not without it’s issues or anything, and technically it’s not supported by Google, but it’s the best solution I’ve found.

Probably the biggest issue I have with the IMAP integration is that mail manipulated on the Gmail site doesn’t get synced to my Blackberry. This isn’t too bad since I usually only use the Gmail site for sending but if I want to keep things in sync I have to treat the Blackberry as the master.

To keep the calendar and contacts in sync requires the use of another Blackberry app called Google Sync. Google Sync it’s keeps your contacts and calendar synced up between Google Calendar and Gmail contacts and your Blackberry.

While it’s not a perfect solution I’ve been mostly happy with IMAP and Gmail on the Blackberry. It’s way, way, better than paying money for an outsourced BES / Exchange account.

Don’t Be A Pussy

Not to worry though; try as hard as you want to not fuck up and it’s just going to happen that much sooner.

I can say with absolute certainty that there’s going to come a time in your career when you fuck up. Big. Like really BIG. The type of mistake that has the potential to sink the company or client you’re working for/with. When it happens it’s going to be bad. So bad that you’ll have the fear of Dad in you. You remember that right? When Dad was coming home and you knew he knew what you did and you knew your life was over. If you didn’t have a Dad; think shear panic mixed with absolute paranoia and terror. Yeah, that’s the stuff.

What you did/will do isn’t important. What is important is how you deal with it. You’re going to have options when it  comes to dealing with the issue(s) and how you act is going to determine how your colleagues and peers look at you for the next few months. Make the wrong call and you’re in for some real uncomfortable silences and some really awkward sidelong glances.

If this has already happened to you; congratulations. Just know it probably won’t be the last. On the other hand if it hasn’t happened yet get ready; it will. You’re going to make some stupid mistakes in your career; mistakes so idiotic and so demoralizing your confidence will shatter and you’ll have a hard time getting back on the horse.

Like I said above, I have absolutely no idea what you do or what you can do to fuck it up so, as anecdotal examples only, I’m going to rely on my personal experience. I can honestly say, with absolute pride, that I have done the following:

• Deleted a database and couldn’t restore the data
• Deleted all the rows in a table and didn’t have a backup
• Deleted a user account and all the email and files associated with it.
• Changed every users password to “password” in a database
• Sent an internal cost analysis report for a client project to the client

And that’s only what was off the top of my head; I’m sure I’ve blocked out some of the worse things. The one constant between the above list (aside from the stupidity involved) was that I owned the mistake. You have to immediately handle the situation whatever that means (it’ll depend on the situation).

After that though a funny thing will happen; it’s very likely your confidence will be shot. This is important because you need confidence (read: balls) to work in IT. There are too many things, that you just don’t know how to do, that you’re going to have to do, and that requires the confidence to know you can do these things. It’s why we make the big bucks.

In my experience the only thing you can do in these situations is get back on the horse ASAP. The sooner you do something, anything, that has consequences the better. You can’t wallow in the past and getting hung up isn’t the answer.

BTW: After reviewing the above I have to say:

Thank fucking God I don’t work in IT anymore.

When Did Performance Stop Mattering?

Which lead to the question of why.  At the moment it seems like a question of speed of development versus performance (which is ironic because Zend Framework is not easy or speedy to develop with).

<disclaimer>
To be fair, it’s not just frameworks that have an uncomfortable overhead. Just take a look at Joomla and Drupal; 2 popular content management systems with an absurd overhead. It’s just easier to focus on my current interest rather than the CMS’es.
</disclaimer>

One thing I’m having a hard time getting comfortable with is how much memory is required when using a php web framework. Out of the box both Zend and Symfony (for example) use around 5MB per request. Understand, this is without any custom code. Just setting up the MVC and Autoloader for the default views and models. Nothing impressive or useful and 5 fucking MB to run that?

After having been on the wrong end of this issue on my own code I’m pretty sensitive to how my code performs; I’ve written some nasty algorithms and watching them crumble in real time has a tendency to turn you around

Researching the issue doesn’t really help. There’s a lot of advice on how to improve the performance but it seems to always center around common sense improvements you should be using anyway.

The most touted improvement I’ve heard is that you have to use a PHP accelerator and opcode cache. I just find that response flawed but not because it’s bad advice but because it’s common sense. Yes, it’s true, but not using a framework in combination with a PHP accelerator and opcode cache is still better in my experience. All relying on those tools does is move the baseline for performance, which you’re supposed to do already, and a framework still consumes a good amount of resources on it’s own.

In my experience you get about a 50% reduction in memory usage when using something like x-cache but using the Zend Framework  still leaves a total of 2.5MB of memory usage to accomplish the bare minimum setup.

One saving grace is that hardware is cheap. Scaling with hardware is usually the go-to escape when the bottleneck is the code but it’s not without it’s own set of issues. For one thing while it’s true that hardware is cheap the labor to maintain that hardware is not. Especially if you want to maintain the server in a proper and responsible manner.

Another option, that’s really only available when using the Zend Framework, is decoupling the project from a direct dependence and not use the MVC components. In anticipation of doing this I’ve been writing a lot my recent code and projects in a style that’ll allow easy(ish) separation when the time comes.

At this point I haven’t used a framework in a production environment so all of this consternation might be for nothing. I just have a hard time accepting the performance hit of half a MB for using something trivial like a content management system (drupal) or, for example, a component like Zend_Navigation compared to the benefit. What are they actually doing to make the cost worth while?

Still another option is to just walk away from this whole OOP thing and head back to the familiar touch of procedural php and using functions and classes as more of decorators to apply than core components.  From my personal experience, and only my experience, using OOP is way more expensive than procedural. At the end of the day I need my programs to work fast, be easy to operate for my users and have a low impact on the server. How does using OOP help that?

At the moment I’m not sure how this is going to work out. I am confident it’ll be an adventure though. Hopefully, I find out how Zend will scale before a project of mine goes viral or gets popular. Hopefully.