Parse Apache Log Files With PHP

This always felt like a good idea at the time the need came up. These days, for some ungodly reason, parsing Apache logs seems to come up a little too frequently to keep this up. In the spirit of making my life a hell of a lot easier for tomorrow I’ve taken a shot at writing an Apache log parser written in PHP.

One thing I decided to implement is a filtering system so you can filter out based on a provided regex. Might not be too useful to everyone but it should be trivial to remove the functionality.

Anyway, I hope someone finds this useful (even to learn from and, of course, use)

Here’s the main class:

<?php
/**
 * Apache Log Parser
 * Parses an Apache log file and runs the strings through filters to find what you're looking for.
 * @author Eric Lamb
 *
 */
class apache_log_parser
{
	/**
	 * The path to the log file
	 * @var string
	 */
	private $file = FALSE;
 
	/**
	 * What filters to apply. Should be in the format of array('KEY_TO_SEARCH' => array('regex' => 'YOUR_REGEX'))
	 * @var array
	 */
	public $filters = FALSE;
 
	/**
	 * Duh.
	 * @param string $file
	 * @return void
	 */
	public function __construct($file)
	{
		if(!is_readable($file))
		{
			return 	FALSE;
		}
 
		$this->file = $file;
	}
 
	/**
	 * Executes the supplied filter to the string
	 * @param $filer
	 * @param $status
	 * @return string
	 */
	private function applyFilters($str)
	{
		if(!$this->filters || !is_array($this->filters))
		{
			return $str;
		}
 
		foreach($this->filters AS $area => $filter)
		{
			if(preg_match($filter['regex'], $str[$area], $matches, PREG_OFFSET_CAPTURE))
			{
				return $str;
			}
		}
	}
 
	/**
	 * Returns an array of all the filtered lines 
	 * @param $limit
	 * @return array
	 */
	public function getData($limit = FALSE)
	{
		$handle = fopen($this->file, 'rb');
		if ($handle) {
			$count = 1;
			$lines = array();
		    while (!feof($handle)) {
		        $buffer = fgets($handle);
		        $data = $this->applyFilters($this->format_line($buffer));
		        if($data)
		        {
		        	$lines[] = $data;
		        }
 
		        if($limit && $count == $limit)
		        {
		        	break;
		        }
		        $count++;
		    }
		    fclose($handle);
		    return $lines;
		}		
	}
 
	/**
	 * Regex to parse the log file line
	 * @param string $line
	 * @return array
	 */
	function format_log_line($line)
	{
		preg_match("/^(\S+) (\S+) (\S+) \[([^:]+):(\d+:\d+:\d+) ([^\]]+)\] \"(\S+) (.*?) (\S+)\" (\S+) (\S+) (\".*?\") (\".*?\")$/", $line, $matches); // pattern to format the line
		return $matches;
	}
 
	/**
	 * Takes the format_log_line array and makes it usable to us stupid humans
	 * @param $line
	 * @return array
	 */
	function format_line($line)
	{
		$logs = $this->format_log_line($line); // format the line
 
		if (isset($logs[0])) // check that it formated OK
		{
			$formated_log = array(); // make an array to store the lin info in
			$formated_log['ip'] = $logs[1];
			$formated_log['identity'] = $logs[2];
			$formated_log['user'] = $logs[2];
			$formated_log['date'] = $logs[4];
			$formated_log['time'] = $logs[5];
			$formated_log['timezone'] = $logs[6];
			$formated_log['method'] = $logs[7];
			$formated_log['path'] = $logs[8];
			$formated_log['protocal'] = $logs[9];
			$formated_log['status'] = $logs[10];
			$formated_log['bytes'] = $logs[11];
			$formated_log['referer'] = $logs[12];
			$formated_log['agent'] = $logs[13];
			return $formated_log; // return the array of info
		}
		else
		{
			$this->badRows++; // if the row is not in the right format add it to the bad rows
			return false;
		}
	}
}
?>

And here’s an example of how to use it:

<?php
$data = new apache_log_parser($d->path.'/'.$entry); // Create an apache log parser
$data->filters = array(
	'path' => array('regex' => '/^.*\.(FLV|flv)$/') //pull only flv files
);
 
$data = $data->getData();
?>

A couple things to note about this script though:

1. The regex and parsing was pretty stolen from the Apache Log Parser on PHPClasses.org.
2. Without filters the script is pretty memory intensive. My needs don’t require anything client facing but heed my adivice; Don’t use this on a public web server.

Failed to Login

Zend Server is supposed to be a complete Web Application Server that is purpose built for php development. It includes application monitoring, problem root cause analysis and and extended caching capabilities. Pretty enticing really.

Unfortunately, Zend Server CE doesn’t include any of the above bells and whistles. Instead, it’s a stripped down version that appears to just match the features and functionality of XAMP or WAMP (Apache, php and mysql wrapped in a nice little installer for Windows).

I’ve used both used both XAMP and WAMP and, with little exception, I’ve always wished I’d gone with a manual installation instead. It’s not that they’re bad programs, it’s nice that they’re available for newbies, but my needs aren’t easy to package up in a “one size fits all” package. I like to try new things and experiment and sometimes what I want to do isn’t easy without breaking something. Admittedly, I haven’t tried to use any one size package for a few years so this may not be the case anymore.

Either way though, I know I have a bias; I might even be a bit of a snob about the issue. Totally possible.

That being said, after installing Zend Server CE, which went very smoothly actually, I was confronted with what appeared to be an incomplete installation of php; php-win just didn’t work. It did nothing in fact; I couldn’t get it to do a damn thing. Since I do a little maintenance scripting with php-cli (and php-win.exe is essential on Windows) this was a pretty big issue.

On top of that, I just couldn’t figure out how to modify the –configure options so changing the setup was obviously going to be an issue. I don’t know if I’m an idiot but I just couldn’t figure it out.

Then the let down happened; I was under the impression that there were going to be some cool profiling toys to play with. Instead, there’s a web GUI for configuring PHP, which is pretty nice I guess, but for me, it’s just easier to edit php.ini directly than navigate through a web interface. Kind of useless. What with the integration with Zend Debugger I was really expecting more.

Ultimately, it seems that if you’re a complete newbie to php Zend Server CE is a worthwhile fit but if you actually know what you’re doing you’re still better off setting up a development environment manually.

This is pretty disappointing. A product from Zend, that’s supposed to ease the pain of php development being released to the community, offering nothing more than you could already get from a dozen other programs kind of seems like posturing. I understand the desire to have a demo of a paid product but it should, you know, be different.

A good change I’d like to see would be to include some of the more advanced features like the Application problem diagnostics and the Application monitoring (alerting) functionality in the CE version. It would benefit the community far better than the current version.

lighttpd

Fast forward a couple years and I’m working on projects that Apache is really struggling with. The point of removing Apache from my toolbox started because one of our projects at StreetWise, the official community for one of our biggest client’s programs, was having serious load issues. The site was very popular, getting over 2,000,000 hits a day, and Apache was having a hard time dealing. Here we were with 8 web servers, on a load balancer, 3 DB servers, and all of the web servers were just getting hammered; it was cascading, where one by one the servers would go into swap and all traffic would be shifted to a different server, only to have the same thing happen again until all servers died. It was a very, very, stressful time for my team and I.

We tried configuring the hell out of Apache, upgrading the server’s RAM and adding additional CPUs but finally came to realize that continuing with Apache meant we’d just have to get more servers. Apache can only handle so many users at one time, like all webservers really, so we needed a web server that could handle more users at one time. It’s a pretty basic While adding additional servers to a project is cheaper than throwing people at it, if it’s a software issue, there’s additional costs with hardware in my team especially because we manage the servers ourselves.

This wasn’t good; it meant explaining to the client that we would need more money outside of our projections, which is embarrassing and, worse, my team would have to manage more servers (we were already managing 18 in total at the time).

It was at this time that lighttpd came back into play. Since Apache wasn’t working out so hot in the load we were under it had to be replaced. I remembered how lighttpd was started as a proof of concept for the C10K problem and had some pretty impressive benchmarks so I started researching more about it.

While lighttpd’s penetration wasn’t even a blip compared to Apache or IIS we installed it on a one of our dev servers and started testing it.

lighttpd Overall Usage

Installing lighttpd was pretty straightforward though setting up php to run in fast-cgi mode was a little more challenging than usual. Right away, one of the biggest challenges was the configuration though. After using Apache for, well, forever really, moving to how lighttpd was configured was pretty difficult.

lighttpd.conf example

$HTTP["host"] =~ "(www.)?your-domain.com" {
 
  server.document-root = "/www/your-domain.com/html"
 
#### auth module
## read authentication.txt for more info
auth.backend               = "htpasswd"
auth.backend.htpasswd.userfile = "/www/your-domain.com/.htpasswd"
#auth.backend.plain.groupfile = "lighttpd.group"
auth.require               = ( "/" =>
                               (
                                 "method"  => "basic",
                                 "realm"   => "Server Info",
                                 "require" => "valid-user"
                               )
                             )
 
}

Compare the above to the below.
httpd.conf example

<VirtualHost *>
    ServerName your-domain.com
    DocumentRoot /www/your-domain.com/html
    ServerAdmin contact@your-domain.com
    <IfModule mod_suphp.c>
        suPHP_UserGroup nobody nobody
    </IfModule>
</VirtualHost>

Right away, you can see how Apache and Lighttpd have different philosophies on layout and structure. Apache takes an XML structure for their configuration files while lighttpd formats its configurations like an array.

Anyway, after familiarizing ourselves in how lighttpd was configured we started the process of moving all the production servers away from Apache over to lighttpd. We went one server at a time, installing lighttpd, configuring, benchmarking and then changing the server’s init so lighttpd was started and Apache wasn’t. All told the process took about a week.

Since then, I haven’t had a single issue related to load on one of lighttpd boxes. If you’re serious about performance and care about how your projects function check out lighttpd ASAP.