Made of Everything You're Not

When I, ahem, "left" StreetWise a couple years ago and started freelancing full time one of the first things I did was change up how I bill from, the traditional, project based scale into an hourly rate. I did this for a variety of reasons, mostly having to do with a lot of issues I see with the project pricing model. It certainly wasn't an easy thing to do, but after a couple years of this I have to say it's been very good to me and something I think every freelance developer should at least look at.

If you don't know, the traditional way of pricing out a web development project (at least in my experience) is for a client to contact an agency (or freelancer), tell them what they want and the agency telling the client how much it'll cost at a flat rate. There's usually a back and forth over cost and features, time and expense, and shaving features accordingly to meet some predetermined budget, and there's certainly a lot of minutia in between the parts, but those are the broad strokes (again, in my experience).

Every agency I've worked for and with, and every freelance project I'd previously taken on, had been priced using this model. From a client perspective it's a winner; the client has an expectation of what a project will cost and the agency can manage expectations and allot resources accordingly. Considering most agencies have their employees on salary, where the employees make a set rate regardless of work, this can work out nicely for the agency, from a bottom line stand point, as well. When things go outside of scope, and they will go out of scope, your employees work more with no additional investment from a raw cost standpoint.

The obvious downside though is that the principals of a project, the people who actually create the damn property or concept, are treated as replaceable commodities. The effect this can have on morale within an agency can not be understated. But, and this is important, within an agency, that's the job. In my opinion, that's why good principals can easily make 6 figure salaries with sick benefits and bonuses; you get worked like a dog.

To be fair, I have heard stories of agencies where treating principals as disposable isn't the norm but those are usually the smaller, boutique firms, and I haven't had much experience with those.

In freelancing though this model doesn't hold up even a little. There is no salary safety net. When a project goes out of scope, and again, it will go out of scope, you're fucked as a freelancer. You're no longer being paid without then negotiating for overages (which carries it's own risk). Plainly, agreeing to a set price as a freelance web developer is a sure fire way to go hungry. Yet so many of us work that way it's simply amazing, in hindsight, when I think about it.

The biggest challenge to charging hourly is actually finding clients who are willing to pay an hourly rate. It was pretty surprising to realize just how ingrained the project pricing model is within client's expectations. Loads of potential clients love having a set price. That said, an unexpected benefit of the hourly model is that those clients who are willing to pay hourly are much more disciplined and knowledgeable about the medium and what's in store. Simply, just having an hourly pricing model weeds out the ignorant and unrealistic clients. I found that it's a nice way to keep the nonsense in check.

When I find a new client who's willing to pay hourly (with a couple exceptions) the first thing I do is get a retainer. The retainer basically gives me the confidence in the client to be able to pay my rate, as well as giving me motivation to start work with the confidence of continued payment. Plus, if a client can't, or won't, pay the initial retainer than I know right away they aren't worth working for. Harsh, maybe, but the initial retainer is important above and beyond the money involved. It shows respect and trust, and provides motivation for the work ahead. The retainer is only required up until trust is built; future projects don't require it.

Once the retainer is agreed to, though not paid, the next thing to do is figure out how long it's going to take to do what the client needs done. Basically, how much are things going to cost. This is where the process is very similar to the project pricing model; you still have to know ahead of time what's expected so you can say how long it'll cost. The big difference here though is that your estimate needs to be flexible. Instead of saying, and sticking to, a project will cost, for example. $15,000 you say it'll take around 100 to 150 hours likely costing $10,000 to $15,000.

The "to 150" hours part above is important. Always overestimate. What I do is imagine how long something will take and then, seriously, multiply that by 3. For example, if I think a project would take me 50 hours I say it'll take 150 hours. Why? Three reasons; shit changes, there's more involved than just development, and experience has taught me that I, personally, suck at estimating timelines (I'm an optimist). You have to take into project design, account emails, phone calls, client visits, project management, quality assurance, and client hand holding. The back and forth can kill a budget but my time isn't free; the client pays for all of that.

Plus, from a client relations stand point, being under budget is always a good thing. If you can get the client to expect, and agree to, a cost that's more than expected, they'll love you to death when their budget isn't tapped.

This leaves the question of accountability. To make sure the client has the confidence in me to do what they need I've found it's important to provide as detailed an accounting as possible for all hours worked. Everything gets documented and passed to the client with invoices. When I send an invoice for 60 hours you can bet there's a breakdown of every single hour worked, what was done, on what project/task, and when. Honestly is critical. If a client decides to audit your invoices, and you need to defend what you've done, you don't want to go head to head with someone who knows it shouldn't take 40 hours to install the core of ExpressionEngine (for example).

In terms of deployment process I'm not a fan of holding projects hostage from a client. By this I mean, keeping the client from having access until payment is received. My thought is that if I have a client that requires this sort of shenanigans than they shouldn't be my client. Even entertaining the idea is a red flag so when/if it does come up I try to recognize the end is near. Further, that's just dirty pool in my opinion.

With all that said, to me, it's a wonder why any freelance dev would ever opt for a project rate (outside of panic mode, "I need money NOW", freaking out). It's just a losing proposition and one that I'm glad is off my table.

Last June I decided to do something unprecedented for me; I dipped my toe into the ExpressionEngine add-on market to try and sell some custom add-ons I'd written. Being the open source loving hippie that I am this felt pretty out of character for me at the time and I got a bit of grief for it from friends and my own id. But I've been talking about moving out of client services for far too long and figured a little experiment by writing some add-ons and selling them through Devot:ee (the de facto marketplace for ExpressionEnigne add-ons) would be a good introduction into full time commercial development. Looking at things 4 months later I realize that deciding do that was one of most important things I could have done for my career.

Those of you who read this blog regularly (which, I admit hasn't been that regular of late) probably noticed that I've been writing more and more about ExpressionEngine over the last year and a half. Obviously, I've been doing a lot of ExpressionEngine development. Mostly for client work but, like most things a good programmer does, I eventually built up a go to library of custom stuff for when new projects started. Just your normal stuff really; a couple extensions and modules to make my client's lives easier and give that little "extra" to make a project sparkle.

Then, my CartThrob project ended. This is a project that will forever have a place in my heart. Not the site specifically mind you, which is pretty awesome BTW, but the clients tight budget combined with needing more from the site than they had initially outlined created a dilemma that presented an opportunity. I could either hand off the site as is, which was to the client's spec but not to their expectations, or I could continue working without getting paid to make the client happy. Then, a third option came to mind.

I proposed to the client that I build an add-on to handle their needs on the condition that I retain ownership of the add-on. The plan was simple; I'd write this add-on and then sell it on Devot:ee in the hopes that eventually I'd recoup my loss in time investment. Hence, the birth of my first commercial add-on now called CT Admin. That... I then sat on for a couple months while I worked up the courage to put it up for sale.

The more I thought about releasing CT Admin the more nervous I became over the prospect of putting out an add-on that's only useful for users of another third party add-on. Starting out with a customer base that's a small slice of another company's customer base isn't exactly a good way to measure an industry. So, I decided to release a second add-on around the same time as CT Admin.

This second add-on, now named Backup Pro, was at the time a specialized backup add-on I'd written for my hosted clients which due to certain configuration conflicts couldn't use any of the existing ExpressionEngine backup add-ons on the market. Releasing Backup Pro was also a gamble though of a different sort; as mentioned there were existing add-ons within that niche and and one of those competitors has fiercely loyalty customers (Hi Tom!). But still, two add-ons are better than one. Right?

So on June 10th I had a stiff drink and posted my add-ons to Devot:ee. And a whole 24 hours later I sold my first add-on. Immediately, I realized I was in. I was hooked. The joy I felt when I got the email notification from Devot:ee about the sale was absolute and total. Someone had paid me money for proprietary code I had written that I was going to be able to sell again. Fuck. That was the sweetest $24 I ever made.

But $24 also came with a weird sense of responsibility I never really felt writing OSS code. With OSS I'd always kept things at my pace and, unless there was anything critical that affected me personally, individual issues other users experienced weren't that big of a priority. But, with the ExpressionEngine add-ons, that someone had paid real cash money for, the paradigm shifted from random and faceless users to clients. In essence my customers became my clients and that's a dynamic I've always been comfortable with in terms of knowing how to respond. Put simple; complete and total support until they're happy.

And that's really the key to everything right there. What initially started out as a small experiment I personally likened to an OSS project morphed into me sharing some responsibility with other developer's client work. Shit got real all of a sudden. People were paying money on my code with the belief that the code would help them solve their client's problems. For those users who had issues (and there were more than a few) my responsibility could not be denied.

Initially, I was naive enough to think I could just post my add-ons to Devot:ee and go about my life. I had no concept of how much care would have to go into support and customer satisfaction or the amount of hours I'd spend in the Devot:ee forums and answering questions on Twitter. To give an idea of just how many updates there have been; at the time of this writing Backup Pro is at version 1.8.1. That's a lot of bug fixes and feature requests in just under 4 months. Almost to the point of ridiculousness.

But it's addictive. Extremely so. The more feedback I got, and the more features and bugs that were fixed or added, the better the products became. The process creates a really nice positive feedback loop wherein we're all winners. The customer gets a better product, I get to create a better product which, in turn, makes my products more appealing to others. It's really something to be a part of.

So after 4 months of progressively improving my products, networking within the community, building a dedicated site for marketing and having an absurd advertising appetite and a library of 7 add-ons now, I look around and realize that I'm a part of something I had never anticipated; the ExpressionEngine community. And it feels like home.

I had a small task come to me recently wherein a site needed to allow for verification of ExpressionEngine credentials but couldn't use the normal controllers for access. The challenge was in how ExpressionEngine encrypts the passwords and replicating that behavior. Taking a look at the Login controllers made things very clear though; as usual ExpressionEngine was very well written.

Here's an example of how to do it (note that this will only work within the CP):

<?php
$user = 'test';
$pass = 'test';
$this->EE->db->select('members.password, members.unique_id, members.member_id, members.group_id, member_groups.can_access_cp');
$this->EE->db->where('username', $user);
$this->EE->db->where('member_groups.site_id', $this->EE->config->item('site_id'));
$this->EE->db->where('members.group_id = '.$this->EE->db->dbprefix('member_groups.group_id'));
$query = $this->EE->db->get(array('members', 'member_groups'));
if ($query->num_rows() != 0)
{
	$password = do_hash($pass);
	if ($query->row('password') == $password)
	{
		//good user credentials <img src="http://blog.ericlamb.net/images/smileys/smile.gif" width="19" height="19" alt="smile" style="border:0;" />
	}
	else
	{
		//bad password/good username
	}
}
else
{
	//bad username
}
 
?>

According to the site admin who passed this my way the above won't work outside the CP. He was kind enough to send along an example that worked fine for their situation:

<?php
$this->EE->load->library('auth');
$this->EE->lang->loadfile('login');
$authorized = $this->EE->auth->authenticate_username($this->EE->input->post('username'), $this->EE->input->post('password'));
if ( ! $authorized)
{
	set_status_header(500);
	exit(lang('unauthorized_request'));
}
?>

Fuck. We spent all this time convincing one of the oldest clients with the agency into upgrading their old and kludgy, PHP 3 era, nightmare website into a snazzy ExpressionEngine 2.2 gem of modern technology. But, like the assholes we are, we didn't quite think through the fact that they have a complicated data structure (having been built when PHP really sucked) that we'd have to import into ExpressionEngine. So, you know, fuck...

But then, after the pants come on and a moment to reflect is had and the obviousness of the overreaction becomes clear. This is ExpressionEngine after all; there's bound to be some utility available to handle this with the least amount of pain possible. And there is: the Member Import Utility and, even better, it's included into the core of ExpressionEngine so it's already there.

And in traditional ExpressionEngine fashion its built in much the same and logical way most would design an import system. There are two different modes to work with: one to import an ExpressionEngine Member XML file and another to create said file from a CSV formatted file.

The CSV format conversion wasn't really all that for the purposes of this project (mostly due to conditional rewriting of specific column entries) but it does have it's uses. For instance, with smaller, self contained, exports it should be a snap to easily hand over the export straight from phpMyAdmin and convert it into an ExpressionEngine Member XML file. One big drawback though is that the CSV import doesn't appear to create new custom member fields on import so manual mapping is required per import.

The flip side of the CSV format convertor is the straight ExpressionEngine Member XML import utility. The big drag with this, and only if you don't use the CSV convertor, is that a script has to be written to handle the conversion. Not too big a deal but, considering most installations will vary, this is going to require manual intervention every time a new system is imported. But, unlike the CSV convertor, the Member XML importer does actually create missing custom fields so updating the system shouldn't be too big  a drag.

For my purposes I wrote a dirty little script to create the XML file for me so it was simply a matter of importing. I'm lazy, in the long term, so keeping as much of this as simple as possible is always a priority for me. But, since it's a throw away thing cleanliness and best practices is less important than getting me more time to do real work. A word of caution though; the XML parser within ExpressionEngine isn't very helpful when it comes to error messages and telling you what went wrong. You will curse.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<?php
$sql = "SELECT * FROM users";
$result = mysql_query($sql);
if(!$result)
{
	echo mysql_error();
	exit;
}
$ee = "<members>\n";
while($row = mysql_fetch_assoc($result))
{
	$ee .= "\t<member>\n";
	$ee .= "\t\t<username><!." ".$row."]]]]><!." ".$row."]]]]><!."]]]]><!]]]><![CDATA[></".$key.">\n";
	}
	$ee .= "\t</member>\n";
}
 
$ee .= '</members>';
 
echo $ee;
exit;
?>

The above code is fine if your existing database is cool but, and this is weird, ExpressionEngine goes full retard when it encounters anything it doesn't like in the XML and won't let you proceed. It's really quite stunning. The XML importer knows that duplicate email addresses are bad in the database but instead of, say, allowing you to skip or force things the importer will just die and scream about a duplicate leaving you the only option of then manually pruning your data. Why they couldn't be bothered with a checkbox or something allowing you to skip bad entries is beyond me but it makes the experience really, really, painful if your data sucks.

Still though, if your data is good to begin with the importer is a nice touch.

An unfortunate issue with ExpressionEngine is how internal messaging is handled. It's really unfortunate that such a nice platform like ExpressionEngine shows such a wart when it comes to managing system pages and templates. Anyone who's ever had to modify the message pages can tell you; it's a dirty and counter intuitive deed requiring a lack of best practices that will leave you feeling pretty gross.

The current method of modifying the message pages works as you would expect; this being an ExpressionEngine site you just go to the Design section and you'll find them under "Message Pages" . There are 3 different types of message pages you can edit (Email Notifications, User Messages and Offline Template) and while it would seem logical on how what to do to get going you'd probably be wrong (if you hadn't ever tried before). There are a couple problems here...

First, you can't use EE template tags within those templates. It's the damnedest thing too because some of those templates (User Messages and Email Notifications) do allow specific tags; they just don't parse any others. So, if you're a good little programmer and you build your site using global embeds for building your templates (head & footer for example) you're hosed. Your only option is to hard code your HTML which, if you've built even a simple site, you'll instantly be filled with remorse and guilt over creating such a maintenance nightmare.

Second, and after the above it's a minor issue though if you're good enough for the above to be an issue this'll probably piss you off too; you can't save those templates as files. This means version control is going to be a bit of a problem. After nearly 10 years of having version control beat into me this hurt pretty bad. To be honest, because of tight deadline and even tighter budgets I could have lived with the guilt and shame of hard coding just to finish the project but taking away my version control was like adding insult to injury.

It's quite the head scratcher when you get right down to it; a smart and clever company redesigns their entire product from the ground up and completely drops the ball when it comes to the messaging system. It's a shame...

There is another option though it does require money; a really nice extension called Custom System Messages. The extension was written by Brian Litzinger and is available for $12 through the ExpressionEngine marketplace Devot:ee.

Custom System Messages works exactly as you would expect ExpressionEngine to work; through a nice interface you choose custom templates for specific messages. You can set a different template for both message scenarios (General Error and Form Submission Error) as well as change the behavior of the {link} template tag so it doesn't use the lame ass JavaScript "back link".

The best part though is that Custom System Messages allows you to set special templates based off of all the actions your site has (even those from other modules and extensions). So, for example, if you have a CartThrob site you can set a special template for when adding an item to your cart fails and separate template for when updating an item within a cart fails. It's really quite slick and for only $12 it's very reasonable. Priced just enough to dissuade any developer from rolling their own solution.

I do have to say how disappointing it is to see ExpressionEngine lacking this built in. To me, it's reminiscent of buying a car and being charged extra for the gas cap. Sure, I can just go buy a gas cap for a couple bucks but it really should be a part of the car to begin with.

I recently had a project where there would be 2 databases being used together; one locally and the other hosted externally by the client. Using traditional PHP this wouldn't be a problem (just pass a link to the database I want to use on a per query basis and I'm done), but, since this project was to use the Zend Framework and I'd never attempted this sort of thing before with it, I was in quite the pickle.

Doing some quick Googling turned up an article that seemed to fit the bill: Zend Framework: Connecting to 2 databases written by Wenbert Del Rosario. Unfortunately though, while it was a good read and very informative, my circumstances were a little different. The database I had to connect to was hosted on an external network that I wouldn't have local access to. I had to go over the wide Internet to connect. Using Wenbert's tutorial would entail a constant connection on every request whether it needed it or not and that just wouldn't do.

Anyone who's ever had to connect to an external database can tell you it's far from an ideal scenario. The latency involved with this strategy is noticeable and caching is pretty much required for any sane strategy. Still, Wenbert's article was definitely helpful for developing my strategy.

As in Wenbert's article I started with putting the connection information into my config.ini file

//config.ini
resources.db.adapter = PDO_MYSQL
resources.db.params.host = localhost
resources.db.params.username = ****
resources.db.params.password = ****
resources.db.params.dbname = local_db 
 
externaldb.adapter = PDO_MYSQL
externaldb.params.host = example.com
externaldb.params.username = ****
externaldb.params.password = ****
externaldb.params.dbname = external_db

Then, I created a standard Zend_Db_Table class that handled the connection:

<?php
//Model/DbTable/External.php
//name made up to protect the client but do yourself a favor and name your shit logically
class Model_DbTable_External extends Model_DbTable_Abstract
{
   /**
     * Doesn't matter for use but ZF demands a $_name variable
     * @var string
     */
	protected $_name = "content";
 
	public function init()
	{
		$settings = Zend_Registry::get('settings');
		$this->db = Zend_Db::factory($settings, $settings);
 
	}
}
?>

With the above out of the way I can now connect to the external database. I just have to pipe all my SQL through the above class. To that end, and to provide a nice wrapper for the caching, I create a Model that has methods for all the SQL my app will need:

<?php
class Model_External extends Model_Abstract
{
	/**
	 * The key to use for the cache items
	 * @var string
	 */
	public $cache_key = 'external_db';
 
	public function __construct()
	{
		parent::__construct();
		$db = new Model_DbTable_External;
		$this->db = $db->db;
	}
 
	public function testSystem($id)
	{
		$key = $cache_key.__FUNCTION__.$id;
		if(!$data = $this->cache->load($key))
		{
			//just a random query with joins and whatnots to show  it's possible <img src="http://blog.ericlamb.net/images/smileys/smile.gif" width="19" height="19" alt="smile" style="border:0;" />
			$sql = $this->db->select()->from(array('e' => 'example_table1'), array('e.id', 'e.title'))->where('e.id = ?', $id)->limit('6');
			$data = $this->db->fetchAll($sql);
			$this->cache->save($data, $key, $this->cache_key));
		}
		return $data;
	}
}
?>

So, using the above class I've abstracted out the SQL and provided caching through the below class Model_Abstract:

<?php
//models/Abstract.php
abstract class Model_Abstract
{
	/**
	 * The database object
	 * @var object
	 */
	public $db;
 
	/**
	 * The Cache Object
	 * @var object
	 */
	public $cache;
 
	/**
	 * The stored cache name
	 * @var string
	 */
	public $cache_key = null;
 
	public function __construct()
	{
		$c = new Model_Cache;
		// getting a Zend_Cache_Core object
		$this->cache = Zend_Cache::factory(
	                    'Core',
	                    'File',
	                    array('lifetime' => 720, 'automatic_serialization' => true),
	                    array('cache_dir' => '/path/to/cache/')
		);
	}
}
?>

Putting it all together I now have a mechanism to connect to an external database, perform queries against said database and can cache the results pretty easily and with minimal code. One of the advantages of this approach is that connections to the database only happen when needed instead of on every page request as in Wenbert's article. So, for example, so long as the cached items exist then no database connection is created and no latency (outside of hitting the filesystem of course) is created.

Still though, there are ways to improve on this approach. For example, using APC instead of the file system for the caching would help as would increasing the cache lifetime from 2 hours to something more but, overall, the above should be a good starting point.

About a year ago I reviewed various shopping cart platforms for what was to be an "upcoming" project. After numerous delays and client hand holding (with the requisite back and forth) one year later that project finally began. And, as has lately become a trend in my professional life, the client's spec combined with a drastically shortened and accelerated deadline forced us to go with something completely unknown and with only cursory inspection; the 3rd party e-commerce ExpressionEngine module CartThrob 2.0 (beta at the time of this review BTW).

Of course this presented the problem of actually using an unknown package to build a site with. Thankfully though, while CartThrob has a steep learning curve and is most certainly missing key functionality, it's an overall nice package that allowed us to do what we needed. Sure, there's pain involved (lots of configuration, testing and set up) but with a little elbow grease it all worked out.

As mentioned, CartThrob is an ExpressionEngine module for managing an online store. It's available for both ExpressionEngine branches (1.x and 2.x) and was voted Module of the Year by Devot:ee in 2010 (for the 1.x version only). This being an ExpressionEngine project CartThrob would, in theory, be a nice fit.

One of the really nice things about using CartThrob is that it uses the ExpressionEngine admin panel for managing your store. This meant there wouldn't be a second administration site that the client would have to learn. (Personally, I always found dual admins to be janky at best and sloppy/lazy at worst so this was a definite bonus in favor CartThrob.)

But this is also double edged because, well, pretty much everything about your store is saved into the entries system within ExpressionEngine. This, of course, causes your Entries panel to become an eye sore whenever you have to do anything there. Instead of having a dedicated section to manage products you use the Entries section. Instead of a dedicated section to manage your orders you use the Entries section. Instead of a dedicated section to manage... well, you get the idea; everything is managed through the entries. This makes sense for the product management but when you have to dig through entries to find orders and coupons and discounts (etc) things get a little jarring for some people (even internally with my team) so it's a concern.

Still, single freaking admin so it was kind of a lesser evil compromise.

The unusual thing though is that, for how thoroughly the management of items was punted (though, to be fair, it looks to be a feature versus bug debate more than "punting"), CartThrob has an extremely deep configuration section where you can customize CartThrob to function just right. CartThrob has all the basic customization functionality one would expect from any modern shopping cart platform (tax, shipping, payments with lots of merchant options, etc) plus a bunch of settings just to get CartThrob working (CartThrob is NOT an out of the box solution). It's all presented in an easy to use interface that's designed to mesh with the ExpressionEngine administration panel design which makes for an easier experience.

As far as merchant accounts go (for accepting payments through your site) there's the usual suspects of Authorize.net, Paypal and SagePay built right in but also some offline solutions for pay by check and credit accounts. For this project we used Authorize.net which went smooth for the most part but I can't speak for the other options though if they're anything like the Authorize.net one they're probably cool. CartThrob also includes some extensions and hooks so writing your own payment gateway is possible if you need.

CartThrob can also hook into the ExpressionEngine member module so your customers can be registered internally. Like pretty much everything else with CartThrob there's a bit of configuration involved to get it up and working though. Unfortunately though, there's not really any out of the box functionality for your members to take advantage of. For example, there isn't any way for your customers to view their past orders when logged in and when using the control panel and viewing a member's profile there's no relationship with CartThrob. You're gonna have to write all that stuff yourself but, and I don't know why, but CartThrob doesn't store any relationships between ExpressionEngine members and it's customers so expect a challenge.

In terms of the user side, the actual storefront, everything is done using ExpressionEngine templates and template tags. There are lots of module methods available and even though it's far from a complete package of the expected functionality. In fact, on our team, we got into a habit of thinking of CartThrob as a Lego set for creating what we want instead of having what we want already.

For example, to view the details about a users cart you'd use the template code below:

{exp:cartthrob:cart_info}
	{total_items} 
	{total_unique_items} 
	{cart_entry_ids} 
	{cart_shipping} 
	{cart_tax_rate} 
	{cart_tax} 
	{cart_subtotal} 
	{cart_total} 
{/exp:cartthrob:cart_info}

On the surface this appears to be nice because it keeps things within the core ExpressionEngine development process. And it would be except, well, CartThrob isn't abstract enough. Too many times I would try to do something that seemed obvious, like checking for shippable items within a users cart, and would come up short; that functionality doesn't exist.

Still though, don't get me wrong. Yes, some of the above sucks. Absolutely. But CartThrob is still under development. Plus, since it's a commercial script ($99) there's financial incentive to improve it. And, truth be told, for as painful as certain points were it's still way more fun to build a site with CartThrob than most of the other options I've used or reviewed like OpenCart or Avactis.

Plus, again, single freaking admin!

Recently, I came across a weird issue while developing an ExpressionEngine 2.0 module. The issue was that, for some reason, my module would output a random string instead of the intended content. For example, instead of displaying the HTML, that the module created, the module would display the string M00o93H7pQ09L8X1t49cHY01Z5j4TT91fGfr instead. Tracking down this issue was symptomatic of the proverbial needle in a haystack but I learned a lot about how ExpressionEngine's internals work and was once again reminded to be pragmatic about my development strategies.

My first and automatic reaction when coming up against some random and meaningless output string is to just Google it and see what happens. Unfortunately though, this particular search, at the time at least, returned around 35,000 sites that were outputting the string and only 1 that had a reference to the issue with a possible explanation. All of a a sudden this became an actual thing.

Putting aside the fact that there are around 35,000 sites with the string within Google for the moment (and that's really a problem to be discussed) it took a bunch of digging but I finally found a few things:

First, this is an issue that has been within ExpressionEngine for years. Doing a search on the ExpressionEngine site lists the first entry of this string at 2007 (sorry for the lack of a link; ExpressionEngine search is cached so old search URLs don't work sometimes). My initial reaction was that this was kind of sloppy; EllisLab updated ExpressionEngine to 2.0 recently, from the ground up to use CodeIgniter I believe, yet this issue got ported over?

After doing further research though, I came to believe the above is more to do with the design of ExpressionEngine than anything else like copying code. Turns out, there's some logic to that random and oblique string.

Basically, that string (M00o93H7pQ09L8X1t49cHY01Z5j4TT91fGfr) is a template holder for the content you want to output. It's a little complicated in how it works but thanks to Derek Jones over at the ExpressionEngine forums there's a succinct, though still a little complicated but less complicated than I could explain it, explanation of what's going on.

...

The template parser is an extremely complex gizmo, so let me try to explain it stripped of what techno mumbo jumbo I can. This will still be long, so bear with me.

The template parser intelligently saves on resources by only processing a given tag once, no matter how many times it occurs on the template. This means that if your tag is exactly the same, including its tagdata, EE only execute’s that modules code once, and replaces all copies of that tag with the output.

So first, when the template parser goes through the template finding tags, it replaces them all with temporary markers (the gibberish), and adds each tag to a list of tags to be parsed, in the order they are encountered on the template, top down. Each marker starts with the letter “M” followed by the number (starting at 0) that tag is sitting in on The List.

Second, the parser goes down its list of tags. It parses tag #0, and then replaces all of #0’s marker with that tag’s output.

——————-

Ok that’s the setup. Here’s why it affected you, and why this side effect is not a problem (and also extremely, extremely rare). Remember, top down, tags will be assigned 0, 1, 2, etc. So I’m abstracting it out to show the nesting and the positioning only. Just focus on the numbers.

{exp:tag0}foo{/exp:tag0}
{exp:tag1}
    bar
    {exp:tag0}foo{/exp:tag0}
{/exp:tag1}

Step 1, the tags are grabbed and replaced with markers. The first tag encountered is replaced first.

M0
{exp:tag1}
    bar
    M0
{/exp:tag1}

And then the next tag:

M0
M1

All tags have been collected and put on The List, so step 2, the parser goes down The List, and parses tag #0:

foo
M1

See the problem? The second M0 is sitting protected inside the marker for tag #1, so it doesn’t get replaced. Then tag #1 gets parsed.

foo
    bar
    M0

And the marker is left in the output. EE doesn’t go back and parse tag #0 again, it’s already done that once, and will not do it again.

So the circumstances required for this problem to exist each are uncommon, and combined, very very rare:

1) Multiple exact copies of a given tag
2) One or more copies nested inside another tag (tag nesting itself is also rare, and usually not recommended)
3) The tag of which there are copies must sit at a position in the template above any occurrences of it being nested in another tag.

From the above it was easy to conclude that the template parsing stuff was breaking down in my module. Why, I had no idea, but at least I had a known starting point. While it was easy to conclude that my issue lay with the template parsing though, it was 100% off point for my particular issue which had to do with, of all things, the case of the module name. But, I digress.

The point is that for years now, and through multiple versions and rewrites, ExpressionEngine has consistently carried the same "issue". Understanding the intent of the design for the template parsing helps put some perspective on things: it's definitely a side effect of the particular approach. It bugged, and surprised, me that ExpressionEngine would have such oblique and useless output on such a widespread basis (35,000 site's with that string within Google is a bit much in my opinion) but the more I thought about it the more I came to understand the complexity of what was involved in being helpful when the issue cropped up. Aside from abandoning the approach what could really be done?

It's a nice reminder that development is often a balancing act between getting what you want and dealing with a counter issue. The ExpressionEngine team developed a strategy to enable them to effectively and efficiently meet their requirements but there's sometimes an issue; a random string gets output instead of the template data. Plus, since the cause could be for anything from bad module installation data (as I had) to an issue within a template there's no real way to provide helpful error messages when the template parsing does fail. But, on the other hand, the template parser does exactly what it was designed to do (and it does it well).

Which would you choose?

I just pushed an update to the WordPress plugin repository for the latest update to Wp-Click-Track. It's been around 9 months since I've had a chance to even look at my own stats with Wp-Click-Track much less fix a couple bugs that have cropped up since but I finally found the time. And then, after spending 10 whole minutes looking into the cause of a couple bugs when I found that 3 lines of code fixed all of them.

3 little lines of code to fix the timezone bug, disappearing stats bug and random timestamp issues.

Also, please keep in mind that Wp-Click-Track is a labor of love; I work full time as a freelance programmer and try to keep a healthy social life so sometimes my priorities are a little wacked. Point being, I try to stay on top of these things but sometimes life gets in the way. But, I will get to it I can promise that.

When I first started using the Zend Framework I didn't give much thought into the way it handles routing URLs to the controllers/action combo. Honestly, while I had become used to so called search engine friendly, or pretty, URLs, I grew up on query strings where the idea of URLs was more about function than form. So when I moved over to Zend Framework as a framework I didn't give a damn how the URLs were parsed and routed to which controller/action. I just accepted the natural order of things.

Recently though, I took on a project to build a sort of quick and dirty CMS with a couple unique requirements:

1. It had to use the Zend Framework
2. URLs would be broken down into a /category/page/subpage scheme.

This was the first time I've ever had a client demand a project use the Zend Framework. Pretty cool. But, this presented a problem because, as mentioned above, by default the Zend Framework has URL schemes based on the controller and action you're looking to access like /controller/action/. This just wouldn't work.

Since the clients URLs would mostly be based on user generated strings, what we were calling URL slugs, the default routing scheme just wouldn't do. I say mostly because there were other areas of the project that would use the normal scheme for a couple modules. Either way, thankfully, using Zend_Controller_Router_Route made this pretty much a snap (a little head bashing but not too much).

First though, a little background. Because of the above requirement from the client on URL schemes, I was thinking the best thing to do was to route all user generated URLs to to a single controller and handle the rest from there. I looked but there didn't seem to be any easy way to route all URLs except the specific modules I wanted to exclude which meant I had to pull out the categories and iterate over them to create each route individually. (If anyone knows of a way to reverse the design and create a route for everything but certain strings let me know.)

To do this, in my bootstrap class I just created a new _init function like the below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

//bootstrap
public function _initRoutes()
{
	$url_data = $this->cat->getCategoryUrls();
	$frontController = Zend_Controller_Front::getInstance();
	$router = $frontController->getRouter();
	foreach($url_data AS $url)
	{
 
		$route = new Zend_Controller_Router_Route (
			$url.'/*',
			array('controller' => 'pages',
			'action' => 'index')
		);
 
		$router->addRoute($url, $route);
	}
}

In a nutshell, all the above is doing is looping over an array of category URL slugs and adding a custom route for each category that sends all requests to the Pages controller and the Index action. From there I parse the URL and proceed accordingly for the rest of the module.

For this example I'm doing a couple thing; using the Zend_Controller_Router_Route class and using wild cards (*). The Zend_Controller_Router_Route class takes two parameters, the first being the route and the second being an array of options (I'm only passing the controller and action but it's possible to send others). For the route, and because it's using the wildcard (*), I'm basically saying take any requests for the "url_slug" and anything after it and use the controller "Pages" and the "Index" action to handle everything.

After passing the Zend_Controller_Router_Route object to the $router, passing the unique name for the route as the first parameter, I was good to go.

That said, there's a lot more that can be done with the custom routing stuff within the Zend Framework above what's outlined above. There's a pretty good article by Jason Gilmore that goes into some detail into how to create default URL parameters to make the URLs look even prettier. Definitely worth checking out.