Made of Everything You're Not

I recently had a project where there would be 2 databases being used together; one locally and the other hosted externally by the client. Using traditional PHP this wouldn't be a problem (just pass a link to the database I want to use on a per query basis and I'm done), but, since this project was to use the Zend Framework and I'd never attempted this sort of thing before with it, I was in quite the pickle.

Doing some quick Googling turned up an article that seemed to fit the bill: Zend Framework: Connecting to 2 databases written by Wenbert Del Rosario. Unfortunately though, while it was a good read and very informative, my circumstances were a little different. The database I had to connect to was hosted on an external network that I wouldn't have local access to. I had to go over the wide Internet to connect. Using Wenbert's tutorial would entail a constant connection on every request whether it needed it or not and that just wouldn't do.

Anyone who's ever had to connect to an external database can tell you it's far from an ideal scenario. The latency involved with this strategy is noticeable and caching is pretty much required for any sane strategy. Still, Wenbert's article was definitely helpful for developing my strategy.

As in Wenbert's article I started with putting the connection information into my config.ini file

//config.ini
resources.db.adapter = PDO_MYSQL
resources.db.params.host = localhost
resources.db.params.username = ****
resources.db.params.password = ****
resources.db.params.dbname = local_db 
 
externaldb.adapter = PDO_MYSQL
externaldb.params.host = example.com
externaldb.params.username = ****
externaldb.params.password = ****
externaldb.params.dbname = external_db

Then, I created a standard Zend_Db_Table class that handled the connection:

<?php
//Model/DbTable/External.php
//name made up to protect the client but do yourself a favor and name your shit logically
class Model_DbTable_External extends Model_DbTable_Abstract
{
   /**
     * Doesn't matter for use but ZF demands a $_name variable
     * @var string
     */
	protected $_name = "content";
 
	public function init()
	{
		$settings = Zend_Registry::get('settings');
		$this->db = Zend_Db::factory($settings, $settings);
 
	}
}
?>

With the above out of the way I can now connect to the external database. I just have to pipe all my SQL through the above class. To that end, and to provide a nice wrapper for the caching, I create a Model that has methods for all the SQL my app will need:

<?php
class Model_External extends Model_Abstract
{
	/**
	 * The key to use for the cache items
	 * @var string
	 */
	public $cache_key = 'external_db';
 
	public function __construct()
	{
		parent::__construct();
		$db = new Model_DbTable_External;
		$this->db = $db->db;
	}
 
	public function testSystem($id)
	{
		$key = $cache_key.__FUNCTION__.$id;
		if(!$data = $this->cache->load($key))
		{
			//just a random query with joins and whatnots to show  it's possible <img src="http://blog.ericlamb.net/images/smileys/smile.gif" width="19" height="19" alt="smile" style="border:0;" />
			$sql = $this->db->select()->from(array('e' => 'example_table1'), array('e.id', 'e.title'))->where('e.id = ?', $id)->limit('6');
			$data = $this->db->fetchAll($sql);
			$this->cache->save($data, $key, $this->cache_key));
		}
		return $data;
	}
}
?>

So, using the above class I've abstracted out the SQL and provided caching through the below class Model_Abstract:

<?php
//models/Abstract.php
abstract class Model_Abstract
{
	/**
	 * The database object
	 * @var object
	 */
	public $db;
 
	/**
	 * The Cache Object
	 * @var object
	 */
	public $cache;
 
	/**
	 * The stored cache name
	 * @var string
	 */
	public $cache_key = null;
 
	public function __construct()
	{
		$c = new Model_Cache;
		// getting a Zend_Cache_Core object
		$this->cache = Zend_Cache::factory(
	                    'Core',
	                    'File',
	                    array('lifetime' => 720, 'automatic_serialization' => true),
	                    array('cache_dir' => '/path/to/cache/')
		);
	}
}
?>

Putting it all together I now have a mechanism to connect to an external database, perform queries against said database and can cache the results pretty easily and with minimal code. One of the advantages of this approach is that connections to the database only happen when needed instead of on every page request as in Wenbert's article. So, for example, so long as the cached items exist then no database connection is created and no latency (outside of hitting the filesystem of course) is created.

Still though, there are ways to improve on this approach. For example, using APC instead of the file system for the caching would help as would increasing the cache lifetime from 2 hours to something more but, overall, the above should be a good starting point.

About a year ago I reviewed various shopping cart platforms for what was to be an "upcoming" project. After numerous delays and client hand holding (with the requisite back and forth) one year later that project finally began. And, as has lately become a trend in my professional life, the client's spec combined with a drastically shortened and accelerated deadline forced us to go with something completely unknown and with only cursory inspection; the 3rd party e-commerce ExpressionEngine module CartThrob 2.0 (beta at the time of this review BTW).

Of course this presented the problem of actually using an unknown package to build a site with. Thankfully though, while CartThrob has a steep learning curve and is most certainly missing key functionality, it's an overall nice package that allowed us to do what we needed. Sure, there's pain involved (lots of configuration, testing and set up) but with a little elbow grease it all worked out.

As mentioned, CartThrob is an ExpressionEngine module for managing an online store. It's available for both ExpressionEngine branches (1.x and 2.x) and was voted Module of the Year by Devot:ee in 2010 (for the 1.x version only). This being an ExpressionEngine project CartThrob would, in theory, be a nice fit.

One of the really nice things about using CartThrob is that it uses the ExpressionEngine admin panel for managing your store. This meant there wouldn't be a second administration site that the client would have to learn. (Personally, I always found dual admins to be janky at best and sloppy/lazy at worst so this was a definite bonus in favor CartThrob.)

But this is also double edged because, well, pretty much everything about your store is saved into the entries system within ExpressionEngine. This, of course, causes your Entries panel to become an eye sore whenever you have to do anything there. Instead of having a dedicated section to manage products you use the Entries section. Instead of a dedicated section to manage your orders you use the Entries section. Instead of a dedicated section to manage... well, you get the idea; everything is managed through the entries. This makes sense for the product management but when you have to dig through entries to find orders and coupons and discounts (etc) things get a little jarring for some people (even internally with my team) so it's a concern.

Still, single freaking admin so it was kind of a lesser evil compromise.

The unusual thing though is that, for how thoroughly the management of items was punted (though, to be fair, it looks to be a feature versus bug debate more than "punting"), CartThrob has an extremely deep configuration section where you can customize CartThrob to function just right. CartThrob has all the basic customization functionality one would expect from any modern shopping cart platform (tax, shipping, payments with lots of merchant options, etc) plus a bunch of settings just to get CartThrob working (CartThrob is NOT an out of the box solution). It's all presented in an easy to use interface that's designed to mesh with the ExpressionEngine administration panel design which makes for an easier experience.

As far as merchant accounts go (for accepting payments through your site) there's the usual suspects of Authorize.net, Paypal and SagePay built right in but also some offline solutions for pay by check and credit accounts. For this project we used Authorize.net which went smooth for the most part but I can't speak for the other options though if they're anything like the Authorize.net one they're probably cool. CartThrob also includes some extensions and hooks so writing your own payment gateway is possible if you need.

CartThrob can also hook into the ExpressionEngine member module so your customers can be registered internally. Like pretty much everything else with CartThrob there's a bit of configuration involved to get it up and working though. Unfortunately though, there's not really any out of the box functionality for your members to take advantage of. For example, there isn't any way for your customers to view their past orders when logged in and when using the control panel and viewing a member's profile there's no relationship with CartThrob. You're gonna have to write all that stuff yourself but, and I don't know why, but CartThrob doesn't store any relationships between ExpressionEngine members and it's customers so expect a challenge.

In terms of the user side, the actual storefront, everything is done using ExpressionEngine templates and template tags. There are lots of module methods available and even though it's far from a complete package of the expected functionality. In fact, on our team, we got into a habit of thinking of CartThrob as a Lego set for creating what we want instead of having what we want already.

For example, to view the details about a users cart you'd use the template code below:

{exp:cartthrob:cart_info}
	{total_items} 
	{total_unique_items} 
	{cart_entry_ids} 
	{cart_shipping} 
	{cart_tax_rate} 
	{cart_tax} 
	{cart_subtotal} 
	{cart_total} 
{/exp:cartthrob:cart_info}

On the surface this appears to be nice because it keeps things within the core ExpressionEngine development process. And it would be except, well, CartThrob isn't abstract enough. Too many times I would try to do something that seemed obvious, like checking for shippable items within a users cart, and would come up short; that functionality doesn't exist.

Still though, don't get me wrong. Yes, some of the above sucks. Absolutely. But CartThrob is still under development. Plus, since it's a commercial script ($99) there's financial incentive to improve it. And, truth be told, for as painful as certain points were it's still way more fun to build a site with CartThrob than most of the other options I've used or reviewed like OpenCart or Avactis.

Plus, again, single freaking admin!

Recently, I came across a weird issue while developing an ExpressionEngine 2.0 module. The issue was that, for some reason, my module would output a random string instead of the intended content. For example, instead of displaying the HTML, that the module created, the module would display the string M00o93H7pQ09L8X1t49cHY01Z5j4TT91fGfr instead. Tracking down this issue was symptomatic of the proverbial needle in a haystack but I learned a lot about how ExpressionEngine's internals work and was once again reminded to be pragmatic about my development strategies.

My first and automatic reaction when coming up against some random and meaningless output string is to just Google it and see what happens. Unfortunately though, this particular search, at the time at least, returned around 35,000 sites that were outputting the string and only 1 that had a reference to the issue with a possible explanation. All of a a sudden this became an actual thing.

Putting aside the fact that there are around 35,000 sites with the string within Google for the moment (and that's really a problem to be discussed) it took a bunch of digging but I finally found a few things:

First, this is an issue that has been within ExpressionEngine for years. Doing a search on the ExpressionEngine site lists the first entry of this string at 2007 (sorry for the lack of a link; ExpressionEngine search is cached so old search URLs don't work sometimes). My initial reaction was that this was kind of sloppy; EllisLab updated ExpressionEngine to 2.0 recently, from the ground up to use CodeIgniter I believe, yet this issue got ported over?

After doing further research though, I came to believe the above is more to do with the design of ExpressionEngine than anything else like copying code. Turns out, there's some logic to that random and oblique string.

Basically, that string (M00o93H7pQ09L8X1t49cHY01Z5j4TT91fGfr) is a template holder for the content you want to output. It's a little complicated in how it works but thanks to Derek Jones over at the ExpressionEngine forums there's a succinct, though still a little complicated but less complicated than I could explain it, explanation of what's going on.

...

The template parser is an extremely complex gizmo, so let me try to explain it stripped of what techno mumbo jumbo I can. This will still be long, so bear with me.

The template parser intelligently saves on resources by only processing a given tag once, no matter how many times it occurs on the template. This means that if your tag is exactly the same, including its tagdata, EE only execute’s that modules code once, and replaces all copies of that tag with the output.

So first, when the template parser goes through the template finding tags, it replaces them all with temporary markers (the gibberish), and adds each tag to a list of tags to be parsed, in the order they are encountered on the template, top down. Each marker starts with the letter “M” followed by the number (starting at 0) that tag is sitting in on The List.

Second, the parser goes down its list of tags. It parses tag #0, and then replaces all of #0’s marker with that tag’s output.

——————-

Ok that’s the setup. Here’s why it affected you, and why this side effect is not a problem (and also extremely, extremely rare). Remember, top down, tags will be assigned 0, 1, 2, etc. So I’m abstracting it out to show the nesting and the positioning only. Just focus on the numbers.

{exp:tag0}foo{/exp:tag0}
{exp:tag1}
    bar
    {exp:tag0}foo{/exp:tag0}
{/exp:tag1}

Step 1, the tags are grabbed and replaced with markers. The first tag encountered is replaced first.

M0
{exp:tag1}
    bar
    M0
{/exp:tag1}

And then the next tag:

M0
M1

All tags have been collected and put on The List, so step 2, the parser goes down The List, and parses tag #0:

foo
M1

See the problem? The second M0 is sitting protected inside the marker for tag #1, so it doesn’t get replaced. Then tag #1 gets parsed.

foo
    bar
    M0

And the marker is left in the output. EE doesn’t go back and parse tag #0 again, it’s already done that once, and will not do it again.

So the circumstances required for this problem to exist each are uncommon, and combined, very very rare:

1) Multiple exact copies of a given tag
2) One or more copies nested inside another tag (tag nesting itself is also rare, and usually not recommended)
3) The tag of which there are copies must sit at a position in the template above any occurrences of it being nested in another tag.

From the above it was easy to conclude that the template parsing stuff was breaking down in my module. Why, I had no idea, but at least I had a known starting point. While it was easy to conclude that my issue lay with the template parsing though, it was 100% off point for my particular issue which had to do with, of all things, the case of the module name. But, I digress.

The point is that for years now, and through multiple versions and rewrites, ExpressionEngine has consistently carried the same "issue". Understanding the intent of the design for the template parsing helps put some perspective on things: it's definitely a side effect of the particular approach. It bugged, and surprised, me that ExpressionEngine would have such oblique and useless output on such a widespread basis (35,000 site's with that string within Google is a bit much in my opinion) but the more I thought about it the more I came to understand the complexity of what was involved in being helpful when the issue cropped up. Aside from abandoning the approach what could really be done?

It's a nice reminder that development is often a balancing act between getting what you want and dealing with a counter issue. The ExpressionEngine team developed a strategy to enable them to effectively and efficiently meet their requirements but there's sometimes an issue; a random string gets output instead of the template data. Plus, since the cause could be for anything from bad module installation data (as I had) to an issue within a template there's no real way to provide helpful error messages when the template parsing does fail. But, on the other hand, the template parser does exactly what it was designed to do (and it does it well).

Which would you choose?