Made of Everything You're Not

About a week ago I was sitting in my office when I needed to find an email I had sent the previous week. As anyone would I go into Outlook (2003) when I'm presented with the below dialog box.

Which, basically, told me that my OST file was corrupted and I should run Scanpst.exe against the file to repair it. "OK", I thought, "simple...".

Where does outlook 2003 store ost files?

My computer had other thoughts though. First, I couldn't find Scanpst.exe anywhere but after about 20 minutes of using Windows freaking awesomely great searching tool I found it.
On Vista (Service Pack 1) Scanpst.exe location was in:

C:\Program Files\Common Files\System\MSMAPI\1033

During the research for the issue I found that it could also be in any of the below directories too:

C:\Program Files\Microsoft Office\Office12
C:\Program Files\Common Files\System\MSMAPI\1033

Now that Scanpst.ext was found I tried it but it only works on PST files, not OST files like I needed so I ran Scanost.exe instead. It returned this error message:

15:44:02 Beginning offline folder file integrity check.
15:44:02 Performing initial synchronization.
15:44:02 Synchronizer Version 11.0.8200
15:44:02 Synchronizing Mailbox 'Eric Lamb'
15:44:02 Synchronizing Hierarchy
15:44:02 1 folder(s) updated in offline store
15:44:02 Synchronizing server changes in folder 'Calendar'
15:44:02 Downloading from server 'my.domain.local'
15:44:02 Synchronizing server changes in folder 'Drafts'
15:44:02 Downloading from server 'my.domain.local'
15:44:03 Synchronizing server changes in folder 'Inbox'
15:44:03 Downloading from server 'my.domain.local'
15:44:03 Synchronizing local changes in folder 'Sent Items'
15:44:03 Uploading to server 'my.domain.local'
15:44:03 1 item(s) updated in online folder
15:44:03 Downloading from server 'my.domain.local'
15:44:03 Terminated in error
15:44:03
15:44:03 The client operation failed.
15:44:03 Microsoft Exchange Server Information Store
15:44:03 For more information on this failure, click the URL below: REMOVED FOR SPACE

Shit...

So I had to run Scanpst.exe against my Inbox.ost file whether it wanted to or not. I had to change the View Type to OST to see the OST files.

Wasn't sure if this would do anything though; just an educated guess because Scanost.exe failed to do anything.

After about 30 minutes it let me know it had found errors and whether I wanted to fix. 10 minutes later and I was good.

One of the coolest features of Xdebug is the Profiler. By enabling the Profiler you can generate what are essentially full traces about your php scripts that are invaluable in fine tuning your code and isolating problem areas effectively.

According to the official page on the xdebug Profiler:

Xdebug's built-in profiler allows you to find bottlenecks in your script and visualize those with an external tool such as KCacheGrind or WinCacheGrind. Xdebug's Profiler is a powerful tool that gives you the ability to analyze your PHP code and determine bottlenecks or generally see which parts of your code are slow and could use a speed boost.

There's some serious information in these little files too. By importing the debug files into a tool called WinCacheGrind you can view the files in a format that won't make you stick a knife in your eyes.

Here's a small example of the exported data:

version: 0.9.6
cmd: C:ProjectFilescat_v2index.php
part: 1
 
events: Time
 
fl=php:internal
fn=php::define
3 7
 
fl=php:internal
fn=php::get_include_path
4 6
 
fl=php:internal
fn=php::set_include_path
4 5

To get started first make sure you have Xdebug 2.0 installed. To check, execute the below in a dos prompt:

php -v

You should see something like the below:

PHP 5.2.6 (cli) (built: May  2 2008 18:02:07)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend Technologies
with Xdebug v2.0.3, Copyright (c) 2002-2007, by Derick Rethans

If not, you'll need to download and install Xdebug.

If you do have Xdebug, modify your php.ini to include the below after the Xdebug init call:

xdebug.profiler_enable = 1
xdebug.profiler_output_dir = "C:/path/to/profiler/storage"

Once you've configured php you might need to restart your server depending on how php is implemented. Either way, to generate a profile output just execute a script; the output should be written to the directory you set in the xdebug.profiler_output_dir.

All that's left is to open the profile output in WinCacheGrind.

Notes:
The profilers seems to only record data if the script is executed from the command line.
Download WinCacheGrind

It's pretty sad how a single bad choice can make for a REALLY bad product. In designing the project the developer's responsible for keeping the integrity of the project intact. There are all sorts of micro choices the developer has to make and if they aren't good the project suffers. This was made very apparent after looking at Mambo Wiki...

My team was recently tasked with building an Intranet for one of our clients. If you're not familiar with the term an Intranet is just a website that lives on an internal network; they're, generally, not publically available. Now, the client just wanted your basic, generic, CMS with a Wiki component and a full administration panel for setting up users and groups. Since we didn't have to work with an existing code-base we decided to take a look at a few OSS projects. Long story short; we settled on Joomla (this was a HUGE mistake).

As a quick aside I have to mention just how crappy the Joomla extension code is. If you thought WordPress was bad take a look at Joomla. Just one of the worst designed applications I've ever seen.

Anyway, Joomla met all of our requirements for a program except for the wiki component. While we were researching a platform we took note of the extensive module, plugin and extension community and were impressed at how much code was out there. There were a couple a couple wiki extensions out there but most were commercial, which we just didn't have the budget for (and philosophically, WTF would I pay for any code for?), though there was one promising extension that seemed to fit the bill; Mambo Wiki.

First off, I want to apologize for what I'm about to do. I'm sure the developers of Mambo Wiki are smart and capable but... well... yeah...

That being said, Mambo Wiki is, hands down, one of the worst programs I have ever seen. Billed as an integration of MediaWiki and Joomla the design of the program actually prohibits usage of the program.

Let me be clear; the design of Mambo Wiki destroys any practical usage of the program as a wiki. How is that possible? By placing the wiki within an iFrame.

First, think about how and why you use a wiki. For me I usually go to a Wiki through a search or because someone sent me a link to a page. By stuffing the wiki into an iFrame this is impossible. The only way to use it is to browse which makes for a very simple wiki with very little content before it becomes impractical.

I'm really at a loss to figure out how this could be ok for the developers. I mean, they're obviously skilled coders and they're pretty active on the official site for the module yet the project could double as an example of what not to do (which I'm doing...).

Maybe it was pure stubbornness. They wanted to use MediaWiki and, dammit, they were going to do it whether it was a good call or not. God, I hope not, but like I said, I'm really at a loss on the "why".

There's a lesson in this though; make sure you don't half ass the design. Be humble. If it looks like your design is going to hurt the usability of the program walk away and start over. It's much better to never publish than to publish crap.

I was curious about the impact of the directory separator and how that affects cross platform compatibilitye ver since I started developing on my Vista machine.

I've noticed how different they are compared to the Linux variety. Here's an example:

File path on *nix:

/path/to/file

File path on Windows:

C:\\path\to\file

As you can see the separator is a slash, either a "/" or "" depending on the OS. Why does this matter? Because if we're dealing with files on the server, and the path is going to be recorded so they can be displayed on the Internet, through a URI, than this is going to introduce compatibility issues if not handled correctly. Sigh..

I've seen open source programs handle this type of issue through concatenation. For example the code:

<?php
//$path = 'C:pathtofile.php'; Windows
$path = '/path/to/file.php';
?>

Is written like:

<?php
//$ds = ''; Windows
$ds = '/';
$path = $ ds.'path'.$ds.'$to.'file.php';
?>

Is this even an issue anymore? Both Firefox 2.0 and IE 7 render HTML assets whether the slash is one way or another. Maybe if all paths are written like the *nix variety it's all good...

I did notice that when you have a dual slash Windows chokes:

<?php
$path = '//path/to//dir';
?>

Just something to think about...

UPDATE::
According to the php manual:

On Windows, both slash (/) and backslash (\) are used as directory separator character. In other environments, it is the forward slash (/).

With that in mind it seems kind of dumb to even worry about this anymore. Just use the *nix variety and you should be fine.

There were 2 companies under consideration for an upcoming project to help with their email marketing business; MailChimp and Emma. I thought other people might find this useful. Below is a breakdown of each.

MailChimp vs. Emma

MailChimp

MailChimp has over 15,000 customers, manages over 65,000 opt-in lists containing over 75 million subscribers, and we deliver millions of emails a day. According to their site they:

strive to make MailChimp easy and affordable enough for a small business to get started, but powerful enough for a large company that's looking for an enterprise level solution. That philosophy is why we attract so many organizations from all over the globe, ranging in diversity from Mozilla Firefox to Chumby, from PeachPit Press to Harvard University, and from American Airlines to Dee Snider's House of Hair. The common bond is they all want a powerful email marketing solution that helps them get their work done.

Pros:

Tracks Bouncebacks
Tracks Open Rates
API
Very fast return on contact requests
Detailed reporting
Slick AJAX interface

Cons:

Click track links aren't white labeled
Requires templated unsubscribe
Email headers are not white labeled

Cost:

For lists ranging from 400,001 to 500,000 recipients, you can send 4 million emails monthly for $1,860.
For lists ranging from 500,001 to 600,000 recipients, you can send 4.8 million emails monthly for $2,220.

Emma Mailing

Emma is a Web-based service that includes everything you need to manage your email marketing and communications from start to finish. It's a unique platform that combines easy self-serve features, a custom-designed brand template, and personal assistance whenever you need it. You might say Emma's changing the face of self-serve email marketing. Seriously, try saying it out loud now.

It should be noted that I wasn't able to perform a hands-on analysis due to time constraints so a lot of this info is from Emma directly. Take with a grain of salt.

Pros:

API
Tracks Bouncebacks
Tracks Open Rates
Tracks click tracks
100% white labeled system
Client interface

Cons:

Custom templates have to be designed by Emma
Lengthy delay in contact requests
Emma requires design options for all internal pages.

Cost

$2500 Custom Licensing one time cost (includes one free sub account, and a free account for a Non-profit (up to 5000 emails/month)
$100 For each additional client account to cover setup/licensing
$250 unlimited Remote Sign-up API
$375 for web services if you needed it.

For up to 5 Million emails Emma charges $.00200 per email. So, 4 million emails would cost 8,000, which is billed retroactively. Send 1 million one month, 6 million the next? The price will fluctuate accordingly.

They provide unlimited storage for email addresses at no charge (ahem....good stuff!) and our delivery team makes sure it de-duplicates when you send it.

Summary

As you can see from the above both companies are pretty comparable in terms of features and pros vs cons breakdown. The biggest advantage that stands out is the white listing option available from Emma. This will allow you to keep your clients in the dark over your methods and allow you to build out your email marketing business.

If you're just looking for a simpler solution but still want the ability to send bulk MailChimp would probably be best.

I've been reading a lot of blogs about reinventing the wheel lately so, in a bit of "Me too! Me too!", and because it's an issue I feel passionate about, I have to weigh in.

The idea of not recreating something that's already been done is especially poignant in programming because our job requires invention. We get a task that involves the development of some piece of code that, hopefully, fixes a problem or meets some need. And there is almost always a deadline. Usually, one that is far from reasonable much less realistic.

With circumstances like that it makes sense you'd want to save as much time as possible by taking advantage of existing solutions. And boy, is there ever a bunch of solutions out there; almost always represented by 3rd party code. I have yet to run into a situation where I wasn't able to find some sort of help from another library or full program.

Take for example a simple request that came across my desk a few months ago. One of the producers at StreetWise came in with a blog site project. After getting the details it became apparent that taking advantage of an existing framework like Wordpress made sense; the client didn't have any expectations of adding additional functionality outside of a basic CRUD app with a permission system so we were good to focus on other things. Unfortunately, this is the exception not the rule. Also, there was the plugin system of WordPress so when the client, eventually, realized the need to update and or upgrade it'd be easier.

Problem

The problem comes in when the developer takes the idea of never reinventing the wheel literally. When this happens growth stops and progress in the craft of programming stagnates. I've seen too many developers rely on external libraries and they can't do anything without them (much less explain them). That process makes for a one dimensional developer who creates crappy products that are painful to maintain and upgrade.

It's especially troubling in the php community because of the sheer number of OSS projects out there utilizing php and MySQL. Need a forum? Use phpBB or Invision Board. Oh, a Blog? Take a look at WordPress, Expression Engine or Serendipity. Survey system? Check out phpQuestionaire. And those are just the popular ones; there are tens of thousands of programs out there for free use.

What you end up with is an amalgam of disparate parts that is the picture of a maintenance nightmare. Seriously, close your eyes and think about the worst possible maintenance scenario; I bet you there's at least a few external libraries being used. Anyone who has ever had to maintain one of these monstrosities will tell you how devastating this approach can be.

My first exposure to maintaining a program using the above approach was like falling in a frozen lake. The code in question was 80% custom with a phpBB forum attached that was heavily modified. phpBB was locked at version 0.9(!!) and hadn't been updated, ever. This created a mess of the database schema and codebase that required a few different philosophy changes and switches. Just a fucking headache to work on. There was code and data duplication everywhere.

A lot of beginners go this route; they're mystified by the possibility of being able to create large programs with little effort. Unfortunately beginners rarely, if ever, take maintainability into account.

It should be said that taking advantage of existing code also means you shouldn't touch it. Ever. Woe betide those developers who modify 3rd party code. Seriously, don't modify existing code if you want to painlessly upgrade an application.

Solution

So, if you shouldn't modify 3rd party code, and if using it can increase maintenance woes, why is there even code out there to use? Because of examples like in the beginning of the post; if your project's needs are within the parameters of the code you're looking to use it's a good idea to use it. Just don't go mixing and matching different code bases.

The only reason I knew the client wouldn't need additional functionality in the example above outside of a basic blog was because I've written blog scripts before. I know what makes up a blog at the basic to advanced levels. This makes interpreting project criteria and minutia a lot easier. If you've already done something you're more capable to make those calls.

I may have implied earlier that only beginners use 3rd party code. That isn't accurate; I use 3rd party code all the time. I'm just more selective about what I use and when and where I do it. Unless I plan on using the full application as a framework or if I'm not confident enough in the code to handle upgrade scenarios I just don't use full applications in my projects. You can be sure that if I use a program like phpBB, WordPress or phpQuestionaire (among many, many, others) it's because I know them inside and out. Issues won't... (clear throat)... be an issue :/

Yet, if I'm not using a full 3rd party project I'm still using 3rd party code. Taking a look at the codebase for my latest project I see the following OSS code:

1. Smarty
2. HTML_QuickForm
3. HTML_Table
4. Geshi
5. Snoopy
6. phpMailer
7. easyRSS
8. phpexifrw
9. etc, etc, etc...

The above are php classes that handle some of the bells and whistles for the project. All have excellent reputations and I am intimately familiar with all of them so any issues can be mediated efficiently. Extend each of the above classes in a wrapper and take care with what you do and making upgrades when they come out is pretty painless too.

It's also important to note that none of the above classes are for the core functionality, just ancillary features. I agree with Jeff Atwood who, in his post Programming Is Hard, Let's Go Shopping!, talks about why he went through all the trouble of writing his own text sanitizer:

But here's the thing: deeply understanding HTML sanitization is a critical part of my business. Users entering markdown isn't just some little tickbox in a feature matrix for me, it is quite literally the entire foundation that our website is built on.

That's the best explanation of why not to use 3rd party code I've heard yet. Outside of that though, just use it in moderation and don't be afraid to take control of the project.

I spent this Christmas with some family doing the usual holiday activities of drinking and merriment. Getting together with family really does recharge (so long as it's short enough; this year it was). As usually happens, I was conscripted to help everyone with their new electronics; ipods, digital cameras and the like. This always turns into a full blown "thing" where first I have to figure out how the gadget works then, as inevitably happens, as soon as I hook the gadget up to the computer I notice it's running really slowly. This always leads to five hours spent trying to clean out spyware and viruses. Usually, there's the standard fare of spyware and a couple viruses but this time I walked away feeling my family could, almost single-handedly, destroy the Internet.

Now, I run the IT department as my day job so I've seen some messed up computers. I don't want to go into too many details so suffice it to say that you can learn a lot about someone by cleaning up viruses on their computer. Anyway, the first thing I noticed on my families computer this time was that automatic updates was off, their anti-virus protection was disabled and the firewall was turned off. My first thought; they're completely screwed.

It took me the better part of the day cleaning everything up; I did it in the usual method of start a process, have drink and laughs, start next process, repeat.

During the cleanup, and the drinking, I started thinking about what to do about the larger issue of making sure they remained secure. I've tried, multiple times, to explain to my family the importance of maintaining their computer(s), but every time I return for another visit I find the pattern above repeating.

I've gone through the entire spiel about how their computer is acting as a hub on a botnet. How, when they hear stories on the news about piracy and spam their computer is usually at fault too. I get the same reaction as when I try to explain my day job; blank eyes and a nodding head but no real comprehension. Obviously, they just aren't computer people.

Which is really the problem, though not really. Here's what I mean:

The problem is that, while the dumbing down of computers for "regular" people brought them into the mainstream the lack of any type of education about security and management created the perfect melting pot for this type of fiasco.

See, regular people just don't care. As far as their concerned if the computer works than WTF is up with all this NAGGING ABOUT VIRUSES AND SPYWARE! I JUST WANT TO USE THE STUPID THING!!! GOD! So long as their computer can access the Internet and they can write email everything is just hunky dory.

And that's a problem because as we all know, computers require maintenance and attention.

The moral?

Computers are complicated creations, programmers, geeks, nerds and IT folk know this; Granny, unfortunately, does not and she doesn't care to. It's important that we instill common sense into our loved ones even if it means BEATING IT INTO THEM.

Tell them:

You need to run regular virus and spyware scans. Uninstalling unused programs is important. Defragging the harddrives is good. Regular backups? Yeah, that too.

Anyway, sorry for the slow Internet and all the spam. I'll be seeing my family around Easter and I'll get everything working around then.

One thing I've been waiting to become practical for a while is video mobile blogging (VMB). In case you don't know, VMB is the practice of taking a video with a mobile device and automatically adding it to a blog or website. It's pretty much exactly like a moblog except instead of sending photos you send a video. Due to poor quality, varied video formats and slow network speeds VMB just hasn't become mainstream yet.

The general idea behind a VMB works like this:

1. Record video
2. Send email with video as attachment
3. Script parses email and extracts the video
4. Process video for online play
5. Extract all meta data
6. Record in database

Really, a basic idea; works GREAT for a moblog . The reason's pretty simple; ever since phones started coming with cameras as a standard accessory they've improved in quality and couple that with a data plan (so email can be sent) and it's a perfect fit.

The same thing can't be said for video though. Video on a cell phone is just too new and, well, clunky to be useful in business level applications. In fact, RIM just recently upgraded the Blackberry Curve OS to include video playback a couple of months ago (yeah, I have a Curve).

Problem is, the quality on the video sucks and it takes me about 15 minutes to send the video as an email on an EDGE network. Now, I'm pretty sure my speed isn't a good baseline; I have AT&T so my service downright sucks. That being said, I did do a couple tests on a couple of my friend's phones from various vendors and even though the speed was a little faster the quality was still crappy.

The above doesn't really matter though because, according to friend I have in the mobile industry, there are about 20 different formats the different phones use to encode videos in. This is a biggie; embedding a video into a webpage requires that videos be encoded to the lowest common denominator. Since phones aren't using a standard format the videos would have to be converted to a standard format on the server which increases the complexity and cost of such a platform.

It should be noted that a lot of phones are encoding their videos using the 3GP format; but because it's not common enough to be a standard codec on users computers I don't consider this a full solution.

I've been working on a new code base for StreetWise these past couple months, getting ready for the upcoming busy season, and as this new code needs to support membership of 1,000,000+ people decided I wanted, no, needed, to rework my pagination algorithm.

The Problem

Previously, I've gone for a basic approach of just listing out the links sequentially with a Next and Prev state. For larger page sets, I would just remove the numeric links. For example they would look like the below:

1 2 3 4 5 etc...

This becomes problematic at higher numbers though; just imagine the page would look if there were 1,000 pages instead of five. Here's an example of just 100:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

So it can be a bit of a problem.

The Replacement

A lot of websites are using a better pagination style that's sometimes called Wordpress pagination. This pagination style presents only a few links at a time while still allowing more freedom than the basic Next Prev style. Here's an example:

« Previous 1 2 3 ... 10 11 12 13 14 ... 118 119 120 Next »

The Solution

Being it's called Wordpress pagination, and that I'm a lazy php programmer, I just stole the code from Wordpress and translated it into something a little more portable.

function paginate_links($format = '?page=%#%', $total = '1', $current = '0', $base = '%_%', $show_all = FALSE, $prev_next = TRUE, $prev_text = '« Previous', $next_text = 'Next »',$end_size = '3',$mid_size = '2',$type = 'plain') {
 
	$total = (int) $total;
	if($total < 2) {
		return;
	}
	$current  = (int) $current;
	$end_size = 0  < (int) $end_size ? (int) $end_size : 1; // Out of bounds?  Make it the default.
	$mid_size = 0 <= (int) $mid_size ? (int) $mid_size : 2;
	$r = '';
	$page_links = array();
	$n = 0;
	$dots = false;
 
	if ( $prev_next && $current && 1 < $current ){
		$link = str_replace('%_%', 2 == $current ? '' : $format, $base);
		$link = str_replace('%#%', $current - 1, $link);
		$link .= $add_fragment;
		$page_links = "<span class='page-numbers current'>$n_display</span>";
			$dots = true;
		} else {
			if ( $show_all || ( $n <= $end_size || ( $current && $n >= $current - $mid_size && $n <= $current + $mid_size ) || $n > $total - $end_size ) ) {
				$link = str_replace('%_%', 1 == $n ? $format : $format, $base);
				$link = str_replace('%#%', $n, $link);
				$link .= $add_fragment;
				$page_links = "<span class='page-numbers dots'>...</span>";
				$dots = false;
			}
		}
	}
 
	if ( $prev_next && $current && ( $current < $total || -1 == $total ) ) {
		$link = str_replace('%_%', $format, $base);
		$link = str_replace('%#%', $current + 1, $link);
		$link .= $add_fragment;
		$page_links[] = "<a class='next page-numbers' href='" . clean_url($link) . "'>$next_text</a>";
	}
 
	switch ( $type ) {
		case 'array' :
			return $page_links;
			break;
		case 'list' :
			$r .= "
<ul class='page-numbers'>\n\t
<li>";
			$r .= join("</li>
 
\n\t
<li>", $page_links);
			$r .= "</li>
 
\n</ul>
 
\n";
			break;
		default :
			$r = join("\n", $page_links);
			break;
	}
 
	return $r;
}

As you can see, especially if you're already familiar with Wordpress, all I've done is take out the Wordpress specific functions and set the $args to function parameters. Pure theft

There are a bunch of paramaters and options you can configure the links by but only the first 3 are really needed to make the function work.

Here's an example:

$numofpages = 20;
$page = '4'
$query_string = 'foo='.$foo.'&bar='.$bar;
echo paginate_links('?'.$query_string.'&page=%#%', $numofpages, $page);

I didn't put too much work into this, which is kinda the point, but I hope it's helpful!

I've updated the repository with the latest version tonight.

This release includes a couple bug fixes and a few improvements.

Improvements

• Added Indexes on tables for better performance
• Updated Install system
• Added pie charts for clicks by day and hour

Bug Fixes

• fixed hour click report formatting
• fixed blank link haunting
• fixed option saving

Screenshots