Made of Everything You're Not

A couple weeks ago I received an email from a client of mine about a bug one of their clients was having using the Mailpress WordPress plugin and wanting to know if I could help. They’re an agency and I always want to make them happy so, even though I didn’t write Mailpress, I decided to dive in a see what was up. Plus, it’s always fun to contribute to open source projects and to get paid to do it is always a win-win.

Mailpress Email Validation Bug

Before getting into the bug I just want to say that I didn’t want to post it in this way; ideally there would be channels available to submit issues but Mailpress doesn’t exactly make that easy. Their site, while having links to the expected destinations like Community and Submitting a patch, doesn’t appear to be finished and those sections are essentially empty at the moment.  The information to put this information out there very well might be in the site but, frankly, the thought of writing this post was less painful than digging through the site looking for info. Plus, this isn’t a security issue at all so there’s that. Ass == Covered.

The issue was that the email validation was returning false even when an email was valid, specifically if the email wasn’t entirely lowercase. The problem with that, in case it’s not clear, is that an email address doesn’t have to be lower case (at least in the name portion). For example the below two emails are valid and, in fact, different:

eric@example.com
Eric@example.com

They look similar and it’s not really advisable to do email addresses in that format but people do it that way and, technically, it is allowed so not sure why Mailpress doesn’t.

Mailpress would throw an error on the second email which was pissing of my client’s client and my client (sigh…). The fix is pretty stratightford and easy; just replace the regular expression in Mailpress with the working one I cribbed from Zaheer.

File: “/wp-content/plugins/mailpress/mp-admin/js/write.js”

219

is_email : function(m) { var pattern = /^[_a-z0-9-]+(\.[_a-z0-9-]+)*@[a-z0-9-]+(\.[a-z0-9-]+)*(\.[a-z]{2,4})$/; return pattern.test(m); },

With:

219

is_email : function(m) { var pattern = /^[a-zA-Z0-9._-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}$/; return pattern.test(m); },

Hopefully, the issue doesn’t go deeper than the javascript validation but the above does allow for a working email validation script. Now we just need Mailpress to update their wonderful plugin with the fix…

Dreamhost is slandering me! They think I suck and have no problem telling people as much. Seriously; I write bad code by them. How do I know? Because Chris Duke over at AppModo told me so. Problem is, Dreamhost was 100% right. Yup; I fucked up.

Wp-Click-Track 0.7.1

On Monday I got an email from Chris asking for some help with one of the WordPress plugins I wrote; wp-click-track. Chris was cool enough to let me post the message he received from Dreamhost:

Hello,

I’m writing you about your database “appmodo”, please write me back as soon as possible.  There seems to be a plugin you are running that is poorly coded (missing table joins) and this 1300+ second query that it is running is causing high server load spikes. If this plugin is not fixed, or disabled, and it causes server problems, we may need to disable the wp_tracking_clicks and wp_tracking_links tables.  You can write me at justin@dreamhost.com.

The problem queries:

# Query_time: 2181  Lock_time: 1115  Rows_sent: 1  Rows_examined: 38703 use appmodo;

SELECT date_format(click_date,"%Y-%m-%d") AS first_click,
date_format(click_date,"%j") AS day_of_year FROM wp_tracking_clicks tc,
wp_tracking_links tl WHERE tc.click_id != '0'  GROUP BY first_click ORDER
BY first_click DESC LIMIT 1;

# Query_time: 1995  Lock_time: 924  Rows_sent: 1  Rows_examined: 38703 use appmodo;

SELECT date_format(click_date,"%Y-%m-%d") AS first_click,
date_format(click_date,"%j") AS day_of_year FROM wp_tracking_clicks tc,
wp_tracking_links tl WHERE tc.click_id != '0'  GROUP BY first_click ORDER
BY first_click DESC LIMIT 1;

# Query_time: 1387  Lock_time: 309  Rows_sent: 1  Rows_examined: 38703 use appmodo;

SELECT date_format(click_date,"%Y-%m-%d") AS first_click,
date_format(click_date,"%j") AS day_of_year FROM wp_tracking_clicks tc,
wp_tracking_links tl WHERE tc.click_id != '0'  GROUP BY first_click ORDER
BY first_click DESC LIMIT 1;

# Query_time: 2016  Lock_time: 930  Rows_sent: 1  Rows_examined: 38703 use appmodo;

SELECT date_format(click_date,"%Y-%m-%d") AS first_click,
date_format(click_date,"%j") AS day_of_year FROM wp_tracking_clicks tc,
wp_tracking_links tl WHERE tc.click_id != '0'  GROUP BY first_click ORDER
BY first_click DESC LIMIT 1;

Thanks!
Justin K

Justin did a nice job highlighting what the issue was; I didn’t join the tables together so it would take a really long time to complete the scans if there were a lot of rows in the tables. Another reason to always develop with large data sets; something I obviously didn’t do with wp-click-track.

Your application is useful and popular. Your users love it. Your users love you. But over the next week, something curious happens. As people use the application, it gets progressively slower and slower. Soon, the complaints start filtering in. Within a few weeks, the app is well-neigh unusable due to all the insufferable delays it subjects users to– and your users turn on you.

Anyway, there was no excuse for that; this was shoddy on my part. The only explanation I have is poor quality on my part.

This particular bug affected the line chart on the admin dashboard. Ever wonder why that particular graph took forever to load? Yup, that’s why.

What’s funny, to me at least, was that I had already fixed this particular issue; I just hadn’t released it yet. This particular bug was planned to be released with 0.8 but 0.8 is taking longer to complete than I initially thought it would. So the bug fix languished and sat there. Unreleased.

So yeah, wp-click-track 0.7.1. It’s a good release to get.

I was working on a project recently that was using WordPress as the base platform and had a need for a scheduled task function. Ordinarily, this would be the type of functionality that I would just setup a cron job for; but since WordPress has a pseudo cron system in place I decided to investigate that as an option. Long story short; fuck that man.

Working With WP Cron

As mentioned, WordPress has a pseudo cron mechanism available to WordPress developers for scheduling tasks called WP Cron. WP Cron works using series of functions available for plugin developers to create their own cron style tasks. In theory WordPress makes this easy(ish) using the built in cron API but the reality is that it’s not only confusing and awkward to implement but once you’re complete you have to allow for a bit of latitude on the scheduling.

Take a look at scheduling a task in WordPress. Basically, it works by adding a function to your activation hook, within a plugin, that like the below:

<?php 
register_activation_hook(__FILE__, 'my_activation');
add_action('my_hourly_event', 'do_this_hourly');
 
function my_activation() {
	wp_schedule_event(time(), 'hourly', 'my_hourly_event');
}
 
function do_this_hourly() {
	// do something every hour
}
?>

Pretty simple implementation really though not very “cron” like in syntax. The important part is the call to “wp_schedule_event()” which takes 4 parameters (start date, recurrence, hook and arguments). According to the API manual the function:

Schedules a hook which will be executed by the WordPress actions core on a specific interval, specified by you. The action will trigger when someone visits your WordPress site, if the scheduled time has passed. See the Plugin API for a list of hooks.

The problem I had was that this function only allows for 3 options on recurrence; hourly, twicedaily and daily. I needed my function to run at a specific time. (It was EXTREMELY important that it ran during a specific interval.)

This isn’t to say that it’s not possible to schedule a task for a specific time; it’s just a pain in the ass. You have to write all the logic to determine when to add a new task and removing expired tasks. Ughh. To me the question becomes “Should I write a couple hundred lines of code or just add a single line to my crontab?”.

Crontab baby.

There’s been a nagging thought in the back of my head ever since I started writing WordPress plugins; the apparent lack of oversight in the WordPress plugin distribution system. I’m not trying to create hype around anything, I really do like WordPress, but I’m worried about the implications of zero oversight.

Lack of Oversight

It would be trivially simple to add an algorithm to a plugin to do all sorts of nefarious things. Off the top of my head I can imagine the following:

1. Send over a blogs comment data (email addresses) to an external server.
2. Add an admin account or change existing password
3. Hell, even deleting the entire database and files

Not a happy thought is it?

Attempting such a thing from a new plugin, with zero initial interest, would be nearly impossible because of the noise from all the other plugins. I say nearly impossible because there was recently a case of a plugin, called Pushit, which sent an email containing passwords to a gmail account upon execution. I first heard about this from a post on Mental Fruition asking the question “Are your WordPress plugins safe?“.

In doing some research on connecting WordPress to SMS gateways for some mobile action, I found what is either a careless remnant of testing or a malicious attempt to steal data from people using a WordPress plugin. Actually, make that carelessly malicious.

Interested in how the Pushit plugin for WordPress was working behind the scenes, I stumbled upon quite a shocker:
…

Yet towards the end of this send function, there’s a call to PHP’s mail function. It sends the following to smart.maxx@gmail.com:

• receiving number
• message sent
• username of SMS service account
• password of SMS service account
• the short number used
• the sender name/number to be displayed on the receiving mobile
• whether the SMS was sent OK

Apparently the authors of this plugin deem that this information is something that someone with the e-mail address smart.maxx@gmail.com should have about every SMS you, or your visitors, attempt to send using their plugin. Are you cool with that?

No, I’m not cool with that and I don’t know anyone who would be cool with that.

Thankfully, the Pushit plugin was only downloaded around 200 times before the issue was discovered. It should be noted that the developer of the plugin swears that it was an honest mistake; the mail() call was leftover code from testing apparently. This doesn’t invalidate the fear though.

No, the fear is from an existing, popular, plugin being compromised and distributed. This is the nightmare scenario.

Take wp-click-track (my most popular plugin) for example. To date, it’s been downloaded a couple thousand times and it’s been written up on a few sites. Were I evil, and my girlfriend says the juries still out on that , I could easily compromise quite a few sites with a simple update. Quite the scary thought isn’t it?

I know it would be nearly impossible to protect WordPress from malicious code because plugins are written in php and, because of that, have access to all the resources WordPress does.  PHP can’t be sandboxed so manual screening would be pretty much required.

The fact that there isn’t any manual screening to date is surprising though. WordPress is owned by Automattic, which has received at least 29.9 million dollars in funding so they have money. Just look at it: $29,900,000. That’s a lot of money so it’s pretty tough to understand the argument, if it was used, that there just isn’t money available for a person to provide manual oversight.

That is inexcusable.

I’m not saying it would be easy by any means, but if a company has 29, fucking, million dollars based on the back of a shoddy system there’s no reason why they couldn’t spend some money adding oversight to the plugin system.

As mentioned above, since WordPress plugins are written in raw php they have full access to all the functionality WordPress does so solving this issue programmatically would be nearly impossible. No, without using some sort of meta language, similar to Smarty, manual oversight is probably the only solution.

At the very least take 2-4 programmers, or, hell, one really talented programmer, and have them write a couple scripts to parse the plugin svn repository looking for suspicious function calls (rm(), mail(), exec(), etc) and algorithms (DROP table) and have the programmer intervene manually if anything suspicious is detected.

Heal thyself WordPress. Now.

It’s taken way too long to do but I’ve finally fixed a bunch of issues people have been experiencing. In fact, this was supposed to just be a bug release but I got a little carried away with the design and added in a couple overdue features.

First, The Bugs

Big Bug!! (by Denis Collette)

To be blunt, there were a couple nasty bugs in 0.3 that pretty much ruined the experience for a lot of people. I’m sorry for that. I swear, it worked on my blog

Anyway, the bugs that were fixed include 2 that prevented the system from tracking (Gary) and a bug that pretty much destroyed a users blog (Joerg) plus a one that prevented the comment body from being parsed as well as a couple spelling issues.

The Gary bug prevented the tracker from working because of a simple pathing issue. I didn’t take into account that people might have their blog in a sub-directory of the domain so my calls were going to the ether instead of the WordPress installation. So, you know, my bad…

The Joreg bug was a little more insidious and harder to track down though. It basically consisted of a couple issues, that when combined, caused a memory failure on the users blog. This is pretty much the worst case scenario when developing for other people (to destroy the system) and I, unfortunately, didn’t experience this myself so it only became apparent on other peoples blogs. This one bugged the shit out of me.

To fix I had to take content from Joreg’s blog and add it to my system. After I did that my computer instantly burst into flames. I didn’t know that was even possible but there you go…

The actual cause of the bug came from 3 different causes:

1. Email (mailto:) links
2. Anchor (#top) links
3. Missing (empty hrefs) links

Turns out that my regex skills were lacking because I didn’t account for any of the above. To fix I just added checking for those types of links to the parser and am ignoring them for now. This is a no brainer for the missing and anchor links (why would you want to track those?) but I can see a reasonable argument for wanting to track mailto links.  I’ll be working on getting that working with the next release.

The comment parsing bug was easily fixed by changing the parse priority in the WordPress hook to a higher level than 1. This was an issue because WordPress uses it’s own plugin system to create links placed into comments to href links automagically so when I was parsing the href wasn’t created yet. I went with a priority of 9 and it worked great.

Now, The Features

New Gold Dream... (by law_keven)

So, while I was trying to figure out the Joerg bug I got a little motivated and started work on a couple new features no really asked for but I felt the system could use. The biggest is a mod to add links for external distribution. If I’m going to be honest, this really started as a bit of  a copout on my part. I originally wanted the entire experience to be seemless to the user so they wouldn’t have to go through the pain of manually adding links to the system but with all the problems people were having with the parser and me not being able to figure it as fast as I wanted I thought having a fallback mechanism to add links made sense. It should be noted that this decision was made at 3AM after a night on the town.

Like most coding decisions made while, ahem…”comfortable“, this mod required more work than I initially thought. I don’t want to bore with the technical crap but suffice it to say  a HUGE refactoring was required so I wouldn’t gouge my eyes out while working on the plugin. Technical debt, you’ve got to pay the price at some point…

I also built in a little configuration panel so blog owners can explicitly state what portions of the code they want to parse and which they don’t. This was, again, meant as a stop gap for the Joerg bug with the idea being that blog owners could lower the parsing coverage if performance issues came up.

And that’s 0.4. I hope this one goes better than 0.3 but I have no idea at this point. I did include the activation monitor so I can keep an eye on how many times it’s activated and/or deactivated so I’ll find out in about a week if people are actually using the plugin.

In the meantime, if anyone experiences any bugs or issues, no matter how small, please don’t hesitate to post a comment on the official WP-Click-Track 0.4 page.

One of the more nagging issues I have with releasing my code into the wild is that once it’s gone I have no idea what it’s being used for. I do have some limited data from various sources; for example I can easily track downloads from my site and the various counters some of the 3rd party sites who distribute my code have but this is painful and, frankly, flawed.

Magnifying Glass

Since I personally reuse downloaded code multiple times, while only downloading it once, I’m assuming there are other people who work the same way. This makes download tracking inaccurate for usage stats. Even if that wasn’t true, I would have to visit a few sites to get a total count and I’m pretty lazy so that kind of sucks…

Taking a cue from desktop programs I’m going to start keeping track of who’s using my shit.

Starting with the next release of Wp-Click-Tracker I’m going to be including a little script that let’s me know when a WordPress plugin is activated and/or deactivated.

For privacy hounds let me say this: the only information I’m going to collect is the URL of the site as well as the name and time of the action. This information will be kept strictly confidential though I will be publishing statistics about my plugin usage in the future. I will never, ever, ever, EVER, collect more data than what’s mentioned above.

To that end, the script I’m going to use is a new WordPress plugin I wrote I’m calling Activation Counter. I’ll be releasing it in the coming weeks for other WordPress plugin developers but I’ve been using it since the 0.3 release of wp-click-tracker and, so far anyway, it’s worked perfectly. Every time someone activates or deactivates wp-click-tracker a little “ping” is sent to my server which records the data.

Right off the bat, it’s allowing me to collect information about how many people are trying wp-click-tracker, obviously, but also how long someone keeps the plugin active. Because of that I’ve become aware of quite a few issues with the existing code that I’ll be fixing in the next release (0.4). I never would have been able to know about the issues if I hadn’t started monitoring the plugin.

I’ve been working on a new code base for StreetWise these past couple months, getting ready for the upcoming busy season, and as this new code needs to support membership of 1,000,000+ people decided I wanted, no, needed, to rework my pagination algorithm.

The Problem

Previously, I’ve gone for a basic approach of just listing out the links sequentially with a Next and Prev state. For larger page sets, I would just remove the numeric links. For example they would look like the below:

1 2 3 4 5 etc…

This becomes problematic at higher numbers though; just imagine the page would look if there were 1,000 pages instead of five. Here’s an example of just 100:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

So it can be a bit of a problem.

The Replacement

A lot of websites are using a better pagination style that’s sometimes called WordPress pagination. This pagination style presents only a few links at a time while still allowing more freedom than the basic Next Prev style. Here’s an example:

« Previous 1 2 3 … 10 11 12 13 14 … 118 119 120 Next »

The Solution

Being it’s called WordPress pagination, and that I’m a lazy php programmer, I just stole the code from WordPress and translated it into something a little more portable.

function paginate_links($format = '?page=%#%', $total = '1', $current = '0', $base = '%_%', $show_all = FALSE, $prev_next = TRUE, $prev_text = '« Previous', $next_text = 'Next »',$end_size = '3',$mid_size = '2',$type = 'plain') {
 
	$total = (int) $total;
	if($total < 2) {
		return;
	}
	$current  = (int) $current;
	$end_size = 0  < (int) $end_size ? (int) $end_size : 1; // Out of bounds?  Make it the default.
	$mid_size = 0 <= (int) $mid_size ? (int) $mid_size : 2;
	$r = '';
	$page_links = array();
	$n = 0;
	$dots = false;
 
	if ( $prev_next && $current && 1 < $current ){
		$link = str_replace('%_%', 2 == $current ? '' : $format, $base);
		$link = str_replace('%#%', $current - 1, $link);
		$link .= $add_fragment;
		$page_links[] = "<a class='prev page-numbers' href='" . clean_url($link) . "'>$prev_text</a>";
	}
 
	for ( $n = 1; $n <= $total; $n++ ) {
		$n_display = number_format($n);
		if ( $n == $current ) {
			$page_links[] = "<span class='page-numbers current'>$n_display</span>";
			$dots = true;
		} else {
			if ( $show_all || ( $n <= $end_size || ( $current && $n >= $current - $mid_size && $n <= $current + $mid_size ) || $n > $total - $end_size ) ) {
				$link = str_replace('%_%', 1 == $n ? $format : $format, $base);
				$link = str_replace('%#%', $n, $link);
				$link .= $add_fragment;
				$page_links[] = "<a class='page-numbers' href='" . clean_url($link) . "'>$n_display</a>";
				$dots = true;
			} elseif ( $dots && !$show_all ) {
				$page_links[] = "<span class='page-numbers dots'>...</span>";
				$dots = false;
			}
		}
	}
 
	if ( $prev_next && $current && ( $current < $total || -1 == $total ) ) {
		$link = str_replace('%_%', $format, $base);
		$link = str_replace('%#%', $current + 1, $link);
		$link .= $add_fragment;
		$page_links[] = "<a class='next page-numbers' href='" . clean_url($link) . "'>$next_text</a>";
	}
 
	switch ( $type ) {
		case 'array' :
			return $page_links;
			break;
		case 'list' :
			$r .= "<ul class='page-numbers'>\n\t<li>";
			$r .= join("</li>\n\t<li>", $page_links);
			$r .= "</li>\n</ul>\n";
			break;
		default :
			$r = join("\n", $page_links);
			break;
	}
 
	return $r;
}

As you can see, especially if you’re already familiar with WordPress, all I’ve done is take out the WordPress specific functions and set the $args to function parameters. Pure theft

There are a bunch of paramaters and options you can configure the links by but only the first 3 are really needed to make the function work.

Here’s an example:

$numofpages = 20;
$page = '4'
$query_string = 'foo='.$foo.'&bar='.$bar;
echo paginate_links('?'.$query_string.'&page=%#%', $numofpages, $page);

I didn’t put too much work into this, which is kinda the point, but I hope it’s helpful!

I’ve updated the repository with the latest version tonight.

This release includes a couple bug fixes and a few improvements.

Improvements

• Added Indexes on tables for better performance
• Updated Install system
• Added pie charts for clicks by day and hour

Bug Fixes

• fixed hour click report formatting
• fixed blank link haunting
• fixed option saving

Screenshots

Clicks By Hour Pie Chart

Day Clicks Pie Chart

Version 0.2 of wp-click-tracker has been out for almost 4 days and it looks like I left a couple bugs in there. I’ve updated the repository with the latest version and you can also download it from the main page:

wp-click-track download

This release is just a bug fix for the below:

• Parsing of attributes with single quotes
• added parsing / tracking of Archive links
• added parsing / tracking of Bookmark links
• added parsing / tracking of Comment Author URL links
• fixed option updates

Well it’s taken a couple weeks but I’m finally done with 0.2. This release includes an extensive reporting area, the addition of tracking on links created in the Links area of the admin, incompatibility checking for Google Analytics plugin and an upgrade notification nag.

Top Referrers

Top Referrers

Individual Clicks

Individual Clicks

Clicks by Hour

Clicks By Hour

Clicks by Day

Clicks By Day

Upgrade Checking

Upgrade Click Tracker

Plugin Incompatibility Checking

Compatibility Warning