Made of Everything You're Not

I really wanted to move away from Bad Behavior; there’s only so much I’m interested in this topic. But the first two posts didn’t cover everything I wanted to talk about so I wasn’t left with that warm fuzzy feeling of completeness.  If, like me, you’re over the whole Bad Behavior series I’m really sorry.

More Bad Behavior; Again

This time I’m going to go over the last little tid-bits so you can protect your sites and make sure the “bad” people stay away. For the most part anyway.

White Listing

You can white list IP addresses and user-agents. IP addresses can be white listed using ranges (in the CIDR format) or single IP by editing the file ‘whitelist.inc.php’. Open it up and edit the  below:

14
15
16
17
18
19
20
21
22

// Includes four examples of whitelisting by IP address and netblock.
$bb2_whitelist_ip_ranges = array(
	"64.191.203.34",	// Digg whitelisted as of 2.0.12
	"208.67.217.130",	// Digg whitelisted as of 2.0.12
	"10.0.0.0/8",
	"172.16.0.0/12",
	"192.168.0.0/16",
//	"127.0.0.1",
);

User-agents can be white listed in the same file but require an exact match to what you add to the array.

37
38
39

$bb2_whitelist_user_agents = array(
//	"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) It's me, let me in",
);

It’s important to use white listing conservatively. Extremely so. You don’t want to use white listing unless you positively, absolutely, have no other option.

Black Listing

There are two different ways to use black listing in Bad Behavior; manually adding entries to your black lists and using the http:BL feature.

To use Bad Behavior’s http:BL features you must have an http:BL Access Key. It’s a fairly simple process that requires registration with Project Honeypot. They’ll give you a BL Access Key and you place it in the settings array of ‘bad-behavior-generic.php’.

47
48
49
50
51
52
53
54
55
56

$bb2_settings_defaults = array(
	'log_table' => 'bb_logs',
	'display_stats' => true,
	'strict' => true,
	'verbose' => true,
	'logging' => true,
	'httpbl_key' => 'PLACE_YOUR_KEY_HERE',
	'httpbl_threat' => '25',
	'httpbl_maxage' => '30',
);

Once that’s done your install of Bad Behavior will use your local black lists as well as the http:BL lists.

On the other hand, you may encounter some rare cases where your site is being spammed by a new agent. In this case you might want to manually add entries to your local black lists.

The black lists are placed within ‘blacklist.inc.php’. It only accepts user-agents, probably because IP address blocking is essentially useless. You’ll need to edit 3 different areas of the script:

The first is for strings that occur at the beginning of the user-agent.

7
8
9
10
11
12

$bb2_spambots_0 = array(
	"<sc",			// XSS exploit attempts
	"8484 Boston Project",	// video poker/porn spam
	"adwords",		// referrer spam
	"autoemailspider",	// spam harvester
	//etc...

The next is for strings that occur anywhere within the user-agent string.

57
58
59
60
61

$bb2_spambots = array(
	"\r",			// A really dumb bot
	"; Widows ",		// misc comment/email spam
	"a href=",		// referrer spam
	//etc...

And, best of all, there’s also a regular expression (regex) array for the really difficult user-agents.

87
88
89
90
91
92
93
94

// These are regular expression matches.
$bb2_spambots_regex = array(
	"/^[A-Z]{10}$/",	// misc email spam
	"/^Mozilla...[05]$/i",	// fake user agent/email spam
	"/[bcdfghjklmnpqrstvwxz ]{8,}/",
//		"/(;\){1,2}$/",		// misc spammers/harvesters
//		"/MSIE.*Windows XP/",	// misc comment spam
);

It’s also possible to add your own blacklists into your Bad Behavior install. This is pretty helpful if you have multiple installs and are sane enough to recognize the absurdity in maintaining multiple lists. Just build a blacklist service and add the info to ‘blackhole.inc.php’.

Yeah, just build a blacklist server. Ummm… it’s easy?

Customizing the Template

Every time a request gets blocked the system doesn’t really know, 100%, that the request is bad; it just can’t. So, instead of just dying, Bad Behavior displays a page with instructions on how to “unblock” yourself usually by just clicking on a link.

Unfortunately, the page looks like ass.

Bad Behavior Blocked Screenshot

The idea, I’m sure, is that the page should have as small a footprint on the server as possible. One of the selling points is to lower bandwidth by blocking spam requests. No images, CSS or pretty allowed at all.

The flip side of this argument is that legitimate users who get this page will have an experience that’s lacking in, ahem, quality.

You can change the look of the page by editing ‘banned.inc.php’. It should be pretty self explanatory once you open the file what needs to be done.

If you want, you can also change the response messages. Those are stored in ‘responses.inc.php’.

Well, that about does it; there’s more to Bad Behavior but this series pretty well covered all the good parts.

This will be my last post on Bad Behavior; I swear.

In the first post in this “series” (wtf? when did I start doing “series”? Oh, right…) I went over the basics of what Bad Behavior is and how to get it installed. Bad Behavior’s advanced setup required some investigation and forethought in order to work out it so it was best to break the post up; so, you know, here you go.

More Bad Behavior

By default, if all you do is follow the instructions laid out in the first post you’ll have a working setup. For some of the cooler logging functionality you’re going to have to edit the included file ‘bad-behavior-generic.php’.

I’ve been feeling kind of down on Bad Behavior about this process for a few days now. It seemed kind of lame that they came up with such a great idea, wrote a really cool script but then killed the implementation. After working with the script for a couple days it’s starting to make some sense but not really enough to convince me it shouldn’t have been done.

To help others who’ve had this delima, I’ve compiled a list of steps to get Bad Behavior logging up and running.

Install Bad Behavior

The first thing you’re going to want to do is create a database and add the connection code to your version of ‘bad-behavior-generic.php’.

Here’s the SQL:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

CREATE TABLE IF NOT EXISTS `bb_logs` (
  `id` int(11) NOT NULL auto_increment,
  `ip` text NOT NULL,
  `date` datetime NOT NULL default '0000-00-00 00:00:00',
  `request_method` text NOT NULL,
  `request_uri` text NOT NULL,
  `server_protocol` text NOT NULL,
  `http_headers` text NOT NULL,
  `user_agent` text NOT NULL,
  `request_entity` text NOT NULL,
  `key` text NOT NULL,
  PRIMARY KEY  (`id`),
  KEY `ip` (`ip`(15)),
  KEY `user_agent` (`user_agent`(10))
)

For this simple demo I’m going to use the native php functions but it’s more than likely you’ll have a database class. I put the below directly past the comments above any function declaration:

33
34
35
36
37
38
39
40
41
42

$link = mysql_connect('localhost', 'user_name', 'password');
if (!$link) {
   die('Not connected : ' . mysql_error());
}
 
// make foo the current db
$db_selected = mysql_select_db('bad_behavior', $link);
if (!$db_selected) {
   die ('Can\'t use foo : ' . mysql_error());
}

Next, you need to populate the skeletal functions with their appropriate code:

60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

// Return current time in the format preferred by your database.
function bb2_db_date() {
	return gmdate('Y-m-d H:i:s');	// Example is MySQL format
}
 
// Return affected rows from most recent query.
function bb2_db_affected_rows() {
	return mysql_affected_rows();
}
 
// Escape a string for database usage
function bb2_db_escape($string) {
	return mysql_real_escape_string($string);
}
 
// Return the number of rows in a particular query.
function bb2_db_num_rows($result) {
	if ($result !== FALSE)
		return mysql_num_rows($result);
	return 0;
}
 
// Run a query and return the results, if any.
// Should return FALSE if an error occurred.
// Bad Behavior will use the return value here in other callbacks.
function bb2_db_query($query) {
	return mysql_query($query);
}
 
// Return all rows in a particular query.
// Should contain an array of all rows generated by calling mysql_fetch_assoc()
// or equivalent and appending the result of each call to an array.
function bb2_db_rows($result) {
	return mysql_fetch_assoc();
}
 
// Return emergency contact email address.
function bb2_email() {
	// return "example@example.com";	// You need to change this.
	return "badbots@ioerror.us";	// You need to change this.
}

Then you’ll also need to place a call to close the database connection at the bottom of the script. Place the below on the very last line:

160

mysql_close();

Doing that will make sure you don’t have any rogue connections eating up your queue.

Once the above is complete you should have a fully setup and working install of Bad Behavior.

In case anyone has any issues with the above I’ve prepared a stand alone version of the script anyone can download bad-behavior-generic.

It should also be noted that incorporating a database into your Bad Bahavior installation ups the load on every request (which may be why it’s not in there by default). You just have to choose whether the need for logging out weighs the increased load. For me, it did.

Anyone with a blog has seen comment spam. This is the stuff that shows up talking about Viagra, written in Russia and are usually stuffed with links.

There are a couple tactics for combating this sort of thing; some sites require registration to comment, some people manually delete the stuff and some sites use technology to help.

Bad Behavior

What to do? What to do…?

Well, I don’t think it’s a good idea to add barriers in front of users participating in a discussion, so registration is out. I’m pretty lazy and don’t want to manually delete comment spam so moderation isn’t going to work. I am a programmer though so I have an innate confidence in technology to deal with this (mostly anyway). To that end I like to use 2 different services to deal with comment spam; Akismet, which I’m not going to talk about now, and Bad Behavior.

According to the official site:

Bad Behavior complements other link spam solutions by acting as a gatekeeper, preventing spammers from ever delivering their junk, and in many cases, from ever reading your site in the first place. This keeps your site’s load down, makes your site logs cleaner, and can help prevent denial of service conditions caused by spammers.

Thankfully, there are already Wordpress plugins for both Akismet and Bad Behavior, so my blog is pretty well protected, but I also work on custom programs and need to protect them too. This got me thinking about how to to get Bad Behavior up and running on your systems; which is why you’re here I’m sure.

Like most things php, installing Bad Behavior is pretty easy. To install just download the files, unzip and place the files somewhere in your applications include path. Then just include the below preferably in a file included in all your pages after you upload the files. Using the below will only protect your site

1
2
3
4

<?php
$path_to_bb = '/path/to/';
require_once("$path_to_bb/bad-behavior-generic.php");
?>

The above is nice and all; your site’s pretty well protected from there but it would be nice to know what was happening behind the scenes. Just how many spam attempts are being blocked?

Bad Behavior does include a logging system but, oddly, at the time of this writing using 2.0.26, they don’t include any sort of install script. The instructions state:

If you just can’t live without logging, you will need to provide a database connection. Bad Behavior uses callbacks whenever it needs to run a database query; in order to provide this functionality, you will need to provide the appropriate hooks into your PHP-based software’s database and add them into the bad-behavior-generic.php file. The code has stub functions which show what is needed, and you can use the bad-behavior-wordpress.php file as an example to work from, though your implementation will necessarily be different.

I’ll go into detail in the next post; I’m still sick so I can’t write anymore.

In How to Not Suck at Email Campaigns I completely left out a HUGE part of not sucking; scrubbing your email list.

Sanitize email list

Scrubbing an email list before sending it out is something that rarely gets done but is actually pretty crucial.

See, email providers, the organizations that provide email like yahoo, gmail and hotmail, and some spam filters, actually look at the email addresses you send to and flag those they deem to be either role accounts or spam traps and they use that to determine part of your spam score. If your spam score is too high; you’re a spammer.

This assumes you have an honest list to begin with. If you’re buying lists of email addresses from people and sending emails to them you’re a spammer and nothing here is going to help you. Oh yeah, everyone else knows you’re a spammer too.

Fuck you.

Anyway, I first learned of the importance of scrubbing a list while preparing one of my clients email databases for a blast (they hired StreetWise to send email on their behalf). During the import process into the mailing solution we were using our account was put in lock down which essentially stopped the entire project dead in it’s tracks. After contacting the mailing solution I got the below response:

I do want to be clear that the list both contains spamtraps, role accounts (such as admin@) and other addresses with aspects that indicate that some part of the list may not be completely opt-in. Spamtrap addresses are those addresses that have generally been retired or are known by the domains postmaster to be inactive and as a result unlikely to be signing up for new mailing lists. These addresses may also be ones planted on a website specifically for email harvesting purposes. Sending to them indicates a lack of an opt-in process, the ideal option being a double opt-in confirmation method where members must both submit their email address and respond to a follow-up email before receiving regular mailings from the list.

Mail sent to spam traps, as with non opt-in mail in general, may result in a loss of deliverability for a sender. This may include domain blocks (refusal of the senders mail by a domain) or blacklisting by spam prevention agenies and further hurts the reputation of the sender.

Combined with negative feedback from the recipients (sent either to the sender, its ESP, or the users postmaster), not honoring opt-out requests, sending to a large amount of non-existent members, spamtrap addresses call into question the entire lists credibility.

So just what do you need to scrub for? Well, duplicates, MX records for the domain and syntax issues obviously but, more importantly, specific keywords in emails.

To start you want to get rid of any email address containing the keywords below (* is wildcard):

junk*
admin@*
root@*
postmaster@*
blackhole*
confirm@*
fuck*
donotreply*
help@*
nobody@*
*@poop.com
support@*
sysadmin@*
*spam*
dev@*
devnull@*

That’s not a complete list of course; just the ones I’ve personally discovered to date. I plan on compiling a database of them once I get more and so should anyone who’s serious about not sucking at bulk email blasts.

There’s also spam trap email addresses. These are email addresses people sign up for that allow them to sign up for accounts and other services without actually giving their real email address.

They are insanely popular (and pretty annoying too).

So far I’ve only had to deal with a few though. You want to remove any email address from the below domain:

*@spamgourmet.com
*@sneakemail.com
*@mailinator.com
*@trashymail.com
*@mailexpire.com
*@temporaryinbox.com
*@spambox.us
*@spamhole.com
*@pookmail.com
*@spamfree24.*
*@kasmail.com

This is especially important on lists you’re sending on behalf of your clients. I have yet to have a client freely admit to having email addresses that haven’t been opted into properly; they always swear that their list is clean. I’ve seen HUGE corporations, we’re talking hundreds of millions of dollar companies, hand over lists that decreased over 20% once it’s scrubbed.

Cool thing is, though, that the client usually has no idea. Usually, someone told them they have a list of email addresses somewhere and, odd as this may sound, they’re actually thrilled to be told their list isn’t “good”. Makes you look like you know your shit

One of the most frequent IT related tasks I have to deal with are email issues. Spam is a big part of my life, on both sides of the fence. I have to deal with legitimate email from clients getting flagged as spam on my network and I also have to make sure email StreetWise sends out on behalf of our clients doesn’t get flagged as spam.

Spam

Over the years I’ve put together a list of requirements for making sure email campaigns go as smoothly as possible. Since no one can seem to get email campaigns right I figured I write it all down and post it and hope someone gets something out of it. Enjoy!

Check Spam Score

A Spam Score is a number assigned by the spam-scanning rules. The higher the score, the more “spamlike” the message appears. It’s very important to how your email ranks so you can fix the before you send out the blast.

There’s a great tool called Mail-Check that will evaluate an email and report on it’s supposed spam score. It has an integrated SpamAssassin engine that performs a series of tests against your email to determine how much like spam it looks like.

The tool is in no way fool proof though. Be sure to use it as just a single source of information and don’t make too many decisions based on the results.

Send Email Slowly

The basic idea behind this idea is that an email provider looks at how many emails is coming in from a particular domain and, if there’s a lot of email coming in at , will assume it’s either a spammer or a virus. Not a good label to have on your domain.

How slowly should email be sent? Unfortunately, that’s a difficult question and the amount of email you need to send greatly affects the answer. For example, if you have 1,000 emails to send you probably don’t need to worry about sending too fast. On the other hand if you have 250,000 emails to send slowing down the send process is HUGELY beneficial.

I’ve heard that the best approach is to throttle your sending by recipient domain. Now, I’ve never done this myself, but throttling by domain is supposed to be helpful because you can keep a consistent flow of email coming from the mail server but since there isn’t a constant stream of email to one provider they don’t know a bulk mailing is being sent to their network.

Get Whitelisted

This ones a bit of a no brainer but you NEED to get your email domain whitelisted by the email service providers you’re sending to. This should include, at the very least, Yahoo!, gMail, MSN (hotmail) and AOL as well as the email recipients. This can be pretty tricky to do because the methods for getting whitelisted by the companies above tend to change fairly frequently. A good method to help would be to send individual requests to “postmaster@DOMAIN.tld”, where DOMAIN.tld is the email providor you want to get whitelisted from, and explain your opt-in process and ask to be added to their whitelist.

It’s also a good idea to ask your email recipients to add your email address to their personal whitelists. This helps because most email providers allow email through that is on their blacklist but on a users personal whitelist.

Setup and Maintain Your Email Server Properly

A lot of newbies make the mistake of thinking that if they can send email they should send email. This is pure folly. There are a few things you should do on the server level to ensure you’re not going to look like a spammer. Off the top of my head here are a couple requirements that have to be implemented:

All e-mail servers must have valid reverse DNS records. Doing this allows email providers the ability to look you up and make sure you’re not sending from a blackhole. A lot of spammers won’t go through the trouble of setting up a reverse DNS record because they have to be ready to move at a moments notice. Setting one up shows you’re serious.

All e-mail servers must be secured to prevent unauthorized or  anonymous use. Preventing Joe Spammer from sending email from your domain shows the email services that the email really came from you. If, for example, they see that anyone can send email from your sever the email services has no reason to trust it actually came from you.

Setup a Sender Policy Framework (SPF) DNS record for your domain. The SPF is an open standard specifying a technical method to prevent sender address forgery. This basically means that setting up an SPF record in your DNS shows the email provider that email coming from a server is being sent on your behalf.

Avoid Evil

This is a topic that’s pretty absurd it even has to be mentioned but, sigh…, it has to be mentioned. (I guess I’m a bit of an idealist, but it seems to me that this stuff is just kind of  evil.)

Anyway, right off, you’ve got to be compliant with the federal Can Spam Act of 2003. If you’re going to be sending bulk email it’s important to understand the rules, especially when sending on behalf of your business and clients.

For the love God, don’t try to hide, forge or misrepresent the sender of the e-mail and sending site of the e-mail in any way. I’ve had this asked of me more times than I can count and it always ends badly; it’s an easy way to get yourself on a blacklist.

The last evil thing might not be too evil but I hate when I get an email and it doesn’t say where they you got the my email address and why they’re sending them an email. It’s an easy thing to do, especially since the act of having that information requires honesty about the list, plus it makes your recipients feel secure.

Write Good Email Content

Much like in Search Engine Optimization a lot of what makes a good email is good content (by content I’m talking about the entire email here; not just the copy). Here’s just a couple must dos:

1. Use alt-tags on ALL images.
2. Use short, descriptive, subject lines. Avoid special characters like the plauge.
3. Avoid using a lot of spammy key words and characters. For example; free, $, Prize, etc.
4. Keep the top header under 100 pixels tall and place a “View Email Online” link first and foremost.
5. Avoid center aligned content; it’s best to keep the email body left aligned on a white background. Spam filters tend to flag colored backgrounds higher than white.
6. Include a plain text version of your email with every blast.

Unsubscribing Should be EASY

This is probably the most important piece in my opinion. It has to be easy, like ridiculously easy, for a user to remove themselves from your list. Bulk mailings should contain simple and obvious unsubscribe mechanisms. AOL recommend that this be in the form of a working link to a one-click unsubscribe system; however, a valid “reply to:” address may be used instead.

There are all sorts of methods for the unsubscribe flow; just pick one that requires the least investment for the user. Avoid the impulse to create a survey asking why a user doesn’t want your email anymore. The fact they don’t want it anymore should be enough. Anything more and you run the risk of pissing people off.

Summary

Looking over the above I’m sure I forgot some things. Still, it should be a good starting point.

It’s important to remember that there really is no silver bullet when it comes to email marketing. Sadly, the reality is that even if all of the above is done there’s still a chance your email will be flagged as spam. Most email providers take their users interests very seriously, while taking the stance that everyone else is evil until proven not evil. Even if they don’t consider you evil they’ll revert a good mailing to evil status after only a handful of complaints.

Additional Reading

AOL Whitelist Information
Email Secrets of a Top Converting Website
Create Better Performing Assets