Made of Everything You're Not

It all started with a simple email from a client with the subject line "Broken Link"; looked like one of my client's sites was down. Since this was the one that used that lovely of loveliest of programs, Dolphin CMS, I didn't really thing much of it. Remember, Dolphin CMS sucks and, yes, it's been known to just crumble on occasion (fucking Boonex...). So, yeah, initially, I didn't think much of this. But then again, I have been pretty happy lately and the universe really does hate us all. I should have known this was gonna be bad...

Looking into the problem it was immediately obvious what was going on; my server was gone. I couldn't access any site, much less my client's site who alerted me to the issue, using any protocol or tool (ssh, ftp, cpanel or whm at least). It was like it didn't exist...

Immediately, I contacted the server provider, HostGator, where I had a dedicated server requisitioned a few years ago. Thinking this was at worst a network issue I contact HostGator and submitted a support ticket; that's the only thing to do since I couldn't personally deal with the issue. That's the trade off for leasing a dedicated server; on the other hand HostGator will fix anything that goes wrong so it's worth it in my opinion.

HostGator, to their credit, got back to me within 20 minutes to inform me that the server account was closed. My first thought:

WHAT THE FUCK??!!??

Looking into just what the hell happened I found out that the client who was paying the bill (I had worked out a "deal" for him where I built and maintained his sites and he would pay the hosting bill) just decided that he was done with the whole Internet thing. Seriously. Apparently, he decided to just stop paying the bill a couple months ago forcing HostGator to cancel the account because he, in his words, "Hadn't made shit from this crap".

Anyway, this all highlighted a failure point in my backup strategy. Yes, I had a  backup strategy and I even had backups locally. No, my problem was that I didn't have access to any backup newer than 2 weeks old. Here's how my backup strategy worked:

1. Daily backups stored to a NAS on the rack.
2. Weekly backups FTPed to another HostGator account
3. Bi-weekly backups were being downloaded to my local network (home) every 2 weeks.

See the problem point? Yeah, keeping too much of an interval on the network instead of within reach and not having access to my recent backups. That. Was. My. Bad.

Two weeks was just too much time to go without a physical backup. It's an infinity on the Internet; too much stuff can happen in that time and ironically my client with the nasty, nasty, Dolphin CMS is proof of that. Unfortunately, two weeks ago her site had no traffic and no real use to anyone. Come two weeks later and she had gotten the membership up to a couple hundred users who were actually using the site. That's the part that really sucked; I had failed my client.

This whole ordeal prompted a number of changes to my overall routine. Here's the new setup.

1. For starters I'm no longer leasing a server; instead I purchased my first server and have it hosted at a data center where I have personal 24/7 access to it and the rack it's on.
2. Daily backups are being stored to a backup drive in my server and being mirrored to an external NAS on the rack.
3. Weekly, I will go to the data center and swap out the backup drive on the server with another keeping the drive at my house.
4. Bi-weekly backups will be downloaded to my laptop and kept close at hand.

Overkill? Probably, but considering that the majority of this is automated I don't see too much of an issue with it. In fact, the only thing I have to actually do is go to the data center once a week and switch out the drive so it's really not too bad.

Hopefully, this'll work out better than the last strategy.

One thing almost all computer users have in common, regardless of vocation, is that we use the computer to achieve some goal. The actual goal doesn't matter so much as the fact that we're using the computer to do something that, otherwise, we wouldn't be able to do. To do so though we use various tools that are, usually, purpose built for the task.

Let's be honest; without the tools we would be useless.

Personally I love my tools; specifically, I find developer tools to be some of the most interesting and fun toys available. I don't want this to turn into a fanboy post but, in the interest of honesty, it just might. You have been warned...

What's in my Web Developer Toolbox?

My toolbox is full of programs that are purpose built to help every step along the way for building Internet applications. Over the years, like pretty much all developers I'd guess,  I've come to rely on the below tools to ease the pain of development as much as possible. I totally vouch for these tools.

Version Control

This one's crucial. If I had to rank these (and I really don't plan to) version control would be at the top of the list. There's a whole slew of options available but, for me, version control starts and ends with Subversion.

Yes, there are all sorts of hype surrounding GIT and Mercurial but, because right now, I work alone my needs are way too simple for anything like distributed version control. Nope; just give me Tortoise and an SVN URL, with credentials, and I'm a happy camper.

Local Development Web Server

Once upon a time the thought of using a local development web server was heresy to my style and philosophy. Now that I've been using one for the last year I have to admit I was dead wrong. Dumb even.

Previously, I would always use an external Linux server for all my development. The idea was that since the finished site would be hosted on a Linux server it was important to develop the site in the same environment. There are 2 big problems with this approach though; one is that the project is more likely to be dependent on the environment which can make relocation a problem, and two, is that continued progress on the project requires a connection to the Internet.

On the other hand, developing your projects on a local machine requires finesse and forethought to ensure porting the site from one environment to another doesn't lead to anarchy. There's also the knowledge and insight gained from setting up an environment by hand; there's so much to gain from doing this it's just silly not to without some edge condition.

Text Editor

Only masochists use Notepad for text editing. In today's world of fast CPUs and large amounts of RAM it's really hard to believe anyone would use Notepad for anything other than the most basic of basic editing tasks. If you plan on having a file open, for editing, for any extended period of time it's just stupid (yes; STUPID) to use it.

Instead, I prefer EditPlus for all my text editing needs. Why? For one killer feature; a right click context menu item. Right click over any file and choose EditPlus to open the file for editing; it makes working very fluid and continuous.  It even handles files in the hundreds of MBs with ease.

There are other options for a text editor (Notepad2 comes to mind) but EditPlus is tough to beat.

Database Tools

Sure, a command line tool is perfectly adequate for administering a database server. The problem though is that I develop on a Windows machine and not using a GUI tool for database administration is kind of silly. It's like the people who only use VIM for text editing; it's like trying to prove a point against all logic.

Using MySQL, the easiest, and most familiar tool, is phpMyAdmin; but for remote administration phpMyAdmin starts to break down. Instead, you can't beat the MySQL Administrator. It offers all the functionality as phpMyAdmin as well as a slew of advanced functionality like the ability to create stored procedures (which you should never, ever, do) and functions.

Remote Connectivity

Since pretty much everything I work on has to go somewhere and I usually need to connect directly to a server for administration I need tools that'll allow me access. These tools really come down to 3; FTP, SSH and RDP.

For simply moving files between servers FTP or SFTP is obviously the choice. There's bunches and bunches of options when it comes to FTP clients but I've been using CuteFTP for years and, pretty much, I swear by it. Yes, it's a paid product but CuteFTP is also low impact, easy to use and, more important, doesn't get in the way of my productivity.

And then there's system administration which requires full control and access to a server or computer.

Linux has SSH which requires a small client utility. There's a shitload of options available here but they're all pretty similar so there's not much difference between using, say, PuTTy or SSH Secure Shell Client (or any of the myriad other SSH clients out there). You just need to have one.

For Windows as far as I'm concerned there's really only 2 options; Remote Desktop Connection (RDP) and VNC. For ease of use and quality of experience RDP is the way to go. HINT: There's a setting to allow mapping of HDDs on the client machine to the server for easier file transfers.

Virtual Machines

In my opinion there's been no bigger advance in quality assurance (QA) then the advent of the virtual machine (VM). QA probably wasn't even a goal when VMs were first conceived but, boy, have they filled the gap well.

I've already gone into detail about my choice of VM tool as being Virtual Box:

VirtualBox handles the resource detail pretty elegantly; in that it doesn’t use the resource until it needs it. This means that instead of instantly having 10GB of your hard drive used up VirtualBox will only use the amount already taken. You can tweak the settings for a VM whenever you want so you can get just the right mix.

Web Browsers

You need pretty much every modern browser under the sun here. Um... duh?

So, there you go; those are the primary tools I use for web development. There are definitely some other tools I use that didn't make the list (RegexBuddy, Photoshop and diff tools come to mind) but I didn't feel they deserved mention because of how rare they're used.

Did I miss anything else?

Since I had to setup a whole new computer I decided to move away from the IIS experiment I've been working on for the last year and try something a little different. I'd heard about Zend Server CE before but after a failed attempt to get it working a few months ago, because of IIS ironically enough, I hadn't really given it the attention I thought it deserved. After having played with it for about a week I have to say I'm completely... underwhelmed.

Zend Server is supposed to be a complete Web Application Server that is purpose built for php development. It includes application monitoring, problem root cause analysis and and extended caching capabilities. Pretty enticing really.

Unfortunately, Zend Server CE doesn't include any of the above bells and whistles. Instead, it's a stripped down version that appears to just match the features and functionality of XAMP or WAMP (Apache, php and mysql wrapped in a nice little installer for Windows).

I've used both used both XAMP and WAMP and, with little exception, I've always wished I'd gone with a manual installation instead. It's not that they're bad programs, it's nice that they're available for newbies, but my needs aren't easy to package up in a "one size fits all" package. I like to try new things and experiment and sometimes what I want to do isn't easy without breaking something. Admittedly, I haven't tried to use any one size package for a few years so this may not be the case anymore.

Either way though, I know I have a bias; I might even be a bit of a snob about the issue. Totally possible.

That being said, after installing Zend Server CE, which went very smoothly actually, I was confronted with what appeared to be an incomplete installation of php; php-win just didn't work. It did nothing in fact; I couldn't get it to do a damn thing. Since I do a little maintenance scripting with php-cli (and php-win.exe is essential on Windows) this was a pretty big issue.

On top of that, I just couldn't figure out how to modify the --configure options so changing the setup was obviously going to be an issue. I don't know if I'm an idiot but I just couldn't figure it out.

Then the let down happened; I was under the impression that there were going to be some cool profiling toys to play with. Instead, there's a web GUI for configuring PHP, which is pretty nice I guess, but for me, it's just easier to edit php.ini directly than navigate through a web interface. Kind of useless. What with the integration with Zend Debugger I was really expecting more.

Ultimately, it seems that if you're a complete newbie to php Zend Server CE is a worthwhile fit but if you actually know what you're doing you're still better off setting up a development environment manually.

This is pretty disappointing. A product from Zend, that's supposed to ease the pain of php development being released to the community, offering nothing more than you could already get from a dozen other programs kind of seems like posturing. I understand the desire to have a demo of a paid product but it should, you know, be different.

A good change I'd like to see would be to include some of the more advanced features like the Application problem diagnostics and the Application monitoring (alerting) functionality in the CE version. It would benefit the community far better than the current version.

The Linux command "top" is one of the most used and powerful jewels in a developers pocket.

In most Unix-like operating systems, the top command is a system monitor tool which produces a frequently-updated list of processes. By default, the processes are ordered by percentage of CPU usage, with only the "top" CPU consumers shown. The top command shows how much processing power and memory are being used, as well as other information about the running processes. Some versions of top allow extensive customization of the display, such as choice of columns or sorting method.

The top command is useful for system administrators, as it shows which users and processes are consuming the most system resources at any given time.

The great thing about "top" also highlights one of it's weaknesses; it's focused on CPU, memory (RAM) and time. Top is wonderful if you want to know how much performance your program is using but if you want to know how much the individual components are using you're out of luck.

Enter mtop.

mtop (MySQL top) monitors a MySQL server showing the queries which are taking the most amount of time to complete. Features include 'zooming' in on a process to show the complete query, 'explaining' the query optimizer information for a query and 'killing' queries. In addition, server performance statistics, configuration information, and tuning tips are provide

mtop is a pretty useful program; it really helps in finding out the trouble spots in queries. There one obstacle to consider before diving into though; mtop is written in PERL so there are a couple module dependancies (Curses, DBI, DBD::mysql, Getopt::Long and Net::Domain

Still, I didn't run into any issues installing the program and, so far anyway, mtop is a nice addition to my tool box.

I've been setting up my own servers for years; it's something I'm pretty passionate about as a programmer. I've learned soooo much about programming from seeing how the computer operates; setting up a server is HUGELY enlightening in that respect.

On the other hand though, I have a lot of things to do with my day and setting up a server, something that's pretty rote, doesn't require my direct attention. Just someone who can really follow instructions.

With that in mind, I started to compile a list so I could delegate this activity to my team. It's pretty useful from a historical stand point too so in case anyone else is interested in just how to setup a linux web server; here you go.

I've broken the setup into multiple sections, each outlining a specific type of setup, so it should be easier to digest. I'm just over the whole "series" thing, so instead of breaking this post into several smaller posts I opted for one REALLY long post. Plus, there's the whole disconnect thing with a series and setting a Linux box up shouldn't be done piecemeal.

It should be noted that the below is just a vanilla setup; it should be considered the bare minimum that needs to be done.

Basics

These are just a couple things that have to be done at no particular time. Ideally, the Firewall (below) would be configured first and then everything else would follow, but the rest could be done in any order. To make life easier it would be a good idea to take care of the basic stuff ASAP.

Setup locate

There's a sweet little command that is extremely helpful in locating files on the server; locate.

From the manual page:

locate reads one or more databases prepared by updatedb(8) and writes
file names matching at least one of the PATTERNs to standard output,
one per line.

It's way faster and efficient than 'find' and it's just what you need to find all sorts of things quickly. A basic use works like so:

locate php.ini

Which outputs:

/scripts/php.ini
/usr/lib/php.ini
/usr/local/lib/php.ini

To use it though, the first thing you need to do is make a call to 'updatedb' which will create the 'locate' database. The first time it's ran it'll take around 5 minutes to complete so you want to get this done ASAP so it doesn't break up your flow when working on something else.

updatedb

Setup Host Name

This part will register the server with the rest of the Internet. It's a different way on quite a few of the Linux servers I've worked with so I won't go into detail on how to do configure the server; I'm just going to go over the steps. You can always ask your host to set it up if needed.

Probably the most important part is choosing the name. There sees to be a couple different camps on this subject. Some like naming it as the role (so if it's a web server it'd be web1 or web5 (whatever) or if it's a db it'd be db1 or db5. Other's feel this is a security risk because it broadcasts what the server does.

Still, others, choose names that are just arbitrary; wizardtower, methlab, etc...

Personally, I find the convenience of recognition (knowing web1 is a web server and db3 is a database server) far outweighs any security concerns so I go this route.

Whatever you choose make sure to register the name with your registrar as an A DNS entry. Once that's done then you can update the server with the new hostname.

Email Forwarding

For every user on the system that receives email, like root for example, you'll want to create a .forward file in that users home directory. This good so email can be sent to you personally outside of logging into the server.

echo you@domain.com > /root/.forward

Security

Just to reiterate the above, this is by no means a complete list of everything that can be done to secure a server. Consider this a list of the minimum that needs to be done; nothing more. You should still research secure based off of your particular situation.

Seriously, your needs may require additional levels of security. You've been warned.

Firewall

Most modern installations of Linux come with a firewall installed called iptables which is really reliable and stable. I use iptables in conjunction with either ConfigServer's csf/lfd or, usually if I'm not on a cPanel or Webmin server, apf.

Personally, I prefer csf/lfd for managing my firewall. Not only does it take care of iptable configuration but it also:

• Sends an email on ssh login or su usage
• Blocks connections with excessive connections
• Login failure notifications for a lot of common services (cpanel, ftp, ssh, etc)
• Port scan tracking and blocking
• Temporary and permanent IP blocking
• System Integrity checking and alerts
• Suspicious process alerts
• Suspicious file alerts

apf requires adding a few other programs on to attain the same amount of coverage; I prefer simple.

To install csf/lfd ConfigServer made the process extremely easy; they even put together step by step instructions that have yet to fail.

Setup Users

Add non privileged user and add to wheel group. This is important because we're going to seriously limit access to the shell. I like to have just one user who can ssh into a server but who can't do anything but use 'su' to up their privileges. No access to anything but 'su', not even 'wget'.

Mount /tmp securely

It's important to mount your tmp directory securely so nothing contained inside can be executed. It really helps when your site allows file uploads or if the server has been exploited (the tmp directory is a prime target).

You want to mount the partition as noexec,nosuid.

Install RKHunter

RKHunter is a tool that scans your system looking for any rootkits. It's a good tool but it does report some false positives; nothing too annoying but it does happen.

It's pretty simple to install and I've yet to see it fail on any flavor of Linux I've used (and I've used a bunch).

wget http://superb-west.dl.sourceforge.net/sourceforge/rkhunter/rkhunter-1.3.4.tar.gz
tar -zxvf rkhunter-1.3.4.tar.gz
cd rkhunter-1.3.4
./installer.sh --layout /usr/local --install

Once RKHunter is installed you'll want to set a daily cron job so your system is checked regularly. To do that just create a shell script and place it in /etc/cron.daily/ as outlined in this tutorial for installing RKHunter

#!/bin/bash
(rkhunter -c --cronjob 2>&1 | mail -s "Daily Rkhunter Scan Report" root)

SSH

SSH is incredibly vulnerable for no bigger reason than visibility; it's the de facto entrance point for most linux servers. I like to do a couple things to secure my SSH installation.

To make any of the below changes you'll need to edit your sshd_config file. On most systems it's going to be in:

/etc/ssh/sshd_config

First, I always disable root login. Most brute force (BF) attacks on your server will be for the user 'root' so simply disabling this allows most BF attacks will be futile for the attacker. If you've added a new user to the system, and that user is the only user who can ssh in, you're in a pretty good spot. The attacker has to know the username in order to even try passwords.

Second, I also change the port ssh listens on. The default, 22, is what most attackers will try for getting into ssh. Change that to something different and you've added another level of complexity onto the system.

It's important to let your host know of the change so they can access ssh when they need to. This shouldn't be a problem for most hosts but you may have a fight on your hands for some.

There are quite a few options you can use to configure ssh for; it's definitely recommended that you research as much about ssh as possible to configure it specifically for your needs.

Disable General Commands

This next one isn't exactly critical but I find it useful and it definately adds peace of mind so there's that.

I first heard about this from a forum for securing a cPanel server.

Many php exploit scritps use common *nix tools to download rootkits or backdoors. By simply chmod'ing the files so that no none-wheel or root user can use them we can eliminate many possible problems. The downside to doing this is that shell users will be inconvenienced by not being able to use the the commands below. Mod_security really removes the need to chmod this, but it is an added layer of protection.

#chmod 750 /usr/bin/rcp
#chmod 750 /usr/bin/wget
#chmod 750 /usr/bin/lynx
#chmod 750 /usr/bin/links
#chmod 750 /usr/bin/scp

As mentioned above, this is probably overkill but it does prevent anyone who does gain access from being able to do much of anything. If you really want to have that warm, fuzzy, feeling of safety you could also just chmod everything under /usr/bin like so

chmod 750 /usr/bin/*

That should really make you feel safe.

Disable Unneeded Services

Chances are that your server is going to be running quite a few services you're just not going to need. For example there's 'cups' the Linux print service. Is your webserver going to be connected to a printer? Probably not.

Leaving these enabled is bad because it's an avoidable entry point into your server by the "bad" people. From my experience I've learned to disable a bunch so I put together a little shell script to just handle it for me. Copy the below and put it into a file on your server called 'disable_services.sh' and chmod it to 0755

#!/bin/bash
service cups stop
chkconfig cups off
 
service xfs stop
chkconfig xfs off
 
service atd stop
chkconfig atd off
 
service nfslock stop
chkconfig nfslock off
 
service canna stop
chkconfig canna off
 
service FreeWnn stop
chkconfig FreeWnn off
 
service cups-config-daemon stop
chkconfig cups-config-daemon off
 
service iiim stop
chkconfig iiim off
 
service mDNSResponder stop
chkconfig mDNSResponder off
 
service nifd stop
chkconfig nifd off
 
service rpcidmapd stop
chkconfig rpcidmapd off
 
service bluetooth stop
chkconfig bluetooth off
 
service anacron stop
chkconfig anacron off
 
service gpm stop
chkconfig gpm off
 
service saslauthd stop
chkconfig saslauthd off
 
service avahi-daemon stop
chkconfig avahi-daemon off
 
service avahi-dnsconfd stop
chkconfig avahi-dnsconfd off
 
service hidd stop
chkconfig hidd off
 
service pcscd stop
chkconfig pcscd off
 
service sbadm stop
chkconfig sbadm off

HTTP Server

I've gotten a renewed appreciation for Apache lately so I'm not going to focus on one more than the other. With the exception of mod_security, everything below should be possible on pretty much all your popular webservers like Apache or Lighttpd.

If you're going to stay with the default web server that's installed on the server you should, at the very least, rebuild it to make sure you're using the most up to date version.

Once you're dealing with a new(ish) installation of a web server the next thing you need to do is create the default site. This is the site people will see when they put either the IP address or the hostname of the server into a browser. I always set it up to use a blank page instead of the standard or default page. This way it doesn't look janky when users stumble upon the server.

Next, you want to disable Indexes. This setting is useful to prevent people from hitting a directory and seeing all the contents. If a user does try to read a directory, "images" for example, they will get a 403 (Forbidden) page.

Another thing I like to do is change the cPanel and http server skeleton files to blank pages. This is nice so when another site is created the site gets setup with blank pages instead of the "advert" pages for the system.

ModSecurity is an open source intrusion detection and prevention engine for web applications. It operates embedded into the web server, acting as a powerful umbrella - shielding applications from attacks. It's really cool.

ModSecurity works with Apache but there's always people out there experimenting so, hopefully, other http servers should get coverage some day. If you're using Apache you should definately, 100%, no excuses not to, install ModSecurity

PHP

For all the jokes about PHP being a sub-par programming language, accoring to the TIOBE Programming Community Index for May 2009 it's the most popular web development language available. So suck it.

It is true that php's the only language with a configuration file. I admit; that's just fucked up man...

Improve PHP

PHP is an interpreted language so right away your scripts are going to have a performance penalty (compared to a compiled language like C# for example). To help alleviate this you should always, always, install some sort of OP code cache. My personal favorite is xCache but there's also eAccelerator to name just one.

Installing xCache is pretty straightforward and setting it up is just as easy. Once it's done you should notice a considerable improvement in performance.

You'll also need to upgrade PEAR to make sure you're using the latest versions of packages and such. It's pretty easy; from a command prompt:

pear upgrade pear

After that you'll want to make sure the below packages are installed and up to date. These are just what I personally use and what the majority of open source php projects I've seen use.

pear install HTML_QuickForm
pear install Table
pear install Cache
pear install Cache_Lite
pear install Mail
pear install Mail_Mime

Secure PHP

Pretty much all of the security stuff is done by configuring PHP in php.ini. If you don't know where it is you can either create a phpinfo() call and look for the path to php.ini or, better, just use the 'locate' command from the command line:

locate php.ini

Unless you're using an old version of php (and why the fuck would you?) it's going to come out of the box with the WORST setting php ever introduced; 'register_globals'. If you require this setting to be active, for new projects, you're an idiot. I do unfortunately know about legacy apps requiring register_globals to be on but you can just set it with 'ini_set' on a per project basis so turn it off FOR EVERYTHING NEW YOU DO.

You're going to want to disable quite a few functions too, if possible. There's definate use in a lot of the below functions and sometimes the project you're hosting is going to require some of them. For example, on php 5.2.9 popen is required for some PEAR packages (and itself I think). This should be done on a case by case basis. But if you don't need to keep these open DON'T.

Look for the string 'disable_functions' in your php.ini and add any of the below to that string.

disable_functions = show_source, system, shell_exec, passthru, exec, phpinfo, popen, proc_open, allow_url_fopen

Configure MySQL

One thing really; change the god damn root password!! Set it to something, anything is better than NOTHING.

I've had some people recommend disabling LOAD DATA LOCAL but while it sounds like a good idea it doesn't really gel with the way I work. I need that enabled to import databases on the server when the file is too big to upload into phpMyAdmin. I'm sure I could just enable it, do my thing, then disable it again but that sounds... troublesome.

Run Benchmarks

Running the benchmarks on the server is probably the most important part of setting up a server. It's important because the results of the benchmarks are going to tell you what you have to do next.

There are a few different tools for benchmarking a server but the most popular is ApacheBench. Accoring to Wikipedia:

ApacheBench is a command line computer program for measuring the performance of HTTP web servers, in particular the Apache HTTP Server. It was designed to give an idea of the performance that a given Apache installation can provide. In particular, it shows how many requests per second the server is capable of serving.

ApacheBench comes standard with the Apache distribution so on most systems it's already going to be there. There's already a really good tutorial on NixCraft on how to use ApacheBench so I won't go into it. Just check out the tutorial above and you'll have a good understanding to start with.

I will say that this portion should take a good a while to do properly because you'll be doing it a lot. You're going to want to tweak your HTTP server configuration to get the optimal performance and every time you make a change you're going to want to confirm the improvement.

It will get old.

Now, as I mentioned above this is by no means a definitive guide or anything. It's just the bare minimum that should be done when you're setting up a Linux web server.

One of the most frequent IT related tasks I have to deal with are email issues. Spam is a big part of my life, on both sides of the fence. I have to deal with legitimate email from clients getting flagged as spam on my network and I also have to make sure email StreetWise sends out on behalf of our clients doesn't get flagged as spam.

Over the years I've put together a list of requirements for making sure email campaigns go as smoothly as possible. Since no one can seem to get email campaigns right I figured I write it all down and post it and hope someone gets something out of it. Enjoy!

Check Spam Score

A Spam Score is a number assigned by the spam-scanning rules. The higher the score, the more "spamlike" the message appears. It's very important to how your email ranks so you can fix the before you send out the blast.

There's a great tool called Mail-Check that will evaluate an email and report on it's supposed spam score. It has an integrated SpamAssassin engine that performs a series of tests against your email to determine how much like spam it looks like.

The tool is in no way fool proof though. Be sure to use it as just a single source of information and don't make too many decisions based on the results.

Send Email Slowly

The basic idea behind this idea is that an email provider looks at how many emails is coming in from a particular domain and, if there's a lot of email coming in at , will assume it's either a spammer or a virus. Not a good label to have on your domain.

How slowly should email be sent? Unfortunately, that's a difficult question and the amount of email you need to send greatly affects the answer. For example, if you have 1,000 emails to send you probably don't need to worry about sending too fast. On the other hand if you have 250,000 emails to send slowing down the send process is HUGELY beneficial.

I've heard that the best approach is to throttle your sending by recipient domain. Now, I've never done this myself, but throttling by domain is supposed to be helpful because you can keep a consistent flow of email coming from the mail server but since there isn't a constant stream of email to one provider they don't know a bulk mailing is being sent to their network.

Get Whitelisted

This ones a bit of a no brainer but you NEED to get your email domain whitelisted by the email service providers you're sending to. This should include, at the very least, Yahoo!, gMail, MSN (hotmail) and AOL as well as the email recipients. This can be pretty tricky to do because the methods for getting whitelisted by the companies above tend to change fairly frequently. A good method to help would be to send individual requests to "postmaster@DOMAIN.tld", where DOMAIN.tld is the email providor you want to get whitelisted from, and explain your opt-in process and ask to be added to their whitelist.

It's also a good idea to ask your email recipients to add your email address to their personal whitelists. This helps because most email providers allow email through that is on their blacklist but on a users personal whitelist.

Setup and Maintain Your Email Server Properly

A lot of newbies make the mistake of thinking that if they can send email they should send email. This is pure folly. There are a few things you should do on the server level to ensure you're not going to look like a spammer. Off the top of my head here are a couple requirements that have to be implemented:

All e-mail servers must have valid reverse DNS records. Doing this allows email providers the ability to look you up and make sure you're not sending from a blackhole. A lot of spammers won't go through the trouble of setting up a reverse DNS record because they have to be ready to move at a moments notice. Setting one up shows you're serious.

All e-mail servers must be secured to prevent unauthorized or  anonymous use. Preventing Joe Spammer from sending email from your domain shows the email services that the email really came from you. If, for example, they see that anyone can send email from your sever the email services has no reason to trust it actually came from you.

Setup a Sender Policy Framework (SPF) DNS record for your domain. The SPF is an open standard specifying a technical method to prevent sender address forgery. This basically means that setting up an SPF record in your DNS shows the email provider that email coming from a server is being sent on your behalf.

Avoid Evil

This is a topic that's pretty absurd it even has to be mentioned but, sigh..., it has to be mentioned. (I guess I'm a bit of an idealist, but it seems to me that this stuff is just kind of  evil.)

Anyway, right off, you've got to be compliant with the federal Can Spam Act of 2003. If you're going to be sending bulk email it's important to understand the rules, especially when sending on behalf of your business and clients.

For the love God, don't try to hide, forge or misrepresent the sender of the e-mail and sending site of the e-mail in any way. I've had this asked of me more times than I can count and it always ends badly; it's an easy way to get yourself on a blacklist.

The last evil thing might not be too evil but I hate when I get an email and it doesn't say where they you got the my email address and why they're sending them an email. It's an easy thing to do, especially since the act of having that information requires honesty about the list, plus it makes your recipients feel secure.

Write Good Email Content

Much like in Search Engine Optimization a lot of what makes a good email is good content (by content I'm talking about the entire email here; not just the copy). Here's just a couple must dos:

1. Use alt-tags on ALL images.
2. Use short, descriptive, subject lines. Avoid special characters like the plauge.
3. Avoid using a lot of spammy key words and characters. For example; free, $, Prize, etc.
4. Keep the top header under 100 pixels tall and place a "View Email Online" link first and foremost.
5. Avoid center aligned content; it's best to keep the email body left aligned on a white background. Spam filters tend to flag colored backgrounds higher than white.
6. Include a plain text version of your email with every blast.

Unsubscribing Should be EASY

This is probably the most important piece in my opinion. It has to be easy, like ridiculously easy, for a user to remove themselves from your list. Bulk mailings should contain simple and obvious unsubscribe mechanisms. AOL recommend that this be in the form of a working link to a one-click unsubscribe system; however, a valid "reply to:" address may be used instead.

There are all sorts of methods for the unsubscribe flow; just pick one that requires the least investment for the user. Avoid the impulse to create a survey asking why a user doesn't want your email anymore. The fact they don't want it anymore should be enough. Anything more and you run the risk of pissing people off.

Summary

Looking over the above I'm sure I forgot some things. Still, it should be a good starting point.

It's important to remember that there really is no silver bullet when it comes to email marketing. Sadly, the reality is that even if all of the above is done there's still a chance your email will be flagged as spam. Most email providers take their users interests very seriously, while taking the stance that everyone else is evil until proven not evil. Even if they don't consider you evil they'll revert a good mailing to evil status after only a handful of complaints.

Additional Reading

AOL Whitelist Information
Email Secrets of a Top Converting Website
Create Better Performing Assets

One thing I've been waiting to become practical for a while is video mobile blogging (VMB). In case you don't know, VMB is the practice of taking a video with a mobile device and automatically adding it to a blog or website. It's pretty much exactly like a moblog except instead of sending photos you send a video. Due to poor quality, varied video formats and slow network speeds VMB just hasn't become mainstream yet.

The general idea behind a VMB works like this:

1. Record video
2. Send email with video as attachment
3. Script parses email and extracts the video
4. Process video for online play
5. Extract all meta data
6. Record in database

Really, a basic idea; works GREAT for a moblog . The reason's pretty simple; ever since phones started coming with cameras as a standard accessory they've improved in quality and couple that with a data plan (so email can be sent) and it's a perfect fit.

The same thing can't be said for video though. Video on a cell phone is just too new and, well, clunky to be useful in business level applications. In fact, RIM just recently upgraded the Blackberry Curve OS to include video playback a couple of months ago (yeah, I have a Curve).

Problem is, the quality on the video sucks and it takes me about 15 minutes to send the video as an email on an EDGE network. Now, I'm pretty sure my speed isn't a good baseline; I have AT&T so my service downright sucks. That being said, I did do a couple tests on a couple of my friend's phones from various vendors and even though the speed was a little faster the quality was still crappy.

The above doesn't really matter though because, according to friend I have in the mobile industry, there are about 20 different formats the different phones use to encode videos in. This is a biggie; embedding a video into a webpage requires that videos be encoded to the lowest common denominator. Since phones aren't using a standard format the videos would have to be converted to a standard format on the server which increases the complexity and cost of such a platform.

It should be noted that a lot of phones are encoding their videos using the 3GP format; but because it's not common enough to be a standard codec on users computers I don't consider this a full solution.

Here's a quick fix for a weird issue I ran into while working with MySQL on Windows.

The first time I moved code from my Windows box to one of my Linux servers I ran into an error about a MySQL table not existing in one of my queries. I checked the table and the code to verify everything was as it should be and noticed that all my tables were lowercase instead of CamelHumped. Odd....

Here's the SQL:

$sql = "SELECT * FROM TableName WHERE id = '".$DB->es($id)."'";

It turns out that MySQL on Windows will convert all tables created to a lowercase name. To remain consistent with Linux you should add the below to your my.cnf file:

set-variable=lower_case_table_names=0

Adding the above line will preserve the naming conventions between both platforms.

Like most web developers, I use Linux on a daily basis. Because I always have to revisit how to do some of the basics from time to time I thought I'd put together a list of some of the commands I use the most along with some of their examples. NOTE: the paths are all from RHEL 4 and CENTOS 4 & 5.

Enjoy!

MySQL

Example of mysqldump::

mysqldump -u USER -pPASSWORD DATABASE > filename.sql

Example of restore DB::

mysql -u USER -p DBNAME < dump.sql

Start the MySQL service::

/etc/init.d/mysqld start

Stop MySQL service::

/etc/init.d/mysqld stop

Add User::

GRANT ALL PRIVILEGES ON *.* TO 'USER'@'localhost' IDENTIFIED BY 'PASSWORD' WITH GRANT OPTION;

Export Single Table::

mysqldump --opt -u USER-pPASSWORD DataBaseName TableName > FileName

rsync

rsync is used primarily synchronize data between two directories or servers.

Sync 2 servers

rsync -p -r -a -z -v /path/to/dir 2.2.2.2:/path/to/dir

Sync 2 directories

rsync -p -r -a -z -v /path/to/dir /path/to/new/dir

Lighttpd

Start the lighttpd service::

/etc/init.d/lighttpd start

Stop the lighttpd service::

/etc/init.d/lighttpd stop

Path to conf::

/etc/lighttpd/lighttpd.conf

SSH

Restart SSH Server

/etc/rc.d/init.d/sshd restart

Start SSH Server

/etc/rc.d/init.d/sshd start

Stop SSH Server

/etc/rc.d/init.d/sshd stop

Tar/GZ

Create tarball

tar -cvf directory.tar directory/

Create gzipped tar file

tar -czvf directory.tgz directory/

Extracting tarball:

tar -xvf directory.tar

Extracing gzipped tar:

tar -xzvf directory.tgz

Misc.

Delete the mail queue

rm /var/spool/mqueue/*

Count files in directory recursively

ls -R -1 proc | wc -l

I've been using lighttpd as my main production web server since the summer of 2007. I first heard about lighttpd from a server administrator I was working with on a project a few years earlier but since my clients weren't getting traffic that Apache couldn't handle I didn't really pay it too much attention. As far as I was concerned it was a cool idea, building a better webserver, but since lighttpd was so new and support and documentation being a little sparce, compared to, say, Apache's documentation, all using lighttpd meant was that I had to stay away from Apache specific functionality and I'd be ok.

Fast forward a couple years and I'm working on projects that Apache is really struggling with. The point of removing Apache from my toolbox started because one of our projects at StreetWise, the official community for one of our biggest client's programs, was having serious load issues. The site was very popular, getting over 2,000,000 hits a day, and Apache was having a hard time dealing. Here we were with 8 web servers, on a load balancer, 3 DB servers, and all of the web servers were just getting hammered; it was cascading, where one by one the servers would go into swap and all traffic would be shifted to a different server, only to have the same thing happen again until all servers died. It was a very, very, stressful time for my team and I.

We tried configuring the hell out of Apache, upgrading the server's RAM and adding additional CPUs but finally came to realize that continuing with Apache meant we'd just have to get more servers. Apache can only handle so many users at one time, like all webservers really, so we needed a web server that could handle more users at one time. It's a pretty basic While adding additional servers to a project is cheaper than throwing people at it, if it's a software issue, there's additional costs with hardware in my team especially because we manage the servers ourselves.

This wasn't good; it meant explaining to the client that we would need more money outside of our projections, which is embarrassing and, worse, my team would have to manage more servers (we were already managing 18 in total at the time).

It was at this time that lighttpd came back into play. Since Apache wasn't working out so hot in the load we were under it had to be replaced. I remembered how lighttpd was started as a proof of concept for the C10K problem and had some pretty impressive benchmarks so I started researching more about it.

While lighttpd's penetration wasn't even a blip compared to Apache or IIS we installed it on a one of our dev servers and started testing it.

Installing lighttpd was pretty straightforward though setting up php to run in fast-cgi mode was a little more challenging than usual. Right away, one of the biggest challenges was the configuration though. After using Apache for, well, forever really, moving to how lighttpd was configured was pretty difficult.

lighttpd.conf example

$HTTP =~ "(www.)?your-domain.com" {
 
  server.document-root = "/www/your-domain.com/html"
 
#### auth module
## read authentication.txt for more info
auth.backend               = "htpasswd"
auth.backend.htpasswd.userfile = "/www/your-domain.com/.htpasswd"
#auth.backend.plain.groupfile = "lighttpd.group"
auth.require               = ( "/" =>
                               (
                                 "method"  => "basic",
                                 "realm"   => "Server Info",
                                 "require" => "valid-user"
                               )
                             )
 
}

Compare the above to the below.
httpd.conf example

<VirtualHost *>
    ServerName your-domain.com
    DocumentRoot /www/your-domain.com/html
    ServerAdmin contact@your-domain.com
    <IfModule mod_suphp.c>
        suPHP_UserGroup nobody nobody
    </IfModule>
</VirtualHost>

Right away, you can see how Apache and Lighttpd have different philosophies on layout and structure. Apache takes an XML structure for their configuration files while lighttpd formats its configurations like an array.

Anyway, after familiarizing ourselves in how lighttpd was configured we started the process of moving all the production servers away from Apache over to lighttpd. We went one server at a time, installing lighttpd, configuring, benchmarking and then changing the server's init so lighttpd was started and Apache wasn't. All told the process took about a week.

Since then, I haven't had a single issue related to load on one of lighttpd boxes. If you're serious about performance and care about how your projects function check out lighttpd ASAP.