This year’s Black Hat broke records yet again with the highest levels of attendance, including highest number of countries represented and, based on the size of the business hall, companies represented as well. While it featured some truly novel technical methods and the advanced security research for which it is so well known, this year’s conference even more than others reflected an institutionalization of the status quo within the security industry. Rather than reflecting the major paradigm shifts that are occurring in the security community, it seemed to perpetuate the insularity for which this community is often criticized.

In her Black Hat keynote speech, Jennifer Granick, lawyer and Director of Civil Liberties at Stanford University, noted that inclusion is at the heart of the hacker’s ethos and called for the security community to take the lead and push forth change within the broader tech sector. She explicitly encouraged the security community to refrain from being so insular, and to transform into a community that not only thinks globally but is also much more participatory in the policies and laws that directly affect them. While she focused on diversity and equality, there are several additional areas where the security community could greatly benefit from a more expansive mindset. Unfortunately, these strategic level discussions were largely absent from the majority of the Black Hat briefings that followed the keynote. The tactical, technical presentations understandably comprise the majority of the dialogue and garner the most attention.  However, given the growing size and expanding representation of disparate parts of the community, there was a noticeable absence of nuanced discussion about the state of the security community, including broader thinking about the three big strategic issues and trends that will define the community for the foreseeable future:

• Where’s the threat? Despite a highly dynamic threat landscape, ranging from foreign governments to terrorist organizations to transnational criminal networks, discussion of these threat actors was embarrassingly absent from the panels this year. Although the security community is often criticized for over-hyping the threat, this was not the case at this year’s Black Hat. Even worse, the majority of discussions of the threat focused on the United States and Western European countries as the greatest security threats. Clearly, technology conferences must focus on the latest technological approaches and trends in the field. However, omitting the international actors and context in which these technologies exist perpetuates an inward-facing bias of the field that leads many to misunderstand the nature, capabilities and magnitude of the greatest threats to corporate and national security.
• Toward détente? Last year’s Black Hat conference was still reeling from the Snowden revelations that shook the security community. A general feeling of distrust of the U.S. government was still apparent in numerous panels, heightening interest in privacy and circular discussions over surveillance. While sentiments of distrust still exist, this no longer appears to be the only perspective. In a few briefings, there was a surprising lack of the hostility toward the government that existed at similar panels a year ago. In fact, the very few panels that had government representation were not only well attended, but also contained civil discourse between the speakers and the audience. This does not mean that there were softball questions. On the contrary, there was blunt conversation about the "trust deficit" between the security community and the government. For instance, the biggest concern expressed regarding data sharing with the government (including the information sharing bill which Congress discussed last week, but is now delayed) was not about information sharing itself, but rather how the security community can trust that the government can protect the shared data in light of OPM and other high-profile breaches. This is a very valid concern and one that ignited a lot of bilateral dialogue. Organizations from the DHS to the Federal Trade Commission requested greater partnerships with the security community. While there are certainly enormous challenges ahead, it was refreshing to see not only signs of a potential thawing of relations between the government and the security community, but also hopefully some baby steps toward mutually beneficial collaboration.
• Diversity. The general lack of diversity at the conference comes as no surprise given the well-publicized statistics of the demographics of the security community, as well as the #ilooklikeanengineer campaign that took off last week. However, diversity is not just about gender – it also pertains to diversity of perspectives, backgrounds and industries. Areas such as human factors, policy and data science seemed to be less represented than in previous years, conflicting with much of the rhetoric that permeated the business hall. In many of the talks that did cover these areas, there were both implicit and explicit requests for a more expansive partnership and role within the community.

Given the vast technological, geopolitical and demographic shifts underway, the security community must transform beyond the traditional mindset and truly begin to think beyond the insular perimeter. Returning to Granick’s key points, the security community can consciously provide leadership not only in shaping the political discourse that impacts the entire tech community, but also lead by example through promoting equality and thinking globally. The security community must play a participatory role in the larger strategic shifts that will continue to impact it instead of remaining an insularly focused island in the desert.

Natural Language Processing (NLP) is a diverse field in computer science dedicated to automatically parsing and processing human language. NLP has been used to perform authorship attribution and sentiment analysis, as well as being a core function of IBM’s Watson and Apple’s Siri. NLP research is thriving due to the massive amounts of diverse text sources (e.g., Twitter and Wikipedia) and multiple disciplines using text analytics to derive insights. However, NLP can be used for more than human language processing and can be applied to any written text. Data scientists at Endgame apply NLP to security by building upon advanced NLP techniques to better identify and understand malicious code, moving toward an NLP methodology specifically designed for malware analysis—a Malicious Language Processing framework. The goal of this Malicious Language Processing framework is to operationalize NLP to address one of the security domain’s most challenging big data problems by automating and expediting the identification of malicious code hidden within benign code.

How is NLP used in InfoSec?

Before we delve into how Endgame leverages NLP, let’s explore a few different ways others have used it to tackle information security problems:

• Domain Generation Algorithm classification– Using NLP to identify malicious domains (e.g., blbwpvcyztrepfue.ru) from benign domains (e.g., cnn.com)
• Source Code Vulnerability Analysis– Determining function patterns associated with known vulnerabilities, then using NLP to identify other potentially vulnerable code segments.
• Phishing Identification– A bag-of-words model determines the probability an email message contains a phishing attempt or not.
• Malware Family Analysis–Topic modeling techniques assign samples of malware to families, as discussed in my colleague Phil Roth’s previous blog.

Over the rest of this post, I’ll discuss how Endgame data scientists are using Malicious Language Processing to discover malicious language hidden within benign code. 

Data Acquisition/Corpus Building

In order to perform NLP you must have a corpus, or collection of documents. While this is relatively straightforward in traditional NLP (e.g., APIs and web scraping) it is not necessarily the same in malware analysis. There are two primary techniques used to get data from malicious binaries: static and dynamic analysis. 

Fig 1. Disassembled source code

Static analysis, also called source code analysis, is performed using a disassembler providing output similar to the above (Fig 1). The disassembler presents a flat view of a binary, however structurally we lose important contextual information by not clearly delineating the logical order of instructions. In disassembly, jmp or call instructions should lead to different blocks of code that a standard flat file misrepresents. Luckily, static analysis tools exist that can provide call graphs that provide logical flow of instructions via a directed graph, like this and this.

Dynamic analysis, often called behavioral analysis, is the collection of metadata from an executed binary in a sandbox environment. Dynamic analysis can provide data such as network access, registry/file activity, and API function monitoring. While dynamic analysis is often more informative, it is also more resource intensive, requiring a suite of collection tools and a sandboxed virtual environment. Alternatively, static analysis can be automated to generate disassembly over a large set of binaries generating a corpus ready for the NLP pipeline. At Endgame we have engineered a hybrid approach that automates the analysis of malicious binaries providing data scientists with metadata from both static and dynamic analysis.

Lexical Parsing

Lexical parsing is paramount to the NLP process as it provides the ability to turn large bodies of text into individual tokens. The goal of Malicious Language Processing is to parse a binary the same way an NLP researcher would parse a document:

To generate the “words” in this process we must perform a few traditional NLP techniques. First is tokenization, the process of breaking down a string of text into meaningful segments called tokens.  Segmenting on whitespace, new line characters, punctuation or regular expressions can generate tokens. (Fig 2)

Fig 2. Tokenized disassembly

The next step in the lexical parsing process is to merge families of derivationally related words with similar meaning or text normalization. The two forms of this process are called stemming and lemmatization.

Stemming seeks to reduce a word to its functional stem. For example, in malware analysis this could reduce SetWindowTextA or SetWindowTextW to SetWindowText (Windows API), or JE, JLE, JNZ to JMP(x86 instructions) accounting for multiple variations of the essentially the same function.

Lemmatization is more difficult in general because it requires context or the part-of-speech tag of a word (e.g., noun, verb, etc.). In English, the word “better” has “good” as its lemma. In malware we do not yet have the luxury of parts-of-speech tagging, so lemmatization is not yet applicable. However, a rules-based dictionary that associates Windows API equivalents of C runtime functions may provide a step towards lemmatization, such as mapping _fread to ReadFile or _popen to CreateProcess.

Semantic Networks

Semantic or associative networks represent the co-occurrence of words within a body of text to gain an understanding of the semantic relationship between words. For each unique word in a corpus, a node is created on a directed graph. Links between words are generated with an associated weight based on the frequency that the two words co-occurred. The resulting graph can then be clustered to derive cliques or communities of functions that have similar behavior.

A malicious language semantic network could aid in the generation of a lexical database capability for malware similar to WordNet. WordNet is a lexical database of English nouns, verbs, and adjectives grouped into sets of cognitive synonyms. Endgame data scientists are in the incipient stages of exploring ways to search and identify synonyms or synsets of malicious functions. Additionally, we hope to leverage our version of WordNet in the development of lemmatization and the Parts-of-Speech tagging within the Malicious Language Processing framework.

Parts-of-Speech Tagging

Parts-of-Speech (POS) tagging is a piece of software capable of tagging a list of tokens in a string of text with the correct language annotation, such as noun, verb, etc. POS Tagging is crucial for gaining a better understanding of text and establishing semantic relationship within a corpus. Above I mentioned that there is currently no representation of POS tagging for malware. Source code may be too abstract to break down into nouns, prepositions or adjectives. However, it is possible to treat subroutines as “sentences” and gain an understanding of functions used as subjects, verb and predicates. Using pseudo code for a process injection in Windows, for example, would yield the following from a Malicious Language Processing POS-Tagger:

Closing Thoughts

While the majority of the concepts mentioned in this post are being leveraged by Endgame today to better understand malware behavior, there is still plenty of work to be done. The concept of Malicious Language Processing is still in its infancy. We are currently working hard to expand the Malicious Language Processing framework by developing a malicious stop word list (a list of the most common words/functions in a corpus of binaries) and creating an anomaly detector capable of determining which function(s) do not belong in a benign block of code. With more research and larger, more diverse corpuses, we will be able to understand the behavior and basic capabilities of a suspicious binary without executing or having a human reverse engineer it. We view NLP as an additional tool in a data scientist’s toolkit, and a powerful means by which we can apply data science to security problems, quickly parsing the malicious from the benign.

Read more blog posts about Data Science.

Follow Bobby on Twitter @filar.

This year’s Pwnie Award goes to ... OPM!

The U.S. Office of Personnel Management (known as OPM) won the “Most Epic Fail” award at the 2015 Black Hat Conference for the worst known data breach in U.S. government history, with more than 22 million employee profiles compromised. Joining OPM as contenders for this award were other victims of high-profile cyber attacks, including Poland's Plus Bank and the website AshleyMadison.com. The truth is, hardly a day goes by without news of cyber intrusions. As an example, according to databreachtoday.com, just in recent months PNI Digital Media and many retailers such as Wal-Mart and Rite-Aid had their photo services compromised, UCLA Health’s network was breached, and information of 4.5 million people may have been exposed. Criminals and nation-state actors break into systems for many reasons with catastrophic and often irremediable consequences for the victims.

Traditionally, security experts are the main force for investigating cyber threats and breaches. Their expertise in computers and network communication provides them with an advantage in identifying suspicious activities. However, with more data being collected, not only in quantity but also in variety, data scientists are beginning to play a more significant role in the adventure of hunting malicious attackers. At Endgame, the data scientist team works closely with the security and malware experts to monitor, track and identify cyber threats, and applies a wide range of data science tools to provide our customers with intelligence and insights. In this post, I’ll explain how we analyze attack data collected from a honeypot network, which provides insight into the locations of attackers behind those activities. The analysis captures those organized attacks from a vast amount of seemingly separated attempts.

This post is divided into three sections. The first section describes the context of the analysis and provides an overview of the hacking activities. The second section focuses on investigating the files that the attackers implanted into the breached systems. Finally, the third section demonstrates how I identified similar attacks through uncovering behavioral characteristics. All of this demonstrates one way that data science can be applied to the security domain. (My previous post explained another application of data science to security.)

Background

Cyber attackers are constantly looking for targets on the Internet. Much like a lion pursuing its prey, an attacker usually conducts a sequence of actions, known as the cyber kill chain, including identifying the footprints of a victim system, scanning the open ports of the system, and probing the holes trying to find an entrance into the system. Professional attackers might be doing this all day long until they find a weak system.

All of this would be bad news for any weak system the attacker finds – unless that weak system is a honeypot. A honeypot is a trap set up on the Internet with minimum security settings so an attacker may easily break into it, without knowing his/her activities are being monitored and tracked. Though honeypots have been used widely by researchers to study the methods of attackers, they can also be very useful to defenders. Compared to sophisticated anomaly detection techniques, honeypots provide intrusion alerts with low false positive rates because no legitimate user should be accessing them. Honeypots set up by a company might also be used to confuse attackers and slow down the attacks against their networks. New techniques are on the way to make setting up and managing honeypots easier and more efficient, and may play an increasingly prominent role in future cyber defense.

A network of honeypots is called a honeynet. The particular honeynet for which I have data logged activities showing that an attacker enumerated pairs of common user names and passwords to enter the system, downloaded malicious files from his/her own hosting servers, changed the privilege over the files and then executed them. During the period from March 2015 through the end of June 2015, there were more than 21,000 attacker IP addresses being detected, and about 36 million SSH attempts being logged. Attackers have tried 34,000 unique user names and almost 1 million unique passwords to break into those honeypots. That’s a lot of effort by the attackers to break into the system. Over time, the honeynet has identified about 500 malicious domains and more than 1000 unique malware samples.

The IP addresses that were owned by the attackers and used to host malware are geographically dispersed. Figure 1 shows that the recorded attacks mostly came from China, the U.S., the Middle East and Europe. While geographic origination doesn’t tell us everything, it still gives us a general idea of potential attacker locations. 

Figure 1. Attacks came from all around the world, color coded on counts of attack. The darker the color, the greater the number of attacks originating from that country.

The frequency of attacks varies daily, as shown in Figure 2, but the trend shows that more attacks were observed during workdays than weekends, and peaks often appear on Wednesday or Thursday. This seems to support the suspicion that humans (other than bots) were behind the scenes, and professionals instead of amateur hobbyists conducted the attacks. 

Figure 2. Daily Attack Counts.

Now that we understand where and when those attacks were orchestrated, we want to understand if any of the attacks were organized. In other words, were they carried out by same person or same group of people over and over again?

Attackers change IP addresses from attack to attack, so looking at the IP addresses alone won’t provide us with much information. To find the answer to the question above, we need to use the knowledge about the files left by the attackers. 

File Similarity

Malware to an attacker is like a hammer and level to a carpenter. We expect that an attacker would use his/her set of malware repeatedly in different attacks, even though the files might have appeared in different names or variants. Therefore, the similarity across the downloaded malware files may provide informative links to associated attacks.

One extreme case is a group of 17 different IPs (shown in Figure 3) used on a variety of days containing exactly the same files and folders organized in exactly the same structure. That finding immediately portrayed a lazy hacker who used the same folder time and time again. However, we would imagine that most attackers might be more diligent. For example, file structures in the hosting server may be different, folders could be rearranged, and the content of a malicious binary file may be tweaked. Therefore, a more robust method is needed to calculate the level of similarity across the files, and then use that information to associate similar attacks.

Figure 3. 17 IPs have exactly the same file structure.

How can we quantitatively and algorithmically do this?

The first step is to find similar files to each of the files in which we are interested. The collected files include different types, such as images, HTML pages, text files, compressed tar balls, and binary files, but we are probably only interested in binary files and tar balls, which are riskier. This reduces the number of files to work on, but the same approach can be applied to all file types.

File similarity computation has been researched extensively in the past two decades but still remains a rich field for new methods. Some mature algorithms to compute file similarities include block-based hashing, Context-Triggered Piecewise (CTP) hashing (also known as fuzzy hashing), and Bloom filter hashing. Endgame uses more advanced file similarity techniques based on file structural and behavioral attributes. However, for this investigation I used fuzzy hashing to compute file similarities for simplicity and since open source code is widely available.

I took each of the unique files based on its fuzzy hashing string and computed the similarity to all the other files. The result is a large symmetric similarity matrix for all files, which we can visualize to check if there are any apparent structures in the similarity data. The way I visualize the matrix is to connect two similar files with a line, and here I would choose an arbitrary threshold of 80, which means that if two files are more than 80% similar, they will be connected. The visualization of the file similarity matrix is shown in Figure 4.

Figure 4. Graph of files based on similarity.

It is visually clear that the files are indeed partitioned into a number of groups. Let’s zoom into one group and see the details in Figure 5. The five files, represented by their fuzzy hash strings, are connected to each other, having mutual similarity of over 90%. If we look at them very carefully, they only differ in one or two letters in the strings, even they have totally different file names and MD5 hashes. VirusTotal recognizes four out of the five malware, and the scan reports indicate that these malware are Linux Trojan. 

Figure 5. One group of similar files.

Identifying Similar Attacks

Now that we have identified the groups of similar files, it’s time to identify the attacks that used similar malware. If I treat each attack as a document, and the malware used in an attack as words, I can construct a document-term matrix to encapsulate all the attack information. To incorporate the malware similarity information into the matrix, I tweaked the matrix a bit. For malware that were not used in a specific attack, but that still share a certain amount of similarity with the malware being used, the malware will assume the value of the similarity level for that attack. For example, if malware M1 was not used in attack A1, but M1 is most similar to malware M2 which was used in attack A1, and the similarity level is 90%, then the element at cell (A1, M1) will be 0.9, while (A1, M2) be 1.0.

For readers who are familiar with NLP (Natural Language Processing) and text mining, the matrix I’ve described above is similar to a document-term matrix, except the values are not computed from TF-IDFs (Term Frequency-Inverse Document Frequency). More on applications of NLP on malware analysis can be found in a post published by my fellow Endgamer Bobby Filar. The essence of such a matrix is to reflect the relationship between data records and features. In this case, data records are attacks and features are malware, while for NLP they are documents and words. The resulting matrix is an attack-malware matrix, which has more than 400 columns representing malware hashes. To get a quick idea of how the attacks (the rows) are dispersed in such a high dimensional space, I plotted the data using the T-SNE (t-Distributed Stochastic Neighbor Embedding) technique and colored the points according to the results from K-means (K=10) clustering. I chose K=10 arbitrarily to illustrate the spatial segmentation of the attacks. The T-SNE graph is shown in Figure 6, and each color represents a cluster labeled by the K-means clustering. T-SNE tries to preserve the topology when projecting data points from a high dimensional space to a much lower dimensional space, and it is widely used for visualizing the clusters within a data set.

Figure 6 shows that K-Means did a decent job of spatially grouping close data points into clusters, but it fell short of providing a quantitative measurement of similarity between any two data points. It is also quite difficult to choose the optimum value for K, the number of clusters. To overcome the challenges that K-Means faces, I will use Latent Semantics Indexing (LSI) to compute the similarity level for the attack pairs, and build a graph to connect similar attacks, and eventually apply social network analytics to determine the clusters of similar attacks.

Figure 6. T-SNE projection of Attack-Malware matrix to 2-D space.

LSI is the application of a particular mathematical technique, called Single Value Decomposition or SVD, to a document-term matrix. SVD projects the original n-dimensional space (with n words in columns) onto a k-dimensional space, where k is much smaller than n. The projection then transforms a document’s vector in n-dimensional space into a vector in the reduced k-dimensional space under the requirement that the Euclidean distance between the original matrix and the resulting matrix after transformation is minimized.

SVD decomposes the attack-malware matrix into three matrices, one of which defines the new dimensions in the order of significance. We call the new dimensions principal components. The components are ordered by the amount of explained variance in the original data. Let’s call this matrix attack-component matrix. With the risk of losing some information, we can plot the attack data points on the 2-d space using the first and the second components just to illustrate the differences between data points, as shown in Figure 7. The vectors pointing to perpendicular directions are most different from each other.

Figure 7. Attack data projected to the first and second principal components.

The similarity between attacks can be computed with the results of LSI, more specifically, by calculating the dot product of the attack-component matrix.

Table 1. Attacks Similar to Attack from 61.160.212.21:5947 on 2015-03-23.

I connect two attacks if their similarity is above a certain threshold, e.g. 90%, and come up with a graph of connected attacks, shown in Figure 8.

Figure 8. Visualization of attacks connected by similarity.

There are a few big component subgraphs in the large graph. A component subgraph represents a group of attacks closely similar to each other. We can examine each of them in terms of what malware were deployed in the given attack group, what IP addresses were used, and how frequently the attacks were conducted.

I plotted the daily counts of attack for the two largest attack groups in Figure 9 and Figure 10. Both of them show that attacks happened more often on weekdays than on weekends. These attacks may have targeted different geo-located honeypots in the system and could be viewed as a widely expanded search for victims.

Figure 9. Daily counts of attack in one group.

Figure 10. Daily counts of attack in another group.

We can easily find out where those attackers’ IPs were located (latitude and longitude), and the who-is data associated with the IPs. But it’s much more difficult to fully investigate the true identity of the attackers.

Summary

In this post, I explained how to apply data science techniques to identify honeypot attackers. Mathematically, I firmed the problem as an Attack-Malware matrix, and used fuzzy hashing to represent files and compute the similarity between files. I then employed latent semantic indexing methods to calculate the similarity between attacks based on file similarity values. Finally, I constructed a network graph where similar attacks are linked so that I could apply social network analytics to cluster the attacks.

As with my last blog post, this post demonstrates that data science can provide a rich set of tools that help security experts make sense of the vast amount of data often seen in cyber security and discover relevant information. Our data science team at Endgame is constantly researching and developing more effective approaches to help our customers defend themselves – because the hunt for attackers never ends.

Read more blog posts by Richard Xie.

Read more blog posts about Data Science.

Follow Richard on Twitter @maximumlike.

The concept of deterrence consistently fails to travel well to the cyber realm. One (among the many) reasons is that, although nuclear deterrence is achieved through nuclear means, cyber deterrence is not achieved solely through cyber means. In fact, any cyber activity meant for deterrence is likely going to be covert, while the more public deterrence activities fall into diplomatic, economic, financial, and legal domains. Less than six months after President Obama  signed an executive order to further expand the range of responses available to penalize individuals or companies conducting “malicious cyber-enabled activities”, there are now reports that it may be put to use in a big and unprecedented way. Numerous news outlets have announced the possibility of sanctions against Chinese individuals and organizations associated with economic espionage within the cyber domain. If the sanctions do come to fruition, it may not be for a few more weeks. Until then, below are some of the immediate questions that may help provide greater insight into what may be one of the most significant policy evolutions in the cyber domain.

1. Why now?  

Many question the timing of the potential Chinese sanctions, especially given President Xi Jinping’s upcoming state visit to Washington. It is likely that a combination of events over the summer in both the US and China have instigated this policy shift:

Chinese domestic factors: China’s stock market has been consistently falling since June, with the most visible plunge occurring at the end of August, which has had global ramifications. A major slowdown in economic growth has also hit China, which by some estimates could be as low as 4% (counter to the ~10% growth of the last few decades, and lower than even the recent record low of 7.4% in 2014). The latest numbers from today reinforce a slowing economy, with the manufacturing sector recording a three-year low. Simultaneously, President Xi continues to consolidate power, leading a purge of Communist Party officials targeted for corruption and asserting greater control of the military. In short, President Xi is looking increasingly vulnerable, handling economic woes as well as continuing a political power grab, which has led to two influential generals to resign and discontent among some of the highest ranks of leadership.

US domestic factors: The most obvious reason for the timing of potential US sanctions seems to be in response to this summer’s OPM breach, which has been largely attributed to China. This is just the latest in an ongoing list of public and private sector hacks attributed to China, including United Airlines and Anthem. The OPM breach certainly helped elevate the discussions over retaliation, but it’s unlikely that it was the sole factor. Instead, the persistent theft of IP and trade secrets, undermining US competitiveness and creating an uneven playing field, is the dominant rationale provided. Ranging from the defense sector to solar energy to pharmaceuticals to tech, virtually no sector remains unscathed by Chinese economic espionage. The continuing onslaught of attacks may have finally reached a tipping point.

The White House also has experienced increased pressure to respond in light of this string of high-profile breaches. Along with pressure from foreign policy groups and the public sector, given the administration’s pursuit of greater public-private partnerships, there is likely similar pressure from powerful parts of the private sector – including the financial sector and Silicon Valley – impacting the risk calculus of economic and financial espionage. For instance, last week, Secretary of Defense Ashton Carter visited Silicon Valley, encouraging greater cooperation and announcing a $171 million joint venture with government, academia and over 160 tech companies. These partnerships have been a high priority for the administration, meaning that the government likely feels pressure to respond when attacks attributed to the Chinese, such as the GitHub attacks this spring, hit America’s tech giants.

2. Why is this different from other sanctions?

Sanctions against Russia and Iran were in response to the aggressive policies of those countries, while those against North Korea were in response to the Sony breach. However, each of these countries lacks the economic interdependence with the US that exists for China.  Mutually assured economic destruction is often used to describe the economic grip the US and China have on each other’s economies. The United States is mainland China’s top trading partner, based on exports plus imports, while China is the United States’ third largest trading partner, following the European Union and Canada. Compare this to the situation in Russia, North Korea, and Iran, the most prominent countries facing US sanctions, none of which have significant trade interdependencies with the US.

Similarly, foreign direct investment (FDI) between China and the US is increasingly significant, with proposals for a bilateral investment treaty (BIT) exchanged this past June, and discussions ongoing in preparation for President Xi’s visit this month. China is also the largest foreign holder of  US Treasury securities, despite its recent unloading of Treasury bonds to help stabilize its currency. Compare this to Russia, North Korea, or Iran, none of which the US economy relied on prior to their respective sanctions. Even in Iran and Russia’s strongest industry – oil and gas– the US has become less reliant and more economically independent, especially given that the US was the world’s largest producer of oil in 2014.

3. Who or what might be targeted?

If sanctions are administered, the US will most likely continue its use of “smart” or targeted sanctions that focus on key individuals and organizations, rather than the entire country. The US sanctions against Russia provide some insight into the approach the administration might take. Russian sanctions are targeted at Putin’s inner circle, including its affiliated companies. These range from defense contractors to the financial sector to the energy sector, and include close allies such as Gennady Timchenko.  Similarly, North Korean sanctions following the Sony hack focused on three organizations and ten individuals. In the case of China, the state-owned enterprises (SOEs)deemed to reap the most benefits from economic espionage will likely be targeted. In fact, the top twelve Chinese companies are SOEs, meaning they have close ties to the government. More specifically, sanctions could include energy giants CNOOC, Sinopec and PetroChina, some of the large banks, or the global tech giant Huawei because of their large role in the economy and their potential to benefit from IP theft. Interestingly, the largest Chinese companies do not include several of their more famous tech companies, such as Alibaba, Tencent, Baidu and Xiaomi. Most of these enterprises have yet to achieve a significant global footprint, which means they are less likely to top any sanctions list. In considering who among Xi’s network might be targeted, some point to the Shaanxi Gang, Xi’s longtime friends, while others look at those most influential within the economy, such as Premier Li Keqiang.

Given President Xi’s upcoming visit, is the talk of sanctions diplomatic maneuvering, or will it be backed by concrete action? If enacted, the administration’s intent will be revealed through the actual targets of the sanctions.  If the objective is to deter future cyber aggression, then sanctions must be targeted at these influential state-owned companies and inner circle of the regime.  Otherwise, it will be perceived as a purely symbolic act both in the United States and in China and lack the teeth to truly enact change. 

Read more blog posts by Andrea Little Limbago.

Follow Andrea on Twitter @limbagoa

A free and open Internet is the cornerstone of net neutrality, advocated by civil liberties groups and the US government alike. A wide range of actors have taken this concept to the extreme by publicly releasing very private information and pictures.  This reflects a laissez-faire approach to the Internet, completely removing government intervention from the equation. Simultaneously, authoritarian regimes have implemented policies and approaches aimed at Internet censorship, reflecting a protectionist view, with the government determining exactly what information will be accessible to its citizens. But many democratic states are testing the waters with a third approach: Minimal intervention in the name of privacy and protecting citizens. Similar to Keynesian economics, where some government intervention is necessary to optimize economic stability, this third way may serve as a harbinger for the future of cyber security—especially as the public now very clearly understands just how fragile their private data is on the Internet.

Too Little: Laissez-Faire Data Freedom

Continuing the economics analogy, a laissez-faire approach to data freedom argues that data wishes to be free, and government intervention to prevent disclosures is anathema to that principle. The release of private information is already becoming a cottage industry from which blackmailers hope to reap financial benefits. One example is the Wikileaks compilation and aggregated release of the emails of Sony employees following last December’s attack. Similarly, last month’s Ashley Madison hack went a step further, publishing information that not only has the potential to ruin the personal and professional lives of its customers, but also their families.

Personally identifiable information (PII) is also a key target of disclosures and has implications both for personal as well as national security. The Islamic State of Iraq and Syria (ISIS) recently released email addresses, phone numbers, passwords and names of US military personnel, calling for attacks against the service members and their families. Other recent cases of PII theft include the Anthem and OPM breaches. Most of these were instigated by nation-states and criminal groups who have yet to publicly release the data, but who nevertheless now have access a vast trove of PII. And it’s not just adversarial groups who are releasing PII, but also data brokers, which are poorly regulated and have released billions of customer records. For instance, in 2013 Experian accidentally sold the PII of close to two-thirds of Americans to a criminal group in Vietnam.

Finally, with the omnipresence of smart phones, very private pictures also are released online without personal consent. This ranges from celebrities’ hacked iPhones, to photos used as a component of cyber bullying or revenge porn.

In each of these broad categories, the argument that data demands to be free often prevails, leading to victim blaming instead of facing the larger issue that – given the slow pace of the legal system – currently perpetrators face few, if any, repercussions for the theft and posting of personal information.

Too Much: Government Intervention & Information Control

Many authoritarian regimes operate at the opposite end of the spectrum when it comes to information freedom. For example, as the Chinese stock market plunged at the end of last month, impacting markets across the globe, the major Chinese newspapers either barely referenced the largest drop in eight years, or failed to mention it at all. The search engine Baidu and the micro-blogging site Weibo blocked much of the content related to information about the crash. Similarly, after promising not to censor the Internet a few years ago, the Malaysian government blocked the Sarawak Report, a UK-based news website, following its publication of an article alleging a bank transfer from a state investment fund into the Malaysian Prime Minister’s personal account. Russia was also busy with censorship last week, removing Russian Wikipedia following similar censorship of Reddit earlier this month.

This behavior is not just limited to authoritarian regimes; democracies are also increasingly censoring material. The Mexican government continues to control the narrative surrounding the students who disappeared en route to a protest last year, recently releasing Twitter bots to squash anti-government activists’ online activity. South Korea seems to be borrowing from this playbook, recently censoring LGBT apps as part of a larger censorship campaign, blocking or deleting almost 100,000 sites in 2013 alone. As these examples illustrate, too much government intervention in authoritarian and democratic regimes alike can lead to extreme infringements on civil liberties.

Just Right?

Complete information freedom is not the panacea many imagined, thanks to malicious actors and profiteers who benefit from the release of private information.  At the same time, the rise of censorship, generally a tool of governments hoping to control the flow of information to their citizens, is a serious concern. Is there a middle ground that can support the freedom of information that promotes development and democracy, while also protecting privacy?

Attaining this middle ground will require creativity and innovation both from the security community and the legal system. For instance, there was some discussion that the Intimate Privacy Protection Act would be introduced this summer, but it continues to be stalled while other countries have successfully passed legislation focusing on protecting individual privacy. The UK passed and has sentenced perpetrators under a new law criminalizing revenge porn. The European Court of Justice ruled in favor of the “right to be forgotten” and has increasingly required compliance by the major search engines.  Most of these laws focus on non-complicit posting of private data, and reveal legislation that addresses the growing concerns over privacy protection. Nevertheless, as we’ve seen with the Ashley Madison hack, because the legal system lags behind technological change, prosecutors may seek more creative solutions to protect personal privacy.  For example, the Canadian government may rely upon recent cyber-bullying legislation to prosecute perpetrators of this behavior.

These examples demonstrate the emerging trend of patchwork legislation focused on protecting private data. However, this kind of legislation often encounters opposition from free-speech groups, many of which worry about a slippery slope toward censorship, as well as concerns about third-party sites being legally liable.  If nothing else, the hacks ranging from OPM to Ashley Madison prove just how insecure private information can be on the Internet. Until the legal system catches up to the pace of technology, there will continue to be a greater need for security solutions. With the watershed hacks of the past year, there will be a greater demand to more actively defend against the range of adversaries and better protect not only personal information but intellectual property as well. It’s unfortunate that we’re increasingly talking about the millions of people affected. Groups and individuals on both sides of this argument must get more comfortable with a middle ground approach that integrates minimal government intervention to protect personal privacy while safeguarding information freedom. An incremental approach based on information sharing legislation is not enough. We should move towards greater protection of privacy, including the ability to prosecute for the theft and non-consensual disclosure of digital data. But until the legal system catches up, technical security solutions remain the main (yet imperfect) safeguard for information protection. This combination of more active security as well as modernized legislation is exactly what is needed to tip the balance back in favor of citizens and privacy protection.

Read more blog posts by Andrea Little Limbago.

Follow Andrea on Twitter @limbagoa

See how we automate the hunt for cyber adversaries.

Stop by Booth #1329 to:

See a Demo of Endgame Products

Sign up here for a private demo to learn how we detect attacks that:

• Use native tools to locate, stage, and exfiltrate customer data
• Exploit application vulnerabilities to install unknown malware
• Install backdoors to gain control of critical servers
   

Join us at 1923 Bourbon Bar!

Join Endgame for an evening of bourbon, cigar rolling, and jazz at 1923 Bourbon Bar on Wednesday, October 7. Registration is required to attend.Learn more and register here.

As demonstrated in an earlier post, establishing relationships (between files, executable behaviors, or network packets, for example) is a key objective of researchers when automating the hunt.  But, the scale of information security data can present a challenge if naïvely measuring pairwise similarity.  Let’s take a look at two prominent methods used in information security to estimate Jaccard similarity at scale, and compare their strengths and weaknesses.  Everyone loves a good head-to-head matchup, right?

Jaccard distance is a metric¹ that measures the similarity of two sets, A and B, by

where J_s denotes the Jaccard similarity, bounded on the interval [0,1].  Jaccard similarity has proven useful in applications such as malware nearest-neighbor search, clustering, and code reuse detection.  In such cases, each set might contain imported functions, byte or mnemonic n-grams, or behavioral properties observed in dynamic analysis of each file.

Since each datapoint (e.g., malware sample) often consists of many feature sets (e.g., imports, exports, strings, etc.) and each set can itself contain many elements, naïve computation of Jaccard similarity can be computationally expensive.  Instead, it’s customary to leverage efficient descriptions of the sets A and B together with a fast comparison mechanism to compute J_d(A,B) or J_s(A,B). Minwise Hashing (MinHash) and bitwise set hashing are two methods to estimate Jaccard similarity.  Bitwise set hashing will be referred to in this blog post as BitShred since it is used as the core similarity estimator in the BitShred system proposed for large-scale malware triage and similarity detection.

First, let’s review some preliminaries.  (Or, jump to experiments or summary.)  First, key ideas behind MinHash and BitShred will be reviewed, with a few observations about each estimator.  Then, these two methods will be compared experimentally on supervised and unsupervised machine learning tasks in information security.

MinHash

MinHash approximates a set with a random sampling (with replacement) of its elements.  A hash function h(a) is used to map any element a from set A to a distinct integer, which mimics (but, with consistency) a draw from a uniform distribution.  For any two sets A and B, Jaccard similarity can be expressed in terms of the probability of hash collisions:

where the min operator acts as the random sampling mechanism.  Approximating the probability by a single MinHash comparison of A and B is actually an unbiased estimator, but has quite large variance—the value is either identically 1 or 0.  To reduce the variance, MinHash averages over m trials to produce an unbiased estimator with variance O(1/m).

Estimating Jaccard similarity via MinHash is particularly efficient if one approximates h(a) using only its least significant bit (LSB).  This of course, introduces collisions between distinct elements since the LSB of h(a) is 1 with 0.5 probability—but the approximation has been shown to be effective if one uses many bits in the code.  Overloading notation a bit, let a (respectively, b) be the bit string of m 1-bit MinHashes for set A (respectively, B). Then Jaccard similarity can be approximated via a CPU-efficient Hamming distance computation (xor and popcount instructions):

It has been shown that the variance of 1-bit MinHash is 2(1-J_s)/m when using m total bits, and indeed the variance of any summary-based Jaccard estimator has variance O(1/m).  Interestingly, the variance of b-bit MinHash does not decrease if one uses more than b=1 bits to describe each hash output h(a) while retaining the same number of bits in the overall description.  With a little arithmetic, one can see that to achieve an estimation error of at most ε J_s with probability exceeding 1/2, one requires m > (1-J_s)/ (ε J_s)² bits of 1-bit Minhash, by Chebyshev’s inequality.

Code (golang) to generate a 1-bit MinHash code and approximate Jaccard similarity from two codes is shown below.

func Hash64(s string, seed uint64) uint64func PopCountUint64(x uint64) intfunc OneBitMinHash(set []string, N_BITS int) []uint64 {
  code := make([]uint64, N_BITS/64)var minhash_value uint64for bitnum := 0; bitnum < N_BITS; bitnum++ {
    minhash_value = math.MaxUint64for _, s := range set {
      minhash_tmp := Hash64(s, uint64(bitnum)) // bitnum as seed
      if minhash_tmp < minhash_value {
        minhash_value = minhash_tmp
      }
    }
    whichword := bitnum / 64   // which uint64 in the slice?
    whichbit := bitnum % 64    // which bit in the uint64?
    if minhash_value&0x1> 0 { // is the bit set?
      code[whichword] = code[whichword] | (1<< uint8(whichbit))
    }
  }return code
}func JaccardSim_OneBitMinHash(codeA []uint64, codeB []uint64) float64 {var hamming int
  N_BITS := len(codeA) * 64for i, a := range codeA {
    hamming += PopCountUint64(a ^ codeB[i])
  }return1.0 - 2.0*float64(hamming)/float64(N_BITS)
}

BitShred: Bitwise Set Hashing

Feature hashing is a space-efficient method to encode feature-value pairs as a sparse vector.  This is useful when the number of features is a priori unknown or when otherwise constructing a feature vector on the fly.  To create an m­-dimensional vector from an arbitrary number of feature/value pairs, one simply applies a hash function and modulo operator for each feature name to retrieve a column index, then updates that column in the vector with the provided value.   Column collisions are a natural consequence in the typical use case where the size of the features space n is much larger than m.

BitShred uses an adaptation of feature hashing in which elements of a set are encoded as a single bit in a bit string.  Since m<<n, a many-to-one mapping between set elements and bit locations introduces collisions.  A concise bit description of set A is created by setting the bit at [h(a) mod m] for all elements a in A.  Overloading notation again, let a (respectively, b) be the BitShred description of set A (respectively, B).  Then Jaccard similarity is estimated efficiently by replacing set operators with bitwise operators:

To make sense of this estimator, let random variable C_i denote the event that one or more elements from each set A and B both map to the ith bit.  Similarly, let random variable U­_i denote that one or more elements from either set A or B (or both) map to the ith bit.  Then, the BitShred similarity estimator J_s can be analyzed by considering the ratio

which is simply the (noisy, with collisions) sum of the intersections divided by the sum of the union.  Estimating the bias of the ratio of random variables will not be detailed here.   But, note that due to the many-to-one mapping, the numerator generally overestimates the true cardinality of the set intersection, while the numerator underestimates the true cardinality of the set union.  So, without cranking laboriously through any math, it’s easy to see from the ratio of “too big” to “too small” that this estimator is biased², and generally overestimates the true Jaccard similarity.

Code (golang) to generate a BitShred code and estimate Jaccard similarity from two BitShred codes is shown below.

func Hash64(s string, seed uint64) uint64func PopCountUint64(x uint64) intfunc BitShred(set []string, N_BITS uint16) []uint64 {
  code := make([]uint64, N_BITS/64)for _, s := range set {
    bitnum := Hash64(s, 0) % uint64(N_BITS)
    whichword := bitnum / 64  // which uint64 in the slice?
    whichbit := bitnum % 64   // which bit in the uint64?
    code[whichword] = code[whichword] | (1<< uint8(whichbit))
  }return code
}func JaccardSim_BitShred(codeA []uint64, codeB []uint64) float64 {var numerator, denominator intfor i, a := range codeA {
    numerator += PopCountUint64(a & codeB[i])
    denominator += PopCountUint64(a | codeB[i])
  }returnfloat64(numerator) / float64(denominator)
}

Estimator Quality

The math is over; let’s look at some plots.

This plot shows the estimated vs. true Jaccard similarity for MinHash and BitShred, for the contrived case where sets A and B consist of randomly generated alphanumeric strings, |A|=|B|=64, and the number of bits m=128.  The mean and 1 standard deviation error bars are plotted from 250 trials for each point on the similarity graph.  The y=x identity line (dotted) is also plotted for reference.

A few things are evident. As expected, MinHash shows its unbiasedness with modest variance.  BitShred is grossly biased, but has low variance.   Note however, that the variance of both estimators vanishes as similarity approaches unity.  In many applications such as approximate nearest-neighbor search, it’s the consistent rank-order of similarities that matter, rather than the actual similarity values.  In this regard, one is concerned about the variance and strict monotonicity of this kind of curve only on the right-hand side, where J_s  approaches 1.  The extent to which the bias and variance near J_s=1 play a role in applications will be explored next.

Nearest Neighbor Search

So, what about nearest-neighbor search?  Let’s compare k-NN recall.

As a function of neighborhood size k, we measure the recall of true nearest neighbors, that is, what fraction of the true k neighbors did we capture in our ­k­-NN query?  The plot above shows recall vs. k averaged over 250 trials with one standard deviation error bars for MinHash vs. BitShred.  The same contrived case is used as before, in which sets A and B consist of randomly generated alphanumeric strings, |A|=|B|=64, and the number of bits m=128.  While it’s mostly a wash for small k, one observes that the lower-variance BitShred estimator general provides better recall.

Note that in this toy dataset, the neighborhood size increases linearly with similarity; but in real datasets the monotonic relationship is far from linear.  For example, the first 3 nearest neighbors may enjoy Jaccard similarity greater than 0.9, while the 4^th neighbor may be very dissimilar (e.g., Jaccard similarity < 0.5).

Applications: Malware Visualization and Classification

Let’s take a look at an application.   In what follows, we form a symmetric nearest neighbor graph of 250 samples from each of five commodity malware families plus a benign set, with k=5 nearest neighbors retrieved via Jaccard similarity (MinHash or BitShred).  For each sample, codes are generated by concatenating five 128-bit codes (640 bits per sample) consisting of a single 128-bit for each of the following feature sets derived from PE files:

• PE file section names;
• language resources (English, simplified Chinese, etc.);
• statically-declared imports;
• runtime modification to the hosts file (Cuckoo sandbox); and
• IP addresses used at runtime (Cuckoo sandbox).

A t-SNE plot of the data—which aims to respect local similarity—for MinHash and BitShred are shown below. (I use the same random initialization for both plots.)

Figure 1: MinHash similarity from k=5 symmetric similarity matrix

Figure 2: BitShred similarity from k=5 symmetric similarity matrix

The effects of BitShred’s positive bias can be observed when comparing to the MinHash plot.  It’s evident that BitShred is merging clusters that are distinct in the MinHash plot.  This turns out to be good for Allaple, but very bad for Ramnit, Sality and benign, which exhibit cleaner separation in the MinHash plot.  Very small, tight clusters of Soltern and Vflooder appear to be purer in the BitShred visualization. Embeddings produced from graphs with higher connectivity (e.g., k=50) show qualitatively similar findings.

For a quantitative comparison, we show results for simple k-NN classification with k=5 neighbors, and measure classification performance.  For MinHash the confusion matrix and a classification summary are:

In this contrived experiment, the numbers agree with our intuition derived from the visualization: BitShred confuses Ramnit, Sality and benign, but shows marginal improvements for Soltern and Vflooder.

Summary

MinHash and BitShred are two useful methods to approximate Jaccard similarity between sets with low memory and computational footprints.  MinHash is unbiased, while BitShred has lower variance with nonnegative bias.  In non-extensive experiments, we verified intuition that BitShred overestimates Jaccard similarity, which can introduce errors for visual clustering and nearest-neighbor classification.  In our contrived experiments (which also plays out in practice), this caused confusion/merging of distinct malware families.

The bias issue of BitShred could be partially ameliorated by using neighbors that fall within a ball of small radius r, where the BitShred bias is small.  (This is in contrast to k-NN approaches in which similarities in the “local” neighborhood can range from 0 to 1, with associated bias/variance.) 

Finally, the Jaccard metric represents a useful measure of similarity.  There are many others based on common or custom similarity measures, which may also be approximated by Hamming distance on compact binary codes.   These, together with efficient search strategies (also not detailed in this blog post) can be employed for powerful large-scale classification, clustering and visualization.

¹How can one show that Jaccard  distance is really a metric?  Nonnegativity, coincidence axiom, and symmetric properties? Check, check and check.  But, triangle inequality?  Tricky!  Alternatively, one can start with a known metric—the symmetric set difference between A and B—then rely on the Stenhaus Transform, to crank through the necessary arithmetic and arrive at Jaccard distance.

²One may reduce the bias of BitShred by employing similar tricks to those used in feature hashing. For example, a second hash function may be employed to determine whether to xor the current bit with a 1 or a 0. This reduces bias at the expense of variance.  For brevity, I do not include this approach in comparisons.

Read more blog posts about Data Science.

Follow Hyrum on Twitter @drhyrum.

As adversaries - whether criminal or otherwise - make use of increasingly sophisticated attack methods, network defenses have not kept pace; they remain focused on signature-based, reactive measures that close the barn door after the horses have escaped. Automated threat detection offers the opportunity for truly proactive network defense, by reducing the amount of time an intruder remains undetected and introducing remedies earlier than otherwise possible. Automation can also enable better use of scarce resources and reduced exposure to network-based threats. This webcast discusses how to automate the hunt for network threats and move an organization's security posture to the next level.

Sign up for this SANS webcast and be among the first to receive an advance copy of a SANS whitepaper discussing the automation of threat detection. Register here.

Last week’s understanding reached between Chinese President Xi Jinping and US President Barack Obama highlighted the attempt to mitigate the growing tension between the countries over espionage. In response, a series of commentaries applauded the agreement for its deterrent effect, and view it as a sign of détente or simply a good first step. This agreement, coupled with Xi’s meeting with top US CEOs, has been interpreted as growing collaboration in both the public and private domains. In contrast, as yesterday’s Senate hearing exemplified, many in and out of the national security view it as a hollow agreement that will not alter Chinese behavior in the cyber domain. Below are three key areas that, when analyzed, illustrate the need to maintain a healthy dose of skepticism when it comes to Sino-American relations in the cyber domain.

An Inflated Threat

Many contend that the Chinese threat to US interests in the cyber domain is inflated because there has yet to be physical destruction as a result of malicious digital activity, or because China has yet to convert the stolen information to their advantage. These arguments often rely on the nebulous term cyber war, which is the wrong gauge of the threat to US national interests. The absence of war does not imply peace. In contrast, conflict in the cyber domain is very similar (although dramatically different by the three Vs: velocity, volume, and variety) to economic conflict of the mercantilist era, where economic warfare was an extension of politics and part of the escalatory path to military conflict. For instance, it’s quite unlikely that Lockheed Martin or Dupont (among numerous private and public organizations) would agree that the Chinese threat is inflated. Similarly, while there has not been physical destruction, intrusions into critical infrastructure already exist and could lead to sabotage during times of heightened tensions. Similarly, the aggregation of health records, background checks, and travel records, to name a few, together provide a vast network view of US citizens that can be used for recruitment, blackmail, and exploitation of vulnerabilities. Just because the full extent of the possible has not occurred, it does not imply that the preparation of the operating environment is not well underway.  

The Tech Community Embraces China

From Cloudflare’s venture with Baidu to Microsoft’s partnerships with politically connected Chinese companies to Google’s latest partnership with Huawei to make the Nexus 6P, one might believe the tech community is openly embracing the world’s largest market. However, the growing concerns of US companies over IP theft and increased restrictions on doing business in China, have led to relations that are increasingly deteriorating. Last week’s forum in Seattle organized by Xi to bring together Chinese tech CEOs with their US counterparts illustrates these growing tensions as well as challenges with doing business in China. For instance, there were notable absentees on the invite list, which is apparent in the forum’s class picture, which lacks Google, Twitter and Uber CEOs. Moreover, this forum normally does not require CEO level attendance, but China threatened regulatory scrutiny that would negatively impact the organizations if the invited companies did not send CEO level representatives, rendering this a mandatory forum if the companies did not want to potentially encounter retaliation. Furthermore, this summer’s announcement that China will be inserting cybersecurity police into tech companies is indicative of their ongoing push for greater control of the internet, which runs counter to the internet freedoms and global norms promoted by the US government and tech companies alike. The tech community increasingly is coming to grips with the tradeoff of access to the largest market with the acknowledgement that the Chinese government could exploit their technologies as part of its ongoing censorship campaign. In addition, China’s crackdown on VPN access, and use of US partnerships to build domestic competitors is evidence of the Chinese strategy to replace all foreign technologies with domestic counterparts by 2020. This is hardly the warm embrace corporations seek.

Deterrence & Credible Commitments

The notion that last week’s agreement could be a deterrent fails to comprehend that deterrence depends on credible commitments, which are strongly lacking in the Sino-American relationship. Xi’s stance that China does not steal IP or PII from the US, despite the ever-growing list of intrusions, sparks little confidence when it comes to his ability to commit to the agreement. Those in the national security community as well as tech community have a hard time taking him at his word.  This skepticism is expounded when noting that the agreement was negotiated while under the threat of sanctions. Leaders are self-interested actors, and Xi was able to shape the agreement to stall (temporarily?) sanctions while enabling him to maintain his stance that China does not conduct cyber espionage. Finally, the agreement not only lacks any compliance mechanism, but it also fails to address the theft of PII and is nebulous in so many areas that the Chinese government can easily continue to lean on proxy actors in and outside of government to feign ignorance regarding any upcoming identification of an intrusion. Clearly, this is not what is meant when discussing deterrence, as there has been little to no impact on the decision calculus of the Chinese, which is at the core of successful deterrence.

Discussion of détente is as ridiculous as comparing Chinese open economic policies to Glasnost, or their anti-corruption campaign to Perestroika. Obviously, it’s important diplomatically to seek to prevent the growing intrusions, but it’s naïve to believe this might be the first step at achieving a deterrent effect. As yesterday’s Senate Armed Services Committee hearing demonstrated, there is little faith in the agreement, and therefore it likely will soon be forgotten as soon as the next major breach is revealed. In that regard, the aspect of the past week that may have the longest media cycle is not so much the idea of a plausible détente, but rather the attire of Silicon Valley’s CEOs, who stunned the Twitter-sphere by proving they do in fact own suits.

While it’s unlikely that Shakespeare had patching in mind when he penned “to be or not to be”, I started thinking about this seemingly simple question the other day when I heard about a recent Microsoft out-of-cycle patch (which means that Microsoft pushed out a critical patch outside of its regular “patch Tuesday”). Patching is always a good idea, but not all patches are created equal - especially if they are received via email.  Those are most likely always malicious.  Since some readers of this blog may not have experienced this first-hand, I’d like to share an example of a malicious campaign with you and explain how a link to a malicious binary, spread via email under the guise of a Microsoft update, can have a catastrophic effect.

The malware link shown below in Figure 1 was spread via an email phishing campaign that purportedly had a cyber spin to it.  In the body, the email notified the recipient that an urgent update was necessary and included a hyperlink to a nefarious link, and the link still hosted the malware so it was easy to obtain for analytical purposes.  It’s important to note that while I’m not at liberty to disclose the actual email, the malicious link was reported by urlquery.net, an online service for testing and analyzing URLs for malicious content, making an excellent resource for researchers (portions of the link have been redacted for security reasons).

Figure 1: urlquery.net displaying a malicious link to malware

After obtaining the malicious ZIP file, closer inspection revealed the compressed archive file contained an executable binary named ‘Mse-Security.Update.exe’ (binary icon included below)—and this is where our story begins.

Upon execution of the binary, the unknowing victim wouldn’t notice anything unusual.  However, underneath the operating system’s hood, the story is quite different.  The user probably wouldn’t have been the wiser that several artifacts were dropped on their system to include a persistent executable binary (.exe) along with two dynamic link library (.dll) files on the victim host.  Nor would the user have known it would capture keystrokes, web activity and document all running applications.  Not only that, the user wouldn’t have been aware that the malware was also grabbing screen captures of their desktop, then disseminating all this collected data back to its Command and Control (CnC) server via FTP.  And this is only the beginning.  The real fun comes when we start dissecting this in greater detail.

‘Mse-Security.Update.exe’ is a dropper that drops four files contained within a newly created directory named ‘LMCAEC’.  This directory is created with System Hidden attributes and it resides in the application data directory for all users.  Here is how it appears on Windows XP:

            Figure 2: LMCAEC Directory Tree

The binary PLG.exe is the persistent implant which has a registry RUN key to insure it runs at startup.  It also has a unique icon. Some will recognize this as the Ardamax Keylogger (see below).

Upon execution, it captures, encodes, then writes four different types of stolen data.  Each of these data types gets stored in another directory named IGW.  This too resides within the same all users path.  The contents of these files are continually written to.  Figure 3, below, shows an example of the directory as well as the type of data stored within each file:

Figure 3: IGW Directory Tree

There’s also some network activity, but we’ll look at that in a moment.  First, let’s take a look at the stolen data files.  At first glance PLG.001, PLG.002 and PLG.004 look similar, but obfuscated.  After closer inspection, however, a few things jump out.

The first thing I noticed was the appearance of an every other byte pattern consisting of the same two-bytes (see red highlights in Figure 4).  These turned out to be extra bytes thrown in probably for obfuscation.   These extra bytes begin appearing regularly at offset 0x12, but they also appear in the first dword (or 4-bytes) of the file (also highlighted in red). 

Second was a 2-dword (or 8 byte) separator, or delimiter (see green highlights in Figure 4).  The first dword of the separator consist of null bytes, while the first byte of the second dword contains the length (in hex) of the data segment to follow (i.e. 0x8a = 138; 0x 6A = 106; 0x 4A = 74).  These data segments are the encoded stolen data.  Additionally, with the exception of the first segment, each concluded with a dword bitmask (see black highlights).

More importantly, however, was the encoding key.  The stolen data is encoded with a 2-byte xor key found interspersed with the extra bytes within the first dword of the file (see blue highlights in Figure 4).  Once the extra bytes are removed, this 2-byte key can be applied to decode the data.  I’ll expand upon this in more detail shortly.   

Figure 4: PLG.001 Encoding Schemes (with decoded data)

Scripting a decoder gives you a quick peek at the stolen data from a command line, as shown in Figure 5.

Figure 5: PLG.001 output from a python decoder

Interestingly, analysis of PLG.004 revealed it followed the exact obfuscation scheme of PLG.001, whereas the extra bytes and encoding key were flip-flopped within the first dword of PLG.002.  Figure 6 illustrates this by highlighting their respective ‘xor keys’ in blue and their ‘extra bytes’ in red.  Notice too that these values are different for each file – they are variables created on the fly during run time.  Another interesting piece to these files can be found at offsets 0x11 through 0x21.  Every other byte decodes to ‘Wonderful’ (see highlighted green below—did you spot that previously in Figure 5?).

Figure 6: File Comparison (key in blue, extra bytes in red)

After examining the three files above, I took a quick peek at the config file ‘PLG.00’ in the LMCAEC directory.  It too began with the same encoding scheme described above, but it followed the path of PLG.002 in that the first dword was:  extra byte : key : extra byte : key (see Figure 7).  Moreover, once the file is decoded, two interesting strings appear between offsets 0x121 and 0x14B: ‘a5XXXX64’ and ‘metXXXXXXt85’ (portions intentionally redacted with Xs).  These strings can be seen below in the pop up box in Figure 7, but we’ll come back to them momentarily.

Figure 7: PLG.00 Decoded

First, let’s get back to the files in the IGW directory for a moment.  These eventually have the ‘PLG’ in their name replaced with a date time stamp.  For example, PLG.001 becomes something like 2015-04-08_13-01-23.001.  These files are still encoded as described above; however, within milliseconds of their respective name changes, they are decoded and the contents are added to an html page for exfiltration.  The html page has the same basic naming convention, but the filename is prepended with a ‘flag’ indicating the type of data contained within the html page.  These flags are: App, Keys, Web and Screen.  This means our file ‘2015-04-08_13-01-23.001’ becomes ‘Keys_2015-04-08_13-01-23.html’.  Figure 8 demonstrates their respective naming conventions before and after.

Figure 8: Stolen Data Files (naming convention: before and after)

The html files are then exfiltrated via FTP to the CnC server ‘aymail[.]site11[.]com’ logging in with the credentials: username ‘a5XXXX64’ and password ‘metXXXXXXt85’ (remember those from our PLG.00 file?).  This login can be seen in Figure 9.

Figure 9: FTP Login Session

Once these are pushed to the CnC server, both versions of these files are deleted from the system.  Interestingly, the html files are in the clear as can be seen in Figures 10 through 13, detailing examples of the exfiltrated data.  This data loss could be quite damaging depending on the unsuspecting user’s activity.

Figure 10: Keystrokes by the Victim User

Figure 11: Websites Visited by the Victim User

Figure 12: Applications Used by the Victim User

Figure 13: Exfiltrated Screen Capture of the Victim User’s Desktop

And there you have it – a day in the life of a malicious update; one that updated nothing except the attacker’s stolen data repository.  Before signing off though, I’d like to leave you with a chronological snippet of the malware during runtime.  Table 1 details the dropper and implant along with their respective operations and results.  The chronology is followed by the file identifying hashes of the malware discussed within this post.  Until next time--patch smartly!

Table 1: Chronological Gist of Malware During Runtime

File Identifiers:

If this sort of analysis interests you, check out our Senior Malware Research Scientist position. We are always looking for great malware research talent to join the Endgame team!

https://www.endgame.com/careers/senior-malware-research-scientist-san-francisco 

A few years ago Jeff Hammerbacher famously claimed  that, “The best minds of my generation are thinking about how to make people click ads.” This seems to have only marginally changed with teams of data scientists in Silicon Valley often devoted to discovering solutions that yield indiscernible improvements within a broader range of recommender engines. In large part, data science within the tech community remains focused on e-commerce and the sharing economy – which largely are at the point of diminishing returns from a customer’s perspective –instead of disrupting industries such as healthcare, education or security. This general lack of integration of data science innovations into products in other realms is anecdotally reinforced at the various data science focused conferences, which overwhelmingly present the incremental changes to driving times, deliveries, or more targeted shopping experiences. Areas awash with data at scale – such as security – rarely even garner a blip on the radar at data science-focused tech conferences.

The failure of data science to extend significantly into products in new industries may be a major contributing factor when looking at data science within the 2015 Gartner Hype Cycle for Emerging Technologies. The 2015 Hype Cycle divides the various approaches within data science, placing each of them just before or after the peak of inflated expectations, including machine learning and NLP. Interestingly, digital security remains in the innovation trigger phase, highlighting the great opportunities that exist in the security space.

Below is a quick synopsis of some observations from a range of data science and technology focused conferences I’ve attended on both coasts this year. In short, Hammerbacher’s admonitions are as relevant today as they were a few years ago. However, this does not need to be the case, with great opportunities for data science to disrupt the security industry.

Current State of Data Science

• Much of the Same: Targeted marketing continues to prevail, with emphasis on fine-tuning the already refined and complex algorithms for better shopping experiences and search results within sites.
• Diminishing Returns: Large teams are focused on incremental improvements to the user experience, creating an ever bigger void between what users understand is being done with their personal data and the reality. Much of this also focuses on social media mining for marketing and e-commerce purposes.
• Black Box Approach: Hailed by the Harvard Business Review as the sexiest job of the twenty-first century, there are signs that many believe current work by data scientists will soon be automated or simply is not the silver bullet as it has been portrayed in marketing and media materials. The prevalent mentality belittles domain expertise of the data and/or data science techniques in favor of a black box approach. This impacts the frequency and kind of data collected, what questions can be addressed with the data, or even the theoretical validity of the multitude of correlations that are bound to occur given a large-scale data environment.
• Chasing Fads: The majority of data science research and development focuses on edge-cases to solve niche problems, instead of the majority of the problems that would have the biggest disruption across an industry.  While the technology may be novel and groundbreaking, it actually may provide little utility for a product. Theoretically interesting breakthroughs that fail to be relevant for a product remain stove piped in the Ivory (or Silicon) Tower.
• General misperception of data science: The less technical conferences with sections on data science or big data generally exhibit lengthy Q&A sessions, which exhibit the ongoing struggles of those outside of the field to comprehend how data science might be applied within their company or industry. In many cases, companies have hired data scientists but aren’t really sure what to do with them. The media portrayal does not help in this regard, arguing that BI tools can serve as nextgen data science.

Data Science’s Next Disruption:  Security

The Gartner Hype Cycle for Emerging Technologies’ bleak outlook for data science highlights the necessity for data science to expand into products in industries beyond the e-commerce, sharing economy, and marketing realms. These markets have greatly benefited from machine learning and other data science techniques, but could very well be at the point of diminishing returns. In contrast, the security community – which is ultimately a key player in both the protection of individual privacy as well as economic and national security – greatly underachieves in integration of vetted and advanced data science techniques into commercial software. The vast majority of security products are based on rules and signatures, which are tenuous and fail to scale or generalize to current environments. While there is arguably a growing emphasis on quantitative approaches to security research, these remain one-off services, with very few actually making their way into products that could truly disrupt an industry that remains focused on Cold War, perimeter based mindsets.

There are great opportunities for data science to play a critical role in the next generation of security research and product instantiation. There is untapped potential for the application of anything from machine learning to natural language processing to dynamic, Bayesian approaches that can be automatically updated with prior and additional knowledge. Similarly, the socio-technical interplay is another under-explored area. For instance, the time series econometric models could help inform repeatable and scalable risk assessment frameworks. Finally, there is the unfortunate perception that security related work is orthogonal to individual privacy. In fact, data science algorithms should help inform the next wave of privacy features – ranging from encryption to fraud detection to preventing the extraction of personally identifiable information by malicious actors.

Join Us at the Data Mining in Cybersecurity Meetup

Data science within security is admittedly difficult, with low tolerance for errors and few open datasets for training and testing. These challenges, however, make the work that much more rewarding and impactful. Endgame’s data science and research and development teams are increasingly pursuing many of the established and bleeding edge techniques in data science across a wide range of data feeds. If you’d like to meet some of the team and hear more about our research, we’ll be hosting the Datamining in Cybersecurity Meetup in San Francisco on November 12^th.

Today Adobe released a patch for CVE-2015-7663[1] that addresses a vulnerability we discovered in Flash Player.

The vulnerability exists due to the improper tracking of freed allocations associated with a “Renderer” object when handling multiple progress bar additions. This can be forced to overflow a Bitmap object corrupting adjacent memory. As we will discuss later, we originally exploited this bug in the lab using the common Vector length corruption target.

In this post I wanted to focus on mitigating the exploitation of Flash Player, and the challenges associated with it, instead of the traditional look at this particular vulnerability in detail.

But first, a little insight into why we see Flash in APT campaigns and exploit markets. From the attacker's perspective, Flash is an amazing access capability.

1.     It’s cross platform

2.     It’s cross browser

3.     It can be embedded in other documents and formats

4.     It has a very rich programming language available

5.     It’s easy to fuzz

6.     There is so much code, vulnerabilities are sure to shake out

Because of this, attackers know that a good Flash exploit can give them reliable access to Windows, Linux, OS X, and Android systems through Chrome, Firefox, IE, Reader, Office, and more! For these reasons Flash exploitation is valuable, and will continue to be so. One capability could easily cover a large majority of all Desktop targets. 

Adobe Vector length corruption technique

The Vector corruption technique we used to exploit CVE-2015-7663 has been publicly known since at least 2013[2]. It is a classic exploitation concept that provides a few “nice to haves” as an exploit writer. 

1.     Corruption gives the attacker read/write of virtual memory

2.     You can allocate arbitrary sizes

3.     It is resistant to corruption and application crash

4.     There is no validation or protection of its contents

Due to the popularity of this technique over the past year, we have seen a rise[3] in Flash exploits[4] using it.

Vector Isolation

In response to the widespread use of this technique, Adobe has strengthened the security posture of Flash by adding two defenses to help reduce the effectiveness of zero day exploits[5].

Previously, Flash used a single heap for allocating all ActionScript objects. Doing this allowed an attacker to target a Vector objects length property when overflowing an adjacent buffer, by coercing the allocation algorithm to position different objects consecutively. Doing this after corrupting memory gave attackers read and write access to virtual memory, making it simple to bypass ASLR and execute code. Here is a graphic showing this memory layout.

However, starting with Flash Player version 18.0.0.209[6], Adobe has made this more difficult.

Now, Flash Player allocates Vector objects in the default runtime heap, instead of the heap associated with ActionScript interpretation. This effectively removes the ability to coerce the allocator into creating adjacent blocks of memory for an attacker to use when corrupting the “length” property. The memory layout before now looks more like this.

This concept of moving specific allocations into separate heaps is called “Heap Isolation”.

The idea of isolating heap allocations for security purposes is not new. In fact, Microsoft[7], Mozilla[8], and others, have been pursuing the idea for years. This approach disrupts some of the steps in the exploitation process that are typical of almost all exploits leveraging memory corruption vulnerabilities.

1.     Allocate memory linearly in a predictable order of specific sizes

2.     Free a subset of those allocations creating predictable “holes”

3.     Trigger a vulnerability that falls into the predictable locations and corrupts adjacent memory allocated in step 1.

Heap isolation effectively breaks Step 1 and 3 by placing certain objects in isolated sections of memory. For example, only Vector objects can be allocated in Heap A, and only ByteArray objects can be allocated in Heap B. Never together. Thus ensuring an attacker cannot allocate memory of another type adjacent to them making it impossible to control virtual memory enough to corrupt adjacent allocations of interest.

Vector Property Guarding 

Additionally, because moving an allocation is not sufficient in some cases, the Vector object now contains a precomputed value[9], often called a “cookie”, that is checked for consistency before the length property is used. If an attacker corrupts this cookie the Flash application aborts and alerts the user. This breaks #4 in the “nice to haves” we discussed earlier.

Unfortunately, we know from experience that preventing a specific technique, such as Vector length corruption, will not stop attackers[10]. Instead, we see a new technique pop up with the same characteristics I have outlined above. There are likely many more “Vector like” objects available in ActionScript/Flash Player beyond ByteArrays and attackers have already found them.

Endgame Heap Isolation

Beyond vulnerability discovery, Endgame's Vulnerability Research and Prevention team is also focused on mitigating and detecting exploits.  We provide protections to customers against whole classes of attacks, without the need for source. We feel particularly aligned to do this because of our extensive experience discovering and exploiting software vulnerabilities.

One of our first research efforts focused on generic enforcement of heap isolation. Instead of enforcing isolation on specific objects like Vectors, we apply it to every object that fits our criteria. This is particularly well suited for prevention of the previously described techniques, as well as vulnerability classes like Use-After-Frees (UAF).

An attacker exploiting a UAF must reallocate a different object into the freed memory location when an object has been released. This reallocation is what eventually gains the attacker code execution by controlling the function pointers in an object.

Forcing heap isolation ensures the attacker can only reallocate the original object, effectively preventing exploitation. The illustration below helps to visualize this effect.

Before

After

This can be a powerful mitigation against specific bug classes and in our testing it has been proven to prevent a large portion of reported vulnerabilities. But we can do more. 

Endgame Control Flow Integrity

In addition to heap isolation we can also enforce control flow integrity (CFI) policies on an application. Whereas heap isolation can be very effective at preventing successful exploitation, a CFI based approach additionally allows us to detect active exploitation attempts since we are inspecting and validating when control flow –- the path that an application executes -- has changed. In the majority of exploits we have studied, there is a point when the attacker must “hijack” control of the process to begin executing a ROP chain – used to bypass DEP -- or arbitrary code.

To accomplish this, Endgame has adapted and expanded on the idea of utilizing processor functionality to determine the exact moment when this happens. Inspired by a novel approach published by researchers at Fudan University[11, 12], we leverage CPU branch misprediction[13], allowing us to introduce control flow integrity policies without expensive binary modifications to a target application such as hooking or dynamic instrumentation. 

We have extended this technique to work on both Linux and Windows 64-bit operating systems and have used it to detect our exploitation of CVE-2015-7663 as well as others, including CVE-2014-0556[14] and the exploit used in the APT campaign Russian Doll CVE-2015-3043[15].

The following output shows our system catching the exploitation of CVE-2014-0556 on a 64-bit Linux host.

  libpepflashplayer.so ----> libpepflashplayer.so                    libpepflashplayer.so

The FROM_IP in the anomolous branch detection is the point when the exploit has control over execution.

The TO_IP is the beginning of the payload. In this case no ROP is used which would be missed by ROP only detection methods.

The following screenshots shows the full system preventing this exploit in real-time.

This work is exciting, as it has already shown its effectiveness at comprehensively detecting unknown exploits regardless of the specific technique used by observing abnormal program execution indicative of exploitation.

Conclusion

We know from experience that vulnerabilities and exploits will continue to make headlines. With the ubiquity of Flash and its high value, attackers will invent creative ways to exploit bugs. We have already seen how Adobe’s recent mitigations are a great step forward, but are not keeping pace with the attackers' ability to exploit vulnerabilities. We understand it’s an iterative process that eventually poses a significant limitation to attackers, but there is still a long way to go.

Endgame is working hard to defend against advanced attacks on all software by developing cutting edge mitigations that work in tandem with strong vendor protections, affording the end user better defense in depth. Our unique experience allows us to test real exploits against real software, something we find necessary to providing adequate protections.

Look for future posts where we cover additional mitigations and share more vulnerabilities!

References

[1] https://helpx.adobe.com/security/products/flash-player/apsb15-28.html

[2] https://sites.google.com/site/zerodayresearch/smashing_the_heap_with_vector_Li.pdf

[3] http://krebsonsecurity.com/2015/07/third-hacking-team-flash-zero-day-found/

[4] https://www.fireeye.com/blog/threat-research/2015/06/operation-clandestine-wolf-adobe-flash-zero-day.html

[5] http://googleprojectzero.blogspot.com/2015/07/significant-flash-exploit-mitigations_16.html

[6] https://helpx.adobe.com/security/products/flash-player/apsb15-19.html

[7] https://labs.mwrinfosecurity.com/blog/2014/06/20/isolated-heap-friends---object-allocation-hardening-in-web-browsers/

[8] http://robert.ocallahan.org/2010/10/mitigating-dangling-pointer-bugs-using_15.html

[9] http://googleprojectzero.blogspot.com/2015/08/three-bypasses-and-fix-for-one-of.html

[10] http://blog.trendmicro.com/trendlabs-security-intelligence/latest-flash-exploit-used-in-pawn-storm-circumvents-mitigation-techniques/

[11] http://ipads.se.sjtu.edu.cn/_media/publications:perf-apsys.pdf

[12] http://ipads.se.sjtu.edu.cn/_media/publications:cfimon.pdf

[13] https://en.wikipedia.org/wiki/Branch_predictor

[14] http://googleprojectzero.blogspot.com/2014/09/exploiting-cve-2014-0556-in-flash.html

[15] https://www.fireeye.com/blog/threat-research/2015/04/probable_apt28_useo.html

For months, there has been sharp criticism of the secret negotiations surrounding the Trans-Pacific Partnership (TPP), which is on track to becoming the world’s largest trade agreement covering 40% of the global economy. If implemented by all twelve countries involved, this trade agreement would have profound geo-political consequences, largely driven by the exclusion of China from the agreement. As the White House website states, the TPP is a means to rewrite the rules of trade, otherwise “competitors who don’t share our values, like China, will step in to fill that void.” Clearly, in addition to the pursuit of trade openness, this agreement is a major geo-political tool to shape global norms to the US advantage. The geo-political consequences – which may very well play out in the cyber domain – have been all but ignored by the tech community, which has focused almost entirely on the agreement’s privacy implications. This is unfortunate and leads to myopic conversations that ignore the agreement’s larger implications for the tech community, and specifically cybersecurity. While Internet privacy absolutely is a high priority, these arguments are completely misplaced. Instead, the tech community – and especially security – should be very wary of how China may respond to this open pursuit of economic containment. In the least, in the short term it does not bode well for the cooperative agreement made last month between China and the US. To help briefly fill this void, below is a cheat sheet of sorts for those unfamiliar with trade agreements, and the TPPs more probable implications in the cyber domain.

1. Misplaced criticism– Critiques focused solely on the TPP and its potential infringement on digital privacy are simply misplaced. In fact, based on the few aspects of the agreement that address the digital domain, the intent is to protect privacy, not erode it. Moreover, trade agreements have a statistically significant relationship with decreasing government repression, and can support democratic consolidation. If the TPP follows suit of other trade agreements (especially those including some democratic members), it likely will also help support Internet openness, not repression. This includes a component that protects organizations from having to submit source code, thus circumventing a major concern of the tech community when working abroad. Even with those arguments aside, the more appropriate place to target international online privacy concerns would be at the rise of bilateral cyber agreements, not trade agreements.
2. Turn the map around: Any global map of the TPP with the Eastern hemisphere on the left quickly highlights the economic containment of China.  This is reinforced when considering China’s pursuit of shaping global economic norms through the Asian Infrastructure Investment Bank, and more recently pursuit of membership in the European development bank.  Although there has been limited talk about including China in the TPP, China would have to adhere to the rules of privacy protection that are completely orthogonal to its domestic interests and policies around the Great Firewall. Viewing the TPP through the lens of China quickly highlights the likelihood that they are feeling encircled, and may respond accordingly.
3. Stumbling blocs– Depending on how the TPP plays out, it could mirror the discriminatory trade blocs prevalent during the Interwar Era trade that helped lay the foundation for future conflict. While trade agreements increase trade between countries, it can also lead to a contraction of global trade, and certainly augment tensions between member states and those excluded from the agreement. Clearly, the current environment is extremely different from that of the Interwar era, but trade agreements that perpetuate geo-political fault lines at best will not improve relations that are already tenuous.

Given the larger geo-political context surrounding the TPP, those in the tech and security communities would be well advised to spend some time looking at the broader implications of the TPP. In reality, the TPP could be a means to expand the US vision of a free and open internet within signatory states, but that gets lost on those who don’t see the whole picture. This would be especially true if China actually becomes a signatory in the future, much as it did with WTO membership. Until then, China likely perceives the agreement as one to spread US influence in the region, and may be considering a range of retaliatory responses, including their persistent reliance on digital statecraft to achieve political objectives. Instead of a myopic focus on Internet privacy, the security community’s broader concerns should focus on China’s potential retaliation in the digital domain, which impacts national security, economic stability and – given the enormous data breaches of the last year – privacy. 

According to the Chinese state-run Xinhua news, the OPM breach, “turned out to be a criminal case rather than a state-sponsored cyber attack as the U.S. previously suspected.” Yesterday, the Washington Post similarly reported the arrest of Chinese criminal hackers, which has since circulated and been sourced across numerous outlets.

Similar to remarks following the US-Sino cyber agreement from September, many pundits are claiming a sea change in Chinese cyber activity. These perceptions unfortunately ignore centuries of theories and data on how states manage the tight balancing act to appease both international as well as domestic audiences. The need to assuage both international and domestic groups leads states to pursue policies that support their own incentive structure and overarching goal of staying in power. By focusing on this latest news from the Chinese government’s perspective, it’s easier to extract insights on their actions and the plausible gap between what is said in the diplomatic arena and what occurs in the nebulous realm of cyberspace.

Below are four assumptions that – when viewed through a strategic, Chinese perspective – should be met with a solid amount of skepticism as the OPM plot thickens:

1. OPM was not state-sponsored. China has devoted significant capital claiming they are not perpetrators of malicious activity in the cyber domain. By allegedly finding the criminal group behind the OPM hack, China is able to save face internationally and maintain the façade of the pursuit of benign activity in cyberspace. Moreover, by identifying the perpetrators as Chinese criminals, the Chinese government rationalizes away any evidence that may point to China, while distancing any government involvement.

2. China is holding domestic criminals accountable. The Chinese government has a long history of leveraging scapegoats, as is evident in the ongoing crackdown on corruption. Accountability and scapegoating are very different, and confounding the two only leads to erroneous interpretations of activities.

3. Norms are working. Xinhua’s announcement supports the ongoing perception that US-shaped global norms may be impacting Chinese digital activity. Unfortunately, this ignores the difficulty in establishing norms, which generally follow a steep S-curve and take significant time and resources to establish in the international system. Moreover, Chinese overt announcements toward cooperation occurred just as the US was about to impose economic sanctions due to the string of breaches attributed to China, including OPM as well as GitHub, United Airlines, Anthem, and the National Oceanic and Atmospheric Administration, to name a few.  This behavior does not change overnight, nor do norms become embedded quickly enough to alter behavior that significantly. Conversely, self-interest (not so-called cyber norms) dominates states behavior, and will continue to rationalize the gap between diplomatic behavior and covert activity.

4. The source is credible. Finally, a dominant source of information on the arrest of Chinese criminals for OPM is Xinhuanet, run by Xinhua News Agency, the official media outlet for the state government. Like virtually all state-run media platforms in non-democracies, Chinese state-run platforms have a reputation as serving as a propaganda tool of the state. In the 2014 World Press Freedom Index, China ranks 175 out of 180, barely edging out Somalia and Syria in press liberties.

As additional details are disclosed over the next few weeks and months regarding the OPM hack, greater scrutiny of the sources and incentive structures should be explored before making grand assertions of strategic behavioral shifts. Diplomatic maneuvering between states to shape both domestic and international perceptions is an omnipresent characteristic of the international system. It would be wise to remain cognizant of motives and activities before believing the next state-sponsored media report.

By Andrew Morris and Andrea Little Limbago

Throughout the second half of this year, there has been much heated debate about proposed changes to the Wassenaar Arrangement, which seeks to expand export controls on dual-use technologies, including those that pertain to intrusion software. While the intention was good, the first iteration of these changes released this past spring was more likely to hurt those who adhere to the arrangement, while empowering non-participants (e.g. China, Iran, North Korea) with an uneven playing field in their favor.

This week’s proposal by the French government to impose a ban on Tor – the most popular anonymous dark net – is just the latest in a series of myopic policy solutions (e.g. encryption debate) that similarly seems to entail undesirable externalities. In the wake of the Paris terrorist attacks, the government likely feels obliged to expedite policy changes to demonstrate a tough stance against terrorist activity. Unfortunately, banning Tor not only will fail to meet those objectives, but will also disrupt the democratic ideals of a free and open Internet. Below are four of the key problematic issues that arise from this initial policy proposal.

• ​Failure to deter terrorist activity. Malicious actors – including terrorists, criminal networks, and lone wolves –will simply adapt and find another venue for their activity. Tor is just one aspect of a multi-pronged OPSEC strategy pursued by ISIS and other groups. In fact, roadblocks are more likely to force adversaries toward more innovative strategies and activity outside of the law enforcement radar.
• Difficult to enforce.At a technical level, it is not difficult to identify whether a specific computer is communicating directly with the Tor network. However, the difficulty arrives in specifically attributing the actual person behind the keyboard, especially given the ability to bounce connections, obfuscate connections via proxy or VPN, and the presence of multiple users at a computer. In fact, banning Tor eliminates a known source of activity and data, thereby making it arguably much harder to monitor and attribute criminal and terrorist behavior.  Identifying whether a particular individual is using Tor inherently involves monitoring Internet usage, which may require additional legal provisioning. Finally, the simple technical logistics of implementing the ban is much more difficult, which even if enacted, returns to the difficulty of who to charge if a given computer is discovered to be using Tor.
• Decreases civil liberties. While the French proposal is in response to terrorist activities, it is more likely to harm those human rights and civil liberties groups who use Tor to express their perspective, collaborate, and coordinate with journalists. The only people that will stop using Tor in France as a result of the ban are people who were using it for legal purposes. Anyone using Tor for an illicit criminal or terrorist agenda will continue to use Tor. The ban therefore decreases an important outlet for these civil liberties groups while enabling illicit activity to persist.
• A global Splinternet.  Despite the widespread perception that state boundaries are obsolete, they do in fact still matter. The ban on Tor in France would accelerate the trend toward a Balkanized Internet, again undermining the spread of a free and open Internet. Moreover, just because Tor is blocked in France, does not mean that malicious actors can’t access it elsewhere. This is especially pertinent and returns to our first point. Because the attacks in Paris were largely planned in Belgium, if this legislation had been in place in France prior to the terrorist attacks, it is extremely unlikely that they would have been prevented based on what we know now.

The policy debates around Wassenaar, encryption, and now the ban on Tor all reflect the naïve belief that a policy can simply make these capabilities disappear. The genie is out of the bottle and instead of placing bans on these technologies, which will only hinder licit while enabling illicit activity, the policy world needs to dig deep and provide innovative solutions that better align with the realities of the modern world system.

By: Margot Koehler

Today, Gary McGraw of Cigital spoke with our CTO Jamie Butler about Endgame's security research, thinking like an adversary, and his specialization in rootkit development. Head over to Cigital and listen in as Gary and Jamie discuss the attribution problem and his research focusing on how to think like a hacker in an effort to turn their work against them with an offensive security stance.

Each of the last several years has been dubbed the “year of the breach,” or more creatively the “year of the mega-breach.” But instead of continuing this trend and calling 2016 the “year of the uber-mega-breach,” Endgame’s team of engineers, researchers and scientists have pulled together their top predictions for the security industry.  We anticipate a threatscape that will continue to grow in complexity and sophistication.  And while policymakers are yet to acknowledge that cyber innovations like encryption, Tor, and intrusion software will not simply go away through legislation, global enterprises should recognize that the “year of the breach” is the new normal.

Increased Focus on the Cloud
Mark Dufresne, Director Malware Research and Threat Intelligence

Cyber attackers will increasingly interact with cloud services to acquire sensitive data from targets. Through compromising credentials and socially engineering their way into access, attackers will successfully gain control of sensitive data and services hosted by commercial cloud providers.  In addition to data exposure, we may see companies that rely heavily on the cloud significantly impacted by ransom demands for restoration of cloud-hosted assets, potentially with new cases of outright destruction of data or services that are often perceived by users as backed-up and secured by their cloud provider.  As part of their continuing effort to evade detection, adversaries will increasingly use these same cloud-based services for command and control as well as exfiltration in order to blend in with the noise in modern Internet-connected environments. Encryption and the heterogeneity of most environments makes drawing a distinction between legitimate and malicious activity very difficult. Attackers will increasingly take advantage of this heterogeneity, leading some organizations to increase investments in securing and controlling their perimeter.

Targeted Malvertising Campaigns
Casey Gately, Cyber Intel/Malware Analyst

State sponsored actors will continue exploiting the social dimension of breaches, focusing on susceptible human vulnerabilities in diverse ways, such as through targeted spear phishing or more widespread malvertising campaigns. Many of these widespread campaigns will become increasingly targeted given the growing sophistication of attacks. Spear-phishing is a very reliable method for a state-sponsored actor to gain a foothold into a given network. In contrast, malvertising is more of a 'spray and pray' approach - where attackers generally hope that some of the millions of attempts will succeed.

Attackers could also take a more targeted malvertising approach by dispersing a series of weaponized ads for a particular item – such as weight training equipment. When someone conducts a search for “barbell sets” those ads would be auto-delivered to the potential victim. If the ads were targeted to fit the output, mission statement or goal of a specific corporation, the chance of victimizing someone from that company would be greater.

Increase in Mobile App Breaches
Adam Harder, Technical Director of Mobile Strategy

The volume of payments and digital transactions via mobile apps will continue to grow as end-users continue to shift from desktops and the web to mobile platforms.  Walmart is in the process of standing up a complete end-to-end mobile payment system, and 15% of all Starbucks revenue is processed through its mobile app.  Unfortunately, more of these apps will likely fall victim to breaches this year. Consider all the apps installed on your mobile device. How many of these are used to make purchases or view credit/loyalty account balances? Several popular consumer apps - Home Depot, Ebay, Marriott, and Starbucks - have been associated with data breaches in the last 24 months.

Public Perception Shift from Security to Safety
Rich Seymour, Data Scientist

People are slowly coming to realize the lack of implicit security in the systems they trust with their data. Many users operate under the false assumption that security is inherently baked into private services.  This isn't a paradigm shift for folks used to untrusted networks (like the manually switched telephone systems of the pre-rotary era), but people who simply assumed their information was stored, retrieved, and archived securely need to recognize that not only must they trust the physical security of a data center, they must also trust the entire connected graph of systems around it.  

Based on some leading literature from last year, including the work of Nancy Leveson, expect to see safety become the buzzword of 2016. There also could be big things from the Rust community (including intermezzOS and nom) and QubesOS. As such, “safety” will likely be the new information security buzzword.

Malicious Activity Focused on Exploiting PII & Critical Infrastructure
Doug Weyrauch, Senior CNO Software Engineer

With the rise in frequency and severity of data breaches, including those at OPM and several health care companies, cyber criminals and hacktivists are increasingly using PII and other sensitive data for extortion, public shaming, and to abuse access to victims’ health records and insurance.  Unlike credit card information, personal health and background information cannot be canceled or voided.  If health records are co-opted and modified by a malicious actor, it is very difficult for a victim correct misinformation.  And with the US Presidential election heating up this year, it’s likely one or more candidates will suffer a breach that will negatively impact their campaign.

As more stories surface regarding the cyber risks unique to critical infrastructure, such as in the energy sector, terror groups will increasingly target these systems. In 2016, there will likely be at least one major cyber attack impacting critical infrastructure or public utilities. Hopefully this propels critical infrastructure organizations and governments to actually put money and resources behind overhauling the digital security of the entire industry.

Image: Maddie King

Last week, our team predicted the biggest cybersecurity trends in the new year – specifically, that as attacks grow in complexity and sophistication, breaches will be the new normal.

Indicative of the growing importance of cybersecurity to critical infrastructure industries, the financial sector is responding to this new normal, and is investing its resources accordingly. In light of high profile breaches like JP Morgan Chase and the Carbanak campaign, current and anticipated spending on cybersecurity in the financial sector exposes the resources required to counter this new normal. To highlight this, we’ve compared cybersecurity spending of four of the largest banks to the GDP of four small countries to demonstrate the vast resources required to manage current and emerging threats.

With the new year kicking off with a high profile attack on the Ukrainian power grid, it is increasingly evident that the new normal is here to stay. Tackling this dynamic and complex threatscape requires organizations – especially those in the highly targeted critical infrastructure sectors –  to think like the adversary. That’s why we’ve built a solution that intimately understands adversarial techniques and tactics – enabling our customers to go from being the hunted to the hunter and identifying threats at the earliest possible moment before damage and loss can occur.

In the State of the Union address on Tuesday, President Obama highlighted the important contributions of women in science and technology fields. Unfortunately, the tech industry on average has less than 30% women in the workforce, which supersedes the paltry 10-12% of women in cybersecurity, at any position. Endgame understands that today’s complex threatscape requires new thinking and diverse perspectives. As we continue to grow, we keep this in mind, with women comprising almost 42% of our recent hires. We value the contributions of all of our team members and continue to bring a diversity of perspectives to ensure our products and research and development are best prepared to tackle adversaries of today and tomorrow. If you want to see first hand presentations of some of our team members, please visit our RSA booth in San Francisco next month, or attend the Women in Cybersecurity conference in Dallas at the end of March.