It has been an exciting summer in the security community for the Event Query Language (EQL) as we delivered presentations at Circle City Con and Bsides San Antonio. These talks showcased creative ways to hunt for adversaries in your environment with EQL. If you couldn't make either of these events, we are sorry we missed you, but good news: EQL is making its way to Blackhat USA in a joint talk with Red Canary called "Fantastic Red-Team Attacks and How to Find Them". We hope to see you there at the presentation or at Endgame’s booth (#1253) to talk more about behavior based detection with EQL. In advance of Blackhat, we’re excited to announce today’s release of more EQL analytics and tooling to make EQL even more usable and powerful.
Before we review the exciting updates to EQL, let's pause for a quick recap. EQL was built by Endgame to express relationships between events. The language is data source and platform agnostic, can answer stateful questions, and enables the hunter by including data stacking pipes to sift through data. If you have structured data, you can start asking questions now. For more background, please see the following posts: Introducing EQL,EQL For the Masses, and Getting Started with EQL.
With today’s update to 0.7, we added an interactive shell so you can better explore your own data. It will dynamically learn your schema, and includes tab-completion, syntax highlighting, table output, helpful error messages, and the ability to export results from a search to a CSV file.
In addition to the improved EQL experience, we also added over 60 new analytics to our Analytics Library. The goal of these new analytics is to help you enrich your data and gain context by matching high-noise MITRE ATT&CK™ techniques against your security events. Finally, we’ve updated our contribution guides, making it easier to contribute to the EQL community.
Interactive Shell
We’re happy to make a powerful shell available to improve your experience with EQL It is included with EQL, so after you install EQL 0.7 (pip install eql), you can use the shell to quickly explore your own data set. It has syntax highlighting and supports queries that span multiple lines. You can define a schema or teach it to learn your schema in order to validate your queries. We provided a powerful table viewer to allow for readability and flexibility of query results. With the shell, you can tab complete through EQL keywords, EQL functions, and even your own data fields.
Let's show some examples using the interactive shell by running the command eql. First, here’s a quick video.
Figure 1: A quick video showing the EQL Shell. Showing features ranging from importing data, tab completion, type/field/syntax checking, table outputs, and more!
Let’s highlight a couple features. For instance, the tab complete is very useful for quickly building your EQL query or knowing the available fields from your data.
Figure 2: Tab completion of EQL functions and data fields within EQL Shell
Additionally, the table is a powerful utility to quickly customize your EQL match results. During the alert triage process, it can be helpful to quickly make pivot tables and narrow your investigation.
Figure 3: EQL table showing query results.
Check out the documentation for more information and ways to use the EQL shell. We hope you enjoy it as much as we do!
Improved Query Validation
In the Endgame product, we recently gave our users the ability to create their own realtime detections with EQL. As part of this work, we updated EQL to include a system for detecting query errors (this functionality is available in the EQL shell as well). If you try to create a query with an unknown function, event type or field, EQL will tell you exactly where you went wrong, with a helpful message indicating the root problem. We also added a type system to the internals of EQL, and now you'll get more concise and useful errors if you pass the wrong type to a function, or try to compare across types.
Figure 4: Schema and type validation in core EQL. The first example showing an error for the field, cmdline, as it did not exist in our data and the second showing a type error, as the length function expected an array/string.
Enriching with EQL
Security isn’t always about alerting. Contextualizing an alert to further comprehend what’s happening is a great way to help security analysts go from reacting to yet another alert to understanding a potential incident. We can provide better context by enriching logs with ATT&CK™ tactics and techniques. This isn't the same thing as positively identifying something as malicious, but serves to enhance raw data with critical information for performing triage or finding trends in lower confidence detections
Often attackers use native-os tools for their own purposes, tradecraft often known as living off the land. It can become difficult for defenders to separate malicious intent from benign usage of an administrative tool, but regardless of the intent, execution of these tools is noteworthy. By enriching this data when it is seen, we may notice some seemingly benign actions representing more nefarious intent when viewed collectively. Or, defenders may use labels to prioritize and rank event collections that include certain enrichments. Either way, the crux is labeling security data, and we provide a solution with EQL.
We are releasing over 60 analytics to our the EQL Analytics Library. These are categorized as enrichments so you can start labeling your data or even start hunting for suspicious behaviors in your environment. Additionally, they are all mapped to MITRE ATT&CK™! These queries are better enrichment labels than alerting detections. A majority have high potential for noise and could represent legitimate activity, but if you’ve been struggling to enhance your coverage of ATT&CK™ without success, this kind of enrichment can help you get a great deal closer. Going farther, if you understand normal baseline activity in your environment, then you could selectively choose which EQL queries could act as detections for you and use them accordingly.
Figure 5: An example of using eqllib to survey a dataset and return analytic matches. We could use this as a means to prioritize our triage by count.
Endgame provides enrichments to our users. Attack timelines which come back with alerts are enriched with additional ATT&CKTM-based context displayed to the user. This helps our users understand their alerts and take appropriate actions.
Figure 6: Enrichment in the Endgame platform (click image for larger view)
How to Contribute to EQL
And finally, we want to collaborate with you, so we updated our contribution guides (EQL and the Analytics Library). Our intention is to provide a clear path for you to contribute to the EQL Analytics Library or the language directly. We recommend everyone following basic git development workflows by creating branches and submitting pull requests, but we added plenty of useful help docs to assist. We also created templates to make the process as painless as possible.
We pledge to be timely in our review of all submitted work. And of course, always feel free to reach out and communicate with the EQL team via chat on Gitter, on Twitter (@eventquerylang) or with Github issues.
We hope to work with you soon!