Ransomware IR with PowerForensics and the USN Journal

Well it's certainly been a while since I made a post!

I last blogged in February about Malware analysis and you can find that post here. http://www.redblue.team/2016/02/a-soft-introduction-to-malware-analysis.html. My thanks go to Dave and Abdul for keeping content coming while I was slacking. It's hard to believe the year is almost over. It was a pretty busy one for me, I changed roles, changed companies, and moved across the country. I'll use that as my excuse for being too lazy to write anything :)

I also wanted to say thanks to everyone who has viewed this blog. In the last year or so, we've managed to accumulate over 100,000 views! Honestly, the sole intent of starting this site was to document our own study efforts but we've had good feedback and hope to share content more consistently in the new year. This field is changing rapidly, and the scope of knowledge is constantly expanding. Hopefully we can help you feel a little less overwhelmed!

For this post, I wanted to document an experience I had on a recent investigation and share some thoughts about how I tackled this challenge, to hopefully benefit your investigations. Specifically, we'll be looking at; Ransomware, live Incident Response using PowerForensics, and the USN Journal.

So let's dig in.

Our Scenario

Customer ABC requested assistance investigating a suspected case of Ransomware. An end user reported that file extensions had changed and they could no longer open documents. They also received a note demanding payment. The organization is curious to identify the source of the infection.

However there are two primary complications (both of which may be readily encountered in the real world). 

1. The users browser history is gone (either wiped by malware or by the user themselves - they may have been browsing inappropriate content and are concerned about the IT staff identifying this).We aren't concerned about recovering this data, we want to look for an alternate method to investigate the infection.
2. The company does not retain centralized logs, nor do they have an adequate local firewall log retention policy. There are no web browsing/firewall logs to obtain. This is of course, a finding in and of itself, but again, our focus here is working around these limitations

In a typical situation, carving browser history, and retrieving proxy/firewall logs would likely be sufficient in identifying malicious traffic but for this post we are assuming it is not available. It's important to have a variety of options when performing digital forensics and incident response. Conditions will often be less than ideal to it's important to find creative alternatives.

For this scenario we will primarily use a single tool to perform live Forensics on the compromised system, PowerForensics (from DFIR guru Jared Atkinson). This is an extremely powerful forensics framework composed of PowerShell (version 2 and higher) scripts that we can use to accomplish common DFIR tasks. Jared has an installation walk through but I found some of the subtle nuances of it weren't addressed in detail so I'd like to cover that here.

Note: Forensic best practices would advise you to take a complete, forensically sound, write-blocked image of the system, if possible, prior to executing any actions. You should always analyze an image and avoid working on a live system if it is confirmed as compromised. In this post we are performing live triage as it is being done in a 'lab' environment and for convenience sake. Depending on the situation, analyzing live systems may be entirely acceptable and preferred but this is completely circumstantial and is based on the complexity and potential legal outcomes of your investigation (and authorization of your superiors/client).

Tool Setup

On the infected machine:

1. Download PowerForensics: https://github.com/Invoke-IR/PowerForensics.
2. Right-Click the ZIP file, check unblock, click apply. This is required to properly install this module.
3. Open a PowerShell command prompt as administrator.
4. Run the command 'Set-ExecutionPolicy Unrestricted'
1. If you have issues here consult this blog post: https://blogs.msdn.microsoft.com/pasen/2011/12/07/set-executionpolicy-windows-powershell-updated-your-execution-policy-successfully-but-the-setting-is-overridden-by-a-policy-defined-at-a-more-specific-scope/
5. Close and re-open the PowerShell prompt as administrator
6. Make note of your PowerShell version using the following command:
   $PSVersionTable.PSVersion
   This is important as you will need to use PowerForensicsv2 depending on the installed version of PowerShell. For example, Windows Server 2008 has version 2 of PowerShell installed by default.
7. Now you need to identify your User Module Path. Use the following command:
   $Env:PSModulePath
   [Environment]::GetEnvironmentVariable("PSModulePath")
   This should print out a path to a series of PowerShell modules folder. There may be a series of paths but you can use one similar to: C:\Users\<username>\Documents\WindowsPowerShell\Modules
8. The path identified above is where you need to unzip PowerForensics too. Remember, if your system is running PowerShell version2, unzip PowerForensicsV2 to that location.
   Note: It looks like the latest version bundles in v2 so you should be able to handle different versions in-line. You may not need to worry about unzipping the correct version.
9. Now you can use the following commands to import PowerForensics and view the available functions it comes packaged with:
   

   Get-Module -ListAvailable -Name PowerForensics
   Import-Module PowerForensics
   Get-Command -Module PowerForensics

Background

There are lots of fantastic modules that Jared has built, but in particular we'll be using the "Get-ForensicUsnJrnl" command. You can pass it a path using the VolumeName switch to retrieve USN artifacts for a specific volume. Use the Get-Help module to obtain usage syntax for each cmdlet (Get-Help Get-ForensicUsnJrnl).

Why are we concerned with this specific cmdlet? To answer that we need a bit of background.

NTFS is a Microsoft proprietary file system, that is to say it is responsible for keeping track of and managing files on disk. As computing requirements have grown more complex, file system standards have evolved to meet those needs and have expanded to include a variety of different features such as hard links, alternate data streams, and so on. 

The USN Journal is a feature of NTFS which maintains a record of changes made to a specified volume. Each volume maintains a USN Journal log. The USN Journal is a system metafile, and can not be viewed in a regular directory listing. On most systems it is enabled by default as Windows uses it for a number of core system operations (like file replication services). When NTFS objects are added, changed, or deleted, an entry is recorded in the USN Journal for a respective volume. This log does not contain enough information to 'reverse' a change, it is simply a record of activity that includes the type of change, the file object impacted.

Each USN entry is a data structure of the following format (Note: there are 3 different USN structure formats depending on the version of Windows you are running, the one described below is the oldest format, however they are all pretty similar):

typedef struct {
  DWORD         RecordLength;
  WORD          MajorVersion;
  WORD          MinorVersion;
  DWORDLONG     FileReferenceNumber;
  DWORDLONG     ParentFileReferenceNumber;
  USN           Usn;
  LARGE_INTEGER TimeStamp;
  DWORD         Reason;
  DWORD         SourceInfo;
  DWORD         SecurityId;
  DWORD         FileAttributes;
  WORD          FileNameLength;
  WORD          FileNameOffset;
  WCHAR         FileName[1];
} USN_RECORD_V2, *PUSN_RECORD_V2, USN_RECORD, *PUSN_RECORD;

There are several important fields here, namely; FileName, Reason, Usn, TimeStamp, FileReferenceNumber.

The reason field in particular records a flag that corresponds with 20+ different possible NTFS operations such as:

• USN_REASON_BASIC_INFO_CHANGE
• USN_REASON_CLOSE
• USN_REASON_DATA_OVERWRITE
• USN_REASON_FILE_CREATE
• USN_REASON_FILE_DELETE

These operations are fairly self-explanatory. 

The USN Journal is of value to our analysis because browser operations create temporary files when accessing pages. These temporary files record Journal entries. This means that even if we lose our firewall/proxy logs and browser history, there may still be a way to identify what actions occurred at the time frame in question.

Note: It is possible to, as with anything, tamper with forensic evidence and as such an attacker could theoretically purge the USN Journal using the following command: fsutil usn deletejournal /d c:

Similarly, CCleaner/PrivaZer/etc may also clear MFT/USN Journal records. Detecting the presence of those tools may be possible through alternative artificats such as AMCache or Prefetch files.

Analysis

Keep one thing in mind - when you are dealing with the USN Journal you will be dealing with hundreds of thousands, if not millions of records. Determining what happened can be a time consuming effort so it is extremely important to narrow your focus down to a very small time frame (possibly an hour or two).

In our scenario, we are dealing with a a Ransomware infection, which typically leaves a ransom note on the desktop. We can perhaps use this to narrow our time frame.

Using PowerForensics we can pull out the MFT record for this file:

File Name: !MADEUPFILENAME.html
Master File Table Record Index: 24151
MFT Entry for Index 24151:
FullName             : C:\Users\greg\Desktop\!MADEUPFILENAME.html
Name                 : !MADEUPFILENAME.html
SequenceNumber       : 32
RecordNumber         : 24151
ParentSequenceNumber : 2
ParentRecordNumber   : 22060
Directory            : False
Deleted              : False
ModifiedTime         : 3/12/1601 1:17:27 PM
AccessedTime         : 3/12/1601 1:17:27 PM
ChangedTime          : 6/8/2016 1:27:40 AM
BornTime             : 3/12/1601 1:17:27 PM
FNModifiedTime       : 6/8/2016 1:27:40 AM
FNAccessedTime       : 6/8/2016 1:27:40 AM
FNChangedTime        : 6/8/2016 1:27:40 AM
FNBornTime           : 6/8/2016 1:27:40 AM

There are some discrepancies here between the FN and SI time stamps which may be due to stomping. However the $FN attributes accurately reflect an infection time of 6/8/2016 1:27:40 AM UTC. This is converted to 6/7/2016 6:27:40 PM EST which is roughly when the user asserts they noticed an issue.

But this might not be the first instance in which malware started executing, as the ransom note is usually dropped after file system encryption routine has been completed. We know that after each file is encrypted, the extension is changed to .crypz in our scenario. Consequently, we also can assert that the time of compromise would have to have happened just prior to the first file extension being renamed. We are essentially working backwards through the attack lifecycle.

1. Ransom Note HTML file dropped
2. Files encrypted, extension changed
3. Malicious payload executed
4. Malicious payload dropped
5. Exploit kit successful exploitation
6. Browser instantiates Flash/JS to execute vulnerability
7. Web page redirect to malicious landing page

So, we can constrain our review of the USN Journal to the first instance of .crypz minus 30 minutes to be safe. It is also useful to grep on specific extensions such as .js, .zip, .exe, .dll, .swf, .htm, and others that may be involved in the attack lifecycle to further constrain your data set.

Using this technique I was able to identify the following sequence of events:

VolumePath               : \\.\C:
Version                  : 2.0
RecordNumber             : 33080
FileSequenceNumber       : 28
ParentFileRecordNumber   : 206296
ParentFileSequenceNumber : 97
Usn                      : 4743722112
TimeStamp                : 6/8/2016 1:22:51 AM
Reason                   : DATA_EXTEND, FILE_CREATE, CLOSE
SourceInfo               : 0
SecurityId               : 0
FileAttributes           : ARCHIVE, NCI
FileName                 : (REDACTEDWEBPAGE).jpg

VolumePath               : \\.\C:
Version                  : 2.0
RecordNumber             : 33299
FileSequenceNumber       : 209
ParentFileRecordNumber   : 206165
ParentFileSequenceNumber : 763
Usn                      : 4743722424
TimeStamp                : 6/8/2016 1:22:51 AM
Reason                   : FILE_CREATE
SourceInfo               : 0
SecurityId               : 0
FileAttributes           : ARCHIVE, NCI

FileName                 : dsF5h3S[1].htm

VolumePath               : \\.\C:
Version                  : 2.0
RecordNumber             : 33707
FileSequenceNumber       : 106
ParentFileRecordNumber   : 899
ParentFileSequenceNumber : 21
Usn                      : 4743725296
TimeStamp                : 6/8/2016 1:22:51 AM
Reason                   : FILE_CREATE
SourceInfo               : 0
SecurityId               : 0

FileAttributes           : 8208

FileName                 : aojwoejfi.obtainwhite.top

VolumePath               : \\.\C:
Version                  : 2.0
RecordNumber             : 33303
FileSequenceNumber       : 620
ParentFileRecordNumber   : 244618
ParentFileSequenceNumber : 3
Usn                      : 4743722896
TimeStamp                : 6/8/2016 1:22:51 AM
Reason                   : FILE_CREATE
SourceInfo               : 0
SecurityId               : 0
FileAttributes           : ARCHIVE, NCI
FileName                 : truck-yield-damage-objection-journey-punish-dizzy-sl

                           ide-argument[1].swf

The HTM file was recovered from the file system, essentially when loaded it redirects to the obtainwhite.top domain which instantiates the SWF flash object. The randomized sub-domain (as well as odd TLD '.top), and randomized SWF file name are both highly suspect.

Not surprisingly the SWF flash object was ZLIB compressed. After unpacking, it was obviously an Exploit Kit landing page used to exploit some older (2014) browser vulnerabilities. The ransomware variant was a much newer iteration at the time.

Virustotal results (almost 6 months later) are somewhat discouraging for this domain:
https://www.virustotal.com/en/url/fce8be210eecf3088d1436a627fd29893872e1e8ae04f2385dcdc92b66cede02/analysis/

Registrant information for this domain:
Updated Date: 2016-06-08T00:20:29Z Creation Date: 2016-06-08T00:09:59Z Registry Expiry Date: 2017-06-08T00:09:59Z Sponsoring Registrar: Alpnames Limited Sponsoring Registrar IANA ID: 1857 Domain Status: clientTransferProhibited https://www.icann.org/epp#clientTransferProhibited Registrant ID: alp_54912665 Registrant Name: Clive Hoetnez Registrant Organization: N/A Registrant Street: N/A Registrant City: Smallvile Registrant State/Province: Arkansas Registrant Postal Code: 43547 Registrant Country: US Registrant Phone: +1.447898 Registrant Phone Ext: Registrant Fax: Registrant Fax Ext: Registrant Email: zizsdcqe@6paq.com

6paq is a random temporary email provider.

Conclusion

I didn't really want to cover much else specifically in this post. This is really only one component of an investigation of this type. There are a number of other artifacts and avenues you would want to go down to draw broader conclusions about your case. Mostly, my intent was to demonstrate some of the thought process in tackling a fairly common problem with a somewhat creative solution and demonstrate the importance of this quality in DFIR work. Hopefully you learned something from this post, please feel free to leave comments as usual.

Computer Security Incident Handling Guide - A presentation based off of the NIST paper

A few years ago during an interview at Mandiant I was asked to create a presentation based on the NIST Computer SecurityIncident Handling Guide, a good primer on incident handling that I would recommend every NetSec  professional to read.

Although the presentation is light in description, the basic outline remains. If the content interests you I would highly recommend reading the NIST report.

Audit Logging Configuration for the Linux Environment

One challenge to performing a proper incident investigation is dealing with missing event logs. Part of a healthy SOC posture is ensuring that you have the proper audit logging settings to ensure that you log what is needed tomorrow.

Windows has a very well defined audit policy, but when I was trying to find an audit policy for the Linux audit system I found it much more difficult than expected. One excellent resource I found is the Defense Information Systems Agency (DISA)'s Security Technical Implementation Guide (STIG) for Red Hat Enterprise Linux 6, which can be found here, and the STIG Viewer can be found here.

To open the STIG, open the JAR file by double clicking on it (or running java -jar <filename> from the CLI.) Next, go to file --> Import STIG and traverse to the STIG file location. Next, change the file extension box from XML to ZIP:

The RHEL STIG will open with 236 security-based rules that outline how the Department of Defense audits their RHEL systems to ensure that they adhere to their security standards, and I would highly recommend going through it if you manage a RHEL system. But today we will be focusing on the RHEL audit configuration recommendations. Enter the word audit in the Filter window pane of the STIG viewer (I typically don't push enter as I find that this works better.) From there you should filter for rules that contain the word, well, audit

The Linux system uses the auditd service to create and log audit events within the system. The first step is to ensure that the auditd service is enabled:

# systemctl is-active auditd.service 
Active: active (running) since Tue 2015-01-27 19:41:23 EST; 22h ago

Next, the auditd service uses the audit.rules configuration file to determine what actions to audit. Details on the CentOS audit configuration file can be found on Digital Ocean's tutorials here and here. As for what to audit, the STIG recommends that we run a search on the file system to find all programs with execution function capabilities and add them to the audit configuration file. To find relevant setuid and setgid programs, use the following command once for each local partition:

find / -xdev \( -perm -4000 -o -perm -2000 \) -type f | awk '{print "-a always,exit -F path=" $1 " -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged" }'

This outputs a series of audit configuration parameters that can be placed in the /etc/audit/rules.d/audit.rules file to ensure that using said commands/binaries is a logged event. An example output for my Linux box shows:

-a always,exit -F path=/usr/bin/wall -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chage -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/newgrp -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chfn -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/write -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chsh -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/mount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/at -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/su -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/umount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/sudo -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/gpasswd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/pkexec -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/crontab -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/locate -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/ssh-agent -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/staprun -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/pam_timestamp_check -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/unix_chkpwd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/netreport -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/usernetctl -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/userhelper -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/postdrop -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/postqueue -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib/polkit-1/polkit-agent-helper-1 -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib64/dbus-1/dbus-daemon-launch-helper -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/libexec/utempter/utempter -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/libexec/openssh/ssh-keysign -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/libexec/abrt-action-install-debuginfo-to-abrt-cache -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged

Going through the rest of the STIG shows eight categories of audit logging settings that I have pasted into this very pretty and convenient table:

 
 
 
 
 


  
Log Group
  
Activity
  
Audit Trigger
  
Details
 

  

1

  
System time alteration
  
settimeofday
  
-a always,exit
  -F arch=b32 -S settimeofday -k audit_time_rules

    -a always,exit -F arch=b64 -S settimeofday -k audit_time_rules
 

  
stime
  
-a
  always,exit -F arch=b32 -S stime -k audit_time_rules 

    -a always,exit -F arch=b64 -S adjtimex -S settimeofday -S clock_settime -k
  audit_time_rules
 

  
clock_settime
  
-a
  always,exit -F arch=b32 -S clock_settime -k audit_time_rules

    -a always,exit -F arch=b64 -S clock_settime -k audit_time_rules
 

  
localtime
  
-w /etc/localtime -p
  wa -k audit_time_rules
 

  

2

  
Account Changes
  
group
  
-w /etc/group -p wa
  -k audit_account_changes
 

  
passwd
  
-w /etc/passwd -p wa
  -k audit_account_changes
 

  
gshadow
  
-w /etc/gshadow -p wa
  -k audit_account_changes
 

  
shadow
  
-w /etc/shadow -p wa
  -k audit_account_changes
 

  
opasswd
  
-w
  /etc/security/opasswd -p wa -k audit_account_changes
 

  

3

  
Network Modifications
  
sethostname
  
-a
  always,exit -F arch=B32 -S sethostname -S setdomainname -k
  audit_network_modifications

    -a always,exit -F arch=B64 -S sethostname -S setdomainname -k
  audit_network_modifications
 

  
issue
  
-w /etc/issue -p wa
  -k audit_network_modifications
 

  
issue.net
  
-w /etc/issue.net -p
  wa -k audit_network_modifications
 

  
hosts
  
-w /etc/hosts -p wa
  -k audit_network_modifications
 

  
network
  
-w
  /etc/sysconfig/network -p wa -k audit_network_modifications
 

  

4

  
Mandatory Access
  Control (MAC) configuration (SELinux)
  
MAC-policy
  
-w /etc/selinux/ -p
  wa -k MAC-policy
 

  

5

  
High Priority Commands
  
chmod
  
-a
  always,exit -F arch=b32 -S chmod -F auid>=500 -F auid!=4294967295 -k
  perm_mod 

    -a always,exit -F arch=b32 -S chmod -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S chmod -F auid>=500 -F auid!=4294967295 -k
  perm_mod 

    -a always,exit -F arch=b64 -S chmod -F auid=0 -k perm_mod
 

  
chown
  
-a
  always,exit -F arch=b32 -S chown -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S chown -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S chown -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b64 -S chown -F auid=0 -k perm_mod
 

  
fchmod
  
-a
  always,exit -F arch=b32 -S fchmod -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S fchmod -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S fchmod -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b64 -S fchmod -F auid=0 -k perm_mod
 

  
fchmodat
  
-a
  always,exit -F arch=b32 -S fchmodat -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S fchmodat -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S fchmodat -F auid>=500 -F auid!=4294967295
  -k perm_mod

    -a always,exit -F arch=b64 -S fchmodat -F auid=0 -k perm_mod
 

  
fchown
  
-a
  always,exit -F arch=b32 -S fchown -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S fchown -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S fchown -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b64 -S fchown -F auid=0 -k perm_mod
 

  
fchownat
  
-a
  always,exit -F arch=b32 -S fchownat -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S fchownat -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S fchownat -F auid>=500 -F auid!=4294967295
  -k perm_mod

    -a always,exit -F arch=b64 -S fchownat -F auid=0 -k perm_mod
 

  
fremovexattr
  
-a
  always,exit -F arch=b32 -S fremovexattr -F auid>=500 -F auid!=4294967295
  -k perm_mod

    -a always,exit -F arch=b32 -S fremovexattr -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S fremovexattr -F auid>=500 -F
  auid!=4294967295 -k perm_mod

    -a always,exit -F arch=b64 -S fremovexattr -F auid=0 -k perm_mod
 

  
fsetxattr
  
-a
  always,exit -F arch=b32 -S fsetxattr -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S fsetxattr -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S fsetxattr -F auid>=500 -F auid!=4294967295
  -k perm_mod

    -a always,exit -F arch=b64 -S fsetxattr -F auid=0 -k perm_mod
 

  
lchown
  
-a
  always,exit -F arch=b32 -S lchown -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S lchown -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S lchown -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b64 -S lchown -F auid=0 -k perm_mod
 

  
lremovexattr
  
-a
  always,exit -F arch=b32 -S lremovexattr -F auid>=500 -F auid!=4294967295
  -k perm_mod

    -a always,exit -F arch=b32 -S lremovexattr -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S lremovexattr -F auid>=500 -F
  auid!=4294967295 -k perm_mod

    -a always,exit -F arch=b64 -S lremovexattr -F auid=0 -k perm_mod
 

  
lsetxattr
  
-a
  always,exit -F arch=b32 -S lsetxattr -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S lsetxattr -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S lsetxattr -F auid>=500 -F auid!=4294967295
  -k perm_mod

    -a always,exit -F arch=b64 -S lsetxattr -F auid=0 -k perm_mod
 

  
removexattr
  
-a
  always,exit -F arch=b32 -S removexattr -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S removexattr -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S removexattr -F auid>=500 -F
  auid!=4294967295 -k perm_mod

    -a always,exit -F arch=b64 -S removexattr -F auid=0 -k perm_mod
 

  
setxattr
  
-a
  always,exit -F arch=b32 -S setxattr -F auid>=500 -F auid!=4294967295 -k
  perm_mod

    -a always,exit -F arch=b32 -S setxattr -F auid=0 -k perm_mod

    -a always,exit -F arch=b64 -S setxattr -F auid>=500 -F auid!=4294967295
  -k perm_mod

    -a always,exit -F arch=b64 -S setxattr -F auid=0 -k perm_mod
 

  
mount
  
-a
  always,exit -F arch=b32 -S mount -F auid>=500 -F auid!=4294967295 -k
  export

    -a always,exit -F arch=b32-S mount -F auid=0 -k export

    -a always,exit -F arch=b64 -S mount -F auid>=500 -F auid!=4294967295 -k
  export

    -a always,exit -F arch=b64 -S mount -F auid=0 -k export
 

  
unlink
  
-a
  always,exit -F arch=b32 -S rmdir -S unlink -S unlinkat -S rename -S renameat
  -F auid>=500 -F auid!=4294967295 -k delete

    -a always,exit -F arch=b32 -S rmdir -S unlink -S unlinkat -S rename -S
  renameat -F auid=0 -k delete

    -a always,exit -F arch=b64 -S rmdir -S unlink -S unlinkat -S rename -S
  renameat -F auid>=500 -F auid!=4294967295 -k delete

    -a always,exit -F arch=b64 -S rmdir -S unlink -S unlinkat -S rename -S
  renameat -F auid=0 -k delete
 

  
lnsmod
  
-w
  /sbin/insmod -p x -k modules

    -w /sbin/rmmod -p x -k modules

    -w /sbin/modprobe -p x -k modules

    -a always,exit -F arch=b32 -S init_module -S delete_module -k modules

    -a always,exit -F arch=b64 -S init_module -S delete_module -k modules
 

  

6

  
Unauthorized File
  Access Attempt
  
open
  
-a
  always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate
  -F exit=-EACCES -F auid>=500 -F auid!=4294967295 -k access

    -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S
  ftruncate -F exit=-EPERM -F auid>=500 -F auid!=4294967295 -k access

    -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S
  ftruncate -F exit=-EACCES -F auid=0 -k access

    -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S
  ftruncate -F exit=-EPERM -F auid=0 -k access

    -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S
  ftruncate -F exit=-EACCES -F auid>=500 -F auid!=4294967295 -k access

    -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S
  ftruncate -F exit=-EPERM -F auid>=500 -F auid!=4294967295 -k access

    -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S
  ftruncate -F exit=-EACCES -F auid=0 -k access

    -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S
  ftruncate -F exit=-EPERM -F auid=0 -k access
 

  

7

  
Sudoers
  
Sudoers
  
-w /etc/sudoers -p wa
  -k actions
 

  

8

  
All privileged
  Commands
  
Priv commands
  
To
  audit use of all privileged commands run:

          # find PART -xdev \( -perm
  -4000 -o -perm -2000 \) -type f | awk '{print \ "-a always,exit -F
  path=" $1 " -F perm=x -F auid>=1000 -F auid!=4294967295 \ -k
  privileged" }'

    

    Next, add those lines to the /etc/audit/audit.rules file.

Now when commands are run /var/log/audit/audit.log. Running the command chmod produces the following output:

[epicism@rhel audit]# chmod 777 test
[epicism@rhel audit]# tail-f /var/log/audit/audit.log
type=SYSCALL msg=audit(1469583828.606:21849): arch=c000003e syscall=268 success=yes exit=0 a0=ffffffffffffff9c a1=bd00f0 a2=1ff a3=7fff145eaaf0 items=1 ppid=6170 pid=13606 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="chmod" exe="/usr/bin/chmod" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="perm_mod"
type=CWD msg=audit(1469583828.606:21849):  cwd="/var/log/audit"
type=PATH msg=audit(1469583828.606:21849): item=0 name="test" inode=67225482 dev=fd:00 mode=0100777 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:auditd_log_t:s0 objtype=NORMAL

No common SIEM can parse this, but Splunk could probably parse it out pretty easily. In the least I would recommend backing up the audit.log files to a repository for audit purposes. In any case, I hope that the post helps to set a baseline audit configuration file on RHEL.

Update: Our friend Dixie Flatline was kind enough to share the Audisp plugin which converts Linux audit events to CEF (ArcSight's Comment Event Format standard) which can be interpreted by ArcSight directly without a parser.

AWS Security: Automating Palo Alto firewall rules with AWS Lambda

With the increased adoption of IaaS cloud services such as Amazon Web Services (AWS) and Microsoft Azure, there is also a greater need for security controls in the cloud. Firewall and IPS vendors such as Palo Alto, Checkpoint and Fortinet have made available virtual instances of their products ready to run in these cloud environments. These tools can provide great advantages on top of the existing security controls inherent in the cloud platform, and can provide more inspection capabilities and filtering, especially at the application level. In addition to firewall capabilities, these products can provide extra features such as intrusion prevention, URL filtering, and other features that are lacking with the native security controls.

However, there is still lots of manual configuration expected from the security or network administrators, such as configuring interfaces or network settings, firewall and threat prevention rules, and so on. But in a cloud environment where speed and flexibility are expected, waiting for a CAB meeting, change window, etc. for a newly created instance can seem to be a step back. Firewall vendors are definitely aware of this, and have introduced features such as Palo Alto’s ability to read AWS attributes (for example tags or instance IDs) and use them in dynamic rules that get updated as changes occur in the cloud environment. For example, you can create a group with the setting to include anything with a certain tag, and then use this group in a security rule to allow Web traffic to it. The gateway will then know to allow web traffic to any new or existing instance that has this tag. A full list of monitored AWS attributes can be found here.

But there are still more features that you might want to configure automatically that are not included by the vendor. The vendors mentioned above all have API interfaces available, and so combining that with the tools from Microsoft or Amazon, we can easily write small pieces of code to automate lots of these tasks. To demonstrate this, I wrote a lambda function that monitors AWS instance starts and stops, as well as security group updates, and then pushes or deletes rules to a Palo Alto gateway based on these changes.

In this post, I will go over the different components of my aws-lambda-paloalto code, its design, and then show it in action. Finally i will list a number of notes and considerations as well as a link to download the code.

Components

Palo Alto

Palo Alto instances can be accessed from the Amazon marketplace. Palo Alto provides excellent documentation on how to set up a gateway in the AWS, and I would recommend to start here for the initial configuration. Another useful case study provided by Palo Alto is on how to configure and use dynamic address groups in rules, where the groups are based on AWS attributes.

For this setup, I had a Palo Alto gateway configured as an Internet gateway in AWS, and so all internet traffic from my instances was passing through it. I also had an elastic IP assigned to the management port, and my lambda function used this IP address to configure the gateway. Using a public IP address to configure the gateway is not the best option, and I would recommend using a private IP instead (I address some of the limitations this might introduce in the notes section below).

AWS CloudTrail/S3

CloudTrail is a service from AWS to log and store the history of all API calls made in an AWS environment. CloudTrail saves all the logs to AWS S3 which is another AWS service that provides object storage. CloudTrail has to be enabled so that we can monitor when changes are made that are relevant to our code, and then act based on that information.

More details on CloudTrail and S3 can be found here and here.

AWS Lambda

Lambda is a service from AWS that lets you upload code, and AWS will run it for you based on triggers you set up,such as triggers from other AWS services or external triggers, without having to run a dedicated instance for your code. In our case, we will use CloudTrail (S3) as a trigger for our code so that whenever a new change is made in the environment, the code can scan the changes and determines if a corresponding change is required in the Palo Alto gateway.

Details on AWS Lambda can be found here.

Code Features/Design

Adding rules

The lambda function that will monitor the following two events for adding new rules:

• StartInstances: Event indicating that a new instance was started
• AuthorizeSecurityGroupIngress: Event indicating that a new rule was adding to an existing security group

Once any of these two events are detected, the Lambda function will extract the relevant information for rules required to the instances affected, and add the corresponding rules to our Palo Alto instance.

The script has been designed to add rules with the name corresponding to the type of event that triggered them. For example, if the rule added is because of an instance with instance-id X, then the rule name is  ‘X-#’ where # is increased with every rule added. Correspondingly,If the Lambda function executes due to a change in a security group with group-id Y, then the naming is ‘Y-#’. This naming convention is used by the code to track the rules it added.

Unnecessary rules

Since the Palo Alto gateway is running as an internet gateway, there are many scenarios that are not relevant, and the code will try to filter out these events so that we don’t make any unnecessary changes. The following scenarios would not introduce changes to the Palo Alto gateway:

• Instances that are started but that don’t use the Palo Alto as their internet gateway. For example, there can be multiple internet gateways configured and we're only concerns with instances that use the Palo Alto to reach the internet.
• In instances with multiple interfaces, the script checks all the interfaces, and only includes those that use the Palo Alto instance as an internet gateway.
• Security group rules that have a source from within the AWS VPC (Virtual Private Cloud) will be filtered out. The Palo Alto gateway in this instance is used as an internet gateway, and so traffic from within the VPC would not pass through it.
• Security group rules that reference other security groups as a source will also not be included. These rules imply that the traffic would be local to the VPC and so would not pass the internet gateway.

Also before adding a rule, a test is made to make sure traffic is not already allowed, and only after making sure that traffic is denied, we will add a new rule.

Rule location

The code will also only add rules at the bottom so that the security administrator can create rules at the top of the rule base that would override anything added dynamically. This can be used to control the rules automatically added. Furthermore, we specify in the code the bottom most rule that the new rules must  go above in order to avoid adding rules below bottom rules such as the cleanup rule.

Cleaning up

When instances are stopped or rules in security groups are removed we want the rules that we added to be removed. To avoid removing any permanent rules added by the security administrator, the code will only remove rules that it added previously to the rule base based on the naming convention mentioned earlier. The following events are monitored as triggers:

• StopInstances: An instance was stopped.
• RevokeSecurityGroupIngress: A rule was removed from a security group.

Imported Modules

I tried not to import any modules that don’t come with a default installation of Python except when needed. The only exceptions are:

Boto3

Boto3 is the AWS SDK for python. Using boto3 we can make API calls to AWS to get relevant information that will help us gather the necessary details to read events from AWS and build the rules and changes we want to push to Palo Alto. More details on boto3 https://boto3.readthedocs.io/en/latest/.

Paloalto.py

These are functions that I wrote to interface with the Palo Alto gateway. The functions include adding/deleting rules or objects, searching rules and getting details, and saving changes.

To download the latest version of this code, refer to the github link. You can also refer to this blog post which goes over the details for writing it.

Netaddr

I used the IPAddress and IPNetwork functions from netaddr to allow quick checks on IP addresses (For example, if an IP address belongs to a certain subnet).

Event Handling Logic

The main function in the code is the lambda_handler function. When a Lambda trigger occurs, AWS calls this function and passes the event details that triggered it, which in this case would be adding a new entry to S3 by CloudTrail. Four things to take into consideration:

1. The event passed by AWS contains the location of the S3 object that has the new cloudtrail entries. Our first step is to extract the name and location of this file.
2. Second we have to retrieve the file using the S3 methods from boto3, and then uncompress it using gzip.
3. The contents are then parsed as json to allow us to read and extract properties easily.
4. Finally we iterate through all the records in the logs provided searching for any of the following events:

StartInstances

1. Call event_StartInstances which returns the list of rules relevant to the Instance in the event. In this function, a list of instance Ids are extracted, and the following is performed for each instance.
1. First a list of relevant subnets is created. Relevant subnets are those that use the Palo Alto as their internet gateway.
2. Second, a list of all interfaces belonging to the instance is created along with the subnets each belongs to.
3. For each interface that belongs to a subnet in the relevant subnets, a list of security groups attached to it is compiled.
4. Finally, the rules of all security groups compiled are parsed through, and the relevant rules are added to a list to be sent back.
2. For each rule returned, first we have to convert the format to something that can be understood by Palo Alto. This means that we need to add zone definition, translate the destination port to a corresponding service and application, and specify the action for the rule. I used the aws_rules_to_pa function to convert the format, which in turn uses aws_to_pa_services to map port numbers to application and service combinations.
3. Once the format is changed, we can now test the existing rule base allows this traffic. If it is already allowed, then we move to the next rule in our list from the first Step, otherwise we add the rule on the Palo Alto gateway and move it to the proper location.
4. Finally, commit to save the changes on the Palo Alto.

StopInstances

1. Call event_StopInstances to get a list of Instance Ids from the log event.
2. For each instance id, call paloalto_rule_findbyname to get a list of all existing rules added by earlier by our code.
3. Remove each rule returned.
4. Commit to save changes

AuthorizeSecurityGroupIngress

1. Call event_AuthorizeSecurityGroupIngress to get a list of all rules to be added. (Similar function to event_StartInstances described above).
2. Convert each rule to Palo Alto format using aws_rules_to_pa.
3. Find if there are already existing rules that would allow the traffic for each rule, and discard any rule that has a match.
4. Add remaining rules and move them to the proper location.
5. Commit to save changes.

RevokeSecurityGroupIngress

1. Call event_RevokeSecurityGroupIngress to get a list of relevant security rules to be removed.
2. Find all rules added by our code for this security group (using the rule names)
3. Compare the matching rules on the Palo Alto with the list of relevant rules from Step 1.
4. Remove rules that match both lists.
5. Commit to save changes.

In action

Setup

To run the code as is, the following will be required:

1. Palo Alto instance configured with a publicly accessible IP address for management.
2. Lambda function created with the following settings (For help configuration the lambda function, refer to this link, and in particular Using AWS Lambda with AWS CloudTrail):
1. Handler should be set to lambda.lambda_handler.
2. No VPC set. (If you would like to set a VPC, refer to the Notes section below for more details).
3. IAM role (with policy attached) to allow the lambda function access to query your S3 and EC2 resources.
4. Timeout value of 25 seconds.
5. Trigger set to the S3 bucket containing the CloudTrail logs.
6. Runtime set to ‘Python 2.7’
3. Finally you will need to upload the code as a zip file to your lambda function. Before doing so, there are some hardcoded variables that need to be set first (All of which are at the top of the lambda_handler function in lambda.py):
1. pa_ip: IP address of your Palo Alto gateway.
2. pa_key: Access key for the Palo Alto gateway (Refer to the Pan-OS XML API User guide for more details on this, and specifically this page).
3. pa_bottom_rule: Name of the rule which the lambda function would be adding on top of. This would usually be the clean up rule in your security policy.
4. pa_zone_untrust: Name of the outside zone configured on the Palo Alto gateway.
5. pa_zone_trust: Name of the inside zone configured on the Palo Alto gateway.
6. pa_sgp: name of security profile group in Palo Alto to be set on rules added.
7. igwId: Instance id of the Palo Alto gateway.

Runtime

In the following example, i had a simple setup of a 3 web server instances that use a Palo Alto instance as their internet gateway. I set the variables for my lambda function to point to my Palo Alto, provided the Access Key, etc.

I had a basic security rule base configured with 4 rules initially:

• Two rules for my web servers. One rule to allow access in (on tcp ports 80,81, and 8000), and one rule to allow access out from the web servers.
• One rule to deny any clear text authentication protocols such as ftp, telnet, etc.
• Finally a clean up rule so that all other dropped traffic is logged.

I set the ‘Clean up’ rule to be my bottom rule in the lambda function, so that all rules created would be added between rules 3 and 4.

Starting an Instance

I then started my 'web server 2' instance which had the IP address 172.20.200.225, and had the webserver_sg security group assigned, which allowed traffic from any internet source to destination ports 80 and 443:

Once the instance is started, you can see the Palo Alto rulebase updated with new rule #4:

Note that only one rule was added (for ssl - tcp port 443) since port 80 was already allowed by rule #2 in the rulebase.

Adding rules to a security group

Next I updated the security group ‘webserver_sg’ and added two new rules:

The lambda function adds two new rules with the security group id as the name:

Removing a rule from a security group

Finally, I removed one of the newly added rules from the security group (for port 22):

And the rulebase was updated accordingly:

Notes and considerations

• There might be a delay from the time of an event to the time the action is seen in the Palo Alto gateway. This is because AWS can have up to 5-15 minutes delay from the time an API call is made to the time it is logged in CloudTrail. I am not aware of an easy way to overcome this other than configuring the lambda function to run on a schedule (for example every 1 to5 minutes,) or moving the code to be run continuously on an instance that has access to CloudTrail and can monitor it in real time.
• In my tests, I used a public IP address of the Palo Alto gateway to configure it. This was easier since I didn’t place my lambda function to run from within my VPC, and so it couldn’t access the private IP address. To have the lambda function access the Palo Alto gateway through a private IP address, the lambda function must be run from within the VPC with security groups assigned to allow it to access the private IP of the Palo Alto. Furthermore, running lambda from within the VPC might interfere with how it accesses S3 objects since those are accessed through the internet. The easiest way to get around this to have an endpoint created in the VPC for accessing S3 (See https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/).
• All ICMP rules from AWS are treated the same when pushed to Palo Alto (configured with the application ‘icmp’ that would allow all types of ICMP regardless of the configuration in the security groups). Modify the function aws_to_pa_services to introduce more granularity.
• Currently only inbound rules from the security groups are examined and added, but I will be adding support for outbound rule access as well.

Download

You can download the latest version of the code from github.

To use the code as is, you only need to upload the zip file to your lambda function. If you want to make modifications, you have to zip all files (lambda.py,paloalto.py, netaddr, adn netaddr-0.7.18.dist-info).

I hope this has been helpful, and note that while the functionality described in this post should be fully functional, there are a number of other features that are in progress, and the github link will be updated as these features are completed.