1. No category

Copyright © 2014 Splunk Inc.
From Tool to Team Member:
Controlling Systems with
Splunk Alert Scripts
George Starcher
Security Engineer,
Peak Hosting
About Me
• George Starcher, Information Security Engineer
• CISSP, Splunk Certified Knowledge Manager and Splunk
Certified Administrator
• Splunk IRC Channel
• Looking to kick off a Nashville, TN - Splunk User Group
• www.georgestarcher.com
• www.peakhosting.com
2
Agenda
Splunk from Tool to a Team Member
How it Works
Getting into the Code
Alert Script to Intrusion Prevention System Control
Alert Script to X-ARF Abuse Reporting
“Using Alert Scripts to take action on our behalf,
we can transform Splunk from a tool to a team
member.”
3
Splunk from Tool to Team Member
Manual Abuse Scanning Process
Reviewed SSH, RDP, VNC etc daily
Consumed 30-45 minutes per day
Permanent blacklist entries
Moved to automated process
Scheduled Splunk Searches driven by any log source
Web Services (REST) calls to the IPS
Greatly reduced time and static blacklist maintenance
4
Splunk from Tool to Team Member
Outlook Web Access - Phishers
Started Feb 10, 2014
• Noticed unexpected Exchange OWA from Nigeria
• Blocked for any access from Nigeria every 5 minutes
Expanded Multi Country Feb 15, 2014
Feb 17, 2014
•
Blocked for combination from certain countries & a
lookup table of hosted providers
5
Splunk from Tool to Team Member
Outlook Web Access - Phishers
Hosted
Lookup users
by src_ip:
Single User by
src_ip_country:
6
How it Works
How it Works
http://blogs.splunk.com/2011/03/15/storing-encrypted-credentials/
http://www.georgestarcher.com/splunk-alert-scripts-automating-control/
$splunk_home/etc/auth/splunk.secret
8
How it Works
Alert Service Account
Setup a service account to own the Alert Searches: svc-alert
Create a role just for the alert account
That role must have ‘admin_all_objects’
The role must have access to all indexes that might have the
data for the scheduled search alert.
Consider restricting access to the rest API port 8089 to only
Splunk servers and any other dedicated systems.
9
How it Works
Intrusion Prevention Appliance
10
How it Works
SPLUNK_ARG
11
How it Works
12
Alert Script in Action
X-ARF Abuse Reporting
Avoids manual reporting
Ensures timely action
Consistent Reporting Format
Accurate Evidence Data
Splunky the Intern works around the clock and
doesn't need coffee
13
Alert Script in Action
X-ARF Abuse Reporting
14
Alert Script in Action
X-ARF Abuse Reporting
15
Crawling into the Code
@SplunkDev Team - THANKS!!
@gblock - Glenn Block
@damiendallimore -Damien Dallimore
David Noble - Twitter App
17
Where can you get the code?
Github Repository
https://github.com/georgestarcher/Splunk-Alert
General Intrusion Prevention System Example Code
Google Spreadsheet Upload Code
X-ARF Abuse Reporting Code
The Google Spreadsheet Example
http://www.georgestarcher.com/splunk-alert-scriptsautomating-control/
18
Arguments Sent to Alert Scripts
http://docs.splunk.com/Documentation/Splunk/6.1.3/Alert/Configuringscriptedalerts
SPLUNK_ARG_0 Script name
SPLUNK_ARG_1 Number of events returned
SPLUNK_ARG_2 Search terms
SPLUNK_ARG_3 Fully qualified query string
SPLUNK_ARG_4 Name of report
SPLUNK_ARG_5 Trigger reason (for example, "The number of events
was greater than 1")
SPLUNK_ARG_6 Browser URL to view the report
SPLUNK_ARG_8 File in which the results for this search are stored
(contains raw results)
19
The Code Modules - IPS
credentialsFromSplunk.py
A Python class to fetch the saved service account
targetlist.py
The Python class for data to be handled
ips.py
The Python class for an IPS rest API interface
alert_script.py
The main Python alert script
20
credentialsFromSplunk.py
a re-usable Python class to fetch stored user credentials from
Splunk
provide the app where credentialed is stored: splunkapp
provide the purpose name used when saving the credentials:
realm
provide the username to be retrieved: username
call the getPassword method
21
credentialsFromSplunk.py
# Define the source in Splunk for the stored credential
splunkapp = "myadmin"
realm = 'ips'
username = 'splunk'
from alert_script.py
# Define the ips connection
ipsCredential = credential(splunkapp,realm,username)
# Get the stored credential from Splunk
try:
ipsCredential.getPassword(sessionKey)
except Exception, e:
logError("Splunk Credential Error: %s" % str(e))
exitAlertScript(_SYS_EXIT_FAILED_SPLUNK_AUTH)
22
targetlist.py
a simple Python class for a single column list of source IPs
populated by the alert search returning only source IPs
takes argument of path to the search results to load the list
# Obtain the path to the alert events compressed file and load the search results to the list
alertEventsFile = os.environ['SPLUNK_ARG_8']
try:
alertTargetList = targetlist(alertEventsFile)
except Exception, e:
logError("Target File Error: %s" % str(e))
exitAlertScript(_SYS_EXIT_FAILED_TARGET_FILE)
23
from alert_script.py
ips.py
an example Python class to interface with our Intrusion
Detection System Rest API
setup and retrieve the credential from splunk: ipsCredential
provide the IPS quarantine policy name: policy_name
provide IP address of the IPS management Interface: ips_ip
activate the IPS rest connection object
loop through the alertTargetList having the IPS quarantine each
IP
Make your Own REST API wrapper class to control other systems.
24
ips.py
from alert_script.py
# Active the ips connection object
try:
ssh_ips = ips(ips_ip,ipsCredential.username,
ipsCredential.password,policy_name)
except Exception, e:
logError("IPS Error: %s" % str(e))
exitAlertScript(_SYS_EXIT_FAILED_IPS)
25
alert_script.py
the main script called by Splunk for our alert search
imports all our classes
parses the sessionKey
connects to our IPS
pulls in the search result list of IP addresses
loops through the IP list and tells the IPS to quarantine
them
26
alert_script.py
The Hash Bang:
#!/opt/splunk/bin/python
# Obtain the Splunk authentication session key
…
# Adjust the returned sessionKey text based on Splunk version
…
# Quarantine each source ip in the alert results table
for address in alertTargetList.targetlist:
try:
ssh_ips.addQuarantine(address)
except Exception, e:
logError("IPS Quarantine Error: %s" % str(e))
exitAlertScript(_SYS_EXIT_FAILED_IPS)
27
Extended Abuse Reporting - X-ARF
http://www.x-arf.org/
S
U
N
O
B
Much more complex code
Search results driving results is a table of data not a simple IP
list
Pulls email settings from Splunk
Builds the email body using the Python Mako template (mail
merge to search results)
Improved alert script action logging sending into
index=_internal
Attaches Alert Event Search results from Splunk REST API Calls
28
The Search for Alerting
tag=authentication action=failure app=sshd NOT src_ip=10.0.0.0/8 | stats count values(user) AS
Users first(_time) AS EndTime last(_time) AS StartTime by src_ip, host, app | where count>4 |
lookup abuseLookup ip AS src_ip| lookup dnsLookup ip AS src_ip OUTPUT hostname AS
src_hostname | eval src_hostname=if(isNull(src_hostname),"unknown",src_hostname) | lookup
dnsLookup hostname AS host OUTPUT ip AS dest_ip | eval dest_host=host | iplocation src_ip | eval
StartTime=strftime(StartTime,"%Y-%m-%d %T %z") | eval EndTime=strftime(EndTime,"%Y-%m-%d %T
%z") | eval City=if(isNull(City),"unknown city",City) | eval Region=if(isNull(Region),"unknown
region",Region) | eval Country=if(isNull(Country),"unknown country",Country)| eval
DistinctUserNames=mvcount(Users) | table src_ip, src_hostname, City, Region, Country,
abusecontact, StartTime, EndTime, dest_ip, dest_host, app, count, DistinctUserNames, Users |
search dest_ip=* NOT abusecontact=spam* NOT abusecontact="[email protected]" NOT
src_hostname=*.shodan.io
29
The Code Modules - X-ARF
abuselist.py
The data to be handled, a different version than for IPS
emailSplunkXARF.py
A python class to for the mail settings and sending reports
xarf-abuse.tmpl
Abuse report Email mako template
alert_to_xarf.py
The main alert script
30
abuselist.py
method getEvidence holds the evidence search executed against
the Splunk REST API
this method also manipulates the earliest/latest timestamp coming
from the search results automatically to go into the detail evidence
search
emailSplunkXARF.py
method getMailSettings is Splunk REST API call to fetch the
settings from your Splunk server
31
Evidence Search
search_query = 'search tag=authentication action=failure app=sshd src_ip='+self.source+'
earliest="'+earliestTimestamp+'" latest="'+latestTimestamp+'" | head '+numEvents+' | eval
eventTime=strftime(_time,"%Y-%m-%d %T %z") | eval eventLog=substr(_raw,timeendpos+1) | eval
eventDetail=eventTime+" "+eventLog | table eventDetail'
32
alert_to_xarf.py
All the X-ARF values are at the top of the script
method getSplunkVersion gets the running Splunk version from
the REST API to help auto adjust the sessionKey
method getSplunkUser gets the username the Alert executed
under from Splunk needed for the evidence search fetch
logging writes with proper timestamp GMT to $SPLUNK_HOME/
var/log/splunk/…
You could use this to make your own highly customized alert
email based on search results
33
Thank You!
Other resources
Splunk IRC ( EFNet #splunk )
Splunk Answers ( http://answers.splunk.com )
Splunk community wiki ( http://wiki.splunk.com )
http://www.georgestarcher.com/
http://blog.splunk.com/
http://www.meetup.com/Splunk/Nashville-TN/
Other “must-see” .conf 2014 presentations
Avoid the SSLippery Slope of Default SSL - Duane Waddle and George Starcher
In Depth With Deployment Server - Dave Shpritz, Aplura
Using Lesser Known Commands in Splunk Search Processing Language (SPL) - Kyle Smith,
The Hershey Company
Masters of IRC - panel talk on the Splunk Community Stage
34
THANK YOU