What's New with Programmability and Scripting Jon Peck
What's New with
Programmability and
Scripting
Jon Peck
Technical Advisor and Principal Software Engineer
Las Vegas, November 2008
Copyright (c) SPSS Inc, 2008
Do you have...
• Repetitive SPSS tasks?
• A need to reduce manual work?
• Problems you can't solve with traditional
syntax?
• A need for statistical procedures not in SPSS
Statistics?
• A need for higher productivity?
• A need to reduce code maintenance time?
See if programmability can help...
Copyright (c) SPSS Inc, 2008
Agenda
• Programmability introduction
• Four examples
– Automating repetitive work:
applySyntaxToFiles
– Integrating programs and scripting:
SPSSINC MODIFY TABLES
– Adding R statistical procedures:
SPSSINC QUANTILE REGRESSION
– Making your own user interface with .NET
The Statistical Explorer
Copyright (c) SPSS Inc, 2008
SPSS Statistics embeds three
programming languages
• Traditional syntax and Basic scripting still
present
• Plug-ins let you extend capabilities using
– Python
–R
– .NET languages (Windows only)
• Free plug-in downloads or get from cd
• SPSS Developer Central web site provides
articles, SPSS-written modules, plug-ins and
user contributions
Copyright (c) SPSS Inc, 2008
My first program
BEGIN PROGRAM PYTHON.
import spss
print "Hello, Las Vegas"
END PROGRAM
• Python or R program code goes in the
normal SPSS Statistics syntax window
Copyright (c) SPSS Inc, 2008
Programmability combines SPSS
Statistics with Python or R
• A program in the input stream can communicate with
SPSS Statistics and control it and use Python or R
facilities and modules (internal mode)
spss.Submit("GET FILE='c:/data/cars.sav'.")
• A Python or .NET application can embed SPSS
Statistics inside itself (external mode)
– User interface does not appear
• There is a lower level C API available in an SDK
Copyright (c) SPSS Inc, 2008
Python and R are open source
software
• Programmability plug-ins are an optional
install
– They are free
– They make possible tapping the work of the
Python and R communities
– Python and R have license agreements
– SPSS has a Freeware license agreement
•
SPSS is not the owner or licensor of the Python or R software. Any user
of Python or R must agree to the terms of the license agreement located
on the Python or R web site. SPSS is not making any statement about
the quality of the Python or R programs. SPSS fully disclaims all liability
associated with your use of the Python or R programs.
Copyright (c) SPSS Inc, 2008
Programmability increases your
power, flexibility, and productivity
• Generalization
– React flexibly to metadata, results, and the environment
• Automation
– Embed program logic in jobs
• Extension
– Add or extend procedures and transformations
– Tap existing R or Python statistical modules
• Integration
– Connect SPSS inputs and outputs to other agents
• And programmability is more fun
Copyright (c) SPSS Inc, 2008
Version 17 has new programmability
and scripting features
V 15
V 16 Additions
V 17 Additions
• Processor
• Processor
• Processor
– SPSS syntax
– Python
programs
– .NET
programs
• Front End
– SaxBasic
scripts
– COM support
– R programs
– Extension
commands
– Dataset class
• Front End
– Basic script
upgrade
– Python
scripts
– Extension
improvements
– Program and
script integration
– R graphics in
Viewer
• Front End
– Custom Dialog
Builder
– New syntax
editor
Copyright (c) SPSS Inc, 2008
Programmability functionality is
fully integrated into SPSS Statistics
• Programs run in the regular syntax stream
• Users can define SPSS Statistics syntax for
program and scripts via Extension
mechanism.
• Users can create dialog boxes and menus
using the Custom Dialog Builder.
– Not just for extensions or programs
• Python and R output appears in the Viewer
– plain text
– pivot tables
– charts
Copyright (c) SPSS Inc, 2008
There are new extension commands
available from Developer Central
• New for version 17
–
–
–
–
–
–
–
–
–
–
–
–
SETSMACRO: syntax for using Variable Sets
SPSSINC APRIORI: association rules (R)
SPSSINC HETCOR: polychoric and serial correlation (R)
SPSSINC MERGE TABLES: merge pivot tables
SPSSINC MODIFY OUTPUT: outline titling and styling
SPSSINC MODIFY TABLES: pivot table styling
SPSSINC QUANTREG: quantile regression (R)
SPSSINC RAKE: adjust weights to control totals
SPSSINC RANFOR/RANPRED: random forests (R)
SPSSINC RASCH: Rasch models (R)
SPSSINC ROBUST REGR: robust regression (R)
SPSSINC TOBIT REGR: Tobit regression (R)
• For version 16 (and 17)
– Seven others
Copyright (c) SPSS Inc, 2008
You can create and share your own
additions to SPSS Statistics
– Write Python or R functions to implement the functionality
• Use input API's to get data to Python or R
• Use output API's to create pivot tables
– For extensions,
• Define the syntax in an xml file
• Use tools in extension.py to receive parsed output and pass to
implementing function
– R extensions can be wrapped in Python code
• Use organization or author name as first word of command
– Use the Custom Dialog Builder to create the interface
• The CDB is not just for extensions
– Test!
– Package and distribute
– Contributions to Developer Central are welcome
• Documentation available at SPSS Developer Central
Copyright (c) SPSS Inc, 2008
Expand the audience by creating SPSS
Statistics syntax and dialog boxes
Copyright (c) SPSS Inc, 2008
Example I
Generalize and automate work
Copyright (c) SPSS Inc, 2008
applySyntaxToFiles
• You have syntax files and need to process
datasets not known in advance every day
• applySyntaxToFiles function applies a syntax file
to each file in input specification
• Optionally saves new sav and output files
• Optionally creates a log
• Can be run in internal or external mode
• Can be integrated into a larger process
Copyright (c) SPSS Inc, 2008
Automating a routine process
• Apply standard processing to an unknown set of files
• Produce processed data and reports
Copyright (c) SPSS Inc, 2008
Use a program to drive processing
begin program.
import spss, spssaux3
spssaux3.applySyntaxToFiles(inputspec="c:/temp/parts/*.sav",
syntax = "c:/userconf2008lasvegas/dailychecks.sps",
outputdatadir = "c:/temp/processed",
outputfiledir = "c:/temp/processed",
logfile ="c:/temp/processed/report.txt")
end program.
• dailychecks.sps could apply data cleaning
rules, modify data, and create reports
• This program could be run daily through
Production Mode or PES job scheduler or used
interactively
Copyright (c) SPSS Inc, 2008
Example II
Use integrated scripting for better table
presentation
Copyright (c) SPSS Inc, 2008
SPSSINC MODIFY TABLES extension
command
• TableLooks provide static formatting for entire areas of
a table
– data cells
– row and column layers
• You want tables with formatting beyond tableLooks
• Many users copy tables to Excel and manually format
them
• Basic and Python Scripting provide programmatic way
to do formatting
• SPSSINC MODIFY TABLES provides syntax for
extensive formatting
– Eliminates need to know scripting
– Uses Extension mechanism for programs and Python
scripting
Copyright (c) SPSS Inc, 2008
Use dynamic highlighting to make
crosstab table easier to read
SPSSINC MODIFY TABLES SUBTYPE='Crosstabulation'
DIMENSION=ROWS SELECT='Std. Residual'
/STYLES TEXTSTYLE=BOLD BACKGROUNDCOLOR=255 0 0
APPLYTO='abs(x) >2'.
Copyright (c) SPSS Inc, 2008
Custom dialog boxes are easy to
create
• Dialog created with
Custom Dialog
Builder
• Generates
extension command
syntax
• Easy to distribute
Copyright (c) SPSS Inc, 2008
Use static formatting to call out
parts of a table
SPSSINC MODIFY TABLES subtype='variables in the equation'
SELECT="B" "Sig."
/STYLES TEXTCOLOR = 0 0 255
BACKGROUNDCOLOR=0 255 0.
Copyright (c) SPSS Inc, 2008
Format CTABLES totals to call them
out
SPSSINC MODIFY TABLES SUBTYPE="Custom Table"
SELECT = "Total" DIMENSION=ROWS
/STYLES BACKGROUNDCOLOR=255 255 88
TEXTSTYLE = BOLD
Copyright (c) SPSS Inc, 2008
Use custom functions for special
effects
SPSSINC MODIFY TABLES SUBTYPE='Report' SELECT="<<ALL>>"
/STYLES APPLYTO=DATACELLS TEXTCOLOR=255 255 255
TEXTSTYLE=BOLD
CUSTOMFUNCTION="customstylefunctions.washColumnsBlue".
def washColumnsBlue(obj, i, j, numrows, numcols, section, more):
mincolor=150.
maxcolor=255.
increment = (maxcolor - mincolor)/(numcols-1)
colorvalue = round(mincolor + increment * j)
obj.SetBackgroundColorAt(i,j, RGB((mincolor, mincolor, colorvalue)))
Copyright (c) SPSS Inc, 2008
It is possible to get carried away with this
Copyright (c) SPSS Inc, 2008
Example III
Add R procedures seamlessly to SPSS
Statistics
Copyright (c) SPSS Inc, 2008
R procedures can be accessed from
SPSS Statistics using the R plug-in
• R
– has many advanced statistical procedures
– is a complex language for programming statistics
– is difficult to learn
• The R plug-in makes it easy to use R packages
– SPSS Statistics datasets and Viewer output can be
processed by R using plug-in
– Graphical, text, and table output appear in the Viewer
– New SPSS datasets can be created from R
– R can communicate with SPSS via 30 API's
– Integration requires writing some R wrapper code
• Plug-in is downloadable from Developer Central
Copyright (c) SPSS Inc, 2008
Quantile regression models
conditional quantiles
• Ordinary regression models conditional mean
• Median regression is 50th quantile
• Estimating quantiles is useful with varying
spread, asymmetries, outliers
• Areas of application include
– empirical finance
• value at risk
• mutual fund investment styles
• credit scoring
– school quality
– demand analysis
– others
Copyright (c) SPSS Inc, 2008
SPSSINC QUANTILE REGRESSION
extension embeds R quantreg
Copyright (c) SPSS Inc, 2008
Plots and pivot tables appear in the
Viewer
Copyright (c) SPSS Inc, 2008
New datasets appear in Data Editor
windows
Copyright (c) SPSS Inc, 2008
Example IV
The .NET plug-in allows an easy .NET
way to do your own user interface for
SPSS Statistics
Copyright (c) SPSS Inc, 2008
The Statistical Explorer uses the
.NET plug-in
Copyright (c) SPSS Inc, 2008
The Statistical Explorer
Copyright (c) SPSS Inc, 2008
Statistics and chart shown depends
on variable measurement level
Copyright (c) SPSS Inc, 2008
Where we have been today
• SPSS Statistics 17 completes
programmability and scripting building blocks
• Unification of programs and scripts
• Custom Dialog Builder
• Extension improvements and new extensions
• R integration and graphics in Viewer
• SPSS Developer Central is your friend
Copyright (c) SPSS Inc, 2008
Questions
?
?
Copyright (c) SPSS Inc, 2008
Programmability increases your
power, flexibility, and productivity
• Generalization and automation
– applySyntaxToFiles
– SPSS MODIFY TABLES
• Extension
– SPSSINC MODIFY TABLES using integrated scripting
– SPSS QUANTREG using R
– Many new extension commands available
• Integration
– applySyntaxToFiles as part of a process
• And it's still more fun
Copyright (c) SPSS Inc, 2008
Contact
Copyright (c) SPSS Inc, 2008
© Copyright 2024