Data-Build Command and TML Reference Guide...18-Datatype-conversion functions todate 217 tointeger 218 toreal 219 tostring 220 19-Functionsforworkingwith dates addcenturies,addcenturiescountbackwards
Post on 07-Sep-2020
10 Views
Preview:
Transcript
Spectrum Miner™
Version 8.0
Data-Build Command and TML Reference Guide
Copyright© 2017 Pitney Bowes Software Inc. All rights reserved. MapInfo andGroup 1 Software are trademarks of Pitney Bowes Software Inc. Allother marks and trademarks are property of their respective holders.
USPS® Notices
Pitney Bowes Inc. holds a non-exclusive license to publish and sellZIP + 4® databases on optical and magnetic media. The followingtrademarks are owned by the United States Postal Service: CASS,CASS Certified, DPV, eLOT, FASTforward, First-Class Mail, IntelligentMail, LACSLink, NCOALink, PAVE, PLANET Code, Postal Service,POSTNET, Post Office, RDI, SuiteLink , United States Postal Service,Standard Mail, United States Post Office, USPS, ZIP Code, and ZIP+ 4. This list is not exhaustive of the trademarks belonging to thePostal Service.
Pitney Bowes Inc. is a non-exclusive licensee of USPS® for NCOALink®
processing.
Prices for Pitney Bowes Software's products, options, and servicesare not established, controlled, or approved by USPS® or UnitedStates Government. When utilizing RDI™ data to determineparcel-shipping costs, the business decision on which parcel deliverycompany to use is not made by the USPS® or United StatesGovernment.
Data Provider and Related Notices
Data Products contained on this media and used within Pitney BowesSoftware applications are protected by various trademarks and byone or more of the following copyrights:© Copyright United States Postal Service. All rights reserved.© 2014 TomTom. All rights reserved. TomTom and the TomTom logoare registered trademarks of TomTom N.V.© 2016 HERE
Fuente: INEGI (Instituto Nacional de Estadística y Geografía)
Based upon electronic data © National Land Survey Sweden.© Copyright United States Census Bureau© Copyright Nova Marketing Group, Inc.
Portions of this program are © Copyright 1993-2007 by Nova Marketing Group Inc. All RightsReserved© Copyright Second Decimal, LLC© Copyright Canada Post Corporation
This CD-ROM contains data from a compilation in which Canada Post Corporation is the copyrightowner.© 2007 Claritas, Inc.
The Geocode Address World data set contains data licensed from the GeoNames Project(www.geonames.org) provided under the Creative Commons Attribution License ("AttributionLicense") located at http://creativecommons.org/licenses/by/3.0/legalcode. Your use of theGeoNames data is governed by the terms of the Attribution License, and any conflict between youragreement with Pitney Bowes Software, Inc. and the Attribution License will be resolved in favor ofthe Attribution License solely as it relates to your use of the GeoNames data.
ICU Notices
Copyright © 1995-2011 International Business Machines Corporation and others.
All rights reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software andassociated documentation files (the "Software"), to deal in the Software without restriction, includingwithout limitation the rights to use, copy, modify, merge, publish, distribute, and/or sell copies of theSoftware, and to permit persons to whom the Software is furnished to do so, provided that the abovecopyright notice(s) and this permission notice appear in all copies of the Software and that both theabove copyright notice(s) and this permission notice appear in supporting documentation.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS ORIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS.IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICEBE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THISSOFTWARE.
Except as contained in this notice, the name of a copyright holder shall not be used in advertisingor otherwise to promote the sale, use or other dealings in this Software without prior writtenauthorization of the copyright holder.
3Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Copyright
C o p y r i g h t
1 - Introduction
Spectrum Miner Overview 10Who should read this book 10
2 - Spectrum Miner data-buildcommands
About Spectrum Miner data-build commands 12Standard command-line options 18The -force command-line option 20The -macro command-line option 21
3 -Managing a SpectrumMinerdata build
qsbuild 24
4 - Commands for importingand exporting data
qsdbaccess 28qsimportdb 29qsdbcreatetable 31qsdbinsert 33qsdbupdate 35qsgenfdd 37qsimportflat 39qsexportflat 41qsimportstat 44
qsexportstat 46qsimportfocus 48How Spectrum Miner imports database types 50Date formats 52
5 - Commands for processingfoci
qssort 59qsderive 60qsmeasure 63qstrack 67qsselect 70qsrenamefields 73qsexportmetadata 76qsimportmetadata 76qsupdate 81
6 - Commands for combiningfoci
About combining foci 83qsjoin 83qsmerge 87
7 - Commands for managingfoci
qscopy 91qslink 92qsmove 93qsremove 93qsarchive 95qsunzip 95
Table of Contents
8 - Commands for producingreports
qssettings 98qsaudit 98qsdescribe 102qsdescribestat 104qshtmlunpack 105qsdtsnapshot, qsscsnapshot 105qsxt 108qsinfo 110qsdescribelicense 111
9 - Commands for buildingmodels
About the Scorecard Wizard 114qsscorecardwizard 115qsdecisiontree 120qsscorecard 122About the Association Rule Wizard 123qsruleminer 125
10 - Commands for workingwith QMML files
qsqmmlview 129qsqmmledit 129qslt 132qsqmml2sas 134qsqmml2sql 134qsqsfmtosas 135
11 - Other commands
qsmapgen 137
12 - Transaction MeasurementLanguage
About Transaction Measurement Language 142TML syntax 142Reserved words in TML 143
13 - TML statements
Field definition: the create statement 146Using aggregation functions and the where and
default clauses 148Splitting aggregations: the by clause 150Evaluating focus statistics: the calculate
statement 153
14 - Aggregation functions
Aggregation functions for measurements andderivations 156
any 159confintlower 160confintupper 160count 161countnonnull/countnonnulls 162countnull/countnulls 163countunique 164countuniquenonnull 164first 165last 166max (one argument) 167mean (one argument) 168median 168min (one argument) 169mode 170moderatio 171percentage 172percentagerate 172segindex 173significance 174stdev 175
5Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
sum (one argument) 175variance 176
15 - Field Derivation Language
About Field Derivation Language 179
16 - FDL syntax
Datatypes 181Expressions 183Conditional expressions 185Variables 187User-defined functions 190Arithmetic operators 193Relational operators 194Logical operators 196Operator precedence 197Built-in functions 198Reserved words in FDL 206
17 - Conditional functions
clamp 209cond 210iff 211ifnull, nvl 212isnull 213isselected 214replace 214
18 - Datatype-conversionfunctions
todate 217tointeger 218toreal 219tostring 220
19 - Functions for workingwithdates
addcenturies, addcenturiescountbackwards 223adddays 224addhours 225addminutes 226addmonths, addmonthscountbackwards 227addseconds 229addweeks 230addyears, addyearscountbackwards 231countcenturies 232countdays 233counthours 234countminutes 235countseconds 236countweeks 237countwholecenturies,
countwholecenturiesbackwards 238countwholedays 240countwholehours 241countwholeminutes 242countwholemonths,
countwholemonthsbackwards 243countwholeseconds 245countwholeweeks 246countwholeyears, countwholeyearsbackwards 247countyears 248day 249dayofweek 250gmt2edt 250hour 251minute 252month 253now 253second 254today 255weekofyear 255year 256
6Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
20 - Functions for workingwithstrings
concat 259endswith 260find 261left 262mid 263right 264soundex 265startswith 266strlen 267strmember 268substitute 269substr 270tolower 272toupper 273trim 273
21 - Regular expressions andassociated functions
Regular expressions 276Basic components of a regular expression 276Regular-expression operators 279match 280replaceall 282replacefirst 283
22 -Mathematical and statisticalfunctions
abs 286ceil 286cos 287exp 288floor 288log 289log10 290logbase 290
max (two or more arguments), maxnonnull 291mean (two or more arguments), meannonnull 292min (two or more arguments), minnonnull 293normalize 294pow 294product, productnonnull 295round 296sgn 297sin 297sqrt 298sum (two or more arguments), sumnonnull 299tan 300
23 - Data-sampling functions
numericTestTrainSplit 302numericTestTrainValidateSplit 302sampleEqualSize 303sampleExactNumber 304sampleExactPercentage 305sampleStratified 306testTrainSplit 307testTrainValidateSplit 308
24 - Random-number functions
About random-number functions in FDL 311rndBinomial 311rndBool 312rndExp 312rndGamma 313rndNormal 314rndPoisson 315rndUniform 315
25 - Return-on-investmentfunctions
ActionROI 318ActionROIAnnualized 319OfferROI 320OfferROIAnnualized 322
7Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
RetentionActionROI 323RetentionActionROIAnnualized 324RetentionOfferROI 326RetentionOfferROIAnnualized 327
26 - Miscellaneous functions
dblookup 330member 332rankOrder, rankOrderApprox 333rankOrderMean, rankOrderApproxMean 335rownum 336
27 - Binnings
bin 338Boolean 339DayFrom, WeekFrom, MonthFrom, YearFrom 340DayMultipleFrom, WeekMultipleFrom,
MonthMultipleFrom, YearMultipleFrom 341DayMultipleNumBins, WeekMultipleNumBins,
MonthMultipleNumBins,YearMultipleNumBins 342
DayMultiplePrePost, WeekMultiplePrePost,MonthMultiplePrePost, YearMultiplePrePost343
DayMultipleTo, WeekMultipleTo, MonthMultipleTo,YearMultipleTo 344
DayMultipleWidth, WeekMultipleWidth,MonthMultipleWidth, YearMultipleWidth 345
DayPrePost, WeekPrePost, MonthPrePost,YearPrePost 346
DayRange, WeekRange, MonthRange,YearRange 347
DayTo, WeekTo, MonthTo, YearTo 348EqualRange 349EqualRangeWidth 350NegativeNonNegative 351PreDuringPost 352PrePost 353Sign 354
28 - XML in Spectrum Miner
XML in Spectrum Miner 356Metadata specification for qsimportmetadata 357Aggregation specification for qsmeasure 366Derivation specification for qsderive 368Selection specification for qsselect 369Crosstab specification for qsxt 371Field name mapping specification for
qsrenamefields 374Decision-tree build specification for
qsdecisiontree 376Scorecard build specification for qsscorecard 379Binning specifications 381Attribute values 390
8Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
1 - Introduction
In this section
Spectrum Miner Overview 10Who should read this book 10
Spectrum Miner Overview
SpectrumMiner is a powerful predictive analytics solution that enables customer insight professionalsand business users alike to achieve a clear picture of their customers for the purpose of greatercustomer understanding, uncovering areas of opportunity, achieving optimal segmentation andpredicting future behavior.
Bridging the gap between standard Business Intelligence tools with a limited scope for exploringdata, and number-crunching solutions which require statistical programmers to build queries andproduce models, Spectrum Customer Analytics is a next-generation solution designed for unparalleledease of use – and fast actionable insight.
The solution utilizes powerful 3D data visualization and rapid modeling automation to uncoverimportant data relationships and deliver propensity scores at the push of a button, boosting predictivemodel accuracy and increasing the speed of analytic results.
Spectrum Miner can be used to predict profit-impacting behaviors and propensities, includingcustomer churn, cross sell and up sell opportunities, campaign planning and segmentation, customersatisfaction and loyalty, and customer lifetime value.
Who should read this book
This book is intended for Spectrum Miner users who are comfortable with the command line andwant to create or modify data builds. Some of it — in particular, the chapter describing the commandsthat are available for managing foci — may also be of interest to Spectrum Miner administrators.
This document is designed for reference and is not a tutorial.
10Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Introduction
2 - Spectrum Minerdata-build commands
In this section
About Spectrum Miner data-build commands 12Standard command-line options 18The -force command-line option 20The -macro command-line option 21
About Spectrum Miner data-build commands
SpectrumMiner data-build commands provide key SpectrumMiner data-build, data-management,and reporting functionality from the command line. While you can enter commands individually toperform one-off tasks, the real advantage of using data-build commands is that you can combinethem in the likes of UNIX shell scripts and Windows batch files to perform complex data builds.
To manage data builds that contain multiple interdependent steps, you can also use SpectrumMiner's Data Build Manager tool or its command-line equivalent, qsbuild. Using a build plan toexpress the sequence of steps that is required saves time and minimizes human error. If you run aregular data-build or deployment process unattended, perhaps overnight, and something goes wrong(such as running out of disk space), you probably want to fix the problem and carry on from whereyou left off, rather than starting all over again. Likewise, if the operations needed for successivebuilds overlap to some extent, you may wish to save on processing time by performing suchoperations once only. Using the Data Build Manager or qsbuild makes it easy to keep track ofdependencies between operations, and remember which operations have already been performed.
Each data-build command requires certain command-line arguments and may acceptcommand-specific optional arguments as well as standard options [see Standard command-lineoptions on page 18]. In the section describing each command is a synopsis of the command, whichuses some special syntactic notation:
• Items in angle brackets are non-literal. For example, where you see <source focus>, you shouldenter the filename of an existing focus.
• Items in square brackets are optional. For example, where you see [-overwrite], you canenter -overwrite or nothing at all.
• Items separated by vertical bars are alternatives: you should choose just one of them. For example,where you see XML | HTML | Full, you can enter XML, HTML, or Full.
• An item followed by an ellipsis can be repeated as required. For example, where you see <field>[, <field> ...], you can enter Income (ignoring the optional item), Income,Gender,Income,Gender,Age, etc.
• Braces {...} are sometimes used to delineate non-optional items. (What counts as an item forthe purposes of alternation and repetition extends to the nearest vertical bar, square bracket, orbrace.) For example, where you see -input <source focus> {,-input <source focus>...}, you can enter -input one.ftr -input two.ftr or -input one.ftr -inputtwo.ftr -input three.ftr but not simply -input one.ftr.
There are data-build commands for importing and exporting data, for processing, joining, andmanaging data in foci, and for reporting on foci and models.
Commands for importing and exporting data
12Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
PurposeCommand
Create, test, or delete a User Database Connection, foraccess to a database.
qsdbaccess
Import data from a database table or from the result of adatabase query.
qsimportdb
Create an empty table in a database based on the fields ina focus.
qsdbcreatetable
Insert records from a focus into a database table.qsdbinsert
Update existing records in a database table using data froma focus.
qsdbupdate
Create a flat-data description file for data in a text file.qsgenfdd
Import data from a flat file.qsimportflat
Export data to a flat file.qsexportflat
Import data from an Excel dataset.qsimportstat
Export data to an Excel file.qsexportstat
Import data (without metadata) from an existing focus.qsimportfocus
Commands for processing foci
PurposeCommand
Sort the records in a focus.qssort
Derive fields in a focus.qsderive
Aggregate records in a focus to produce a new focus.qsmeasure
Derive fields in a focus using state-tracking derivations.qstrack
Add a record selection to a focus.qsselect
13Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
PurposeCommand
Rename fields in a focus.qsrenamefields
Export metadata from a focus.qsexportmetadata
Import metadata from a file to a focus.qsimportmetadata
Copy metadata from one focus to another.qsupdate
Commands for combining foci
PurposeCommand
Join fields from one or more secondary foci to a primaryfocus, matching records using key fields.
qsjoin
Merge records from similar foci to create a new focus,keeping records sorted by key fields.
qsmerge
Commands for managing foci
PurposeCommand
Copy a focus to create a new, independent focus.qscopy
Copy a focus, sharing the underlying data.qslink
Move or rename a focus.qsmove
Delete foci.qsremove
Create an archive file from foci or folders.qsarchive
Extract from an archive file.qsunzip
Commands for producing reports
14Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
PurposeCommand
Set common formatting options for various reports.qssettings
Produce a Profile and Audit of a focus.qsaudit
Display summary information about a focus.qsdescribe
Display summary information about an Excel dataset.qsdescribestat
Unpack an HTML or XML archive produced byqsaudit,qsdtsnapshot, or qsscsnapshot.
qshtmlunpack
Produce a Model Snapshot of a decision tree.qsdtsnapshot
Produce a Model Snapshot of a scorecard.qsscsnapshot
Apply a crosstab specification to a focus to produce a newcrosstab.
qsxt
Display information on a focus's constituent files and itsrelationships to other foci.
qsinfo
Display information about a Spectrum Miner license.qsdescribelicense
Commands for building models
PurposeCommand
Apply a decision tree specification to a focus to produce anew decision tree model report.
qsdecisiontree
Apply a scorecard specification to a focus to produce a newscorecard model report.
qsscorecard
Build a scorecard on a specified focus by using a parametersfile.
qsscorecardwizard
Commands for working with QMML files
15Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
PurposeCommand
Show the contents of a QMML rules file.qsqmmlview
Modify a QMML rules file.qsqmmledit
Transform an FDL, QMML, or generic XML file to a differentformat.
qslt
Convert a QMML file to a SAS file.qsqmml2sas
Convert a QMML file to an SQL file.qsqmml2sql
Convert derivations from a metadata file to a SAS file.qsqsfmtosas
Other commands
PurposeCommand
Execute a build plan.qsbuild
Create a set of Decision Studio maps from the binnings inone or more categorical hierarchy files.
qsmapgen
Because of the potential interdependence of foci, you should not attempt to copy,move, rename, or delete foci using standard operating-system utilities. Instead, youshould always use the commands for managing foci listed above (or SpectrumMiner).
Data-build commands that generate files generally overwrite any existing files ofthe same name. Take care not to overwrite foci that are dependent on other focior that have dependants themselves, as you may render those other foci unusable.
Note: • To use the data-build commands, you must have command-line access to SpectrumMiner.See the Spectrum Miner Administration Guide for information on setting up your commandpath for using data-build commands (or ask your Spectrum Miner administrator).
• Command shells generally require you to quote command-line arguments that containcertain characters, notably spaces and special punctuation (such as single and doublequotation marks, "<," ">," "*," "=," and "#").
Some options take arguments that are comma-separated lists, for example, the -fieldsoption in several commands. Such a list is in fact a single command-line argument, so youshould either quote it or ensure that there are no spaces following the commas (or anywhereelse in the list).
16Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
• When you use the -verbose or -logfile option [see Standard command-line optionson page 18], the output of a data-build command (on screen or in the log file, according tothe option used) includes a line similar to the following:
11:55:21:
{P1/4/4Hm4096Pd2047Pi63Pw27Jm64Um-1Fs991Fh1784Fc0Fp0}
The cryptic string of characters encodes information about the memory and processorresources used by the command; if you are having performance problems with data-buildcommands, this information may help SpectrumMiner Support to resolve matters for you.
• The data-build commands qssort, qsjoin and qsmerge display or record data-specificmessages when you use the -verbose or -logfile option. These can help you tounderstand exactly how the command has acted on your data.
• The exit status returned by each command is zero if it succeeds or a non-zero value if itfails. When a command fails, it always produces an error message of the following form:
*** Error: <message>
Error messages appear on standard error — and also in a log file if you use the -logfileoption [see Standard command-line options on page 18]. Commands may also producewarning messages of the following form:
*** Warning: <message>
Warning messages appear either on standard error or in a log file (but not both).
• If a command fails because a focus is read-only, copy the read-only focus to a new locationusing qslink or qscopy and run the command again.
• When creating UDCs, you use the command qsdbaccess with the -add argument.Because it prompts you for information, it is not appropriate to use this form of the commandin a shell script or batch file.
• By default, data-build commands that read foci use the default subfocus if there is one, andthe top-level ("root") focus otherwise. You can override this behavior for a given commandby using the common -subfocus option.
• Many of the data-build commands take -field , -xfield , -tag and -xtag options tospecify lists of fields. Fields can be specified by using combinations of -field and -xfieldor -tag and -xtag but the field options and tag options cannot be used in combination.
17Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
Standard command-line options
Standalone options: use one of the following options in place of a data-build command's normalarguments:
EffectOption
Show usage text, including a summary of all validcommand-line options other than standard options.
-help
Show usage text, including a summary of all validcommand-line options.
-helpall
Display the version of Spectrum Miner and associatedlicense information.
-version
Other options common to all commands
EffectOption
Use the specified preferences file (see the Spectrum MinerAdministration Guide).
Settings in this file override those in the standardsystem-wide and user-specific preferences files.
-config <preferences file>
On completion of the command, send an e-mail messageto the recipient specified in the Email preference
emailaddress (see theSpectrumMiner AdministrationGuide).
The e-mail message includes an HTML-formatted statussummary, like that generated by the -statusfileoption.
Write detailed progress information complete with operationtimings, and other command-specific information about
-logfile <log file>
execution including any warning and error messages, to thespecified file, overwriting any existing file of that name.Continue to show error messages on standard output. Thisoption implies the -verbose option.
18Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
EffectOption
Use at most the specified amount of memory. The commandwill try to keep within the limit if possible, but will definitely
-memory <number of megabytes>
not exceed it. In cases where a command requires morememory than the maximum amount, the command fails.This option overrides the setting of the focus memorysoft limit preference (see the Spectrum MinerAdministration Guide).
Use the specified number of processors (if available), foroperations that can make use of more than one processor.
-parallel <number of processors>
This overrides the setting of the parallelismpreference (see the SpectrumMiner Administration Guide).
Show simple progress information, as a line of . and +characters.
-progress
Use the settings in the specified file to control numeric anddate formatting in a generated report. This option has an
-settings <settings file>
effect only for the report-generating commands, namelyqsaudit,qsdtsnapshot, andqsscsnapshot.
In the user-specific configuration directory on a SpectrumMiner client PC, Decision Studio creates a file calledsettings.xml when you make changes to settingsusing Edit->View Preferences. You can copy this file tothe server for use with the -settings option. Or youcan create a settings file on the server using the commandqssettings.
By default, the date format is European, the number ofdecimal places is 2, trailing zeros are not stripped, nolocale-specific thousand separator is used, and the timeformat is 24-hour.
Produce an HTML-formatted command-status summary inthe status file.
If you use the -email option, the status e-mail messageincludes a similar summary.
-statusfile <status file>
Show detailed progress information complete with operationtimings, and other command-specific information aboutexecution.
This option overrides the -progress option.
-verbose
19Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
Note: • If you specify the use of more than one processor, using the -parallel option or thecorresponding preference, the available memory, as specified using the -memory optionor the corresponding preference, is shared between the processors.
• Command-specific information about execution may include, for example: the number ofrecords that were input, output, or matched; information about database datatypeconversions; or diagnostic codes of interest to Spectrum Miner Support.
Examples
Display the help text for qsremove:
qsremove -help
Given the preferences file, audit.ini, containing the following:
[Audits and Snapshots]imageheight = 200barcolor = #FF0000
Use these preferences to create a Profile and Audit with images of height 200 pixels and red bars:
qsaudit -config audit.ini -input RetailCustApril.ftr
Use 512 megabytes of memory when creating a new focus, RetailTransAprilSorted.ftr,by sorting the RetailTransApril.ftr focus:
qssort -memory 512 -output RetailTransAprilSorted.ftr-input RetailTransApril.ftr -keys CustomerID
Use four processors (and default memory settings) when creating a new focus,RetailTransMaySorted.ftr, by sorting the RetailTransMay.ftr focus:
qssort -parallel 4 -output RetailTransMaySorted.ftr-input RetailTransMay.ftr -keys CustomerID
The -force command-line option
This option applies to commands that create foci, namely:
qscopy, qsderive, qsimportdb, qsimportflat, qsimportfocus,qsimportmetadata, qsimportstat, qsjoin, qslink, qsmeasure, qsmerge,qsrenamefields, qsselect, qssort, qstrack, qsupdate, qsscorecard, andqsdecisiontree.
20Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
EffectOption
Allow a new focus specified using the -output or -tooption to overwrite an existing focus.
Without the -force option, if the focus specified by-output or -to already exists, the command doesnothing (except issue a warning).
-force
Note: • The -force option should only be used when absolutely necessary, and you shouldensure that you no longer require either the focus that you are overwriting or any foci thatare linked to it [see qsinfo on page 110]. In most cases, it would be better to move theoriginal focus out of the way using qsmove before using a data-build command to createa new focus.
• The -force option only has an effect on explicitly specified output foci. If you are usinga data-build command to modify an existing focus implicitly, there is no need for -force.
Examples Create a new focus RetailTransAprilSorted.ftr, overwriting any existing focusof the same name, containing all the records from the focus RetailTransApril.ftr, sorted bythe field CustomerID:
qssort -force -output RetailTransAprilSorted.ftr-input RetailTransApril.ftr -keys CustomerID
The -macro command-line option
This option applies to the commands:
qsderive, qsmeasure, qstrack, and qsselect
PurposeCommand
Before parsing a TML or corresponding XML input file,replace all occurrences of $<name> with the given value.
You can use this option repeatedly on the same commandline to specify expansions for multiple macros.
-macro <name>=<value>
21Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
PurposeCommand
Perform macro substitution in TML and corresponding XMLinput files, reading <name>=<value> pairs fromsuccessive lines of the macros file.
The macros file may also be in XML format.
-macro @<macros file>
Note: • Names for macros must begin with a letter and contain only letters, digits, and underscore("_").
• The Data Build Manager provides similar, but independent, support for macros.
Examples
Given the selections file, selections-macro.tml, containing the following:
create ${monthName}Trans${year} :=month(PurchaseDate) = ${month} and year(PurchaseDate) = ${year};
Use macro substitutions to select records for April 1999 from the RetailTransApril.ftr focus:
qsselect -macro monthName=April -macro year=1999 -macro month=4-selections selections-macro.tml-input RetailTransApril.ftr -outputRetailTransAprilSelection1.ftr
Or, alternatively, given the macros file, macros.tml, containing the following:
monthName=Aprilyear=1999month=4
Apply these macros to achieve the same result:
qsselect -macro @macros.tml -selections selections-macro.tml-input RetailTransApril.ftr -outputRetailTransAprilSelection2.ftr
See also
XML in Spectrum Miner on page 356
22Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Spectrum Miner data-build commands
3 - Managing aSpectrum Miner databuild
In this section
qsbuild 24
qsbuild
Synopsis qsbuild
[-input <build plan>]
[-targets <target> [, <target> ...]
[-skip <target> [, <target> ...]
[-D<name>=<value>]
[-propertyfile <properties file>] [-describe] [-dryrun]
[-validate full | -validate warn | -validate none]
[-schema <schema file>]
[-lib <class path>]
[-preprocess] [-warn]
Description: execute the build plan qsbuild.qsb in the current directory (unless a different buildplan is specified using the -input option).
Optional arguments
EffectOption
Assign the given value to the named parameter or otherproperty. (Note that there is no space after the -D.)
-D<name>=<value>
Instead of executing the build plan, describe it, listing themain targets and their descriptions.
-describe
Execute the build plan as usual, but display data-buildcommands instead of executing them. (Any tasks that arenot data-build commands are executed as usual.)
-dryrun
Use the specified build plan instead of the fileqsbuild.qsb in the current directory.
-input <build plan>
Use the specified class path as an additional search pathfor Java classes.
-lib <class path>
Validate the build plan, and generate an intermediate buildfile.
-preprocess
24Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Managing a Spectrum Miner data build
EffectOption
Assign values to parameters or other properties, by reading<name>=<value> pairs from successive lines of theproperties file.
-propertyfile <properties file>
Validate the build plan against the specified schema (inRELAX NG syntax) instead of the schema referred to in the
-schema <schema file>
build plan. You can use a URI to specify the location of anon-local schema file.
Do not build the specified targets, even if other targetsdepend on them.
-skip <target> [, <target> ...]
Build the specified targets instead of the default target.-targets <target> [, <target> ...]
Validate the build plan against its schema, executing theplan only if it is valid (the default if you do not specify a-validate option).
-validate fail
Do not validate the build plan against its schema.-validate none
Validate the build plan against its schema; if there are errors,issue warnings, but attempt to execute the plan regardlessof its validity.
-validate warn
Rather than aborting the build plan when a target fails tobuild, issue a warning, and attempt to build other targetsthat do not depend on the failed target.
-warn
Besides these command-line options, qsbuild accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • The build plan describes how to build an analysis dataset or deploy a model. It containsa number of "targets" with dependencies between them, and the sequences of operationsrequired to build the targets. The file uses an XML-based file format. For more informationon the Data Build Manager and the format of build plans, see the Spectrum Miner DataBuild Manager Reference Guide.
• You can also access most of the functionality of qsbuild through the Data Build Manager, available from Spectrum Miner.
Examples Execute a data build using the build plan qsbuild-starter.qsb:
qsbuild -input qsbuild-starter.qsb
25Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Managing a Spectrum Miner data build
Only execute the Clean task within the build plan qsbuild.qsb:
qsbuild -targets Clean -input qsbuild.qsb
Execute the build plan qsbuild.qsb with the Month parameter set to June:
qsbuild -DMonth=June -input qsbuild.qsb
See also
XML in Spectrum Miner on page 356
26Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Managing a Spectrum Miner data build
4 - Commands forimporting andexporting data
In this section
qsdbaccess 28qsimportdb 29qsdbcreatetable 31qsdbinsert 33qsdbupdate 35qsgenfdd 37qsimportflat 39qsexportflat 41qsimportstat 44qsexportstat 46qsimportfocus 48How Spectrum Miner imports database types 50Date formats 52
qsdbaccess
Synopsis qsdbaccess {,-add
| -list
| -test <UDC>
| -delete <UDC>,}[-system]
Description: interactively create a UDC (User Database Connection), list UDCs, test a UDC, ordelete a UDC.
Using -add, qsdbaccess prompts you to enter a DSN, username, and password.
UDCs are used in database-related data-build commands and other parts of Spectrum Miner toconnect to databases. See the Spectrum Miner Administration Guide for further information.
Optional arguments
EffectOption
Apply the operation (add, list, test, or delete) to the systemUDCs instead of your personal UDCs.
When you use -add with -system, qsdbaccessadditionally prompts you for a URL, a class, and a DRL.
-system
You should normally accept the defaults for these threeoptions.
The -system option is intended for use only by systemadministrators.
Besides this command-line option, qsdbaccess accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • A UDC is encrypted: if it contains user-authentication details, only you can use them toaccess the database in question.
• When you create a UDC, qsdbaccess overwrites any existing UDC of the same name(and therefore overwrites the associated user-authentication details).
• The name of a UDC is case-significant (fred and Fred are distinct), even when the DSNis not case-significant to the ODBC driver.
• Using SpectrumMiner, you can access the functionality of qsdbaccess through the EditDatabase Connection dialog box. .
28Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Examples: create a UDC for the user user to access the retail database DSN:
qsdbaccess Please enter required text at the ">" prompt:DSN> retailUser> userPassword>Updating connection user@retailUpdate OKTesting connection user@retailODBC OKJDBC OKConnection OK
Create a system UDC (as the administrator user) for any user to access the retail database DSN:
qsdbaccess -system Please enter required text at the ">" prompt:DSN> retailUser>[Just pressing ENTER will use the system default]URL>Class>DRL>Updating connection dbuser@dbserver:dbUpdate OK
See also
qsdbcreatetable on page 31
qsdbinsert on page 33
qsdbupdate on page 35
qsimportdb on page 29
qsimportdb
Synopsis qsimportdb -udc <UDC>
-table <database table or view>
-fields <column>> [, <column> ...] | -fields @<fields file>]
-xfields <column> [, <column> ...] | -xfields @<fields file>] [-catalog<catalog name>] [-schema <schema name>]-output <destination focus> [-force]qsimportdb -udc <UDC>
-sql <SQL file>
-output <destination focus> [-force]
29Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Description: create the destination focus from the specified database table or view, which is in thedatabase with the given UDC [see qsdbaccess on page 28].
Alternatively, create the focus from the result of an SQL SELECT statement (given in an SQL file).
Optional arguments
EffectOption
Locate the table in this database catalog.-catalog <catalog name>
Create fields in the destination focus corresponding only tothe specified columns in the table.
-fields <column> [, <column> ...]
Create fields in the destination focus corresponding only tocolumns in the table that are listed (one per line) in the fieldsfile.
-fields @<fields file>
See The -force command-line option on page 20.-force
Locate the table in this database schema.-schema <schema name>
Create fields in the destination focus corresponding to allcolumns in the table except the specified columns.
-xfields <column> [, <column> ...]
Create fields in the destination focus corresponding to allcolumns in the table except the columns that are listed (oneper line) in the fields file.
-xfields @<fields file>
Besides these command-line options, qsimportdb accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • If you specify an identifier (for example, a field or table name) as a command-line argument,qsimportdb passes it to the ODBC driver in quoted form, and your database may treatthe identifier as case-significant.
• SQL specified with the -sql option is passed as-is to the database as a single statement;a terminating semicolon is not required (and may be invalid syntax for some databasesystems).
• Using qsimportdb, it is possible to create a focus that does not have Spectrum Miner— compliant field names. To convert the focus into a form that you can use in DecisionStudio or with other data-build commands, you can rename fields using qsrenamefields.
• Using Spectrum Miner, you can access most of the database-query functionality ofqsimportdb through the Import from Query dialog box .
30Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Examples Create a focus,RetailCustApril.ftr, containing all the records from theRetailCustApril table, in the DSN referred to by the user@retail UDC:
qsimportdb -table RetailCustApril-udc user@retail -output RetailCustApril.ftr
Create a focus, RetailTransAprilCA.ftr, containing only records from the RetailCustApriltable for which the value of the PaymentMethod column equals CA:
Given the SQL query file, select-statement.sql, containing the following:
select *from RetailTransAprilwhere (PaymentMethod = 'CA')
Apply this query to create a new focus, RetailTransAprilCA.ftr:
qsimportdb -sql select-statement.sql-udc user@retail -output RetailTransAprilCA.ftr
See also
qsdbcreatetable on page 31
qsdbinsert on page 33
qsdbupdate on page 35
qsdbcreatetable
Synopsis qsdbcreatetable -udc < UDC >
-focus <template focus> [-subfocus <subfocus>]
-table <destination table>
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]
-xfields <field> [, <field> ...] | -xfields @<fields file> | -xtags <tag>[, <tag> ...]]
[-catalog <catalog name>]
[-schema <schema name>]
[-output <SQL output file>]
qsdbcreatetable -udc <UDC>
-sql <SQL file>
31Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Description: use the given template focus to define the field names and types for a new, emptydatabase table with the given table name, in the database with the given UDC [see qsdbaccesson page 28].
Alternatively, create a new table using an SQL CREATE TABLE statement (given in an SQL file).
Optional arguments
EffectOption
Create the table in this database catalog.-catalog <catalog name>
Create table columns corresponding only to the specifiedfields in the focus.
-fields <field> [, <field> ...]
Create table columns corresponding only to fields in thefocus that are listed (one per line) in the fields file.
-fields @<fields file>
Instead of creating a table in the database, create an SQLoutput file containing the appropriate CREATE TABLEstatement.
You can later use the SQL file as input toqsdbcreatetable (with the -sql option).
[-output <SQL output file>]
Create the table in this database schema.-schema <schema name>
Use the specified subfocus of the template focus.-subfocus <subfocus>
Create table columns corresponding only to the fields in thefocus that have the specified tags.
-tags <tag> [, <tag> ...]
Create table columns corresponding to all fields in the focusexcept the specified fields.
-xfields <field> [, <field> ...]
Create table columns corresponding to all fields in the focusexcept the fields that are listed (one per line) in the fieldsfile.
-xfields @<fields file>
Create table columns corresponding to all fields in the focusexcept the fields that have the specified tags.
-xtags <tag> [, <tag> ...]
Besides these command-line options, qsdbcreatetable accepts the options common to alldata-build commands [see Standard command-line options on page 18].
32Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Note: • If you specify an identifier (for example, a field or table name) as a command-line argument,qsdbcreatetable passes it to the ODBC driver in quoted form, and your database maytreat the identifier as case-significant.
• SQL specified with the -sql option is passed as-is to the database as a single statement;a terminating semicolon is not required (and may be invalid syntax for some databasesystems).
Example
Create an empty database table, RETAILAGGREGATIONS, from theRetailAggregationsApril.ftr focus, connecting via the retail UDC:
qsdbcreatetable -focus RetailAggregationsApril.ftr-table RETAILAGGREGATIONS -udc user@retail
See also
qsdbinsert on page 33
qsdbupdate on page 35
qsimportdb on page 29
qsdbinsert
Synopsis qsdbinsert-udc <UDC>
-input <source focus> [-subfocus <subfocus>]
-table <destination table>
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]
-xfields <field> [, <field> ...] | -xfields @<fields file> | -xtags <tag>[, <tag> ...]]
[-catalog <catalog name>]
[-schema <schema name>]
Description: insert records from the source focus into the database table (using an SQL INSERTstatement). Locate the table in the database with the given UDC [see qsdbaccess on page 28].
Optional arguments
33Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
EffectOption
Locate the table in this database catalog.-catalog <catalog name>
Insert data only from the specified fields in the focus.-fields <field> [, <field> ...]
Insert data only from fields in the focus that are listed (oneper line) in the fields file.
-fields @<fields file>
Locate the table in this database schema.-schema <schema name>
Use the specified subfocus of the source focus.-subfocus <subfocus>
Insert data only from fields in the focus that have thespecified tags.
-tags <tag> [, <tag> ...]
Insert data from all fields in the focus except the specifiedfields.
-xfields <field> [, <field> ...]
Insert data from all fields in the focus except the fields thatare listed (one per line) in the fields file.
-xfields @<fields file>
Insert data from all fields in the focus except the fields thathave the specified tags.
-xtags <tag> [, <tag> ...]
Besides these command-line options, qsdbinsert accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • To create an empty table into which you can insert records, use qsdbcreatetable.• The qsdbinsert command inserts additional records into a database table rather than
updating existing records. It doesn't check for duplicate records (or records with duplicatekeys), though your database may be configured to enforce such checking.
• The fields in the focus (or those specified using the -fields or -xfields options) mustcorrespond in name and be type-compatible with the columns in the database table.
• If you specify an identifier (for example, a field or table name) as a command-line argument,qsdbinsert passes it to the ODBC driver in quoted form, and your database may treatthe identifier as case-significant.
Example
34Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Insert data from the RetailAggregationsApril.ftr focus into the RETAILAGGREGATIONSdatabase table:
qsdbinsert -input RetailAggregationsApril.ftr-table RETAILAGGREGATIONS -udc user@retail
See also
qsdbupdate on page 35
qsdbupdate
Synopsis qsdbupdate -udc <UDC>
-input <source focus> [-subfocus <subfocus>]
-table <destination table>
{-keys <key field> [, <key field>] [, <key field>] | -key @<key file>}
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]
-xfields <field> [, <field> ...] | -xfields @<fields file> | -xtags <tag>[, <tag> ...]]
[-catalog <catalog name>]
[-schema <schema name>]
Description: update records in the database table with data from the specified fields in the sourcefocus (using an SQL UPDATE statement). Each record in the focus updates all records with the samekey-field values in the database table. Locate the table in the database with the given UDC [seeqsdbaccess on page 28].
Optional arguments
EffectOption
Locate the table in this database catalog.-catalog <catalog name>
Update the table using only data from the specified fieldsin the focus.
-fields <field> [, <field> ...]
Update the table using only data from fields in the focus thatare listed (one per line) in the fields file.
-fields @<fields file>
35Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
EffectOption
Locate the table in this database schema.-schema <schema name>
Use the specified subfocus of the source focus.-subfocus <subfocus>
Update the table using only data from the fields in the focusthat have the specified tags.
-tags <tag> [, <tag> ...]
Update the table using data from all fields in the focus exceptthe specified fields.
-xfields <field> [, <field> ...]
Update the table using data from all fields in the focus exceptthe fields that are listed (one per line) in the fields file.
-xfields @<fields file>
Update the table using data from all fields in the focus exceptthe fields that have the specified tags.
-xtags <tag> [, <tag> ...]
Besides these command-line options, qsdbupdate accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • The qsdbupdate command does not insert any additional records into a database table.• The fields in the focus (or those specified using the -fields or -xfields options) must
correspond in name and be type-compatible with columns in the database table. However,you don't have to update the values in all columns of the table.
• If you specify an identifier (for example, a field or table name) as a command-line argument,qsdbupdate passes it to the ODBC driver in quoted form, and your database may treatthe identifier as case-significant.
Example
Update only the fields numberPurchases, averagePurchase, and totalPointsRedeemedin the RETAILAGGREGATIONS table using the RetailAggregationsMay.ftr focus:
qsdbupdate -udc user@retail -keys CustomerID-input RetailAggregationsMay.ftr -table RETAILAGGREGATIONS-fields "numberPurchases, averagePurchase,totalPointsRedeemed"
See also
qsdbcreatetable on page 31
qsdbinsert on page 33
36Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
qsgenfdd
Synopsis qsgenfdd -input <flat file>
[-template <template FDD file>]
[-headers]
[-separator <field separator>]
[-null <null marker>]
[-stringmarker <string marker>]
[-datemarker <date marker>]
[-dateformat <date format>] [-defaultday <day number>] [-defaultmonth<month number>] [-comment <table comment>]
Description: create a flat-data description file (.fdd file) for the flat file, by examining a sample ofthe records to work out the file format. Name the flat-data description file using the basename ofthe flat file and create it in the same directory as the flat file.
Optional arguments
EffectOption
Include this table comment in the flat-data description file.-comment <table comment>
Override the automatically-chosen date format [see Dateformats on page 52].
-dateformat <date format>
Override the automatically-chosen date-marker character.-datemarker <date marker>
Include this day number in dates that don't have a daycomponent.
-defaultday <day number>
Include this month number in dates that don't have a monthcomponent.
-defaultmonth <month number>
Include a description of the data as having an initial headersline, which contains field names.
-headers
Override the automatically chosen null-marker string.-null <null marker>
37Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
EffectOption
Override the automatically chosen field-separator character.-separator <field separator>
Override the automatically chosen string-marker character.-stringmarker <string marker>
Base the flat-data description file on this existing templateflat-data description file.
-template <template FDD file>
See Spectrum Miner Online Help for further information on flat-file options.
Besides these command-line options, qsgenfdd accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • If a valid flat-data description file already exists, qsgenfdd does nothing.• In most cases, qsgenfdd automatically chooses appropriate values for the metadata
describing a flat file. However, because it only scans a sample of the data in the flat file —and because there may be ambiguities in the flat-file format — it can guess wrongly. Evenso, you normally only have to override one or two parameters to arrive at a correctinterpretation of the data file.
• You can control the size of samples used in automatic format detection using the Flat FileDatabase preferences data format sample size and data sample size (seeSpectrum Miner Online Help).
• For automatic format detection to work, fields in the flat file must be delimited by a separatorcharacter rather than being distinguished by position (that is, the file cannot be in fixed-widthformat).
• If you use a template FDD file, it must contain a description of all the fields in a tablecomponent.
• Formatting options specified on the command line override global formatting options in atemplate FDD file, which in turn override automatically detected global formatting options.Command-line options do not override field-specific options.
• You can also access the functionality of qsgenfdd through theNew Focuswizard availablefrom Decision Studio, or Spectrum Miner.
Examples
Create a flat-data description file for the RetailCustMay.txt text file:
qsgenfdd -input RetailCustMay.txt
Use the resulting FDD file to create a new focus, RetailCustMay.ftr:
qsimportflat -input RetailCustMay.fdd -output RetailCustMay.ftr
Use a custom data-format string to create a FDD file for the RetailCustMay.txt text file:
qsgenfdd -dateformat "%d-%b-%Y %H:%M:%S" -input RetailCustMay.txt
38Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
See also
qsexportflat on page 41 qsimportflat on page 39
qsimportflat
Synopsis qsimportflat
-input <flat file>
-output <destination focus> [-force]
-encoding <encoding>
-fields <field> [, <field> ...] | -fields @<fields file>]
-fields <field> [, <field> ...] | -fields @<fields file>]
[-maxwarnings <number of warnings>] | [-nowarnings]
[-maxerrors <number of errors>]
Description Create the destination focus from data in the flat file (which may be a text file or aflat-data description file).
Optional arguments
EffectOption
Input file encoding (UTF-8, Shift-JIS, LATIN1, EUC_JP,UTF16-LE, UTF_BE etc.). .
-encoding <encoding>
Create fields in the destination focus corresponding only tothe specified fields in the flat file.
-fields <field> [, <field> ...]
Create fields in the destination focus corresponding only tofields in the flat file that are listed (one per line) in the fieldsfile.
-fields @<fields file>
See The -force command-line option on page 20.-force
Abort the flat-file import if the process generates more thanthe specified number of errors regarding the format of theflat file.
-maxerrors <number of errors>
39Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
EffectOption
Abort the flat-file import if the process generates more thanthe specified number of warnings regarding the format ofthe flat file.
-maxwarnings <number of warnings>
Do not generate any warnings regarding the format of theflat file.
-nowarnings
Create fields in the destination focus corresponding to allfields in the flat file except the specified fields.
-xfields <field> [, <field> ...]
Create fields in the destination focus corresponding to allfields in the flat file except the fields that are listed (one perline) in the fields file.
-xfields @<fields file>
Besides these command-line options, qsimportflat accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • You should specify a flat-data description (FDD) file in preference to a text file as the inputto qsimportflat. You can create an FDD file from a text file using qsgenfdd.
• By default, the Flat File Database preference permit data format errors is set totrue (see SpectrumMiner Onlline Help). This means that all flat-file format errors are treatedas warnings and the -maxerrors option has no effect.
• The -maxerrors, -maxwarnings, and -nowarnings options override the settings ofthe Flat File Database preferences extraction max errors, extraction maxwarnings, and ignore extraction warnings respectively.
• You can also access the functionality of qsimportflat through the New Focus wizardavailable from Decision Studio or Spectrum Miner.
Examples
Create a new focus, RetailTransApril.ftr, containing all the fields from theRetailTransApril.txt flat file:
qsimportflat -input RetailTransApril.txt -output RetailTransApril.ftr
Create a new focus, RetailCustAprilSomeFields.ftr, containing fields CustomerID,StartDate, Age, Postcode and Gender from the RetailCustApril.txt flat file, while alsocreating an FDD file:
qsgenfdd -input RetailCustApril.txtqsimportflat -fields "CustomerID, StartDate, Age, Postcode, Gender"-input RetailCustApril.fdd-output RetailCustAprilSomeFields.ftr
40Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
then create another focus RetailCustAprilOtherFields.ftr, containing all fields exceptCustomerID and Postcode, from the flat file RetailCustApril.txt, using the same FDD file:
qsimportflat -xfields "CustomerID, Postcode"-input RetailCustApril.fdd-output RetailCustAprilOtherFields.ftr
See also
qsexportflat on page 41
qsexportflat
Synopsis qsexportflat
-input <source focus> [-subfocus <subfocus>]
-output <flat file>
[-alwaysquotestrings]
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]
-xfields <field> [, <field> ...] | -xfields @<fields file> | -xtags <tag>[, <tag> ...]]
[-records <FDL expression> | -records @<FDL file>]
[-template <template FDD file>]
[-fdd]
[-headers]
[-separator <field separator>]
[-null <null marker>]
[-stringmarker <string marker>]
[-datemarker <date marker>]
[-dateformat <date format>]
[-defaultday <day number>]
[-defaultmonth <month number>]
[-fixedformat]
Description create the flat file from data in the source focus.
Optional arguments
41Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
EffectOption
Quote all string values in the flat file, rather than only quotingstrings that contain special characters (such as theseparator).
-alwaysquotestrings
Use this format [see Date formats on page 52] for datesin the flat file.
-dateformat <date format>
Use this marker character to introduce dates in the flat file.-datemarker <date marker>
Include this day number in any generated flat-datadescription file (to describe dates formatted without a daycomponent).
-defaultday <day number>
Include this month number in any generated flat-datadescription file (to describe dates formatted without a monthcomponent).
-defaultmonth <month number>
As well as creating the flat file, create a flat-data descriptionfile (.fdd file) to describe the data format.
-fdd
Create fields in the flat file corresponding only to thespecified fields in the source focus.
-fields <field> [, <field> ...]
Create fields in the flat file corresponding only to fields inthe source focus that are listed (one per line) in the fieldsfile.
-fields @<fields file>
Use a fixed number of characters for each field (chosen toaccommodate all possible values of that datatype).
You can use this in conjunction with an empty field separatorto create a flat file in which fields are distinguished byposition instead of using a separator character.
-fixedformat
Include an initial headers line in the flat file, containing fieldnames.
-headers
Use this null-marker string instead of a blank to indicateoccurrences of the null value in the flat file.
-null <null marker>
Create records in the flat file corresponding only to recordsin the source focus for which the numeric FDL expressionis non-zero ("true").
-records <FDL expression>
42Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
EffectOption
Create records in the flat file corresponding only to recordsin the source focus for which the FDL expression in thespecified file is non-zero ("true").
-records @<FDL file>
Use this field-separator character instead of a comma.-separator <field separator>
Use this string-marker character instead of double quotationmarks.
-stringmarker <string marker>
Use the specified subfocus of the source focus.-subfocus <subfocus>
Create fields in the flat file corresponding only to the fieldsin the focus that have the specified tags.
-tags <tag> [, <tag> ...]
Base the flat-file format on this existing template flat-datadescription file.
-template <template FDD file>
Create fields in the flat file corresponding to all fields in thesource focus except the specified fields.
-xfields <field> [, <field> ...]
Create fields in the flat file corresponding to all fields in thesource focus except the fields that are listed (one per line)in the fields file.
-xfields @<fields file>
Create fields in the flat file corresponding to all fields in thesource focus except the fields that have the specified tags.
-xtags <tag> [, <tag> ...]
See Spectrum Miner Online Help for further information on flat-file options.
Besides these command-line options, qsexportflat accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note:
• If you use an FDL selection expression (to export only a subset of the records in the focus), youmust export any fields used in the expression.
• If you specify a flat file with the filename extension .fdd, qsexportflat treats it as a flat-datadescription file (regardless of whether you specify the -fdd option) and creates a correspondingflat file with the filename extension .txt.
• If you use a template FDD file, it must contain a description of all the fields in a table component.• Formatting options specified on the command line override global formatting options in a template
FDD file. Command-line options do not override field-specific options.• If a flat file of the same name as the output file already exists, qsexportflat makes a backup copy
of the existing file (with "~" appended to the filename) rather than simply overwriting it.
43Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
• You can access most of the functionality of qsexportflat through the Export Focus dialog box inDecision Studio , or the Export Text wizard available from Spectrum Miner .
Examples
Create a new flat file, RetailTransApril.csv that contains all the records from theRetailTransApril.ftr focus:
qsexportflat -input RetailTransApril.ftr -output RetailTransApril.csv
Create a new flat file, RetailTransAprilDates.csv that contains only the fields CustomerIDand PurchaseDate from the RetailTransApril.ftr focus, using a particular date format:
qsexportflat -fields "CustomerID, PurchaseDate"-dateformat "%e %h %Y"-input RetailTransApril.ftr -output RetailTransAprilDates.csv
Given the template flat-data description file, template.fdd, containing the following:
separator |null (###)dateformat ((%e %h %Y))skip headertable \CustomerID string(18) \PurchaseDate date \Store integer \Amount real \PaymentMethod string(2) \PointsRedeemed integer
Apply this template to create a flat file, RetailTransAprilDates.txt:
qsexportflat -template template.fdd-input RetailTransApril.ftr -output RetailTransAprilDates.txt
See also
qsgenfdd on page 37 qsimportflat on page 39
qsimportstat
Synopsis qsimportstat
-input <source dataset>
-output <destination focus> [-force]
44Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
[-type <dataset type>]
-fields <field> [, <field> ...] | -fields @<fields file>]
-fields <field> [, <field> ...] | -fields @<fields file>][-metadata<metadata file>]
Description: create the destination focus from Excel datasets.
Optional arguments
EffectOption
Create fields in the destination focus corresponding only tothe specified fields in the source dataset.
-fields <field> [, <field> ...]
Create fields in the destination focus corresponding only tofields in the source dataset that are listed (one per line) inthe fields file.
-fields @<fields file>
See The -force command-line option on page 20.-force
Create the metadata file from the source Excel dataset.-metadata <metadata file>
Interpret the Excel dataset as the specified dataset type,overriding the default interpretation (which depends on thefilename extension).
-type <dataset type>
Create fields in the destination focus corresponding to allfields in the source dataset except the specified fields.
-xfields <field> [, <field> ...]
Create fields in the destination focus corresponding to allfields in the source dataset except the fields that are listed(one per line) in the fields file.
-xfields @<fields file>
Besides these command-line options, qsimportstat accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • If you attempt to import from an Excel dataset and the metadata file already exists, theoperation fails. To avoid this, use the -force argument.
• Using Spectrum Miner, you can access most of the functionality of qsimportstatthrough the Focus Import dialog box.
Examples
45Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Create a new focus RetailTransApril.ftr, containing all the fields from the Excel fileRetailTransApril.xlsx:
qsimportstat -input RetailTransApril.xlsx-output RetailTransApril.ftr
Create a new focus RetailCustAprilSomeFields.ftr, containing the fields CustomerID,StartDate, Age, Postcode, and Gender from the Excel file RetailCustApril.xlsm:
qsimportstat -fields "CustomerID, StartDate, Age, Postcode, Gender"-input RetailCustApril.xlsm -type xls-output RetailCustAprilSomeFields.ftr
See also
qsdescribestat on page 104
qsexportstat on page 46
qsexportstat
Synopsis qsexportstat
-input <source focus> [-subfocus <subfocus>]
-output <destination dataset> [-overwrite]
[-type <dataset type>]
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]
-xfields <field> [, <field> ...] | -xfields @<fields file> | -xtags <tag>[, <tag> ...]] [-records <FDL expression> | -records @<FDL file>]
Description: create the destination dataset from an Excel focus.
Optional arguments
EffectOption
Create fields in the destination dataset corresponding onlyto the specified fields in the source focus.
-fields <field> [, <field> ...]
Create fields in the destination dataset corresponding onlyto fields in the source focus that are listed (one per line) inthe fields file.
-fields @<fields file>
46Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
EffectOption
Allow a new dataset specified using the -output optionto overwrite an existing file.
Without the-overwrite option, if the dataset specifiedusing-output already exists, the command does nothing(except issue a warning).
-overwrite
Create records in the destination dataset correspondingonly to records in the source focus for which the numericFDL expression is non-zero ("true").
-records <FDL expression>
Create records in the destination dataset correspondingonly to records in the source focus for which the FDLexpression in the specified file is non-zero ("true").
-records @<FDL file>
Use the specified subfocus of the source focus.-subfocus <subfocus>
Create fields in the destination dataset corresponding onlyto the fields in the source focus that have the specified tags.
-tags <tag> [, <tag> ...]
Create an Excel dataset of the specified dataset type,overriding the default interpretation (which depends on thefilename extension).
-type <dataset type>
Create fields in the destination focus corresponding to allfields in the source dataset except the specified fields.
-xfields <field> [, <field> ...]
Create fields in the destination focus corresponding to allfields in the source dataset except the fields that are listed(one per line) in the fields file.
-xfields @<fields file>
Create fields in the destination focus corresponding to allfields in the source dataset except the fields that have thespecified tags.
-xtags <tag> [, <tag> ...]
Besides these command-line options, qsexportstat accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Using Spectrum Miner, you can access most of the functionality of qsexportstat throughthe Export for Another Program dialog box.
Note:
47Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Examples Create a new Excel file RetailTransAprilStore800.xlsx, that contains a subsetof records (only purchases from store 800) and only spend fields, from the focusRetailTransApril.ftr:
qsexportstat -input RetailTransApril.ftr-output RetailTransAprilStore800.xlsx-fields "CustomerID, PurchaseDate, Store, Amount"-records "Store = \"800\""
See also
qsdescribestat on page 104 qsimportstat on page 44
qsimportfocus
Synopsis qsimportfocus
-input <source focus> [-subfocus <subfocus>]
-output <destination focus> [-force]
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]
-xfields <field> [, <field>> ...] | -xfields @<fields file> | -xtags <tag>[, <tag> ...]]
[-records <FDL expression> | -records @<FDL file>] [-preservetypes]
Description: create the destination focus from the data in the source focus. Do not copy themetadata.
Optional arguments
EffectOption
Create fields in the destination focus corresponding only tothe specified fields in the source focus.
-fields <field> [, <field> ...]
Create fields in the destination focus corresponding only tofields in the source focus that are listed (one per line) in thefields file.
-fields @<fields file>
See The -force command-line option on page 20.-force
48Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
EffectOption
Preserve legacy datatypes from a focus created using anearlier version of Spectrum Miner.
-preservetypes
Create records in the destination focus corresponding onlyto records in the source focus for which the numeric FDLexpression is non-zero ("true").
-records <FDL expression>
Create records in the destination focus corresponding onlyto records in the source focus for which the FDL expressionin the specified file is non-zero ("true").
-records @<FDL file>
Use the specified subfocus of the source focus.-subfocus <subfocus>
Create fields in the destination focus corresponding only tothe fields in the source focus that have the specified tags.
-tags <tag> [, <tag> ...]
Create fields in the destination focus corresponding to allfields in the source focus except the specified fields.
-xfields <field> [, <field> ...]
Create fields in the destination focus corresponding to allfields in the source focus except the fields that are listed(one per line) in the fields file.
-xfields @<fields file>
Create fields in the destination focus corresponding to allfields in the source focus except the fields that have thespecified tags.
-xtags <tag> [, <tag> ...]
Besides these command-line options, qsimportfocus accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • To create a copy of a focus including metadata, use qscopy.• The destination focus is independent of the source focus, that is, it does not share underlying
data.• If the source focus was created using an earlier version of SpectrumMiner, it may contain
fields with legacy datatypes. By default, qsimportfocus converts such fields into fieldswith the standard integer, real, date, and string datatypes. You can force qsimportfocusto retain the original datatypes in the destination focus, by using the -preservetypesoption.
• Using SpectrumMiner, you can access most of the functionality of qsimportfocus throughthe Export as Focus dialog box .
49Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Examples Create a new focus RetailTransAprilStandardTypes.ftr, that contains all therecords from the focus RetailTransApril.ftr, but converts fields with legacy datatypes to thestandard datatypes:
qsimportfocus -input RetailTransApril.ftr-output RetailTransAprilStandardTypes.ftr
Create a new focus RetailTransAprilStore800.ftr, that contains a subset of records (onlypurchases from store 800) and only spend fields, from the focus RetailTransApril.ftr:
qsimportfocus -input RetailTransApril.ftr-output RetailTransAprilStore800.ftr-fields "CustomerID, PurchaseDate, Store, Amount"-records "Store = \"800\""
Create a new focus, RetailTransApril2Customers.ftr, containing only the transactions forcustomers 20450000000036004 and 20450000000043009:
qsimportfocus -records "strmember(CustomerID, \"20450000000036004\",\"20450000000043009\")"-input RetailTransApril.ftr-output RetailTransApril2Customers.ftr
How Spectrum Miner imports database types
Importing to Spectrum Miner from a database
Spectrum Miner maps the following database types into its datatypes, based on the size of thedatabase type and the Databases preference settings.
SpectrumMiner field type
preservenumeric typeson import
represent largeintegers asstrings
PrecisionScaleDatabase type
stringchar, char2,varchar, varchar2,nchar, nchar2,nvarchar,nvarchar2
50Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
SpectrumMiner field type
preservenumeric typeson import
represent largeintegers asstrings
PrecisionScaleDatabase type
datedate, time,timestamp,datetime,smalldatetime
integerint, integer, tinyint,smallint, byteint, bit
real
string
true
false
false
false
true
bigint
string
real
real
true
false
15
> 15
> 15
< 0decimal, dec,numeric, number
string
real
integer
true
false
false
0
10
> 10 15
> 15
0decimal, dec,numeric, number
real
realfalsefalse
true> 15
> 10
string
string
realfloat, double, real
real
realtrue
false
15
> 15
> 15
> 0decimal, dec,numeric, number
string
real
string
true
false
money
realsmallmoney
Note: • In rows marked with , there may be some loss of precision.• In rows marked with , values too large to store as integer will be converted to Null.
51Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Exporting from Spectrum Miner to a database
The database types that Spectrum Miner exports depend on both the target database and theODBC driver. Spectrum Miner suggests a suitable type for the database that you specify, whilethe ODBC driver interprets that suggestion into one of types in that database.
Date formats
When working with flat files and FDL expressions, you can specify how Spectrum Miner shouldinterpret or export dates and times, using either a standard date format [see Standard date formatson page 52] or a customized date format [see Customized date formats on page 52].
Standard date formats
Date formatName
MM/DD/YYYY:hh:mm:ss, for example, 12/31/2000:13:36:59American
DD/MM/YYYY:hh:mm:ss, for example, 31/12/2000:13:36:59European
YYYY/MM/DD:hh:mm:ss, for example, 2000/12/31:13:36:59YMD
Note: • All three of the standard date formats also apply to dates without a time component andto pure times ("dates" with only a time component). For date-only and time-only dates, thetime and date parts respectively are omitted.
• American, European, and YMD formats also encompass data in the forms MMDDYYYY,DDMMYYYY, and YYYYMMDD respectively (for input only).
Customized date formats
You can specify a customized date format by using special character codes, each beginning witha percent sign %:
52Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Component of formatted dateCode
The percent sign itself: %%%
Matches any single character%?
Matches any number of characters; for example, to ignoretrailing characters, use %Y/%m/%d%*
%*
The day name, in abbreviated form (Mon, Tue, ...,Sun)
%a
The day name, in full (Monday, Tuesday, ...,Sunday)
%A
The month name, in abbreviated form (Jan, Feb,..., Dec)
%b
The month name, in full (January, February,..., December)
%B
(Input) Equivalent to %a %b %e %T %Y
(Output) The date and time in the standard representationfor the current locale
%c
Equivalent to %a %b %e %T %Z %Y%C
The day of the month, using two digits (01, 02, ...,31)
%d
Equivalent to %m/%d/%y%D
The day of the month, with single digits preceded by a space( 1, 2, ..., 31)
%e
Equivalent to %b%h
The hour of the day, using the 24-hour clock (00, 02,..., 23)
%H
53Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Component of formatted dateCode
The hour of the day, using the 12-hour clock (01, 02,..., 12)
%I
The day of the year, using three digits (Julian dates) (001,002, ..., 366)
%j
The month number, using two digits (01, 02, ...,12)
%m
The minute of the hour, using two digits (00, 01, ...,59)
%M
AM or PM, according to the time of day%p
Equivalent to %I:%M:%S %p%r
Equivalent to %H:%M%R
The second of the minute, using two digits (00, 01,..., 61)
%S
Equivalent to %H:%M:%S%T
The date formatted according to the x ordering preference%x
Equivalent to %x%X
The year of the century, using two digits (00, 01,..., 99)
%y
The year, using four digits%Y
The name of the current locale's time zone%Z
Note:
• To use %D or %y, you must also specify either a pivot year or a maximum future year (but not both).
54Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
If you set a pivot equal to a valid year Y, Spectrum Miner interprets two-digit years as yearsbetween Y - 99 and Y. For example, if you set the preference to 2077, SpectrumMiner interpretsthe two-digit years 76, 77, and 78 as 2076, 2077, and 1978 respectively.
If you set a maximum future year equal to a positive value n and the current year is Y, SpectrumMiner interprets all two-digit years as years between Y + n - 99 and Y + n. For example, ifyou set the preference to 50 and the current year is 2002, SpectrumMiner interprets the two-digityears 51, 52, and 53 as 2051, 2052, and 1953 respectively.
• If you set the write in locale-specific format preference, the codes %a, %A, %b, %B, %C, %p,%x, %X, and %Z may produce different results when displaying dates (but not when interpretingthem).
• As well as /, any other character (except for the field separator or null marker) may be used as aseparator within the date.
• For a date format that lacks an explicit day or month component, you can specify a default byusing the "day" or "month" qualifier.
• The special character codes are those of the standard C library function strftime.
Example R–1: Date Formatting Strings
Date format stringDate example
European02/04/2003
27/10/1940
American04/02/2003
10/27/1940
YMD2003/04/02
1940/10/27
%d-%b-%Y02-Apr-2003
27-Oct-1940
((%d %b %Y))02 Apr 2003
27 Oct 1940
((%Y-%m-%d %H:%M))2003-04-02 13:25
1940-10-27 08:23
55Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Date format stringDate example
%Y-%m-%d:%H:%M2003-04-02:13:25
1940-10-27:08:23
((%e %b %Y))2 Apr 2003
27 Oct 1940
((%B %e, %Y))April 2, 2003
((%d/%m/%Y %H:%M))02/04/2003 13:25
27/10/1940 08:23
((%Y-%m-%d %H:%M:%S%*))2003-04-02 13:25:15000000
((%B %d, %Y))April 02, 2003
%e-%b-%Y2-Apr-2003
27-Oct-1940
(European pivot=2010)02/04/03
27/10/40
(American pivot=2010)04/02/03
10/27/40
(YMD pivot=2010)03/04/02
40/10/27
(%d-%b-%y pivot=2010)02-Apr-03
27-Oct-40
((%B %d, %y) pivot=2010)April 02, 03
(%e-%b-%y pivot=2010)2-Apr-03
56Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
Date format stringDate example
27-Oct-03
((%B %e, %y) pivot=2010)April 2, 03
((%I:%M %p))01:25 PM
08:23 AM
%H:%M13:25
08:23
57Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for importing and exporting data
5 - Commands forprocessing foci
In this section
qssort 59qsderive 60qsmeasure 63qstrack 67qsselect 70qsrenamefields 73qsexportmetadata 76qsimportmetadata 76qsupdate 81
qssort
Synopsis qssort
-input <source focus>> [-subfocus <subfocus>]
-output <destination focus> [-force]
{-keys <key field> [, <key field>] [, <key field>]| -key @<key file>}}qssort-input <source focus> [-subfocus <subfocus>]
-check
{-keys <key field> [, <key field>] [, <key field>]| -key @<key file>}}
Description: sort the records of the source focus by the key fields, to produce the destination focus.
Alternatively, check that the source focus is already sorted.
Optional arguments
EffectOption
See The -force command-line option on page 20.-force
Use the specified subfocus of the source focus.-subfocus <subfocus>
Besides this command-line option, qssort accepts the options common to all data-build commands[see Standard command-line options on page 18].
Note: • You can use at most three key fields with qssort.• Spectrum Miner collation order, as used by qssort, may differ in detail from the collation
order used by databases or third-party file-sorting utilities. Where it is important for a focusto be sorted, you should check the sort order — and if necessary re-sort — using qssort.
• It is, in general, much faster to check the sort order of a focus than to sort it.• It is, in general, slightly faster to sort a focus if it is already nearly sorted.• Even when the focus (or subfocus) contains only a selection of records, the destination
focus contains all the records from the source focus.• If you use the -verbose option, qssort reports the number of unique key values, and the
number of records containing the most frequent duplicate key value.• The command qssort requires temporary disk space comparable to the size of the source
focus as well as space for the destination focus.• The destination focus is independent of the source focus, that is, it does not share underlying
data.
59Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
• Using SpectrumMiner, you can access most of the functionality of qssort through the SortFocus wizard .
Examples Check that the RetailCustApril.ftr focus is sorted by the CustomerID field:
qssort -check -input RetailCustApril.ftr -keys CustomerID
Create a new focus, RetailTransAprilSorted.ftr, containing all the records from theRetailTransApril.ftr focus, sorted by the CustomerID field:
qssort -output RetailTransAprilSorted.ftr-input RetailTransApril.ftr -keys CustomerID
qsderive
Synopsis qsderive -derivations <derivations file>
-input <source focus> [-subfocus <subfocus>]
[-macro <name>=<value>[-macro <name>=<value> ...] | -macro @<name>=<macrofile>]
[-output <destination focus>] [-force]
-fields <field> [, <field> ...] | -fields @<fields file>]
-fields <field> [, <field> ...] | -fields @<fields file>]
[-random <integer seed>]
[-warn] [-savexml]
Description: copy all fields (by default) from the source focus to the destination focus; derive fieldsin the destination focus according to the field definitions in the derivations file, which contains createstatements or corresponding XML representations. By default, append the derived fields to thesource focus.
Optional arguments
PurposeCommand
EffectOption
Derive only the specified fields.-fields <field> [, <field>> ...]
60Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
PurposeCommand
Derive only fields that are listed (one per line) in the fieldsfile.
-fields @<fields file>>
See The -force command-line option on page 20.-force
See The -macro command-line option on page 21.-macro <name>=<value>
See The -macro command-line option on page 21.-macro @<macros file>
Instead of appending derived fields to the source focus,copy the source focus to the destination focus and appendthe derived fields to the destination focus.
-output <destination focus>
Use the integer seed, instead of 0, for any FDLrandom-number functions occurring in derivationexpressions.
-random <integer seed>
In addition to deriving one or more fields, write the derivationexpression(s) in TML and XML formats to files
-savexml
<output>.tml and <output>.xml, in the samedirectory as the destination focus, where <output> isthe basename of the destination focus.
Use the specified subfocus of the source focus.-subfocus <subfocus>
If a field derivation fails, display a warning and continue toderive other fields (instead of stopping).
-warn
Derive all fields in the derivations file with the exception ofthe specified fields.
-xfields <field>> [, <field> ...]
Derive all fields in the derivations file with the exception ofthe fields that are listed (one per line) in the fields file.
-xfields @<fields file>
Besides these command-line options, qsderive accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • The order in which derived fields appear in the destination focus is the order in which theyappear in the derivations file.
• If you specify a destination focus, it shares underlying data with the source focus.• XML for qsderive takes the form of a <derivations> element, containing <field>
elements, each of which must in turn contain an FDL expression in an <fdl> element.
61Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
Each <fdl> element optionally includes an integer seed attribute to specify therandom-number seed for that FDL expression.
• You can use the XML file saved by qsderive using the -savexml option as a derivationsfile.
• Using Spectrum Miner, you can access most of the functionality of qsderive throughthe Derive Fields wizard .
Example
Given the derivations file, derivations-purchases.tml, containing the following:
create TransMonth := month(PurchaseDate);create CashRecd := if PaymentMethod = "CA" then Amount else 0;
apply these derivations to the RetailTransApril.ftr focus, to create a new focus,RetailTransAprilDeriv.ftr, containing all the fields of the RetailTransApril.ftr focus,plus two derived fields:
qsderive -derivations derivations-purchases.tml-input RetailTransApril.ftr -output RetailTransAprilDeriv.ftr
Apply all but the first of these derivations to the focus RetailTransApril.ftr, to create a newfocus RetailTransAprilDerivAllButFirstOne.ftr, containing all the fields of theRetailTransApril.ftr focus plus all but the first derived field:
qsderive -xfields TransMonth -derivations derivations-purchases.tml-input RetailTransApril.ftr-output RetailTransAprilDerivAllButFirstOne.ftr
Create a new focus,RetailCustAprilScored.ftr, containing all the fields of theRetailCustApril.ftr focus, plus the score fields created in Decision Studio's Scorecard Builderand saved as QMML file, derivations-score.qmml:
qsderive -derivations derivations-score.qmml-input RetailCustApril.ftr-output RetailCustAprilScored.ftr
Given the XML derivations file derivations-purchases.xml containing the following:
<?xml version="1.0" encoding="UTF-8"?><derivations xmlns="http://www.quadstone.com/xml"><field name="TransMonth"><fdl>month(PurchaseDate)</fdl></field><field name="CashRecd"><fdl>if PaymentMethod = "CA" then Amount else 0</fdl></field><field name="Random"><fdl seed="123">rndUniform()</fdl>
62Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
</field></derivations>
apply these derivations to the focus RetailTransApril.ftr, to create a new focusRetailTransAprilDeriv.ftr, containing all the fields of the focus RetailTransApril.ftrplus two derived fields:
qsderive -derivations derivations-purchases.xml-input RetailTransApril.ftr -output RetailTransAprilDeriv.ftr
See also
Derivation specification for qsderive on page 368
qstrack on page 67
qsmeasure
Synopsis qsmeasure
-aggregations <aggregations file> [, <aggregations file> ...]
-input <source focus>
-output <destination focus> [-force]
{-keys <key field> [, <key field>] [, <key field>] | -key @<key file>}
[-statistics <statistics file> [, <statistics file> ...]]
[-library <FDL functions file> [, <FDL functions file> ...]]
[-macro <name>=<value> [-macro <name>=<value> ...] | -macro @<name>=<macrofile>]
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]
-xfields <field> [, <field> ...] | -xfields @<fields file> | -xtags <tag>[, <tag> ...]]
Description: aggregate records in the source focus, to produce fields in the destination focusaccording to the field definitions in the aggregations files. The aggregations files contain createstatements, or corresponding XML representations. Use the key fields to identify groups of recordsfor aggregation [see Using aggregation functions and the where and default clauses on page148].
Optional arguments
63Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
EffectOption
Consider only the specified fields from the source focus (tooptimize performance of the command by avoidingconsideration of unused fields).
-fields <field> [, <field> ...]
Consider only fields from the source focus that are listed(one per line) in the fields file.
-fields @<fields file>
See The -force command-line option on page 20.-force
Include FDL function definitions [see User-definedfunctions on page 190].
Expressions in the aggregations files may involve thesefunctions.
-library <FDL functions file> [,<FDL functions file> ...]
See The -macro command-line option on page 21.-macro <name>=<value>
See The -macro command-line option on page 21.-macro @<macros file>
Compute statistics for the input focus according to thedefinitions in the statistics files, which containcalculate
-statistics <statistics file> [,<statistics file> ...]
statements [see Evaluating focus statistics: the calculatestatement on page 153] or corresponding XMLrepresentations.
Expressions in the aggregations files may refer to thesestatistics.
Consider only the fields from the source focus that have thespecified tags (to optimize performance of the command byavoiding consideration of unused fields).
-tags <tag> [, <tag> ...]
Consider all fields from the source focus, except thespecified fields (to optimize performance of the commandby avoiding consideration of unused fields).
-xfields <field> [, <field> ...]
Consider all fields from the source focus, except the fieldsthat are listed (one per line) in the fields file.
-xfields @<fields file>
Consider all fields from the source focus, except the fieldsthat have the specified tag (to optimize performance of thecommand by avoiding consideration of unused fields).
-xtags <>tag> [, <tag>> ...]
64Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
Besides these command-line options, qsmeasure accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • The source focus must be sorted by the key fields [see qssort on page 59]. However,qsmeasure does not check that this is the case, and does not issue a warning or error ifthe focus is not sorted correctly.
• The destination focus contains key fields corresponding to the key fields in the source focus,followed by the aggregated fields (in the same order in which they appear in the aggregationsfile), followed by any derived fields (again, in the same order in which they appear in theaggregations file). To force an aggregated field to appear after a derived field in thedestination focus, you can use a temporary statement to create the aggregated field, andthen derive a field from that.
• The destination focus is independent of the source focus, that is, it does not share underlyingdata.
• To compute aggregations involving the states between transactions rather than just thevalues in the transaction records themselves, derive suitable intermediate fields usingqstrack before using qsmeasure.
• XML for qsmeasure takes the form of an <aggregations> element, containing <fieldcontext="aggregation"> elements, each of which must in turn contain a TMLexpression in an <fdl> element. You can use a <by> element to specify groupings, anda <where> element to filter the input records.
• Using Spectrum Miner, you can access most of the functionality of qsmeasure throughthe Aggregate Records wizard .
ExamplesGiven the aggregations file, aggregations-purchases.tml, containing the following:
create numberPurchases := count();create averagePurchase := mean(Amount);create totalPointsRedeemed := sum(PointsRedeemed);create averagePointsPerPurchase :=totalPointsRedeemed / numberPurchases;
Apply these aggregations and the derivation to the RetailTransAprilSorted.ftr focus tocreate the RetailAggregationsApril.ftr focus containing the fields CustomerID,numberPurchases, averagePurchase, totalPointsRedeemed andaveragePointsPerPurchase:
qsmeasure -aggregations aggregations-purchases.tml-input RetailTransAprilSorted.ftr-output RetailAggregationsApril.ftr-keys CustomerID
Given the FDL functions file, fdl-functions-storeSplit.fdl, containing the following:
function StoreSplitFunction( Store )[element_names = "0,800,600,700,300,400,100,500,900,Other"
65Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
]{caseStore = "0000" : 1;Store = "0800" : 2;Store = "0600" : 3;Store = "0700" : 4;Store = "0300" : 5;Store = "0400" : 6;Store = "0100" : 7;Store = "0500" : 8;Store = "0900" : 9;default : 10;}
Given the aggregations file, aggregations-mostCommon.tml, containing the following:
create mostCommonStore := mode(Store);create averageSpendInStore_ := mean(Amount)by StoreSplitFunction(Store);
Apply these aggregations to the RetailTransAprilSorted.ftr focus, to create theRetailStoreSplitsApril.ftr focus, containing the fields CustomerID, mostCommonStore,averageSpendInStore_0, averageSpendInStore_800, averageSpendInStore_600,averageSpendInStore_700, averageSpendInStore_300, averageSpendInStore_400,averageSpendInStore_100, averageSpendInStore_500, averageSpendInStore_900and averageSpendInStore_Other:
qsmeasure -library fdl-functions-storeSplit.fdl-aggregations aggregations-mostCommon.tml-input RetailTransAprilSorted.ftr-output RetailStoreSplitsApril.ftr -keys CustomerID
Alternatively, given the XML aggregations file aggregations-mostCommon.xml, containing thefollowing:
<?xml version="1.0" encoding="UTF-8"?><aggregations xmlns="http://www.quadstone.com/xml">
<field name="mostCommonStore" context="aggregation"><fdl>mode(Store)</fdl>
</field><field name="averageSpendInStore_" context="aggregation">
<fdl>mean(Amount)</fdl><by>StoreSplitFunction(Store)</by>
</field></aggregations>
66Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
apply these aggregations to the focus RetailTransAprilSorted.ftr, to create the focusRetailStoreSplitsApril.ftr, containing the same fields as the example above:
qsmeasure -library fdl-functions-storeSplit.fdl-aggregations aggregations-mostCommon.xml-input RetailTransAprilSorted.ftr-output RetailStoreSplitsApril.ftr -keys CustomerID
Given the statistics file, statistics-amount.tml, containing the following:
calculate averageAmount := mean(Amount);
and given the aggregations file, aggregations-statistic.tml, containing the following:
create averageSpend := mean(Amount);create bigSpender := averageSpend > STATISTIC.averageAmount;
apply this aggregation and derivation to the RetailTransAprilSorted.ftr focus to create theRetailRankApril1.ftr focus, containing the fields CustomerID, averageSpend, andbigSpender:
qsmeasure -statistics statistics-amount.tml-aggregations aggregations-statistic.tml-input RetailTransAprilSorted.ftr-output RetailRankApril1.ftr-keys CustomerID
Alternatively, speed up the aggregation by importing only the fields CustomerID and Amount fromthe RetailTransAprilSorted.ftr focus:
qsmeasure -fields "CustomerID, Amount"-statistics statistics-amount.tml-aggregations aggregations-statistic.tml-input RetailTransAprilSorted.ftr-output RetailRankApril2.ftr-keys CustomerID
See also
Aggregation specification for qsmeasure on page 366
qstrack
Synopsis qstrack
67Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
-trackers <trackers file> [, <trackers file> ...] -input <source focus>-output <destination focus> [-force]-key <key field> | -key @<keyfile>[-statistics <statistics file> [, <statistics file> ...]] [-library<FDL functions file> [, <FDL functions file> ...]]
[-macro <name>=<value> [-macro <name>=<value> ...] | -macro @<name>=<macrofile>]
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]-xfields <field> [, <field> ...] | -xfields @<fields file> |-xtags <tag> [, <tag> ...]]
Description: copy all fields (by default) from the source focus to the destination focus; derive fieldsin the destination focus according to the field definitions in the trackers file, which contains createstatements or corresponding XML representations. These definitions typically involve state variables[seeVariables on page 187]. Use the key field to identify groups of records for tracking state: qstrackresets all state variables at the start of a new group.
Optional arguments
EffectOption
Consider only the specified fields from the source focus (tooptimize performance of the command by avoidingconsideration of unused fields).
-fields <field> [, <field> ...]
Consider only fields from the source focus that are listed(one per line) in the fields file.
-fields @<fields file>
See The -force command-line option on page 20.-force
Include FDL function definitions [see User-definedfunctions on page 190].
Expressions in the trackers files may involve these functions.
-library <FDL functions file> [,<FDL functions file> ...]
See The -macro command-line option on page 21.-macro <name>=<value>
See The -macro command-line option on page 21.-macro @<macros file>
Compute statistics for the input focus according to thedefinitions in the statistics files, which containcalculate
-statistics <statistics file> [,<statistics file> ...]
statements [see Evaluating focus statistics: the calculatestatement on page 153] or corresponding XMLrepresentations.
Expressions in the trackers files may refer to these statistics.
68Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
EffectOption
Consider only the fields from the source focus that have thespecified tags (to optimize performance of the command byavoiding consideration of unused fields).
-tags <tag> [, <tag> ...]
Consider all fields from the source focus, except thespecified fields (to optimize performance of the commandby avoiding consideration of unused fields).
-xfields <field> [, <field> ...]
Consider all fields from the source focus, except the fieldsthat are listed (one per line) in the fields file.
-xfields @<fields file>
Consider all fields from the source focus, except the fieldsthat have the specified tags (to optimize performance of thecommand by avoiding consideration of unused fields).
-xtags <tag> [, <tag> ...]
Besides these command-line options, qstrack accepts the options common to all data-build commands[see Standard command-line options on page 18].
Note: • The order in which derived fields appear in the destination focus is the order in which theyappear in the trackers file.
• The source focus must be sorted by the key field [see qssort on page 59]. However,qstrack does not check that this is the case, and does not issue a warning or error if thefocus is not sorted correctly.
• The destination focus is independent of the source focus, that is, it does not share underlyingdata.
• XML for qstrack takes the form of a <trackers> element, containing <fieldcontext="tracker"> elements, each of which must in turn contain an FDL expressionin an <fdl> element.
Examples
Given the trackers file, trackers-monthlySpend.tml, containing the following:
create monthlySpend := (state currentMonth := null;state currentYear := null;state monthSpend := 0;if (currentMonth = month(PurchaseDate) andcurrentYear = year(PurchaseDate))then (monthSpend := monthSpend + Amount)else (monthSpend := Amount);currentMonth := month(PurchaseDate);currentYear := year(PurchaseDate);monthSpend;);
69Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
apply these derivations to the focus RetailTransAprilSorted.ftr to create a new focusRetailTransAprilRunningSpend.ftr:
qstrack -trackers trackers-monthlySpend.tml -key CustomerID-input RetailTransAprilSorted.ftr-output RetailTransAprilRunningSpend.ftr
Alternatively, given the XML trackers file trackers-monthlySpend.xml, containing the following:
<?xml version="1.0" encoding="UTF-8"?><trackers xmlns="http://www.quadstone.com/xml">
<field name="monthlySpend" context="tracker"><fdl>
state currentMonth := null;state currentYear := null;state monthSpend := null;if (currentMonth = month(PurchaseDate) and currentYear =
year(PurchaseDate))then (monthSpend := monthSpend + Amount)else (monthSpend := Amount);currentMonth := month(PurchaseDate);currentYear := year(PurchaseDate);path>monthSpend;h
</fdl></field>
</trackers>
apply these derivations to the focus RetailTransAprilSorted.ftr to create a new focusRetailTransAprilRunningSpend.ftr:
qstrack -trackers trackers-monthlySpend.xml -key CustomerID-input RetailTransAprilSorted.ftr-output RetailTransAprilRunningSpend.ftr
See also
XML in Spectrum Miner on page 356
qsderive on page 60
qsselect
Synopsis qsselect -selections <selections file>
-input <source focus> [-subfocus <subfocus>]
[-macro <name>=< value>[-macro <name>=<value> ...] | -macro @<name>=<macrofile>]
70Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
[-output <destination focus>>] [-force][-random <integer seed>] [-savexml][-selection <field>]
Description: copy all fields (by default) from the source focus to the destination focus; derive anumeric field in the destination focus according to the first field definition (by default) in the selectionsfile, which is a file containing create statements or a corresponding XML file. By default, appendthe derived field to the source focus.
Apply a record selection to the new field, selecting only records with the value 1.
Optional arguments
EffectOption
See The -force command-line option on page 20.-force
See The -macro command-line option on page 21.-macro <name>=<value>
See The -macro command-line option on page 21.-macro @<macros file>
Instead of appending derived fields to the source focus,copy the source focus to the destination focus and appendthe derived fields to the destination focus.
-output <destination focus>
Use the integer seed, instead of 0, for any FDLrandom-number functions occurring in derivationexpressions.
-random <integer seed>
In addition to deriving one or more fields and applying aselection, write the derivation expression(s) from the
-savexml
selections file in TML and XML formats to files<output>.tml and <output>.xml, in the samedirectory as the destination focus, where <output> is thebasename of the destination focus.
Instead of using the first field definition in the selections file,use the definition of the specified field.
-selection <field>
Use the specified subfocus of the source focus.-subfocus <subfocus>
Besides these command-line options, qsselect accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • Typically, the derivation expressions in the selections file are applications of relational orlogical operators, which produce the values 1 for "true" and 0 for "false." In such cases,qsselect selects those records for which the logical expression is true.
71Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
• If you specify a destination focus, it shares underlying data with the source focus.• XML for qsselect takes the form of a <selections> element, containing <fieldcontext="selection"> elements, each of which must in turn contain an FDL expressionin an <fdl> element.
• You can use the XML file saved by qsselect using the -savexml option as a selectionsfile.
Examples,
Given the selections file, selections-sixmonth.tml, containing the following:
create lastSixMonths :=countwholemonths(PurchaseDate, #1999/07/01) < 6;
apply this selection to the RetailTransApril.ftr focus:
qsselect -selections selections-sixmonth.tml-input RetailTransApril.ftr-output RetailTransApril6Months.ftr
Given the selections file, selections-random.tml, containing the following:
create random := rndUniform() < 0.1;
apply this selection to select 10% of the records in the RetailTransApril.ftr focus, with aknown seed (12345678), to allow the same series of records to be selected on subsequentapplications:
qsselect -selections selections-random.tml -random 12345678-input RetailTransApril.ftr-output RetailTransAprilSampleSelect.ftr
Alternatively, given the XML selections file, selections-random.xml, containing the following:
<?xml version="1.0" encoding="UTF-8"?><selections xmlns="http://www.quadstone.com/xml"?>
<field name="random" context="selection"><fdl seed="12345678">rndUniform() < 0.1</fdl>
</field></selections>
Apply this selection to select the same 10% of the records in the RetailTransApril.ftr focus,using the same seed:
qsselect -selections selections-random.xml-input RetailTransApril.ftr-output RetailTransAprilSampleSelect.ftr
See also
72Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
Selection specification for qsselect on page 369
qsrenamefields
Synopsis qsrenamefields
-input <source focus>
{-map <old name>=<new name> | -map @<input mapping file>} [{-map <oldname>=<new name> | -map @<input mapping file>} ...]
[-output <destination focus>] [-force]
[-invert]
[-mapping <output mapping file>]
qsrenamefields
-input <source focus>
[-map QSCompliant]
[-output <destination focus>] [-force]
[-invert]
[-mapping <output mapping file>]
Description: rename fields in the source focus, applying <old name>=<new name> mappingsand mappings in input mapping files.
Alternatively, rename fields in the source focus to unique, SpectrumMiner— compliant equivalents,using a built-in algorithm .
Optional arguments
PurposeCommand
See The -force command-line option on page 20.-force
Reverse the sense of the field-name mappings.-invert
Create an XML mapping file, describing the actual mappingused from old to new field names.
You can use this file as an input mapping file forqsrenamefields. In conjunction with the -invert option,
-mapping <output mapping file>
you can then use a reverse field-name mapping even in the
73Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
PurposeCommand
case where the original field-name mapping was algorithmic(and not necessarily one-to-one).
Instead of directly renaming the fields in the source focus,copy the source focus to the destination focus and renamethe fields in the destination focus.
-output <destination focus>
Besides these command-line options, qsrenamefields accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • Omitting the -map argument altogether has the same effect as specifying -mapQSCompliant.
• An input mapping file is either in XML format or contains <old name>=<new name> pairson successive lines of the file.
• If you use more than one -map argument, qsrenamefields applies the field-name mappingsin the order that they appear on the command line.
• If you try to rename fields so that more than one field has the same name, qsrenamefieldswarns you and does nothing.
• If you specify a destination focus, it shares underlying data with the source focus.• You cannot rename a field more than once in a single invocation of the command.• If the source focus includes subfoci, qsrenamefields renames fields across the entire
hierarchy of subfoci.• XML for qsrenamefields takes the form of a <mappingset> element containing <map>
elements, each of which must contain a <name> and <alias> element.• Given a focus produced by qsimportdb, which may include field names that are notSpectrum Miner — compliant, you can use qsrenamefields (with -map QSCompliant)to convert the focus into a form that you can use in Decision Studio or with other data-buildcommands.
• You can access some of the functionality of qsrenamefields through the Table Viewer inDecision Studio , or the Rename Fields dialog box available from Spectrum Miner .
Examples
Create a focus, RetailCustAprilRenamed1.ftr, with the StartDate field renamed asInitialized, Age renamed as CurrentAge, and Gender renamed as Sex:
qsrenamefields -map StartDate=Initialized-map Age=CurrentAge -map Gender=Sex-input RetailCustApril.ftr -output RetailCustAprilRenamed1.ftr
74Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
Alternatively, given the input mapping file,mapping-file.txt, with the following contents:
StartDate=InitializedAge=CurrentAgeGender=Sex
apply this mapping file to create the focus RetailCustAprilRenamed2.ftr, with field StartDaterenamed as Initialized, Age renamed as CurrentAge, and Gender renamed as Sex:
qsrenamefields -map @mapping-file.txt-input RetailCustApril.ftr -output RetailCustAprilRenamed2.ftr
or, alternatively, given the XML input mapping file mapping-file.xml, with the following contents:
<?xml version="1.0" encoding="UTF-8"?><mappingset xmlns="http://www.quadstone.com/xml">
<map><name>StartDate</name><alias>Initialized</alias>
</map><map>
<name>Age</name><alias>CurrentAge</alias>
</map><map>
<name>Gender</name><alias>Sex</alias>
</map></mappingset>
apply this mapping file to create the focus RetailCustAprilRenamed2.ftr, with field StartDaterenamed as Initialized, Age renamed as CurrentAge, and Gender renamed as Sex:
qsrenamefields -map @mapping-file.xml-input RetailCustApril.ftr -output RetailCustAprilRenamed2.ftr
Create a focus,RetailCustAprilQSCompliant.ftr, with Spectrum Miner-compliant field names:
qsrenamefields-input RetailCustApril.ftr-output RetailCustAprilQSCompliant.ftr
See also
Field name mapping specification for qsrenamefields on page 374
75Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
qsexportmetadata
Synopsis qsexportmetadata -input <source focus> [-output <metadata file>]
Description: export metadata from the source focus, including focus history, binnings, comments,derivations, interpretations, record selections, subfocus structure, and default subfocus. By default,write the information to standard output.
Optional arguments
EffectOption
Create this metadata file instead of writing to standardoutput.
-output <metadata file>
Besides this command-line option, qsexportmetadata accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • The metadata output by qsexportmetadata is in an XML-based file format [see Metadataspecification for qsimportmetadata on page 357].
• Using SpectrumMiner, you can access the functionality of qsexportmetadata through theExport Metadata dialog box .
Examples
Create a new XML file RetailTransApril.xml, that contains the metadata from the focusRetailTransApril.ftr:
qsexportmetadata -input RetailTransApril.ftr-output RetailTransApril.xml
See also
qsimportmetadata on page 76
qsimportmetadata
Synopsis qsimportmetadata
-input <source focus> -metadata <metadata file>[-output <destinationfocus>] [-force] [-details <kind> [, <kind> ...] -fields <field> [, <field>
76Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
...] | -fields @<fields file>] -fields <field> [, <field> ...] | -fields@<fields file>] [-dryrun] [-warn]
Description: import metadata, including binnings, comments, derivations, interpretations, recordselections, subfocus structure and default subfocus from the metadata file to the source focus.
Optional arguments
EffectOption
Import the specified kinds of metadata, which can includebinnings, comments, derivations,
-details <kind> [, <kind> ...]
history, interpretations,selections, launch (default subfocus) andsubfoci (subfocus structure). Categorical interpretationsare classified as binnings, not interpretations. In the absenceof this option, all kinds of metadata except history areimported.
Do not import metadata, but display information about themetadata that would have been imported.
-dryrun
Instead of importing field metadata from all fields, importmetadata from just the specified fields.
-fields <field> [, <field> ...]
Instead of importing field metadata from all fields, importmetadata from just the fields listed (one per line) in the fieldsfile.
-fields @<fields file>
See The -force command-line option on page 20.-force
Instead of directly applying the metadata to the source focus,copy the source focus to the destination focus and applythe metadata to the destination focus.
-output <destination focus>
Rather than aborting the operation if import of somemetadata fails, issue a warning, and attempt to import theremaining metadata.
-warn
Import field metadata from all fields except the specifiedfields.
-xfields <field> [, <field> ...]
Import field metadata from all fields except the fields thatare listed (one per line) in the fields file.
-xfields @<fields file>
77Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
Besides these command-line options, qsimportmetadata accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • If you import history, and the metadata file includes non-empty history metadata,qsimportmetadata replaces the history metadata in the focus; if the history metadata in themetadata file is missing or blank, qsimportmetadata does not replace the history metadatain the focus.
• If you specify a destination focus, it shares underlying data with the source focus.• The metadata file uses an XML-based file format [see Metadata specification forqsimportmetadata on page 357]. Metadata XML for qsimportmetadata takes the form of a<metadata> element, containing <focus> elements, which can contain <comment>,<history>, nested <focus> and <field> elements. The <field> elements cancontain <comment>, <binning>, <recordselection> and <fdl> elements.
• If you import a categorical binning to a field that does not have a categorical interpretation,qsimportmetadata first interprets the field as categorical, creating base categories from thevalues contained in the field (and a binning called "unnamed node"). If the base categoriesin the imported binning refer to values that are not referred to by the base categories of thefield, qsimportmetadata adds these as base categories. Finally, it adds the categories (ifany) from the imported binning directly below the base categories of the field (unless theimported binning has the same set of base categories as the field, in which caseqsimportmetadata omits the first level of the imported binning, adding the other levelsdirectly below the base categories of the field).
• If you import a categorical binning to a field that has a categorical interpretation, and theimported binning contains at least one <category> element, qsimportmetadata clears thecategorical interpretation on the field (deleting all existing categorical binnings on the field)before it imports the binning. (If the imported binning contains no <category> elements,qsimportmetadata does nothing.)
• If you import a subfocus structure, the newly created subfocus automatically inherits fieldattributes (such as analysis candidates and binnings) from the parent subfocus. This occurseven if you do not explicitly specify the attributes in the subfocus definition because theattributes are set in the process of applying the subfocus structure.
• A subfocus defined in the metadata file takes precedence over a subfocus of the samename in the source file, so any existing metadata associated with such a subfocus isoverwritten.
• Using Spectrum Miner, you can access most of the functionality of qsimportmetadatathrough the Import Metadata to Focus dialog box .
Examples
Given the focus metadata file metadata-fieldfocus.xml, with the following contents:
<?xml version="1.0" encoding="UTF-8"?><metadata xmlns="http://www.quadstone.com/xml">
<focus><comment>This focus was created for the Retail Analysis
project.</comment>
78Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
<field name="Age"><comment>Imported from database field DOBin CUSTINFO.</comment>
</field><field name="Gender"><comment>Imported from CUSTINFO:SEX,mapping 1="M" and 2="F".</comment></field>
</focus></metadata>
import this metadata file to create a focus RetailCustAprilCommented2.ftr, containing fieldcomments and a focus comment:
qsimportmetadata -metadata metadata-fieldfocus.xml-input RetailCustApril.ftr-output RetailCustAprilCommented2.ftr
Alternatively, import the same metadata file to create a focus RetailCustAprilCommented3.ftr,with a field comment on the field Age only:
qsimportmetadata -fields Age -metadata metadata-fieldfocus.xml-input RetailCustApril.ftr-output RetailCustAprilCommented3.ftr
Given the focus metadata file metadata-focushtml.xml, with the following contents:
<?xml version="1.0" encoding="UTF-8"?><metadata xmlns="http://www.quadstone.com/xml">
<focus><comment xhtml="true">
<div>This focus was created for theRetail Analysis project.<br/>An audit is available on the<a href="http://intranet.company.com/audits">intranet</a>.
</div></comment>
</focus></metadata>
import this metadata file to create a focus RetailCustAprilCommented4.ftr, containing afocus comment that has HTML formatting:
qsimportmetadata -metadata metadata-focushtml.xml-input RetailCustApril.ftr-output RetailCustAprilCommented4.ftr
79Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
Given the focus metadata file metadata-catbinning1.xml, with the following contents:
<?xml version="1.0" encoding="UTF-8"?><metadata xmlns="http://www.quadstone.com/xml">
<field name="MaritalStatus"><binning>
<categorical><categories>
<category name="Single" levelname="Summarized"><category name="Single" value="1"
levelname="Detail"/></category>
<category name="Married"><category name="Married" value="2"/>
</category><category name="Other">
<category name="Divorced" value="3"/><category name="Widowed" value="4"/>
<category name="Separated" value="5"/></category>
</categories></categorical>
</binning></field>
</metadata>
import this metadata file to create a focus Lion1.ftr, in which MaritalStatus has a categoricalbinning with two levels (one of which names the base categories, and the other of which mergesthree categories):
qsimportmetadata -metadata metadata-catbinning1.xml-input Lion.ftr -output Lion1.ftr
Given the focus metadata file metadata-catbinning2.xml, with the following contents:
<?xml version="1.0" encoding="UTF-8"?><metadata xmlns="http://www.quadstone.com/xml">
<field name="SRVOverallSatisfaction"><binning>
<categorical><categories>
<category name="Very Dissatisfied" value="VD"/><category name="Dissatisfied" value="D"/><category name="Neutral" value="N"/><category name="Fairly Satisfied" value="FS"/><category name="Very Satisfied" value="VS"/>
</categories></categorical>
</binning></field>
</metadata>
80Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
import this metadata file to create a focus Lion2.ftr, in which SRVOverallSatisfaction hasa categorical binning with a particular ordering of the categories, from "Very Dissatisfied" through"Very Satisfied:"
qsimportmetadata -metadata metadata-catbinning2.xml-input Lion1.ftr -output Lion2.ftr
See also
qsexportmetadata on page 76
qsupdate
Synopsis qsupdate -from <template focus> -to <destination focus> [-force]
Description: apply the focus metadata in the template focus to the destination focus, includingdefinitions of subfoci and derived fields, as well as any field interpretations, binnings and recordselections.
Optional arguments
EffectOption
See The -force command-line option on page 20.-force
Besides this command-line option, qsupdate accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • The destination focus cannot have any existing field interpretations, binnings, or recordselections.
• If the template focus and destination focus have differing field names or datatypes,qsupdate may be unable to copy some aspects of the template focus's metadata.
• This data-build command is deprecated: use qsexportmetadata and qsimportmetadatainstead.
Example
Apply derivations, interpretations, binnings, and record selections from the RetailCustApril.ftrpreprocessed focus to the RetailCustMay.ftr newly-imported focus:
qslink -from RetailCustMay.ftr -to RetailCustMayUpdated.ftrqsupdate -from RetailCustApril.ftr -to RetailCustMayUpdated.ftr
81Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for processing foci
6 - Commands forcombining foci
In this section
About combining foci 83qsjoin 83qsmerge 87
About combining foci
Spectrum Miner provides two ways of combining the fields from multiple foci to create a newfocus, the first of which focus join is a variant of a left join. You can use focus join in SpectrumMiner or via the qsjoin Spectrum Miner data-build command.
The second method uses Decision Studio to import fields from a focus in a variant of a two-tableleft join.
You can also merge the records of multiple foci, in SpectrumMiner or via the qsmerge SpectrumMiner data-build command.
See also
dblookup on page 330
qsjoin
Synopsis qsjoin
-input <primary focus> [-subfocus <subfocus>]
{-keys <key field> [, <key field>] [, <key field>] | -key @<key file>}
[-output <destination focus>] [-force]
[-equalnulls] [-importmeta] [-onetoone]
[-match <prefix>] [-unmatched <prefix>]
-join <secondary focus> [ -subfocus <subfocus>] -fields <field> [, <field>...] | -fields @<fields file> | -tags <tag> [, <tag> ...]] -xfields <field>[, <field> ...] | -xfields @<fields file> | -xtags <tag> [, <tag> ...]][-join <secondary focus> [ -subfocus <subfocus>] -fields <field> [, <field>...] | -fields @<fields file> | -tags <tag> [, <tag> ...]] -xfields <field>[, <field> ...] | -xfields @<fields file> | -xtags <tag> [, <tag> ...]]...]
Description: join fields from the secondary foci to the primary focus, matching records using thekey fields. By default, join all non-key fields from the secondary foci.
Unless you use the -onetoone option, qsjoin effectively performs a series of left outer joins (wherethe primary focus is the left table), incorporating the fields from each of the secondary foci in turn.The key values in a record in the primary focus uniquely identify a corresponding record in eachsecondary focus (where one exists).
Optional arguments
83Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for combining foci
EffectOption
When matching records, treat the null value in a key fieldas an ordinary value (that is, consider all occurrences of the
-equalnulls
null value to be equal to one another but distinct from allother values).
Instead of joining all fields from the secondary focus, joinjust the specified fields.
-fields <field> [, <field> ...]
Instead of joining all fields from the secondary focus, joinjust the fields listed (one per line) in the fields file.
-fields @<fields file>
See The -force command-line option on page 20. Thisalso affects foci created using the -unmatched option.
-force
Import field metadata from the secondary foci.-importmeta
For each secondary focus, create an additional integer fieldin the output focus, containing the value 1 for records that
-match <prefix>
include data from the secondary focus and the value 0 forall other records. Use the given prefix together with the nameof the secondary focus to name the field.
Instead of performing a left outer join, attempt to perform aone-to-one match of groups of records with the same keys.
-onetoone
Instead of joining fields to the primary focus, copy theprimary focus to the destination focus and then join fieldsto the destination focus.
-output <destination focus>
Use the specified subfocus of the primary or secondaryfocus last mentioned on the command line (using-inputor -join).
-subfocus <subfocus>
Instead of joining all fields from the secondary focus, joinjust the fields that have the specified tags.
-tags <tag> [, <tag> ...]
For each secondary focus, create an alternative output focuspreserving those records from the secondary focus whose
-unmatched <prefix>
key fields do not match records in the primary focus. Usethe given prefix together with the name of the secondaryfocus to name the alternative output focus.
Join all fields in the focus except the specified fields.-xfields <field> [, <field> ...]
84Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for combining foci
EffectOption
Join all fields in the focus except the fields that are listed(one per line) in the fields file.
-xfields @<fields file>
Join all fields in the focus except the fields that have thespecified tags.
-xtags <tag> [, <tag> ...]
Besides these command-line options, qsjoin accepts the options common to all data-build commands[see Standard command-line options on page 18].
Note:
• For each named key field, the fields of that name in the primary and secondary foci must all be ofcompatible type, that is, all string fields, all date fields, or all numeric fields.
• The primary and secondary foci must be sorted by the key fields [see qssort on page 59]. If thefoci are not sorted appropriately, qsjoin fails with an error message to that effect.
• For a left outer join (the default join type for qsjoin), no two records in a secondary focus can sharethe same keys.
• For a left outer join (the default join type for qsjoin), if a secondary focus contains no recordcorresponding to a given record in the primary focus, qsjoin uses a null record instead.
• For a left outer join (the default join type for qsjoin), if more than one record in the primary focushas the same set of keys, copies of the corresponding records (if any) from the secondary fociare joined to each of these records.
• If you use the -onetoone option, for each set of records in the primary focus that share a givencombination of key-field values, qsjoin attempts to join fields from successive records in a secondaryfocus that share the same combination of key-field values, in such a way that records are matchedone-to-one.
If, for a given key, there are fewer records in a secondary focus than in the primary focus, qsjoinuses null records in place of missing records in the secondary focus.
If, for a given key, there are more records in a secondary focus than in the primary focus, qsjointreats surplus records in the secondary focus as unmatched. If you use the -unmatched option,qsjoin preserves these records in an alternative output focus.
• If the join operation would otherwise result in duplicate field names, qsjoin renames fields asneeded, by adding numeric suffixes.
• If you specify a destination focus, it shares underlying data with the primary focus (but isindependent of the secondary focus).
• When matching records, qsjoin normally treats each occurrence of the null value in a key field asa new, distinct value: a record with the null value in a key field never corresponds to a record inanother focus. You can override this behavior using the -equalnulls option.
• For backward compatibility, if you specify only one secondary focus, and use the -match or-unmatched option, the name of the secondary focus is not used to name the match field oralternative output focus.
85Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for combining foci
• Using SpectrumMiner, you can access most of the functionality of qsjoin through the Join Fieldsto Focus wizard .
Examples
Create a focus, RetailAprilAnalysis1.ftr, by combining records from theRetailCustApril.ftr and RetailAggregationsApril.ftr foci:
qsjoin -input RetailCustApril.ftr -join RetailAggregationsApril.ftr-keys CustomerID -output RetailAprilAnalysis1.ftr
Alternatively, add only the fields numberPurchases and totalAmount from theRetailAggregationsApril.ftr focus:
qsjoin -fields numberPurchases,totalAmount-input RetailCustApril.ftr -join RetailAggregationsApril.ftr-keys CustomerID -output RetailAprilAnalysis2.ftr
Create the focus RetailAprilAnalysis3.ftr, by combining records from the fociRetailCustApril.ftr, RetailAggregationsApril.ftr, andRetailAggregationsMay.ftr:
qsjoin -input RetailCustApril.ftr -join RetailAggregationsApril.ftr-join RetailAggregationsMay.ftr-keys CustomerID -output RetailAprilAnalysis3.ftr
Repeat the above, and write any unmatched transaction records to the fociunmatchedRetailAggregationsApril.ftr or unmatchedRetailAggregationsMay.ftr:
qsjoin -unmatched unmatched-input RetailCustApril.ftr -join RetailAggregationsApril.ftr-join RetailAggregationsMay.ftr-keys CustomerID -output RetailAprilAnalysis4.ftr
Repeat the above, joining only a subset of the fields from the secondary foci:
qsjoin -unmatched unmatched-input RetailCustApril.ftr-join RetailAggregationsApril.ftr-fields numberPurchasesApr,totalAmountApr-join RetailAggregationsMay.ftr-fields numberPurchasesMay,totalAmountMay-keys CustomerID -output RetailAprilAnalysis5.ftr
See also
qsmerge on page 87
86Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for combining foci
qsmerge
Synopsis qsmerge
-input <primary focus> [-subfocus <subfocus>]
{-keys <key field> [, <key field>] [, <key field>] | -key @<key file>}
[-equalnulls]
-output <destination focus> [-force]
[-nodups]
-merge <secondary focus> [ -subfocus <subfocus>][-merge <secondary focus>[ -subfocus <subfocus>] ...]
Description Merge records from the primary and secondary foci, interleaving them to create thedestination focus, in such a way that it remains sorted by the key fields [see qssort on page 59].
Optional arguments
EffectOption
When testing for duplicate keys, treat the null value in akey field as an ordinary value (that is, consider all
-equalnulls
occurrences of the null value to be equal to one another butdistinct from all other values).
See The -force command-line option on page 20.-force
Avoid duplicating key-field values: if two or more recordsshare key-field values, retain the first such record from the
-nodups
first focus to contain such a record (treating foci in the order:primary, earliest-mentioned secondary, etc.); discard theremainder of the records with the same key-field values.
Use the specified subfocus of the primary or secondaryfocus last mentioned on the command line (using-inputor -merge).
-subfocus <subfocus>
Besides these command-line options, qsmerge accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • The primary and secondary foci must be sorted by the key fields [see qssort on page 59].If the foci are not sorted appropriately, qsmerge fails with an error message to that effect.
87Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for combining foci
• Corresponding fields in the primary and secondary foci (identified by name) must be of thesame exact type, but they do not need to be arranged in the same order.
• The destination focus contains the same set of fields as the primary focus, arranged in thesame order. If a secondary focus contains fields that are not in the primary focus, qsmergeignores them. If the primary focus contains fields that are not in a secondary focus, qsmergeuses the null value in place of missing values in the secondary focus.
• If two or more records share the same key-field values, those from the primary focus appearfirst in the destination focus, followed by those from each secondary focus in turn. Unlessyou use the -nodups option, the merge operation does not remove any records withduplicate keys.
• By using the same focus twice on the command line — as the argument to both -inputand -merge — and using the -nodups option, you can use qsmerge to remove recordswith duplicate key values from a focus.
• The destination focus is independent of the primary and secondary foci, that is, it does notshare underlying data.
• When ordering records or testing for duplicate keys, qsmerge normally treats the null valuein a key field as a new, distinct value: a record with the null value in a key field nevercorresponds to a record in another focus. You can override this behavior using the-equalnulls option.
• Using Spectrum Miner, you can access most of the functionality of qsmerge through theMerge Records dialog box .
• If the merged foci contains legacy datatypes, the legacy types are converted to the mostappropriate supported datatype in the output focus.
If conversion fails because the datatype does not have sufficient storage, rather thantruncating the content the field will contain the null value. In this case, a summary messageshowing the effected field and number of entries on completion.
• In order to merge two foci, the fields to be merged must contain compatible data types.
Examples
Create a focus, RetailTransAprilMayJune.ftr, containing all the records in the fociRetailTransAprilSorted.ftr, RetailTransMaySorted.ftr, andRetailTransJuneSorted.ftr:
qsmerge -input RetailTransAprilSorted.ftr-merge RetailTransMaySorted.ftr-merge RetailTransJuneSorted.ftr-keys CustomerID -output RetailTransAprilMayJune.ftr
Alternatively, include only records from the RetailTransAprilSorted.ftr focus whose keyvalues also appear in the RetailTransMaySorted.ftr focus:
qsmerge -nodups-input RetailTransAprilSorted.ftr
88Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for combining foci
-merge RetailTransMaySorted.ftr-keys CustomerID -output RetailTransAprilMayNodups.ftr
See also
qsjoin on page 83
89Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for combining foci
7 - Commands formanaging foci
In this section
qscopy 91qslink 92qsmove 93qsremove 93qsarchive 95qsunzip 95
qscopy
Synopsis qscopy -from <source focus>
{-to <destination focus> | -to <directory>} [-force]
Description: copy the source focus to the destination focus or to a focus in the specified directory(preserving the name of the source focus). Copy the underlying data so that the new focus isindependent of other foci.
Optional arguments
EffectOption
See The -force command-line option on page 20.-force
Besides this command-line option, qscopy accepts the options common to all data-build commands[see Standard command-line options on page 18].
Because of the potential interdependence of foci, you should not attempt to copya focus using standard operating-system utilities. Instead, you should always useqscopy or qslink (or Spectrum Miner).
Note: • If you try to copy a focus on top of an existing focus, qscopy warns you and does nothing.• To create a linked copy of a focus, in which the copy shares the underlying data with the
original focus, use qslink.• To create a copy of a focus, or of a subset of a focus, not including any metadata , useqsimportfocus.
• You can also access the functionality of qscopy through Spectrum Miner .
Example
Create a new focus, RetailArchiveMay.ftr, archiving the results of the May analysis sessionin the RetailAnalysisMay.ftr focus:
qscopy -from RetailAnalysisMay.ftr -to RetailArchiveMay.ftr
See also
qsmove on page 93
qsremove on page 93
91Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for managing foci
qslink
Synopsis qslink -from <source focus>
{-to <destination focus> | -to <directory>} [-force]
Description: copy the source focus to the destination focus or to a focus in the specified directory(preserving the name of the source focus). Share the underlying data with the original focus. (Thenew focus is dependent on the original focus.)
Optional arguments
EffectOption
See The -force command-line option on page 20.-force
Besides this command-line option, qslink accepts the options common to all data-build commands[see Standard command-line options on page 18].
Note: • If you try to copy a focus on top of an existing focus, qslink warns you and does nothing.• To create an independent copy of a focus, in which the copy does not share the underlying
data with the original focus, use qscopy.• You can also access the functionality of qslink through Spectrum Miner .
Example
Create a focus RetailLoyaltyProjectMay.ftr, that shares underlying data with theRetailAnalysisMay.ftr focus:
qslink -from RetailAnalysisMay.ftr-to RetailLoyaltyProjectMay.ftr
See also
qsimportfocus on page 48
qsmove on page 93
qsremove on page 93
92Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for managing foci
qsmove
Synopsis qsmove -from <source focus>
-to {,<destination focus> | <directory>,}
Description: rename the source focus to the destination focus, or move the source focus to thespecified directory, taking into account possible data dependencies.
Optional arguments Besides the required arguments, qscopy accepts the options common to alldata-build commands [see Standard command-line options on page 18].
Note: • If you try to rename or move a focus on top of an existing focus, qsmove warns you anddoes nothing.
• You can also access the functionality of qsmove through Spectrum Miner .
Because of the potential interdependence of foci, you should not attempt to renameor move a focus using standard operating-system utilities. Instead, you shouldalways use qsmove (or Spectrum Miner).
Examples
Rename the RetailTransAprilSorted.ftr focus to RetailTransAprilSort.ftr:
qsmove -from RetailTransAprilSorted.ftr-to RetailTransAprilSort.ftr
Move the RetailArchiveMay.ftr focus to the Retail directory:
qsmove -from RetailArchiveMay.ftr -to Retail
See also
qscopy on page 91
qslink on page 92
qsremove on page 93
qsremove
Synopsis qsremove -focus <focus file>
[-focus <focus file> ...]
93Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for managing foci
[-force]
[-recursive]
Description: delete one or more focus files, taking into account possible data dependencies.
Optional arguments
EffectOption
Ignore any errors caused by failure to update links withrelated foci.
-force
Delete the focus and all dependent foci.-recursive
In additon to these command-line options, qsremove accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Because of the potential interdependence of foci, you should not attempt to deletea focus using standard operating-system utilities. Instead, you should always useqsremove (or Spectrum Miner).
Note: • If there is a circular dependency in the chain, or the parent of the focus chain is not writeable,an error will be displayed. In this case, you need to use both -force and -recursiveto delete the focus.
• If you use more than one -focus argument and qsremove fails to delete one of the foci,it nevertheless continues to delete subsequent foci (but produces an error message andreturns a non-zero exit status).
• Removing a focus with qsremove will fail if you do not have write permission to the .ftrfocus file.
• After removing a focus using qsremove, there may be a backup .ftr file left behind. Youcan safely delete this as you would delete any file.
• You can also access the functionality of qsremove through Spectrum Miner .
Examples
Delete the RetailTransApril6Months.ftr focus and all dependents:
qsremove -focus RetailTransApril6Months.ftr
Delete two foci RetailTransAprilCA.ftr and RetailTransApril2Customers.ftr:
qsremove -focus RetailTransAprilCA.ftr-focus RetailTransApril2Customers.ftr
See also
94Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for managing foci
qscopy on page 91
qslink on page 92
qsmove on page 93
qsarchive
Synopsis qsarchive -input <focus or folder>
[, -input <focus or folder> ...]-output <archive file>
Description: create an archive file from a list of files or folders.
Optional arguments: besides the required arguments, qsarchive accepts the options commonto all data-build commands [see Standard command-line options on page 18].
Note: • If you create an archive file from a folder that contains foci that link to data outside thefolder, the external data is copied into the archive.
• You can access most of the functionality of the data-build command qsarchive by usingthe Archive dialog box in Spectrum Miner.
Example
Create an archive file FirstQuarter.zip that shares underlying data with the January.ftr,February.ftr, and March.ftr foci:
qsarchive -input January.ftr -input February.ftr -input March.ftr-output FirstQuarter.zip
See also
qsunzip on page 95
qsunzip
Synopsis qsunzip -input <archive file> [-output <output directory>]
[-overwrite]
Description: extract files and folders from an archive file.
Optional arguments
95Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for managing foci
EffectOption
Extract to an alternative directory to the one in which thearchive file is located. If you don't provide an -output
-output <output directorygt;
option, the files and folders will be extracted to the currentdirectory.
Overwrites files and folders if you extract to a location thatcontains the same content as the zip archive.
-overwrite
In addition to these command-line options, qsunzip accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • If you don't use the -overwrite option and extract to a location that contains the samesource files and folders, the extract will fail.
• You can access most of the functionality of the data-build command qsunzip by using theExtract dialog box in Spectrum Miner.
Example Extract from the archive file firstquarter.zip into the same location asfirstquarter.zip:
qsunzip -input firstquarter.zip
See also
qsarchive on page 95
96Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for managing foci
8 - Commands forproducing reports
In this section
qssettings 98qsaudit 98qsdescribe 102qsdescribestat 104qshtmlunpack 105qsdtsnapshot, qsscsnapshot 105qsxt 108qsinfo 110qsdescribelicense 111
qssettings
Synopsis qssettings [<property>=<value> ...]
Description: set the value of each listed property, to control aspects of the output of thereport-generating commands qsaudit, qsdtsnapshot, and qsscsnapshot.
You can set values of the following properties:
• DateFormat, with 0 corresponding to European, 1 corresponding to American, and 2 correspondingto YMD
• NumDecimalPlaces• StripTrailingZeroes• ThousandSeparators• TwelveHour
Optional arguments Besides the required arguments, qssettings accepts the options commonto all data-build commands [see Standard command-line options on page 18].
The qssettings command creates or modifies the file settings.xml in your user-specificSpectrum Miner configuration directory. Use this file in conjunction with qsaudit,
Note:
qsdtsnapshot, or qsscsnapshot by specifying the -settings option [see Standardcommand-line options on page 18].
Example Create a settings file to display dates with YMD date formats, 12-hour time format andnumbers with trailing zeros, to four decimal places and a locale-specific thousand separator:
qssettings DateFormat=2 TwelveHour=true StripTrailingZeroes=falseNumDecimalPlaces=4 ThousandSeparators=true
See also
Date formats on page 52 qsaudit on page 98 qsdtsnapshot, qsscsnapshot on page 105
qsaudit
Synopsis qsaudit [-generate Full]
-input <focus> [-subfocus <subfocus>]
[-reference <archived HTML report> | -reference <archived XML report>]
-output <archived HTML report>
98Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]
-xfields <field> [, <field> ...] | -xfields @<fields file> | -xtags <tag>[, <tag> ...]]
[-records <FDL expression> | -records @<FDL file>>]
[-targets <field> [, <field> ...] | -targets @<fields file> | -notarget]
[-overwrite]
[-htmlimages largepng | -htmlimages smallpng | -htmlimages svg |-htmlimages none]
[-paginate]
[-nopartition]
[-partitionfield <field>]
qsaudit -generate XML
-input <focus> [-subfocus <subfocus>]
[-reference <archived HTML report> | -reference <archived XML report>]
-output <archived XML report>
-fields <field> [, <field> ...] | -fields @<fields file> | -tags <tag> [,<tag> ...]]
-xfields <field> [, <field> ...] | -xfields @<fields file> | -xtags <tag>[, <tag> ...]]
[-records <FDL expression> | -records @<FDL file>]
[-targets <field> [, <field> ...] | -targets @<fields file> | -notarget]
[-overwrite]
[-nopartition]
[-partitionfield <field>] qsaudit -generate HTML
{-input <archived HTML repor> | -input <archived XML report>}
-output <archived HTML report>
[-overwrite]
[-htmlimages largepng | -htmlimages smallpng | -htmlimages svg |-htmlimages none] [-paginate]
Description: create a Profile and Audit report on the specified focus, or a comparison reportdescribing differences between the specified focus and the focus described in a reference report(with the -reference option). By default, audit all the fields in the focus, in relation to the focusobjective if there is one. This is a two-stage process: first the command generates an intermediateXML file (packaged in a .qsxml archive) and then from this intermediate file it creates the finishedHTML-formatted report (packaged in a .qshtml archive).
99Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
Alternatively, create just the intermediate .qsxml archive — or create a .qshtml archive from anexisting .qshtml or .qsxml archive.
To unpack .qshtml and .qsxml archives, for viewing or reuse outside Spectrum Miner, useqshtmlunpack.
Optional arguments
EffectOption
Audit only the specified fields (except any fields excludedusing -xfields).
-fields <field> [, <field> ...]
Audit only the fields listed (one per line) in the fields file.-fields @<fields file>
Create an HTML report from an XML or HTML report.-generate HTML
Create an HTML report directly (the default if you do notspecify a -generate option).
-generate Full
Create an XML report.-generate XML
Include printer- and screen-optimized PNG bitmaps andSVG images.
-htmlimages largepng
Do not include any images.-htmlimages none
Include screen-optimized PNG bitmaps and SVG images(the default if you do not specify a-htmlimages option).
-htmlimages smallpng
Include only SVG images.-htmlimages svg
Create a non-uplift Profile and Audit, ignoring any partitioninterpretation in the focus.
-nopartition
Don't audit fields in the focus in relation to any target fields,even if the focus has an objective.
-notarget
Overwrite any existing report.
Otherwise, if there is already a report of the same name,qsaudit warns you and does nothing.
-overwrite
Create a separate HTML page for each audited field.-paginate
100Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
EffectOption
Use the specificed field as the partition field.-partitionfield <field>
Report only on the records for which the numeric FDLexpression is non-zero ("true").
-records <FDL expression>
Report only on the records for which the FDL expression inthe specified file is non-zero ("true").
-records @<>FDL file>
Use the previously created archived HTML report as areference report, and create a comparison report.
-reference <archived HTML report>
Use the specified subfocus of the focus.-subfocus <subfocus>
Audit only the fields that have the specified tags (except anyfields excluded using -xtags).
-tags <tag> [, <tag> ...]
Audit fields in the focus in relation to the specified targetfields, instead of the focus objective.
-targets <field> [, <field> ...]
Audit fields in the focus in relation to the target fields listed(one per line) in the fields file.
-targets @<fields file>
Do not audit the specified fields.-xfields <field> [, <field> ...]
Do not audit the fields listed (one per line) in the fields file.-xfields @<fields file>
Do not audit the fields that have the specified tags.-xtags <tag> [, <tag> ...]
Besides these command-line options, qsaudit accepts the -settings option, as well as optionscommon to all data-build commands [see Standard command-line options on page 18].
Note: • By default, a comparison report includes the same fields as in the reference report. If youexplicitly choose fields to include in the comparison report (using the -fields option),they are added to the previously profiled fields.
• A comparison report uses the same target fields as in the reference report. Any explicitlyspecified target fields are ignored.
• Additional report-formatting options are available through Audits and Snapshotspreferences (see Spectrum Miner Online Help).
• If you do not include an objective and -target option, qsaudit will fail to audit the focus.• Using Spectrum Miner, you can access most of the functionality of qsaudit through theCreate Profile and Audit dialog box .
101Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
Examples
Create a new Profile and Audit RetailCustApril.qshtml from the focusRetailCustApril.ftr:
qsaudit -input RetailCustApril.ftr -output RetailCustApril.qshtml
Create a new Profile and Audit RetailCustMay.qshtml from the focus RetailCustMay.ftr,auditing all fields by Age:
qsaudit -targets Age -input RetailCustMay.ftr-output RetailCustMay.qshtml
Overwrite this to create a Profile and Audit in which each field audit is on a separate HTML page:
qsaudit -paginate -targets Age -overwrite -input RetailCustMay.ftr-output RetailCustMay.qshtml
Create a Profile and Audit RetailCustAprilnoimages.qshtml without images, from the focusRetailCustApril.ftr, only for the fields StartDate, Age, and Gender:
qsaudit -fields "StartDate, Age, Gender" -htmlimages none-input RetailCustApril.ftr-output RetailCustAprilnoimages.qshtml
Re-create the HTML output, with one field audit per HTML page (and without recreating the underlyingaudit data):
qsaudit -generate HTML -paginate-input RetailCustApril.qshtml-output RetailCustAprilpaginate.qshtml
Create a comparison report RetailCustMayApril.qshtml from the RetailCustMay.ftrfocus in comparison with the reference report RetailCustApril.qshtml:
qsaudit -reference RetailCustApril.qshtml-input RetailCustMay.ftr -output RetailCustMayApril.qshtml
See also
XML in SpectrumMiner on page 356 qsdescribe on page 102 qsdtsnapshot, qsscsnapshot onpage 105 qssettings on page 98
qsdescribe
Synopsis qsdescribe
102Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
-input <source focus> [-subfocus <subfocus>]
[-output <report file>]
[-fields [-detail]]
Description: display summary information about the focus in plain-text format, including the numberof fields and records and the focus history.
Optional arguments
EffectOption
Report more detailed information on the fields in the focus,including field statistics, derivation expressions, and seedsfor derived fields.
-detail
Report information on each of the fields in the focus.-fields
Write the report to the specified file instead of standardoutput.
-output <report file>
Use the specified subfocus of the source focus.-subfocus <subfocus>
Besides these command-line options, qsdescribe accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • By default, qsdescribe reports on the entire hierarchy of subfoci within a focus. If you usethe -subfocus option, qsdescribe only reports on the subfocus that you specify.
• Using Spectrum Miner, you can access most of the functionality of qsdescribe throughthe Display Focus Information dialog box .
Examples
Report the number of fields, number of records, and history of the RetailCustApril.ftr focus:
qsdescribe -input RetailCustApril.ftr
Additionally, report the field names and types:
qsdescribe -fields -input RetailCustApril.ftr
Finally, report the field statistics, selections, comments, and derivations on the fields:
qsdescribe -detail -fields -input RetailCustApril.ftr
See also
qsaudit on page 98
103Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
qssettings on page 98
qsdescribestat
Synopsis qsdescribestat -input <source dataset>
[-output <report file>]
[-type <dataset type>]
[-detail]
Description: display a list of the field names in the Excel dataset in plain-text format.
Optional arguments
EffectOption
List the datatypes of fields alongside their names.-detail
Write the report to the specified file instead of standardoutput.
-output <report file>
Interpret the Excel dataset as the specified dataset type,overriding the default interpretation (which depends on thefilename extension).
-type <dataset type>
Besides these command-line options, qsdescribestat accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Examples
Report the field names of the Excel dataset RetailCustApril.xlsx:
qsdescribestat -input RetailCustApril.xlsx
Additionally, report the field types:
qsdescribestat -detail -input RetailCustApril.xlsx
See also
qsexportstat on page 46
qsimportstat on page 44
104Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
qshtmlunpack
Synopsisqshtmlunpack <archived report> <output directory> [<HTML filename>]
Description: unpack a Spectrum Miner-generated report in the form of a .qshtml or .qsxmlarchive, creating the specified output directory to contain the components of the report. In the caseof an archived HTML report, the HTML file itself (within the output directory) is by default namedqsreport.html.
Optional arguments
EffectOption
Use this filename for the HTML file in an unpacked HTMLreport, instead of qsreport.html.
When unpacking an archived XML report,qshtmlunpack ignores this argument.
<HTML filename>
Using Spectrum Miner, you can access most of the functionality of qshtmlunpack usingthe right-click option Unpack to Folder.
Note:
Example
Unpack the archive RetailCustApril.qshtml as an HTML Profile and AuditRetailCustApril.html in subdirectory www:
qshtmlunpack RetailCustApril.qshtml www RetailCustApril.html
See also
qsaudit on page 98
qsdtsnapshot, qsscsnapshot on page 105
qsdtsnapshot, qsscsnapshot
Synopsis
{qsdtsnapshot -input <decision tree> | qsscsnapshot -input <scorecard>}[-generate Full]
105Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
-output <archived HTML report>
[-focus <focus> [-subfocus<subfocus>]]
[-audit modeled | -audit all | -audit none]
[-description <text file>]
[-overwrite]
[-htmlimages largepng | -htmlimages smallpng | -htmlimages svg |-htmlimages none]
{qsdtsnapshot -input <decision tree> | qsscsnapshot -input <scorecard>}
-generate XML
-output <archived XML report>
[-focus <focus> [-subfocus<subfocus>]]
[-audit modeled | -audit all | -audit none]
[-overwrite]
[-htmlimages largepng | -htmlimages smallpng | -htmlimages svg |-htmlimages none]
{qsdtsnapshot | qsscsnapshot}
-generate HTML
{-input <archived HTML report> | -input<archived XML report>}
-output <archived HTML report>
[-description <text file>
[-overwrite]
[-htmlimages largepng | -htmlimages smallpng | -htmlimages svg |-htmlimages none]
Description: create a Model Snapshot of the specified decision tree (.qsdt file) or scorecard(.qssc file). By default, audit all the fields used in the model. This is a two-stage process: first thecommand generates an intermediate XML file (packaged in a .qsxml archive), and then from thisintermediate file it creates the finished HTML-formatted report (packaged in a .qshtml archive).
Alternatively, create just the intermediate .qsxml archive — or create a .qshtml archive from anexisting .qshtml or .qsxml archive.
To unpack .qshtml and .qsxml archives, for viewing or reuse outside Spectrum Miner, useqshtmlunpack.
Optional arguments
106Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
EffectOption
Audit all the fields in the focus.-audit all
Audit the fields used in the model (the default if you do notspecify a -audit option).
-audit modeled
Do not audit any fields.-audit none
Include the contents of this text file as a description of themodel in the "User Supplied Metadata" section of the report.
-description <text file>
Report on the model as applied to the specified focus, ratherthan the focus used to build the model.
-focus <focus>
Create an HTML report from an XML or HTML report.-generate HTML
Create an HTML report directly (the default if you do notspecify a -generate option).
-generate Full
Create an XML report.-generate XML
Include printer- and screen-optimized PNG bitmaps andSVG images.
-htmlimages largepng
Do not include any images.-htmlimages none
Include screen-optimized PNG bitmaps and SVG images(the default if you do not specify a-htmlimages option).
-htmlimages smallpng
Include only SVG images.-htmlimages svg
Overwrite any existing report.
Otherwise, if there is already a report of the same name,qsdtsnapshot qsscsnapshot warns you anddoes nothing.
-overwrite
Use the specified subfocus of the focus.-subfocus <subfocus>
Besides these command-line options, qsdtsnapshot and qsscsnapshot accept the -settingsoption, as well as options common to all data-build commands [see Standard command-lineoptions on page 18].
107Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
Note: • If you do not use the -focus option, the focus used to build the model must still exist.• Additional report-formatting options are available through audits and snapshots
preferences (see Spectrum Miner Online Help).• Using Spectrum Miner, you can access most of the functionality of qsdtsnapshot andqsscsnapshot through the Decision Tree Snapshot or Scorecard Snapshot dialogbox .
Examples
Create a Model Snapshot from the decision-tree file early-adopter-modelApril.qsdt, builton the focus RetailCustApril.ftr, and applied to the same focus:
qsdtsnapshot -input early-adopter-modelApril.qsdt-output early-adopter-modelApril.qshtml
Apply this model to the RetailCustMay.ftr focus, but this time auditing over all fields, ratherthan only those involved in defining the model:
qsdtsnapshot -focus RetailCustMay.ftr -audit all-input early-adopter-modelApril.qsdt-output early-adopter-modelMay.qshtml
See also
XML in Spectrum Miner on page 356
qsaudit on page 98
qssettings on page 98
qsxt
Synopsis qsxt -focus <focus> [-subfocus <subfocus>]
[-spec <crosstab file>] [-comparable]
[-output <crosstab file>]
[-description <text file>]
Description: apply a crosstab specification (supplied by default on standard input) to the specifiedfocus, to create a new crosstab (by default on standard output).
Optional arguments
108Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
EffectOption
Where relevant, use binnings from the result part of thesupplied crosstab rather than from the specification part.
-comparable
Include the contents of this text file as a description of thecrosstab.
-description <text file>
Create this crosstab file instead of writing to standard output.-output <crosstab file>
Use this crosstab file instead of reading from standard input.-spec <crosstab file>
Use the specified subfocus of the focus.-subfocus <subfocus>
Besides these command-line options, qsxt accepts the options common to all data-build commands[see Standard command-line options on page 18].
Note: • In the output crosstab, qsxt bins each field using one of the following (in decreasing orderof preference): a binning for the field in the supplied crosstab specification (or result partif you use the -comparable option); a binning for the field in the supplied focus; the defaultbinning parameters, as configured by binning preferences (see Spectrum Miner OnlineHelp).
• Using Spectrum Miner, you can access most of the functionality of qsxt through theApply Crosstab dialog box .
Examples
Create a new crosstab, ageGenderMay.qstv, from the RetailCustMay.ftr focus, using theageGenderApril.qstv crosstab specification file (generated from Decision Studio):
qsxt -focus RetailCustMay.ftr -spec ageGenderApril.qstv-output ageGenderMay.qstv
Additionally, use the binning contained in the ageGenderApril.qstv crosstab file to create theageGenderMay-binbyApril.qstv result:
qsxt -comparable -focus RetailCustMay.ftr -spec ageGenderApril.qstv-output ageGenderMay-binbyApril.qstv
See also
Crosstab specification for qsxt on page 371
109Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
qsinfo
Synopsis qsinfo -input <source focus>
[-generate HTML | -generate XML]
[-output <report file>]
Description: display information on the locations and sizes of the files that constitute the sourcefocus, including any data files shared with other foci. Display the source focus's relationships toother foci (listing both foci that depend on it and foci on which it depends). By default, write the reportto standard output in an XML-based file format.
Optional arguments
EffectOption
Create the report in HTML format.-generate HTML
Create the report in XML-based file format (the default ifyou do not specify a -generate option).
-generate XML
Write the report to the specified file instead of standardoutput.
-output <report file>
Besides these command-line options, qsinfo accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • A focus is stored on disk as a .ftr file (the "focus file") together with a collection of datafiles and other files in two folders, with names ending in .rdx ("Raw Data eXtract") and.xtr ("eXTRa").
A new focus that is created by copying an existing focus [see qscopy on page 91] or byimporting data from another kind of dataset is independent of other foci. On the other hand,a focus that is saved from Decision Studio with a new name, or one that is created bymaking a linked copy [see qslink on page 92] is dependent on data folders belonging toother foci.
• Using Spectrum Miner, you can access most of the functionality of qsinfo through theDisplay Focus Properties dialog box .
Example
110Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
Create a new HTML focus properties report RetailCustApril.html, from the focusRetailCustApril.ftr:
qsinfo -input RetailAprilAnalysis.ftr -generate HTML-output RetailAprilAnalysis.html
See also
XML in Spectrum Miner on page 356
qsdescribelicense
Synopsis qsdescribelicense
[-input <license file>]
[-output <report file>]
Description: display a report about the installed Spectrum Miner license in plain-text format.
Optional arguments
EffectOption
Report on the specified license file instead of the installedlicense.
-input <license file>
Write the report to the specified file instead of standardoutput.
-output <report file>
Besides these command-line options, qsdescribelicense accepts the options common to alldata-build commands [see Standard command-line options on page 18].
The report may include the following components:Note:
• the name of the license holder• the IP address or subnet for which the license is valid• the maximum focus size for which the license is valid• start and expiration dates for the license• a list of licensed Spectrum Miner components, with possible per-component overrides
for IP address/subnet, maximum focus size, and dates of validity
Example
111Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
Show the license keys in the current Spectrum Miner license file:
qsdescribelicense
112Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for producing reports
9 - Commands forbuilding models
In this section
About the Scorecard Wizard 114qsscorecardwizard 115qsdecisiontree 120qsscorecard 122About the Association Rule Wizard 123qsruleminer 125
About the Scorecard Wizard
The Scorecard Wizard automates scorecard modeling using best-practice methodology: performingoptimized binning, then variable reduction, and finally building a sequence of models on test andtraining data to establish the best "model size." The modeling methodology consists of the followingstages.
Test/training methodology: part of the source data is used for training, and part for testing. Thissplit can be generated at random or using a designated binary indicator field.
Optimized binning: the training data is used to select a binning for each analysis candidate thatmaximizes its univariate predictive power against the objective.
For non-categorical fields, the optimized binning wizard is used iteratively to choose the best binning.The number of bins is increased until the number of "turning points" in the plot of the averageobjective value by bin exceeds a configurable threshold (a "turning point" being a change of sign inthe slope of the graph). Intuitively, this ensures that the profile of average objective value as afunction of each analysis candidate is relatively "smooth," with not too many ups and downs.
For categorical fields, categories where the average objective value is not significantly different fromthe overall mean are merged together. A standard difference of means test [see significance onpage 174] is used, with a Bonferroni correction applied based on the total number of categories, c,that appear for the field. Thus, all categories where the confidence level are merged (with aconfigurable significance threshold).
During this phase, fields with suspiciously high correlation to the objective are excluded from furtheranalysis (with a warning message). A warning is also issued for fields where any individual bincontains a small number of records. Both thresholds are configurable.
Variable Reduction: the variable reduction phase reduces the pool of analysis candidates(independent predictors) from those initially marked to a smaller set of n, in order to decreaserun-time of the final "right-sizing" phase.
The process partitions the analysis candidates into groups of up to 2n fields and builds a scorecard(using the training data) on each group, discarding the worst fields by stepwise exclusion until onlythe best n fields of the group remain. By repeating this procedure, eventually the n best overall fieldsare identified.
Right-sizing: the right-sizing phase chooses the scorecard that best balances complexity (numberof predictors) against generalizability (as measured by test-set performance). The training data isused to build a sequence of n scorecards with the first containing all n analysis candidates, andeach subsequent model excluding the worst predictor. The performance of each model is thenassessed on the test data, with the best model size chosen using a configurable criterion. As a finalstep, the weights in the selected model are refit based on all data (i.e. both test and training) toimprove accuracy. All interim models can be saved along with the final model if desired.
Results: various outputs are (optionally) generated, including:
114Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
• An HTML report summarizing how the methodology was applied, including the input parametersand links to the detailed execution log. This includes a table of performance data that can be easilycharted within Excel.
• The final recommended model (in .qssc and QMML formats).• A Model Snapshot report for the recommended model.• The final dataset including the selected analysis candidates, optimized binnings, and model
predictions. Intermediate datasets can also be saved if desired, containing the metadata generatedby the methodology to that point. These are often useful to drill down and explore the decisionsmade during the process.
See also
qsscorecardwizard on page 115
qsscorecardwizard
Synopsis qsscorecardwizard <parameters file>
Description: build a scorecard on a specified focus by using a parameter file.
You can also access the functionality of the data-build command qsscorecardwizard byusing the Scorecard Wizard dialog box in Spectrum Miner.
Note:
Modeling parameters: the modeling process executed by qsscorecardwizard is controlled bythe parameter file you specify as a command-line argument. For example, with a file calledparameters.ini containing:
[globals]inputFocus=D:/Data/DirectBank/DirectBank.ftrobjectiveField=CardVisaxmlReport=D:/Data/DirectBank/DirectBank.xml[variable reduction]maxVariables=5[model development]finalModel=D:/Data/DirectBank/DirectBank_auto.qsscfinalFocus=D:/Data/DirectBank/DirectBank_auto_3rightsized.ftr
the following command
qsscorecardwizard parameters.ini
would create a scorecard model DirectBank_auto.qssc along with a new focusDirectBank_auto_3rightsized.ftr and a model report DirectBank.xml.
The parameter file has a similar format to a Windows .ini file:
115Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
• Each section is indicated by a section heading enclosed in square brackets, e.g. [globals].The four valid section names are listed below.
• Parameters within each section are defined by specifying the parameter name, an equal sign andthe parameter value, e.g. maxBins = 20.
• Parameter names are not case-sensitive, and leading and trailing whitespace on each line isignored.
• Blank lines and lines starting with the comment characters # or ; (along with optional leadingwhitespace) are ignored.
There are default values for several of the parameters, which can be modified on a site-wide orper-user basis, but these are always overridden by parameters specified in the command-line .inifile. Specifying a parameter value in any of the following locations overrides any setting in a priorlocation (where <smhome> is the Spectrum Miner installation directory):
• The required standard defaults in
<smhome>/ext/scorecardwizard/defaults.ini.
• Optional site-wide defaults in
<smhome>/ext/scorecardwizard/scorecardwizard.ini
• Optional per-user defaults in
<smhome>/shared/users/<user>@<domain>/scorecardwizard.ini
• The required parameter file specified on the command line.
Parameters in the [globals] section
DescriptionDefaultParameter
The location of the input focus.(required)inputFocus
The objective field name. Required unless the focus contains an objectiveinterpretation. If set, this setting overrides any focus interpretation.
(none)objectiveField
A comma-separated list of analysis candidate fields. If set, any analysiscandidate interpretations in the focus are ignored.
(none)useFields
A comma-separated list of fields to exclude from the list of analysiscandidates.
(none)ignoreFields
116Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
DescriptionDefaultParameter
The field to use as the test-training indicator. The field must be a numericbinary field with 1 indicating training data, 0 indicating test. If not set, asplit will be generated automatically.
(none)testTrainingField
The proportion of records (between 0 and 1) to select for training, unlesstestTrainingField is specified.
0.5trainingProportion
The random seed to use. If not set, a random seed will be generatedautomatically. You can replicate a previous run by using the seedreported by that run (in the "parameters" section).
(none)randomSeed
Set to true to output additional diagnostic information to the console log.This extra detail is always logged to the file log; see logFile.
falseverbose
The location to log execution progress. If not specified, the log is createdin <params>-<YYYY-mm-dd-HH-MM-SS>.log alongside
(automatic)logFile
the <params>.ini parameter file specified on the command line.(When launched via Spectrum Miner, the log is created in<smhome>/shared/logs/qsscorecardwizard-<user>-<YYYY-mm-dd-HH-MM-SS>.log.)
The location to create the XML summary report. By default the reportis saved at <params>_report.xml alongside the<params>.ini parameter file specified on the command line.
(automatic)xmlReport
Parameters in the [optimized binning] section
DescriptionDefaultParameter
Set to false to skip optimized binning.truedoOptimizedBinning
The minimum number of records thatmust appear in a non-null bin during
0minBinSize
optimized binning of non-categoricalfields. Any bin (including null andcategorical bins) with fewer records willtrigger a warning in the final report.
The maximum number of bins thatshould be created for non-categorical
20maxBins
fields. The selected number of binsmight be lower, based on the"turning-point" constraint (see below).
117Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
DescriptionDefaultParameter
The maximum number of changes ofsign in the difference of the mean of the
2maxTurningPoints
objective field between consecutive bins(excluding null, unclassified). A valueof 0 forces the final binning to bemonotone in the mean of the objectivefield. A value ofmaxBins (or greater)removes this constraint. Typically avalue of 1 or 2 is reasonable.
The maximum "believable" Gini (or R^2)percentage. Fields with predictive
95maxQuality
power above this threshold will beexcluded from further analysis. Notethis check is always carried out (evenwhen optimized binning is disabled),unless a value of 100 is specified.
The significance level that determineswhether a categorical bin has a
0.001pValue
response rate significantly different fromthe overall population. The value isspecified on a 0 to 1 scale, typicallynear 0. Note that a Bonferronicorrection is also applied.
The location to save the interim focusafter optimized binning, if desired.
(none)binnedFocus
Parameters in the [variable reduction] section
DescriptionDefaultParameter
Set to false to skip variable reduction(and use all analysis candidates in themodel right-sizing phase).
truedoVariableReduction
The maximum number of analysiscandidates to allow in the final model(typically 20 — 100).
100maxVariables
The location to save the interim focusafter variable reduction, if desired.
(none)reducedFocus
118Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
Parameters in the [model development] section
DescriptionDefaultParameter
Set to false to skip model right-sizing.truedoModelDevelopment
Specifies the type of model, for labelingpurposes. Valid values are:
responsemodelType
response, risk, churn,satisfaction,dissatisfaction.
Specifies whether scorecard modelsshould be fit with logistic
logisticregressionMethod
regression or a (faster) linearapproximation.
Specifies either field or bin levelparameter estimation in the scorecard.
fieldregressionLevel
Bin-level weighting can be moreaccurate on training data but is morelikely to over-fit on test data.
Specifies the right-sizing criterion usedto choose the best number of fields.
minrightSizeCriteria
Valid values are : BIC for BayesianInformation Criteria, AIC for AkaikeInformation Criteria (typically weakerthan BIC in the sense of allowing morefields), delta for diminishing returnsin quality measure for binary outcomes(seeminQualityIncreasePctbelow), min or max for the minimum(or maximum) of all recommendedcriteria (excluding delta in thecontinuous case).
This is a parameter used by thedelta form of
0.5minQualityIncreasePct
rightSizeCriteria: therecommended model is the one withthe most fields such that the worst fieldadds at least the specified amount tothe test-set Gini (or R^2) percentage.
119Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
DescriptionDefaultParameter
Set to true to save intermediate modelsas.qssc and.qmml files, otherwise
falsesaveIntermediateModels
only the final recommendation will besaved.
Specify the name of the prediction fieldfor the final model. If not set, a unique
(automatic)predictionField
field name based on the model type isused, e.g.scorecardwizard__response.
The location to save the focuscontaining the final analysis candidatesand recommended model, if desired.
(none)finalFocus
The location stem to save the finalscorecard model file (in both .qsscand .qmml format), if desired.
(none)finalModel
If both finalModel andfinalFocus are specified, and the
truerunModelSnapshot
option is true, a Model Snapshot(.qshtml file) is created alongsidethe finalModel files.
See also
About the Scorecard Wizard on page 114
qsdecisiontree
Synopsis qsdecisiontree -input <source focus> [-subfocus<subfocus>]
-build <decision-tree build specification> -result <decision-tree buildreport> [-output <destination focus>] [-force]
Description: use a decision-tree build specification to build a decision tree on the specified focus,and create a new decision-tree build report (containing decision-tree statistics, a decision-treeprediction score and a calibration crosstab) in an XML-based file format.
Optional arguments
120Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
EffectOption
See The -force command-line option on page 20.-force
Create this focus, containing an additional derived field thatapplies the new decision-tree prediction score to the inputfocus.
-output <destination focus>
Use the specified subfocus of the focus.-subfocus <subfocus>
Besides these command-line options, qsdecisiontree accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Example: given the decision-tree build specification, decisiontree-specification.xml, withthe following contents:
<?xml version="1.0" encoding="UTF-8"?><decisiontree xmlns="http://www.quadstone.com/xml">
<specification nlevels="3"><objectivefield>
<field name="Age"/></objectivefield><analysiscandidates>
<field name="Gender"/><field name="MaritalStatus"/><field name="HomeOwner"/>
</analysiscandidates><resultfield>
<field name="PredictedAge"/></resultfield>
</specification></decisiontree>
use this specification to create a decision-tree build report decisiontree-report.xml, andcreate a new focus RetailCustAprilPredictedAge.ftr, containing the derived decision-treeprediction field PredictedAge:
qsdecisiontree -build decisiontree-specification.xml-input RetailCustApril.ftr -result decisiontree-report.xml-output RetailCustAprilPredictedAge.ftr
See also
Decision-tree build specification for qsdecisiontree on page 376
qsscorecard on page 122
121Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
qsscorecard
Synopsis qsscorecard -input <source focus> [-subfocus <subfocus>]
-build <scorecard build specification> -result <scorecard build report>[-output <destination focus>] [-force]
Description: use a scorecard build specification to build a scorecard on the specified focus, andcreate a new scorecard build report (containing scorecard statistics, a scorecard prediction scoreand a calibration crosstab) in an XML-based file format.
Optional arguments
EffectOption
See The -force command-line option on page 20.-force
Create this focus, containing an additional derived field thatapplies the new scorecard prediction score to the inputfocus.
-output <destination focus>
Use the specified subfocus of the focus.-subfocus <subfocus>
Besides these command-line options, qsscorecard accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • For a binary objective, qsscorecard uses logistic regression; for a continuous objective, ituses linear regression.
Example
Given the scorecard build specification, scorecard-specification.xml, with the followingcontents:
<?xml version="1.0" encoding="UTF-8"?><scorecard xmlns="http://www.quadstone.com/xml">
<specification modeltype="value" nfields="2" iscontinuous="true">
<objectivefield><field name="Age"/>
</objectivefield><analysiscandidates>
<field name="Gender"/><field name="MaritalStatus"/><field name="HomeOwner"/>
122Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
</analysiscandidates><resultfield>
<field name="PredictedAge"/></resultfield>
</specification></scorecard>
use this specification to create a scorecard build report scorecard-report.xml, and create anew focus RetailCustAprilPredictedAge.ftr, containing the derived scorecard predictionfield PredictedAge:
qsscorecard -build scorecard-specification.xml-input RetailCustApril.ftr -result scorecard-report.xml-output RetailCustAprilPredictedAge.ftr
See also
qsdecisiontree on page 120
Scorecard build specification for qsscorecard on page 379
About the Association Rule Wizard
The Association Rule Wizard automates discovery of association and sequencing rules. This istypically used to explore detailed customer transaction, basket or event data for applications suchas:
Market basket analysis: identify combinations of products that commonly appear (or don't appear)together within a single purchase transaction (basket of items).
Best next product: determine the most common sequence in which products or services arepurchased; for example, banking customers are more likely to open an investment/trading accountafter opening a high-interest savings account.
Marketing impact analysis: identify patterns of marketing communications that are more likely tooccur before a purchase
Data preparation: Association Rule Wizard runs directly against transaction or event data storedin a focus dataset, benefitting from Spectrum Miner's high-performance data access.
• For Market Basket analysis, you should use an item-level dataset containing one record per item.The dataset should contain a basket identifier field that groups items in the same basket, as wellas a product identifier field (binned and categorical) that indicates the product name or code foreach item. The dataset should be sorted by the basket identifier key field.
• For Next Product analysis, you should pick a dataset with one record per product or service. Thedataset should contain a basket identifier — often the customer identifier — that groups productstogether, along with the product identifier (binned and categorical) and a product timestamp (integer
123Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
or date) that indicates the order in which products were acquired. The dataset should be sortedby the basket identifier key field.
Algorithm parameters: Association Rule Wizard is based on the industry-standard apriori algorithmand includes a broad range of functionality to control rule discovery.
You can restrict rule discovery to particular products of interest, and control various parameters thatdefine what an interesting rule is (support, confidence, rule complexity, extended rule selectionmeasures). Note that these settings can impact execution time: in particular, reducing the minimumsupport value will increase run time unless a maximum rule complexity (number of antecedents) isalso specified.
Terminology: Association Rule Wizard uses a variety of terms that might be unfamiliar, including:
Rule: in general, a rule asserts that if a set of conditions is satisfied (the left-hand side, or antecedentof the rule), then some other condition will occur (the right-hand side, or consequent of the rule). Inassociation and sequencing rules, the conditions in the rule correspond to the presence of particularitems in the basket. For example, a simple rule might be: if a basket contains bread, then it will alsocontain jam.
Antecedent: the left-hand side of a rule, i.e. the set of items that must appear in the basket for therule to be relevent.
Consequent: the right-hand side of a rule, i.e. the set of items that the rule predicts will be presentwhen the left-hand side (antecedent) is satisfied.
Support: the (absolute) support of an item set is the number of baskets that contain all items in theset. Relative support is the same number expressed as a percentage of the total number of baskets.
Rule Support: the support of a rule is the support of the rule's antecedent, i.e. the proportion ofbaskets for which the left-hand side of the rule applies. This measures how often the rule is relevant.
Rule Confidence: the confidence of a rule is the support of the set of all items that appear in therule (on left or right), divided by the support of the rule's antecedent. It measures the likelihood thatthe rule's consequent (right-hand side) is true given that the antecedent of the rule is satisfied, i.e.the likelihood that the rule is correct.
Absolute Confidence Difference to Prior: the difference in the posterior and the prior confidencemeasures (with and without the antecedent items). This measures how much more (or less) likelythe rule's consequent is when the left-hand side of the rule is true. For example, the rule "if bread,then jam" is not very interesting if it's confidence is similar to the rule "if (anything) then jam".
Lift Value: the ratio of the posterior and the prior confidence measures (with and without theantecedent items).
Results: various outputs are (optionally) generated, including:
• An interactive HTML model report summarizing the rules created, their metrics and the option tofilter on interesting rules.
• A scored new dataset creating an aggregated dataset (one record per basket identifier) indicatingwhether each selected rule is satisfied for that group of records.
• The rules as deployable code in Transaction Measurement Language format.
124Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
qsruleminer
Synopsisqsruleminer -ini <configuration file> -output <rules file> [-force]
Description: find association or sequencing rules.
Parameters: the configuration file controls how the rule mining process executes. Each line of thefile can specify one of the options listed below. Lines starting with # or lacking an equals sign ( =) are ignored. (Note trailing comments are not supported.) A line starting with eof causes the restof the file to be ignored.
For example, the following ini file would seek rules displaying lift as the additional measure withat least three and at most 10 items, with minimum confidence of 20% and minimum support of 5%,and minimum lift of 25%:
focus = C:/association/MarketBasket.ftr
task = market basket
key fields = TRANS_NO
product field = ProductName
options = -tr -el -d25 -s5.0 -c20.0 -n10 -m3
eof
DescriptionDefaultParameter
The location of the input focus.(required)focus
Specifies the discovery task. Validvalues are: segment, marketbasket, next product.
(required)task
Specifies a key field or composite key(comma-separated list of fields) that
(required)key fields
define a basket. The input datasetshould be in sorted order with respectto the key.
Specifies the product field identifier. Thefield should be binned and categorical.
(required)product field
Each bin defines a separate product(so a categorical binning hierarchy canbe used to work at different levels of
125Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
DescriptionDefaultParameter
detail on the same underlying productdata). Composite keys are notsupported.
Comma-separated list of items thatrestricts the items that can appear on
(none)rhs
the right-hand side (consequent) of arule. Note that no excess whitespaceis permitted — - the labels itemA ,itemB and itemC need to matchexactly the corresponding bin labels forthe product field.
(next product task only) Specifies thefield to use as the timestamp for each
(none)timestamp field
item. Must be integer or date type.Composite time-stamp keys are notsupported.
(segment task only) Specifies acomma-separated list of fields to use in
(none)segment fields
the segment discovery task. Bin labelswithin the specified fields willcorrespond to items in the discoverytask.
Pass everything after the equal signdirectly to the apriori algorithm.
(none)options
Support apriori options include (also seehttp://www.borgelt.net/doc/apriori/apriori.html#options):
DescriptionOption
Discover association rules-tr
Minimum support value (default: 10%)-s<#>
Maximum support value: positive: percentage; negative:absolute number (default: 100%)
-S<#>
Minimum confidence value (default: 80%)-c<#>
126Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
DescriptionOption
Minimum number of items per rule (default: 1)-m<#>
Maximum number of items per rule (default: no limit)-n<#>
Do not use an additional rule evaluation measure-ex
Minimum value for the additional rule evaluation measure(default 10%)
-d<#>
To specify the additional rule evaluation measure:
Rule Confidence-ec
Absolute Confidence Difference to Prior-ed
Lift Value-el
Difference of Lift Value to 1-ea
Difference of Lift Quotient to 1-eq
Conviction-ev
Difference of Conviction to 1-ee
Difference of Conviction Quotient to 1-er
Certainty Factor-ef
Normalized Chi Squared Measure-en
P-value from Chi Squared Measure-ep
Information Difference to Prior-ei
P-value from G Statistic-eg
Binary Log of Support Quotient-eb
127Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for building models
10 - Commands forworking with QMMLfiles
In this section
qsqmmlview 129qsqmmledit 129qslt 132qsqmml2sas 134qsqmml2sql 134qsqsfmtosas 135
qsqmmlview
Synopsis qsqmmlview -input <source QMML file>
[-prettyprint | -rulesets | -ruleset <ruleset>]
Description: display information about the source QMML file: show the contents as neatly-formattedXML (the -prettyprint option). Show summary information for all the rulesets in the file (the-rulesets option); or show detailed information for a specified ruleset (the -ruleset option).
By default, qsqmmlview shows detailed information for all rulesets in the file.
Optional arguments: besides the required arguments, qsqmmlview accepts the options commonto all data-build commands [see Standard command-line options on page 18].
Note: • If you specify the -rulesets or the -ruleset option, and the QMML file is not of theqmml:rules form, qsqmmlview fails.
• If you use the -ruleset option, and the QMML file does not contain a ruleset with thespecified name, qsqmmlview fails.
• For information on QMML rules and rulesets, seeQuadstoneMetadataMarkup Language.
Examples
Check which (if any) of the rulesets in the QMML rules document model.qmml are live:
qsqmmlview -input model.qmml -rulesets
View the names and types of the input and output fields for the (compiled) ruleset OutcomeRulesin the QMML rules document model.qmml:
qsqmmlview -input model.qmml -ruleset OutcomeRules
See also
XML in Spectrum Miner on page 356
qsqmmledit on page 129
qsqmmledit
Synopsis qsqmmledit -input <source QMML file> [-output <destination QMMLfile>]
129Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for working with QMML files
-addrule <rules file>
qsqmmledit -input <source QMML file> [-output <destination QMML file>]-addruleset <ruleset> <rule> [<rule> ...]
qsqmmledit -input <source QMML file> [-output <destination QMML file>]
-deleteruleset <ruleset> qsqmmledit -input <source QMML file>
[-output <destination QMML file>]
-parse
[-s]
qsqmmledit -input <source QMML file> [-output <destination QMML file>]
{-compile | -compile <ruleset> [<ruleset> ...] }
[-l] [-s]
qsqmmledit
-input <source QMML file> [-output <destination QMML file>]
{-setlive | -setlive <ruleset> | -setlive <ruleset> true | -setlive<ruleset> false}
Description: modify the source QMML file (if the -output option is not used) performing one ofthe following operations:
• Add a rule from the rules file, which is either a standard XML file containing <field> elements,or a QMML FDL interchange file containing rules in <fdl-derivation> and <fdl-function>elements.
An unnamed rule is given the name Rule. If the source QMML file is not of the qmml:rules form,or already contains a rule with the same name as any of the rules in the rules file, qsqmmleditfails.
• Add a ruleset containing one or more rules (already present as <field> elements in the sourceQMML file).
If the source QMML file is not of the qmml:rules form, or already contains a ruleset with thespecified ruleset name, or does not contain rules with all of the specified names, qsqmmleditfails.
• Delete a ruleset (but not the rules that it contains).
If the source QMML file is not of the qmml:rules form, or does not contain a ruleset with thespecified name, qsqmmledit fails.
• Parse the file.
If you use File->Export as QMML in Decision Studio to create the source QMML file, SpectrumMiner parses the QMML file automatically. In this case, you do not need to use the -parsecommand. Otherwise, the -parse command is a prerequisite for compilation.
130Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for working with QMML files
If the source QMML file is not of the qmml:rules form, qsqmmledit does nothing (unless youuse the -s option, in which case it fails). If the source QMML file is of the qmml:rules form,qsqmmledit attempts to parse all unparsed rules. If a rule cannot be parsed, qsqmmleditproduces a warning message and continues with the next rule (unless you use the -s option, inwhich case it fails).
• Compile all rulesets or the specified rulesets (a prerequisite for deployment).
If the source QMML file is not of the qmml:rules form, qsqmmledit does nothing (unless youuse the -s option, in which case it fails). If a rule is not parsed, qsqmmledit fails.
• Mark all rulesets as live (ready for deployment), or mark a specified ruleset as live (true/default)or not (false).
If the source QMML file is not of the qmml:rules form, or does not contain a ruleset with thespecified name, or the specified ruleset is not compiled, qsqmmledit fails.
Optional arguments
EffectOption
As each ruleset is compiled, mark it as live.-l
Instead of modifying the source QMML file, create thespecified destination QMML file, and apply the requestedchanges to it.
-output <destination QMML file>
Treat the file strictly, disallowing empty rulesets. When usedwith the -parse option, fail if the source QMML file is not
-s
of the qmml:rules form, or if any unparsed rule in thesource file cannot be parsed. When used with the-compile option, fail if the source QMML file is not ofthe qmml:rules form.
Besides these command-line options qsqmmledit accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note: • For information on QMML rules and rulesets, seeQuadstoneMetadataMarkup Language.• A rule can only be parsed if it is supported by the RealTime variant of FDL.
Examples
Add rules from the QMML FDL interchange file model.xfdl to the QMML rules documentmodel.qmml:
qsqmmledit -input model.qmml -addrule model.xfdl
131Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for working with QMML files
Assign a rule Outcome in the QMML rules document model.qmml to a new ruleset OutcomeRules:
qsqmmledit -input model.qmml-addruleset OutcomeRules Outcome
Deploy the QMML rules document model.qmml, by parsing all embedded FDL, compiling allrulesets, and marking the ruleset OutcomeRules as live:
qsqmmledit -input model.qmml -parseqsqmmledit -input model.qmml -compileqsqmmledit -input model.qmml -setlive OutcomeRules
See also
XML in Spectrum Miner on page 356
qslt on page 132
qsqmmlview on page 129
qslt
Synopsis qslt {-input <source file> | -input -}
[-output <destination file> | -output -]
[[-source <source type>] [-target <destination type>] | -transform <namedtransformation>]
[-spec <specification file>]
Description: transform the specified source file or standard input (with -input -) to a differentformat, writing the result to standard output by default.
By default, qslt deduces the source and destination file types from the filename extensions (ifpresent), and from the file contents in the case of an XML source file. It then attempts to find andapply a transformation that matches these types. If none exists, or more than one possibletransformation exists, the command does nothing except issue an error message and fail.
The standard transformations matching source and destination file types are:
• From fdl (an FDL expression, or a set of TML create statements) to qmml:expressions (fullyparsed FDL in QMML)
• From fdl (an FDL expression, or a set of TML create statements) to qmml:x-fdl (QMML FDLinterchange format) and vice versa
• From qmml:expressions (fully parsed FDL in QMML) to qmml:x-fdl (QMML FDL interchangeformat)
• From qmml:rules (QMML rules format) to fdd (flat-data description format)
132Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for working with QMML files
The remaining standard transformation is the named transformation prettyprint, which transformsarbitrary XML into neatly-formatted XML (with line breaks and indentation). You can only choose itby using the -transform option.
Optional arguments
EffectOption
Write the transformed data to this destination file (overwritingany existing file of the same name).
-output <destination file>
Write the transformed data to standard output (the defaultif you do not specify a -output option).
-output -
Consider only transformations matching this source type(overriding a deduced source type, if any). You must specifythis option if qslt is reading from standard input.
-source <source type>
Locate the required transformation in this file, ignoring thestandard transformations and any system-wide oruser-specified customizations.
-spec <specification file>
Consider only transformations matching this destinationtype (overriding a deduced destination type, if any). You
-target <destination type>
must specify this option if qslt is writing to standardoutput.
Use the named transformation instead of selecting atransformation on the basis of source and destination types.
-transform <named transformation>
Besides these command-line options qslt accepts the options common to all data-build commands[see Standard command-line options on page 18].
Note: • When converting between the fdl type and QMML, TML create statements correspondto named field derivations in QMML, while single FDL expressions correspond to unnamedfield derivations in QMML.
• The conversion from qmml:rules to the fdd type omits derived fields.• As well as the standard transformation types, you can define custom transformations, on
a system-wide or per-user basis. Custom transformations can accept additionalcommand-line options.
• For details of the QMML forms qmml:expressions, qmml:rules, and qmml:x-fdl,see Quadstone Metadata Markup Language.
Examples
133Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for working with QMML files
Transform the field metadata represented by the FDD file model.fdd into an "empty" QMML rulesdocument model.qmml (describing a set of fields, but containing no rules), using a user-specifiedtransformation described in the file ext/qmml/qslt.xml:
qslt -source fdd -target qmml:rules -input model.fdd-output model.qmml -spec ext/qmml/qslt.xml
Transform an FDL expression in the file model.fdl to QMML FDL interchange format, writing theresult to the file model.xfdl:
qslt -source fdl -target qmml:x-fdl -input model.fdl-output model.xfdl
See also
XML in Spectrum Miner on page 356 qsqmmledit on page 129
qsqmml2sas
Synopsis qsqmml2sas <QMML file>
Description: convert a QMML file to a SAS file that has a separate SAS statement for each fieldderivation.
Note: • The command is experimental.• You can access most of the functionality of the data-build command qsqmml2sas by using
the Convert to SAS dialog box in Spectrum Miner.
Example
Convert model.qmml to model.sas:
qsqmml2sas model.qmml
qsqmml2sql
Synopsis qsqmml2sql <QMML file>
{Oracle | Teradata V2Rx | MS SQL Server}
<source table>
[<field> [, <field> ... ]]}
134Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for working with QMML files
Description: use a supported database type and a source table to convert a QMML file to an SQLfile that has a separate SQL SELECT statement for each field derivation.
Optional arguments
EffectOption
Converts only derived fields.<field> [, <field> ... ]
Note: • The command is experimental.• You can access most of the functionality of the data-build command qsqmml2sql by using
the Convert to SQL dialog box in Spectrum Miner.
Example
Convert model.qmml to model.sql, create SELECT statements for Microsoft SQL Server, anduse prod.customer as the source table for any scoring.
qsqmml2sas model.qmml "MS SQL Server" prod.customer
qsqsfmtosas
Synopsis qsqsfmtosas <metadata file>
Description: convert derivations from a metadata file to a SAS file that has a separate SAS statementfor each field derivation.
Note: • The command is experimental.• You can access most of the functionality of the data-build command qsqsfmtosas by
using the Convert to SAS dialog box in Spectrum Miner.
Example
Convert metadata.qsfm to metadata.sas:
qsqsfmtosas metadata.qsfm
135Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Commands for working with QMML files
11 - Other commands
In this section
qsmapgen 137
qsmapgen
Synopsis qsmapgen
-input <categorical hierarchy> [-input < categorical hierarchy> ...]
-name <map name> -output <directory> [-overwrite]
[-style radial_drill | -style radial_layers]
Description: from one or more categorical hierarchy files (typically with filename extension .hrc),create a representation of the categorical hierarchies suitable for use in the Decision Studio MapViewer, with a region corresponding to each category. Create the map or maps in the specifieddirectory, using the specified map name as the basis for filenames.
Optional arguments
EffectOption
Allow new map files specified using the -name and-output options to overwrite existing files.
Without the -overwrite option, if any of the map filesto be created already exists, the command does nothing(except issue a warning).
-overwrite
Create a hierarchy of maps suitable for drilling down in theDecision Studio Map Viewer, with the top-level map (shown
-style radial_drill
initially) based on the coarsest-grained binning in eachcategorical binning hierarchy (the default if you do notspecify a -style option).
Create an independent map for each level in the categoricalbinning hierarchies, so that a Decision Studio user can
-style radial_layers
choose the degree of coarseness at which to view fields inthe Map Viewer, though without the option to drill down.
Besides these command-line options, qsmapgen accepts the options common to all data-buildcommands [see Standard command-line options on page 18].
Note:
• If a categorical hierarchy file does not specify a correct hierarchy, qsmapgen fails.• You should not use a field value in more than one categorical hierarchy file. If you do, the behavior
of qsmapgen is undefined.
137Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Other commands
• If you specify a category name in more than one categorical hierarchy file, or in more than onelevel of the same file, qsmapgen adds one or more suffixes to the region names to ensureuniqueness.
• For a map produced using -style radial_drill (or no specified -style option), the top-levelmap consists of .mif and .mid files, named using the specified map name. Drill-down maps forthe next level of detail are contained in the next subdirectory. Maps at the subsequent level ofdetail (if any) are contained in the next subdirectory of that directory, and so on.
• The maps produced using -style radial_layers consist of pairs of .mif and .mid files,each named using the given map name together with a numeric suffix corresponding to the levelwithin the hierarchy.
• Using SpectrumMiner, you can access most of the functionality of qsmapgen through theCreateCategory Map dialog box .
Examples: given the categorical hierarchy file Referrer.hrc, containing the following:
B, Banner Ad, Marketing
D, Direct, Direct
E, E-mail Link, Marketing
G, Google, Search Engine
M, MSN, Search Engine
Y, Yahoo, Search Engine
L, Lycos, Search Engine
and the categorical hierarchy file PurchasePage.hrc, containing the following:
01, Menswear
02, Womens
03, Kids
04, Home
create a map file clickthru.mif (and supporting files) suitable for drilling down, in a new directorywebmaps:
qsmapgen -input Referrer.hrc -input PurchasePage.hrc-output webmaps -name clickthru
The top-level map has the following regions (where "spikes" on the circumference indicate regionsthat support drilldown to the next level):
138Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Other commands
Drilldown on the "SearchEngine" region produces a map with the following regions:
Alternatively, create independent map files clickthru_layer_1.mif andclickthru_layer_2.mif (and supporting files) in the directory webmaps:
qsmapgen -input Referrer.hrc -input PurchasePage.hrc-output webmaps -name clickthru_layer -style radial_layers
The clickthru_layer_1.mif map has the following regions:
139Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Other commands
and the clickthru_layer_2.mif map has the following regions:
140Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Other commands
12 - TransactionMeasurementLanguage
In this section
About Transaction Measurement Language 142TML syntax 142Reserved words in TML 143
About Transaction Measurement Language
Transaction Measurement Language (TML) is a collection of lightweight syntactic wrappers aroundFDL expressions, used to specify field derivations and aggregations in conjunction with the DeriveFields andAggregate Recordswizards, and theSpectrumMiner data-build commands qsderive,qsmeasure, qstrack and qsselect.
See also
About Field Derivation Language on page 179
qsderive on page 60
qsmeasure on page 63
qsselect on page 70
qstrack on page 67
TML syntax on page 142
TML syntax
A derivations, trackers, selections, or aggregations file contains one or more TML create statements,while a statistics file contains one or more TML calculate statements.
• TML is a case-sensitive language:
• TML keywords (create, by, string etc.) are written in lowercase letters, with the exceptionof STATISTIC.
• The names of all aggregation functions [see Aggregation functions for measurements andderivations on page 156] are written in lowercase letters.
• The names Average, average, and AVERAGE all refer to different fields or statistics.
• Names for fields and statistics must begin with a letter ("A" — "Z" or "a" — "z"), contain only letters,digits, and underscore ("_"), and be no longer than 128 characters; optionally quoted in singlequotation marks, for example, 'CustomerID'. You must enclose the name of a field or statisticin single quotation marks if it coincides with a TML or FDL reserved word, or differs from one onlyin case.
• TML syntax is described using railroad diagrams like this:
142Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Transaction Measurement Language
As long as you follow the arrows and don't backtrack, any path through a railroad diagram leadsto syntactically valid (though not necessarily meaningful) TML:
• Words in oval/circular boxes (like create) are TML keywords; you should type them exactly asthey appear.
• You should also type punctuation in oval boxes (like ":" and ":=") exactly as it appears (youcan't have a space between ":" and "=" in ":=").
• Don't forget the semicolon that ends each TML statement.• Terms in rectangular boxes (like expression) refer to other railroad diagrams or to forms described
elsewhere in words; you should not type these literally.
• TML is a freely formatted language: you can break lines or insert extra spaces between syntacticelements (in oval/circular boxes in railroad diagrams) without affecting the interpretation of aderivations, trackers, selections, aggregations, or statistics file.
• As well as introducing spaces between syntactic elements, you can include comments, whichbegin with a double slash ("//") and continue to the end of the same line.
• Wherever TML syntax requires an expression, you can use any FDL expression [see Expressionson page 183], except that an expression involving a TML reserved word, a semicolon, or braces{...} must be enclosed in parentheses (...).
See also
About Transaction Measurement Language on page 142
Evaluating focus statistics: the calculate statement on page 153
Field definition: the create statement on page 146
Reserved words in FDL on page 206
Reserved words in TML on page 143
Reserved words in TML
The following tokens are reserved words in TML:
aggregation, all, and, as, auto, automatic, bin, binning, bool, boolean,by, calculate, constant, create, crosstab, date, default, description,dimension, double, drl, enum, except, fields, file, float, format, from,full, function, group, include, inplace, input, int, integer, join, left,long, merge, meta, modifiable, normal, on, open outer, outfile, output,parameter, pragma, real, ref, reference, rename, right, separate_functions,short, show, show_empty_segments, show_summaries, sort, state, statistic,strbytes, string, suppress, table, temp, temporary, track, tracker, view,where
143Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Transaction Measurement Language
To use a reserved word, or a word that differs only in case from a reserved word, as a field or statisticname in TML, or as an identifier [see Expressions on page 183] in FDL, you must enclose it in singlequotation marks — for example, 'Date'.
See also
About Transaction Measurement Language on page 142
Reserved words in FDL on page 206
144Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Transaction Measurement Language
13 - TML statements
In this section
Field definition: the create statement 146Using aggregation functions and the where and default clauses 148Splitting aggregations: the by clause 150Evaluating focus statistics: the calculate statement 153
Field definition: the create statement
The create statement defines a field by derivation or aggregation, depending on the context inwhich it appears.
Derivations: the derivations file for qsderive, or the equivalentDerive Fieldswindow, the trackersfile for qstrack, and the selections file for qsselect must contain create statements of thefollowing form:
The aggregations file for qsmeasure, or the equivalent Aggregate Records window can optionallycontain create statements of the following form:
In both cases, the create statement defines a field called name in terms of an FDL expression[see Expressions on page 183]. The datatype of the field is the datatype of the expression, unlessyou override it by specifying a compatible type [see Datatypes on page 181].
• For each successive create statement in the derivations file, qsderive, or the Derive Fieldswindow derives a field in the output focus using the given expression, which can refer both to fieldsin the source focus and to fields derived earlier in the derivations file.
• For each successive create statement in the trackers file, qstrack derives a field using thegiven expression, which can refer to fields in the source focus and to fields derived earlier in thetrackers file, as well as to focus statistics defined in any statistics file. The expression typicallyinvolves state variables [see Variables on page 187].
• Given a selections file, qsselect derives a field corresponding by default to the first createstatement and ignores any other statements in the file. The defining expression must be of boolean(or compatible) type.
• For each successive create statement of this form in the aggregations file, qsmeasure, or theAggregate Records window derives a field in the output focus using the given expression, whichcan refer to fields derived earlier in the aggregations file, and to focus statistics defined in anystatistics file.
• A temporary statement has the same effect as a create statement, except that the field doesn'tappear in the output focus. However you can still refer to the temporary field in subsequentexpressions in the aggregations file.
Aggregations The aggregations file for qsmeasure or the equivalentAggregate Recordswindowcan contain create statements of the following form:
146Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
TML statements
For each create statement of this form in the aggregations file, qsmeasure or the AggregateRecords window effectively derives a virtual field corresponding to the expression following theaggregation function. This expression can refer to fields in the source focus.
After deriving the virtual field, qsmeasure or theAggregate Recordswindow applies the aggregationfunction [see Using aggregation functions and the where and default clauses on page 148].
The default datatype of the resulting field(s) depends on the aggregation function and the values,operators, and functions involved in the expression to which it applies, but you can override it byspecifying a compatible type.
A temporary statement has the same effect as a create statement, except that the field doesn'tappear in the output focus. However you can still refer to the temporary field in subsequentexpressions in the aggregations file.
Note: • In qsderive, or Derive Fields derivations files, qsselect selections files, and qstracktrackers files, the expression can involve field statistics, global variables, and functionsdefined in terms of these. These FDL features are not available in a qsmeasure orAggregate Records aggregations file.
• You must put extra parentheses around the expression if it contains a semicolon, braces,or a TML reserved word [see Reserved words in TML on page 143].
• Each create statement must start on a new line.
Examples
Given the derivations file, derivations-customer.tml, containing the following:
// Length of standing of customer in monthscreate CustMonths := countwholemonths(StartDate, today());// A key field of extended lengthcreate CustomerID_24 : string(24) := CustomerID;
Apply these derivations to the focus RetailCustApril.ftr to create a new focusRetailCustAprilDeriv.ftr:
qsderive -derivations derivations-customer.tml-input RetailCustApril.ftr -output RetailCustAprilDeriv.ftr
147Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
TML statements
Given the selections file, selections-customer.tml, containing the following:
// Select male customers aged between 18 and 65 inclusive.create Selection := Gender = 1 and Age >= 18 and Age <= 65;
Apply this selection to the focus RetailCustApril.ftr to create a new focusRetailCustAprilSelection.ftr:
qsselect -selections selections-customer.tml-input RetailCustApril.ftr%-output RetailCustAprilSelection.ftr
Given the aggregations file, aggregations-mostRecent.tml, containing the following:
temporary LastPurchase := max(PurchaseDate);create AvgTransVal := mean(Amount - 0.01 * PointsRedeemed);create NumTrans := count();create MonthsSinceLastPurchase :=countwholemonths(LastPurchase, today());
Apply these aggregations to the focus RetailTransApril.ftr to create a customer-orientedfocus RetailAggRecentApril:
qsmeasure -aggregations aggregations-mostRecent.tml-input RetailTransAprilSorted.ftr -output RetailAggRecentApril%-keys CustomerID
See also
Aggregation functions for measurements and derivations on page 156
qsderive on page 60
qsmeasure on page 63
qsselect on page 70
qstrack on page 67
Splitting aggregations: the by clause on page 150
Using aggregation functions and the where and defaultclauses
You can aggregate records using qsmeasure or the equivalent Aggregate Records window.
148Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
TML statements
When you use a create statement in an aggregations file [see Field definition: the createstatement on page 146], qsmeasure or theAggregate Recordswindow effectively derives a virtualfield corresponding to the expression in parentheses following the aggregation function.
Conceptually, qsmeasure or the Aggregate Records window divides the records in the sourcefocus — augmented by the virtual field — into groups of records with the same key-field value. Foreach group of records (corresponding to a single key-field value), qsmeasure or the AggregateRecords window applies the aggregation function to the values of the virtual field within thoserecords, to produce a single result value.
The result values for all the groups taken together consitute a new field, with the name given in thecreate statement — unless the create statement contains a by clause [see Splittingaggregations: the by clause on page 150].
Within each group of records with a common key-field value, certain aggregation functions ignorerecords for which the virtual field contains null [see The null value on page 181].
If the create statement contains a where clause, the aggregation function ignores records forwhich the associated expression evaluates to 0 ("false"). This expression can refer to fields in thesource focus.
Because aggregation functions can ignore some of the records in a group with a common key-fieldvalue, it's possible for all the records in the group to be skipped. Many aggregation functions returnthe null value in this case. By including a default clause, you can specify a constant expressionto be evaluated instead of the aggregation in the event that it returns null.
When you use the sum aggregation function [see sum (one argument) on page 175], youshould usually specify a default sum of 0. If you don't use a default clause, the sum overan empty record selection is null, which is unlikely to be what you want.
Note:
Examples Given the aggregations file, aggregations-purchaseSummary.tml, containing thefollowing:
create firstPurchaseDate := first(PurchaseDate);create commonPaymentMethod := mode(PaymentMethod);create Spend := sum(Amount) default 0;create SpendInStore800 := sum(Amount)
149Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
TML statements
where Store = "0800" default 0;create proportionSpendInStore800 := SpendInStore800 / Spend;create NumTransCRRedeemed := count()
where PaymentMethod = "CR" and not isnull(PointsRedeemed);
Apply these aggregations to the focus RetailTransApril.ftr to create a customer-orientedfocus RetailAggSummaryApril:
qsmeasure -aggregations aggregations-purchaseSummary.tml-input RetailTransAprilSorted.ftr -output RetailAggSummaryApril-keys CustomerID
See also
Aggregation functions for measurements and derivations on page 156
Expressions on page 183
Splitting aggregations: the by clause
The aggregations file for qsmeasure, or the equivalent Aggregate Records window contains asequence of create statements [see Field definition: the create statement on page 146]. As partof such a create statement, you can optionally include a by clause, to break your aggregationdown according to the application of either a split function or the bin function to an expressionderived from other fields.
For each possible integer i returned by the split function or bin function, qsmeasure, or theAggregate Records window creates a separate field, containing values aggregated from just thoserecords for which the function application evaluates to i. The name of each of these related fieldsis formed from the name in the create statement together with a suffix that depends on the functionused for splitting the aggregation.
• The following built-in split functions are available (with given field-name suffixes corresponding tosuccessive function values):
Field-name suffixesFunction
"_1," "_2," "_3," ..., "_31"day
"Sun," "Mon," "Tue," ..., "Sat"dayofweek
"0000," "0100," "0200" ..., "2300"hour
150Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
TML statements
Field-name suffixesFunction
"NonNull," "Null"isnull
"MisMatch," "Match"match
"NotMember," "Member"member
"0," "1," "2," ..., "59"minute
"Jan," "Feb," "Mar," ..., "Dec"month
"0," "1," "2," ..., "59"second
"Negative," "Zero," "Positive"sgn
"NotMember," "Member"strmember
"_1," "_2," "_3," ..., "_53"weekofyear
• You can use the bin function to apply an FDL binning to an expression. Used as a split function,the bin function gives rise to field-name suffixes "_1," "_2," ..., "_n," "_null," and "_other" (wheren depends on the number of bins in the binning).
• You can also define your own split function [see User-defined functions on page 190]. Theelement_names attribute defines the field-name suffixes corresponding to the function returnvalues.
Examples Given the aggregations file, aggregations-builtinSplit.tml, containing thefollowing:
create numberPurchases := count();create numberPurchasesOn_ := count() by dayofweek(PurchaseDate);create proportionPurchases_Mon :=numberPurchasesOn_Mon / numberPurchases;create purchaseAmountIn_ :=sum(Amount) by month(PurchaseDate) default 0;
Apply these aggregations to the focus RetailTransApril.ftr to create a customer-orientedfocus RetailDateSplitsApril.ftr containing fields CustomerID, numberPurchases,numberPurchasesOn_Sun, numberPurchasesOn_Mon, ..., numberPurchasesOn_Sat,proportionPurchases_Mon and purchaseAmountIn_Jan, purchaseAmountIn_Feb,..., purchaseAmountIn_Dec:
qsmeasure -aggregations aggregations-builtinSplit.tml -keys CustomerID -input RetailTransApril.ftr-output RetailDateSplitsApril.ftr
151Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
TML statements
Given the aggregations file, aggregations-binSplit.tml, containing the following:
create numberPurchasesAmountBand :=count() by bin(EqualRange(20,100,4),Amount);create numberPurchasesDateBand := count() bybin(PrePost(todate(19990401), todate(19990501),todate(19990601)), PurchaseDate);create ratioPrePost :=numberPurchasesDateBand_3 / numberPurchasesDateBand_2;
Apply these aggregations to the RetailTransApril.ftr focus to create a customer-orientedfocus, RetailRangeSplitsApril.ftr, containing fields CustomerID,numberPurchasesAmountBand_1, numberPurchasesAmountBand_2, ...,numberPurchasesAmountBand_6, numberPurchasesAmountBand_null,numberPurchasesAmountBand_other (corresponding to two end bins, four internal bins andthe null and unclassified bin), numberPurchasesDateBand_1, numberPurchasesDateBand_2,..., numberPurchasesDateBand_4, numberPurchasesDateBand_null,numberPurchasesDateBand_other and ratioPrePost:
qsmeasure -aggregations aggregations-binSplit.tml -keys CustomerID -input RetailTransApril.ftr-output RetailRangeSplitsApril.ftr
Given the FDL functions file, fdl-functions-storeGroups.fdl, containing the following:
function StoreGroupFunction( Store )[
element_names = "North, South, East, West"]{case
Store = "0000" or Store = "0800" : 1;Store = "0300" or Store = "0600" or Store = "0700" : 2;Store = "0100" or Store = "0400" : 3;default : 4;
}
Given the aggregations file, aggregations-userDefinedSplit.tml, containing the following:
create totalPurchases := sum(Amount);create totalPurchasesByStore_ :=sum(Amount) by StoreGroupFunction( Store );create proportionPurchases_North :=totalPurchasesByStore_North / totalPurchases;
Apply these aggregations to the RetailTransApril.ftr focus to create a customer-orientedfocus, RetailGroupSplitsApril.ftr, containing fields CustomerID, totalPurchases,totalPurchasesByStore_North, totalPurchasesByStore_South,
152Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
TML statements
totalPurchasesByStore_East, totalPurchasesByStore_West andproportionPurchases_North:
qsmeasure -aggregations aggregations-userDefinedSplit.tml-library fdl-functions-storeGroups.fdl-keys CustomerID-input RetailTransApril.ftr -output RetailGroupSplitsApril.ftr
See also
Aggregation functions for measurements and derivations on page 156
Expressions on page 183
Using aggregation functions and the where and default clauses on page 148
Evaluating focus statistics: the calculate statement
The statistics file for qsmeasure, the equivalentAggregate Recordswindow, or qstrack containscalculate statements of the following form:
The calculate statement defines a focus statistic called name in terms of an aggregation functionapplied to an FDL expression [see Expressions on page 183], which can refer to fields in the sourcefocus.
As in the case of a measurement [see Using aggregation functions and the where and defaultclauses on page 148], qsmeasure, qstrack, or the Aggregate Records window effectivelyderives a virtual field corresponding to the expression in parentheses following the aggregationfunction.
The calculate statement gives rise to a single result value, obtained by applying the aggregationfunction to all the values in the virtual field. (In contrast, a create statement in an aggregations filegives rise to a field, in which each value is obtained by applying the aggregation function to thevalues in the virtual field for a group of records.)
The default datatype of a focus statistic depends on the aggregation function and the values,operators, and functions involved in the expression to which it applies, but you can override it byspecifying a compatible type [see Datatypes on page 181].
153Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
TML statements
If the calculate statement contains a where clause, the aggregation function ignores records forwhich the associated expression evaluates to 0 ("false"). This expression can refer to fields in thesource focus.
Once you have defined a focus statistic in the statistics file, you can refer to it in expressions in theaggregations or trackers file using the special form STATISTIC.name. You can use this formanywhere that you can refer to a field in the source focus or trackers file.
Examples Given the statistics file, statistics-amount.tml, containing the following:
calculate averageAmount := mean(Amount);
Given the aggregations file, aggregations-statistic.tml, containing the following:
create averageSpend := mean(Amount);create bigSpender := averageSpend > STATISTIC.averageAmount;
Apply these aggregations to the focus RetailTransApril.ftr to create a customer-orientedfocus RetailAprilBigSpenders.ftr containing fields CustomerID, averageSpend, andbigSpender:
qsmeasure -statistics statistics-amount.tml-aggregations aggregations-statistic.tml-keys CustomerID-input RetailTransApril.ftr -output RetailAprilBigSpenders.ftr
See also
Aggregation functions for measurements and derivations on page 156
154Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
TML statements
14 - Aggregationfunctions
In this section
Aggregation functions for measurements and derivations 156any 159confintlower 160confintupper 160count 161countnonnull/countnonnulls 162countnull/countnulls 163countunique 164countuniquenonnull 164first 165last 166max (one argument) 167mean (one argument) 168median 168min (one argument) 169mode 170moderatio 171percentage 172percentagerate 172segindex 173significance 174stdev 175sum (one argument) 175variance 176
Aggregation functions for measurements and derivations
There are a number of aggregation functions available for use in measurements and derivations.
To use a function in a measurement context means to use it in an aggregations file [see Usingaggregation functions and the where and default clauses on page 148] or statistics file [seeEvaluating focus statistics: the calculate statement on page 153] for qsmeasure or the equivalentAggregate Records window, or in a statistics file for qstrack..
To use a function in a derivation context means to use it in a Decision Studio derivation, in thederivations file for qsderive or the equivalentDerive Fieldswindow, in the trackers file for qstrack,or in a selections file for qsselect..
Except where otherwise noted, an aggregation function takes a single numeric, string, or date fieldas an argument.
Aggregation functions for both measurements and derivations You can use the followingaggregation functions in both measurement and derivation contexts:
ResultFunction
The number of records in the group/segment
(This function does not take an argument.)
count
The maximum non-null value in the field for thegroup/segment
(In a derivation context, this function does not accept astring-valued expression as an argument.)
max [see max (one argument) on page 167]
The mean (common average) of the non-null values in thefield for the group/segment
(This function requires a numeric expression as anargument.)
mean [see mean (one argument) on page 168]
The minimum non-null value in the field for thegroup/segment
(In a derivation context, this function does not accept astring-valued expression as an argument.)
min [see min (one argument) on page 169]
156Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
ResultFunction
The standard deviation of the non-null values in the field forthe group/segment
(This function requires a numeric expression as anargument.)
stdev
The sum of the non-null values in the field for thegroup/segment
(This function requires a numeric expression as anargument.)
sum [see sum (one argument) on page 175]
The variance of the non-null values in the field for thegroup/segment
(This function requires a numeric expression as anargument.)
variance
Aggregation functions for measurements only You can use the following aggregation functionsin a measurement context only:
ResultFunction
A non-null value in the field, drawn from an arbitrary non-nullrecord in the group/segment
any
The lower bound of a 95% confidence interval for the meanof the field in the group
confintlower
The upper bound of a 95% confidence interval for the meanof the field in the group
confintupper
The number of records in the group for which the field hasnon-null values [see The null value on page 181]
In derivations, use countnonnulls instead.
countnonnull [see countnonnull/countnonnulls on page162]
The number of records in the group for which the fieldcontains null [see The null value on page 181] In derivations,use countnulls instead.
countnull [see countnull/countnulls on page 163]
The number of unique values that occur in the field,potentially including the null value
countunique
157Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
ResultFunction
The number of unique values, excluding the null value, thatoccur in the field
countuniquenonnull
The first non-null value in the field for the groupfirst
The last non-null value in the field for the grouplast
The median (middle value) of the non-null values in the fieldfor the group
(This function requires a numeric or string-valued expressionas an argument.)
median
The mode (most-common value) of the non-null values inthe field for the group
mode
The proportion of non-null records in the group that containthe most-common non-null value in the field
moderatio
Aggregation functions for derivations only You can use the following aggregation functions ina derivation context only:
ResultFunction
The number of records in the segment for which the fieldhas non-null values [see The null value on page 181]
In measurements, use countnonnull instead.
countnonnulls [see countnonnull/countnonnulls on page162]
The number of records in the segment for which the fieldcontains null [see The null value on page 181]
In measurements, use countnull instead.
countnulls [see countnull/countnulls on page 163]
The number of records in the segment, as a percentage ofthe total number of records
(This function does not take an argument.)
percentage
The percentage of non-null records in the segment thatcontain the value 1 in the field
The field must contain only the values 0 and 1.
percentagerate
158Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
ResultFunction
A positive number uniquely identifying the segment withinthe crosstab formed by the breakdown fields
(This function does not take an argument.)
segindex
The statistical significance of any discrepancy between themean value of the field in the segment and the mean value
significance
of the field for the whole population, with a sign indicatingthe direction of the deviation
(This function requires a numeric expression as anargument.)
any
Purpose: pick an arbitrary non-null value from a field (possibly per group or segment).
Syntax any(x)
Arguments
DescriptionNameType
The field from which to pick a valuexinteger, real, date, or string
Result
DescriptionType
An arbitrary non-null value from the field x (possibly pergroup or segment)
as input
You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
See also
Aggregation functions for measurements and derivations on page 156
159Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
first on page 165
last on page 166
confintlower
Purpose: calculate the lower bound of a 95% confidence interval for the mean of a field.
Syntax confintlower(x)
Arguments
DescriptionNameType
The field to be consideredxnumeric
Result
DescriptionType
The lower bound of an interval within which the segment'spopulation mean for the field x is expected to fall with 95%confidence
real
You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
See also
Aggregation functions for measurements and derivations on page 156
confintupper on page 160
significance on page 174
confintupper
Purpose: calculate the upper bound of a 95% confidence interval for the mean of a field.
160Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
Syntax confintupper(x)
Arguments
DescriptionNameType
The field to be consideredxnumeric
Result
DescriptionType
The upper bound of an interval within which the segment'spopulation mean for the field x is expected to fall with 95%confidence
real
You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
See also
Aggregation functions for measurements and derivations on page 156
confintlower on page 160
significance on page 174
count
Purpose: count the number of records (possibly per group or segment).
Syntax count()
Arguments None
Result
DescriptionType
The number of records (possibly per group or segment)integer
161Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
See also
Aggregation functions for measurements and derivations on page 156
countnonnull/countnonnulls on page 162
countnull/countnulls on page 163
countunique on page 164
countuniquenonnull on page 164
countnonnull/countnonnulls
Purpose: count the number of records for which a field has non-null values (possibly per group orsegment).
Syntax countnonnull(x)
Arguments
DescriptionNameType
The field to be consideredxinteger, real, date, or string
Result
DescriptionType
The number of records for which the field x has non-nullvalues (possibly per group or segment)
integer
In Decision Studio crosstabs and in an measurement context [see Aggregation functionsfor measurements and derivations on page 156] this function is countnonnull, while ina derivation context it is countnonnulls.
Note:
See also
Aggregation functions for measurements and derivations on page 156
count on page 161
countnull/countnulls on page 163
countunique on page 164
162Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
countuniquenonnull on page 164
countnull/countnulls
Purpose: count the number of records for which a field contains the null value (possibly per groupor segment).
Syntax countnull(x)
Arguments
DescriptionNameType
The field to be consideredxinteger, real, date, or string
Result
DescriptionType
The number of records for which the field x is null (possiblyper group or segment)
integer
In Decision Studio crosstabs and in an measurement context [see Aggregation functionsfor measurements and derivations on page 156] this function is countnull, while in aderivation context it is countnulls.
Note:
See also
Aggregation functions for measurements and derivations on page 156
count on page 161
countnonnull/countnonnulls on page 162
countunique on page 164
countuniquenonnull on page 164
163Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
countunique
Purpose: count the number of unique values occurring in a field, potentially including the null value(possibly per group or segment).
Syntax countunique(x)
Arguments
DescriptionNameType
The field to be consideredxinteger, real, date, or string
Result
DescriptionType
The number of unique values, including null, that occur inx (possibly per group or segment)
integer
You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
See also
Aggregation functions for measurements and derivations on page 156
count on page 161
countnonnull/countnonnulls on page 162
countnull/countnulls on page 163
countuniquenonnull on page 164
countuniquenonnull
Purpose: count the number of non-null unique values occurring in a field (possibly per group orsegment).
164Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
Syntax countuniquenonnull(x)
Arguments
DescriptionNameType
The field to be consideredxinteger, real, date, or string
Result
DescriptionType
The number of unique values, excluding null, that occur inx (possibly per group or segment)
integer
You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
See also
Aggregation functions for measurements and derivations on page 156
count on page 161
countnonnull/countnonnulls on page 162
countnull/countnulls on page 163
countunique on page 164
first
Purpose: pick the first non-null value from a field (possibly per group or segment).
Syntax first(x)
Arguments
DescriptionNameType
The field from which to pick a valuexinteger, real, date, or string
165Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
Result
DescriptionType
The first non-null value encountered in the field x (possiblyper group or segment)
as input
You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
See also
Aggregation functions for measurements and derivations on page 156
any on page 159
last on page 166
last
Purpose: pick the last non-null value from a field (possibly per group or segment).
Syntax last(x)
Arguments
DescriptionNameType
The field from which to pick a valuexinteger, real, date, or string
Result
DescriptionType
The last non-null value encountered in the field x (possiblyper group or segment)
as input
You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
166Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
See also
Aggregation functions for measurements and derivations on page 156
any on page 159
first on page 165
max (one argument)
Purpose: calculate the maximum non-null value in a field (possibly per group or segment).
Syntax max(x)
Arguments
DescriptionNameType
The field whose values are to becompared
xinteger, real, date, or string
Result
DescriptionType
The maximum non-null value in the field x (possibly pergroup or segment)
as input
Note: • Be careful not to confuse this aggregation function with the multi-argument function [seemax (two or more arguments), maxnonnull on page 291] of the same name.
• In a derivation context, this function does not accept a string- or date-valued field as anargument.
See also
Aggregation functions for measurements and derivations on page 156
min (one argument) on page 169
167Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
mean (one argument)
Purpose: calculate the mean (common average) of the non-null values in a field (possibly per groupor segment).
Syntax mean(x)
Arguments
DescriptionNameType
The field whose mean is to becalculated
xnumeric
Result
DescriptionType
The mean of the non-null values in the field x (possibly pergroup or segment)
real
Be careful not to confuse this aggregation function with the multi-argument function [seemean (two or more arguments), meannonnull on page 292] of the same name.
Note:
See also
Aggregation functions for measurements and derivations on page 156
median on page 168
mode on page 170
median
Purpose: calculate the median (middle value) of the non-null values in a field (possibly per groupor segment).
Syntax median(x)
Arguments
168Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
DescriptionNameType
The field whose median is to becalculated
xinteger, real, or string
Result
DescriptionType
The median value of the non-null values in the field x(possibly per group or segment)
as input
You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
See also
Aggregation functions for measurements and derivations on page 156
mean (one argument) on page 168
mode on page 170
min (one argument)
Purpose: calculate the minimum non-null value in a field (possibly per group or segment).
Syntax min(x)
Arguments
DescriptionNameType
The field whose values are to becompared
xinteger, real, date, or string
Result
169Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
DescriptionType
The minimum non-null value in the field x (possibly pergroup or segment)
as input
Note: • Be careful not to confuse this aggregation function with the multi-argument function [seemin (two or more arguments), minnonnull on page 293] of the same name.
• In a derivation context, this function does not accept a string- or date-valued field as anargument.
See also
Aggregation functions for measurements and derivations on page 156
max (one argument) on page 167
mode
Purpose: calculate the mode (most common value) of the non-null values in a field (possibly pergroup or segment).
Syntax mode(x)
Arguments
DescriptionNameType
The field whose mode is to becalculated
xinteger, real, date, or string
Result
DescriptionType
The mode of the non-null values in the field x (possibly pergroup or segment)
as input
Note: • In the event of a tie for the most common value, the function returns the least tying number,earliest tying date, or alphabetically earliest tying string.
170Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
• You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or ina selections file for qsselect.
See also
Aggregation functions for measurements and derivations on page 156
mean (one argument) on page 168
median on page 168
moderatio on page 171
moderatio
Purpose: calculate the proportion of non-null records containing the most common non-null valuein a field (possibly per group or segment).
Syntax moderatio(x)
Arguments
DescriptionNameType
The field whose mode ratio is to becalculated
xinteger, real, date, or string
Result
DescriptionType
The mode ratio of the fieldx (possibly per group or segment)real
You cannot use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
See also
Aggregation functions for measurements and derivations on page 156
mode on page 170
171Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
percentage
Syntax percentage()
Purpose: calculate the number of records in a crosstab segment, as a percentage of the total.
Arguments None
Result
DescriptionType
The percentage of records (possibly per group or segment)real
You can only use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
See also
Aggregation functions for measurements and derivations on page 156
percentagerate
Purpose: for a binary (1/0) field, calculate the proportion of non-null records containing the value1, as a percentage.
Syntax percentagerate(x)
Arguments
DescriptionNameType
The binary field to be consideredxnumeric
Result
172Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
DescriptionType
The percentage of records in a binary field having the value1 (possibly per group or segment)
real
You cannot use this function in an aggregations file [see Using aggregation functions andthe where and default clauses on page 148] or statistics file [see Evaluating focus
Note:
statistics: the calculate statement on page 153] for qsmeasure or the equivalentAggregateRecords window, or in a statistics file for qstrack.
See also
Aggregation functions for measurements and derivations on page 156
segindex
Purpose: generate an index number representing a segment.
Syntax segindex()
Arguments None
Result
DescriptionType
A positive number uniquely identifying a segment within acrosstab
integer
Note: • If null, unclassified, or empty segments are hidden in a crosstab view, they still receive asegment index, so there may appear to be gaps in the sequence of visible indices.
• You cannot use this function in an aggregations file [see Using aggregation functionsand the where and default clauses on page 148] or statistics file [see Evaluating focusstatistics: the calculate statement on page 153] for qsmeasure or the equivalentAggregate Records window, or in a statistics file for qstrack.
Example R–2: Marking approximate deciles
You can bin a field "myfield" into ten equal-population bins and then derive a field "decile":
segindex() by myfield
See also
173Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
Aggregation functions for measurements and derivations on page 156
bin on page 338
significance
Purpose: calculate the statistical significance of any discrepancy between the mean value of a fieldfor a crosstab segment and the mean value of the same field for the whole population.
Syntax significance(x)
Arguments
DescriptionNameType
The field to be consideredxnumeric
Result
DescriptionType
A value between -1 and 1, whose sign indicates thedirection in which the segment mean of x deviates from the
real
population mean of x and whose absolute value indicatesthe statistical significance of that deviation, taking segmentsize into account
You cannot use this function in an aggregations file [see Using aggregation functions andthe where and default clauses on page 148] or statistics file [see Evaluating focus
Note:
statistics: the calculate statement on page 153] for qsmeasure or the equivalentAggregateRecords window, or in a statistics file for qstrack.
See also
Aggregation functions for measurements and derivations on page 156
confintlower on page 160
confintupper on page 160
174Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
stdev
Purpose: calculate the standard deviation of the non-null values in a field (possibly per group orsegment).
Syntax stdev(x)
Arguments
DescriptionNameType
The field whose standard deviation isto be computed
xnumeric
Result
DescriptionType
The standard deviation of the non-null values in the field x(possibly per group or segment)
real
See also
Aggregation functions for measurements and derivations on page 156
sum (one argument)
Purpose: calculate the sum of the non-null values in a field (possibly per group or segment).
Syntax sum(x)
Arguments
DescriptionNameType
The field whose values are to besummed
xnumeric
175Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
Result
DescriptionType
The sum of the non-null values in the field x (possibly pergroup or segment)
as input
Note: • Be careful not to confuse this aggregation function with the multi-argument function [seesum (two or more arguments), sumnonnull on page 299] of the same name.
• Note that when the sum aggregation function is applied to an empty set of records (forexample, as a result of record filtering), the result is null. You should normally override thisbehavior, by setting the default value to 0.
See also
Aggregation functions for measurements and derivations on page 156
variance
Purpose: calculate the statistical variance of the non-null values in a field (possibly per group orsegment).
Syntax variance(x)
Arguments
DescriptionNameType
The field whose variance is to becomputed
xnumeric
Result
DescriptionType
The variance of the non-null values in the field x (possiblyper group or segment)
real
See also
176Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
Aggregation functions for measurements and derivations on page 156
177Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Aggregation functions
15 - Field DerivationLanguage
In this section
About Field Derivation Language 179
About Field Derivation Language
Field Derivation Language (FDL) is a simple language for specifying the creation of new fields inSpectrum Miner foci. You can use FDL in the Table Viewer in Decision Studio, the Derive Fieldsand Aggregate Records wizards, and the Spectrum Miner data-build commands: qsderive,qsmeasure, qstrack and qsselect.
You can employ FDL to:
• Transform, combine, and perform a variety of other computations on existing fields in a focus• Partition a focus into a set of random samples• Define segmentations and predictive models• Select records according to a criterion of your choice
You typically use it in simple, "one-line" expressions such as those described in Basic expressionson page 184, but you can also use it to construct scripts with complex, data-dependent executionpaths.
FDL syntax includes:
• Operators for arithmetic [see Arithmetic operators on page 193].• Operators for comparison [see Relational operators on page 194] and for combining logical
expressions [see Logical operators on page 196].• Two alternative constructs for conditional evaluation of statements: if...then...else andcase [see Conditional expressions on page 185].
• A comprehensive set of built-in functions [see Built-in functions on page 198].• The ability to encapsulate common transformations as user-defined functions [see User-definedfunctions on page 190], for re-use and sharing.
• Variables, to hold values for re-use within an FDL script, including (in some contexts) global"accumulator" variables [see Global variables in Decision Studio on page 189].
• A facility for incorporating crosstabs into derivations .
See also
About Transaction Measurement Language on page 142
Expressions on page 183
179Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Field Derivation Language
16 - FDL syntax
In this section
Datatypes 181Expressions 183Conditional expressions 185Variables 187User-defined functions 190Arithmetic operators 193Relational operators 194Logical operators 196Operator precedence 197Built-in functions 198Reserved words in FDL 206
Datatypes
The following basic datatypes are supported in FDL [see About Field Derivation Language onpage 179]:
Examples (as FDL literals)DescriptionType
78004, -100A number without a fractional part(between -2 147 483 647 and 2 147483 646)
integer
42.3, -0.56A number with a fractional part,represented internally as a
real
"floating-point" number with 15-digitprecision (between and
, or 0)
#2009/01/01,#2009/01/01:16:01:39
A date/time between AD 1507 and AD3015, to one-second precision
date
"hello"A character string, representing textualdata
string
42, -1A small number without a fractional part(between qsminsmallint andqsmaxsmallint)
smallinteger
Note: • If you use a literal integer value (such as 12345678910) that is too large for an integerdatatype, you will get an overflow. You can often work around this by using a real valueinstead: (in this case, 12345678910.0).
• The smallinteger type has limited applicability within Spectrum Miner, and should only beused where absolutely necessary. In general, you should use the integer type instead.
The null value
In addition to the ordinary values for each of the basic datatypes, a field or expression of integer,real, date, or string type can take the special value null. The null value always represents an unknown
181Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
value (due to missing data, because of an overflow in a calculation — such as division by zero —or for some other reason). It is distinct from both zero and the empty string "".
FDL functions and operators preserve the uncertainty introduced by a null argument or operand,often giving the null value as a result. There are, however, functions — notably isnull and someaggregation functions — that deal specially with the null value.
Examples
?
1 + null equals null.
1 != null equals null.
1 < null equals null.
null = null equals null.
null != null equals null.
false or null equals null.
true and null equals null.
true or null equals true.
false and null equals false.
See also
ifnull, nvl on page 212
Boolean data
There is no formal boolean (true/false) datatype in FDL. However, you can treat an integer field (orvariable) containing only the values 0 and 1, or a logical expression such as Income < 10000, asthough it were of boolean type.
Logical expressions evaluate to 1 when they are true and 0 when they are false. Conditionalexpressions use these values to determine which of two or more other expressions to evaluate.Logical operators also work with these values. A value of 0 in an integer field or variable isinterpreted as "false" and any other (non-null) value is interpreted as "true."
Additionally, you can use the keywords true and false as boolean literals (which stand for 1 and0 respectively).
182Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
Type-compatibility
The integer and real datatypes are said to be type-compatible, because you can use an integervalue wherever a real value is required (but not vice versa), for example, as an argument to afunction. Date and string datatypes are only type-compatible with themselves.
The operands of a binary arithmetic operator [see Arithmetic operators on page 193] or relationaloperator [see Relational operators on page 194] have to be type-compatible. Indeed, in the caseof a binary arithmetic operator, both operands must be numeric; the result is real unless bothoperands are integers.
Likewise, the result expressions in an if or case expression [see Conditional expressions on page185] have to be type-compatible, so that the type of the result can be determined before evaluation.If integer- and real-valued expressions are mixed, the result is real-valued.
Type conversion
As well as using implicit datatype conversion (which occurs when the datatype of an expressionforces it), you can explicitly convert between datatypes with the tointeger, toreal, todate andtostring FDL functions.
When converting from a string to a numeric, any trailing non-numeric characters are ignored.Note:
Expressions
An FDL expression is one of the following:
• A basic expression [see Basic expressions on page 184]• A conditional expression [see Conditional expressions on page 185]• A variable initialization or assignment [see Variables on page 187]• A semicolon-separated list of expressions (an expression list), enclosed in parentheses
Every expression has an associated datatype [see Datatypes on page 181] and can be evaluated.
To evaluate an expression list in a given context, evaluate each constituent expression in turn, fromfirst to last; the value of the expression list as a whole is the last computed value.
Note: • The datatype is inferred automatically. You cannot specify it other than by using a localvariable of the required type, or by using an expression that forces the type. For example,the following expression forces the type to be real:
183Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
if false then 0.0 else ...
• You typically use expression lists in combination with variable assignment.
Basic expressions
A basic expression is one of the following:
• An identifier, optionally quoted in single quotation marks, for example, 'CustomerID'. You mustenclose an identifier in single quotation marks if it does not begin with a letter ("A" — "Z" or "a" —"z"), contains characters other than letters, digits, and underscores, or coincides with a TML orFDL reserved word (or differs from one only in case). An identifier must be no longer than 128characters.
• A numeric literal, for example, 30, -3.28, or 3.1e8 (the last of these representing )• A string literal, enclosed in double quotation marks, for example, "A string"
• A date literal, prefixed by "#," for example, #2004/03/16 or #2004/03/16:09:12:13• The keyword "null" (see The null value on page 181)• A function application, consisting of a function name (an identifier) followed by a comma-separated
list of arguments (expressions) in parentheses; for example, mean(IncomeA + IncomeB,TotalEstIncome)
• An arithmetic expression, consisting of numeric-valued expressions connected by arithmeticoperators [see Arithmetic operators on page 193], for example, 365 / 52
• A relation, consisting of numeric- or string-valued expressions connected by relational operators[see Relational operators on page 194], for example, Age > 30 or 'State' = "New York"
• A logical expression, consisting of relations or other boolean-valued expressions connected bylogical operators [see Logical operators on page 196], for example, (Income > 250000) and(MaritalStatus = "Single")
• A statistic identifier, consisting of the TML keyword STATISTIC followed by a period followed byan identifier, for example, STATISTIC.AverageIncome (only in a TML aggregations file)
• A field statistic — only in a Decision Studio derivation, a qsderive derivations file, or a qsselectselections file
Note: • FDL identifiers are case-significant. For example, the expressions CustomerID andCUSTOMERID refer to different fields, while the expressions isnull(Age) andIsNull(Age) involve different functions.
• A numeric literal is evaluated as an integer if it contains no decimal point and doesn't usescientific notation. Otherwise it is evaluated as a real number.
• A string literal can contain any characters other than double quotation marks (") andnewline characters.
• A date literal is initially evaluated in YMD [see Date formats on page 52] format; if thisfails, it is evaluated according to the setting of your read preference .
184Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
• As well as introducing spaces between syntactic elements, you can include comments,which begin with a double slash ("//") and continue to the end of the line. For example,
// This is a comment.2 + 2 // This is a comment.
See also
Reserved words in FDL on page 206
Reserved words in TML on page 143
Conditional expressions
In FDL, there are two main ways of conditionally evaluating an expression: the if expression andthe case expression.
See also
About Field Derivation Language on page 179
Expressions on page 183
The if expression
If the condition (an FDL expression returning a boolean [see Boolean data on page 182] value) istrue, the expression following then is evaluated; otherwise, the expression following else isevaluated. The value of the if expression is the value of the evaluated expression.
Note: • The expressions following then and else must be type-compatible [see Datatypes onpage 181].
• If you omit the "else" part of an if expression and the condition is false, the value of the ifexpression is null [see The null value on page 181].
• A null [see The null value on page 181] condition is treated by the if expression as thoughit were false. This is an exception to the general rule that the uncertainty introduced by thenull value is preserved by FDL expressions. This can lead to unexpected results: for example,when Age is null, the following (at first sight equivalent) expressions generate the null value,0 and 1 respectively:
185Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
Age > 40
if Age > 40 then 1 else 0
if Age <= 40 then 0 else 1
• You can test for multiple conditions using nested if expressions — if expressions in whichthe expression following then or else is itself an if expression. If the expression followingthen is an if expression, you should enclose it in parentheses to avoid ambiguity.Alternatively, you can use the case expression [see The case expression on page 186].
• If your condition can be expressed as x = 0 (or ) for some number x, you can also usethe built-in FDL function cond instead of an if expression.
Examples
if Age < 40 then "Young"
if Age < 40 then "Young" else "Senior"
if Age > 40 then "Young" else if Age < 60 then "MiddleAged" else "Senior"
if (Age >= 13 and Age < 20) then "Teenager"
if isnull(Age) then mean(Age) else Age
if Responder then RespondDate else StartDate
if match(Postcode, "^EH") then "Y" else "N"
if StartDate < #1998/09/08 then "Loyal" else "Recent"
The case expression
If the first condition (an FDL expression returning a boolean [see Boolean data on page 182] value)is true, the corresponding expression (to the right of the colon) is evaluated and returned as thevalue of the case expression. Otherwise, the next condition is tested; if it is true, the correspondingexpression is evaluated and returned as the value of the case expression. Otherwise, the nextcondition is tested, and so on.
The special default keyword is interpreted as an always-true expression: you would normally useit as the last condition in a case expression (as any subsequent conditions would be ignored).
Note: • All the expressions occurring to the right of colons must be type-compatible [seeDatatypeson page 181].
186Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
• If none of the conditions is true (and there is no default keyword), the value of the caseexpression is null [see The null value on page 181].
• A null [see The null value on page 181] condition is treated by the case expression asthough it were false. This is an exception to the general rule that the uncertainty introducedby the null value is preserved by FDL expressions. For example, when Age is null, thefollowing expression generates the value 3:
case
Age > 40 : 1;
Age <= 40 : 2;
default : 3;
• If you use a case expression in an expression list or nest a case expression as a condition(inside an if expression or another case expression), you should enclose it in parenthesesto avoid ambiguity.
Examples
case
Age < 40: "Young";
default: "NotYoung";
case
Response = "Exc": 7;
Response = "V. Sat": 6
Response = "Sat": 5;
Response = "F. Sat": 4;
Response = "Poor": 3;
Response = "Bad": 2;
default: 1;
Variables
A variable in FDL is a temporary storage location for a real, integer, string, or date value. You canuse a local variable to evaluate a complicated subexpression% — particularly useful if the samesubexpression occurs more than once% — or to evaluate an expression involving a random-numberfunction, to avoid its value changing from one evaluation to the next. You can also use global
187Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
variables in Decision Studio [see Global variables in Decision Studio on page 189] and statevariables in TML trackers files [see State variables in Transaction Measurement Language onpage 189] to carry information from one record to the next.
To assign a value to a variable, use the following syntax:
The first such assignment is an initialization and determines the class of the variable% — local (nokeyword), global, or state. The keyword global or state is only used in conjunction with the firstassignment to a global or state variable (the initialization).
The identifier [see Basic expressions on page 184] is the name of the variable (which must notcoincide with a field name in the focus).
For any class of variable, its datatype [see Datatypes on page 181] is that of the first value that isassigned to it.
The value of an assignment expression is simply the value of the expression to the right of the ":="sign.
For example, the following expression list randomly assigns the value 1 (to 20% of the records), 2(to another 20% of the records), or 0 (to the remainder of the records):
x := rndUniform();
case
x < 0.2 : 1;
x < 0.4 : 2;
default : 0;
It could be written equivalently in this more compact form:
case
( x := rndUniform() ) < 0.2 : 1;
x < 0.4 : 2;
default : 0;
However, the following expression list generates a different (and probably unwanted) result:
case
rndUniform() < 0.2 : 1;
rndUniform() < 0.4 : 2;
default : 0;
188Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
Global variables in Decision Studio
Variables are local by default. A local variable is destroyed as soon as the last expression in the(outermost enclosing) expression list [see Expressions on page 183] is evaluated — The next timethat the same variable is used, it is re-initialized. A local variable cannot carry information from onerecord to the next. For example, the following field-derivation expression list produces a field with1 in every record:
i := 0;
i := i + 1
A global variable differs from a local variable in that it is not destroyed until Decision Studio or thedata-build command has calculated the field values for all records. The initialization only happensonce, for the first record in the focus. For example, the following field-derivation expression listproduces an index field, with 1 in the first record, 2 in the second, and so on:
global i := 0;
i := i + 1
You can use a global variable in a field-derivation expression list in a Decision Studio derivation, inthe derivations file for qsderive or the equivalent Derive Fields window, in a trackers file for qstrack,or in a selections file for qsselect.
State variables in Transaction Measurement Language
You can use a state variable in a TML field definition in the qstrack trackers file.
A state variable differs from a local variable or a global variable in that it is not destroyed until qstrackhas calculated all the field values for the current group (as defined by the key field). The initializationhappens once for each group.
For example, the following field definition defines a running-balance field, assuming that the field"Deposit" contains an amount deposited in a transaction (or a negative value for a withdrawal) andthe field "InitialBalance" contains the initial balance for the customer (perhaps from a previouslyjoined customer table):
create Balance := (
state bal := InitialBalance;
bal := bal + Deposit
)
189Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
User-defined functions
To define an FDL function that you can use in the same way as a built-in function [see Built-infunctions on page 198], use the following syntax:
The identifier [seeBasic expressions on page 184] following the "function" keyword is the nameof the function. The list of (formal) arguments comes next:
Each formal argument (other than one preceded by a "field" keyword) acts like a local variable[see Variables on page 187] that is initialized with the value of the corresponding actual argumentwhen the function is applied. The type [seeDatatypes on page 181] of a formal argument is the typespecified in the function definition, if present; otherwise it is that of the actual argument used whenthe function is applied. Note that the enclosing parentheses are required even if there are no formalarguments.
Following the list of arguments comes an optional list of function attributes [see Function attributeson page 192], and then the defining body of the function. The expression list [see Expressions onpage 183] is effectively prefixed with local variable assignments corresponding to the formal arguments(other than those preceded by a "field" keyword), before being evaluated to give the return valuefor the function. The type of the return value is that of the last expression.
Examples Define a function to replace zeros in the result of one expression (of any type) with theresult of another expression:
function replaceZeros(expr1, expr2)
{
if expr1 = 0 then expr2 else expr1
}
A function which takes a string argument and returns it with the first letter capitalized:
function titleCase(string text)
190Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
{
concat(toupper(left(text, 1)), right(text, strlen(text) - 1));
}
Note:
• When specifying the type of a function argument, use "real", "integer", "string" or "date."• You can include field statistics in a function definition, but each function argument that is used in
a field statistic must be preceded by a "field" keyword. For example, to compare a value witha summary over the whole focus:
function MyBand(real field x)
{
averageX := mean(x);
stdevX := stdev(x);
case
x < averageX - stdevX : "Lower";
x > averageX + stdevX : "Upper";
default: "Mid";
}
• You can include function definitions at the beginning of the expression list that defines a derivedfield. Functions so defined are local to the field derivation.
You can also define globally accessible functions by storing them in a file (or files) in a folder listedin the derivation libraries preference . In Decision Studio, in the Function Family paneof the Derivation Functions dialog box in the Table Viewer, functions in a file named class.fdlappear in the class named class. For example, functions in a file named Conditional.fdl appear inthe Conditional class. You can reuse an existing class name, to augment a class with your ownuser-defined functions, or create your own class by using a new name.
When using the Spectrum Miner data-build command qsmeasure, the equivalent AggregateRecords window, or the data-build command qstrack, you can specify a file containing additionalglobal function definitions.
• You cannot define aggregation functions in a measurement context, that is, in an aggregationsfile [see Using aggregation functions and the where and default clauses on page 148] orstatistics file [see Evaluating focus statistics: the calculate statement on page 153] forqsmeasure or the equivalent Aggregate Records window, or in a statistics file for qstrack..
• If the expression list ends with an alphabetic character, it must be separated from the closingbrace by a semicolon or whitespace.
191Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
Function attributes
Specify function attributes as follows:
Useful attributes are:
synopsis and description. These attributes provide a standard way of making the function"self-documenting." If a function is installed in a folder listed in the derivation librariespreference , the synopsis is shown when you highlight the function in the Functions pane of theDerivation Functions dialog box in the Table Viewer. When you click on the Details button, thedescription is shown. The synopsis text should be short, but the description text can be longer.
For example:
function nvl(testNull, replaceValue)
[
synopsis = "Replace null values"
description = "The nvl() function replaces null values...strings."
]
(Note that the description attribute must all be on the same line.)
element_names. In an aggregations file, you can use a function with the element_names attributeto split an aggregation to create multiple fields [see Splitting aggregations: the by clause on page150].
The value of this attribute is a string containing a comma-separated list of names, which areassociated with function return values 1, 2, ..., in the order that they are listed.
For example, the element_names attribute of the following function associates the names "Low","Medium" and "High" with return values 1, 2, and 3 respectively:
function band(x)
[ element_names = "Low,Medium,High" ]
{
clamp(floor(x / 1000), 0, 2) + 1
}
The element_names attribute of the following function associates the names "North", "South","East" and "West" with return values 1, 2, 3 and 4 respectively:
192Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
function StoreGroupFunction( Store )
[
element_names = "North, South, East, West"
]
{
case
Store = "0000" or Store = "0800" : 1;
Store = "0300" or Store = "0600" or Store = "0700" : 2;
Store = "0100" or Store = "0400" : 3;
default : 4;
}
Arithmetic operators
Except for unary minus, which prefixes its single operand, all FDL arithmetic operators are infixbinary operators, that is, they are written between their two operands.
ResultOperandsOperator
The sum of the operandsnumeric+ (addition)
The result of subtracting the secondoperand from the first
numeric- (subtraction)
The negative of the operandnumeric- (unary minus)
The product of the operandsnumeric* (multiplication)
The (real) result of dividing the firstoperand by the second
numeric/ (division)
The (integer) result of dividing the firstoperand by the second, ignoring anyremainder
integerdiv (integer division)
193Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
ResultOperandsOperator
The remainder on integer-dividing thefirst operand by the second (the
integermod (modulo)
modulus), with the same sign as thefirst operand — for example, 9 % 7returns 2 but -9 % 7 returns -2
Examples
2 + 5 equals 7.
2 - 5 equals -3.
- (2 + 5) equals -7.
2 * 5 equals 10.
2 / 5 equals 0.4.
2 div 5 equals 0.
2 mod 5 equals 2.
5 % 2 equals 1.
Note: • The addition, subtraction, unary minus, and multiplication operators preserve the type oftheir operands (which are type-compatible [see Type-compatibility on page 183]). If oneoperand is an integer and the other is a real number, the result is real.
• If the second operand of the division, integer-division, or modulo operator is zero, the resultis null [see The null value on page 181], as division by zero is undefined.
• mod requires integer operands.• You can also use % instead of mod.
See also
Logical operators on page 196
Operator precedence on page 197
Relational operators on page 194
Relational operators
All FDL relational operators are infix binary operators, that is, they are written between their twooperands.
194Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
ResultOperandsOperator
True if the first operand is less than thesecond; false if the first operand isgreater than or equal to the second
integer, real, date, or string< (less than)
True if the first operand is less than orequal to the second; false if the firstoperand is greater than the second
integer, real, date, or string<= (less than or equal to)
True if the first operand is greater thanthe second; false if the first operand isless than or equal to the second
integer, real, date, or string> (greater than)
True if the first operand is greater thanor equal to the second; false if the firstoperand is less than the second
integer, real, date, or string>= (greater than or equal to)
True if both operands have the samevalue; false if the operands have distinct
integer, real, date, or string= (equality)
values. For string types, the comparisonignores trailing space characters in bothoperands.
True if the operands have distinctvalues; false if both operands have the
integer, real, date, or string!= (inequality)
same value. For string types, thecomparison ignores trailing spacecharacters in both operands.
Examples
2 < 5 equals 1 (TRUE).
5 < 5 equals 0 (FALSE).
"A" < "B" equals 1 (TRUE).
5 <= 5 equals 1 (TRUE).
5 > 5 equals 0 (FALSE).
7 > 5 equals 1 (TRUE).
#2003/12/31 > #2003/11/30 equals 1 (TRUE).
5 >= 5 equals 1 (TRUE).
"A" = "A" equals 1 (TRUE).
"A" = "a" equals 0 (FALSE).
195Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
"A" != "A" equals 0 (FALSE).
"A" != "a" equals 1 (TRUE).
Note: • The operands of a binary relational operator must be type-compatible [seeType-compatibility on page 183].
• When comparing dates, "less than" means earlier. When comparing strings, "less than"means earlier in the alphabet (or more specifically, earlier in the underlying characterrepresentation). Likewise for "greater than" etc.
• You can also use == or eq instead of = and <> instead of !=.
See also
Arithmetic operators on page 193
Boolean data on page 182
Logical operators on page 196
Logical operators
The logical negation operator prefixes its single operand, while the logical conjunction and disjunctionoperators are infix binary operators, that is, they are written between their two operands.
ResultOperandsOperator
True if both operands are true; false ifeither operand is false
booleanand (logical conjunction)
True if either operand is true; false ifboth operands are false
booleanor (logical disjunction)
True if the operand is false; false if theoperand is true
booleannot (logical negation)
Examples
("A" = "A") and (5 = 5) equals 1 (TRUE).
("A" = "B") and (5 = 5) equals 0 (FALSE).
("A" = "B") and (5 = 2) equals 0 (FALSE).
("A" = "A") or (5 = 5) equals 1 (TRUE).
("A" = "B") || (5 = 5) equals 1 (TRUE).
196Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
("A" = "B") or (5 = 2) equals 0 (FALSE).
not ("A" = "A") equals 0 (FALSE).
!(("A" = "B") or (5 = 2)) equals 1 (TRUE).
Note: • There is currently no boolean [see Boolean data on page 182] datatype in FDL, so theoperands for logical operators are actually integers. You should not, however, apply theseoperators to arbitrary integers.
• You can also use and instead of and, || instead of or, and ! instead of not.
See also
Arithmetic operators on page 193
Operator precedence on page 197
Relational operators on page 194
Operator precedence
In the absence of parentheses, FDL operations in an expression are carried out in order of operatorprecedence, from highest to lowest:
OperatorsClass
-Unary minus
* / div modMultiplication-style operators
+ -Addition-style operators
< <= > >= != =Relational operators
notLogical negation
and orBinary logical operators
If there is a tie for precedence, operations are carried out from left to right.
Subexpressions in parentheses are always evaluated first.
Examples -5 + 12 / 4 * 2 + 20 * 3 equals 61.
197Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
that is, the expression is evaluated as: -5 + ((12 / 4) * 2) + (20 * 3)
Moving the parentheses:
-(5 + 12 / (4 * 2) + 20) * 3 equals -79.5.
See also
Arithmetic operators on page 193
Logical operators on page 196
Relational operators on page 194
Built-in functions
FDL includes a comprehensive set of built-in functions.
Conditional functions
DescriptionFunction
Constrain a number to lie within a given interval.clamp
Produce one of two results depending on whether anexpression is zero or non-zero.
cond
Produce one of two results depending on whether anexpression is zero or non-zero.
iff
Replace the null value.ifnull, nvl
Test whether a value is null.isnull
Flag the records that are currently selected.isselected
Replace a number if it does not lie within a given interval.replace
Datatype-conversion functions
198Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
PurposeFunction
Convert a value to a date.todate
Convert a value to an integer.tointeger
Convert a value to a real number.toreal
Convert a value to a string.tostring
Functions for working with dates
PurposeFunction
Calculate a date from another date using an offset incenturies.
addcenturies, addcenturiescountbackwards
Calculate a date from another date using an offset in days.adddays
Calculate a date from another date using an offset in hours.addhours
Calculate a date from another date using an offset inminutes.
addminutes
Calculate a date from another date using an offset inmonths.
addmonths, addmonthscountbackwards
Calculate a date from another date using an offset inseconds.
addseconds
Calculate a date from another date using an offset in weeks.addweeks
Calculate a date from another date using an offset in years.addyears, addyearscountbackwards
Calculate the period between two dates in centuries.countcenturies
Calculate the period between two dates in days.countdays
Calculate the period between two dates in hours.counthours
Calculate the period between two dates in minutes.countminutes
199Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
PurposeFunction
Count the number of seconds between two dates.countseconds
Calculate the period between two dates in weeks.countweeks
Count the number of complete centuries between two dates.countwholecenturies, countwholecenturiesbackwards
Count the number of complete days between two dates.countwholedays
Count the number of complete hours between two dates.countwholehours
Count the number of complete minutes between two dates.countwholeminutes
Count the number of complete months between two dates.countwholemonths, countwholemonthsbackwards
Count the number of seconds between two dates.countwholeseconds
Count the number of complete weeks between two dates.countwholeweeks
Count the number of complete years between two dates.countwholeyears, countwholeyearsbackwards
Calculate the period between two dates in years.countyears
Obtain the day-of-month part of a date.day
Obtain a number representing the day-of-week of a date.dayofweek
Convert a Greenwich Mean Time (GMT) date to EasternDaylight Time (EDT).
gmt2edt
Obtain the hours part of a date.hour
Obtain the minutes part of a date.minute
Obtain the month part of a date.month
Obtain the current date and time.now
Obtain the seconds part of a date.second
200Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
PurposeFunction
Obtain the current date.today
Calculate the week-of-year of a date, relative to a specifiedstart date.
weekofyear
Obtain the year part of a date.year
Functions for working with strings
PurposeFunction
Concatenate two or more strings.concat
Test whether one string ends with another.endswith
Test whether one string is contained within another.find
Return a substring of specified length from the beginningof a string.
left
Return a substring of specified length from the middle of astring.
mid
Return a substring of specified length from the end of astring.
right
Reduce each word to a four-character string for indexingpurposes.
soundex
Test whether one string starts with another.startswith
Obtain the length of a string.strlen
Determine set membership.strmember
Replace one string with another.substitute
Obtain a substring from a string.substr
201Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
PurposeFunction
Convert a string to lowercase text.tolower
Convert a string to uppercase text.toupper
Remove all spaces from a string except for single spacesbetween words.
trim
Regular expressions and associated functions
PurposeFunction
Test a string for a regular expression match [see Regularexpressions on page 276].
match
Replace all substrings that match a regular expression [seeRegular expressions on page 276].
replaceall
Replace the first substring that matches a regular expression[see Regular expressions on page 276].
replacefirst
Mathematical and statistical functions
DescriptionType
Calculate the absolute value of a number.abs
Round a number up to the nearest integer.ceil
Calculate the cosine of an angle.cos
Calculate the exponential of a number.exp
Round a number down to the nearest integer.floor
Calculate the natural (base-e) logarithm of a number.log
Calculate the base-10 logarithm of a number.log10
202Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
DescriptionType
Calculate the logarithm of a number, to a specified base.logbase
Calculate the maximum of two or more numbers, or thelatest of two or more dates, or the alphabetically latest oftwo or more strings.
max (two or more arguments), maxnonnull
Calculate the mean (common average) of two or morenumbers.
mean (two or more arguments), meannonnull
Calculate the minimum of two or more numbers, or theearliest of two or more dates, or the alphabetically earliestof two or more strings.
min (two or more arguments), minnonnull
Normalize field values to lie in the interval [0,1].normalize
Calculate the result of raising one number to the power ofanother number.
pow
Calculate the product of two or more numbers.product, productnonnull
Round a number to the nearest integer.round
Calculate the signum (sign) of a number.sgn
Calculate the sine of an angle.sin
Calculate the square root of a number.sqrt
Calculate the sum of two or more numbers.sum (two or more arguments), sumnonnull
Calculate the tangent of an angle.tan
Data-sampling functions
PurposeFunction
Create a test/training segmentation for use in modelvalidation.
numericTestTrainSplit
203Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
PurposeFunction
Create a test/training/validation segmentation for use inmodel validation.
numericTestTrainValidateSplit
Create, at random, a segmentation index for groups of equalsize.
sampleEqualSize
Flag, at random, an exact number of customers from aselected population.
sampleExactNumber
Flag, at random, an exact percentage of customers from aselected population.
sampleExactPercentage
Flag, at random, a specified number of customers from asegment and the rest of the population.
sampleStratified
Create a test/training segmentation for use in modelvalidation.
testTrainSplit
Create a test/training/validation segmentation for use inmodel validation.
testTrainValidateSplit
Random-number functions
PurposeFunction
Generate a random integer based on a binomial distribution.rndBinomial
Generate either 0 or 1 randomly (with equal probability).rndBool
Generate a random positive number based on anexponential distribution.
rndExp
Generate a random positive number based on a gammadistribution.
rndGamma
Generate a random number based on a normal distribution.rndNormal
Generate a random non-negative integer based on adiscrete Poisson distribution.
rndPoisson
204Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
PurposeFunction
Generate a random number between 0 and 1 based on auniform distribution.
rndUniform
Return-on-investment functions
PurposeFunction
Estimate the (money) per-customer return on investmentcontribution from taking action designed to generate adefinite response.
ActionROI
Estimate the annualized per-customer return on investmentmultiple from taking action designed to generate a definiteresponse from a customer.
ActionROIAnnualized
Estimate the per-customer return on investment contributionfrom making an offer designed to generate a definiteresponse.
OfferROI
Estimate the annualized per-customer return on investmentmultiple from making an offer designed to generate a definiteresponse from a customer.
OfferROIAnnualized
Estimate the per-customer (money) return on investmentcontribution from taking action designed to prevent attrition.
RetentionActionROI
Estimate the annualized per-customer return on investmentcontribution from taking action designed to prevent attrition.
RetentionActionROIAnnualized
Estimate the per-customer return on investment from makingan offer designed to prevent attrition.
RetentionOfferROI
Estimate the annualized per-customer return on investmentfrom making an offer designed to prevent attrition.
RetentionOfferROIAnnualized
Miscellaneous functions
PurposeFunction
Look up values in a reference table (stored as a focus).dblookup
205Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
PurposeFunction
Determine set membership.member
Identify the rank of a value within a list of values.rankOrder, rankOrderApprox
Identify the rank of a value within a list of values.rankOrderMean, rankOrderApproxMean
Generate the number of each row in the table.rownum
Binnings
PurposeFunction
Obtain a bin index corresponding to a value.bin
See also
User-defined functions on page 190
Reserved words in FDL
The following tokens are reserved words in FDL: accum, agg, aggregate, aggregation,and, by, case, date, default, div, double, else, eq, false, field, float,function, global, if, int, integer, long, mod, not, null, number, numeric,or, real, short, state, string, then, true, wheredate, default, div,double, else, eq, false, field, float, function, global, if, int, integer,long, mod, not, null, number, numeric, or, real, short, state, string,then, true, where and while
To use a reserved word, or a word that differs only in case from a reserved word, as an identifier[see Expressions on page 183] in FDL, or as a field or statistic name in TML, you must enclose itin single quotation marks — for example, 'State'.
Although identifiers in FDL are case-significant (fred and Fred are distinct), the reservedwords are not case-significant and are therefore reserved in any case combination: for
Note:
example, if, If, iF, and IF are all representations of the same token; all suchcombinations are reserved and may not be used as identifiers.
See also
About Field Derivation Language on page 179
206Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
Reserved words in TML on page 143
207Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
FDL syntax
17 - Conditionalfunctions
In this section
clamp 209cond 210iff 211ifnull, nvl 212isnull 213isselected 214replace 214
clamp
Purpose: constrain a number to lie within a given interval.
Syntax clamp(x, min, max)
Arguments
DescriptionNameType
The number to be clampedxnumeric
The lower bound of the intervalminas x
The upper bound of the intervalmaxas x
Result
DescriptionType
x if ; min if x < min; max if x > maxas input
Examples
To clamp values within a range from 18 to 65 inclusive, replacing values outside the range with theappropriate range boundaries:
clamp(CustomerAge,18,65)CustomerAge
189
1818
6566
4343
See also
209Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Conditional functions
replace on page 214
cond
Purpose: produce one of two results depending on whether an expression is zero or non-zero.
Syntax cond(cond, trueExpr, falseExpr)
Arguments
DescriptionNameType
The expression to testcondinteger
"True" (non-zero) branchtrueExprinteger, real, date, or string
"False" (zero) branchfalseExpras trueExpr
Result
DescriptionType
trueExpr if cond 0; falseExpr if cond = 0as trueExpr, falseExpr
Unlike an if expression [see The if expression on page 185] or case expression [see Thecase expression on page 186], both branches are evaluated by the cond function.
Note:
More significantly, cond preserves the uncertainty of a null value [see The null value onpage 181] in the condition.
If you don't need this behavior, use an if or case expression for speed.
Examples
To target customers with a zero spend in Quarter 2 for a mailing campaign:
cond(SpendQtr2, "Don't Mail","Mail")
SpendQtr2
Don't Mail100.39
210Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Conditional functions
cond(SpendQtr2, "Don't Mail","Mail")
SpendQtr2
Mail0
Mail-5.72
nullnull
See also
clamp on page 209
iff on page 211
ifnull, nvl on page 212
replace on page 214
iff
Purpose: produce one of two results depending on whether an expression is zero or non-zero.
Syntax iff(logical_text, value_if_true, value_if_false)
Arguments
DescriptionNameType
The expression to testlogical_textreal
"True" (non-zero) branchvalue_if_trueinteger, real, date, or string
"False" (zero or null) branchvalue_if_falseas value_if_true
Result
211Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Conditional functions
DescriptionType
value_if_true if logical_text 0;value_if_false if logical_text = 0 orlogical_text is null
as value_if_true, value_if_false
Examples iff(Score > 650, "Mail", "No Mail")
See also
cond on page 210
ifnull, nvl
Purpose: replace the null value. You can use nvl as an alias for ifnull.
Syntax ifnull(testNull, replaceValue)
nvl(testNull, replaceValue)
Arguments
DescriptionNameType
The value to be testedtestNullinteger, real, date, or string
The replacement for the null valuereplaceValueas testNull
Result
DescriptionType
A copy of the value testNull with replaceValuereplacing the null value
as input
Examples:
To replace missing values with a value derived from another field:
212Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Conditional functions
ifnull(SpendQtr1,AnnualSpend/4)
AnnualSpendSpendQtr1
45.21178.9745.21
37.64150.56null
196.07784.27null
59.05274.1859.05
See also
cond on page 210
isnull on page 213
isnull
Purpose: test whether a value is null.
Syntax isnull(x)
Arguments
DescriptionNameType
The value to be testedxinteger, real, date, or string
Result
DescriptionType
1 if x is null; 0 otherwiseinteger
Examples
To replace null value entries in a field with that field's mean value:
213Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Conditional functions
if isnull(Age) then mean(Age) elseAge
Age
3232
44null
5959
2727
See also
Splitting aggregations: the by clause on page 150
isselected
Purpose: flag the records that are currently selected.
Syntax isselected()
Arguments None
Result
DescriptionType
1 if the record is currently selected, otherwise 0integer
The Table Viewer has special support for immediate expansion of this function.Note:
Examples if isselected() then "LowRisk" else "HighRisk"
replace
Purpose: replace a number if it does not lie within a given interval.
Syntax replace(x, min, max, y)
214Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Conditional functions
Arguments
DescriptionNameType
The number to be testedxnumeric
The lower bound of the intervalminas x
The upper bound of the intervalmaxas x
The replacement valueyas x
Result
DescriptionType
x if ; y otherwiseas input
Examples
To clamp values within a range from 18 to 65 inclusive, replacing values outside the range with themean value:
replace (CustomerAge, 18, 65,mean(CustomerAge))
CustomerAge
389
1818
3866
4343
See also
clamp on page 209
215Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Conditional functions
18 -Datatype-conversionfunctions
In this section
todate 217tointeger 218toreal 219tostring 220
todate
Purpose: convert a value to a date.
Syntax todate(x [, format] )
Arguments
DescriptionNameType
A value to convertxinteger, real, date, or string
A date format [see Date formats onpage 52] to use for the conversion
formatstring
(optional and only allowed if x is ofstring type)
Result
DescriptionType
x, as a datedate
Note: • If the argument is of integer type, it is assumed to represent the date as YYYYMMDD.• If the argument is of real type, it is assumed to represent the date as
YYYYMMDD.HHMMSS.• If the argument is of string type, and no date format argument is provided, the string is
converted according to the setting of your read preference .
Examples (Date values shown with European date format.)
todate(20040109) equals 09/01/2004:00:00:00.
todate(20040109.122853) equals 09/01/2004:12:08:52.
todate("09/01/2004") equals 09/01/2004:00:00:00.
todate("09-Jan-2004","%d-%b-%Y") equals 09/01/2004:00:00:00.
todate(#09/01/2004:12:28:53) equals 09/01/2004:12:28:53.
See also
tointeger on page 218
217Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Datatype-conversion functions
toreal on page 219
tostring on page 220
Type conversion on page 183
tointeger
Purpose: convert a value to an integer.
Syntax tointeger(x)
Arguments
DescriptionNameType
A value to convertxinteger, real, date, or string
Result
DescriptionType
x, as an integerinteger
Note: • If the argument is of real type, digits after the decimal point are ignored.• If the argument is of date type, the result is of the form YYYYMMDD (the time component
is ignored).• If the argument is of string type, any trailing non-numeric characters are ignored.
Examples
tointeger(100) equals 100.
tointeger(100.65) equals 100.
tointeger("100.65") equals 100.
tointeger(#09/01/2004:12:28:53) equals 20040109.
See also
todate on page 217
toreal on page 219
218Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Datatype-conversion functions
tostring on page 220
Type conversion on page 183
toreal
Purpose: convert a value to a real number.
Syntax toreal(x)
Arguments
DescriptionNameType
A value to convertxinteger, real, date, or string
Result
DescriptionType
x, as a real numberreal
Note: • If the argument is of date type, the result is of the form YYYYMMDD.HHMMSS.• If the argument is of string type, any trailing non-numeric characters are ignored.
Examples
toreal(100) equals 100.0.
toreal(100.65) equals 100.65.
toreal("100.65") equals 100.65.
toreal(#09/01/2004:12:28:53) equals 20040109.122853.
See also
todate on page 217
tointeger on page 218
tostring on page 220
Type conversion on page 183
219Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Datatype-conversion functions
tostring
Purpose: convert a value to a string.
Syntax tostring(x)
Arguments
DescriptionNameType
A value to convertxinteger, real, date, or string
Result
DescriptionType
x, as a stringstring
Note: • If the argument is of real type, the result is in ordinary decimal (not scientific) format.• If the argument is of date type, the result is formatted according to the setting of your datewrite preference .
Examples
tostring(100) equals "100".
tostring(1.0065e2) equals "100.650000".
tostring("100.65") equals "100.65".
tostring(#09/01/2004:12:28:53) equals "09/01/2004:12:28:53".
See also
todate on page 217
tointeger on page 218
toreal on page 219
Type conversion on page 183
220Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Datatype-conversion functions
19 - Functions forworking with dates
In this section
addcenturies, addcenturiescountbackwards 223adddays 224addhours 225addminutes 226addmonths, addmonthscountbackwards 227addseconds 229addweeks 230addyears, addyearscountbackwards 231countcenturies 232countdays 233counthours 234countminutes 235countseconds 236countweeks 237countwholecenturies, countwholecenturiesbackwards 238countwholedays 240countwholehours 241countwholeminutes 242countwholemonths, countwholemonthsbackwards 243countwholeseconds 245countwholeweeks 246countwholeyears, countwholeyearsbackwards 247countyears 248day 249dayofweek 250gmt2edt 250hour 251minute 252month 253now 253
second 254today 255weekofyear 255year 256
222Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
addcenturies, addcenturiescountbackwards
Purpose: calculate a date from another date using an offset in centuries.
Syntax addcenturies(date, n)
addcenturiescountbackwards(date, n)
Arguments
DescriptionNameType
The starting datedatedate
The number of centuries to add (whichcan be negative)
ninteger
Result
DescriptionType
The date n calendar centuries after the date date(interpreted as |n| centuries before date if n < 0).
The time part of the result is the same as the time part ofdate.
date
See addmonths, addmonthscountbackwards on page227 for an explanation of the two variants.
The result of adding n centuries using the addcenturies (respectivelyaddcenturiescountbackwards) function is defined to be the result of adding 1200n monthswith the addmonths (respectively addmonthscountbackwards) function.
Note:
Examples
To calculate a date one century later than d, showing the difference between the addcenturies andaddcenturiescountbackwards variants (YMD date format):
223Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
addcenturiescountbackwards(d,1)addcenturies(d,1)d
2000/02/152000/02/141900/02/14
See also
countcenturies on page 232
countwholecenturies, countwholecenturiesbackwards on page 238
Date formats on page 52
adddays
Purpose: calculate a date from another date using an offset in days.
Syntax adddays(date, n)
Arguments
DescriptionNameType
The starting datedatedate
The number of days to add (which canbe negative)
ninteger
Result
DescriptionType
The date n days after the date date (interpreted as |n|days before date if n < 0).
The time part of the result is the same as the time part ofdate.
date
Examples
European date format: adddays(#09/01/2004, 25) equals 03/02/2004.
European date format:adddays(#09/01/2004:11:43:46, 25) equals03/02/2004:11:43:46.
224Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
American date format: adddays(#01/09/2004, 25) equals 02/03/2004.
American date format:adddays(#01/09/2004:11:43:46, 25) equals02/03/2004:11:43:46.
YMD date format: adddays(#2004/01/09, 25) equals 2004/02/03.
YMD date format: adddays(#2004/01/09:11:43:46, 25) equals 2004/02/03:11:43:46.
See also
countdays on page 233
countwholedays on page 240
Date formats on page 52
addhours
Purpose: calculate a date from another date using an offset in hours.
Syntax addhours(date, n)
Arguments
DescriptionNameType
The starting datedatedate
The number of hours to add (which canbe negative)
ninteger
Result
DescriptionType
The date n hours after the date date (interpreted as |n|hours before date if n < 0)
date
All dates in FDL include a time component.Note:
Examples
European date format: addhours(#09/01/2004:12:28:53, 25) equals10/01/2004:13:28:53.
225Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
American date format: addhours(#01/09/2004:12:28:53, 25) equals01/10/2004:13:28:53.
YMD date format: addhours(#2004/01/09:12:28:53, 25) equals 2004/01/10:13:28:53.
See also
counthours on page 234
countwholehours on page 241
Date formats on page 52
addminutes
Purpose: calculate a date from another date using an offset in minutes.
Syntax addminutes(date, n)
Arguments
DescriptionNameType
The starting datedatedate
The number of minutes to add (whichcan be negative)
ninteger
Result
DescriptionType
The date n minutes after the date date (interpreted as|n| minutes before date if n < 0)
date
All dates in FDL include a time component.Note:
Examples
European date format: addminutes(#09/01/2004:12:28:53, 25) equals09/01/2004:12:53:53.
American date format: addminutes(#01/09/2004:12:28:53, 25) equals01/09/2004:12:53:53.
226Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
YMD date format:addminutes(#2004/01/09:12:28:53, 25) equals2004/01/09:12:53:53.
See also
countminutes on page 235
countwholeminutes on page 242
Date formats on page 52
addmonths, addmonthscountbackwards
Purpose: calculate a date from another date using an offset in months.
Syntax addmonths(date, n)
addmonthscountbackwards(date, n)
Arguments
DescriptionNameType
The starting datedatedate
The number of months to add (whichcan be negative)
ninteger
Result
DescriptionType
The date n calendar months after the date date(interpreted as |n| months before date if n < 0).
The month and year parts of the result are determined bythe offset n.
date
Then the day of month of the result is set to be the sameas it is in the date date (for addmonths) or the samenumber of days from the end of the month as it is in the datedate (for addmonthscountbackwards).
If the day of month would end up being before the beginningor after the end of the month (because the result month isa shorter month than the month containing date), it isclamped to the first or last day of the month as appropriate.
227Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionType
The time part of the result is the same as the time part ofdate.
Note: • If you add n months using addmonths (or addmonthscountbackwards), you won'tnecessarily get the same result as you would get by adding one month n times, as the dayof month might be constrained by an intermediate short month in the latter case.
• The results of adding to a date with addmonths and addmonthscountbackwards maydiffer if the month part of the date and the month part of the result contain different numbersof days.
Examples
To calculate a date one month later than d, showing the propagation of time parts (YMD date format):
addmonths(d,1)d
2007/09/14:00:00:002007/08/14:00:00:00
2007/09/14:12:34:562007/08/14:12:34:56
To calculate a date one month later than d, showing the difference between the addmonths andaddmonthscountbackwards variants (YMD date format):
addmonthscountbackwards(d,1)addmonths(d,1)d
2007/09/132007/09/142007/08/14
To calculate a date one month later than d, showing the clamping behavior of addmonths at theend of a month (YMD date format):
addmonths(d,1)d
2007/09/302007/08/31
To calculate a date one month later than d, showing the clamping behavior ofaddmonthscountbackwards at the start of the month (YMD date format):
228Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
addmonthscountbackwards(d,1)d
2007/09/012007/08/02
2007/09/012007/08/01
To calculate the six-month anniversary of each customer's acquisition, using the customer acquisitiondate field StartDate:
addmonths(StartDate, 6)
See also
countwholemonths, countwholemonthsbackwards on page 243
Date formats on page 52
addseconds
Purpose: calculate a date from another date using an offset in seconds.
Syntax addseconds(date, n)
Arguments
DescriptionNameType
The starting datedatedate
The number of seconds to add (whichcan be negative)
ninteger
Result
DescriptionType
The date n seconds after the date date (interpreted as|n| seconds before date if n < 0)
date
All dates in FDL include a time component.Note:
229Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
Examples
European date format: addseconds(#09/01/2004:12:28:53, 25) equals09/01/2004:12:54:18.
American date format: addseconds(#01/09/2004:12:28:53, 25) equals01/09/2004:12:54:18.
YMD date format:addseconds(#2004/01/09:12:28:53, 25) equals2004/01/09:12:54:18.
See also
countseconds on page 236
countwholeseconds on page 245
Date formats on page 52
addweeks
Purpose: calculate a date from another date using an offset in weeks.
Syntax addweeks(date, n)
Arguments
DescriptionNameType
The starting datedatedate
The number of weeks to add (which canbe negative)
ninteger
Result
DescriptionType
The date n weeks after the datedate (interpreted as|n|weeks before date if n < 0).
The time part of the result is the same as the time part ofdate.
date
Examples
230Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
European date format: addweeks(#09/01/2004, 8) equals 05/03/2004.
American date format: addweeks(#01/09/2004, 8) equals 03/05/2004.
YMD date format: addweeks(#2004/01/09, 8) equals 2004/03/05.
See also
countweeks on page 237
countwholeweeks on page 246
Date formats on page 52
addyears, addyearscountbackwards
Purpose: calculate a date from another date using an offset in years.
Syntax addyears(date, n)
addyearscountbackwards(date, n)
Arguments
DescriptionNameType
The starting datedatedate
The number of years to add (which canbe negative)
ninteger
Result
DescriptionType
The date n calendar years after the datedate (interpretedas |n| years before date if n < 0).
The time part of the result is the same as the time part ofdate.
date
See addmonths, addmonthscountbackwards on page227 for an explanation of the two variants.
231Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
The result of adding n years using the addyears (respectively addyearscountbackwards)function is defined to be the result of adding 12n months with the addmonths (respectivelyaddmonthscountbackwards) function.
Note:
Examples
To calculate a date one year later than d, showing the difference between the addyears andaddyearscountbackwards variants (YMD date format):
addyearscountbackwards(d,1)addyears(d,1)d
2008/02/152008/02/142007/02/14
To calculate the one-year anniversary of each customer's acquisition, using the customer acquisitiondate field StartDate:
addyears(StartDate, 1)
See also
countwholeyears, countwholeyearsbackwards on page 247
countyears on page 248
Date formats on page 52
countcenturies
Purpose: calculate the period between two dates in centuries.
Syntax countcenturies(date1, date2)
Arguments
DescriptionNameType
the first datedate1date
the second datedate2date
Result
232Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionType
The period between date1 and date2 in centuries (anegative number if date1 is later than date2)
real
Note: • If the number is not an exact multiple of 100 years (taking time parts of dates into account),the result includes a fractional part.
• The result of the countcenturies function is defined to be the result of the countyearsfunction, divided by 100.
Examples
European date format: countcenturies(#09/01/2004, #09/01/2054) equals 0.5.
American date format: countcenturies(#01/09/2004, #01/09/2054) equals 0.5.
YMD date format: countcenturies(#2004/01/09, #2054/01/09) equals 0.5.
See also
countwholecenturies, countwholecenturiesbackwards on page 238
Date formats on page 52
countdays
Purpose: calculate the period between two dates in days.
Syntax countdays(date1, date2)
Arguments
DescriptionNameType
the first datedate1date
the second datedate2date
Result
233Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionType
The period between date1 and date2 in days (anegative number if date1 is later than date2)
real
If the time parts of the two dates differ, the result includes a fractional part.Note:
Examples
European date format:
countdays(#09/01/2004, #03/02/2004) equals 25.
countdays(#09/01/2004:10:00:00, #03/02/2004:11:00:00) equals 25.04.
countdays(#09/01/2004:11:00:00, #03/02/2004:10:00:00) equals 24.96.
American date format:
countdays(#01/09/2004, #02/03/2004) equals 25.
countdays(#01/09/2004:10:00:00, #02/03/2004:11:00:00) equals 25.04.
countdays(#01/09/2004:11:00:00, #02/03/2004:10:00:00) equals 24.96.
YMD date format:
countdays(#2004/01/09, #2004/02/03) equals 25.
countdays(#2004/01/09:10:00:00, #2004/02/03:11:00:00) equals 25.04.
countdays(#2004/01/09:11:00:00, #2004/02/03:10:00:00) equals 24.96.
See also
adddays on page 224
countwholedays on page 240
Date formats on page 52
counthours
Purpose: calculate the period between two dates in hours.
Syntax counthours(time1, time2)
Arguments
234Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionNameType
the first date/timetime1date
the second date/timetime2date
Result
DescriptionType
The period between time1 and time2 in hours (anegative number if time1 is later than time2)
real
If the minutes or seconds parts of the two dates differ, the result includes a fractional part.Note:
Examples
European date format: counthours(#09/01/2004:12:28:53, #10/01/2004:13:28:53)equals 25.
American date format: counthours(#01/09/2004:12:28:53, #01/10/2004:13:28:53)equals 25.
YMD date format: counthours(#2004/01/09:12:28:53, #2004/01/10:13:28:53) equals25.
See also
countwholehours on page 241
Date formats on page 52
countminutes
Purpose: calculate the period between two dates in minutes.
Syntax countminutes(time1, time2)
Arguments
235Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionNameType
the first date/timetime1date
the second date/timetime2date
Result
DescriptionType
The period between time1 and time2 in minutes (anegative number if time1 is later than time2)
real
If the seconds parts of the two dates differ, the result includes a fractional part.Note:
Examples
European date format: countminutes(#09/01/2004:12:28:53, #09/01/2004:12:53:53)equals 25.
American date format: countminutes(#01/09/2004:12:28:53, #01/09/2004:12:53:53)equals 25.
YMD date format: countminutes(#2004/01/09:12:28:53, #2004/01/09:12:53:53)equals 25.
See also
countwholeminutes on page 242
Date formats on page 52
countseconds
Purpose Count the number of seconds between two dates.
Syntax countseconds(time1, time2)
Arguments
236Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionNameType
the first date/timetime1date
the second date/timetime2date
Result
DescriptionType
The number of seconds between time1 and time2 (anegative number if time1 is later than time2)
real
The only difference between this function and countwholeseconds is that the result of thelatter is of type integer.
Note:
Examples
European date format: countseconds(#09/01/2004:12:28:53, #09/01/2004:12:29:18)equals 25.
American date format: countseconds(#01/09/2004:12:28:53, #01/09/2004:12:29:18)equals 25.
YMD date format: countseconds(#2004/01/09:12:28:53, #2004/01/09:12:29:18)equals 25.
See also
Date formats on page 52
countweeks
Purpose: calculate the period between two dates in weeks.
Syntax countweeks(date1, date2)
Arguments
DescriptionNameType
the first datedate1date
237Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionNameType
the second datedate2date
Result
DescriptionType
The period between date1 and date2 in weeks (anegative number if date1 is later than date2)
real
If the period is not an exact multiple of seven days (taking time parts of dates into account),the result includes a fractional part.
Note:
Examples
European date format: countweeks(#09/01/2004, #05/03/2004) equals 8.
American date format: countweeks(#01/09/2004, #03/05/2004) equals 8.
YMD date format: countweeks(#2004/01/09, #2004/03/05) equals 8.
See also
countwholeweeks on page 246
Date formats on page 52
countwholecenturies, countwholecenturiesbackwards
Purpose: count the number of complete centuries between two dates.
Syntaxcountwholecenturies(date1, date2)countwholecenturiesbackwards(date1,date2)
Arguments
DescriptionNameType
the first datedate1date
the second datedate2date
238Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
Result
DescriptionType
The number of complete centuries elapsed betweendate1and date2 (a negative number if date1 is later thandate2).
See countwholemonths, countwholemonthsbackwardson page 243 for an explanation of the two variants.
integer
Note: • The result of the countwholecenturies (respectivelycountwholecenturiesbackwards) function is defined as the result of thecountwholemonths (respectively countwholemonthsbackwards) function dividedby 1200, ignoring any remainder.
• The results of comparing two dates with countwholecenturies andcountwholecenturiesbackwardsmay differ, but only when both dates lie in February,one date lies in a year divisible by 400, and the other lies in a year divisible by 100 but notdivisible by 400.
Examples
To count the number of complete centuries between d1 and d2, showing the difference betweenthe countwholecenturies and countwholecenturiesbackwards variants (YMD date format):
countwholecenturiesbackwards(d1,d2)countwholecenturies(d1,d2)d2d1
002000/02/131900/02/14
012000/02/141900/02/14
112000/02/151900/02/14
See also
addcenturies, addcenturiescountbackwards on page 223
countcenturies on page 232
Date formats on page 52
239Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
countwholedays
Purpose: count the number of complete days between two dates.
Syntax countwholedays(date1, date2)
Arguments
DescriptionNameType
the first datedate1date
the second datedate2date
Result
DescriptionType
The number of complete 24-hour periods elapsed betweendate1 anddate2 (a negative number ifdate1 is laterthan date2)
integer
Examples
European date format:
countdays(#09/01/2004, #03/02/2004) equals 25.
countdays(#09/01/2004:10:00:00, #03/02/2004:11:00:00) equals 25.
countdays(#09/01/2004:11:00:00, #03/02/2004:10:00:00) equals 24.
American date format:
countdays(#01/09/2004, #02/03/2004) equals 25.
countdays(#01/09/2004:10:00:00, #02/03/2004:11:00:00) equals 25.
countdays(#01/09/2004:11:00:00, #02/03/2004:10:00:00) equals 24.
YMD date format:
countdays(#2004/01/09, #2004/02/03) equals 25.
countdays(#2004/01/09:10:00:00, #2004/02/03:11:00:00) equals 25.
240Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
countdays(#2004/01/09:11:00:00, #2004/02/03:10:00:00) equals 24.
See also
adddays on page 224
countdays on page 233
Date formats on page 52
countwholehours
Purpose: count the number of complete hours between two dates.
Syntax countwholehours(time1, time2)
Arguments
DescriptionNameType
the first date/timetime1date
the second date/timetime2date
Result
DescriptionType
The number of complete hours elapsed between time1and time2 (a negative number if time1 is later thantime2)
integer
Examples
European date format: countwholehours(#09/01/2004:12:28:53,#10/01/2004:13:28:53) equals 24.
American date format: countwholehours(#01/09/2004:12:28:53,#01/10/2004:13:28:53) equals 24.
YMD date format: countwholehours(#2004/01/09:12:28:53, #2004/01/10:13:28:53)equals 24.
See also
241Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
counthours on page 234
Date formats on page 52
countwholeminutes
Purpose: count the number of complete minutes between two dates.
Syntax countwholeminutes(time1, time2)
Arguments
DescriptionNameType
the first date/timetime1date
the second date/timetime2date
Result
DescriptionType
The number of complete minutes elapsed betweentime1and time2 (a negative number if time1 is later thantime2)
integer
Examples
European date format: countwholeminutes(#09/01/2004:12:28:53,#09/01/2004:12:53:53) equals 24.
American date format: countwholeminutes(#01/09/2004:12:28:53,#01/09/2004:12:53:53) equals 24.
YMD date format:countwholeminutes(#2004/01/09:12:28:53, #2004/01/09:12:53:53)equals 24.
See also
countminutes on page 235
Date formats on page 52
242Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
countwholemonths, countwholemonthsbackwards
Purpose: count the number of complete months between two dates.
Syntax countwholemonths(date1, date2) countwholemonthsbackwards(date1,date2)
Arguments
DescriptionNameType
the first datedate1date
the second datedate2date
Result
DescriptionType
The number of complete months elapsed betweendate1and date2 (a negative number if date1 is later thandate2)
integer
Note: • The time parts of date1 and date2 are ignored.• The number of complete months between two dates is the greatest number of months that
can be added to the first date (truncating time parts, and using addmonths in the case ofcountwholemonths, and addmonthscountbackwards in the case ofcountwholemonthsbackwards) such that if the second date is later than the first date,the result is no later than the second date, and if the second date is earlier than the firstdate, the result is no earlier than the second date.
• The results of comparing two dates with countwholemonths andcountwholemonthsbackwards may differ when the dates lie in months of differentlengths.
• Because of the clamping behavior of addmonths (and addmonthscountbackwards),the absolute value of the results of comparing two dates with countwholemonths (orcountwholemonthsbackwards) may depend on the order of the arguments when thedates lie in months of different lengths.
Examples
243Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
To count the number of complete months between d1 and d2, showing the truncation of time parts(YMD date format):
countwholemonths(d1,d2)d2d1
12007/09/14:00:00:002007/08/14:00:00:00
12007/09/14:00:00:002007/08/14:12:34:56
To count the number of complete months between d1 and d2, showing the difference between thecountwholemonths and countwholemonthsbackwards variants (YMD date format):
countwholemonthsbackwards(d1,d2)countwholemonths(d1,d2)d2d1
002007/09/122007/08/14
102007/09/132007/08/14
112007/09/142007/08/14
To count the number of complete months between d1 and d2, showing the clamping behavior ofcountwholemonths at the end of the month (YMD date format):
countwholemonths(d2,d1)countwholemonths(d1,d2)d2d1
012007/09/302007/08/31
To count the number of complete months between d1 and d2, showing the clamping behavior ofcountwholemonths at the start of the month (YMD date format):
countwholemonthsbackwards(d2,d1)countwholemonthsbackwards(d1,d2)d2d1
0-12007/06/012007/07/01
To count the number of monthly e-mail communications each customer has received, wherecommunications are sent on the fifth day of each month (the final communication having been senton March 5th 2006), and using the customer acquisition date field StartDate:
-countwholemonths(#2006/03/05, StartDate)
244Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
To count the number of monthly statements each customer has received, where statement are sentseven days before the end of each month (the final statement having been sent on March 24th2006), and using the customer acquisition date field StartDate:
-countwholemonthsbackwards(#2006/03/24, StartDate)
See also
addmonths, addmonthscountbackwards on page 227
Date formats on page 52
countwholeseconds
Purpose: count the number of seconds between two dates.
Syntax countwholeseconds(time1, time2)
Arguments
DescriptionNameType
the first date/timetime1date
the second date/timetime2date
Result
DescriptionType
The number of seconds between time1 and time2 (anegative number if time1 is later than time2)
integer
The only difference between this function and countseconds is that the result of the latteris of type real.
Note:
Examples
European date format: countwholeseconds(#09/01/2004:12:28:53,#09/01/2004:12:29:18) equals 25.
American date format: countwholeseconds(#01/09/2004:12:28:53,#01/09/2004:12:29:18) equals 25.
245Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
YMD date format:countwholeseconds(#2004/01/09:12:28:53, #2004/01/09:12:29:18)equals 25.
See also
Date formats on page 52
countwholeweeks
Purpose: count the number of complete weeks between two dates.
Syntax countwholeweeks(date1, date2)
Arguments
DescriptionNameType
the first datedate1date
the second datedate2date
Result
DescriptionType
The number of complete weeks elapsed between date1and date2 (a negative number if date1 is later thandate2)
integer
Examples
European date format: countwholeweeks(#09/01/2004, #09/03/2004) equals 8.
American date format: countwholeweeks(#01/09/2004, #03/09/2004) equals 8.
YMD date format: countwholeweeks(#2004/01/09, #2004/03/09) equals 8.
See also
countweeks on page 237
Date formats on page 52
246Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
countwholeyears, countwholeyearsbackwards
Purpose: count the number of complete years between two dates.
Syntax countwholeyears(date1, date2)
countwholeyearsbackwards(date1, date2)
Arguments
DescriptionNameType
the first datedate1date
the second datedate2date
Result
DescriptionType
The number of complete years elapsed between date1and date2 (a negative number if date1 is later thandate2).
See countwholemonths, countwholemonthsbackwardson page 243 for an explanation of the two variants.
integer
Note: • The result of the countwholeyears (respectively countwholeyearsbackwards)function is defined as the result of the countwholemonths (respectivelycountwholemonthsbackwards) function divided by 12, ignoring any remainder.
• The results of comparing two dates with countwholeyears andcountwholeyearsbackwards may differ, but only when both dates lie in February, onedate lies in a leap year, and the other does not.
Examples
To count the number of complete years between d1 and d2, showing the difference between thecountwholeyears and countwholeyearsbackwards variants (YMD date format):
247Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
countwholeyearsbackwards(d1,d2)countwholeyears(d1,d2)d2d1
002008/02/132007/02/14
012008/02/142007/02/14
112008/02/152007/02/14
To compute the age of each customer at March 1st 2007 from a date-of-birth field DOB (YMD dateformat):
countwholeyears(DOB, #2007/03/01)
See also
addyears, addyearscountbackwards on page 231
countyears on page 248
Date formats on page 52
countyears
Purpose: calculate the period between two dates in years.
Syntax countyears(date1, date2)
Arguments
DescriptionNameType
the first datedate1date
the second datedate2date
Result
DescriptionType
The period between date1 and date2 in years (anegative number if date1 is later than date2)
real
248Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
Note: • If the month, day, or time parts of the two dates differ, the result includes a fractional part,counting the incomplete year as a fraction of the number of days (365 or 366) in the yearof the later of the two dates.
Examples
European date format: countyears(#09/01/2004, #09/01/1994) equals -10.
American date format: countyears(#01/09/2004, #01/09/1994) equals -10.
YMD date format: countyears(#2004/01/09, #1994/01/09) equals -10.
See also
countwholeyears, countwholeyearsbackwards on page 247
Date formats on page 52
day
Purpose: obtain the day-of-month part of a date.
Syntax day(date)
Arguments
DescriptionNameType
A datedatedate
Result
DescriptionType
The day-of-month part of dateinteger
Examples
European date format: day(#09/01/2004) equals 9.
American date format: day(#01/09/2004) equals 9.
YMD date format: day(#2004/01/09) equals 9.
See also
249Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
Date formats on page 52
Splitting aggregations: the by clause on page 150
dayofweek
Purpose: obtain a number representing the day-of-week of a date.
Syntax dayofweek(date)
Arguments
DescriptionNameType
A datedatedate
Result
DescriptionType
The day-of-week ofdate (a number between 0 and 6, with0 representing Sunday)
integer
Examples
European date format: dayofweek(#09/01/2004) equals 5 (that is, Friday).
American date format: dayofweek(#01/09/2004) equals 5 (that is, Friday).
YMD date format: dayofweek(#2004/01/09) equals 5 (that is, Friday).
See also
Date formats on page 52
Splitting aggregations: the by clause on page 150
gmt2edt
Purpose: convert a Greenwhich Mean Time (GMT) date to Eastern Daylight Time (EDT).
250Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
Syntax gmt2edt(datetimeGMT)
Arguments
DescriptionNameType
The GMT date to be converteddatetimeGMTdate
Result
DescriptionType
The EDT date, accounting for daylight savingsdate
Examples
Winter: GMT2EDT(#09/01/2004:12:28:53) equals 09/01/2004:07:28:53.
Summer: GMT2EDT(#09/07/2004:12:28:53) equals 09/07/2004:08:28:53.
hour
Purpose: obtain the hours part of a date.
Syntax hour(time)
Arguments
DescriptionNameType
A date/timetimedate
Result
DescriptionType
The hours part of timeinteger
Examples
251Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
European date format: hour(#09/01/2004:12:28:53) equals 12.
American date format: hour(#01/09/2004:12:28:53) equals 12.
YMD date format: hour(#2004/01/09:12:28:53) equals 12.
See also
Date formats on page 52
Splitting aggregations: the by clause on page 150
minute
Purpose: obtain the minutes part of a date.
Syntax minute(time)
Arguments
DescriptionNameType
A date/timetimedate
Result
DescriptionType
The minutes part of timeinteger
Examples
European date format: minute(#09/01/2004:12:28:53) equals 28.
American date format: minute(#01/09/2004:12:28:53) equals 28.
YMD date format: minute(#2004/01/09:12:28:53 equals 28.
See also
Date formats on page 52
Splitting aggregations: the by clause on page 150
252Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
month
Purpose: obtain the month part of a date.
Syntax month(date)
Arguments
DescriptionNameType
A datedatedate
Result
DescriptionType
The month part of dateinteger
Examples
European date format: month(#09/01/2004) equals 1.
American date format: month(#01/09/2004) equals 1.
YMD date format: month(#2004/01/09) equals 1.
See also
Date formats on page 52
Splitting aggregations: the by clause on page 150
now
Purpose: obtain the current date and time.
Syntax now()
Arguments None
Result
253Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionType
The current date (including time), obtained from the systemclock
date
Examples
European date format: now() equals 09/01/2004:12:28:53.
American date format: now() equals 01/09/2004:12:28:53.
YMD date format: now() equals 2004/01/09:12:28:53.
See also
Date formats on page 52
today on page 255
second
Purpose: obtain the seconds part of a date.
Syntax second(time)
Arguments
DescriptionNameType
A date/timetimedate
Result
DescriptionType
The seconds part of timeinteger
Examples
European date format: second(#09/01/2004:12:28:53) equals 53.
American date format: second(#01/09/2004:12:28:53) equals 53.
YMD date format: second(#2004/01/09:12:28:53) equals 53.
254Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
See also
Date formats on page 52
Splitting aggregations: the by clause on page 150
today
Purpose: obtain the current date.
Syntax today()
Arguments None
Result
DescriptionType
The current date obtained from the system clock, with thetime part set to 00:00:00
date
Examples
European date format: today() equals 09/01/2004:00:00:00.
American date format: today() equals 01/09/2004:00:00:00.
YMD date format: today() equals 2004/01/09:00:00:00.
See also
Date formats on page 52
now on page 253
weekofyear
Purpose: calculate the week-of-year of a date, relative to a specified start date.
Syntax weekofyear(date, yearStart)
Arguments
255Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionNameType
The date whose week-of-year is to beobtained
datedate
The start date (for example, of afinancial year)
yearStartdate
Result
DescriptionType
The week-of-year of date, relative to yearStartinteger
Only the month and day parts of the start date are used.Note:
Examples
European date format: weekofyear(#09/07/2004, #01/04/2004) equals 14.
American date format: weekofyear(#07/09/2004, #04/01/2004) equals 14.
YMD date format: weekofyear(#2004/07/09, #2004/04/01) equals 14.
See also
Date formats on page 52
Splitting aggregations: the by clause on page 150
year
Purpose: obtain the year part of a date.
Syntax year(date)
Arguments
DescriptionNameType
A datedatedate
Result
256Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
DescriptionType
The year part of dateinteger
Examples
European date format: year(#09/01/2004) equals 2004.
American date format: year(#01/09/2004) equals 2004.
YMD date format: year(#2004/01/09) equals 2004.
See also
Date formats on page 52
257Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with dates
20 - Functions forworking with strings
In this section
concat 259endswith 260find 261left 262mid 263right 264soundex 265startswith 266strlen 267strmember 268substitute 269substr 270tolower 272toupper 273trim 273
concat
Purpose: concatenate two or more strings.
Syntax concat( , , ...)
Arguments
DescriptionNameType
Strings to be concatenated, , ...string
Result
DescriptionType
The string obtained by writing the characters of , followedby the characters of , etc.
string
Examples
Derive a customer's complete name from two fields FirstName and Surname:
concat(FirstName,"",Surname)
SurnameFirstName
John BrownBrownJohn
David SmithSmithDavid
Derive a customer's complete name, but using only their initial:
concat(substr(FirstName,0,0),".",Surname)
SurnameFirstName
J. BrownBrownJohn
D. SmithSmithDavid
259Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
endswith
Purpose: test whether one string ends with another.
Syntax endswith(find_text, within_text)
Arguments
DescriptionNameType
The string to be found at the endfind_textstring
The string to be testedwithin_textstring
Result
DescriptionType
1 (true) if find_text is found at the end ofwithin_text; 0 (false) otherwise
integer
The endswith function is based on the match function. It appends a $ character tofind_text before treating the result as a regular expression [see Regular expressionson page 276].
Note:
Examples
To derive a field that flags e-mail addresses from .net domains:
endswith(".net", EmailAddress)EmailAddress
1J.Brown@glob.net
0D.Smith@spectrumsoftware.com
See also
find on page 261
startswith on page 266
260Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
find
Purpose Test whether one string is contained within another.
Syntax find(find_text, within_text)
Arguments
DescriptionNameType
The string to be foundfind_textstring
The string to be testedwithin_textstring
Result
DescriptionType
1 (true) if find_text is found in within_text; 0(false) otherwise
integer
The find function is based on the match function, and it treats find_text as a regularexpression [see Regular expressions on page 276].
Note:
Examples
To flag e-mail addresses containing the text "minersoftware" in upper, lower, or mixed case:
find("minersoftware",tolower(EmailAddress))
EmailAddress
1J.BROWN@PITNEYBOWES.COM
0J.Brown@glob.net
1D.Smith@pitneybowes.com
See also
endswith on page 260
261Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
startswith on page 266
left
Purpose: return a substring of specified length from the start of a string.
Syntax left(text, num_chars)
Arguments
DescriptionNameType
The string to extract fromtextstring
The number of charactersnum_charsinteger
Result
DescriptionType
The substring formed by num_chars characters at thestart of text
string
The left function is based on the substr function.Note:
Examples
To use the first two characters of a StateCode field to identify west-coast customers:
member (left (StateCode, 2), "CA","OR", WA")
StateCode
1OR009
1CA043
0UT005
1WA027
262Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
See also
mid on page 263
right on page 264
trim on page 273
mid
Purpose: return a substring of specified length from the middle of a string.
Syntax mid(text, start_num, num_chars)
Arguments
DescriptionNameType
The string to extract fromtextstring
The first indexstart_numstring
The number of charactersnum_charsstring
Result
DescriptionType
The substring formed by thenum_chars characters fromcharacter position start_num (inclusive) of text,where the character position are numbered starting from 1
string
The mid function is based on the substr function.Note:
Examples
To derive a location from the fifth through seventh characters of a store code:
263Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
mid(StoreCode, 5, 3)StoreCode
EDI0114EDI256
LON1863LON836
EDI9326EDI039
BOS0387BOS041
See also
left on page 262
right on page 264
trim on page 273
right
Purpose: return a substring of specified length from the end of a string.
Syntax right(text, num_chars)
Arguments
DescriptionNameType
The string to extract fromtextstring
The number of charactersnum_charsinteger
Result
DescriptionType
The substring formed by num_chars characters at theend of text
string
The right function is based on the substr function.Note:
264Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
Examples
To return the house/street component of a UK postcode field, that is, the last three characters:
right(Postcode, 3)Postcode
0AYNE25 0AY
7RAEH3 7RA
1QXSL4 1QX
0NJEH9 0NJ
See also
left on page 262
mid on page 263
trim on page 273
soundex
Purpose: reduce each word to a four-character string for indexing purposes.
Syntax soundex(text)
Arguments
DescriptionNameType
The text to reducetextstring
Result
DescriptionType
A four-character reduced index codingstring
265Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
Examples
To derive the soundex code for a family name:
soundex(Surname)Surname
B650Brown
S530Smythe
S530Smith
B650Bruno
The Soundex function is not defined for non ascii characters.Note:
startswith
Purpose: test whether one string starts with another.
Syntax startswith(find_text, within_text)
Arguments
DescriptionNameType
The string to look for at the startfind_textstring
The string to be testedwithin_textstring
Result
DescriptionType
1 iffind_text is found at the start ofwithin_text;0 otherwise
integer
266Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
The startswith function is based on the match function. It prepends a ^ character tofind_text before treating the result as a regular expression [see Regular expressionson page 276].
Note:
Examples
To derive the type of credit card used in a transaction, such as "Discover", "AmEx", "Visa" or"MasterCard", based on the card prefix numbers:
case
startswith ("6011", CC_Number): "Discover";
startswith ("4", CC_Number): "Visa";
startswith ("5", CC_Number): "MasterCard";
startswith ("34",CC_Number) or startswith ("37", CC_Number): "AmEx";
default: null;
CC_TypeCC_Number
Visa4111-1111-1111-1111
MasterCard5431-1111-1111-1111
AmEx341-1111-1111-1111
Discover6011-6011-6011-6611
See also
endswith on page 260
find on page 261
strlen
Purpose: obtain the length of a string.
Syntax strlen(string)
Arguments
267Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
DescriptionNameType
The string whose length is to bedetermined
stringstring
Result
DescriptionType
The number of characters in stringinteger
Examples
To derive a string field of exactly four characters, by padding a shorter string field with leading zeros:
case
strlen(AccountID) = 1: concat("000", AccountID);
strlen(AccountID) = 2: concat("00", AccountID);
strlen(AccountID) = 3: concat("0", AccountID);
default: AccountID;
AccountAccountID
00011
002222
0333333
44444444
strmember
Purpose: determine set membership.
Syntax strmember(x, , , ...)
Arguments
268Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
DescriptionNameType
The value to be testedxstring
The elements of the set, , ...as x
Result
DescriptionType
1 (true) if x is in the set ; 0 (false) otherwiseinteger
Note: • You can only use string literal values for the list of set elements.• Trailing whitespace in the values is not ignored.• This function is deprecated in favor of the more generalmember function.
See also
Splitting aggregations: the by clause on page 150
substitute
Purpose: replace one string with another.
Syntax substitute(text, old_text, new_text)
Arguments
DescriptionNameType
The string to be searchedtextstring
The string to be replacedold_textstring
The replacement stringnew_textstring
Result
269Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
DescriptionType
A copy of the stringtextwithnew_text replacing eachnon-overlapping instance of old_text
string
The substitute function is based on the replaceall function, and it treats old_text as aregular expression [see Regular expressions on page 276].
Note:
Examples
To remove whitespace from a field named PostalCode:
substitute(PostalCode,"[[:blank:]]", "")
PostalCode
ND069ND 069
IN099IN 099
UT005UT 0 0 5
OR009O R 009
See also
replacefirst on page 283
substr
Purpose: obtain a substring from a string.
Syntax substr(string, start, end)
Arguments
DescriptionNameType
The initial stringstringstring
The first indexstartinteger
270Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
DescriptionNameType
The second indexendinteger
Result
DescriptionType
The substring formed from charactersstart throughend(inclusive) ofstring, where the characters are numbered
string
starting from 0. Positive index values count from the startof the string, and negative index values count from the endof the string.
Examples
To select the first two and last three characters of a PostalCode field to give the state code:
substr(PostalCode, -3,-1)
substr(PostalCode, 0, 1)PostalCode
015MAMA015
055WAWA055
005UTUT005
119MNMN119
To derive the type of credit card used in a transaction, such as "Discover", "AmEx", "Visa" or"MasterCard", based on the card prefix numbers:
case
substr(CC_Number, 0, 3) = "6011": "Discover";
substr(CC_Number, 0, 0) = "4": "Visa";
substr(CC_Number, 0, 0) = "5": "MasterCard";
substr(CC_Number, 0, 1) = "34" or substr(CC_Number, 0, 1) = "37": "AmEx";
default: null;
271Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
CC_TypeCC_Number
Visa4111-1111-1111-1111
MasterCard5431-1111-1111-1111
AmEx341-1111-1111-1111
Discover6011-6011-6011-6611
tolower
Purpose: convert a string to lowercase text.
Syntax tolower(string)
Arguments
DescriptionNameType
A string to be convertedstringstring
Result
DescriptionType
The string obtained from string by replacing everyuppercase letter in string with the correspondinglowercase letter
string
Examples tolower("ZYXWV") equals "zyxwv"
See also
toupper on page 273
272Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
toupper
Purpose: convert a string to uppercase text.
Syntax toupper(string)
Arguments
DescriptionNameType
A string to be convertedstringstring
Result
DescriptionType
The string obtained from string by replacing everylowercase letter in string with the correspondinguppercase letter
string
Examples toupper("abcde") equals "ABCDE"
See also
tolower on page 272
trim
Purpose: remove all spaces from a string except for single spaces between words.
Syntax trim(text)
Arguments
DescriptionNameType
The string to be trimmedtextstring
273Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
Result
DescriptionType
The string formed from text removing all spaces exceptfor single spaces between words
string
The trim function is based on the replaceall function.Note:
Examples
To derive a field that replace multiple whitespace characters with a single space:
trim(CommentField)CommentField
Returned faulty goodsReturned faulty goods
Missed appointmentMissed appointment
Engineer to visit: MondayEngineer to visit: Monday
ResolvedResolved
274Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Functions for working with strings
21 - Regularexpressions andassociated functions
In this section
Regular expressions 276Basic components of a regular expression 276Regular-expression operators 279match 280replaceall 282replacefirst 283
Regular expressions
Regular expressions are for matching patterns in strings [see Datatypes on page 181]. A regularexpression is a specially formatted string that represents a pattern of characters. For example, theregular expression "ar{1,2}y?$" represents "the letter 'a,' followed by one or two 'r's, possibly followedby 'y,' followed by the end of a string." (That pattern occurs in strings like "marry" and "jar" — butnot in "may" or "jars.")
A regular expression consists of basic components [seeBasic components of a regular expressionon page 276] combined using operators [see Regular-expression operators on page 279].
If a regular expression (other than a subexpression) could match more than one portion of a string,it always matches the leftmost, longest candidate portion. For example, the regular expression"a[^ad]+" (representing "the letter 'a', followed by one or more letters other than 'a' or 'd'") matches"adaptationally." The matched portion is not "ation," as there are candidate portions further to theleft. Neither is it "ap," because, although there are no candidates further to the left, there is a longercandidate portion starting at the same character.
• To test a string for a regular expression match, use the match function.• To replace matched portions of a string with another string, use the replacefirst or replaceall
function.
Note: • Regular expressions are typically (but not necessarily) literals, in which case they must beenclosed in double quotation marks, for example, "ar{1,2}y?$".
• Miner/FDL uses Regular expression provided by the ICU library. (we can link tohttp://userguide.icu-project.org/strings/regexp)
• ICU Regular Expressions conform toUnicode Technical Standard #18 , Unicode RegularExpressions, level 1, and in addition include Default Word boundaries and Name Propertiesfrom level 2.
Basic components of a regular expression
The basic components of a regular expression [see Regular expressions on page 276] are atoms,anchors, and back-references.
AtomsAn atom matches exactly one of a specified set of characters:
276Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Regular expressions and associated functions
Character matchedAtom
Any character. (match-any)
Any one of the specified characters[...] (match-list)
Any character other than one of the specified characters[^...] (non-match-list)
The character itselfA character other than ., ^, $, \, [, ], or *,occurring outside a match-list or non-match-list
The character itself., ^, $, \, [, ], or *, occurring outside amatch-list or non-match-list
A match-list or non-match-list can include simple characters, character ranges, and character classes.You cannot use character ranges or character classes except in a match-list or non-match-list.
A character range is a pair of characters separated by a hyphen and is equivalent to a list of all thecharacters in that range (within the character set). For example, "a-e" is equivalent to "abcde" ina match-list.
A character class is one of the following pre-defined tokens (where the enclosing brackets areadditional to those enclosing the match-list or non-match-list):
MatchesToken
An alphanumeric character, that is an alphabetic characteror decimal digit
[:alnum:]
An alphabetic character, that is, "A" — "Z" and "a" — "z"[:alpha:]
A tab or space character[:blank:]
A control character[:cntrl:]
A decimal digit, that is, "0" — "9"[:digit:]
A printable character other than space[:graph:]
A lowercase letter[:lower:]
A printable character or space[:print:]
277Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Regular expressions and associated functions
MatchesToken
A punctuation character[:punct:]
A whitespace character[:space:]
An uppercase character[:upper:]
A hexadecimal digit, that is, "0" — "9," "A" — "F," and "a"— "f"
[:xdigit:]
Note: • You can include a mixture of simple characters, character ranges, and character classesin a single match-list or non-match-list. For example, the match-list "[a-ex[:digit:]]"matches the lowercase letters "a" — "e" and "x," as well as decimal digits ("0" — "9").
• A hyphen in a match-list or non-match-list stands for itself when it is (a) at the end of arange, (b) at the start of the match-list or non-match-list (in which case it can also serveas the start of a range), or (c) at the end of the match-list or non-match-list.
AnchorsThe characters ^ and $ are anchors; they match the beginning and end of a string respectively.
For example, to match "The" occurring at the start of a string and nowhere else, you could use theregular expression "^The."
Back-references
A back-reference is a backslash followed by a single digit n (other than 0). It matches exactly thesame characters as the nth subexpression enclosed in \(...\).
For example,to match "yoyo," "dodo," etc. — but not "dojo" — you could use the regular expression"\(.o\)$1."
Note: • The nth back-referenced subexpression is always the subexpression beginning with thenth opening backslash/parenthesis (even if subexpressions are nested).
• The empty subexpression "\(\)" can be useful in regular expressions involving alternation[see The alternation operator on page 280]. For example,
\(fish\(\)\|profit\(\)\)-and-\(chips$2\|loss$3\)
matches both "fish-and-chips" and "profit-and-loss," but not "fish-and-loss" or"profit-and-chips."
278Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Regular expressions and associated functions
Regular-expression operators
An operator acts on one or two regular expressions [see Regular expressions on page 276] tocreate a new regular expression. The operators, from highest to lowest precedence, are the repetitionoperators [see Repetition operators on page 279], the concatenation operator [see Theconcatenation operator on page 280], and the alternation operator [see The alternation operatoron page 280].
If there is a tie for precedence, operations are carried out from left to right.
Subexpressions enclosed in (...) are always evaluated first.
Repetition operators
The repetition operators are unary and are written following their single operand.
DescriptionOperator
If a regular expression a matches one or more adjacentportions of a string, the regular expression a* matches the
* (zero-or-more)
concatenation of those portions. It also matches any emptysubstring.
If a regular expression a matches one or more adjacentportions of a string, the regular expression a\+ matchesthe concatenation of those portions.
\+ (one-or-more)
If a regular expression a matches at least m and no morethan n adjacent portions of a string, the regular expression
{m,n} (0 m n)
a{m,n} matches the concatenation of those portions. Ifm=0, it also matches any empty substring.
Equivalent to a{0,1}, the regular expression a\?matches any empty substring, or any portion of a string thatthe regular expression a matches.
\? (optional)
The "adjacent portions" mentioned in the definitions of *, \+, and {...} are not necessarilyidentical strings. If you need to match exact repetitions, use back-references [seeBack-references on page 278].
Note:
279Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Regular expressions and associated functions
The concatenation operator
The concatenation operator is not actually written, but is implied whenever you write two regularexpressions next to each other.
If regular expressions a and b match adjacent portions x and y of a string z (with x preceding y),the concatenation of a and b (written as a followed by b) matches the portion of z containing x andy.
Any string of non-special characters, that is, characters with no special meaning in a regularexpression, serves as a regular expression that matches that string, because of the implicitconcatenation operator.
Note:
In practice, most regular expressions consist of simple character strings, interspersed witha few special characters. For example, the regular expression "[SsDd]imple" matches thestrings "Simple," "simple," "Dimple," and "dimple."
The alternation operator
The alternation operator "|" is a binary operator, written between its operands.
If either of the regular expressions a and b matches a portion of a string, the alternation a|b alsomatches that portion.
match
Purpose: test a string for a regular expression match [see Regular expressions on page 276].
Syntax match(regexp, string)
Arguments
DescriptionNameType
The regular expressionregexpstring
The string to be testedstringstring
Result
280Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Regular expressions and associated functions
DescriptionType
1 (true) if regexpmatches string; 0 (false) otherwiseinteger
Examples
To flag customers whose home postcode and bank branch postcode match in the first two characters(using the substr function):
match(substr(HomePostcode,0, 1),substr(BranchPostcode,0,1))
BranchPostCodeHomePostcode
1NE30 1QXNE25 0AY
0NG18 1HTSW3
1CF31 1HYCF36
0E10 8AJEH9 0NJ
To look in a company name field for the first three characters from a first-name field, followed byany text and then the contents of a family name field (ignoring case):
rx := concat(substr(toupper(FirstName),0,2),"[[:print:]]*", toupper(FamilyName);match(rx,toupper(CompanyName))
CompanyNameFamilyNameFirstName
11234 Pat Smith LtdSmithPatrick
11234 Pat J Smith LtdSmithPatrick
To do the same, but with only some letters or digits and a single space between the first name andfamily name:
281Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Regular expressions and associated functions
rx := concat(substr(toupper(FirstName),0,2),"[[:alnum:]]* ",toupper(FamilyName);match(rx,toupper(CompanyName))
CompanyNameFamilyNameFirstName
11234 Pat Smith LtdSmithPatrick
01234 Pat J Smith LtdSmithPatrick
See also
replaceall on page 282
replacefirst on page 283
Splitting aggregations: the by clause on page 150
replaceall
Purpose: replace all substrings that match a regular expression [see Regular expressions onpage 276].
Syntax replaceall(regexp, replacement, string)
Arguments
DescriptionNameType
The regular expressionregexpstring
The replacement stringreplacementstring
The string to be searchedstringstring
Result
282Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Regular expressions and associated functions
DescriptionType
A copy of the string string with replacementreplacing each non-overlapping portion that regexpmatches
string
The first matching portion of the string to be searched is the leftmost, longest portion thatthe regular expression matches. The next search applies to the remainder of the string tobe searched, and so on. For example,
Note:
replaceall("i..", "X", "initiation")
yields the string "XtXX" (rather than, say, "inXatX"). And
replaceall("i.*i", "X", "initiation")
gives the result "Xon" (rather than, say, "XtXon").
Examples
To detect and remove whitespace from a PostalCode field using a regular expression match:
replaceall("[[:blank:]]","",PostalCode))PostalCode
ND069ND 069
IN099IN 099
UT005UT 0 0 5
OR009O R 009
See also
match on page 280
replacefirst on page 283
replacefirst
Purpose: replace the first substring that matches a regular expression [see Regular expressionson page 276].
Syntax replacefirst(regexp, replacement, string)
283Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Regular expressions and associated functions
Arguments
DescriptionNameType
The regular expressionregexpstring
The replacement stringreplacementstring
The string to be searchedstringstring
Result
DescriptionType
A copy of the string string with replacementreplacing the first portion that regexp matches (if any)
string
Examples
To extract a customer's name from an e-mail address field in the format"Firstname.Surname@company.com", by removing everything from the "@" onwards andreplacing the "." with a space:
replacefirst("@.*", "",(replacefirst ("\.", " ", Email)))
John BrownJohn.Brown@myco.com
Angela SmithAngela.Smith@yourco.com
See also
match on page 280
replaceall on page 282
284Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Regular expressions and associated functions
22 - Mathematical andstatistical functions
In this section
abs 286ceil 286cos 287exp 288floor 288log 289log10 290logbase 290max (two or more arguments), maxnonnull 291mean (two or more arguments), meannonnull 292min (two or more arguments), minnonnull 293normalize 294pow 294product, productnonnull 295round 296sgn 297sin 297sqrt 298sum (two or more arguments), sumnonnull 299tan 300
abs
Purpose: calculate the absolute value of a number.
Syntax abs(x)
Arguments
DescriptionNameType
A signed numberxnumeric
Result
DescriptionType
, the absolute value, or magnitude, of xas input
Examples
abs(10000) equals 10000.
abs(-25000) equals 25000.
To calculate the absolute error between a predicted value and a historical value: abs(Value -PredictedValue)
See also
sgn on page 297
ceil
Purpose: round a number up to the nearest integer.
Syntax ceil(x)
Arguments
286Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
DescriptionNameType
The number to be roundedxnumeric
Result
DescriptionType
, the integer such thatinteger
Examples
ceil(1.25) equals 2.
ceil(-1.25) equals -1.
To generate a random integer between 1 and 10 inclusive: ceil(rndUniform()*10)
See also
floor on page 288
round on page 296
cos
Purpose: calculate the cosine of an angle.
Syntax cos(x)
Arguments
DescriptionNameType
An angle in radiansxnumeric
Result
DescriptionType
cos x, the cosine of xreal
287Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
Examples cos(0) equals 1.
exp
Purpose: calculate the exponential of a number.
Syntax exp(x)
Arguments
DescriptionNameType
The number whose exponential is to becomputed
xnumeric
Result
DescriptionType
, the exponential of xreal
Examples exp(0) equals 1.
floor
Purpose: round a number down to the nearest integer.
Syntax floor(x)
Arguments
DescriptionNameType
The number to be roundedxnumeric
Result
288Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
DescriptionType
, the integer such thatinteger
Examples
floor(1.25) equals 1.
floor(-1.25) equals -2.
To generate a random integer between 0 and 9 inclusive: floor(rndUniform()*10)
See also
ceil on page 286
round on page 296
log
Purpose: calculate the natural (base-e) logarithm of a number.
Syntax log(x)
Arguments
DescriptionNameType
A positive numberxnumeric
Result
DescriptionType
, the natural logarithm of xreal
Examples log(1) equals 0.
289Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
log10
Purpose: calculate the base-10 logarithm of a number.
Syntax log10(x)
Arguments
DescriptionNameType
A positive numberxnumeric
Result
DescriptionType
, the base-10 logarithm of xreal
Examples log10(100) equals 2.
logbase
Purpose: calculate the logarithm of a number, to a specified base.
Syntax logbase(x, base)
Arguments
DescriptionNameType
The positive number whose logarithmis to be computed
xnumeric
The base to be usedbasenumeric
Result
290Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
DescriptionType
, the base-base logarithm of xreal
Examples logbase(100,10) equals 2.
max (two or more arguments), maxnonnull
Purpose: calculate the maximum of two or more numbers, or the latest of two or more dates, orthe alphabetically latest of two or more strings. The max variant returns the null value if any of itsarguments is null; the maxnonnull variant ignores null arguments (but still returns the null value ifall of its arguments are null).
Syntax max( , , ...)
maxnonnull( , , ...)
Arguments
DescriptionNameType
The values to be compared, , ...integer, real, date, or string (all thesame type)
Result
DescriptionType
The maximum of , , ...as input
Note: • Be careful not to confuse this multi-argument max function with the aggregation function[see max (one argument) on page 167] of the same name.
• For strings, "alphabetically latest" means "latest in the underlying character representation."
Examples
max(5.5, -3, 7, -8.5) equals 7.
max(5.5, -3, null, -8.5) equals null.
maxnonnull(5.5, -3, 7, -8.5) equals 7.
291Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
maxnonnull(5.5, -3, null, -8.5) equals 5.5.
To determine the next best action for a customer based on different ROI measures:
maxvalue := max (ROI1, ROI2, ROI3);
case
maxvalue = ROI1: "Product1";
maxvalue = ROI2: "Product2";
maxvalue = ROI3: "Product3";
mean (two or more arguments), meannonnull
Purpose: calculate the mean (common average) of two or more numbers. The mean variant returnsthe null value if any of its arguments is null; the meannonnull variant ignores null arguments (butstill returns the null value if all of its arguments are null).
Syntax mean( , , ...)
meannonnull( , , ...)
Arguments
DescriptionNameType
The numbers to average, , ...numeric
Result
DescriptionType
The mean of , , ...real
Be careful not to confuse this multi-argument mean function with the aggregation function[see mean (one argument) on page 168] of the same name.
Note:
Examples
mean(5.5, -3, 7, -8.5) equals 0.25.
mean(5.5, -3, null, -8.5) equals null.
292Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
meannonnull(5.5, -3, 7, -8.5) equals 0.25.
meannonnull(5.5, -3, null, -8.5) equals -2.
To calculate the average yearly spend from four separate quarter spend fields:
mean(SpendQtr1, SpendQtr2, SpendQtr3, SpendQtr4)
min (two or more arguments), minnonnull
Purpose: calculate the minimum of two or more numbers, or the earliest of two or more dates, orthe alphabetically earliest of two or more strings. The min variant returns the null value if any of itsarguments is null; the minnonnull variant ignores null arguments (but still returns the null valueif all of its arguments are null).
Syntax
min( , , ...)
minnonnull( , , ...)
Arguments
DescriptionNameType
The values to be compared, , ...integer, real, date, or string (all thesame type)
Result
DescriptionType
The minimum of , , ...as input
Note: • Be careful not to confuse this multi-argument min function with the aggregation function[see min (one argument) on page 169] of the same name.
• For strings, "alphabetically earliest" means "earliest in the underlying characterrepresentation."
Examples
min(5.5, -3, 7, -8.5) equals -8.5.
min(5.5, -3, null, -8.5) equals null.
293Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
minnonnull(5.5, -3, 7, -8.5) equals -8.5.
minnonnull(5.5, -3, null, -8.5) equals -8.5.
normalize
Purpose: normalize field values to lie in the interval [0,1].
Syntax normalize(x)
Arguments
DescriptionNameType
The number to be normalizedxnumeric
Result
DescriptionType
Value between 0 and 1 given byreal
You can only use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
pow
Purpose: calculate the result of raising one number to the power of another number.
Syntax pow(x, exponent)
Arguments
294Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
DescriptionNameType
The number to be raised to a powerxnumeric
The exponentexponentnumeric
Result
DescriptionType
, x raised to the power exponentreal
Examples pow(10, 2) equals 100.
product, productnonnull
Purpose: calculate the product of two or more numbers. The product variant returns the null valueif any of its arguments is null; the productnonnull variant ignores null arguments (but still returnsthe null value if all of its arguments are null).
Syntax product( , , ...)
productnonnull( , , ...)>
Arguments
DescriptionNameType
The numbers to be multiplied, , ...numeric
Result
DescriptionType
The product of , , ...as input
Examples
product(5.5, -3, 7, -8.5) equals 981.75.
295Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
product(5.5, -3, null, -8.5) equals null.
productnonnull(5.5, -3, 7, -8.5) equals 981.75.
productnonnull(5.5, -3, null, -8.5) equals 140.25.
round
Purpose: round a number to the nearest integer.
Syntax round(x)
Arguments
DescriptionNameType
The number to be roundedxnumeric
Result
DescriptionType
the nearest integer to x, or, in the event of a tie, the greaterof the two nearest integers (in absolute value)
integer
Examples
round(1.25) equals 1.
round(-1.25) equals -1.
round(1.5) equals 2.
round(-1.5) equals -2.
See also
ceil on page 286
floor on page 288
296Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
sgn
Purpose: calculate the signum (sign) of a number.
Syntax sgn(x)
Arguments
DescriptionNameType
A signed numberxnumeric
Result
DescriptionType
A number representing the sign of x:as input
Examples
sgn(25) equals 1.
sgn(-25) equals -1.
See also
abs on page 286
Splitting aggregations: the by clause on page 150
sin
Purpose: calculate the sine of an angle.
Syntax sin(x)
Arguments
297Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
DescriptionNameType
An angle in radiansxnumeric
Result
DescriptionType
sin x, the sine of xreal
Examples sin(3.14159/2) equals 1.
sqrt
Purpose: calculate the square root of a number.
Syntax sqrt(x)
Arguments
DescriptionNameType
A non-negative numberxnumeric
Result
DescriptionType
, the square root of xreal
Examples
sqrt(25) equals 5.
sqrt(-25) equals null.
298Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
sum (two or more arguments), sumnonnull
Purpose: calculate the sum of two or more numbers. The sum variant returns the null value if anyof its arguments is null; the sumnonnull variant ignores null arguments (but still returns the nullvalue if all of its arguments are null).
Syntax sum( , , ...)
sumnonnull( , , ...)
Arguments
DescriptionNameType
The numbers to be added, , ...numeric
Result
DescriptionType
The sum of , , ...as input
Be careful not to confuse this multi-argument sum function with the aggregation function[see sum (one argument) on page 175] of the same name.
Note:
Examples
sum(5.5, -3, 7, -8.5) equals 1.
sum(5.5, -3, null, -8.5) equals null.
sumnonnull(5.5, -3, 7, -8.5) equals 1.
sumnonnull(5.5, -3, null, -8.5) equals -6.
To derive the total spend from six weekly spend fields:
sum(SpendWk1, SpendWk2, SpendWk3, SpendWk4, SpendWk5, SpendWk6)
299Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
tan
Purpose: calculate the tangent of an angle.
Syntax tan(x)
Arguments
DescriptionNameType
An angle in radiansxnumeric
Result
DescriptionType
tan x, the tangent of xreal
If x is of the form for some integer n, tan x is undefined. In that case, the FDL functionreturns null.
Note:
Examples tan(3.14159/4) equals 1.
300Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Mathematical and statistical functions
23 - Data-samplingfunctions
In this section
numericTestTrainSplit 302numericTestTrainValidateSplit 302sampleEqualSize 303sampleExactNumber 304sampleExactPercentage 305sampleStratified 306testTrainSplit 307testTrainValidateSplit 308
numericTestTrainSplit
Purpose: create a test/training segmentation for use in model validation.
Syntax numericTestTrainSplit(testFraction)
Arguments
DescriptionNameType
The fraction of the data to be used asthe test set
testFractionreal
Result
DescriptionType
Values 0 for the training set or 1 for the test setinteger
Examples
To derive a field that allocates the value 0 to approximately 70% of the records (chosen randomly)and the value 1 to the remainder: numericTestTrainSplit(0.3)
See also
numericTestTrainValidateSplit on page 302
sampleEqualSize on page 303
testTrainSplit on page 307
testTrainValidateSplit on page 308
numericTestTrainValidateSplit
Purpose: create a test/training/validation segmentation for use in model validation.
Syntax numericTestTrainValidateSplit(testFraction, validateFraction)
Arguments
302Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Data-sampling functions
DescriptionNameType
The fraction of the data to be used asthe test set
testFractionreal
The fraction of the data to be used asthe hold-out set
validateFractionreal
Result
DescriptionType
Values 0 for the training set, 1 for the test set or 2 for thevalidation set
integer
Examples
To derive a field that allocates the value 0 to approximately 50% of the records (chosen randomly),the value 1 to approximately 30% of the records (again chosen randomly), and the value 2 to theremainder: numericTestTrainValidateSplit(0.3, 0.2)
See also
numericTestTrainSplit on page 302
sampleEqualSize on page 303
testTrainSplit on page 307
testTrainValidateSplit on page 308
sampleEqualSize
Purpose: create a segmentation index that randomly assigns records to segments of approximatelyequal size.
Syntax sampleEqualSize(NumberOfSegments)
Arguments
DescriptionNameType
The number of segments to createNumberOfSegmentsinteger
303Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Data-sampling functions
Result
DescriptionType
A number between 1 and NumberOfSegments(inclusive)
integer
Examples
To derive a field enumerating five (approximately) equal-sized populations: sampleEqualSize(5)
See also
numericTestTrainSplit on page 302
numericTestTrainValidateSplit on page 302
testTrainSplit on page 307
testTrainValidateSplit on page 308
sampleExactNumber
Purpose: create a random sample of an exact size, specified as a number of records.
Syntax sampleExactNumber(NumberToSelect)
Arguments
DescriptionNameType
The size of sample to create, as anumber of records
NumberToSelectinteger
Result
DescriptionType
1 if a record is chosen for the sample; 0 otherwiseinteger
Note: • Records that are not currently selected in the focus are assigned the null value.
304Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Data-sampling functions
• You can only use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or ina selections file for qsselect.
Examples
To choose 1,250 records for a sample: sampleExactNumber(1250)
See also
sampleExactPercentage on page 305
sampleStratified on page 306
sampleExactPercentage
Purpose: create a random sample of an exact size, specified as a percentage of the size of thecurrently-selected population.
Syntax sampleExactPercentage(PercentageToSelect)
Arguments
DescriptionNameType
The size of sample to create, as apercentage of the size of the currentpopulation
PercentageToSelectinteger
Result
DescriptionType
1 if a record is chosen for the sample; 0 otherwiseinteger
Note: • Records that are not currently selected in the focus are assigned the null value.• If PercentageToSelect multiplied by the population size is not a whole number, the
number chosen will be the nearest whole number above or below PercentageToSelectmultiplied by the population size, and will on average give the specified percentage.
• You can only use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or ina selections file for qsselect.
305Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Data-sampling functions
Examples To choose 25% of the population for a sample:
sampleExactPercentage(25)
See also
sampleExactNumber on page 304
sampleStratified on page 306
sampleStratified
Purpose: create a random sample that contains an exact number of records from a specifiedsegment, together with an exact number of records from the remainder of the currently-selectedpopulation.
Syntax sampleStratified(NumberFromSegmentToSelect, SegmentTotalSize,NumberOutsideSegmentToSelect, OutsideSegmentTotalSize, InSegmentExpression)
Arguments
DescriptionNameType
The number of records in the sampleto be chosen from the specifiedsegment
NumberFromSegmentToSelectinteger
The total number of records in thespecified segment
SegmentTotalSizeinteger
The number of records in the sampleto be chosen from the remainder of thepopulation
NumberOutsideSegmentToSelectinteger
The total number of records in theremainder of the population
OutsideSegmentTotalSizeinteger
The (boolean) expression thatdetermines whether or not a record isin the specified segment
InSegmentExpressioninteger
Result
306Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Data-sampling functions
DescriptionType
1 if a record is chosen for the sample; 0 otherwiseinteger
Note: • Records that are not currently selected in the focus are assigned the null value.• You can only use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or ina selections file for qsselect.
Examples To create a sample comprising 2,500 records from a total of 25,000 females, and 5,000records from a total of 75,000 non-females:
sampleStratified(2500, 25000, 5000, 75000, Gender = "F")
See also
sampleExactNumber on page 304
sampleExactPercentage on page 305
testTrainSplit
Purpose: create a test/training segmentation for use in model validation.
Syntax testTrainSplit(testFraction)
Arguments
DescriptionNameType
The fraction of the data to be used asthe test set
testFractionreal
Result
DescriptionType
String values TestSet or TrainingSetstring
Examples
307Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Data-sampling functions
To derive a field that allocates the value "TrainingSet" to approximately 70% of the records(chosen randomly) and the value "TestSet" to the remainder:
testTrainSplit(0.3)
See also
numericTestTrainSplit on page 302
numericTestTrainValidateSplit on page 302
sampleEqualSize on page 303
testTrainValidateSplit on page 308
testTrainValidateSplit
Purpose: Create a test/training/validation segmentation for use in model validation.
Syntax testTrainValidateSplit(testFraction, validateFraction)
Arguments
DescriptionNameType
The fraction of the data to be used asthe test set
testFractionreal
The fraction of the data to be used asthe hold-out set
validateFractionreal
Result
DescriptionType
String values TestSet, TrainingSet orValidationSet
string
Examples
To derive a field that allocates the value "TrainingSet" to approximately 50% of the records(chosen randomly), the value "TestSet" to approximately 30% of the records (again chosenrandomly), and the value "ValidationSet" to the remainder:
308Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Data-sampling functions
testTrainValidateSplit(0.3, 0.2)
See also
numericTestTrainSplit on page 302
numericTestTrainValidateSplit on page 302
sampleEqualSize on page 303
testTrainSplit on page 307
309Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Data-sampling functions
24 - Random-numberfunctions
In this section
About random-number functions in FDL 311rndBinomial 311rndBool 312rndExp 312rndGamma 313rndNormal 314rndPoisson 315rndUniform 315
About random-number functions in FDL
When evaluating an FDL expression involving a random number function (numericTestTrainSplit,numericTestTrainValidateSplit, rndBinomial, rndBool, rndExp, rndGamma, rndNormal,rndPoisson, or rndUniform sampleEqualSize, sampleExactNumber, sampleExactPercentage,sampleStratified, testTrainSplit, and testTrainValidateSplit), Spectrum Miner needs an integer"seed" to initialize its sequence of random numbers. With the same seed, the same expression willalways produce the same sequence of results.
Note: • The seed is stored as part of the field information.• The (pseudo)random number generation is based on that of Marsaglia, Zaman, and Tsang
(1990).
rndBinomial
Purpose: generate a random integer based on a binomial distribution.
Syntax rndBinomial(N, p)
Arguments
DescriptionNameType
The (positive) number of trialsNinteger
The probability of success for each trial(between 0 and 1)
pnumeric
Result
DescriptionType
A random non-negative integer sampled from the binomialdistribution, with probability given by
integer
311Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Random-number functions
The expectation and variance of the binomial distribution are and respectively.Note:
Examples rndBinomial (50, 0.05)
See also
About random-number functions in FDL on page 311
rndBool
Purpose: generate either 0 or 1 randomly (with equal probability).
Syntax rndBool()
Arguments None
Result
DescriptionType
Either 0 or 1, with probabilitiesinteger
Examples
To derive a field that allocates the value 0 to approximately 50% of the records (chosen randomly)and the value 1 to the remainder: rndBool()
See also
About random-number functions in FDL on page 311
rndExp
Purpose: generate a random positive number based on an exponential distribution.
Syntax rndExp(b)
Arguments
312Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Random-number functions
DescriptionNameType
The (positive) scale parameterbnumeric
Result
DescriptionType
A positive random number sampled from the exponentialdistribution, with probability density function given by
(x > 0)
real
The expectation and variance of the exponential distribution are and respectively.Note:
Examples rndExp(10)
See also
About random-number functions in FDL on page 311
rndGamma
Purpose: generate a random positive number based on a gamma distribution.
Syntax rndGamma(a, b)
Arguments
DescriptionNameType
The (positive) number of independentevents
ainteger
The (positive) event ratebnumeric
Result
313Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Random-number functions
DescriptionType
A positive random number sampled from the gammadistribution , with probability density function given by
real
The expectation and variance of the gamma distribution are and respectively.Note:
Examples rndGamma (50, 0.05)
See also
About random-number functions in FDL on page 311
rndNormal
Purpose: generate a random number based on a normal distribution.
Syntax rndNormal( , )
Arguments
DescriptionNameType
The mean of the distributionnumeric
The standard deviation of thedistribution
numeric
Result
DescriptionType
A random number sampled from the normal distribution, with probability density function given by
real
The expectation and variance of the normal distribution are and respectively.Note:
314Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Random-number functions
Examples
To derive a Gaussian distribution with zero mean and unit standard deviation: rndNormal(0,1)
See also
About random-number functions in FDL on page 311
rndPoisson
Purpose: generate a random non-negative integer based on a discrete Poisson distribution.
Syntax rndPoisson(mean)
Arguments
DescriptionNameType
The (positive) mean of the distributionnumeric
Result
DescriptionType
A random non-negative integer sampled from the discretePoisson distribution, with probability given by( )
integer
The expectation and variance of the discrete Poisson distribution are both equal to the mean,.
Note:
Examples rndPoisson(10)
See also
About random-number functions in FDL on page 311
rndUniform
Purpose: generate a random number between 0 and 1 based on a uniform distribution.
315Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Random-number functions
Syntax rndUniform()
Arguments None
Result
DescriptionType
A random number sampled from a uniform distribution over[0, 1)
real
Examples
To generate a random real number in the range [0,10): rndUniform() * 10
See also
About random-number functions in FDL on page 311
316Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Random-number functions
25 -Return-on-investmentfunctions
In this section
ActionROI 318ActionROIAnnualized 319OfferROI 320OfferROIAnnualized 322RetentionActionROI 323RetentionActionROIAnnualized 324RetentionOfferROI 326RetentionOfferROIAnnualized 327
ActionROI
Purpose: estimate the per-customer (money) return on investment contribution from taking actiondesigned to generate a definite response. The action happens whether or not the customer responds,so there is only a cost of action, not of fulfillment.
Syntax ActionROI(ResponseProbability, ValueOfResponse, CostOfAction)
Arguments
DescriptionNameType
The estimated probability that thecustomer will respond to the action byexhibiting some response behavior
ResponseProbabilitynumeric
The estimated value of the customer'sresponse to the marketing action (if itoccurs)
ValueOfResponsenumeric
The estimated cost of the action(whether or not the customer responds)
CostOfActionnumeric
Result
DescriptionType
The estimated per-customer (money) return on investmentcontribution from taking action
numeric
Note: • The response probability should be an "uplift," that is, the probability that the customer willexhibit that behavior when they would not otherwise have done so. If there is any possibilityof the customer's exhibiting the desired behavior without the action being taken, you shouldmodel this by using an uplift model or, failing that, by subtracting the likelihood of responsegiven the action from the likelihood without the action.
• The value of response should be a net revenue that ignores the cost of the marketingaction.
See also
ActionROIAnnualized on page 319
318Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
OfferROI on page 320
OfferROIAnnualized on page 322
RetentionActionROI on page 323
RetentionActionROIAnnualized on page 324
RetentionOfferROI on page 326
RetentionOfferROIAnnualized on page 327
ActionROIAnnualized
Purpose: estimate the annualized per-customer return on investment multiple from taking actiondesigned to generate a definite response from a customer. The action happens whether or not thecustomer responds, so there is only a cost of action, not of fulfillment.
Syntax ActionROIAnnualized(ResponseProbability, ValueOfResponse,CostOfAction, DaysToROI)
Arguments
DescriptionNameType
The estimated probability that thecustomer will respond to the action byexhibiting some response behavior
ResponseProbabilitynumeric
The estimated value of the customer'sresponse to the marketing action (if itoccurs)
ValueOfResponsenumeric
The estimated cost of the action(whether or not the customer responds)
CostOfActionnumeric
The number of days expected to berequired to achieve the given return oninvestment
DaysToROInumeric
Result
319Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
DescriptionType
The estimated annualized per-customer return on investmentmultiple from taking action
numeric
Note: • The response probability should be an "uplift," that is, the probability that the customer willexhibit that behavior when they would not otherwise have done so. If there is any possibilityof the customer's exhibiting the desired behavior without the action being taken, you shouldmodel this by using an uplift model or, failing that, by subtracting the likelihood of responsegiven the action from the likelihood without the action.
• The value of response should be a net revenue that ignores the cost of the marketingaction.
See also
ActionROI on page 318
OfferROI on page 320
OfferROIAnnualized on page 322
RetentionActionROI on page 323
RetentionActionROIAnnualized on page 324
RetentionOfferROI on page 326
RetentionOfferROIAnnualized on page 327
OfferROI
Purpose: estimate the per-customer return on investment contribution from making an offer designedto generate a definite response.
Syntax OfferROI(ResponseProbability, ValueOfResponse, CostOfOffer,CostOfFulfilment)
Arguments
DescriptionNameType
The estimated probability that thecustomer will respond to the action byexhibiting some response behavior
ResponseProbabilitynumeric
320Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
DescriptionNameType
The estimated value of the customer'sresponse to the marketing action (if itoccurs)
ValueOfResponsenumeric
The estimated cost of the action(whether or not the customer responds)
CostOfOffernumeric
The estimated cost of a fulfilling apositive response (zero if there is no(additional) cost of fulfillment)
CostOfFulfilmentnumeric
Result
DescriptionType
The estimated per-customer return on investment frommaking an offer
numeric
Note: • The response probability should be an "uplift," that is, the probability that the customer willexhibit that behavior when they would not otherwise have done so. If there is any possibilityof the customer's exhibiting the desired behavior without the action being taken, you shouldmodel this by using an uplift model or, failing that, by subtracting the likelihood of responsegiven the action from the likelihood without the action.
• The value of response should be a net revenue that ignores the cost of the marketingaction and the cost of fulfillment to the customer's response.
See also
ActionROI on page 318
ActionROIAnnualized on page 319
OfferROIAnnualized on page 322
RetentionActionROI on page 323
RetentionActionROIAnnualized on page 324
RetentionOfferROI on page 326
RetentionOfferROIAnnualized on page 327
321Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
OfferROIAnnualized
Purpose: estimate the annualized per-customer return on investment multiple from making an offerdesigned to generate a definite response from a customer.
SyntaxOfferROIAnnualized(ResponseProbability, ValueOfResponse, CostOfOffer,CostOfFulfilment, DaysToROI)
Arguments
DescriptionNameType
The estimated probability that thecustomer will respond to the action byexhibiting some response behavior
ResponseProbabilitynumeric
The estimated value of the customer'sresponse to the marketing action (if itoccurs)
ValueOfResponsenumeric
The estimated cost of the action(whether or not the customer responds)
CostOfOffernumeric
The estimated cost of a fulfilling apositive response (zero if there is no(additional) cost of fulfillment)
CostOfFulfilmentnumeric
The number of days expected to berequired to achieve the given return oninvestment
DaysToROInumeric
Result
DescriptionType
The estimated per-customer annualized return on investmentmultiple from making an offer
numeric
Note: • The response probability should be an "uplift," that is, the probability that the customer willexhibit that behavior when they would not otherwise have done so. If there is any possibilityof the customer's exhibiting the desired behavior without the action being taken, you should
322Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
model this by using an uplift model or, failing that, by subtracting the likelihood of responsegiven the action from the likelihood without the action.
• The value of response should be a net revenue that ignores the cost of the marketingaction and the cost of fulfillment to the customer's response.
See also
ActionROI on page 318
ActionROIAnnualized on page 319
OfferROI on page 320
RetentionActionROI on page 323
RetentionActionROIAnnualized on page 324
RetentionOfferROI on page 326
RetentionOfferROIAnnualized on page 327
RetentionActionROI
Purpose: estimate the per-customer (money) return on investment contribution from taking actiondesigned to prevent attrition. The action is assumed to have a fixed cost, regardless of whether thecustomer is saved or not, and whether or not a customer "accepts" the offer.
Syntax RetentionActionROI(SaveProbability, NetRevenueAtRisk, CostOfAction)
Arguments
DescriptionNameType
The estimated probability that thecustomer will be "saved" (for the period
SaveProbabilitynumeric
under consideration) as a result of theaction taken
The estimated net revenue that will belost if the customer undergoes attritionover the period under consideration
NetRevenueAtRisknumeric
The estimated cost of the action(whether or not the customer responds)
CostOfActionnumeric
Result
323Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
DescriptionType
The estimated per-customer (money) return on investmentcontribution from action designed to prevent attrition
numeric
Note: • To say a customer is "saved" means "caused not to leave when the customer otherwisewould have done so." This value is the negative of the uplift in attrition rate produced byan uplift model. Alternatively, it can be calculated as P(attrition|no action) - P(attrition|action).
• Net revenue at risk should be a net revenue that ignores the cost of the action designedto prevent attrition.
See also
ActionROI on page 318
ActionROIAnnualized on page 319
OfferROI on page 320
OfferROIAnnualized on page 322
RetentionActionROIAnnualized on page 324
RetentionOfferROI on page 326
RetentionOfferROIAnnualized on page 327
RetentionActionROIAnnualized
Purpose: estimate the annualized per-customer return on investment contribution from taking actiondesigned to prevent attrition. The action is assumed to have a fixed cost, regardless of whether thecustomer is saved or not, and whether or not a customer "accepts" the offer.
Syntax RetentionActionROIAnnualized(SaveProbability, NetRevenueAtRisk,CostOfAction, DaysToROI)
Arguments
DescriptionNameType
The estimated probability that thecustomer will be "saved" (for the period
SaveProbabilitynumeric
under consideration) as a result of theaction taken
324Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
DescriptionNameType
The estimated net revenue that will belost if the customer undergoes attritionover the period under consideration
NetRevenueAtRisknumeric
The estimated cost of the action(whether or not the customer responds)
CostOfActionnumeric
The number of days expected to berequired to achieve the given return oninvestment
DaysToROInumeric
Result
DescriptionType
The estimated annualized per-customer return on investmentcontribution from action designed to prevent attrition
numeric
Note: • To say a customer is "saved" means "caused not to leave when the customer otherwisewould have done so." This value is the negative of the uplift in attrition rate produced byan uplift model. Alternatively, it can be calculated as P(attrition|no action) - P(attrition|action).
• Net revenue at risk should be a net revenue that ignores the cost of the action designedto prevent attrition.
See also
ActionROI on page 318
ActionROIAnnualized on page 319
OfferROI on page 320
OfferROIAnnualized on page 322
RetentionActionROI on page 323
RetentionOfferROI on page 326
RetentionOfferROIAnnualized on page 327
325Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
RetentionOfferROI
Purpose: estimate the per-customer return on investment from making an offer designed to preventattrition. An offer, as distinct from an action, has a different cost according to whether or not it isaccepted by the customer.
Syntax RetentionOfferROI(SaveProbability, ResponseProbability,NetRevenueAtRisk, CostOfOffer, CostOfFulfilment)
Arguments
DescriptionNameType
The estimated probability that thecustomer will be "saved" (for the period
SaveProbabilitynumeric
under consideration) as a result of theaction taken
The estimated probability that thecustomer will respond to the offer
ResponseProbabilitynumeric
The estimated net revenue that will belost if the customer undergoes attritionover the period under consideration
NetRevenueAtRisknumeric
The estimated cost of the offer (whetheror not the customer responds)
CostOfOffernumeric
The estimated cost of fulfilling a positiveresponse to the offer
CostOfFulfilmentnumeric
Result
DescriptionType
The estimated per-customer return on investment from anoffer designed to prevent attrition
numeric
Note: • To say a customer is "saved" means "caused not to leave when the customer otherwisewould have done so". This value is minus the uplift in attrition rate produced by an upliftmodel. Alternatively, it can be calculated as P(attrition|no action) - P(attrition|action).
326Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
• Net revenue at risk should be a net revenue ignoring the cost of the action designed toprevent attrition.
See also
ActionROI on page 318
ActionROIAnnualized on page 319
OfferROI on page 320
OfferROIAnnualized on page 322
RetentionActionROI on page 323
RetentionActionROIAnnualized on page 324
RetentionOfferROIAnnualized on page 327
RetentionOfferROIAnnualized
Purpose: estimate the annualized per-customer return on investment from making an offer designedto prevent attrition. An offer, as distinct from an action, has a different cost according to whether ornot it is accepted by the customer.
Syntax RetentionOfferROIAnnualized(SaveProbability, ResponseProbability,NetRevenueAtRisk, CostOfOffer, CostOfFulfilment, DaysToROI)
Arguments
DescriptionNameType
The estimated probability that thecustomer will be "saved" (for the period
SaveProbabilitynumeric
under consideration) as a result of theaction taken
The estimated probability that thecustomer will respond to the offer
ResponseProbabilitynumeric
The estimated net revenue that will belost if the customer undergoes attritionover the period under consideration
NetRevenueAtRisknumeric
The estimated cost of the offer (whetheror not the customer responds)
CostOfOffernumeric
327Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
DescriptionNameType
The estimated cost of fulfilling a positiveresponse to the offer
CostOfFulfilmentnumeric
The number of days expected to berequired to achieve the given return oninvestment
DaysToROInumeric
Result
DescriptionType
The estimated annualized per-customer return on investmentfrom an offer designed to prevent attrition
numeric
Note: • To say a customer is "saved" means "caused not to leave when the customer otherwisewould have done so". This value is minus the uplift in attrition rate produced by an upliftmodel. Alternatively, it can be calculated as P(attrition|no action) - P(attrition|action).
• Net revenue at risk should be a net revenue ignoring the cost of the action designed toprevent attrition.
See also
ActionROI on page 318
ActionROIAnnualized on page 319
OfferROI on page 320
OfferROIAnnualized on page 322
RetentionActionROI on page 323
RetentionActionROIAnnualized on page 324
RetentionOfferROI on page 326
328Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Return-on-investment functions
26 - Miscellaneousfunctions
In this section
dblookup 330member 332rankOrder, rankOrderApprox 333rankOrderMean, rankOrderApproxMean 335rownum 336
dblookup
Purpose: look up values in a reference table (stored as a focus).
Syntax dblookup(d, k, f, x)
Arguments
DescriptionNameType
The pathname of the lookup tabledstring
The name of the key field in the lookuptable
kstring
The name of another field in the lookuptable
fstring
The lookup valuexas field k
Result
DescriptionType
The value y in the field f of table d, for the record indexedby the value x occurring in the key field k
as field f
330Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Miscellaneous functions
Note: • You should only use this function for small lookup tables. If the lookup table is comparablein size to a customer or transaction table, you should consider performing a join instead.
• If the lookup value doesn't occur in the key field, the null value is returned.• If the lookup value occurs more than once in the key field, the first value is used.• Trailing whitespace is stripped from the lookup value, but not from the values of the key
field in the lookup table. This may produce unexpected behavior if the key field in the lookuptable contains values with trailing whitespace.
• For backwards compatibility, you can prefix the pathname of the lookup table with the stringfocus:.
Example
In a focus, the store at which a customer most frequently shops is recorded in the field PopularShop.Another focus stores.ftr comprises a lookup table that describes facilities at each store: in thisfocus, StoreCode is the key field, and the field HasFuel indicates whether each store has a gasstation. Store 105 has a gas station, stores 299 and 204 do not, and so on; store 189 is not in thelookup table. To derive a field that flags whether each customer's favorite store has a gas station:
dblookup("focus:stores.ftr","StoreCode", "HasFuel", PopularShop)
PopularShop
1105
0299
0204
331Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Miscellaneous functions
dblookup("focus:stores.ftr","StoreCode", "HasFuel", PopularShop)
PopularShop
null189
member
Purpose: determine set membership.
Syntax member(x, , , ...)
Arguments
DescriptionNameType
The value to be testedxinteger, real, date, or string
The elements of the set, , ...as x
Result
DescriptionType
1 (true) ifx is in the set ; 0 (false) otherwiseinteger
Note: • You can only use literal values (numbers, strings, or dates) for the list of set elements.• It is in general more efficient to use the member function than a disjunction of equality
tests. For example, the expression strmember(F, "a", "b", "c") is quicker toevaluate than F = "a" or F = "b" or F = "c".
• For backwards compatibility, you can use strmember instead of member when thearguments are strings.
• Trailing whitespace in string values is not ignored.
Examples
To use the first two characters of a StateCode field to identify West-coast customers:
332Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Miscellaneous functions
member (substr (StateCode, 0, 1),"CA", "OR", WA")
StateCode
1OR009
1CA043
0UT005
1WA027
To flag customers whose 10-digit numeric phone numbers have Edinburgh (131) or Glasgow (141)area codes (by determining the first three digits using the div [see Arithmetic operators on page193] operator):
if member (TelNo div 10000000, 131,141) then "Edin/Glas" else null
TelNo
Edin/Glas1312204491
null1753833999
null1611234567
Edin/Glas1417654321
See also
Splitting aggregations: the by clause on page 150
rankOrder, rankOrderApprox
Purpose: identify the rank of a value within a list of values.
Syntax
rankOrder(x, , , ...)
rankOrderApprox(x, , , ...)
Arguments
333Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Miscellaneous functions
DescriptionNameType
The value to be rankedxinteger, real, date, or string
The list of values for comparison, , ...as x
Result
DescriptionType
The position of the first occurrence of x within the list ofvalues obtained by sorting , , ... in descending order (latestto earliest for dates; reverse alphabetical order for strings)
If x doesn't occur in the list, rankOrder returns null, whilerankOrderApprox returns the position of x within the list ofvalues obtained by sorting x, , , ... in descending order.
integer
Note: • When the value to be ranked occurs exactly once in the list of values, rankOrder,rankOrderApprox, rankOrderMean, and rankOrderApproxMean all give the same result.
• When the value to be ranked occurs more than once in the list of values, rankOrderApproxgives the same result as rankOrder.
• When the value to be ranked doesn't occur in the list of values, rankOrderApprox givesthe same result as rankOrderApproxMean.
• For strings, "alphabetical order" means the natural order of the underlying characterrepresentation.
Examples
rankOrder(6.74, 6.74, 1.99, 0.00, 9.27) equals 2.
rankOrderApprox(6.74, 1.99, 0.00, 9.27) equals 2.
To check where a customer's number of phone calls in Quarter 4 ranks in a year:
rankOrder(CallsQ4, CallsQ1, CallsQ2, CallsQ3, CallsQ4)
To check where a customer's number of phone calls in month 5 ranks with the previous four:
RankOrderApprox(CallsM5, CallsM1, CallsM2, CallsM3, CallsM4)
See also
rankOrderMean, rankOrderApproxMean on page 335
334Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Miscellaneous functions
rankOrderMean, rankOrderApproxMean
Purpose: identify the rank of a value within a list of values, or its average rank if it occurs more thanonce.
Syntax rankOrderMean(x, , , ...)
rankOrderApproxMean(x, , , ...)
Arguments
DescriptionNameType
The value to be rankedxinteger, real, date, or string
The list of values for comparison, , ...as x
Result
DescriptionType
The mean rank of x within the list of values obtained bysorting , , ... in descending order (latest to earliest fordates; reverse alphabetical order for strings)
If x doesn't occur in the list, rankOrderMean returnsnull, while rankOrderApproxMean returns the
real
position of x within the list of values obtained by sorting x,, , ... in descending order.
Note: • When the value to be ranked occurs exactly once in the list of values, rankOrder,rankOrderApprox, rankOrderMean, and rankOrderApproxMean all give the sameresult.
• When the value to be ranked occurs more than once in the list of values,rankOrderApproxMean gives the same result as rankOrderMean.
• When the value to be ranked doesn't occur in the list of values, rankOrderApproxMeangives the same result as rankOrderApprox.
• For strings, "alphabetical order" means the natural order of the underlying characterrepresentation.
Examples
335Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Miscellaneous functions
rankOrderMean(6.74, 6.74, 1.99, 6.74, 9.27) equals 2.5.
rankOrderApproxMean(6.74, 1.99, 6.74, 9.27, 6.74) equals 2.5.
See also
rankOrder, rankOrderApprox on page 333
rownum
Purpose: generate the number of each row in the table.
Syntax rownum()
Arguments None
Result
DescriptionType
An index number from 1 to the number of records in thetable
integer
You can only use this function in a Decision Studio derivation, in the derivations file forqsderive or the equivalent Derive Fields window, in the trackers file for qstrack, or in aselections file for qsselect.
Note:
Examples To add an index column to an N-record dataset, with values :
rownum()ID
1AZ195
2DI539
3JD974
4BY643
336Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Miscellaneous functions
27 - Binnings
In this section
bin 338Boolean 339DayFrom, WeekFrom, MonthFrom, YearFrom 340DayMultipleFrom, WeekMultipleFrom, MonthMultipleFrom,
YearMultipleFrom 341DayMultipleNumBins, WeekMultipleNumBins, MonthMultipleNumBins,
YearMultipleNumBins 342DayMultiplePrePost, WeekMultiplePrePost, MonthMultiplePrePost,
YearMultiplePrePost 343DayMultipleTo, WeekMultipleTo, MonthMultipleTo, YearMultipleTo 344DayMultipleWidth, WeekMultipleWidth, MonthMultipleWidth,
YearMultipleWidth 345DayPrePost, WeekPrePost, MonthPrePost, YearPrePost 346DayRange, WeekRange, MonthRange, YearRange 347DayTo, WeekTo, MonthTo, YearTo 348EqualRange 349EqualRangeWidth 350NegativeNonNegative 351PreDuringPost 352PrePost 353Sign 354
bin
Purpose: obtain a bin index corresponding to a value.
Syntax bin(binning, x)
Arguments
DescriptionNameType
A named binning, with parametervalues (which must be literal values) inparentheses if required
binningbinning
A value compatible with the binningxinteger, real, date, or string
Result
DescriptionType
The bin index of the bin in which x liesinteger
Note: • Bin indices are positive integers. For a numeric binning with n nominal bins (numbered 2to n + 1), the lower and upper end bins are numbered 1 and n + 2 respectively, while thenull and unclassified bins are numbered n + 3 and n + 4.
• This function is not available in Decision Studio. However, if x is a field and b is the primarybinning associated with x, you can achieve the same result using segindex:
segindex() by x
Examples
Split aggregations using the equal-range binning EqualRange on the Amount field, resulting in aneight-way segmented aggregation (four internal bins between amount values 20 and 100 plus twoend bins, a null bin and an unclassified bin):
create totalPointsByAmountBands :=sum(PointsRedeemed) by bin(EqualRange(20, 100, 4), Amount);
338Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
Split aggregations into eight segments based on the campaign binning PrePost, starting on theMay 1999, and with aggregates from the month before and after the campaign:
create totalSpendPrePostMayCampaign_ := sum(Amount) bybin(PrePost(todate(19990401), todate(19990501),todate(19990601)), PurchaseDate);
See also
Boolean on page 339
DayFrom, WeekFrom, MonthFrom, YearFrom on page 340
DayMultipleFrom, WeekMultipleFrom, MonthMultipleFrom, YearMultipleFrom on page 341
DayMultipleNumBins,WeekMultipleNumBins, MonthMultipleNumBins, YearMultipleNumBinson page 342
DayMultiplePrePost, WeekMultiplePrePost, MonthMultiplePrePost, YearMultiplePrePost onpage 343
DayMultipleTo, WeekMultipleTo, MonthMultipleTo, YearMultipleTo on page 344
DayMultipleWidth, WeekMultipleWidth, MonthMultipleWidth, YearMultipleWidth on page 345
DayPrePost, WeekPrePost, MonthPrePost, YearPrePost on page 346
DayRange, WeekRange, MonthRange, YearRange on page 347
DayTo, WeekTo, MonthTo, YearTo on page 348
EqualRange on page 349
EqualRangeWidth on page 350
NegativeNonNegative on page 351
PreDuringPost on page 352
PrePost on page 353
Sign on page 354
Splitting aggregations: the by clause on page 150
Boolean
Purpose: bin boolean [see Boolean data on page 182] values.
Parameters None
Result
339Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
ValuesBin index
false1
true2
null [see The null value on page 181]3
unclassified4
DayFrom, WeekFrom, MonthFrom, YearFrom
Purpose: bin date values into a specified number of periods (days, weeks, months, or years)following a given date.
Parameters
DescriptionNameType
The start of the periodic portion of thebinning
a ("start_date")date
The number of periods (days, weeks,months, or years) in the periodic portionof the binning
n ("num_bins")integer
Result
ValuesBin index
dates earlier than a; the lower end bin1
the n consecutive periods following a2, ..., n + 1
dates n periods after a and later; the upper end binn + 2
null [see The null value on page 181]n + 3
340Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
ValuesBin index
unclassifiedn + 4
See also
DayRange, WeekRange, MonthRange, YearRange on page 347
DayTo, WeekTo, MonthTo, YearTo on page 348
DayMultipleFrom,WeekMultipleFrom,MonthMultipleFrom,YearMultipleFrom
Purpose: bin date values into a specified number of periods following a given date, where a periodis a specified number of days, weeks, months, or years.
Parameters
DescriptionNameType
The start of the periodic portion of thebinning
a ("start_date")date
The number of units (days, weeks,months, or years) in a period
m ("width")integer
The number of bins in the periodicportion of the binning
n ("num_bins")integer
Result
ValuesBin index
dates earlier than a; the lower end bin1
the n consecutive periods (each of m units) following a2, ..., n + 1
dates n x m units after a and later; the upper end binn + 2
341Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
ValuesBin index
null [see The null value on page 181]n + 3
unclassifiedn + 4
See also
DayMultiplePrePost, WeekMultiplePrePost, MonthMultiplePrePost, YearMultiplePrePost onpage 343
DayMultipleTo, WeekMultipleTo, MonthMultipleTo, YearMultipleTo on page 344
DayMultipleNumBins, WeekMultipleNumBins,MonthMultipleNumBins, YearMultipleNumBins
Purpose: bin date values into approximately the specified number of periods between two givendates, where the period is a fixed multiple of the unit (day, week, month, or year).
Parameters
DescriptionNameType
The start of the periodic portion of thebinning
a ("start_date")date
The end of the periodic portion of thebinning
b ("end_date")date
The target number of periods (multiplesof a day, week, month, or year) in theperiodic portion of the binning
n ("num_bins")integer
Result
ValuesBin index
dates earlier than a; the lower end bin1
342Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
ValuesBin index
periods between a and b2, ..., N - 3
dates b and later; the upper end binN - 2
null [see The null value on page 181]N - 1
unclassifiedN
Note: • These binnings are like the DayRange, WeekRange, MonthRange, and YearRange [seeDayRange, WeekRange, MonthRange, YearRange on page 347] binnings, except thatthe period may be two or more units (days, weeks, months, or years), to yield approximatelythe target number of bins.
• When the period is two or more units, bin N - 3 can be shorter than bins 2 to N - 4.
See also
DayMultipleWidth, WeekMultipleWidth, MonthMultipleWidth, YearMultipleWidth on page 345
DayMultiplePrePost, WeekMultiplePrePost,MonthMultiplePrePost, YearMultiplePrePost
Purpose: bin date values into specified numbers of bins before and after a given reference date,where each bin is a given multiple of the unit (day, week, month, or year).
Parameters
DescriptionNameType
The reference date for the binninga ("ref_date")date
The number of bins before and after thereference date
m ("width")integer
The number of units (days, weeks,months, or years) in each bin
n ("num_bins")integer
Result
343Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
ValuesBin index
dates earlier than n x m units before a; the lower end bin1
the n consecutive periods (each of m units) preceding a2, ..., n + 1
the n consecutive periods (each of m units) following an + 2, ..., 2n + 1
dates n x m units after a and later; the upper end bin2n + 2
null [see The null value on page 181]2n + 3
unclassified2n + 4
See also
DayMultipleFrom, WeekMultipleFrom, MonthMultipleFrom, YearMultipleFrom on page 341
DayMultipleTo, WeekMultipleTo, MonthMultipleTo, YearMultipleTo on page 344
DayPrePost, WeekPrePost, MonthPrePost, YearPrePost on page 346
PreDuringPost on page 352
PrePost on page 353
DayMultipleTo, WeekMultipleTo, MonthMultipleTo,YearMultipleTo
Purpose: bin date values into a specified number of periods preceding a given date, where a periodis a specified number of days, weeks, months, or years.
Parameters
DescriptionNameType
The end of the periodic portion of thebinning
b ("end_date")date
The number of units (days, weeks,months, or years) in a period
m ("width")integer
344Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
DescriptionNameType
The number of bins in the periodicportion of the binning
n ("num_bins")integer
Result
ValuesBin index
dates earlier than n x m units before a; the lower end bin1
the n consecutive periods (each of m units) preceding a2, ..., n + 1
dates a and later; the upper end binn + 2
null [see The null value on page 181]n + 3
unclassifiedn + 4
See also
DayMultipleFrom, WeekMultipleFrom, MonthMultipleFrom, YearMultipleFrom on page 341
DayMultiplePrePost, WeekMultiplePrePost, MonthMultiplePrePost, YearMultiplePrePost onpage 343
DayMultipleWidth, WeekMultipleWidth,MonthMultipleWidth, YearMultipleWidth
Purpose: bin date values into periods between two given dates, where the period is a specifiedmultiple of the unit (day, week, month, or year).
Parameters
DescriptionNameType
The start of the periodic portion of thebinning
a ("start_date")date
345Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
DescriptionNameType
The end of the periodic portion of thebinning
b ("end_date")date
The number of units (days, weeks,months, or years) in a period
m ("width")integer
Result
ValuesBin index
dates earlier than a; the lower end bin1
periods between a and b2, ..., N - 3
dates b and later; the upper end binN - 2
null [see The null value on page 181]N - 1
unclassifiedN
Note: • These binnings are like the DayRange, WeekRange, MonthRange, and YearRange [seeDayRange, WeekRange, MonthRange, YearRange on page 347] binnings, except thatthe period may be two or more units (days, weeks, months, or years).
• When the period is two or more units, bin N - 3 can be shorter than bins 2 to N - 4.
See also
DayMultipleNumBins,WeekMultipleNumBins, MonthMultipleNumBins, YearMultipleNumBinson page 342
DayPrePost, WeekPrePost, MonthPrePost, YearPrePost
Purpose: bin date values into specified numbers of periods (days, weeks, months, or years)preceding and following a given reference date.
Parameters
346Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
DescriptionNameType
The reference date for the binninga ("ref_date")date
The number of periods (days, weeks,months, or years) to bin before thereference date
m ("before")integer
The number of periods to bin after thereference date
n ("after")integer
Result
ValuesBin index
dates earlier than m periods before a; the lower end bin1
the mconsecutive periods preceding a2, ..., m + 1
the n consecutive periods following am + 2, ..., m + n + 1
dates n periods after a and later; the upper end binm + n + 2
null [see The null value on page 181]m + n + 3
unclassifiedm + n + 4
See also
DayMultiplePrePost, WeekMultiplePrePost, MonthMultiplePrePost, YearMultiplePrePost onpage 343
PreDuringPost on page 352
PrePost on page 353
DayRange, WeekRange, MonthRange, YearRange
Purpose: bin date values into periods (days, weeks, months, or years) between two given dates.
Parameters
347Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
DescriptionNameType
The start of the periodic portion of thebinning
a ("start_date")date
The end of the periodic portion of thebinning
b ("end_date")date
Result
ValuesBin index
dates earlier than a; the lower end bin1
periods (days, weeks, months, or years) between a and b2, ..., N - 3
dates b and later; the upper end binN - 2
null [see The null value on page 181]N - 1
unclassifiedN
Note: • If there is less than a complete period between a and b, bin 2 contains dates between a(inclusive) and b (exclusive).
• If there is not an exact number of periods between a and b (but there is at least onecomplete period), the bin immediately preceding b is lengthened to accommodate theexcess.
See also
DayFrom, WeekFrom, MonthFrom, YearFrom on page 340
DayTo, WeekTo, MonthTo, YearTo on page 348
DayTo, WeekTo, MonthTo, YearTo
Purpose: bin date values into a specified number of periods (days, weeks, months, or years)preceding a given date.
Parameters
348Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
DescriptionNameType
The end of the periodic portion of thebinning
b ("end_date")date
The number of periods (days, weeks,months, or years) in the periodic portionof the binning
n ("num_bins")integer
Result
ValuesBin index
dates earlier than n periods before b; the lower end bin1
the n consecutive periods preceding b2, ..., n + 1
dates b and later; the upper end binn + 2
null [see The null value on page 181]n + 3
unclassifiedn + 4
See also
DayFrom, WeekFrom, MonthFrom, YearFrom on page 340
DayRange, WeekRange, MonthRange, YearRange on page 347
EqualRange
Purpose: bin numeric values into a specified number of equal-width intervals.
Parameters
DescriptionNameType
The lower end of the equal-rangeportion of the binning
a ("from")real
349Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
DescriptionNameType
The upper end of the equal-rangeportion of the binning
b ("to")real
The number of bins in the equal-rangeportion of the binning
n ("num_bins")integer
Result
ValuesBin index
; the lower end bin1
[a, a + (b - a)/n), ..., [b - (b - a)/n, b)2, ..., n + 1
; the upper end binn + 2
null [see The null value on page 181]n + 3
unclassifiedn + 4
See also
EqualRangeWidth on page 350
EqualRangeWidth
Purpose: bin numeric values into intervals of a specified width.
Parameters
DescriptionNameType
The lower end of the equal-rangeportion of the binning
a ("from")real
The upper end of the equal-rangeportion of the binning
b ("to")real
350Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
DescriptionNameType
The width of bins in the equal-rangeportion of the binning
r ("width")real
Result
ValuesBin index
; the lower end bin1
equal-width bins of approximate width r2, ..., N - 3
the upper end binN - 2
null [see The null value on page 181]N - 1
unclassifiedN
If a - b isn't an exact multiple of r, an approximation to r is used, so that the bins between aand b are of equal width.
Note:
See also
EqualRange on page 349
NegativeNonNegative
Purpose: bin numeric values according to sign, grouping 0 with positive values.
Parameters None
Result
ValuesBin index
; that is, negative values1
; that is, non-negative values2
351Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
ValuesBin index
null [see The null value on page 181]3
unclassified4
See also
Sign on page 354
PreDuringPost
Purpose: bin date values into pre-campaign, during-campaign, and post-campaign periods.
Parameters
DescriptionNameType
The start of the pre-campaign perioda ("pre_date")date
The start of the campaignb ("campaign_start_date")date
The end of the campaignc ("campaign_end_date")date
The end of the post-campaign periodd ("post_date")date
Result
ValuesBin index
dates earlier than a; the lower end bin1
dates between a (inclusive) and b (exclusive)2
dates between b (inclusive) and c (exclusive)3
dates between c (inclusive) and d (exclusive)4
352Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
ValuesBin index
dates d and later; the upper end bin5
null [see The null value on page 181]6
unclassified7
See also
DayMultiplePrePost, WeekMultiplePrePost, MonthMultiplePrePost, YearMultiplePrePost onpage 343
DayPrePost, WeekPrePost, MonthPrePost, YearPrePost on page 346
PrePost on page 353
PrePost
Purpose: bin date values into pre-campaign and post-campaign periods.
Parameters
DescriptionNameType
The start of the pre-campaign perioda ("pre_date")date
The campaign dateb ("campaign_date")date
The end of the post-campaign periodc ("post_date")date
Result
ValuesBin index
dates earlier than a; the lower end bin1
dates between a (inclusive) and b (exclusive)2
353Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
ValuesBin index
dates between b (inclusive) and c (exclusive)3
dates c and later; the upper end bin4
null [see The null value on page 181]5
unclassified6
See also
DayMultiplePrePost, WeekMultiplePrePost, MonthMultiplePrePost, YearMultiplePrePost onpage 343
DayPrePost, WeekPrePost, MonthPrePost, YearPrePost on page 346
PreDuringPost on page 352
Sign
Purpose: bin numeric values according to sign.
Parameters None
Result
ValuesBin index
; that is, negative values1
02
; that is, positive values3
null [see The null value on page 181]4
unclassified5
See also
NegativeNonNegative on page 351
354Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
Binnings
28 - XML in SpectrumMiner
In this section
XML in Spectrum Miner 356Metadata specification for qsimportmetadata 357Aggregation specification for qsmeasure 366Derivation specification for qsderive 368Selection specification for qsselect 369Crosstab specification for qsxt 371Field name mapping specification for qsrenamefields 374Decision-tree build specification for qsdecisiontree 376Scorecard build specification for qsscorecard 379Binning specifications 381Attribute values 390
XML in Spectrum Miner
To allow easy interchange of results with other applications, XML is used throughout SpectrumMiner.
All XML inputs to the Spectrum Miner data-build commands (and any Spectrum Miner functionthat is implemented via a Spectrum Miner data-build command) must be in the Spectrum Minernamespace. The easiest way to ensure this is to include the attributexmlns="http://www.quadstone.com/xml" in the root element of the XML document.
XML files created by SpectrumMiner automatically include an optional schemaLocation attribute.If you use a schema-aware editor to create XML files, you can access these schemas directly fromyour Spectrum Miner installation at:
<smhome>/server/qs8.0/integration/schemas
(where <smhome> is the SpectrumMiner installation directory). You can also find examples of howto process these XML files at:
<smhome>/server/qs8.0/etc/xslt
See also
Aggregation specification for qsmeasure on page 366
Binning specifications on page 381
Crosstab specification for qsxt on page 371
Decision-tree build specification for qsdecisiontree on page 376
Derivation specification for qsderive on page 368
Field name mapping specification for qsrenamefields on page 374
Metadata specification for qsimportmetadata on page 357
Scorecard build specification for qsscorecard on page 379
Selection specification for qsselect on page 369
356Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Metadata specification for qsimportmetadata
metadata
DescriptionValueRequired
The root focuselementfocus
Examples
Describe the metadata for importing.
<?xml version="1.0" encoding="UTF-8"?><metadata xmlns="http://www.quadstone.com/xml"><focus><comment>This focus was created for the Retail Analysisproject.</comment>
</focus></metadata>
focus
DescriptionValueOptional
The name of the default subfocusfocus name attributelaunch
The name for a subfocusfocus name attributename
The name of the objective fieldfield name attributeobjective
The name of the partition fieldfield name attributepartition
The focus commentelementcomment
357Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
DescriptionValueOptional
The focus historyelementhistory
The fields to import metadata intoelementfield
Subfoci definitionselementfocus
The launch attribute can only appear in the root focus. All other subfoci are required tohave a name attribute.
Note:
Example Describe the root focus.
<?xml version="1.0" encoding="UTF-8"?><metadata xmlns="http://www.quadstone.com/xml"><focus><comment>This focus was created for the Retail Analysisproject.</comment></focus>
</metadata>
field in metadata
DescriptionValueRequired
The name of the field being definedfield name attributename
the name of the field being referencedfield name attributeref
DescriptionValueOptional
The data type representing this fieldsee field typetype
The length of the string representationinteger attributelength
Whether this field is an analysiscandidate
binary attributeanalysis
Whether this field is an export fieldbinary attributeexport
358Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
DescriptionValueOptional
The field commentelementcomment
The field binningelementbinning
The field derivation FDL expressionelementfdl
The field record selectionelementrecordselection
Note: • When a field is being defined, for example as a derivation, it is required to have a nameattribute. When it is being referenced, for example for inclusion in a subfocus, it is requiredto have a ref attribute.
• length and type attributes can only be used when defining a field (when the nameattribute has been given).
Examples Describe a string field:
<field name="HomePostCode" type="string" length="8"/>
Describe an analysis candidate:
<field name="TimeAtAddressMonths" type="integer" analysis="true"/>
See also
Binning specifications on page 381
fdl
DescriptionValueRequired
The FDL expression to describe thefield derivation
textcontent
DescriptionValueOptional
The numeric seed for the randomnumber generator
integer attributeseed
359Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Examples
<fdl>mode(Store)</fdl><fdl>if PaymentMethod = "CA" then Amount else 0</fdl><fdl seed="123">rndUniform()</fdl>
comment
DescriptionValueRequired
The text description for the commenttextcontent
or
DescriptionValueRequired
A comment is formatted as XHTMLbinary attributexhtml
The formatted text description for thecomment, which can contain any
string elementdiv
XHTML markup that is permitted toappear in a<div> element
When the xhtml attribute is used, it always takes the value true.Note:
Examples
Add a comment to a focus:
<?xml version="1.0" encoding="UTF-8"?><metadata xmlns="http://www.quadstone.com/xml"><focus><comment>This focus was created for the Retail Analysisproject.</comment>
</focus></metadata>
Add a formatted comment to a focus:
<?xml version="1.0" encoding="UTF-8"?><metadata xmlns="http://www.quadstone.com/xml">
360Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
<focus><comment xhtml="true">
<div>This focus was created for theRetail Analysis project.<br/>An audit is available on the<a href="http://intranet.company.com/audits">intranet</a>.
</div></comment>
</focus></metadata>
history
Focus history is a single history element containing a non-empty sequence of operation elements.Each operation element is just plain text representing a single Spectrum Miner data-build commandapplied to the focus.
DescriptionValueRequired
One particular action in the historystring elementsoperation
Example
Describe the history of all actions on the focus:
<history><operation>qsimportflat -input DB.fdd -outputDB.ftr</operation><operation>qsupdate -from template.ftr -toDB.ftr</operation>
</history>
recordselection
DescriptionValueOptional
The numeric record selectionelementnumeric
361Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
DescriptionValueOptional
The date record selectionelementdate
The categorical record selectionelementcategorical
The recordselection element takes one of the numeric, date, or categoricalsubelements.
Note:
Examples
Integer range selection between -2677 (inclusive) and 16808 (exclusive), and excluding Nulls:
<field name="Income" type="integer"><recordselection><numeric nullhandling="exclude"><range min="-2677" minisincluded="true" max="16808"maxisincluded="false"/></numeric>
</recordselection></field>
numeric
DescriptionValueRequired
Specify how the record selectionhandles null values
see nullhandlingnullhandling
DescriptionValueOptional
Specify the numeric range selectionelementrange
The numeric element accepts multiple range subelements.Note:
Example
Remove an existing field selection, and any Null selection:
<field name="Age"><recordselection>
362Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
<numeric nullhandling="nomention"/></recordselection></field>
date
DescriptionValueRequired
Specify how the record selectionhandles null values
see nullhandlingnullhandling
DescriptionValueOptional
Specify the date range selectionelementrange
The date element accepts multiple range subelements.Note:
Example
Date range selection, only available with inclusive end points, and dates specified in a fixed format:
<field name="DefaultDate"><recordselection><date nullhandling="nomention"><range min="1940-01-03T16:22:46Z"max="1948-05-19T00:29:17Z"/><range min="1956-04-01T20:35:25Z"max="1973-09-01T04:48:59Z"/><range min="1981-03-15T16:54:52Z"max="2002-08-22T01:11:34Z"/><range min="2012-09-08T01:19:24Z"max="2029-12-08T05:32:51Z"/>
</date></recordselection>
</field>
363Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
categorical
DescriptionValueRequired
Specify how the record selectionhandles null values
see nullhandlingnullhandling
Specify whether the other category isselected or deselected.
see otherstatusotherstatus
DescriptionValueOptional
The category values in the selectionelementselect
The categorical element accepts multiple select subelements.Note:
Example
A categorical selection that uses the otherstatus to deselect any field not explicitly mentioned,for example, select only categories D and E:
<field name="Lifestage"><recordselection><categorical nullhandling="selected" otherstatus="deselected"><select status="selected" value="D"/><select status="selected" value="E"/>
</categorical></recordselection>
</field>
Removing the selection from a categorical field:
<field ref="Lifestage"><recordselection><categorical nullhandling="selected"otherstatus="selected">
</categorical></recordselection>
</field>
364Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
range
DescriptionValueOptional
The minimum range of the recordselection
numeric or date attributemin
The maximum range of the recordselection
numeric or date attributemax
Include this value in the minimum rangebinary attributeminisincluded
Include this value in the maximumrange
binary attributemaxisincluded
The range element requires at least one min and one max attribute.Note:
select
DescriptionValueRequired
The name of the categorystring attributevalue
Include in or exclude from selectionsee statusstatus
365Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Aggregation specification for qsmeasure
aggregations
DescriptionValueRequired
The field definitionselementsfield
Example
Describe the creation of two new aggregates:
<?xml version="1.0" encoding="UTF-8"?><aggregations xmlns="http://www.quadstone.com/xml">
<field name="mostCommonStore" context="aggregation"><fdl>mode(Store)</fdl>
</field><field name="averageSpendInStore_" context="aggregation">
<fdl>mean(Amount)</fdl><by>StoreSplitFunction(Store)</by>
</field></aggregations>
field in aggregations
DescriptionValueRequired
The name of the field to be createdfield name attributename
Whether this field is to be used in anaggregation, statistic, or tracker file
see measure contextcontext
The FDL expression to describe theaggregation
elementfdl
366Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
DescriptionValueOptional
The data type ofthe field to becreated
see field typetype
The length of astring field
integer attributelength
Whether this fieldis created in theoutput focus
binary attributetemporary
The split-byfunction
string elementby
The FDLexpression recordfilter
string elementwhere
The default valueof the aggregation
string elementdefault
Examples
Describe a field with the mode aggregation on field Store:
<field name="mostCommonStore" context="aggregation"><fdl>mode(Store)</fdl></field>
Describe a field with the mean aggregation on field Amount, split by function StoreSplitFunction:
<field name="averageSpendInStore_" context="aggregation"><fdl>mean(Amount)</fdl><by>StoreSplitFunction(Store)</by></field>
367Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Derivation specification for qsderive
derivations
DescriptionValueRequired
The field definitionselementsfield
Example
Describe a set of three derivations:
<?xml version="1.0" encoding="UTF-8"?><derivations xmlns="http://www.quadstone.com/xml">
<field name="TransMonth"><fdl>month(PurchaseDate)</fdl>
</field><field name="CashRecd">
<fdl>if PaymentMethod = "CA" then Amount else 0</fdl></field><field name="Random">
<fdl seed="123">rndUniform()</fdl></field>
</derivations>
field in derivations
DescriptionValueRequired
The name of the field to be createdfieled name attributename
The FDL expression to describe thefield to be created
elementfdl
368Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
DescriptionValueOptional
The data type of the field to be createdsee field typetype
The length of a string fieldinteger attributelength
Examples
Describe a field using an in-built function:
<field name="TransMonth"><fdl>month(PurchaseDate)</fdl></field>
Describe a field using a simple control statement:
<field name="CashRecd"><fdl>if PaymentMethod = "CA" then Amount else 0</fdl></field>
Describe a random field with a specific seed:
<field name="Random"><fdl seed="123">rndUniform()</fdl></field>
Selection specification for qsselect
selections
DescriptionValueOptional
The fields to create this selectionelementsfield
Example
369Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Describe a random selection:
<xml version="1.0" encoding="UTF-8"?><selections xmlns="http://www.quadstone.com/xml"?>
<field name="random" context="selection"><fdl seed="12345678">rndUniform() \< 0.1</fdl>
</field></selections>
field in selections
DescriptionValueRequired
The name of the field to be createdfield name attributename
FDL expression to describe the recordselection.
Must produce an integer valued zeroor one result
elementfdl
This field is for use with qsselectselectioncontext
Example
Describe a random selection:
<field name="random" context="selection"><fdl seed="12345678">rndUniform() \< 0.1</fdl></field>
370Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Crosstab specification for qsxt
crosstabset
DescriptionValueRequired
The crosstabs to be evaluatedelementscrosstab
DescriptionValueOptional
Whether the crosstabset represents aprofile, rather than a crosstab
binary attributeprofile
The description of the crosstabsetstring elementdescription
Example
Describe a single crosstab crosstabset:
<?xml version="1.0" encoding="UTF-8"?><crosstabset xmlns="http://www.quadstone.com/xml"><description>No Description</description><crosstab indexscheme="compact1">
<description>No Description</description><specification>
<field name="Age"/><field name="Gender"/><function name="count"/><function name="mode">
<parameter>MaritalStatus</parameter></function>
</specification></crosstab></crosstabset>
371Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
crosstab
DescriptionValueRequired
The indexing scheme used in thiscrosstab
compact1indexscheme
The specification of the crosstabelementspecification
DescriptionValueOptional
The description of the crosstabstring elementsdescription
Example
Describe a single crosstab crosstabset:
<?xml version="1.0" encoding="UTF-8"?><crosstabset xmlns="http://www.quadstone.com/xml">
<description>No Description</description><crosstab indexscheme="compact1">
<description>No Description</description><specification>
<field name="Age"/><field name="Gender"/><function name="count"/><function name="mode">
<parameter>MaritalStatus</parameter></function>
</specification></crosstab>
</crosstabset>
372Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
specification in crosstab
DescriptionValueRequired
The functions to be used as crosstabmeasurements
elementsfunction
DescriptionValueOptional
The fields to be used as segmentationbreakdowns
elementsfield
Example
Describe a crosstab of fields Age and Gender with measures count() andmode(MaritalStatus):
<specification><field name="Age"/><field name="Gender"/><function name="count"/><function name="mode">
<parameter>MaritalStatus</parameter></function>
</specification>
function
DescriptionValueRequired
Identify the name of the measurefield name attributename
DescriptionValueOptional
Identify the field parameter to thefunction
field name attributeparameter
373Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Examples
Describe a function with no parameters:
<function name="count"/>
Describe a function with a single parameter MaritalStatus:
<function name="mode"><parameter>MaritalStatus</parameter></function>
field in crosstab
DescriptionValueRequired
The name of the field to be used in thecrosstab
field name attributename
Example
Specify field Age for use in the crosstab:
<field name="Age"/>
Field name mapping specification for qsrenamefields
mappingset
DescriptionValueRequired
The mapping of field nameselementsmap
Example
374Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Describe the renaming of three fields:
<?xml version="1.0" encoding="UTF-8"?><mappingset xmlns="http://www.quadstone.com/xml">
<map><name>StartDate</name><alias>Initialized</alias>
</map><map>
<name>Age</name><alias>CurrentAge</alias>
</map><map>
<name>Gender</name><alias>Sex</alias>
</map></mappingset>
map
DescriptionValueRequired
The name of the field to renamefield name elementname
The new field namefield name elementalias
Example
Rename field StartDate to Initialized:
<map><name>StartDate</name><alias>Initialized</alias>
</map>
375Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Decision-tree build specification for qsdecisiontree
decisiontree
DescriptionValueRequired
The decision-tree build specificationelementspecification
Example
Describe a decision-tree build specification with objective field Age and three analysis candidates,creating new result field PredictedAge:
<?xml version="1.0" encoding="UTF-8"?><decisiontree xmlns="http://www.quadstone.com/xml">
<specification nlevels="3"><objectivefield>
<field name="Age"/></objectivefield><analysiscandidates>
<field name="Gender"/><field name="MaritalStatus"/><field name="HomeOwner"/>
</analysiscandidates><resultfield>
<field name="PredictedAge"/></resultfield>
</specification></decisiontree>
specification in decisiontree
DescriptionValueRequired
The depth of the tree to be builtinteger attributenlevels
376Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
DescriptionValueRequired
The objective fieldfield name elementobjectivefield
The analysis candidatesfield names elementanalysiscandidates
The name for the created results fieldfield name elementresultfield
DescriptionValueOptional
The constraints on the decision-treebuild
elementsplitconstraints
The k-way split fieldfield name elementkwaysplitfield
The partition fieldfield name elementpartitionfield
The test/training methodology to beused
elementtesttrain
Example
Describe a decision-tree build specification with objective field Age and three analysis candidates,creating new result field PredictedAge:
<specification nlevels="3"><objectivefield>
<field name="Age"/></objectivefield><analysiscandidates>
<field name="Gender"/><field name="MaritalStatus"/><field name="HomeOwner"/>
</analysiscandidates><resultfield>
<field name="PredictedAge"/></resultfield></specification>
Note: • If the <partitionfield> element is empty, qsdecisiontree will ignore any existingpartition prior to building the tree.
• If the <kwaysplitfield> element is empty, qsdecisiontree will ignore any existingk-way split prior to building the tree.
377Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
splitconstraints
DescriptionValueRequired
The minimum size of a segmentpopulation
integer attributeminpopsize
The minimum size of the treatedsegment population
integer attributeminpopsizetreated
The minimum size of the controlsegment population
integer attributeminpopsizecontrol
DescriptionValueOptional
The combining function used to selecthow the final split is chosen
see combining functioncombiningfunction
Note: • When a partitionfield element is given, the minpopsizetreated orminpopsizecontrol are valid splitconstraint attributes. Otherwise, only attributeminpopsize should be set.
• The combiningfunction attribute should only be used when the kwaysplitfieldelement is defined.
testtrain
DescriptionValueRequired
The proportion of records, as a numberbetween zero and one, to be used totrain the decision-tree build
real attributethreshold
The random field to define the test trainsplit
field name elementnamefield
378Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Example
<testtrain threshold="0.5"><field name="randomField"/>
</testtrain>
Scorecard build specification for qsscorecard
scorecard
DescriptionValueRequired
The scorecard build specificationelementspecification
Example
Describe a scorecard build specification with objective field Age and three analysis candidates,creating new result field PredictedAge:
<?xml version="1.0" encoding="UTF-8"?><scorecard xmlns="http://www.quadstone.com/xml">
<specification modeltype="value" nfields="2" iscontinuous="true"><objectivefield>
<field name="Age"/></objectivefield><analysiscandidates>
<field name="Gender"/><field name="MaritalStatus"/><field name="HomeOwner"/>
</analysiscandidates><resultfield>
<field name="PredictedAge"/></resultfield>
</specification></scorecard>
379Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
specification in scorecard
DescriptionValueRequired
Build a continuous-objective scorecard,regardless of the setting ofmodeltype
binary attributeiscontinuous
What type of (binary) model to build(ignored if iscontinuous is true);
see model typesmodeltype
if this is unset, build acontinuous-objective scorecard
The number of fields in the createdscorecard
integer attributenfields
The objective fieldfield name elementobjectivefield
The analysis candidatesfield names elementanalysiscandidates
The name for the created result fieldfield name elementresultfield
DescriptionValueOptional
The threshold above which analysiscandidates are excluded for being toocorrelated with the objective
real attributecorrelationthreshold
The test/training methodology to beused
elementtesttrain
Example
<specification modeltype="value" nfields="2" iscontinuous="true"><objectivefield>
<field name="Age"/></objectivefield><analysiscandidates>
<field name="Gender"/><field name="MaritalStatus"/><field name="HomeOwner"/>
380Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
</analysiscandidates><resultfield>
<field name="PredictedAge"/></resultfield>
</specification>
Binning specifications
binning
DescriptionValueOptional
The name of the binningstring attributename
The numeric binningelementnumeric
The date binningelementdate
The geographical binningelementgeographic
The categorical binningelementcategorical
The binning element takes only one of either numeric, date, geographical orcategorical subelements.
Note:
Examples
Add a numeric binning to field LoanAmount:
<field name="LoanAmount"><binning name="LoanAmount-1">
<numeric targetnbins="10" style="equalwidth"/></binning>
</field>
381Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Add a categorical binning to field Gender:
<field name="Gender"><binning name="Gender-1">
<categorical><categories>
<category name="male" levelname="unnamed node"><category name="1" levelname="base categories"
value="1"/></category>
<category name="female"><category name="2" value="2"/></category>
</categories></categorical>
</binning></field>
Add a geographic field binning to field StateCode:
<field name="StateCode"><binning>
<geographic><maplocation>/home/data/maps/us.mif</maplocation>
</geographic></binning>
</field>
numeric
DescriptionValueOptional
The required number of binsinteger attributetargetnbins
Automatically adjust the number of binsto create nice bin boundaries. Defaultsto true
binary attributeadjustnbins
Whether a binning on a real value fieldshould have whole numberedboundaries
binary attributeforcewholenumbered
The type of binning appliedsee bin stylestyle
382Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
DescriptionValueOptional
The binning displaysee binning displaybinning
The lower end bin definition. By defaultequal width binning useauto endbins,while equal population use none
elementminendparam
The upper end bin definition. By defaultequal width binning useauto endbins,while equal population use none
elementmaxendparam
The bin boundaries for user-definedbinning style
elementboundaries
Example
Add an equal population numeric binning with ten ranges to a field:
<binning name="LoanAmount-1"><numeric targetnbins="10" style="equalwidth"/>
</binning>
date
EffectOptional
The required number of binsinteger attributetargetnbins
The binning time periodsee date periodbinby
Which date to bin by date period fromsee date referencereference
The number of time periods for a bininteger attributebinwidth
The type of binning appliedsee bin stylestyle
How the binning should be displayedsee binning displaybinning
383Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
EffectOptional
The lower end bin definition. By defaultequal width binning useauto endbins,while equal population use none
elementminendparam
The upper end bin definition. By defaultequal width binning useauto endbins,while equal population use none
elementmaxendparam
The bin boundaries for user-definedbinning style
elementboundaries
Example
Create an date binning, by qaurters, starting from 1st March 1992.
<date display="equalwidth" binwidth="1" binby="quarter"reference="auto" style="equalwidth"><minendparam type="fixed">
<boundary>1992-03-01T00:00:00Z</boundary></minendparam><maxendparam type="none"/><boundaries endbins="low">
<boundary>1992-03-01T00:00:00Z</boundary><boundary>1992-04-01T00:00:00Z</boundary><boundary>1992-07-01T00:00:00Z</boundary><boundary>1992-10-01T00:00:00Z</boundary><boundary>1993-01-01T00:00:00Z</boundary><boundary>1993-04-01T00:00:00Z</boundary><boundary>1993-07-01T00:00:00Z</boundary><boundary>1993-10-01T00:00:00Z</boundary><boundary>1994-01-01T00:00:00Z</boundary><boundary>1994-04-01T00:00:00Z</boundary><boundary>1994-07-01T00:00:00Z</boundary><boundary>1994-10-01T00:00:00Z</boundary><boundary>1995-01-01T00:00:00Z</boundary><boundary>1995-04-01T00:00:00Z</boundary>
</boundaries></date>
384Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
categorical
DescriptionValueRequired
The description ofthe categoryhierarchy
elementcategories
The location ofthe file containing
file elementhierarchylocation
the categoricalhierarchydescription
The number ofcategories tocreate
elementselectn
Only one of these elements is required to define a categorical binning.Note:
Example Describe a categorical binning:
<binning name="Gender-1"><categorical>
<categories><category name="male" levelname="unnamed node">
<category name="1" levelname="base categories"value="1"/>
</category><category name="female">
<category name="2" value="2"/></category>
</categories></categorical>
</binning>
385Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
categories
DescriptionValueRequired
A category in the hierarchyelementscategory
Example
Describe the categorical hierarchy:
<binning name="Gender-1"><categorical>
<categories><category name="male" levelname="unnamed node">
<category name="1" levelname="base categories"value="1"/>
</category><category name="female">
<category name="2" value="2"/></category>
</categories></categorical>
</binning>
category
DescriptionValueRequired
The name of the categorystring attributename
The value for this categorynumeric, date or string attributevalue
A subcategory descriptionelementscategory
386Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
DescriptionValueOptional
The name of the level of the categoricalhierarchy, required only on the firstcategory in each set
string attributelevelname
Note: • A category element either has a value attribute, or a category element.• All categories should recurse to the same depth.
Examples
Describe the category mapping between field value and label:
<category name="female"><category name="1" value="1"/>
</category>
Describe the category grouping of many field values to a single label:
<category name="InArrears"><category name="2" value="2"/><category name="6" value="6"/><category name="14" value="14"/><category name="15" value="15"/><category name="18" value="18"/><category name="25" value="25"/></category>
selectn
DescriptionValueRequired
The number of categories to createinteger attributen
DescriptionValueOptional
The minimum number of records ineach category
integer attributemincatsize
387Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
geographic
DescriptionValueRequired
The location of the map filefile elementmaplocation
Example
Add a geographic field binning to field StateCode:
<geographic><maplocation>/home/data/maps/us.mif</maplocation>
</geographic>
minendparam
DescriptionValueOptional
The type of end binsee end bin styletype
The fixed end bin valueelementboundary
Example
Describe a binning with the lower end bin fixed to value 18 :
<binning name="Age-1"><numeric targetnbins="10" style="equalwidth">
<minendparam type="fixed"><boundary falls="right">18.0</boundary>
</minendparam></numeric>
</binning>
388Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
maxendparam
DescriptionValueOptional
The type of end binsee end bin styletype
The fixed end bin valueelementboundary
Example
Describe a binning with the upper end bin set to be automatically set by Spectrum Miner:
<binning name="Age-1"><numeric targetnbins="10" style="equalpopulation">
<maxendparam type="auto"/></numeric>
</binning>
boundaries
DescriptionValueOptional
Which end bins are enabledsee end binsendbins
The bin boundary definitionelementsboundary
Example
Defining the boundaries of a user defined binning:
<binning name="Age-1"><numeric style="userdefined">
<boundaries endbins="both"><boundary falls="right">20.0</boundary><boundary falls="right">30.0</boundary><boundary falls="right">40.0</boundary>
</boundaries></numeric>
</binning>
389Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
boundary
DescriptionValueRequired
The boundary valuenumeric or datecontent
DescriptionValueOptional
The boundary typesee end bin styletype
Whether the boundary is inclusive orexclusive
see boundary inclusionfalls
Date values should be provided in the standard YYYY-MM-DDTHH:MM:SSZ format.Note:
Example
Define the boundary which includes the value 20:
<boundary falls="right">20.0</boundary>
Define the boundary which does not include the value 20:
<boundary falls="left">30.0</boundary>
Define a date boundary:
<boundary>1992-03-01T00:00:00Z</boundary>
Attribute values
ValuesName
nomention | include | include_only| exclude | selected | deselected
nullhandling
390Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
ValuesName
selected | deselectedstatus
selected | deselectedotherstatus
integer | real | string | date |unknown | numeric
field type
none | auto | fixedend bin style
equalpopulation | equalwidth |userdefined
bin style
none | both | high | lowend bins
left | rightboundary inclusion
equalwidth | compactendsbinning display
auto | hour | minute | second | day| week | month | quarter | year |decade | century
date period
auto | minend | maxenddate reference
aggregation | statistic | trackermeasure context
unused | mean | meanlessstddev |meanlesssqrtstddev | min
combining function
risk | response | churn |satisfaction | dissatisfaction
model types
391Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
XML in Spectrum Miner
Index
* (FDL operator) 193%, mod (FDL operator) 193
Aabout railroad diagrams 142About the Scorecard Wizard 114abs (FDL function) 286absolute value of a number 286ActionROI (FDL function) 318ActionROIAnnualized (FDL function) 319addcenturies, addcenturiescountbackwards (FDL functions)
223adddays (FDL function) 224addhours (FDL function) 225addition 193FDL operator 193addminutes (FDL function) 226addmonths, addmonthscountbackwards (FDL functions) 227addseconds (FDL function) 229addweeks (FDL function) 230addyears, addyearscountbackwards (FDL functions) 231aggregating a field 146in a TML aggregations file 146aggregation 63using qsmeasure 63aggregation functions 156for use in measurements and derivations 156anchor 278in an FDL regular expression 278and (FDL operator) 196any (aggregation function) 156, 159in measurements and derivations 156archiving a focus file 95using qsarchive 95arithmetic expression 184in FDL 184arithmetic operators 193in FDL 193assigning a value to a variable 187in FDL 187atom 276in an FDL regular expression 276auditing a focus 98using qsaudit 98automating data builds 24using qsbuild 24average 168, 292of a list of numbers 292of values in a field 168
Bback-reference 278in an FDL regular expression 278basic expression 184in FDL 184bin (FDL function) 338
binnings 150for breaking down aggregations 150binomial distribution 311Boolean (binning) 339boolean data 182in FDL 182building a scorecard 115using qsscorecardwizard (linked copy) 115building association rules!using qsruleminer 125build plan 24using in qsbuild 24by clause (TML) 150
Ccalculate statement (TML) 153case 272–273
lower 272converting a string to 272upper 273converting a string to 273
case expression 186in FDL 186case expression (FDL) 186categorical binning hierarchy 137creating a category map using qsmapgen 137category map 137creating from a categorical binning hierarchy using qsmapgen
137ceil (FDL function) 286changing the name of a focus file 93using qsmove 93character class 276in an FDL regular expression 276character range 276in an FDL regular expression 276clamp (FDL function) 209comments 76, 184exporting from a focus to an XML file 76importing into a focus from an XML file 76in FDL 184compatibility 183of datatypes in FDL 183compiling a QMML document using qsqmmledit 129concat (FDL function) 259cond (FDL function) 210conditional evaluation in FDL 185, 210–211using if and case expressions 185using the cond function 210using the iff function 211confintlower (aggregation function) 156, 160in measurements and derivations 156confintupper (aggregation function) 156, 160in measurements and derivations 156constraining a number to an interval 209converting a field 217–220to a date 217to an integer 218
392Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
converting a field (continued)to a real 219to a string 220converting a metadata file 135using qsqsfmtosas 135converting a QMML file 134using qsqmml2sas 134using qsqmml2sql 134copying a focus file 91–92using qscopy (independent copy) 91using qslink (linked copy) 92copying focus metadata 81using qsupdate 81cos (FDL function) 287count (aggregation function) 156, 161in measurements and derivations 156countcenturies (FDL function) 232countdays (FDL function) 233counthours (FDL function) 234countminutes (FDL function) 235countnonnull/countnonnulls (aggregation function) 156, 162in measurements and derivations 156countnull/countnulls (aggregation function) 156, 163in measurements and derivations 156countseconds (FDL function) 236countunique (aggregation function) 156, 164in measurements and derivations 156countuniquenonnull (aggregation function) 156, 164in measurements and derivations 156countweeks (FDL function) 237countwholecenturies, countwholecenturiesbackwards (FDL
functions) 238countwholedays (FDL function) 240countwholehours (FDL function) 241countwholeminutes (FDL function) 242countwholemonths, countwholemonthsbackwards (FDL
functions) 243countwholeseconds (FDL function) 245countwholeweeks (FDL function) 246countwholeyears, countwholeyearsbackwards (FDL functions)
247countyears (FDL function) 248create statement (TML) 146crosstab 108creating by applying a specification using qsxt 108
Ddatabase connection 28creating or modifying using qsdbaccess 28database table 29, 31, 33, 35creating using qsdbcreatetable 31importing using qsimportdb 29inserting data using qsdbinsert 33updating data using qsdbupdate 35data-build commands 12, 18about 12standard command-line options 18datatype-compatibility 183in FDL 183datatype conversion 183in FDL 183
datatypes 181in FDL 181date 217converting a field to 217date formatting 52for flat files and FDL expressions 52date literal 184in FDL 184date type 181in FDL 181day (FDL function) 249DayFrom (binning) 340DayMultipleFrom (binning) 341DayMultipleNumBins (binning) 342DayMultiplePrePost (binning) 343DayMultipleTo (binning) 344DayMultipleWidth (binning) 345dayofweek (FDL function) 250DayPrePost (binning) 346DayRange (binning) 347DayTo (binning) 348dblookup (FDL function) 330deciles 173in FDL 173decision tree 105, 120building with qsdecisiontree 120creating a Model Snapshot using qsdtsnapshot 105default clause (TML) 148deleting a focus file 93using qsremove 93dependencies between foci 110viewing using qsinfo 110dependency tracking in data builds 24using qsbuild 24deriving a field 60, 67, 146in a TML derivations, trackers, selections, or aggregations file
146using qsderive 60using qstrack 67div (FDL operator) 193division 193FDL operators 193duplicate records 87removing using qsmerge 87
Eediting a QMML document using qsqmmledit 129endswith (FDL function) 260equality 194FDL operator 194EqualRange (binning) 349EqualRangeWidth (binning) 350Excel dataset 44, 46, 104creating from a focus using qsexportstat 46describing using qsdescribestat 104importing using qsimportstat 44exp (FDL function) 288exponential distribution 312exporting data 33, 35, 41, 46to a database table using qsdbinsert 33to a database table using qsdbupdate 35
exporting data (continued)to a flat file using qsexportflat 41to a third-party dataset using qsexportstat 46exporting metadata from a focus to an XML file 76expression, expression list 183in FDL 183extracting from an archive file!using qsunzip 95
Ffalse and true values 182in FDL 182FDL 132, 179, 181, 183, 185, 187, 190, 193–194, 196–198,
206, 276about 179datatypes 181expression, expression list 183
functions 190, 198built-in 198user-defined 190
if and case expressions 185operators 193–194, 196–197arithmetic 193logical 196precedence 197relational 194
reserved words 206string matching 276transforming to or from QMML using qslt 132variables 187field 60, 67, 73, 146aggregating in a TML aggregations file 146deriving in a TML derivations, trackers, selections, or
aggregations file 146deriving using qsderive 60deriving using qstrack 67renaming using qsrenamefields 73Field Derivation Language 179see FDL 179filtering records 148in a TML aggregations file 148find (FDL function) 261first (aggregation function) 156, 165in measurements and derivations 156flag field 214creating from a selection 214flat-data description file 37, 132creating from a flat file using qsgenfdd 37creating from QMML using qslt 132flat file 37, 39, 41creating from a focus using qsexportflat 41creating metadata using qsgenfdd 37importing using qsimportflat 39floor (FDL function) 288focus 12, 29, 39, 41, 44, 46, 48, 59, 76, 87, 98creating a Profile and Audit using qsaudit 98creating from a database table using qsimportdb 29creating from a flat file using qsimportflat 39creating from a third-party dataset using qsimportstat 44creating from data in another focus using qsimportfocus 48exporting metadata to an XML file 76exporting to a flat file using qsexportflat 41
focus (continued)exporting to a third-party dataset using qsexportstat 46importing metadata from an XML file 76read-only 12removing duplicate records using qsmerge 87sorting records using qssort 59focus!describing using qsdescribe 102focus file 91–93, 110copying using qscopy (independent copy) 91copying using qslink (linked copy) 92deleting using qsremove 93moving/renaming using qsmove 93viewing dependencies and file sizes using qsinfo 110focus join 83using qsjoin 83focus merge 87using qsmerge 87focus statistic 153evaluating in a TML statistics file 153function 184, 190, 198
in FDL 184, 198applying 184built-in 198
in FDL!defining 190function arguments (FDL) 190function attributes (FDL) 192function definition (FDL) 190
Ggamma distribution 313global variable 187in FDL 187gmt2edt (FDL functions) 250greater than (or equal to) 194FDL operators 194
Hhour (FDL function) 251
Iidentifier 184in FDL 184if expression 185in FDL 185if expression (FDL) 185iff (FDL function) 211ifnull (FDL function) 212importing data 29, 39, 44, 48from a database table using qsimportdb 29from a flat file using qsimportflat 39from an existing focus using qsimportfocus 48from a third-party dataset using qsimportstat 44importing metadata from an XML file into a focus 76importing metadata from one focus into another 81using qsupdate 81index number 173, 338for a bin 338for a crosstab segment 173
394Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
inequality 194FDL operator 194initializing a variable 187in FDL 187integer 218converting a field to 218integer type 181in FDL 181isnull (FDL function) 213isselected (FDL function) 214
Jjoining fields 83using qsjoin 83joining strings together 259
Llast (aggregation function) 156, 166in measurements and derivations 156left (FDL function) 262length of a string 267less than (or equal to) 194FDL operators 194linked copy of a focus file 92using qslink 92literal value 184in FDL 184local variable 187in FDL 187log (FDL function) 289log10 (FDL function) 290logbase (FDL function) 290logical expression 184in FDL 184logical operators 196in FDL 196lookup 330of values in a reference table 330lower case 272converting a string to 272
Mmagnitude of a number 286map of categories 137creating from a categorical binning hierarchy using qsmapgen
137market-basket analysis!using qsruleminer 125match (FDL function) 280matching patterns 276in FDL 276max, maxnonnull (multi-argument FDL functions) 291max (aggregation function) 156, 167in measurements and derivations 156mean, meannonnull (multi-argument FDL functions) 292mean (aggregation function) 156, 168in measurements and derivations 156measurement 63using qsmeasure 63
median (aggregation function) 156, 168in measurements and derivations 156member (FDL function) 332merging records 87using qsmerge 87metadata 37, 76, 81copying from one focus to another using qsupdate 81creating flat-file metadata using qsgenfdd 37exporting from a focus to an XML file 76importing into a focus from an XML file 76mid (FDL function) 263min, minnonnull (multi-argument FDL functions) 293min (aggregation function) 156, 169in measurements and derivations 156minus 193FDL operators 193minute (FDL function) 252mod, % (FDL operator) 193mode (aggregation function) 156, 170in measurements and derivations 156model 105creating a Model Snapshot using qsdtsnapshot or
qsscsnapshot 105Model Snapshot 105unpacking using qshtmlunpack 105moderatio (aggregation function) 156, 171in measurements and derivations 156month (FDL function) 253MonthFrom (binning) 340MonthMultipleFrom (binning) 341MonthMultipleNumBins (binning) 342MonthMultiplePrePost (binning) 343MonthMultipleTo (binning) 344MonthMultipleWidth (binning) 345MonthPrePost (binning) 346MonthRange (binning) 347MonthTo (binning) 348moving a focus file 93using qsmove 93multiplication 193FDL operator 193
NNegativeNonNegative (binning) 351normal distribution 314normalize (FDL function) 294normalize field values 294not (FDL operator) 196now (FDL function) 253null value 181, 212–213in FDL 181replacement using ifnull or nvl 212testing for 213numeric datatypes 181in FDL 181numeric literal 184in FDL 184numericTestTrainSplit (FDL function) 302numericTestTrainValidateSplit (FDL function) 302nvl (FDL function) 212
OOfferROI (FDL function) 320OfferROIAnnualized (FDL function) 322Open Database Connectivity (ODBC) 28creating or modifying using qsdbaccess 28operator precedence 197in FDL 197operators 193–194, 196, 279
arithmetic 193in FDL 193
in an FDL regular expression 279logical 196in FDL 196relational 194in FDL 194
or (FDL operator) 196
Pparametrization of data builds 24using qsbuild 24parsing a QMML document using qsqmmledit 129percentage (aggregation function) 156, 172in derivations 156percentagerate (aggregation function) 156, 172in derivations 156Poisson distribution 315pow (FDL function) 294precedence of operators 197in FDL 197PreDuringPost (binning) 352PrePost (binning) 353product, productnonnull (FDL functions) 295Profile and Audit 105unpacking using qshtmlunpack 105profiling a focus 98using qsaudit 98
QQMML document 129, 132editing or compiling using qsqmmledit 129transforming using qslt 132viewing using qsqmmlview 129qsarchive (data-build command) 95qsaudit (data-build command) 98qsbuild (data-build command) 24qscopy (data-build command) 91qsdbaccess (data-build command) 28qsdbcreatetable 31qsdbinsert 33qsdbupdate 35qsdecisiontree (data-build command) 120qsderive (data-build command) 60qsdescribe (data-build command) 102qsdescribelicense (data-build command) 111qsdescribestat (data-build command) 104qsdtsnapshot (data-build command) 105qsexportflat 41qsexportmetadata (data-build command) 76qsexportstat (data-build command) 46
qsgenfdd 37qshtmlunpack (data-build command) 105qsimportdb 29qsimportflat 39qsimportfocus 48qsimportmetadata (data-build command) 76qsimportstat (data-build command) 44qsinfo (data-build command) 110qsjoin (data-build command) 83qslink (data-build command) 92qslt (data-build command) 132qsmapgen (data-build command) 137qsmeasure (data-build command) 63qsmerge (data-build command) 87qsmove (data-build command) 93qsqmml2sas (data-build command) 134qsqmmledit (data-build command) 129qsqmmltosql (data-build command) 134qsqmmlview (data-build command) 129qsqsfmtosas (data-build command) 135qsremove (data-build command) 93qsruleminer (data-build command) 125qsscorecard (data-build command) 122qsscorecardwizard (data-build command) 115qsscsnapshot (data-build command) 105qsselect (data-build command) 70qssettings (data-build command) 98qssort (data-build command) 59qstrack (data-build command) 67qsupdate (data-build command) 81qsxt (data-build command) 108
Rrandom number 311–315simulating the toss of a fair coin 312with binomial distribution 311with exponential distribution 312with gamma distribution 313with normal distribution 314with Poisson distribution 315with uniform distribution in an interval 315random numbers 311in FDL 311rankOrder (FDL function) 333rankOrderApprox (FDL function) 333rankOrderApproxMean (FDL function) 335rankOrderMean (FDL function) 335read-only focus 12real 219converting a field to 219real type 181in FDL 181record aggregation 63using qsmeasure 63record filtering 148in a TML aggregations file 148record selection 70using qsselect 70reference table lookup 330regular expression 276, 280, 282–283composition 276
396Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
regular expression (continued)replacing all matching substrings 282replacing the first matching substring 283testing a string for a match 280relation 184in FDL 184relational operators 194in FDL 194remainder 193FDL operator 193removing a focus file 93using qsremove 93renaming a field 73using qsrenamefields 73renaming a focus file 93using qsmove 93replace (FDL function) 214replaceall (FDL function) 282replacefirst (FDL function) 283reserved words 143, 206FDL 206TML 143RetentionActionROI (FDL function) 323RetentionActionROIAnnualized (FDL function) 324RetentionOfferROI (FDL function) 326RetentionOfferROIAnnualized (FDL function) 327return on investment 318–320, 322–324, 326–327using ActionROI 318using ActionROIAnnualized 319using OfferROI 320using OfferROIAnnualized 322using RetentionActionROI 323using RetentionActionROIAnnualized 324using RetentionOfferROI 326using RetentionOfferROIAnnualized 327right (FDL function) 264rndBinomial (FDL function) 311rndBool (FDL function) 312rndExp (FDL function) 312rndGamma (FDL function) 313rndNormal (FDL function) 314rndPoisson (FDL function) 315rndUniform (FDL function) 315round (FDL function) 296rounding to an integer 286, 288, 296nearest-integer rounding 296rounding down 288rounding up 286rownum (FDL function) 336rule 129adding to a QMML document using qsqmmledit 129ruleset 129adding to or deleting from a QMML document using
qsqmmledit 129
SsampleEqualSize (FDL function) 303sampleExactNumber (FDL function) 304sampleExactPercentage (FDL function) 305sampleStratified (FDL function) 306
sampling 302–308choosing an exact number of records 304choosing an exact percentage of records 305
partitioning 303into approximately equal-sized subpopulations 303test/training 302, 307using numericTestTrainSplit 302using testTrainSplit 307test/training/validation 302, 308using numericTestTrainValidateSplit 302using testTrainValidateSplit 308
two-way stratified 306scorecard 105, 122building with qsscorecard 122creating a Model Snapshot using qsscsnapshot 105second (FDL function) 254seed 311
for random number generation in FDL 311about 311
segindex (aggregation function) 156, 173in derivations 156selecting records 70using qsselect 70selection 214creating a flag field 214sequence mining!using qsruleminer 125set membership testing 332sgn (FDL function) 297Sign (binning) 354significance (aggregation function) 156, 174in derivations 156signum (sign) of a number 297sin (FDL function) 297smallinteger type 181in FDL 181sorting records in a focus 59using qssort 59soundex (FDL function) 265Spectrum Miner data-build commands 12, 18about 12standard command-line options 18split functions 150for breaking down aggregations 150sqrt (FDL function) 298square root of a number 298standard deviation of a field 175startswith (FDL function) 266state-change tracking 67using qstrack 67state variable 187in FDL 187statistic 153evaluating in a TML statistics file 153statistic identifier 184in FDL 184stdev (aggregation function) 156, 175in measurements and derivations 156stratified two-way sampling 306string 220, 260–267, 269–270, 273converting a field to 220extracting a final substring using FDL 264extracting an initial substring using FDL 262
string (continued)extracting a substring by position and length using FDL 263extracting a substring by start and end position using FDL
270length 267looking for a string at the end using FDL 260looking for a string at the start using FDL 266removing whitespace using FDL 273soundex encoding 265substitute 269testing for a substring using FDL 261string literal 184in FDL 184string matching 276in FDL 276string type 181in FDL 181strlen (FDL function) 267strmember (FDL function) 268substitute (FDL function) 269substr (FDL function) 270subtraction 193FDL operator 193sum, sumnonnull (multi-argument FDL functions) 299sum (aggregation function) 156, 175in measurements and derivations 156
Ttan (FDL function) 300template focus 81in qsupdate 81temporary statement (TML) 146test/training 302, 307–308using numericTestTrainSplit 302using numericTestTrainValidateSplit 302using testTrainSplit 307using testTrainValidateSplit 308testTrainSplit (FDL function) 307testTrainValidateSplit (FDL function) 308text 260–264, 266, 270, 273extracting a final substring using FDL 264extracting an initial substring using FDL 262extracting a substring by position and length using FDL 263extracting a substring by start and end position using FDL
270looking for a string at the end using FDL 260looking for a string at the start using FDL 266removing whitespace using FDL 273testing for a substring using FDL 261text data 181in FDL 181text file 37, 39creating metadata using qsgenfdd 37importing using qsimportflat 39time data 181in FDL 181TML 142–143, 156about 142aggregation functions 156reserved words 143todate (FDL function) 217
today (FDL function) 255tointeger (FDL function) 218tolower (FDL function) 272toreal (FDL function) 219tostring (FDL function) 220toupper (FDL function) 273tracking state changes 67using qstrack 67transaction measurement 63using qsmeasure 63Transaction Measurement Language 142see TML 142trim (FDL function) 273true and false values 182in FDL 182type-compatibility 183in FDL 183types of data 181in FDL 181
Uupper case 273converting a string to 273User Database Connection 28creating or modifying using qsdbaccess 28
Vvariable assignment (FDL) 187variable initialization and assignment 187in FDL 187variance (aggregation function) 156, 176in measurements and derivations 156
WWeekFrom (binning) 340WeekMultipleFrom (binning) 341WeekMultipleNumBins (binning) 342WeekMultiplePrePost (binning) 343WeekMultipleTo (binning) 344WeekMultipleWidth (binning) 345weekofyear (FDL function) 255WeekPrePost (binning) 346WeekRange (binning) 347WeekTo (binning) 348where clause (TML) 148whitespace 273removing using FDL 273
XXML 132formatting for legibility 132XML in Spectrum Miner 356
Yyear (FDL function) 256YearFrom (binning) 340
398Spectrum Miner™ 8.0 Data-Build Command and TML Reference Guide
YearMultipleFrom (binning) 341YearMultipleNumBins (binning) 342YearMultiplePrePost (binning) 343YearMultipleTo (binning) 344
YearMultipleWidth (binning) 345YearPrePost (binning) 346YearRange (binning) 347YearTo (binning) 348
3001 Summer Street
Stamford CT 06926-0700
USA
www.pitneybowes.com
© 2017 Pitney Bowes Software Inc.
All rights reserved
top related