Earthquake data in geonet.org - math.unm.edu

Earthquake data in geonet.org.nz

There is are large gaps in the 2012 and 2013 data, so let’s not use it.Instead we’ll use a previous year. Go tohttp://http://quakesearch.geonet.org.nz/ At the screen, click on theCLEAR button, so that it doesn’t assume you want data from last week,last month, or last year. Then enter in a date range starting in 2010. Ifyour birthday is December 10, then use December 10, 2010 to January 09,2010.

SAS Programming September 11, 2014 1 / 72

Earthquake data in geonet.org.nz

Towards the bottom of the page, click on the botton that says ”BuildQuery” instead of ”Show of Map”. This way, it will generate a data file foryou. Use the drop-down menu to select CSV as the file format.


Earthquake data

After you click the search button, you should be given a url for a websitewhere you can download the data as a .csv file. If it opens upautomatically in Excel, I recommend closing Excel without saving. Insteadget the file from your Downloads or Desktop or wherever it was saved sothat Excel doesn’t change anything.


Earthquake

An alternative is to download a dataset I’ve downloaded of all earthquakesfrom January 1st, 2010 to January 30th, 2011, and extract the month thatyou need. The dataset has about 28000 observations, and it will be up toyou to figure out how to get the exact dates you need. You could, forexample, open in Excel to figure out which rows you need, then usefirstobs= and obs= in SAS to get just those rows. In linux you can dothis

$ grep "2010-08" quake2010-2011.csv > AugSep.csv

$ grep "2010-09" quake2010-2011.csv >> AugSep.csv

This will grab all lines in the files that have earthquakes in August into afile, then append that file with all earthquakes in September.


Earthquake

A quick way to determine the line number that your data should start withis this, assuming that your birthday is say, August 15th.

$ nl quake2010-2011.csv > foo

$ grep "2010-08-15" foo | head -1

$ grep "2010-09-14" foo | tail -1

Here nl creates a version of the file with line numbers in the first column,and this saved to the file foo, which you can later delete. The first grepand head will spit out the first line in foo that has your birthday. Thesecond grep and tail will spit out the last line that has an event in themonth starting on your birthday.


Earthquake data

I tried uploading a file with 5000 lines of earthquakes in SAS Studio, andthis worked fine.


Earthquake data



Earthquake data



The SUM statement

Because the data step processes one observation at a time, it can beawkward to write functions of an entire column of data unless using aspecial procedure like PROC MEANS.

The SUM statement is a helpful way around this to create a counter orcumulutive sum in a data set.


SUM statements

The idea with the sum statement is to have a new variable which is acounter, and which is usually incremented conditionally.As an example, we’ll keep track of the number of cars that have more than100,000 miles in the cars.txt data set.


SUM statement with cars.txt dataset


SUM Statement

Some new aspects of the code are

I the syntax of using high + 1 defines a new variable and have itincrement by 1 each time the condition is true.

I In most languages, you would write something like high = high +1

instead, but that doesn’t work and just creates missing values

I the SUM function can also be used but requires the variable high tobe initialized first, so here just gives 1 for every nonmissing value





21:00 Monday, September 8, 2014 121:00 Monday, September 8, 2014 1


SUM and RETAIN statements with cars.txt dataset

You can fix previous code with a retain statement to initialize variables


SUM and RETAIN statements with cars.txt dataset

Note the order of the columns. The RETAIN statement effects the orderof the columns defined in this datastep, but additional RETAIN statementswon’t change the order of the variables read in from the SET statement.


MIN and RETAIN statements with cars.txt dataset

What will this do? Notice that lag1() returns the value of the variable inthe previous observation.


MIN and RETAIN statements with cars.txt dataset

Notice that the MIN function behaves intelligently for missing values—itreturns missing only when all values are missing. The minimjm of amissing value and a nonmissing value is the nonmissing value. minmiles

returns the minimum value encountered so far for the value of year.


First and Last observations for a given category (Chapter24).

Often we want to only output one value for each level of a given category.For example, I might only want to know the minimum mileage for eachmodel year of the cars. Or I might want to know how many cars therewere in a given year, with one observation per car.

In a medical setting, it is common to have datasets with patientID as onevariable with one record per doctor visit. The number of visits per patientcan vary widely, so sometimes it is useful to summarize this with a datasetwith one record (observation) per patientID. For this one record, youmight want to keep track of, for example, the total number of visits, thesum of the fees for all visits, or the time of the most recent visit. Formedical information, you might want to keep track of the highest bloodpressure recorded on all visits.


First and Last observations

We’ll practice using the car data again. Let’s create a dataset that has theyear and the minimum mileage for that year.

If you use a BY statement after a SET statement in a data set, SAScreates internal variables that start with FIRST. and LAST.. These allowyou to determine when a variable in the last one of its category (assumingthe data is already sorted on the same BY variable).







Here is simpler code that doesn’t use the lag1 function. However, it is abit less general and assumes more sorting. With the lag1 function, I couldkeep track of both the minimum and maximum at the same time and putthose into the data set on one row.




Example with Earthquake data: related to next homework!

As an example, we’ll use some earthquake data from New Zealand. Thisdata set was downloaded from http://www.geonet.org.nz, which has adatabase of New Zealand earthquakes. You specify a geographic range(using latitude and longitude) and a time frame. These were allearthquakes recorded from Jan 1st to March 31st in 2011. There were4892 recorded earthquakes in that interval.The goal here will be to keep a cumulative count of the number ofearthquakes that were at least 4.0 on the Richter Scale. The data is sorteddecreasing by time, and we want it to be sorted increasing by time.


Cumulative sums: Earthquake data

The data comes as a .csv file. It is hard to read by eye as a plain text file,but easy to read into computer programs like SAS.


Cumulative sums: Earthquake data

Here is what part of it looks like read into Excel. Several variables on theright were chopped off. As is typical for empirical data, the data setincludes many variables that you might not be interested in.


Dates and Times in the earthquake data

Something a little tricky about the earthquake data is that the times arenot in a standard format. They have a T in the middle separating datesfrom hours, minutes, and seconds, and they have a Z at the end. Thereare different things you could do to deal with this. You could replace T

and Z with spaces in the original data file and hope that this doesn’tinterfere with other variables such as eventtype.I’m not really sure a good way to do this without linux tools. In emacs Iwould do a global replace of T with a space, and it would tell me howmany replacements took place, which, if it is 2 per observation and thereare no missing values, then it worked.What we’ll do as a SAS solution is to read in the dates as characterstrings, but only read the first 10 characters, and this will chop off thehour-minute-second information. We’ll have to convert the string to adate (an integer), which can be done using the input() function.


SAS code for dealing with earthquake times


SAS code for dealing with earthquake times


input() function

The input function can also be used to convert a string iwith commas forlarge numbers into a numeric value. For example

data popsize;

input population $9.;

pop=input(population,comma9.);

datalines;

2,115,353

;

run;

It is often convenient to be able go back and forth between thinking of avalue as either a string or a number.


Earthquake counter


Earthquake counter


More on First and Last observations for a given value










Example: average of top three salaries

For one retirement plan at UNM, employees are given a proportion of theaverage of their top three years of salary, where the proportion depends onthe number of years of service. Here we want to determine which threeyears have the highest salary for each employee and get the average salaryfor those three years. How can you do this in SAS?



A bit tricky: need a RETAIN so that high3 is not missing.



After sorting on ID and SALARY (not on year), we want a counter to rankthe salaries for each ID.



Here we used a subsetting IF to only include the observation with theaverage of the top three salaries.



How could we have done this if salaries were in ascending order instead?In this case you could use LAST.ID to determine the highest salary. Youcould also LAG1 and LAG2 to determine previous salaries (second andthird highest).You’d have some code that looked something like this

salary1 = lag1(salary);

salary2 = lag2(salary);

highest3 = mean(salary,salary1,salary2);

if last.id;

This code also assumes that each individual has at least three years worthof salary, which might be a reasonable assumption if the person qualifiesfor retirement, but still something to be careful about.




A caution about LAG functions

Use of LAGS can be very useful but can be a tricky when they are usedconditionally (in IF or WHILE statements). The safest way of using themis to create a variable which is the lagged variable, and then do moresophisticated stuff with them. The following code generates incorrectoutput (but not errors)


Imitating SAS in R

The following code imitates the first.id and last.id variables in SAS, and Iuse it on the cars data set.


Imitating SAS in R


Imitating SAS in R


DO loops

An important part of programming is loops. SAS automatically loopsthrough datasets that are read in and processes each observation one at atime. However, sometimes you need to control loops yourself, create loopswithout reading in data, loop through subsets of a dataset, and so forth.


DO loops

In the previous example, the data step generated simulated normal randomvariables and outputted them to the dataset called n01–because it wasNormal(0,1) values. The value in parentheses is the random number seed.You will get different values depending on the seed, which allows you toreproduce results or generate new results.

The syntax requires that the DO loop be closed with END; which you canthink of as being like a closing parentheses.


DO loops

To get into SAS’s head so to speak, consider the following code and try toanticipate what it does. The data set address.txt has three observations(the addresses used for Homework problem 1.2).


Using DO to create plots

Note that log is the natural log.


Using DO to create plots

This plot is not very good. Axis labels are too small, title is oddly large,and there is no legend. But blue is for odds, and orange is for log-odds.


Using DO with LAG functions

Some recursive problems can be solved using LAG functions within DOloops. An example from Ross, A First Course in Probability Chapter 2, is:

Let fn denote the number of ways of tossing a coint n times such thatsuccessive heads never appear. Argue that

fn = fn−1 + fn−2 n ≥ 2, where f0 ≡ 1, f1 ≡ 2

HINT: How many outcomes are there that start with a head, and howmany that start with a tail?If Pn denotes the probability that successive heads never appear when acoin is tossed n times, find Pn (in terms of fn) when all possible outcomesof the n tosses are equally likely. Compute P10.



The recursion fn = fn−1 + fn−2 works for the first term because if the firsttoss is tails, then we have n − 1 tosses without restrictions, and there arefn−1 ways to not get successive heads with the remaining tosses. If the firsttoss is heads, the second toss must be tails (if n > 2), so there are nown − 2 tosses without restrictions, and there are fn−2 ways to accomplishthis. The probability of not getting successive heads in n tosses is thereforePn = fn/2n, since each of the 2n sequences of tosses is equally likely.

To compute this, we use a LAG function within a DO loop.







This code doesn’t work, and the problem seems to be putting the LAGfunction in an IF statement.


DO WHILE

You can also DO something conditionally until you have the desirednumber of successes or observations. Suppose I want to simulate normalrandom variables conditional on having values greater than 3. I might wantto use this to get, say the median of X |X > 3, which is not easy to dealwith analytically. The following accomplishes this, although it is somewhatinefficient. To do this more efficiently, you could use importance sampling.


DO WHILE caution

Because the condition that you require might never occur or might take avery long time to execute, your program could run a VERY long time andappear to be hanging, creating an infinite loop or just very long loop. If wewanted 1000 observations of standard normal variables being larger than 3,we’d need generate an average of 740,000 random numbers. If we wanted1000 normal variables being larger than 6, we’d need to generate anaverage of 1,013,594,635,000 random variables, which probably wouldn’twork, and you might need something more efficient like importancesampling.If you aren’t sure whether something will take too long, you could add acounter and stop after a maximum number of iterations.


DO WHILE testing


DO UNTIL testing

An alternative approach is to use DO UNTIL which executes the code inthe block first, then tests the condition to see whether it should continueto execute. It is mostly a matter of style whether you use DO WHILE orDO UNTIL. For both DO WHILE and DO UNTIL, infinite loops are easyto mistakenly creep into your code. A DO WHILE will be an infinite loopif the condition is always true, and a DO UNTIL will be an infinite loop ifthe condition is always false. The following uses a DO UNTIL instead ofDO WHILE to get the same results.


LEAVE and CONTINUE with DO loops

In addition to testing at the top (WHILE) or bottom (UNTIL) of a loop,you can break out of a loop using the LEAVE and CONTINUE statements.With the LEAVE statement, you break out of the DO loop entirely andmove past the END statement. For the CONTINUE statement, you returnto the top of the loop to do the next iteration of the loop. This is analternative to putting extra testing conditions in the WHILE or UNTILconditions.


LEAVE and CONTINUE


CONTINUE

Continue will mostly be useful if you have more complicated loops wherethere is computational savings if you jump to the next iteration once youhave done everything needed. This can also be useful for running anexample with many iterations, but only printing a few of them. In Markovchain Monte Carlo experiments for example, you might want to run aMarkov chain for several million iterations, but this is too many to plot.Instead of plotting each iteration, you could just plot every 1000thiteration. This is most easily achieved with the MODULUS function,mod(), which returns the remainder when divided by the modulus.The modulus function can also be useful for separating a dataset into itsodd and even observations numbers, or some other unusual grouping.


CONTINUE example with modulus


MODULUS example


MODULUS operator to splice up a dataset


Another way to do the phone data...


Application of modulus to longitude data

The New Zealand data includes longitudinal coordinates on both sides ofthe international date line, although they are primarily to the the West ofit. Some of the longitude values are recorded as negative, so you mighthave one earthquake with longitude 179 (meaning 179 degrees east ofGreenwich) and another with longitude -179 (meaning 179 degrees west ofGreenwich), but really there are only two degrees of longitude separatingthem. On a scatterplot, it would make sense to have them close together.

How can we fix this problem?


Application of modulus to longitude data: no modulus

This data is for eventtype = earthquake only


Application of modulus to longitude data: with modulus

When I first read in the data, I added the line in the datasteplongitude = mod(longitude+360,360); This doesn’t effect positivelongitudes, but allows something with a longitude of -179 to have alongitude of 181, which is where we want it.


Earthquake data in geonet.org - math.unm.edu

Documents