Matlab DIY Lesson 3: Parsing data. Today's Lesson “ parse ” = analyze into its parts o Sort the data you want MATLAB to use o Designate different segments.

Matlab DIY

Lesson 3: Parsing data

Today's Lesson

• “parse”= analyze into its partso Sort the data you want MATLAB to useo Designate different segments of the datao Tell MATLAB to do different things with different

parts of the data

• “batch” = process a group of records as a single unit (one command processes multiple files)

Why Parse?

Let's look at a few example files:

- data output may not be in evenly spaced rows

- you might be interested in a subset of the data and want to separate it from the rest

- you might want to run different analysis on different parts of the data

Your first .m program

During previous lessons you typed individual commands and entered them in the command window

.m allows you to access a series of commands by entering the name of the program in the command window

Open MATLAB set current directoryopen current directory windowdouble click on L3_csvread_pc.m

L3_csvread_pc.m

review commands from previous lessons

Parse option 1: choose rows and columns of data where Matlab begins to read data

Reading in the data file data = csvread([inpath,infilename],1,1);       

tsvreadxlsreaddlmread

These all read in numbers NOT text

0,0 = first cell in upper left corner1,0 = can be used to eliminate header data10,5 = starts reading on 10th line, 5th column

Common csvread errors>> d = csvread([inpath,filename],1,0);??? Error using ==> csvreadFilename must be a string.

Step 1: see what Matlab thinks filename is:

1) type filename in command window2) check class of filename in workspace

Step 2: adjust so that Matlab sees filename as a string

1) redefine filename variable by adding ‘’ or []2) type filename{1} this turns it into a string

d=csvread([inpath,filename{1}],1,0);

Common csvread errors>> u = csvread([inpath,filename{1}],10,0);

??? Error using ==> csvreadFile not found.

Step 1: check the path-is the name what you think it is?-have you been consistent in spelling for

inpath and filename?Step 2: check your data file

-have you used the right name?-should it be in a folder?-have you used the right suffix?

u = csvread([data_inpath,filename{1}],10,0);

Common csvread errors

>> c = csvread([data_inpath,filename{1}],0,0);??? Error using ==> textscanMismatch between file and format string.Trouble reading number from file (row 1, field 1) ==> time,

Step 1: open data file and check for letters in the data

Step 2: adjust to avoid problem area--adjust where Matlab begins reading or--adjust organization of the file

Parse option #2 Create a new smaller array with a subset of data>> c = csvread([data_inpath,filename{1}],1,1);

>> k = csvread([data_inpath,filename{1}],2:8,1:10);??? Error using ==> textscanHeader lines must be a scalar integer.

Error in ==> csvread at 45    m=dlmread(filename, ',', r, c);

>> k = c(2:8,1:10);

Parse option #3: Naming segments of the data

x_pos=data(:,1);y_pos=data(:,2);z_pos=data(:,3);mus1=data(:,4);mus2=data(:,5);mus3=data(:,6);event=data(:,7);time=data(:,8); %notice that we started reading the data at 1,1 so the file data is missing the first row and first column, each of these column numbers have been adjusted accordingly

Now you can perform simple formulas

mn_x=mean(x_pos);

SD_y=std(y_pos);

Two ways to find more info on abbreviations for mathematical functions

1) command window >>help mean2) Go to Matlab help and search for mean

Parse option #4:create program to select a subset of data

onstamp=find(event>4);ontime=onstamp(1); offstamp=find(event<0);offtime=offstamp(1); mv_time=(offtime-ontime)*1/84; %sampling rate =84Hz eyes_open=data(ontime:offtime,:); %this creates a subset of data from rows that begin with event onset and end with event offset, the : means all columns are included

subset of rowsall columns

Printing Data to a File

outfilename=[date,'L3_output.csv'];out_fid=fopen([outpath,outfilename],'w'); fprintf(out_fid,'filename,mn_x,std_y,mv_time,mn_x_EO\n');

fprintf(out_fid,'%s,%s,%s,\n',infilename,mn_x,SD_y,);

%Outputting just a String, useful for Headers. fprintf(out_fid,‘filename,mn_x, SD_y,\n');

%Outputting data from variables. fprintf(out_fid, '%s,%s,%s\n', filename,mn_x, SD_y,); 

Format specifier for how information is to be written, %s= string%d= decimal %f=fixed point decimal

Look these up by typing “ help fprintf”

Values to go in spaces

L3.m

Special Characters you will likely use:• \n = The newline character• \t = The tab character• \\ = The backslash character

 fprintf(out_fid, '%s,%s,%s\n', subj_num,group,AP_RMS); 

fprintf(out_fid, '%s\t%s\t%s\n', subj_num,group,AP_RMS); 

'\n'?

fprintf(out_fid, '%s,%f,%5.3f,,/n', filename, mn_x,MT,);

fprintf(out_fid, '%g,%g,/n', eyes_open(1,:), eyes_open(2,:));

Output per data setOne line vs. multiple lines

Single variable output

entire column of datafrom an array

Run the .m file1) Save any changes to .m program2) Type the name of the program (but not .m) in command window3) Enter4) Did you get any errors?5) Did you get any messages at all?6) Check your workspace7) Are all your variables in place?8) Check your desktop files9) Did Matlab create an output folder?10)Did Matlab create any output files?11)Open the files12)Do they look correct?13)Which number specifier do you want to use? f, g, e, d?

Reading data with textscaninfile=[inpath,infilename]; fid=fopen(infile,'r') ; if fid == -1 [infile,' could not be opened'] returnend d=textscan(fid, '%s', 'delimiter', '\n'); %scans textfile and loads up as a string, delimiter chops it by each new line fclose(fid); %line above read in contents and stored it in "d", d is array of all info line by linetextlines=d{1}; %this puts a wrapper on the data, begins with the first piece that is my data set

Creating a data array with textscan

for p=2:length(textlines) herenow=strread(textlines{p},'%s','delimiter',',');%grabs the text lines and reads the whole line kind of turns it into csv with text) for q=1:length(herenow) big_array((p-1),q)=herenow(q); %(i-1) so it creates the new file from row 1 otherwise have a blank where old labels were end end

Creating data columns with textscan

subj_num = cell2mat(big_array(:,1));

x_pos_t= str2num(cell2mat(big_array(:,2)));

Processing multiple data files

Open L3_combine_batch_textscan.m

This is a template file

It is not set up to work on a current data set

It has multiple inpaths to combine different files

It has multiple subj_nums to allow batch processing

inroute='/Users/saavedra/Desktop/CP_eye_hand/key_variables/';inroute2='/Users/saavedra/Desktop/CP_eye_hand/left_overs/';outroute='/Users/saavedra/Desktop/CP_eye_hand/data_combine/';if exist(outroute)==0 mkdir(outroute);end subj_num={'u11','t04','t19',};%creates an array with 1 row and a cell with a string for each subject name, you must have a folder for each subject for subj_index=1:length(subj_num)%creates an index of all the subjects inpath=cell2mat([inroute,subj_num(subj_index),'/']);% creates a string with subjnum imbedded infilename=cell2mat([subj_num(subj_index),'results.csv']); %creates an infilename with subj_num imbedded

Processing multiple files

• Counting is critical for output files

• Play with some common things that create weirdness in your output file

1) More headers than variables

2) Less headers than variables

3) More % s (or d or f) than variables

4) Put less %s than variables

5) Take the /n off the end of fprintf on data file

Practice making errors

• Open one of your own data files

• Label and sort data

• Keep records of the error messages• Bring a record of errors to class• Bring solution if you found it• IF you did NOT find solution please send the error

to Wayne and Sandy before 10 am Monday morning.

Practice opening files