2012 03 08_dbi

FBW21-02-2008

Wim Van Criekinge

RELOADED2

Three Basic Data Types

• Scalars - $

• Arrays of scalars - @

• Associative arrays of scalers or Hashes - %

• [m]/PATTERN/[g][i][o]

• s/PATTERN/PATTERN/[g][i][e][o]

• tr/PATTERNLIST/PATTERNLIST/[c][d][s]

The ‘structure’ of a Hash

• An array looks something like this:

• A hash looks something like this:

@array =Index

Value

0 1 2

'val1' 'val2' 'val3'

Rob Matt Joe_A

353-7236 353-7122 555-1212

Key (name)

Value%phone =

$a=5;$b=9;$sum=Optellen(5,9);print "The SUM is $sum\n";

sub Optellen(){ $d=@_[0];

$e=@_[1]; #alternatively we could do this: my($a,$b)=@_;

my($answer)=$d+$e; return $answer;}

Sub routine

Overview

• Advanced data structures in Perl

• Object-oriented Programming in Perl

• Bioperl: is a large collection of Perl software for bioinformatics

• Motivation:– Simple extension: “Multiline parsing“

more difficult than expected

• Goal: to make software modular, easier to maintain, more reliable, and easier to reuse

Multi-line parsing

use strict;use Bio::SeqIO;

my $filename="sw.txt";my $sequence_object;

my $seqio = Bio::SeqIO -> new ( '-format' => 'swiss', '-file' => $filename );

while ($sequence_object = $seqio -> next_seq) {my $sequentie = $sequence_object-> seq(); print $sequentie."\n";}

Perl 00

• A class is a package

• An object is a reference to a data structure (usually a hash) in a class

• A method is a subroutine in the class

Perl Classes

• Modules/Packages – A Perl module is a file that uses a package

declaration

– Packages provide a separate namespace for different parts of program

– A namespace protects the variable of one part of a program from unwanted modification by another part of the program

– The module must always have a last line that evaluates to true, e.g. 1;

– The module must be in “known” directory (environment variable)

• Eg … site/lib/bio/Sequentie.pm

Installation on Windows (ActiveState)

• Using PPM shell to install BioPerl– Get the number of the BioPerl repository:

– PPM>repository

– Set the BioPerl repository, find BioPerl, install BioPerl:

– PPM>repository set <BioPerl repository number>– PPM>search * – PPM>install <BioPerl package number>

• Download BioPerl in archive form from – http://www.BioPerl.org/Core/Latest/index.shtml– Use winzip to uncompress and install

Directory Structure

• BioPerl directory structure organization:– Bio/ BioPerl modules

– models/ UML for BioPerl classes

– t/ Perl built-in tests

– t/data/ Data files used for the tests

– scripts/ Reusable scripts that use BioPerl

– scripts/contributed/ Contributed scripts not necessarily integrated into BioPerl.

– doc/ "How To" files and the FAQ as XML

Live.pl

#!e:\Perl\bin\perl.exe -w# script for looping over genbank entries, printing out nameuse Bio::DB::Genbank;use Data::Dumper;

$gb = new Bio::DB::GenBank();

$sequence_object = $gb->get_Seq_by_id('MUSIGHBA1');print Dumper ($sequence_object);

$seq1_id = $sequence_object->display_id();$seq1_s = $sequence_object->seq();print "seq1 display id is $seq1_id \n";print "seq1 sequence is $seq1_s \n";

File converter

#!/opt/perl/bin/perl -w

#genbank_to_fasta.pl

use Bio::SeqIO;

my $input = Bio::SeqIO::new->(‘-file’ => $ARGV[0],

‘-format’ => ‘GenBank’);

my $output = Bio::SeqIO::new->(‘-file’ => ‘>output.fasta’,

‘-format’ => ‘Fasta’);

while (my $seq = $input->next_seq()){

$output->write_seq($seq)

}

• Bptutorial.pl

• It includes the written tutorial as well as runnable scripts

• 2 ESSENTIAL TOOLS– Data::Dumper to find out what class your

in– Perl bptutorial (100 Bio::Seq) to find the

available methods for that class

Run Needleman-Wunsch-monte-carlo.pl

– my $MATCH = 1; # +1 for letters that match– my $MISMATCH = -1; # -1 for letters that mismatch– my $GAP = -1; # -1 for any gap

Score (-64)

Score = f($MATCH,$MISMATCH,$GAP)

f ?Implement convergence criteriaStore in DATABASE, make graphs in Excel

Oefening 1

A Guide to MySQL & DBI

Objectives

• Start MySQL and learn how to use the MySQL Reference Manual

• Create a database

• Change (activate) a database

• Create tables using MySQL

• Create and run SQL commands in MySQL

Objectives (continued)

• Identify and use data types to define columns in tables

• Understand and use nulls

• Add rows to tables

• View table data

• Correct errors in a database

Opvolger voor MySQL Front

• MySQL-Front was destijds een van de meest populaire MySQL-management applicaties. Wat PHPMyAdmin voor webapplicaties is, was MySQL-Front dat voor de desktop. Helaas kon /wilde de auteur niet langer doorgaan met het project en werd het project stilgelegd.

• In begin April 2006 heeft de originele auteur besloten om de laatste broncode voor MySQL-Front beschikbaar te maken onder de naam HeidiSQL en de eerste beta is te downloaden vanaf de nieuwe site: http://www.heidisql.com .

Starting MySQL

• Windows XP– Click Start button

– Point to All Programs

– Point to MySQL on menu

– Point to MySQL Server 4.1

– Click MySQL Command Line Client • Must enter password in Command Line Client

window

Obtaining Help in MySQL

• Type \h at MySQL> prompt

• Type “help” followed by name of command

– help contents

– help union

Creating a Database

• Must create a database before creating tables

• Use CREATE DATABASE command

• Include database name

Creating a Database (continued)

Changing the Default Database

• Default database: database to which all subsequent commands pertain

• USE command, followed by database name:– Changes the default database

– Execute at the start of every session

Creating a Table

• Describe the layout of each table in the database

• Use CREATE TABLE command

• TABLE is followed by the table name

• Follow this with the names and data types of the columns in the table

• Data types define type and size of data

Table and Column Name Restrictions

• Names cannot exceed 18 characters

• Must start with a letter

• Can contain letters, numbers, and underscores (_)

• Cannot contain spaces

Creating the REP Table

Entering Commands in MySQL

• Commands are free-format; no rules stating specific words in specific positions

• Press ENTER to move to the next line in a command

• Indicate the end of a command by typing a semicolon

• Commands are not case sensitive

Running SQL Commands

Editing SQL Commands

• Statement history: stores most recently used command

• Editing commands:– Use arrow keys to move up, down, left, and right– Use Ctrl+A to move to beginning of line– Use Ctrl+E to move to end of line– Use Backspace and Delete keys

Errors in SQL Commands

Editing MySQL Commands

• Press Up arrow key to go to top line

• Press Enter key to move to next line if line is correct

• Use Right and Left arrow keys to move to location of error

• Press ENTER key when line is correct

• If Enter is not pressed on a line, line not part of the revised command

Dropping a Table

• Can correct errors by dropping (deleting) a table and starting over

• Useful when table is created before errors are discovered

• Command is followed by the table to be dropped and a semicolon

• Any data in table also deleted

Data Types

• For each table column, type of data must be defined

• Common data types:

– CHAR(n)

– VARCHAR(n)

– DATE

– DECIMAL(p,q)

– INT

– SMALLINT

Nulls

• A special value to represent situation when actual value is not known for a column

• Can specify whether to allow nulls in the individual columns

• Should not allow nulls for primary key columns

Implementation of Nulls

• Use NOT NULL clause in CREATE TABLE command to exclude the use of nulls in a column

• Default is to allow null values

• If a column is defined as NOT NULL, system will reject any attempt to store a null value there

Adding Rows to a Table

• INSERT command:

– INSERT INTO followed by table name

– VALUES command followed by specific values in parentheses

– Values for character columns in single quotation marks

The Insert Command

Modifying the INSERT Command

• To add new rows modify previous INSERT command

• Use same editing techniques as those used to correct errors

Adding Additional Rows

The INSERT Command with Nulls

• Use a special format of INSERT command to enter a null value in a table

• Identify the names of the columns that accept non-null values, then list only the non-null values after the VALUES command

The INSERT Command with Nulls

• Enter only non-null values• Precisely indicate values you are entering by listing

the columns

The INSERT Command with Nulls (continued)

Viewing Table Data

• Use SELECT command to display all the rows and columns in a table

• SELECT * FROM followed by the name of the table

• Ends with a semicolon

Viewing Table Data (continued)

Viewing Table Data (continued)

Correcting Errors In the Database

• UPDATE command is used to update a value in a table

• DELETE command allows you to delete a record

• INSERT command allows you to add a record

Correcting Errors in the Database

• UPDATE: change the value in a table• DELETE: delete a row from a table

Correcting Errors in the Database (continued)

Correcting Errors in the Database (continued)

Saving SQL Commands

• Allows you to use commands again without retyping

• Different methods for each SQL implementation you are using

– Oracle SQL*Plus and SQL*Plus Worksheet use a script file

– Access saves queries as objects

– MySQL uses an editor to save text files

Saving SQL Commands

• Script file:– File containing SQL commands

– Use a text editor or word processor to create

– Save with a .txt file name extension

– Run in MySQL:• SOURCE file name• \. file name

– Include full path if file is in folder other than default

Creating the Remaining Database Tables

• Execute appropriate CREATE TABLE and INSERT commands

• Save these commands to a secondary storage device

Describing a Table

Summary

• Use MySQL Command Line Client window to enter commands

• Type \h or help to obtain help at the mysql> prompt• Use MySQL Reference Manual for more detailed

help

Summary (continued)

• Use the CREATE DATABASE command to create a database

• Use the USE command to change the default database

• Use the CREATE TABLE command to create tables

• Use the DROP TABLE command to delete a table

Summary (continued)

• CHAR, VARCHAR, DATE, DECIMAL, INT and SMALLINT data types

• Use INSERT command to add rows• Use NOT Null clause to identify columns that cannot

have a null value• Use SELECT command to view data in a table

Summary (continued)

• Use UPDATE command to change the value in a column

• Use DELETE command to delete a row• Use SHOW COLUMNS command to display a

table’s structure

• DBI

• use DBI;

• my $dbh = DBI->connect( 'dbi:mysql:guestdb',• 'root',• '',• ) || die "Database connection not made: $DBI::errstr";

• $sth = $dbh->prepare('SELECT * FROM demo');• $sth->execute();• while (my @row = $sth->fetchrow_array) {

• print join(":",@row),"\n";• }• $sth->finish();

• $dbh->disconnect();

The Players

• Perl – a programming language

• DBMS – software to manage datat storage

• SQL – a language to talk to a DBMS

• DBI – Perl extensions to send SQL to a DBMS

• DBD – software DBI uses for specific DBMSs

• $dbh – a DBI object for course-grained access

• $sth – a DBI object for fine-grained access

• What is DBI ?

• DBI is a DataBase Interface– It is the way Perl talks to Databases

• DBI is a module by Tim Bunce

• DBI is a community of modules & developers

• What is an interface ?

• The overlap where two phenomeba affect each other

• A point at which independent systems interact

• A boundary across which two systems communicate

• A Sample Interface (the bedrock of DBI)

Fred Wilma

Bone

Dino

• Characteristics of the DINO interface

• Separation of knowledge– Fred doesn’t need to know how to find Wilma– Dino doesn’t need to know how to read

• Generalizability– Fred can send any message– Fred can communicate with anyone

• The DBI interface

Perl DBMS

SQL

DBI

• Characteristics of the DBI interface

• Separation of knowledge– You don’t need to know how to connect– DBI doesn’t need to know SQL

• Generalizeability– You can send any SQL– You can communicate with any DBMS

• The ingredients of a DBI App– 1: A perl script that uses DBI– 2: A DBMS– 3: SQL statements

Outline of a basic DBI script

Set the Perl Environment

Connect to a DBMS

Perform data-affecting SQL instructions

Perform data-returning SQL requests

Disconnect from the DBMS

• $dbh = DataBase Handle

• Done by DBI– Connect

• Done by $dbh, The Database Handle– Perform SQL instructions– Perform SQL request– Disconnect

• Set the Perl Environment– use warnings;– use strict;– Use DBI;

• Connect to a DBMS

my $dbh = DBI -> connect (‘dbi:DBM:’)

$dbh is a Database Handke

An object created by DBI to handle access to this specific connection

• Perform data-affecting Instructions

• $dbh->do($sql_string);

• $dbh->do(“ INSERT INTO geography VALUES (‘Nepal’,’Asia’)” );

• Perform data-returning requests

• My @row = $dbh->selectrow_array($sql_string)

• Disconnect from DBMS

• $dbh->disconnect()

A complete script

• use strict;• use warnings;• use DBI;

• my $dbh=DBI->connect("dbi:mysql:test","root","");• $dbh->do("CREATE TABLE geography (country Text, region

Text)");• $dbh->do("INSERT INTO geography VALUES

('Nepal','Asia')");• $dbh->do("INSERT INTO geography VALUES

('Portugal','Europe')");• print $dbh->selectrow_array("SELECT * FROM geography");• $dbh->disconnect

• The script output

• Only one row

• No seperation of the fields

• No metadata

• Improvements

• DBI– Connect to DBMS– Creates a database handle ($dbh)

• $dbh– Provides course-grained access to the DBMS– Creates a statement handle ($sth)

• $sth– Provides fine-grained access to the DBMS

• Life-cycle of a statement handle ($sth)

• Prepare– Creates the handle, sends SQL to the DBMS to

be analyzed and optimized

• Execute– Instructs the DBMS to perform operations

• Fetch– Brings data from the DBMS into a script

• Life-cycle of a statement handle ($sth)

• My $sth = $dbh->prepare($sql_string);

• $sth->execute();

• Print $sth->fetchrow_array();

• Fecthing rows in a loop – the snippet

• My $sth=$dbh->prepare(“SELECT * FROM geography”);

• $sth->execute();

• While (my @row = $sth->fetchrow_array){

• Print join(“:”,@row),”\n”;

• }

• Output– Nepal:Asia– Portugal:Europe

• All data retrieved

• Colums seperated

• Rows seperated

• Still no metadata

• Finding Metadata – Handle Attributes

• $handle->{$key}=$value;

• Print $handle->{$key};

• $dbh->{RaiseError}=1;

• Print $dbh->{RaiseError};

• My $column_names = $sth->{NAME};

• Finding Metadata with $sth->{NAME}

• my $sth=$dbh->prepare(“SELECT * FROM geography”);

• $sth->execute();

• my @column_names=@{$sth->{NAME}};

• my $num_cols = scaler @column_names;

• print join “:”,@column_names;

• print “(there are $num_cols columns)”;

• Errors

• $dbh->do (“Junk”);

• Print “I Got here!”;

• Checking Errors with RaiseError

• my $dbh=DBI->connect >..

• $dbh->{RaiseError}=1;

• $dbh->do(“Junk”);

• Print “Here ?”;

Number of rows affected

$rows=$dbh->do(“DELETE FROM user WHERE age <42”);

# undef = error

# 3 = 3 rows affected

# 0E0 = no error; no rows affected

# -1 = unknown

• Summary so far

• DBI connect($data_source)

• $dbh do($sql_instruction)• Prepare ($sql_request)• Disconnect()• {RaiseError}

• $sth execute()– Fetchrow_array()– {NAMEM}

• A Deeper look at connection

Perl DBI

DBD#1

DBD#2

MySQL

Oracle

• DBDs- Database Drivers

• DRIVER DBMS• DBD::DBM DBM• DBD::Pg postgreSQL• DBD::mysql MySQL• DBD::Oracle Oracle• DBD::ODBC Ms-Access, MS-SQL-

Server• …

• Variation in DBDs & DBMSs

• Driver-specific connection parameters

• Driver-specific attributes and methods

• SQL implementaion

• Optimization Plans

• Driver-Specific Connection Params – driver name – user name and password

• My $dbh = DBI->connect(• “DBI:$driver:”,• “root”,• “password”;• {• RaiseError => 1,• PrinError => 0,• AutoCommit =>1,• }

• );

Finish() – fetchus interuptus

While (my @row=$sth->fetchrow_array){

Last if $row[0] eq $some_conditions;

}

$sth->finish();

• Alternate fecthes

• My @row=$sth->fetchrow_array();– Print $row[1];

• My @row=$sth->fetchrow_arrayref();– Print $row->[1]

• My @row=$sth->fetchrow_hashref();– Print $row->{region};

• Placeholders !

• my $sth = $dbh -> prepare (“SELECT name from user WHERE country = ? AND city = ? AND age > ?”);

• $sth-> execute(‘Venezuela’,’Caracas’,21);

• DBDs that don’t need a separate DBMS

• DBD::CSV, DBD::Excel

• DBD::Amazon DBD::Google

• use DBI; my $dbh = DBI->connect("dbi:Google:", $KEY); my $sth = $dbh->prepare(qq[ SELECT title, URL FROM google WHERE q = "perl" ]); while (my $r = $sth->fetchrow_hashref) { ...

Step1: Getting DriversEssential for SQL Querying

• A driver is a piece of software that lets your operating system talk to a database– Installed drivers visible in ODBC manager

• “data connectivity” tool

• Each database engine (Oracle, MySQL, etc) requires its own driver– Generally must be installed by user

• Drivers are needed by Data Source Name tool and querying programs

• Require (simple) installation

MySQL Driver: Needed to Query MySQL Databases

• Windows: Download MySQL Connector/ODBC 3.51 here

• Must be installed for direct querying using e.g. Excel – Not necessary if you are using the MySQL

Query Browser

Rat versus mouse RBP

Rat versus bacteriallipocalin

Oefening 2

Fetch a sequence by adapting live.pl and do remote blast using 3 different scoring matrices (summarize results) and perform “controls” using adaptation of shuffle …

Parsing BLAST Using BPlite, BPpsilite, and BPbl2seq

• Similar to Search and SearchIO in basic functionality

• However:– Older and will likely be phased out in the

near future– Substantially limited advanced

functionality compared to Search and SearchIO

– Important to know about because many legacy scripts utilize these objects and either need to be converted

Parse BLAST output

#!/opt/perl/bin/perl -w

#bioperl_blast_parse.pl

# program prints out query, and all hits with scores for each blast result

use Bio::SearchIO;

my $record = Bio::SearchIO->new(-format => ‘blast’, -file => $ARGV[0]);

while (my $result = $record->next_result){

print “>”, $result->query_name, “ “, $result->query_description, “\n”;

my $seen = 0;

while (my $hit = $result->next_hit){

print “\t”, $hit->name, “\t”, $hit->bits, “\t”, $hit->significance, “\n”;$seen++ }

if ($seen == 0 ) { print “No Hits Found\n” }

}

Parse BLAST in a little more detail

#!/opt/perl/bin/perl -w

#bioperl_blast_parse_hsp.pl

# program prints out query, and all hsps with scores for each blast result

use Bio::SearchIO;

my $record = Bio::SearchIO->new(-format => ‘blast’, -file => $ARGV[0]);

while (my $result = $record->next_result){

print “>”, $result->query_name, “ “, $result->query_description, “\n”;

my $seen = 0;

while (my $hit = $result->next_hit{

$seen++;

while (my $hsp = $hit->next_hsp){

print “\t”, $hit->name, “has an HSP with an evalue of: “, $hsp->evalue, “\n”;}

if ($seen == 0 ) { print “No Hits Found\n” }

}

Shuffle

#!/usr/bin/perl -wuse strict;

my ($def, @seq) = <>;print $def;chomp @seq;@seq = split(//, join("", @seq));my $count = 0;while (@seq) {

my $index = rand(@seq);my $base = splice(@seq, $index, 1);print $base;print "\n" if ++$count % 60 == 0;

}print "\n" unless $count %60 == 0;

Searching for Sequence Similarity

• BLAST with BioPerl

• Parsing Blast and FASTA Reports– Search and SearchIO– BPLite, BPpsilite, BPbl2seq

• Parsing HMM Reports

• Standalone BioPerl BLAST

Remote Execution of BLAST

• BioPerl has built in capability of running BLAST jobs remotely using RemoteBlast.pm

• Runs these jobs at NCBI automatically– NCBI has dynamic configurations (server side) to “always” be up and

ready– Automatically updated for new BioPerl Releases

• Convenient for independent researchers who do not have access to huge computing resources

• Quick submission of Blast jobs without tying up local resources (especially if working from standalone workstation)

• Legal Restrictions!!!

Example of Remote Blast

A script to run a remote blast would be something like the following skeleton:

$remote_blast = Bio::Tools::Run::RemoteBlast->new( '-prog' => 'blastp','-data' => 'ecoli','-expect' => '1e-10' );

$r = $remote_blast->submit_blast("t/data/ecolitst.fa"); while (@rids = $remote_blast->each_rid ) { foreach $rid

( @rids ) {$rc = $remote_blast->retrieve_blast($rid);}}

In this example we are running a blastp (pairwise comparison) using the ecoli database and a e-value threshold of 1e-10. The sequences that are being compared are located in the file “t/data/ecolist.fa”.

Example

It is important to note that all command line options that fall under the blastall umbrella are available under BlastRemote.pm.

For example you can change some parameters of the remote job.

Consider the following example:

$Bio::Tools::Run::RemoteBlast::HEADER{'MATRIX_NAME'} = 'BLOSUM25';

This basically allows you to change the matrix used to BLOSUM 25, rather than the default of BLOSUM 62.

Parsing Blast Reports

• One of the strengths of BioPerl is its ability to parse complex data structures. Like a blast report.

• Unfortunately, there is a bit of arcane terminology.

• Also, you have to ‘think like bioperl’, in order to figure out the syntax.

• This next script might get you started

Sample Script to Read and Parse BLAST Report

# Get the report $searchio = new Bio::SearchIO (-format => 'blast', -file => $blast_report);

$result = $searchio->next_result; # Get info about the entire report $result->database_name;

$algorithm_type = $result->algorithm; # get info about the first hit $hit = $result->next_hit; $hit_name = $hit->name ; # get info about the first hsp of the first hit $hsp = $hit->next_hsp; $hsp_start = $hsp->query->start;