Transcript
12.04.23 - Page 1
DépartementOffice
DBIx::Classvs.
DBIx::DataModel
FPW 2012, Strasbourg
laurent.dami@justice.ge.chDépartement
Office
Agenda
• Introduction• Schema definition• Data retrieval• Joins• Classes and methods• Resultset/Statement objects• Customization• Conclusion
Introduction
ORM layer
Database
DBD driver
DBI
Object-Relational Mapper
Perl program
ORM useful for …
• dynamic SQL generation– navigation between tables– generate complex SQL queries from Perl datastructures– better than phrasebook or string concatenation
• automatic data conversions (inflation / deflation)• transaction encapsulation • data validation• computed fields• caching• expansion of tree data structures coded in the relational
model• …
CPAN ORM Landscape
• Discussed here– DBIx::Class (a.k.a. DBIC)– DBIx::DataModel (a.k.a. DBIDM)
• Many other– Rose::DB, Jifty::DBI, Fey::ORM, ORM,
DBIx::ORM::Declarative, Tangram, Coat::Persistent, ORLite,DBR, DBIx::Sunny, DBIx::Skinny, DBI::Easy, …
DISCLAIMER- I'm not an expert of DBIC- I'll try to be neutral in comparisons, but …- Won't cover all topics
A bit of 2005 historyClass::DBI (1999)
Class::DBI::Sweet (29.05.05)- SQL::Abstract
DBIx::DataModel 0.10 (16.09.05)
SQL::Abstract (2001)
DBIx::Class 0.01 (08.08.05)- SQL::Abstract
DBIx::Class 0.03 (19.09.05)- prefetch joins
Class::DBI::Sweet 0.02 (29.06.05)-prefetch joins
DBIx::DataModel private (feb.05)
Some figures
• DBIC– thousands of users– dozens of contributors– 162 files– 167 packages– 1042 subs/methods– 16759 lines of Perl– 1817 comment lines– 19589 lines of POD– 70 transitive dependencies
• DBIDM– a couple of users– 1 contributor– 31 files– 40 packages– 175 subs/methods– 3098 lines of Perl– 613 comment lines– 8146 lines of POD– 54 transitive dependencies
straight from CPAN::SQLite
Our example : CPAN model
Author
Distribution Module
1
*
1 1..*
* *
contains ►
depends_on ►
12.04.23 - Page 1
DépartementOffice
Schema definition
12.04.23 - Page 1
DépartementOffice
DBIC Schema class
use utf8;package FPW12::DBIC;use strict;use warnings;
use base 'DBIx::Class::Schema';
__PACKAGE__->load_namespaces;
1;
12.04.23 - Page 1
DépartementOffice
DBIC 'Dist' table class
use utf8;package FPW12::DBIC::Result::Dist;use strict; use warnings;use base 'DBIx::Class::Core';
__PACKAGE__->table("dists");__PACKAGE__->add_columns(
dist_id => { data_type => "integer", is_nullable => 0, is_auto_increment => 1 },
dist_vers => { data_type => "varchar", is_nullable => 1, size => 20 }, ... # other columns );__PACKAGE__->set_primary_key("dist_id");__PACKAGE__->belongs_to ('auth', 'FPW12::DBIC::Result::Auth', 'auth_id');__PACKAGE__->has_many ('mods', 'FPW12::DBIC::Result::Mod', 'dist_id');__PACKAGE__->has_many ('depends', 'FPW12::DBIC::Result::Depends', 'dist_id');__PACKAGE__->many_to_many(prereq_mods => 'depends', 'mod');
# idem for classes Auth, Mod, etc.
DBIDM centralized declarations
use DBIx::DataModel;DBIx::DataModel->Schema("FPW12::DBIDM")# class table primary key # ===== ===== ===========->Table(qw/Auth auths auth_id /)->Table(qw/Dist dists dist_id /)->Table(qw/Mod mods mod_id /)->Table(qw/Depends depends dist_id mod_id/)# class role multipl. (optional join keys) # ===== ==== ======= ====================->Association([qw/Auth auth 1 auth_id /], [qw/Dist dists * auth_id /])->Composition([qw/Dist dist 1 /], [qw/Mod mods * /])->Association([qw/Dist dist 1 /], [qw/Depends depends * /])->Association([qw/Mod mod 1 /], [qw/Depends depends * /])->Association([qw/Dist used_in_dist * depends distrib /], [qw/Mod prereq_mods * depends mod /]);
12.04.23 - Page 1
DépartementOffice
Schema def. comparison
• DBIC– one file for each class– regular Perl classes– full column info– 1-way "relationships"
• belongs_to• has_many• may_have• ….
– custom join conditions• any operator• 'foreign' and 'self'
• DBIDM– centralized file– dynamic class creation – no column info (except pkey)– 2-ways "associations" with
• multiplicities• role names• diff association/composition
– custom column names• only equalities
12.04.23 - Page 1
DépartementOffice
Data retrieval
12.04.23 - Page 1
DépartementOffice
DBIC: Search
use FPW12::DBIC;
my $schema = FPW12::DBIC->connect($data_source);
my @dists = $schema->resultset('Dist')->search( {dist_name => {-like => 'DBIx%'}}, {columns => [qw/dist_name dist_vers/]},);
foreach my $dist (@dists) { printf "%s (%s)\n", $dist->dist_name, $dist->dist_vers;}
12.04.23 - Page 1
DépartementOffice
DBIDM: Search
use FPW12::DBIDM;use DBI;
my $datasource = "dbi:SQLite:dbname=../cpandb.sql";my $dbh = DBI->connect($datasource, "", "",
{RaiseError => 1, AutoCommit => 1});FPW12::DBIDM->dbh($dbh);
my $dists = FPW12::DBIDM::Dist->select( -columns => [qw/dist_name dist_vers/], -where => {dist_name => {-like => 'DBIx%'}},);
foreach my $dist (@$dists) { print "$dist->{dist_name} ($dist->{dist_vers})\n";}
12.04.23 - Page 1
DépartementOffice
Simple search comparison
• DBIC– schema is an object– result is a list or a resultset
object– accessor methods for
columns– uses SQL::Abstract
• mostly hidden
• DBIDM– schema is a class (default) or an
object– result is an arrayref (default)– hash entries for columns– uses SQL::Abstract::More
• user can supply a custom $sqla obj
12.04.23 - Page 1
DépartementOffice
SQL::Abstract new()
• special operators
– ex: DBMS-independent fulltext search
# where {field => {-contains => [qw/foo bar/]}
my $sqla = SQL::Abstract->new(special_ops => [ {regex => qr/^contains(:?_all|_any)?$/i, handler => sub { my ($self, $field, $op, $arg) = @_; my $connector = ($op =~ /any$/) ? ' | ' : ' & '; my @vals = ref $arg ? @$arg : ($arg); @vals = map { split /\s+/ } grep {$_} @vals; my $sql = sprintf "CONTAINS($field, '%s') > 0",
join $connector, @vals; return ($sql); # no @bind
}]);
12.04.23 - Page 1
DépartementOffice
DBIC: Find single record
my $auth1 = $schema->resultset('Auth')->find(123);
my $auth2 = $schema->resultset('Auth') ->find({cpanid => 'DAMI'});
12.04.23 - Page 1
DépartementOffice
DBIDM: Find single record
my $auth1 = FPW12::DBIDM::Auth->fetch(123);
my $auth2 = FPW12::DBIDM::Auth->search( -where => {cpanid => 'DAMI'}, -result_as => 'firstrow',);
12.04.23 - Page 1
DépartementOffice
DBIC: Search args
• 1st arg : "where" criteria• 2nd arg : attributes
– distinct– order_by– group_by, having– columns # list of columns to retrieve from main table– +columns # additional columns from joined tables or
from functions – join # see later– prefetch # see later– for– page, rows, offset, etc.– cache
DBIDM: Select args
my $result = $source->select( -columns => \@columns,, -where => \%where, -group_by => \@groupings, -having => \%criteria, -order_by => \@order, -for => 'read only', -post_SQL => sub { … }, -pre_exec => sub { … }, -post_exec => sub { … }, -post_bless => sub { … }, -page_size => …, -page_index => …, -limit => …, -offset => …, -column_types => \%types, -result_as => $result_type,);
DBIDM: Polymorphic result
-result_as =>– 'rows' (default) : arrayref of row objects– 'firstrow' : a single row object (or undef)– 'hashref' : hashref keyed by primary keys– [hashref => @cols] : cascaded hashref– 'flat_arrayref' : flattened values from each row– 'statement' : a statement object (iterator)– 'fast_statement' : statement reusing same memory– 'sth' : DBI statement handle– 'sql' : ($sql, @bind_values)– 'subquery' : \["($sql)", @bind]
don't need method variants : select_hashref(), select_arrayref(), etc.
DBIDM: Fast statement
• like a regular statement– but reuses the same memory location for each row– see DBI::bind_col()
my $statement = $source->select( . . . , -result_as => 'fast_statement');
while (my $row = $statement->next) { . . . # DO THIS : print $row->{col1}, $row->{col2} # BUT DON'T DO THIS : push @results, $row;}
Advanced search comparison
• DBIC– 'find' by any unique
constraint– "inner" vs "outer" columns– result is context-dependent– optional caching
• DBIDM– 'fetch' by primary key only– all columns equal citizens– polymorphic result– no caching– statement-specific inflators– callbacks
Joins
Task : list distributions and their authors
Author
Distribution Module
1
*
1 1..*
* *
contains ►
depends_on ►
DBIC: Join
my @dists = $schema->resultset('Dist')->search( {dist_name => {-like => 'DBIx%'}}, {columns => [qw/dist_name dist_vers/], join => 'auth',
'+columns' => [{'fullname' => 'auth.fullname'}], },);
foreach my $dist (@dists) { printf "%s (%s) by %s\n",
$dist->dist_name, $dist->dist_vers, $dist->get_column('fullname'); # no accessor meth}
DBIC: Join with prefetch
my @dists = $schema->resultset('Dist')->search( {dist_name => {-like => 'DBIx%'}}, {columns => [qw/dist_name dist_vers/], prefetch => 'auth', },);
foreach my $dist (@dists) { printf "%s (%s) by %s\n",
$dist->dist_name, $dist->dist_vers, $dist->auth->fullname; # already in memory}
DBIDM: Join
my $rows = FPW12::DBIDM->join(qw/Dist auth/)->select( -columns => [qw/dist_name dist_vers fullname/], -where => {dist_name => {-like => 'DBIx%'}},);
foreach my $row (@$rows) { print "$row->{dist_name} ($row->{dist_vers}) " . "by $row->{fullname}\n";}
Dist Auth
..::AutoJoin::…
DBIDM::Source::Join
new class created on the fly
Multiple joins
• DBIC
join => [ { abc => { def => 'ghi' } }, { jkl => 'mno' },
'pqr' ]
• DBIDM
->join(qw/Root abc def ghi jkl mno pqr/)
Join from an existing record
• DBIC
my $rs = $auth->dists(undef, {join|prefetch => 'mods'});
• DBIDM
my $rows = $auth->join(qw/dists mods/)->select;
DBIC: left/inner joins
• attribute when declaring relationships# in a Book class (where Author has_many Books)
__PACKAGE__->belongs_to( author => 'My::DBIC::Schema::Author', 'author', { join_type => 'left' } );
• cannot change later (when invoking the relationship)
DBIDM: Left / inner joins
->Association([qw/Author author 1 /], [qw/Distribution distribs 0..* /])
# default : LEFT OUTER JOIN
->Composition([qw/Distribution distrib 1 /], [qw/Module modules 1..* /]);
# default : INNER JOIN
# but defaults can be overriddenMy::DB->join([qw/Author <=> distribs/)-> . . . My::DB->join([qw/Distribution => modules /)-> . . .
Join comparison
• DBIC– rows belong to 1 table class
only– joined columns are either
• "side-products", • prefetched (all of them, no
choice)
– fixed join type
• DBIDM– rows inherit from all joined
tables– flattening of all columns
– default join type, can be overridden
Speed
12.04.23 - Page 1
DépartementOffice
Select speed
list mods(109349 rows)
join Auth dist mods(113895 rows)
DBI 0.43 secs 1.36 secs
DBIC regular 11.09 secs 15.50 secs
DBIC prefetch n.a. 146.29 secs
DBIC hashref inflator
10.06 secs 14.17 secs
DBIDM regular 4.00 secs 5.01 secs
DBIDM fast_statement
2.25 secs 3.28 secs
12.04.23 - Page 1
DépartementOffice
Insert speed
insert (72993 rows)
DBI 5.8 secs
DBIC regular 40.35 secs
DBIC populate 5.6 secs
DBIDM regular
32.81 secs
DBIDM bulk 32.57 secs
12.04.23 - Page 1
DépartementOffice
Classes and methods
DBIC : methods of a row
DB<1> m $auth # 152 methods_auth_id_accessor_cpanid_accessor_email_accessor_fullname_accessor_result_source_instance_accessoradd_to_distsauth_idcpaniddistsdists_rsemailfullnameresult_source_instancevia DBIx::Class::Core -> DBIx::Class::Relationship ->
DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasMany: has_many
via DBIx::Class::Core -> DBIx::Class::Relationship -> DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasOne: _get_primary_key
via DBIx::Class::Core -> DBIx::Class::Relationship -> DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasOne: _has_one
via DBIx::Class::Core -> DBIx::Class::Relationship -> DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasOne: _validate_has_one_condition
via DBIx::Class::Core -> DBIx::Class::Relationship -> DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasOne: has_one
...
DB<4> x mro::get_linear_isa(ref $auth)0 ARRAY(0x13826c4) 0 'FPW12::DBIC::Result::Auth' 1 'DBIx::Class::Core' 2 'DBIx::Class::Relationship' 3 'DBIx::Class::Relationship::Helpers' 4 'DBIx::Class::Relationship::HasMany' 5 'DBIx::Class::Relationship::HasOne' 6 'DBIx::Class::Relationship::BelongsTo' 7 'DBIx::Class::Relationship::ManyToMany' 8 'DBIx::Class' 9 'DBIx::Class::Componentised' 10 'Class::C3::Componentised' 11 'DBIx::Class::AccessorGroup' 12 'Class::Accessor::Grouped' 13 'DBIx::Class::Relationship::Accessor' 14 'DBIx::Class::Relationship::CascadeActions' 15 'DBIx::Class::Relationship::ProxyMethods' 16 'DBIx::Class::Relationship::Base' 17 'DBIx::Class::InflateColumn' 18 'DBIx::Class::Row' 19 'DBIx::Class::PK::Auto' 20 'DBIx::Class::PK' 21 'DBIx::Class::ResultSourceProxy::Table' 22 'DBIx::Class::ResultSourceProxy'
DBIDM : methods of a row
DB<1> m $auth # 36 methodsdistsinsert_into_distsmetadmvia DBIx::DataModel::Source::Table:
_get_last_insert_idvia DBIx::DataModel::Source::Table:
_insert_subtreesvia DBIx::DataModel::Source::Table: _rawInsertvia DBIx::DataModel::Source::Table: _singleInsertvia DBIx::DataModel::Source::Table:
_weed_out_subtreesvia DBIx::DataModel::Source::Table: deletevia DBIx::DataModel::Source::Table:
has_invalid_columnsvia DBIx::DataModel::Source::Table: insertvia DBIx::DataModel::Source::Table: updatevia DBIx::DataModel::Source::Table ->
DBIx::DataModel::Source: TO_JSONvia DBIx::DataModel::Source::Table ->
DBIx::DataModel::Source: apply_column_handlervia DBIx::DataModel::Source::Table ->
DBIx::DataModel::Source: auto_expandvia DBIx::DataModel::Source::Table ->
DBIx::DataModel::Source: bless_from_DBvia DBIx::DataModel::Source::Table ->
DBIx::DataModel::Source: expandvia DBIx::DataModel::Source::Table ->
DBIx::DataModel::Source: fetch
DB<4> x mro::get_linear_isa(ref $auth)
0 ARRAY(0x145275c)
0 'FPW12::DBIDM::Auth'
1 'DBIx::DataModel::Source::Table'
2 'DBIx::DataModel::Source'
BelongsTo, HasMany, etc.
Schema Core ResultSet
My::DB My::DB::Result::Table_n
row resultset
DBIC classes
application classes
objects
schema
Class
Row
Componentised AccessorGroup
My::DB::ResultSet::Table_n
Relationship InflateColumn PK::Auto PK RSProxy::Table
BelongsTo, HasMany, etc.
RSProxyRLHelpers RLBase ResultSource
RS::Table
inheritanceinstantiation delegation
Schema
Source
Table Join Statement
My::DB My::DB::Table_n
My::DB::AutoJoin::
row statementrow
DBIDM classes
applicationclasses
objects
schema
DBIDM Meta-Architecture
Schema Table Join
My::DB
My::DB::Table_n
My::DB::Auto_join
Meta::Source
Meta::Table Meta::Join
meta::table
meta::join
meta::schema
Meta::Schema Meta::PathMeta::Association Meta::Type
meta::assoc meta::path
meta::type
DBIC introspection methods
• Schema– sources(), source($source_name)
• ResultSource– primary_key– columns(), column_info($column_name)– relationships(), relationship_info($relname)
Architecture comparison
• DBIC– very complex class structure– no distinction front/meta
layser– methods for
• CRUD• navigation to related objs• introspection• setting up columns• setting up relationships• …
• DBIDM– classes quite close to DBI
concepts– front classes and Meta
classes– methods for
• CRUD• navigation to related objs• access to meta
12.04.23 - Page 1
DépartementOffice
Resultset/Statement objects
12.04.23 - Page 1
DépartementOffice
Example task
• list names of authors – of distribs starting with 'DBIx' – and version number > 2
12.04.23 - Page 1
DépartementOffice
DBIC ResultSet chaining
my $dists = $schema->resultset('Dist')->search( {dist_name => {-like => 'DBIx%'}},);my $big_vers = $dists->search({dist_vers => { ">" => 2}});my @auths = $big_vers->search_related('auth', undef,
{distinct => 1, order_by => 'fullname'});
say $_->fullname foreach @auths;
# Magical join ! Magical "group by" !SELECT auth.auth_id, auth.email, auth.fullname, auth.cpanidFROM dists me JOIN auths auth ON auth.auth_id = me.auth_id WHERE ( ( dist_vers > ? AND dist_name LIKE ? ) ) GROUP BY auth.auth_id, auth.email, auth.fullname, auth.cpanid ORDER BY fullname
DBIDM Statement lifecycle
new
sqlized
prepared
executed
schema + source
data row(s)
new()
sqlize()
prepare()
execute()
bind()refine()
bind()
bind()
bind()execute()
next() / all()
blessedcolumn types applied
-post_bless
-pre_exec
-post_exec
-post_SQL
DBIDM Statement::refine()
my $stmt = FPW12::DBIDM->join(qw/Dist auth/)->select( -where => {dist_name => {-like => 'DBIx%'}}, -result_as => 'statement',);
$stmt->refine( -columns => [-DISTINCT => 'fullname'], -where => {dist_vers => { ">" => 0}}, -order_by => 'fullname',);
say $_->{fullname} while $_ = $stmt->next;
DBIDM subquery
my $stmt = FPW12::DBIDM::Dist->select( -where => {dist_name => {-like => 'DBIx%'}}, -result_as => 'statement',);
my $subquery = $stmt->select( -columns => 'auth_id', -where => {dist_vers => { ">" => 2}}, -result_as => 'subquery',);
my $auths = FPW12::DBIDM::Auth->select( -columns => 'fullname', -where => {auth_id => {-in => $subquery}},);
say $_->{fullname} foreach @$rows;
Statement comparison
• DBIC– powerful refinement
constructs• more "where" criteria• navigation to related source• column restriction• aggregation operators (e.g.
"count")
• DBIDM– limited refinement constructs– explicit control of status
12.04.23 - Page 1
DépartementOffice
Other features
12.04.23 - Page 1
DépartementOffice
Transactions
• DBIC
$schema->txn_do($sub);
– can be nested– can have intermediate
savepoints
• DBIDM
$sch->do_transaction($sub);
– can be nested– no intermediate savepoints
DBIC inflation/deflation
__PACKAGE__->inflate_column('insert_time', {inflate => sub { DateTime::Format::Pg->parse_datetime(shift); }, deflate => sub { DateTime::Format::Pg->format_datetime(shift) }, });
DBIDM Types (inflate/deflate)
# declare a TypeMy::DB->Type(Multivalue => from_DB => sub {$_[0] = [split /;/, $_[0]] }, to_DB => sub {$_[0] = join ";", @$_[0] },);
# apply it to some columns in a tableMy::DB::Author->metadm->define_column_type( Multivalue => qw/hobbies languages/,);
Extending/customizing DBIC
• Subclassing– result classes– resultset classes– storage– sqlmaker
Extending / customizing DBIDM
• Schema hooks for– SQL dialects (join syntax, alias syntax, limit / offset, etc.)– last_insert_id
• Ad hoc subclasses for– SQL::Abstract– Table– Join– Statements
• Statement callbacks• Extending table classes
– additional methods– redefining _singleInsert method
Not covered here
• inserts, updates, deletes• Schema generator• Schema versioning• inflation/deflation
12.04.23 - Page 1
DépartementOffice
THANK YOU FOR YOUR ATTENTION
12.04.23 - Page 1
DépartementOffice
Bonus slides
Why hashref instead of OO accessors ?
• Perl builtin rich API for hashes (keys, values, slices, string interpolation)
• good for import / export in YAML/XML/JSON• easier to follow steps in Perl debugger• faster than OO accessor methods• visually clear distinction between lvalue / rvalue
– my $val = $hashref->{column};– $hashref->{column} = $val;
• visually clear distinction between – $row->{column} / $row->remote_table()
SQL::Abstract::More : extensions
• -columns => [qw/col1|alias1 max(col2)|alias2/]– SELECT col1 AS alias1, max(col2) AS alias2
• -columns => [-DISTINCT => qw/col1 col2 col3/]– SELECT DISTINCT col1, col2, col3
• -order_by => [qw/col1 +col2 –col3/]– SELECT … ORDER BY col1, col2 ASC, col3 DESC
• -for => "update" || "read only"– SELECT … FOR UPDATE
top related