Sql abstract from_query

10.04.23 - Page 1

DépartementOfficeSQL::Abstract::FromQu

eryBuilding DB requests from Web queries

YAPC::EU::2014, Sofia

[email protected] de Genève, Pouvoir Judiciaire Département

Office

Power search : the "widget" way

Source: https://rt.cpan.org/Search/Build.html

Power search : the "parsed" way

Name : Smi*Gender : MSalary : > 4000 Job : ! programmer, analyst Birth : BETWEEN 01.01.1970 AND 31.12.1990Address: ! NULL

Agenda

• Background : SQL::Abstract & SQL::Abstract::More

• SQL::Abstract::FromQuery– purpose– API

• internals– Regexp::Grammars– multiple inheritance

Background : SQL::AbstractSQL::Abstract::More

Translating Perl datastructuresinto SQL

SQL::Abstract ('SQLA') : example

my $sqla = SQL::Abstract->new;my ($sql, @bind) = $sqla->select( $table_name, [qw/col1 col2 col3/], { col1 => 'val1', col2 => {-in => [qw/foo bar/]}, col3 => {-between => [qw/2012-01-01 2012-07-01/]}, col4 => {">" => 999}, col5 => {-like => 'foo%'}, }, [qw/col1 col2/],);

# SELECT col1, col2, col3 FROM $table_name# WHERE col1 = ? AND col2 IN (?, ?) AND col3 … etc# ORDER BY col1, col2

Purpose

• generate SQL from Perl data structures– direct from config or from user input– avoid painful string manipulations with regex / join / etc.

• placeholders and bind values– avoid datatype distinctions (strings / numbers /dates / etc.)– avoid SQL injection

used by DBIx::Class, DBIx::DataModel, …

"Special operators" in SQLA

• support for user-defined SQL extensions. For ex : – fulltext index– multivalued columns– case-insensitive– LIKE / GLOB / etc.

• useful for DBMS-independent queries in apps– the "special operator" translates to DBMS-specific syntax

{ col1 => {-contains => [qw/YAPC Sofia/]}, col2 => {-anyval => [12, 34, 56]}, col3 => {-glob => 'foo*'}, }

SQL::Abstract::More ('SQLAM')

• fully compatible with SQLA• named parameters• more SQL constructs (group by, limit/offset, union/intersect, etc.)• more syntax (aliases, joins, ordering direction, etc.)

my $sqla = SQL::Abstract::More->new;my ($sql, @bind) = $sqla->select( -columns => [qw/table1.col|ct1 table2.col|ct2 … /], -from => $table_name, -where => { col1 => 'val1', col2 => {-in => [qw/foo bar/]}, … }, -union => [-columns => …, -from => …, -where => …], -order_by => [qw/+col1 -col2/],);

10.04.23 - Page 1

DépartementOffice

SQLA, SQLAM are for programmers

what about users ?

SQLPerl

datastructure

SQLA, SQLAMHTTPquery

SQLA::FromQuery

HTTP query

Name : Smi*Gender : MSalary : > 4000 Job : ! programmer, analyst Birth : BETWEEN 01.01.1970 AND 31.12.1990Address: ! NULL

Expected SQLA structure

{ Name => {-like => 'Smi%'}, Gender => 'M', Salary => {'>' => 4000}, Job => {-not_in => [qw/programmer analyst/]}, Birth => {-between => [qw/1970-01-01 1990-12-31/]}, Address => {'<>' => undef}, }

SELECT * FROM people WHERE Name LIKE 'Smi%' AND Gender = 'M' AND Salary > 4000 AND Job NOT IN ('programmer', 'analyst') AND Birth BETWEEN 1970-01-01 AND 1990-12-31 AND Address IS NOT NULL

SYNOPSIS

my $parser = SQL::Abstract::FromQuery->new();

my $criteria = $parser->parse($http_query); # $http_query is an object with a ->param() method # .. or it can be just a hashref

my ($sql, @bind) = $sqla->select($table, \@cols, $criteria);

# ormy ($sql, @bind) = $sqlam->select( -columns => \@cols, -from => $table, -where => $criteria, );

Options

my $parser = SQL::Abstract::FromQuery->new(

# additional components (optional) -components => [qw/FR Oracle/],

# grammar rules for some specific fields (optional) -fields => { standard => [qw/field1 field2 .../], bool => [qw/bool_field1/], ... } );

Components

• FR– dates au format dd.mm.yyyy– mots-clés en français (VRAI, FAUX, ENTRE, NUL, etc.)– messages d'erreur en français

• Oracle– queries on date and time

• \ ["to_date(?, '$date_format')", $date]• \ ["to_date(?, '$time_format')", $date]

WHERE D_BIRTH > TO_DATE('DD.MM.YYYY', ?)

• Contains– generate fulltext queries

• … (user-defined)

Field types : which grammar to apply

• Standard– a plain value (number, string, date or time).– a list of values, separated by ','. – a negated value or list of values; negation is expressed by !

or != or - or <>– a comparison operator <=, <, >=, > followed by a plain value– the special word NULL– BETWEEN val1 AND val2– boolean values YES, NO, TRUE or FALSE

• Restricted– string / bool / date / … (user-defined)

Internals

Regexp::Grammars

use Regexp::Grammars; qr{

<grammar: SQL::Abstract::FromQuery>

<rule: standard> <MATCH=between> | <MATCH=op_and_value> | <MATCH=values>

<rule: values> <[value]>+ % ,

<token: compare> <= | < | >= | >

<rule: value> <MATCH=null> | <MATCH=date> | <MATCH=time> | <MATCH=string> <rule: date> <day=(\d\d?)>\.<month=(\d\d?)>\.<year=(\d\d\d?\d?)> | <year=(\d\d\d?\d?)>-<month=(\d\d?)>-<day=(\d\d?)>

Autoactions hooked to the grammar

sub between { my ($self, $h) = @_; return {-between => [$h->{min}, $h->{max}]};}

sub values { my ($self, $h) = @_; my $n_values = @{$h->{value}}; return $n_values > 1 ? {-in => $h->{value}} : $h->{value}[0];}

Grammar Inheritance

<grammar: SQL::Abstract::FromQuery::FR> <extends: SQL::Abstract::FromQuery>

<rule: null> NULL?

<rule: between> <SQL::Abstract::FromQuery::between> | ENTRE (*COMMIT) (?: <min=value> ET <max=value> | <error:> )

<rule: bool> O(?:UI)? (?{ $MATCH = 1 }) | V(?:RAI)? (?{ $MATCH = 1 }) | NO?N? (?{ $MATCH = 0 }) | F(?:AUX|ALSE)? (?{ $MATCH = 0 }) | Y(?:ES)? (?{ $MATCH = 1 })

Rules inheritance

package SQL::Abstract::FromQuery::Oracle;use parent 'SQL::Abstract::FromQuery';use mro 'c3';

¨sub date { my ($self, $h) = @_; my $date_format = $self->{date_format} || 'YYYY-MM-DD'; my $date = $self->next::method($h);

return \ ["to_date(?, '$date_format')", $date];}

10.04.23 - Page 1

DépartementOffice

Conclusion

Features

• flexible syntax for user input– plain value– list of values– comparison operators– patterns– special constructs (BETWEEN, MATCH, etc.)– Including SQLA "special ops"

• internationalization• automate specific behaviours for some form fields

– data conversion– inject special ops

Usage

• See App::AutoCRUD come to tomorrow's talk