SQLAlchemy Primer (extra content) for Kobe Python Meetup #13 2017/09/15 Kobe Japan
SQLAlchemy Primer(extra content)
for Kobe Python Meetup #13 2017/09/15 Kobe Japan
Yasushi Masuda PhD ( @whosaysni )
Tech team, Core IT grp. IT Dept. MonotaRO Co., LTD.
Pythonista since 2001 (2.0~) • elaphe (barcode library) • oikami.py (老神.py) • PyCon JP founder
Japanese Translation works
Agenda
Myths
Core concepts in SQLAlchemy
Engine basics (+hands-on)
ORM primer (+hans-on)
References
Online Document:http://docs.sqlalchemy.org/
en/rel_1_1/
(Old) Japanese translation:http://omake.accense.com/static/doc-ja/sqlalchemy/
Preparationsakila DB on SQLite http://bit.ly/2fdeeft
https://github.com/jOOQ/jOOQ/jOOQ-examples/Sakila/sqlite-sakila-db/sqlite-sakila.sq
Sakila • Demonstration
DB for MySQL • Models a rental
video shop • BSD License
Schema described at: https://dev.mysql.com/doc/sakila/en/sakila-structure-tables.html
Myths[WRONG!] It's just an ORM libraryNot limited to. SQLAlchemy is a DB manipulation framework.
[WRONG!] SA is built on ORMNO. Connection management and SQL expression framework are its core.ORM is built on them.[WRONG!] SA cannot handle raw SQLTable definition is not required. Even textual SQL is available.
[WRONG!] SA automatically escapes value for youSQLAlchemy relies value escaping on DB-API, while it escapes schema, table, column names. SQLAlchemy generates parameterized query that helps DB-API level escaping.
[WRONG!] Only Python adept can handle SASQLAlchemy is EASY. You may need SQL and RDBMS experience. Let's see!
[AGREE] Documentation is somewhat difficult to understandAgreed.
Core concepts
http://docs.sqlalchemy.org/en/latest/index.html
http://docs.sqlalchemy.org/en/latest/index.html
SQLAlchemy Core
Engine (connection) Schema definitionSQL Expression
SQLAlchemy ORM
Mapper Declarative Mapper
Session
Dialect DB Backend-specific functionalities
Engine manages DB connection(s)
SQL Expressiondescribes SQL statement in Python
Mappingreflects DB record with Python object
Dialect DB Backend specific functionalities
DatabaseYour program
SQL construction
Query execution
Typeconversion
Parameter binding
Driver Setup
Connection management
Result data structure
Value escaping
Schema name
Type conversion
Dialect-specific
Query Construction
Query Execution
Result
Schema object Object-relational mapping
High-level Interface
DB Session
Engine
DB API DB ServerProgram
Issues around DB-API
Which DB-API to use How to initialize the DB-API
How to generate valid query for it How to handle cursor on the DB-API How to execute query on the DB-API How to retrieve result from DB-API How to reserve/reuse connection
DB API DB ServerProgram
Engine resolves issues
Select DB-API from DSN URL Initialize DB-API for you
Accept string and SQL expression Manage cursor for you
Unify execution/transaction API Handle result via ResultProxy Pool connection automatically
Engine
Engine DB API DB ServerProgram
>>>fromsqlalchemyimportcreate_engine>>>
Engine DB API DB ServerProgram
>>>fromsqlalchemyimportcreate_engine>>>e=create_engine('sqlite://')#SQLitememoryengine>>>
Engine DB API DB ServerProgram
#UseURLforspecifyingdatabase>>>e=create_engine('sqlite://')#SQLitememoryengine>>>e=create_engine('sqlite:///path_to_db.sqlite3')#sqlite3>>>e=create_engine('mysql://scott:tiger@dbserv/dbname')#mysql>>>e=create_engine('mssql://bill:gates@dbserv/dbname')#mssql>>>
Engine DB API DB ServerProgram
#Becarefulwithnumberofslashes>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>e=create_engine('sqlite:////<absolute_path>/sqlite-sakila.sq')
Engine DB API DB ServerProgram
>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>eEngine(sqlite:///sakila-data.sq)
#executereturnsResultProxy>>>q='selecttitlefromfilmlimit5')>>>res=e.execute(q)>>>res<sqlalchemy.engine.result.ResultProxyobjectat0x10da96990>>>>
Engine DB API DB ServerProgram
>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>eEngine(sqlite:///sakila-data.sq)
#executereturnsResultProxy>>>q='selecttitlefromfilmlimit5')>>>res=e.execute(q)>>>res<sqlalchemy.engine.result.ResultProxyobjectat0x10da96990>>>forrowinres:#ResultProxycanbeiterated/namedtupleaccess...print(row.title)...ACADEMYDINOSAURACEGOLDFINGERADAPTATIONHOLESAFFAIRPREJUDICEAFRICANEGG>>>
Engine DB API DB ServerProgram
>>>q='selectfilm_id,titlefromfilmlimit10'>>>rows=list(e.execute(q))>>>rows[2][1]#eachrowisaccessiblelikeastuple'ADAPTATIONHOLES'>>>rows[4]['title']#rowcanbeaccessiblelikeasdictionary'AFRICANEGG'
>>>res=e.execute(q)>>>forfid,titleinres:#canbeexpandedasnormaltuple...print((fid,title))...(1,'ACADEMYDINOSAUR')(2,'ACEGOLDFINGER')(3,'ADAPTATIONHOLES')(4,'AFFAIRPREJUDICE')(5,'AFRICANEGG')>>>rows=list(e.execute(q))>>>
transaction
>>>t=e.begin()>>>t.transaction<sqlalchemy...RootTransactionobjectat...>>>>t.transaction.commit()
#withstatementhandlestransactionsmart>>>withe.begin():e.execute(...)>>>#(transactioncommittedautomatically)
HANDS ON: Engine basics
Connecttosqlite-sakila.sqdatabase
Listactorsinfilm"DINOSAURSECRETARY"
HANDS ON: Engine basics
>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>q='''selecta.first_name,a.last_name...fromfilmasf...innerjoinfilm_actorasfa...onf.film_id=fa.film_id...innerjoinactorasa...onfa.actor_id=a.actor_id...wheref.title="DINOSAURSECRETARY"'''>>>forfirst_name,last_nameine.execute(q):...print('{}{}'.format(first_name,last_name))...LUCILLETRACYBURTDUKAKISJAYNENEESONRUSSELLBACALLPENELOPEMONROEMINNIEKILMER
SQL expression
Remember: SQL is a language
SELECT[ALL|DISTINCT[ON(expression[,...])]]
[*|expression[[AS]output_name][,...]]
[FROMfrom_item[,...]]
[WHEREcondition]
[GROUPBYgrouping_element[,...]]
[ORDERBYexpression[ASC|DESC|USINGoperator][NULLS{FIRST|LAST}][,...]]
[LIMIT{count|ALL}]
Remember: SQL is a language
SELECT[ALL|DISTINCT[ON(expression[,...])]]
[*|expression[[AS]output_name][,...]]
[FROMfrom_item[,...]]
[WHEREcondition]
[GROUPBYgrouping_element[,...]]
[ORDERBYexpression[ASC|DESC|USINGoperator][NULLS{FIRST|LAST}][,...]]
[LIMIT{count|ALL}]
STATEMENT
Remember: SQL is a language
SELECT[ALL|DISTINCT[ON(expression[,...])]]
[*|expression[[AS]output_name][,...]]
[FROMfrom_item[,...]]
[WHEREcondition]
[GROUPBYgrouping_element[,...]]
[ORDERBYexpression[ASC|DESC|USINGoperator][NULLS{FIRST|LAST}][,...]]
[LIMIT{count|ALL}]
CLAUSE
CLAUSE
CLAUSE
CLAUSE
CLAUSE
CLAUSE
Remember: SQL is a language
SELECT[ALL|DISTINCT[ON(expression[,...])]]
[*|expression[[AS]output_name][,...]]
[FROMfrom_item[,...]]
[WHEREcondition]
[GROUPBYgrouping_element[,...]]
[ORDERBYexpression[ASC|DESC|USINGoperator][NULLS{FIRST|LAST}][,...]]
[LIMIT{count|ALL}]
parameter
parameter
expression
expression
expression
expression
expression
expression
Remember: SQL is a Language
SELECT
FROM selectablesselectable
join
WHERE
selectable
selectable
GROUP BY ORDER BY
expression
expressioncondition
expression
expressionsexpression
expression
SELECT statement
SELECTA.name,B.title,C.title
FROMartistsasAINNERJOINalbumasBONA.id=B.artist_idINNERJOINmusicasCONB.id=C.album
WHEREA.nameLIKE'%Astor%'ANDA.yearBETWEEN1970AND2010ANDC.titleLIKE'%Tango%'
From SQL to Python
SELECT<column>,<column>,...
FROM<selectable>INNERJOIN<selectable>ON<condition>INNERJOIN<selectable>ON<condition>...
WHERE<condition>AND<condition>AND<condition>
GROUPBY...LIMIT...
Clauses
SELECT<[<column>,<column>,<column>]>
FROM<join(<selectable>,<selectable>,...)>
WHERE<and(<condition>,<condition>,...)>
Statement and subjects
SELECT<expressions>
FROM<selectable>
WHERE<conditions>
... simplifed
<selectstatement>
..., finally
engine.execute(<selectstatement>)
If query is "an object"...
Query object
>>>query=select(...)
>>>engine.execute(query)
... it can be "execute()-able"
Engine "compiles" query into string and execute it
(according to dialect)
query=select(
<expression>,
from_obj=<selectables>,
whereclause=<conditions>,
)
clauses as parameters
columns=[col1,col2,...]
fromobj=join(tbl1,tbl2,...)
where=and_(expr1,expr2,...)
query_expr=select(columns,from_obj=fromobj,whereclause=where)
query with SQL expression
>>>fromsqlalchemy.sqlimportselect,text>>>q=select([text('*')])
building sql statement with basic sql expression
>>>fromsqlalchemy.sqlimportselect,text>>>q=select([text('*')])>>>q<sqlalchemy.....Selectat...;Selectobject>>>>str(q)'SELECT*'>>>q=select([text('*')],...from_obj=text('foo'),...whereclause=text('id=3'))>>>str(q)'SELECT*\nFROMfoo\nWHEREid=3'
building sql statement with basic sql expression
>>>fromsqlalchemy.sqlimportselect,text>>>q=select([text('*')])>>>q<sqlalchemy.....Selectat...;Selectobject>>>>str(q)'SELECT*'>>>q=select([text('*')],...from_obj=text('foo'),...whereclause=text('id=3'))>>>str(q)'SELECT*\nFROMfoo\nWHEREid=3'
building sql statement with basic sql expression
elements: table and columnSchema
Table Table Table
...
Column
Column
Column
...
Column
Column
Column
...
Column
Column
Column
...
SchemaTable Table Table
Column
Column
Column
Column
Column
Column
...
elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER
elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER>>>c=column('name')#simplest>>>c=column('name',type_=INTEGER)
elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER>>>c=column('name')#simplest>>>c=column('name',type_=INTEGER)>>>c<sqlalchemy....ColumnClauseat...;name>>>>str(c)'name'
elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER>>>c=column('name')#simplest>>>c=column('name',type_=INTEGER)>>>c<sqlalchemy....ColumnClauseat...;name>>>>str(c)'name'>>>c.table#None>>>t1=table('artist')>>>c.table=t1>>>str(c)'artist.name'
>>>t=table('tbl1',column('col1'),...)
defining table with columnsTable name List of columns
>>>t=table('tbl1',column('col1'),...)
>>>t.c.col1
<sqlalchemy.....ColumnClauseat...;col1>
defining table with columnsTable name List of columns
>>>t=table('tbl1',column('col1'),...)
>>>t.c.col1
<sqlalchemy.....ColumnClauseat...;col1>
>>>t.schema='db1'
>>>str(t.c.col1)
'db1.tbl1.col1'
defining table with columnsTable name List of columns
>>>t=table('tbl1',column('col1'),column('col2'))
select() with table element
>>>t=table('tbl1',column('col1'),column('col2'))>>>print(select([t]))SELECTtbl1.col1,tbl1.col2FROMtbl1
select() with table element
>>>t=table('tbl1',column('col1'),column('col2'))>>>print(select([t]))SELECTtbl1.col1,tbl1.col2FROMtbl1
#columnlabeling>>>print(select([t.c.col1.label('col_alias1')])SELECTtbl1.col1as"col_alias1"FROMtbl1
select() with table element
>>>t=table('tbl1',column('col1'),column('col2'))>>>print(select([t]))SELECTtbl1.col1,tbl1.col2FROMtbl1
#columnlabeling>>>print(select([t.c.col1.label('col_alias1')])SELECTtbl1.col1as"col_alias1"FROMtbl1
#tablealias>>>t_A,t_B=alias(t,'A'),alias(t,'B')>>>print(select([t_A.c.col1,t_B.c.col2]))SELECT"A".col1,"B".col2FROMtbl1AS"A",tbl1AS"B"
select() with table element
howtoworkwith
wheretbl1.col1=42
conditionals
howtoworkwith
wheretbl1.col1=42
conditionals
conditional expression (compare operation)
>>>cond=text('last_nameLIKE%KOV')>>>str(cond)'last_nameLIKE%KOV'
conditional by text()
>>>cond=column('last_name').like('%KOV')>>>cond<sqlalchemy....BinaryExpressionobjectat...>
conditional by like() method
>>>cond=column('last_name').like('%KOV')>>>cond<sqlalchemy....BinaryExpressionobjectat...>>>>str(cond)'last_nameLIKE:last_name_1'
conditional by like() method
placeholder for right value of LIKE
>>>cond=column('last_name').like('%KOV')>>>cond<sqlalchemy....BinaryExpressionobjectat...>>>>str(cond)'last_nameLIKE:last_name_1
>>>cond.rightBindParameter('%(4339977744last_name)s','%KOV',type_=String())
conditional by like() method
>>>column('first_name')='DAVID'File"<stdin>",line1SyntaxError:can'tassigntofunctioncall
conditional by operation
>>>column('first_name')='DAVID'File"<stdin>",line1SyntaxError:can'tassigntofunctioncall
>>>column('first_name')=='DAVID'<sqlalchemy...BinaryExpressionobjectat...>
conditional by operation
>>>column('first_name')='DAVID'File"<stdin>",line1SyntaxError:can'tassigntofunctioncall
>>>column('first_name')=='DAVID'<sqlalchemy...BinaryExpressionobjectat...>
>>>cond=column('first_name')=='DAVID'>>>str(cond)>>>'first_name=:first_name_1'>>>cond.right>>>BindParameter('%(...)s','DAVID',type_=...)
conditional by operation
>>>column('first_name')='DAVID'File"<stdin>",line1SyntaxError:can'tassigntofunctioncall
>>>column('first_name')=='DAVID'<sqlalchemy...BinaryExpressionobjectat...>
>>>cond=column('first_name')=='DAVID'>>>str(cond)>>>'first_name=:first_name_1'>>>cond.right>>>BindParameter('%(...)s','DAVID',type_=...)
>>>str(column('last_name')==None)>>>'last_nameisNULL'
conditional by operation
>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))
conditionals in select
>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))>>>query=select([actor_tbl],...whereclause=(actor_tbl.c.last_name=='TRACY'))
conditionals in select
>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))>>>query=select([actor_tbl],...whereclause=(actor_tbl.c.last_name=='TRACY'))>>>query<sqlalchemy.....Selectat...;Selectobject>>>>print(query)SELECTactor.first_name,actor.last_nameFROMactorWHEREactor.last_name=:last_name_1>>>query.compile().params{'last_name_1':'TRACY'}
conditionals in select
>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))>>>query=select([actor_tbl],...whereclause=(actor_tbl.c.last_name=='TRACY'))>>>query<sqlalchemy.....Selectat...;Selectobject>>>>print(query)SELECTactor.actor_id,actor.first_name,actor.last_nameFROMactorWHEREactor.last_name=:last_name_1>>>query.compile().params{'last_name_1':'TRACY'}>>>list(e.execute(query))[(20,'LUCILLE','TRACY'),(117,'RENEE','TRACY')]
conditionals in select
select(...)
SELECT...
select(...).select_from(...)
SELECT...FROM...
select(...).where(...)
SELECT...WHERE...
select(...).where(...).where(...)
SELECT...WHERE...AND...
select(...).where(...).order_by(...)
SELECT...WHERE...ORDERBY...
generative method
actor_tbl.select()SELECT...FROMactor
actor_tbl.select(actor_tbl.c.first_name='BEN')SELECT...FROMactorWHEREfirst_name=...
generative method
HANDS ON: SQL expression
• define actor table with table()/column()
• build query with sql expression to searchactor having initial A.H. (result should include actor_id)
Schema definition
howtoCREATEtable
schema definition
howtoCREATEtable
howtodefinecolumndetailhowtodefineconstraints
schema definition
>>>fromsqlalchemyimportColumn,MetaData,Table>>>user_table=Table(...'user',MetaData(),...Column('id',INTEGER,primary_key=True),...Column('first_name',VARCHAR(45)),...Column('last_name',VARCHAR(45)))>>>
defining table
>>>fromsqlalchemyimportColumn,MetaData,Table>>>user_table=Table(...'user',MetaData(),...Column('id',INTEGER,primary_key=True),...Column('first_name',VARCHAR(45)),...Column('last_name',VARCHAR(45)))>>>user_tableTable('user',MetaData(bind=None),Column...)
defining table
>>>fromsqlalchemyimportColumn,MetaData,Table>>>user_table=Table(...'user',MetaData(),...Column('id',INTEGER,primary_key=True),...Column('first_name',VARCHAR(45)),...Column('last_name',VARCHAR(45)))>>>user_tableTable('user',MetaData(bind=None),Column...)
#CREATETABLEbycrete()>>>user_table.create(bind=e)
#DROPTABLEwithdrop()>>>user_table.drop(bind=e)
defining table
Table/Column vs table/columnVisitable (base type)
ClauseElement
Selectable
FromClause [ table() ]
ColumnElement
ColumnClause [ column() ]
Column Table
SchemaItem
Table/Column vs table/columnVisitable (base type)
ClauseElement
Selectable
FromClause [ table() ]
ColumnElement
ColumnClause [ column() ]
Column Table
SchemaItem
Supprorts same operation
HANDS ON: Schema definition
• define "user" table with Table()/Column()
• create table with .create()
• insert records
• drop table with .drop()
name type options
id INTEGER primary_key
username VARCHAR (64) nullable=False
email VARCHAR(128) nullable=False
ORM basics
Object-Relational Mapping
• Map DB records to objects
• ActiveRecord pattern
• 1 table ~> 1 class, 1 record ~> 1 object
• FK reference -> reference to other object
• column -> property/attribute of object
ORM in SQLAlchemy
• mapper maps a table with Python classmapped class will have instrumented attribute
• Session manipulates DB for you to retrieve / save mapped objects
mappingTable
Column foo
Column bar
Column baz
...
entity class
"mapped" entity class
instrumented fooinstrumented barinstrumented baz
...
method quxmethod quux
method quxmethod quux
...
mapper
classic mapping
[new] declarative mapping
mapping patterns
>>>fromsqlalchemy.ormimportmapper>>>actor_tbl=Table(...)>>>classActor(object):pass>>>mapper(Actor,actor_tbl)>>>dir(Actor)['__class__',...'_sa_class_manager','actor_id','first_name','last_name']>>>Actor.actor_id<...InstrumentedAttributeobjectat...>
classic mapping
>>>fromsqlalchemy.ext.declarativeimportdeclarative_base>>>fromsqlalchmyimportDateTime,Integer,String>>>Base=declarative_base()>>>>>>classActor(Base):...__tablename__='actor'...actor_id=Column(Integer,primary_key=True)...first_name=Column(String)...last_name=Column(String)...last_update=Column(DateTime)
declarative mapping
>>>fromsqlalchemy.ext.declarativeimportdeclarative_base>>>fromsqlalchmyimportDateTime,Integer,String>>>Base=declarative_base()>>>>>>classActor(Base):...__tablename__='actor'...actor_id=Column(Integer,primary_key=True)...first_name=Column(String)...last_name=Column(String)...last_update=Column(DateTime)>>>dir(Actor)['__class__','_decl_class_registry','_sa_class_manager','actor_id','first_name','last_name','last_update','metadata']
declarative mapping
ORM in SQLAlchemy
• mapper maps a table with Python classmapped class will have instrumented attribute
• Session manipulates DB for you to retrieve / save mapped objects
sessionEngineProgram
query Aselect A
record Aobject Bstart
tracking A
...
(updates) flag A as "dirty"
reflect new/dirty changes
flush
starttracking B
...
add object B
update A insert B
commit
begin
Using Session
>>>fromsqlalchemy.ormimportsessionmaker>>>Session=sessionmaker(bind=e)>>>session=Session()>>>>>>
Using Session
>>>fromsqlalchemy.ormimportsessionmaker>>>Session=sessionmaker(bind=e)>>>session=Session()>>>query=session.query(Actor)>>>query<sqlalchemy.orm.query.Queryobjectat...>
Using Session
>>>fromsqlalchemy.ormimportsessionmaker>>>Session=sessionmaker(bind=e)>>>session=Session()>>>query=session.query(Actor)>>>query<sqlalchemy.orm.query.Queryobjectat...>>>>actor=query.first()>>>actor<...ActorObjectat...>
Query object methods: query filtering
>>>q=session.query(Actor)>>>str(q)>>>'SELECT...\nFROMactor'>>>q1=q.filter(Actor.first_name=='BEN')>>>str(q1)'SELECT...\nFROMactor\nWHEREactor.first_name=?'>>>q2=q1.filter_by(last_name='SMITH')>>>str(q2)'SELECT...\nFROMactor\nWHEREactor.first_name=?ANDactor.last_name=?'
Query object methods: fetching record(s)
>>>q=session.query(Actor)>>>q.first()<...Actorobjectat...>
>>>q.all()#WARNING:SELECTsallrecords<...Actorobjectat...>,<...Actorobjectat...>,...
>>>q[:3]<...Actorobjectat...>,<...Actorobjectat...>,...
>>>q.count()200
Update/Insert in Session
>>>session=Session()>>>actor_a=Actor(first_name='KEN',...last_name='WATANABE')>>>session.add(actor_a)>>>session.commit()
>>>actor_b=session.query(Actor).get(1)>>>actor_b.last_name='GEORGE'>>>session.commit()
HANDS ON: ORM basics
• define Category model with declarative ORM
• select any record
• add "Nature" category
• delete "Nature" category