MOLGENIS For rapid prototyping of web applications biologists want to have

|[email protected]

2009

2009

|

MOLGENIS.org

For rapid prototyping of web applications biologists want to have

Morris Swertz Joeri van der Velde

Joris Lops

NBIC/BioAssist programmers meetingUtrecht, 18 December 2009

1

EBIBiobanking platform

|[email protected] |

Outline

› Introduction Motivation Conceptual strategy MOLGENIS implementation Expected results Underlying architecture

› Practical Install a MOLGENIS virtual machine Work through examples in manual See what it can do for you

2

|[email protected]

2009

2009

|

Motivation

3


Biological challenges4

inbreedinbreed

100

10.000

1,000,000

100,000

10,000

10

10,000,00

QTL profiles

QTL profiles

networknetwork

correlatecorrelate

genomegenomestrainsstrains

individualsindividuals

markersmarkers

expressionsexpressions preprocesspreprocess

probesprobesmicroarraysmicroarrays

100

hybridizehybridize

100,000

genotypegenotype genotypesgenotypes

norm exprs.norm exprs.

10.000

mapmap

Main work flow

Data dependency

Biomaterial/result

Lab/analysis process

Scale of information

Associated data files

processprocess

materialmaterial

10,000

! Many materials! HTP processes! Complex, distr. workflows! Large data! Trace dependencies! Adapt to new protocols

|[email protected] | [email protected] / 5

For example

Dozens of samplesComplex relationships

Large datasetsProcessing


For example

Swertz et al (2004) Bioinformatics 20, 2075-83


For example



For example



Informatic challenges9

Input markers,traits, genotypes

AMap QTLs(qtlMapTwoPart)

QTL profiles{qtlProfiles}

QTL threshold{qtlThres}

BSimulation/FDR(qtlThreshold/qtlFDR)

QTL summary{qtlSumm}

Input mass/chargepeaks

CQTL summary(qtlSummary) Significant QTLs

DZero-ordercorrelation

Peak multiplicity{peakMultiplicity}

HPeak multiplicity(findPeakultiplicity)

Correlation matrix{corrZeroOrder}

FPermutation(qtlCorrThreshold)

E2nd-order correlation(qtlCorrSecondOrder )

Correlation matrix{corrSecondOrder}

Correlation threshold{corrThres}

GCreate network(createCytoFiles)

Network files[network.sif,

network.eda]

researchers

Communication + integrationinfrastructure

datainfrastructure

processinginfrastructure

user interactioninfrastructure

bioinformatician

Swertz & Jansen (2007) Nature Reviews Genetics


… try hitting a moving target10

biologistbiologist

bioinformatician softw engineers

arab 220903

100 200 300 400 500 600 700 800 900 1000m /z0

100

%

Koornneef0007 526 (11.117) AM (Top,4, Ar,10000.0,556.28,0.70,LS 10); Sm (Mn, 2x1.00); Sb (1,40.00 )1.40e3171.1702

1396

649.3804551

526.3066248172.1795

162

650.3882224

809.4496;80

inbreedinbreed

100

100.000

10,000,000

1000

10,000

10

1000genotypegenotypeindividualsindividuals

mass peaks

mass peaks

genotypesgenotypes QTL profiles

QTL profiles

strainsstrains

networknetwork

SNP arraysSNP arrays

correlatecorrelate

LC/MSLC/MS

genomegenome

mapmap

preprocesspreprocess aligned peaks

aligned peaks

€

biological challenges suitable infrastructure


… repeatedly

http://www.molgenis.orgSwertz & Jansen (2007) Nature Revies Genetics 8, 235-243

NextGenSeq

MassSpec/Metab.

Experiments

WorkBiologist challenges Software solutions

Metabolomics support platform

11

|[email protected]

2009

2009

|

Strategy

12


Standardization + Variation 13

http://www.molgenis.orgSwertz & Jansen (2007) Nature Reviews Genetics 8, 235-243

http://www.molgenis.org


Model of a variant<entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"


Small model<entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"

Small model<entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"

Model variation, Reuse software

Code generationCode generation

+

inbreed

100

10.000

1,000,000

100,000

10,000

10,000,00

QTL profiles

network

correlate

genomestrains

individuals

markers

expressions

preprocess

probesmicroarrays

100

hybridize

100,000

genotype genotypes

norm exprs.

10.000

map

Step 1:Model variation Points(biology)

Step 2. Automate commonPatterns(informatics)

Step 3. Reuse in family of projects


Your model<entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"


Add new biology to model15

Reusable framework and generators



Model of a variant<entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int" +

Model new protocols

15

Use new protocols

inbreed

100

10.000

1,000,000

100,000

10,000

10,000,00

QTL profiles

network

correlate

genomestrains

individuals

markers

expressions

preprocess

probesmicroarrays

100

hybridize

100,000

genotype genotypes

norm exprs.

10.000

map




Add new features to software16




Model of a variant<entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int" +

inbreed

100

10.000

1,000,000

100,000

10,000

10,000,00

QTL profiles

network

correlate

genomestrains

individuals

markers

expressions

preprocess

probesmicroarrays

100

hybridize

100,000

genotype genotypes

norm exprs.

10.000

map

Add features once

16

Added automatically


… repeatedly

http://www.molgenis.orgSwertz & Jansen (2007) Nature Revies Genetics 8, 235-243

NextGenSeq

MassSpec/Metab.

Experiments

WorkBiologist challenges Software solutions

Metabolomics support platform

17

|[email protected]

2009

2009

|

MOLGENIS in practice

18

|[email protected] | 19


Step 0: Brainstorm20

Observedvalue

Observedvalue*

Observationtarget

Observationtarget

time

Observablefeature

Observablefeature

*

PanelPanel IndividualIndividual* Observed Relation

Observed Relation

Inferred Value

Inferred Value

*


Step 1a: define the data model

1: molgenis_db.xml

506 lines of XML code16 entities, 67 fields


Step 1b: add ui model

1: molgenis_db.xml

50 lines of XML code(forms, menu’s and plugins)


Language elements<?xml version="1.0" encoding="UTF-8"?>

<!ELEMENT molgenis (description? , (module | entity |form | menu | plugin)*)>

<!ATTLIST molgenis name CDATA #REQUIRED>

<!ATTLIST molgenis label CDATA #IMPLIED>

<!ATTLIST molgenis version CDATA #IMPLIED>

<!ELEMENT description ANY>

<!ELEMENT module (description?, entity+)>

<!ATTLIST module name CDATA #REQUIRED>

<!ELEMENT entity (description?,field*,unique*)>

<!ATTLIST entity name CDATA #REQUIRED>

<!ATTLIST entity abstract (true|false) #IMPLIED>

<!ATTLIST entity implements CDATA #IMPLIED>

<!ATTLIST entity extends CDATA #IMPLIED>

<!ATTLIST entity decorator CDATA #IMPLIED>

<!ATTLIST entity description CDATA #IMPLIED>

<!ELEMENT field EMPTY>

<!ATTLIST field name CDATA #REQUIRED>

<!ATTLIST field type CDATA #IMPLIED>

<!ATTLIST field label CDATA #IMPLIED>

<!ATTLIST field length CDATA #IMPLIED>

<!ATTLIST field xref_field CDATA #IMPLIED>

<!ATTLIST field xref_label CDATA #IMPLIED>

<!ATTLIST field enum_options CDATA #IMPLIED>

<!ATTLIST field default CDATA #IMPLIED>

<!ATTLIST field auto (true|false) #IMPLIED>

<!ATTLIST field nillable (true|false) #IMPLIED>

<!ATTLIST field unique (true|false) #IMPLIED>

<!ATTLIST field readonly (true|false) #IMPLIED>

<!ATTLIST field hidden (true|false) #IMPLIED>

<!ATTLIST field description CDATA #IMPLIED>

<!ELEMENT unique EMPTY>

<!ATTLIST unique fields CDATA #REQUIRED>

<!ATTLIST unique subclass (true|false) #IMPLIED>

<!ATTLIST unique description CDATA #IMPLIED>

<!ELEMENT form (form*,menu*,plugin*)>

<!ATTLIST form name CDATA #REQUIRED>

<!ATTLIST form entity CDATA #REQUIRED>

<!ATTLIST form label CDATA #IMPLIED>

<!ATTLIST form view (list|record) #IMPLIED>

<!ATTLIST form readonly (yes|no) #IMPLIED>

<!ELEMENT menu (form*,menu*,plugin*)>

<!ATTLIST menu name CDATA #REQUIRED>

<!ATTLIST menu label CDATA #IMPLIED>

<!ELEMENT plugin (form*,menu*,plugin*)>

<!ATTLIST plugin name CDATA #REQUIRED>

<!ATTLIST plugin type CDATA #REQUIRED>

<!ATTLIST plugin label CDATA #IMPLIED>

See practical.

23


Generates 150 files, 30k lines of Java, SQL and R code + docs(tomcat/mysql; hsqldb, psql, jpa/hibernate, jetty in alpha )

0 INFO [myFactory] working dir: D:\Development\Molgenis33Workspace\molgenis4phenotype78 INFO [myFactory] MOLGENIS version 3.3.0-testing94 INFO [myFactory] Using options:model_database = [pheno_db.xml] #File with data structure specification (in MOLGENIS DSL).model_userinterface = pheno_ui.xml #File with user interface specification (in MOLGENIS DSL). Can be same file as model_databaseoutput_src = generated/java #Output-directory for the generated project.output_hand = handwritten/java #Output-directory for the generated project.output_sql = generated/sql #Output-directory for the generated sql files.output_doc = WebContent/doc #Output-directory for the generated documentation.output_type = #Output type of the project, either war (for use in tomcat) or jar (standalone).output_web = WebContent #Output-directory for any generated web resourcesdb_driver = com.mysql.jdbc.Driver #Driver of database. Any JDBC compatible driver should work.db_user = molgenis #Username for database. db_password = xxxxxx #Password for database. db_uri = jdbc:mysql://localhost/pheno #Uri of the database. Default: localhostdb_filepath = attachedfiles #Path where the database should store file attachements. Default: nulldb_jndiname = jdbc/molgenisdb #Used to create a JDBC database resource for the applicationobject_relational_mapping = subclass_per_table #Expert option: Choosing OR strategy. Either 'class_per_table', 'subclass_per_table', 'hierarchy_per_table'. Default: class_per_tablemapper_implementation = multiquery #Expert option: Choosing wether multiquery is used instead of prepared statements. Default: falseexclude_system = true #Expert option: Whether system tables should be excluded from generation. Default: trueforce_molgenis_package = false #Expert option. Whether the generated package should be 'molgenis' or the name specified in the model. Default: falseauth_loginclass = org.molgenis.framework.security.SimpleSecurity #Expert option.verbose = true #This switch turns the verbose-mode on.compile = false #This switch makes the factory also compile (usefull outside IDE).mail_smtp_protocol = #Sets the email protocol, either smtp, smtps or null. Default: null meaning email disabledmail_smtp_hostname = localhost #SMTP host server. Default: localhostmail_smtp_port = 25 #SMTP host server port. Default: 25mail_smtp_user = #SMTP user for authenticated emailing. Default: null.mail_smtp_password = #SMTP user for authenticated emailing. Default: null.

110 INFO [MolgenisLanguage] parsing db-schema from [pheno_db.xml]780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable797 WARN [Entity] [WARNING]: missing key 0 for entity Nameable797 WARN [Entity] [WARNING]: missing key 0 for entity Nameable844 INFO [MolgenisLanguage] parsing ui-schema937 INFO [main] generating ....1717 INFO [TableDocGen] generated WebContent\doc\tabledoc.html2076 INFO [EntityDocGen] generated WebContent\doc\objectmodel.html2436 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram.dot2545 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.system.dot2748 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.observation.dot2842 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.target.dot2998 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.variable.dot3138 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.protocol.dot3997 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-minimal-diagram.dot4184 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.system.dot4388 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.observation.dot4606 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.target.dot4731 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.variable.dot4887 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.protocol.dot5184 INFO [ClassDocGen] generated WebContent\doc\classmodel.html5293 INFO [InMemoryDatabaseGen] generated generated\java\ui\data\InMemoryDatabase.java5609 INFO [MySqlCreateSubclassPerTableGen] generated generated\sql\create_tables.sql5671 INFO [JDBCDatabaseGen] generated generated\java\ui\JDBCDatabase.java5921 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Identifiable.java5921 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Nameable.java5968 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\OntologySource.java6014 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\OntologyTerm.java6030 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Investigation.java6061 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservableFeature.java6124 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservedValue.java6170 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservedRelationship.java6217 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\InferredValue.java6233 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservationTarget.java6280 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Individual.java6311 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Panel.java6326 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\CodeList.java6327 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Code.java6374 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Protocol.java6390 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ProtocolApplication.java6405 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ProtocolParameter.java6437 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ParameterValue.java6452 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\InferredValue_derivedFrom.java6468 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Panel_individuals.java6483 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Protocol_observableFeatures.java6499 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Protocol_protocolComponents.java6624 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\OntologySourceMapper.java6655 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\OntologyTermMapper.java6671 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\InvestigationMapper.java6702 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservableFeatureMapper.java6733 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservedValueMapper.java6780 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservedRelationshipMapper.java6827 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\InferredValueMapper.java6842 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservationTargetMapper.java6873 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\IndividualMapper.java6889 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\PanelMapper.java6905 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\CodeListMapper.java6936 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\CodeMapper.java6951 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ProtocolMapper.java6983 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ProtocolApplicationMapper.java6998 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ProtocolParameterMapper.java7029 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ParameterValueMapper.java7045 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\InferredValue_derivedFromMapper.java7061 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\Panel_individualsMapper.java7076 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\Protocol_observableFeaturesMapper.java7092 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\Protocol_protocolComponentsMapper.java7217 INFO [JDBCMetaDatabaseGen] generated generated\java\ui\JDBCMetaDatabase.java7263 INFO [CountPerEntityGen] generated generated\sql\count_per_entity.sql7310 INFO [CountPerTableGen] generated generated\sql\count_per_table.sql7341 INFO [FillMetadataTablesGen] generated generated\sql\insert_metadata.sql7405 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\OntologySourceCsvReader.java7420 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\OntologyTermCsvReader.java7420 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\InvestigationCsvReader.java7436 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservableFeatureCsvReader.java7452 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservedValueCsvReader.java7467 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservedRelationshipCsvReader.java7483 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\InferredValueCsvReader.java7498 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservationTargetCsvReader.java7514 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\IndividualCsvReader.java7514 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\PanelCsvReader.java7530 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\CodeListCsvReader.java7545 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\CodeCsvReader.java7545 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ProtocolCsvReader.java7561 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ProtocolApplicationCsvReader.java7561 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ProtocolParameterCsvReader.java7576 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ParameterValueCsvReader.java7576 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\InferredValue_derivedFromCsvReader.java7592 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\Panel_individualsCsvReader.java7608 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\Protocol_observableFeaturesCsvReader.java7608 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\Protocol_protocolComponentsCsvReader.java7748 INFO [REntityGen] generated generated\java\pheno\core\R\OntologySource.R7748 INFO [REntityGen] generated generated\java\pheno\core\R\OntologyTerm.R7764 INFO [REntityGen] generated generated\java\pheno\core\R\Investigation.R7779 INFO [REntityGen] generated generated\java\pheno\core\R\ObservableFeature.R7779 INFO [REntityGen] generated generated\java\pheno\core\R\ObservedValue.R7795 INFO [REntityGen] generated generated\java\pheno\core\R\ObservedRelationship.R7795 INFO [REntityGen] generated generated\java\pheno\core\R\InferredValue.R7810 INFO [REntityGen] generated generated\java\pheno\core\R\ObservationTarget.R7810 INFO [REntityGen] generated generated\java\pheno\core\R\Individual.R7826 INFO [REntityGen] generated generated\java\pheno\core\R\Panel.R7826 INFO [REntityGen] generated generated\java\pheno\core\R\CodeList.R7842 INFO [REntityGen] generated generated\java\pheno\core\R\Code.R7857 INFO [REntityGen] generated generated\java\pheno\core\R\Protocol.R7857 INFO [REntityGen] generated generated\java\pheno\core\R\ProtocolApplication.R7873 INFO [REntityGen] generated generated\java\pheno\core\R\ProtocolParameter.R7873 INFO [REntityGen] generated generated\java\pheno\core\R\ParameterValue.R7888 INFO [REntityGen] generated generated\java\pheno\core\R\InferredValue_derivedFrom.R7888 INFO [REntityGen] generated generated\java\pheno\core\R\Panel_individuals.R7888 INFO [REntityGen] generated generated\java\pheno\core\R\Protocol_observableFeatures.R7904 INFO [REntityGen] generated generated\java\pheno\core\R\Protocol_protocolComponents.R7998 INFO [RApi] generated generated\java\source.R8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\IdentifiableHtmlForm.java8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\NameableHtmlForm.java8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\OntologySourceHtmlForm.java8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\OntologyTermHtmlForm.java8060 INFO [HtmlFormGen] generated generated\java\pheno\core\html\InvestigationHtmlForm.java8060 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservableFeatureHtmlForm.java8076 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservedValueHtmlForm.java8076 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservedRelationshipHtmlForm.java8076 INFO [HtmlFormGen] generated generated\java\pheno\core\html\InferredValueHtmlForm.java8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservationTargetHtmlForm.java8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\IndividualHtmlForm.java8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\PanelHtmlForm.java8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\CodeListHtmlForm.java8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\CodeHtmlForm.java8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ProtocolHtmlForm.java8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ProtocolApplicationHtmlForm.java8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ProtocolParameterHtmlForm.java8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ParameterValueHtmlForm.java8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\InferredValue_derivedFromHtmlForm.java8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\Panel_individualsHtmlForm.java8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\Protocol_observableFeaturesHtmlForm.java8138 INFO [HtmlFormGen] generated generated\java\pheno\core\html\Protocol_protocolComponentsHtmlForm.java8138 INFO [MolgenisServletContextGen] generated WebContent\META-INF\context.xml8169 INFO [MolgenisContextListenerGen] generated generated\java\servlet\ContextListener.java8232 INFO [MolgenisServletGen] generated generated\java\MolgenisServlet.java8403 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\InvestigationsForm.java8560 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ObservableFeaturesForm.java8591 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\PanelsForm.java8654 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\Panels\IndividualsForm.java8701 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ObservedValuesForm.java8732 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplicationsForm.java8825 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenu\ParameterValuesForm.java8857 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenu\ObservedValuesForm.java8888 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenu\InferredValuesForm.java9013 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\InferredValuesForm.java9044 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservableFeaturesForm.java9137 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservationTargets\IndividualsForm.java9169 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservationTargets\PanelsForm.java9200 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ProtocolsForm.java9293 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Protocols\ProtocolMenu\ParametersForm.java9325 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Protocols\ProtocolMenu\ProtocolComponentsForm.java9496 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\OntologyTermsForm.java9528 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\OntologySourcesForm.java9606 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\OntologySources\OntologyTermsForm.java9638 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\CodeListsForm.java9700 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\CodeLists\CodesForm.java9965 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenuMenu.java10012 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\MainMenu.java10059 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenuMenu.java10152 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenuMenu.java10230 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservationTargetsMenu.java10293 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\Protocols\ProtocolMenuMenu.java10324 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\OntologiesMenu.java11354 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\Main\ReportPlugin.java11557 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\Main\Ontologies\OntologyManagerPlugin.java11604 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\Model_documentationPlugin.java11604 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\RprojectApiPlugin.java11620 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\HttpApiPlugin.java11635 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\WebServicesApiPlugin.java11651 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\report\InvestigationOverview.ftl11807 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\OntologyBrowser\OntologyBrowserPlugin.ftl11807 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\DocumentationScreen.ftl11807 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\RprojectApiScreen.ftl11823 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\HttpAPiScreen.ftl11823 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\SoapApiScreen.ftl11854 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\report\InvestigationOverview.java12057 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\OntologyBrowser\OntologyBrowserPlugin.java12072 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\DocumentationScreen.java12088 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\RprojectApiScreen.java12088 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\HttpAPiScreen.java12088 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\SoapApiScreen.java12103 INFO [MolgenisServletContextGen] generated WebContent\META-INF\context.xml12259 INFO [SoapApiGen] generated generated\java\ui\SoapApi.java12353 INFO [CsvExportGen] generated generated\java\tools\CsvExport.java12431 INFO [CsvImportByNameGen] generated generated\java\tools\CsvImportByName.java12636 INFO [CopyMemoryToDatabaseGen] generated generated\java\ui\tools\CopyMemoryToDatabase.java

Step 2: Generate (MOLGENIS + Eclipse)24


Step 2: Generate25

customize... Model file XML

Generate

APIs in Java, R, Web services and HTTP

MyScriptPlugins

FormGen

MenuGenTreeGen

PluginGenMatrixGen

JTypeGenJDBCMapGen

JListGenJReadCsvGen

HSQLGen

JDatabaseGen

MySQLGen

RMatrixGen

WSGen

RListGen

datainfrastructure


Communicationinfrastructure


Step 3: Use and evaluate

1: molgenis_db.xml

2: generate

3: use

Demo: http://wwwdev.ebi.ac.uk/microarray-srv/pheno/ Source: https://svn.gene.le.ac.uk/gen2phen/pheno-model


Step 4: optionally create plugins27


Alternative: existing database28

Existing db

Xml model

Generate

Connnects to

MolgenisExtractModel(automatic)

|[email protected]

2009

2009

|

Expected results

29


Result: Web GUI

1: molgenis_db.xml

2: brainstorm doc*

3: exchange format*

4: back and frontend*

Demo: http://wwwdev.ebi.ac.uk/microarray-srv/pheno/ Source: https://svn.gene.le.ac.uk/gen2phen/pheno-model

*autogenerated


Result: Documentation (generated-docs/objectmodel.html)

1: molgenis_db.xml

2: documentation*

*autogenerated

|[email protected]

2009

2009

|

Result: Exchange format

my GaP

researcherresearcher

my Data

my Data

Investigation, ProtocolMaterial, Data annotations

Raw and processed data

Between labs

Central access

32


Result: Exchange format

1: molgenis_db.xml

2: documentation*

3: exchange format*

*autogenerated

Tools•Db•Import•Export


Result: Exchange format1: molgenis_db.xml

2: documentation*

3: exchange format*

*autogenerated


REST interface (/api/find and /api/add)

› http://localhost:8080/molgenis_distro/api/find/example.Data?name=mydata

35


/ 36

Result: R interfaces(/api/R)

Swertz et al, CASIMIR consortium, GEN2PHEN consortium (submitted)

source(“http://localhost:8080/xgap/api/R”)MOLGENIS is connected

#download data traits <- get.metabolitedata(name=“mytraits”)25 metabolite downloaded in 30ms

genotypes <- get.markerdata(name=“mygenotypes")744 marker downloaded in 30ms

#calculate ...

#upload results for others to useadd.data(qtls, name=“myqtls”) 18.600 data items added in 2sec

36

|[email protected] | / 37

Smedley, Swertz, Wolstencroft et al (2008) Brief. in Bioinf.

Result: SOAP interface(/api/soap?wsdl)

|[email protected] |38

Connect to R statistics

Connect to R statistics

Workflow ready web-services

Workflow ready web-services

UML documentation of your model

UML documentation of your model

Edit & trace your data

Edit & trace your data

Import/export to Excel

Import/export to Excel

plugin your own scripts (OntBrowse)

plugin your own scripts (OntBrowse)

Tech keywords: object oriented data models, multi-platform java, tomcat/glassfish web server, mysql/postgresql database, Eclipse/Netbeans IDE, Java API, WSDL/SOAP API, R-project API, MVC, freemarker templates and css for custom layout, open source.

find.investigation()102 downloaded

obs<-find.observedvalue(43,920 downloaded

#some calculationadd.inferredvalue(res)36 added

|[email protected]

2009

2009

|

Under the hood

39


Architecture40

customize... Model file XML

Generate

APIs in Java, R, Web services and HTTP

MyScriptPlugins

FormGen

MenuGenTreeGen

PluginGenMatrixGen

JTypeGenJDBCMapGen

JListGenJReadCsvGen

HSQLGen

JDatabaseGen

MySQLGen

RMatrixGen

WSGen

RListGen

datainfrastructure


Communicationinfrastructure


Generators?

<#list entities as entity>

CREATE TABLE ${SqlName(entity)} (

<#list dbFields(entity) as f>

<#if f_index != 0>, </#if>

${SqlName(f)} ${mysql_type(model,f)}

<#if !f.nillable> NOT </#if>NULL

<#if f.getDefaultValue()?exists>${f.getDefaultValue()}</#if>

<#if f.auto && f.type == "int"> AUTO_INCREMENT</#if>

<#if f.type == "xref">

, FOREIGN KEY (${SqlName(f)}) REFERENCES ${SqlName(model.getEntity(f.XRefEntity).getRootAncestor() )} (${SqlName(f.XRefField)}) ON DELETE RESTRICT

</#if>

</#list>

<#list entity.getAllKeys() as key>

<#if key_index == 0>

, PRIMARY KEY(${csv(key.fields)})

<#else>

, UNIQUE(${csv(key.fields)})

</#if>

</#list>

<#list entity.getAllIndices() as i>

, INDEX ${SqlName(i)} (${csv(i.fields)})

</#list>

) ENGINE=InnoDB;

41


Generator results

CREATE TABLE Investigation (

`

id INTEGER NOT NULL AUTO_INCREMENT

, name VARCHAR(255) NOT NULL

, description TEXT NULL

, PRIMARY KEY(id)

, UNIQUE(name)

, UNIQUE(id)

) ENGINE=InnoDB;

42


Database API:

› db.Database- count(filter)- find(filter)- add(entity)- add(list)- update(entity)- update(list)- remove(entity)- remove(list)- beginTx(),commitTx()- add(CsvReader)- find(CsvWriter)

› db.Query- equals(field,value),- greaterThan(f,v)- lessThan(f,v)- limit(f,v)- offset(f,v)- or()- etc.

› util.Entity- find(db,filter)- get(db,id)- query(db)

43


//findDatabase db = new app.JDBCDatabase("molgenis.properties");for(Contact c: db.find(Contact.class)) System.out.println(c);

//queryQuery<Address> q = db.query(Address.class);q.like(“lastname",“Lops");for(Address t: q.find()) System.out.println(t);

//updateContact one = new Contact();one.setFirstName(“Joeri");db.add(one);

//update in batchList<Contact> many = new ArrayList<Contact>();for(int i = 0; i < 10; i++){

Contact c = new Contact();c.setLastName(“friend_"+i);many.add(c);

}db.add(many);


form

MVC user interface framework

Tree MVC triple

45

FormModel.java

(state)

FormView.ftl

(layout)

FormController.java

(logic)

request(target,action)

handleRequest()reload()

response

layout

Database API

main

Menu

Form

menu

form


Plugin: your own MV(C)

public class FileServingPlugin extends PluginModel(AndController)

{

public String getViewTemplate()

{

return "plugin/examples/file/FileServingPlugin.ftl";

}

public String getViewName()

{

return "plugin_examples_file_FileServingPlugin";

}

public void handleRequest(Database db, Tuple request)

{

//if( request.getAction("do_add") )

}

public void reload(Database db)

{

//Query q = db.query(Experiment.class);

//q.like("name", "test");

//List<Experiment> recentExperiments = q.find();

}

}

46

File with view

Name of view (next slide)

What do on request?

What do on page refresh/after

request?


Plugin: view in freemarker<#macro plugin_examples_file_FileServingPlugin screen>

<form method="post" enctype="multipart/form-data" name="${screen.name}">

<input type="hidden" name="__target" value="${screen.name}"" />

<input type="hidden" name="__action" />



<div class="formscreen">

<div class="form_header" id="${screen.getName()}">

${screen.label}

</div>

<#--optional: mechanism to show messages-->

<#list screen.getMessages() as message>

<#if message.success>

<p class="successmessage">${message.text}</p>

<#else>

<p class="errormessage">${message.text}</p>

</#if>

</#list>

<div class="screenbody"><div class="screenpadding">

<#--begin your plugin-->

Create your plugin layout code here.

<#--end of your plugin-->

</div></div>

</div>

</form>

</#macro>

47


Future

› ‘Cooler’ UI framework?- GWT, ServerFaces, YUI, Flex?

› Semantic integration?- /api/rdf, semantically enhanced model

› Large files- Hadoop, BinaryMatrix, …

› Computation/services/workflows- Galaxy integration, GridGain, …

› Documentation!

› Your wish list here

48

|[email protected]

2009

2009

|

Practical

49


Practical

› Copy and install VirtualBox (dvd/usb)› Copy and run molgenis-vm.vdi image› Pick up a manual› Start generating

50

|[email protected]

2009

2009

|

Acknowledgements

Morris SwertzJoris LopsJoeri van der VeldeRichard ScheltemaMartijn DijkstraRudi AlbertsDanny ArendsBruno M. TessonGonzalo Vera RodriguezTomasz AdamuziakJuha MuiluDamian SmedleyKaty WolstencroftAndrew R. Jones …and others

Ritsert C. JansenCisca WijmengaHelen E. Parkinson Carole GoblePaul Schofield John M. HancockKlaus Schughartthe NBIC/BioAssist consortiumThe LifeLines consortiumthe GEN2PHEN consortium (Mm)the CASIMIR consortium (Hs)the Brassica Nutr. consortium (At)The PANACEA consortium (Ce)

51

NLNLEBI



Add WSDL to Taverna


Add WSDL to Taverna


Add WSDL to Taverna


Find in MOLGENIS from Taverna

MOLGENIS For rapid prototyping of web applications biologists want to have

Documents