Building ontologies using Jenkins

Building ontologies using Jenkins

Changing requirements for ontology engineering

• Original ontology build pipeline– What pipeline?– Life on the bleeding edge: Editors version and public

version were identical• Ontology editing has become ontology engineering– We’re applying software engineering methodology– Automated test suites– Build process: classification using Elk, subset generation,

obo/owl, derived files– Versioned releases

Jenkins: a Continuous Integration Server

• Continuous Integration (CI):– Builds, tests executed upfront– Concept from agile programming– We’re applying to ontology engineering

• CI server:– Monitors source code repository– Integration tests triggered whenever

developer commits changes• Configurable

– Builds final product• Different solutions available

– Jenkins, Hudson

Example basic workflow• Local (e.g. EBI):

– Developer makes local modifications using OBO Edit

– Commits changes to SVN– (optionally) checks dashboard in web

browser

Example basic workflow• Local (e.g. EBI):

– Developer makes local modifications using OBO Edit

– Commits changes to SVN– (optionally) checks dashboard in web

browser

• build-go job:– Load main ontology– Import:

• disjointness axioms• Taxon constraints

– Launch reasoner: Elk or hermit– Write reasoner report– Fail if unsatisfiable classes found– Run additional perl checks, ensure external xrefs resolve, etc

• Server:– Jenkins polls SVN– External commit triggers Jenkins

to launch the build-go job (using Oort)

Example basic workflow

FAILSUCCESS

• Jenkins sends email alert to mail list• GO editor debugs using Protégé 4, fixes

using oboedit then recommits

• Write reasoner report• If previous build was fail, Jenkins

sends ‘service resumed’ email • Downstream jobs are triggered

• (e.g. bigger integrated builds, deployment)

Current status

• Jenkins is being used for validation– Invaluable for ontology developers– System relatively easy to set up and maintain

• Release process still (mostly) run by central cron– Only running limited subset of checks– Missing classification step– Need to consolidate

• Extend to annotations

Release pipeline changes• Current:

– Go_ext is direct nightly copy of editors file– Other files are filtered

• No inter-ontology links

• Immediate changes:– Editors file will include additional axiom tracking info

• This will be removed from all release versions– Using obolibrary standard versioning scheme– New obo2owl conversion

• Deprecate version that is built using MySQL pipeline• Encourage bioinformaticians to use this

– http://wiki.geneontology.org/index.php/Ontology_Release_Files_Proposal

Future changes

• Editors version will eventually switch to OWL– Advantages• owl:imports• Direct use of reasoning

– Disadvantages• Protégé lacks many OE features

Future changes

• Editors version will eventually switch to OWL– Advantages• Owl:imports• Direct use of reasoning

– Disadvantages• Protégé lacks many OE features

Thanks

• Heiko• Seth• Sebastian Bauer (Charite.de)• Ontology editors