Kettle Cookbook Pentaho Community Gathering 2010

8/2/2019 Kettle Cookbook Pentaho Community Gathering 2010

1/28

Roland Bouman: http://rpbouman.blogspot.com/ Twitter: @rolandboumankettle-cookbook: http://code.google.com/p/kettle-cookbook/1

Auto-documentationforKettle jobs and transformations

Kettle-Cookbook
http://rpbouman.blogspot.com/http://rpbouman.blogspot.com/


2/28

Roland Bouman: http://rpbouman.blogspot.com/ Twitter: @rolandboumankettle-cookbook: http://code.google.com/p/kettle-cookbook/

2

Thanks for attending!

Roland Bouman; Leiden, Netherlands

Ex MySQL AB, Sun Microsystems

Web and BI Developer

Co-author of Pentaho Solutions

...and Pentaho Kettle Solutions

Blog: http://rpbouman.blogspot.com/ Twitter: @rolandbouman
http://rpbouman.blogspot.com/http://rpbouman.blogspot.com/http://rpbouman.blogspot.com/http://rpbouman.blogspot.com/


3/28


3

Agenda

Documentation

Introducing Kettle-Cookbook

Demonstration Roadmap

Questions and answers

Links and resources


4/28


4

Agenda

Documentation




Links and resources


5/28


5

Documentation


6/28


6

Documentation


7/28


7

Documentation

j


8/28


8

Documentation


9/28


9

Documentation


10/28


10

Documentation


11/28


11

Documentation Benefits

Allows an ETL solution to be verified againstdesign documents

If done right, can help to train developers

Can be used to understand data lineage

Facilitate auditing processes


12/28

Roland Bouman: http://rpbouman.blogspot.com/ Twitter: @rolandbouman

kettle-cookbook: http://code.google.com/p/kettle-cookbook/

12

Documentation?Whaddya mean, documentation?


13/28



13

WTH isn't there anydocumentation?

Benefits are not immediate

Not popular w/ developers

Documentation Myths My software is self-explanatory

Documentation is always outdated

Who reads documentation anyway?


14/28



14

Documentation Myths:My Software is self-explanatory

I already explained, it's self-explanatory.

Software is only self-describing in the sensethat it may be clear *what* it does.

By itself, software cannot explain *why* itwas built this way.


15/28



15

Documentation Myths:Docs are always outdated

Yeah, documentation is always outdated.Let's blame documentation

Documenting should be part of the

development process You can test documentation like you can test

software


16/28



16

Documentation Myths:Who reads docs anyway?

Is theredocumentation?

Waddaya mean, yes?Well, *I* am not

going to read that

Yes

Find an excuseto not write any

Of course not!

No docs to read.self-fulfilling prophecy proved true

Well done! :)

Start

Who reads thatstuff anyway?


17/28



17

Agenda

Documentation




Links and resources


18/28



18

Kettle-cookbook:What is it?

A documentation generator for Kettle ETLsolutions

Built in Kettle

Inspired by Benjamin Kallman's Kettledocumentation generator (Mainz, 2008)

Open Source (LGPL)

Available on google code


19/28



19

Kettle-cookbook:How to use

While creating/designing, enter descriptions:

Job and Transformation Settings

Description

Extended Description

Job entry, Transformation Step:

Description

Run kettle-cookbook. Parameters: INPUT_DIR

OUTPUT_DIR


20/28



20

Kettle-cookbook:How it works

Kettle job scans a directory for .ktr and .kjbfiles creating an XML index

XSLT is applied to XML, outputs HTML


21/28



21

Kettle-cookbook:Features

Table of contents to navigate docs

Exposes value of description fields

Data flow Diagram Crosslinks

Overviews: Variables, Connections, Fields

Syntax highlighting (SQL, Javascript)


22/28



22

Kettle-cookbook:Hacking and Extending

It's built on Kettle. Change jobs andtransformations in the pdi directory to addcustom processing

Documentation generated with XSLT. Editthe kettle-report.xslt file to add customoverviews / HTML rendering

HTML uses externalized CSS and Javacript.Hint: you'll find it in the css and js directories

Icons in the images directory


23/28



23

Agenda

Documentation


Demonstration

Roadmap


Links and resources


24/28



24

Agenda

Documentation


Demonstration

Roadmap


Links and resources


25/28



25

Roadmap

High level data flow diagrams Overviews (variables, connections) across

ETL solution

Replace Kettle Job with Kettle API (BenjaminKallman)

Dependencies / where-used list

Not just ETL, entire Pentaho Solution (Actionsequences, Mondrian Cubes, Reports)

Data lineage


26/28



26

Agenda

Documentation


Demonstration

Roadmap


Links and resources


27/28



27

Agenda

Documentation


Demonstration

Roadmap


Links and resources


28/28



28

Links and resources Project: http://code.google.com/p/kettle-cookbook/

Getting Started: see the project wiki

Issues: http://code.google.com/p/kettle-cookbook/issues/list

Downloads: https://code.google.com/p/kettle-cookbook/downloads/list

Source: http://code.google.com/p/kettle-cookbook/source/checkout
http://rpbouman.blogspot.com/http://code.google.com/p/kettle-cookbook/http://code.google.com/p/kettle-cookbook/issues/listhttps://code.google.com/p/kettle-cookbook/downloads/listhttps://code.google.com/p/kettle-cookbook/downloads/listhttp://code.google.com/p/kettle-cookbook/issues/listhttp://code.google.com/p/kettle-cookbook/http://rpbouman.blogspot.com/

Kettle Cookbook Pentaho Community Gathering 2010

Documents