Continuous integration with business intelligence and analytics
Post on 16-Jul-2015
147 Views
Preview:
Transcript
1
Continuous IntegrationFor
Business IntelligenceAnd Analytics
Alex MeadowsPrincipal ConsultantCSpring
2
Agenda
● About Me
● Waterfall vs Agile
● Continuous Integration
● Software Release Process
● References
Feel free to ask questions at any point in the presentation!
3
Alex Meadows
● Worked in various industries (manufacturing, social/email marketing, consulting, software, agriculture)
● Open source agilest
● Loves data :)
4
Let's Talk Releases...
●What's your team's release candance?●How many bug releases go out afterwards?●How much new development gets cut?
5
Waterfall Approach
Source: http://covalentmarketing.com/blog/2013/02/25/software-development-lifecycle-waterfall-vs-agile/
6
Agile Approach
Source: http://www.mountaingoatsoftware.com/system/asset/file/17/ScrumLargeLabelled.png
7
Continuous Integration
● Release Early, Release Often!
● Bundle work together into cohesive “builds”
– Server Configuration
– Code
– Database
– Tests, tests, and more tests
– Approval Process
8
Continuous Integration
● Agile XP principal since 1994
● Used in software everywhere*
– Google● Tests 100 million test cases every day● Entire code base changes 50% monthly
– Amazon, Yahoo, Microsoft, etc.
* Used in mature agile software shops
9
Software Release Process
Source: http://www.360logica.com/blog/2014/06/continuous-integration.html
10
Plan
● Break tasks out into manageable chunks
– Can that ETL code be broken down by:● Source/Target● Table● Process
– Can that dashboard be broken down by:● Object● Type of data/visual/interaction
● Business must prioritize, team must commit
11
BI Challenges
● Work is difficult to break apart
– Most tools are designed for waterfall
– Complex moving components
12
Code
● Team works and develops “artifacts”
– “Code” - Reports, ETL, Visualizations, Database changes, etc.
– Documentation
– Research
13
Build
● Developed work is
– Run through unit tests
– Peer reviewed
– “Compiled” - tested for readiness for QA
– Pushed to QA
14
BI Challenges
● Coordination across disparate tools, components
● Version control doesn't coordinate between tools
● Database version control is hard to introduce
15
Test
● Verify all functionality, edge cases, and user requirements
● 10% manual, 90% automation work
● Automated gate keeper (pass/fail the build)
16
BI Challenges
● Test tools not written for many components
– Database – exists, but minimalist
– ETL – database tools and etlTest (shameless plug ;)
– Visualization – Selenium
– Reports – mix of the above
– Analytics – mix of the above
17
Release/Deploy
● Coordinate and push to next environment
● Approvals act as gate keeping mechanisim
● Cycle starts back and the code phase as needed
18
BI Challenges
● Coordination is possible, but challenging
● Tools might not provide integration points
19
Operate/Monitor
● Release into Production
● Monitor processes, logs, etc for aberrations
● Bugs and other issues are added into backlog for future releases
20
References
21
Sources
● http://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/2013-03-24%20Continuous%20Integration%20at%20Google%20Scale.pdf
●
top related