BeyeNetwork Open Source Research Report

8/14/2019 BeyeNetwork Open Source Research Report

1/33

BeyeNETWORK 1790 30th Street Suite 310 Boulder, CO 80301 303-339-7255 www.BeyeNETWORK.com

BeyeNETWORK RESEARCH REPORT

Open Source Solutions:

Managing, Analyzing and

Delivering Business Information

by

Mark Madsen


2/33

BeyeNETWORK 1790 30th Street Suite 310 Boulder, CO 80301 303-339-7255 www.BeyeNETWORK.com

TABLEOFCONTENTSAbouttheSurvey .......................................................................................................... 1

Methodology........................................................................................................... 1RespondentProfile ................................................................................................. 1

ExecutiveSummaryandKeyFindings........................................................................... 2Introduction .................................................................................................................. 4

APrimeronOpenSource ....................................................................................... 4CommunityVersusCommercialOpenSource........................................................ 4OpenSourceintheBusinessIntelligenceandDataWarehouseMarket............... 5

DetailedFindings........................................................................................................... 7Open

Source

is

Maturing ........................................................................................ 7

IndicationsofOpenSourceGrowthintheBusinessIntelligenceMarket........ 7UseisMainlyforNewProjects ......................................................................... 8

PeopleareUsingOpenSourceAcrossAllSoftwareCategories ............................. 9Databases........................................................................................................ 10DataIntegration.............................................................................................. 12BusinessIntelligence....................................................................................... 14Advanced

Analytics ......................................................................................... 16

WhoisUsingOpenSourceandHowareTheyUsingIt? ...................................... 17OrganizationSizeandtheUseofOpenSource .............................................. 17ScopeofUse.................................................................................................... 17WhatareOrganizationsBuying? .................................................................... 18UsebyConsultantsandSystemsIntegrators ................................................. 19RationaleforUse ............................................................................................ 19

ProblemsEncounteredWhenAdoptingOpenSource ......................................... 21InformationResourcesFoundUseful ................................................................... 24

Recommendations ...................................................................................................... 26ProfileofSurveyParticipants...................................................................................... 27


3/33

Open Source Solutions: Managing, Analyzing and Delivering Business Information Page 1 BeyeNETWORK and Third Nature 2009

About the SurveyThereportpresentsconclusionsandrecommendationsbasedonasurveyaboutopen

sourcesoftwareforreportingandanalytics.Itcoversallpartsofthedatawarehousestackfromthedatabasetoenduserdelivery.Itiswrittenforbusinessandtechnical

managerswhoareresponsiblefordeliveringreporting,businessintelligence(BI)or

analytics,whether

part

of

aBI

program

or

embedded

in

applications

and

websites.

Theresearchevaluatedtherationale,practicesandbenefitsthataredrivinguseofopen

sourceasanalternativetothetraditionalvendorsinthismarket.Italsolookedatthespecificsoftwareprojects,thescopeandstatusofitsdeployment,andthechallenges

andpracticesofparticipatingorganizations.

Thebusinessintelligence,reportingandanalyticsmarkethasdifferentdriversand

requirementsfromthetypicalITdevelopmentandapplicationsmarket.Mostopensourcestudiestargetopensourceimpactsonoperatingsystems,developmenttoolsand

applicationinfrastructure.ThepointoftheresearchwastogetabetterpictureofthefactorsinfluencingITadoptionintheBIanddatawarehousesegment.

Methodology

TheresearchforthisreportisbasedonasurveyandinterviewswithbothconsultantsandITprofessionalsthatThirdNatureconductedbetweenJulyandAugustof2009in

additiontosolicitedsurveyparticipationviatheBeyeNETWORK,sponsors'emaildistributionlistsandwebsitesandtheannualMySQLconference.Morethan1,000

peoplecompletedthesurvey,althoughnotallrespondentsansweredeveryquestion.

Theaimwastogatherabroadperspectiveoftheevaluation,useandpracticesinbothopensourcecentriccommunitiesandinthebroaderITmarket.

Respondent Profile

ThemajorityofsurveyrespondentsarecorporateITprofessionalsacrossfirmsofallsizes,withconsultantsbeingthenextlargestgroup.Thecompositionofrolesisshown

inFigure1.MostrespondentsareinNorthAmericaandEurope,with81countriesrepresentedinthesample.Computerhardware,softwareandservicecompaniesare

thelargestindustryrepresentedwith22%ofthetotal,withtherestspreadacross15otherindustrycategories.

Figure1:Rolesofsurveyrespondents


4/33


Executive Summary and Key FindingsVenturecapitalfloodingintoopensourcestartupsoverthepastseveralyearsresulted

inanexplosionofenterprisereadytoolsandapplications.Manyofthesestartupsarefocusedonthebusinessintelligencemarket.Opensourcerosequicklyinthe

informationmanagement

market,

from

almost

nothing

afew

years

ago

to

community

andcommerciallysupportedprojectsforeverypossibleuse.

Thegoalofthisreportistoexplainaspectsoftheusage,challengesandpracticesof

organizationsadoptingopensourceinthebusinessintelligenceanddatawarehousemarket.Keyquestionsexploredinthisresearchwere:

WhatorganizationsareusingopensourceintheBI/DWsegmentofthemarket?

Whatsoftwareisbeingdeployed? Whatarethebenefitsandchallenges?

Thesurvey

found

that

interest

and

adoption

are

widespread.

One

third

of

respondents

statedtheyuseopensourcereporting,dataintegrationordatabasesoftwareforanalyticuses.Morethanonethirdarecurrentlyevaluatingopensourcealternatives.

Only12%reportednoplanstouseopensource.

Thetopreasonforadoptionisstillcostsavings,althoughreducedvendordependenceandeaseofintegrationfollowedcloselybehind.Somecompaniesusedopensourcedeploymentsasameansofkeepingtheirincumbentvendorshonest.

Highlightsofthesurveyfindingsinclude:

Whendealingwithdatabaseperformanceproblems,peoplearemorethan

twice

as

likely

to

migrate

a

data

warehouse

to

an

analytic

database

astheyaretoadifferenttraditionaldatabase,opensourceornot.

Whilethisisgoodnewsforanalyticdatabasevendors,it'snotthatgoodbecausepeoplearestillmarriedtotheircurrentchoiceofdatabase.They

aremorelikelytochange,redesignorreplaceeveryothertoolintheBIstackbeforereplacingthedatabase.

Inallsoftwarecategoriesexceptadvancedanalytics,themostcommonlyusedopensourceprojectswerefromcommercialopensourcevendors.Theperceptionthatopensourceisdonelargelybyamateursand

volunteersisnottrueinthismarket.

Experiencebreedsadoption.Organizationswithlessthanoneyearofexperiencewithopensourceuseonlyoneopensourceproduct,i.e.,aBI

tool,whiletherestofthesystemisbuiltfromproprietarysoftware.Iftheorganizationhasbeenusingopensourcelonger,itislikelytobeusing

moretoolsindifferentcategories.Allorganizationswithmorethanthreeyearsofproductionuseareusingmorethanoneopensourceproduct.


5/33


Opensourceisallaboutnewprojects.Morethanhalftheusageofopensourcewasfornewprojects,withminimalfocusonreplacingexistingtools.Thisisgoodnewsforopensourceprojectsandvendors,and

potentiallybadnewsfortraditionalvendors.Itmeansopensourceisbeingadoptedinthegrowthareasofthemarket,andthatcouldbe

takingnewcustomersfromtraditionalvendorsortakingawaythemid

tosmaller

sized

organizations

which

have

previously

been

priced

out

of

themarket.

TraditionalBIanddataintegrationvendorshavebeenintroducing

midmarketprogramsastheylookforgrowth.Bylosingnewprojectsormidmarketcompanies,theyloserevenueinthefastestgrowingpartof

theBImarket.

There'safinelinebetweenacommunityeditionofanopensourceprojectandcrippleware,andsomevendorsarecrossingthatline.By

holdingbackfeaturesinthecommunityeditioninordertoenticepeopletopayfortheprofessionalversion,somevendorsareinadvertently

turningaway

customers.

Survey

respondents

complained

that

some

communityversionswerefeaturelimitedorscalelimitedtothepoint

wheretheycouldn'tbeusedonarealproject.

Theprimarycomplaintsaboutopensourcearerelatedtomaturityofthesoftware,lack

ofinternalskillsandavailabilityofconsultingservices.Giventherapidpaceofinnovationinopensourceprojects,thegapinbothcorefeaturesandmaturitybetween

opensourceandtraditionalvendorsisquicklyclosing.

Roughlyonethirdofopensourceusersarepurchasingservicesandsupportfromopensourcevendorstoday.Basedonthispatterninwhatislargelyanearlyadoptersegment,

expectthecommercialopensourcevendorstocontinuetheirgrowth.


6/33


Introduction

A Primer on Open Source

Opensource

software

(OSS)

is

released

under

alicense

that

differs

from

traditional

softwarelicenses.Thelicenseguaranteesseveralfreedoms:accesstothesourcecode,

theabilitysharetothecodewithothersandthefreedomtomodifyordeployitasyouwish.

Onemisconceptionisthatyoumustshareanychangesyoumaketothecode.Therequirementtoshareonlyappliesifyougiveorsellthesoftwaretoothersoutsideyourorganization.Ifyouredistribute,thenanychangesoradditionsyoumademustbeprovidedassourcecode.Ifyoudon'tredistribute,youdonotneedtoshareyourwork.

Opensourcesoftwareisavailableasaprojectwhichismaintainedbyacommunityof

peoplewhowritethecodeanddocumentation,providequalityassuranceandhelpto

managedistribution.

These

people

may

be

independent

volunteers,

contract

programmersortheymayallworkforasoftwarecompanythatmaintainsthesoftware.

Vendorsuseopensourcetoenableameansofsoftwareproductionanddistributionthatprovidesloweroperatingcostsandotherbenefitsbacktothevendor.

Community Versus Commercial Open Source

Therearetwomodelsofsupportforopensource.Oneiscommunitybasedopensource,oftencall"freeandopensource"orFOSSforshort.Theotheriscommercialopensourcesoftware,usuallyabbreviatedasCOSS.

Mostpeople

are

familiar

with

the

free

and

open

source

model

because

it's

been

around

thelongestandreceivesthemostpresscoverage.Inthismodel,volunteerscontributetheireffortstodevelopmentandmaintenance.Insomecases,theymaybefulltime

employeesofanonprofitorganizationowningthesoftware,buttheprojectdoesnotoperatelikeatraditionalsoftwarecompany.Thereisnoprofitmotive.Thesoftwareis

availablefreeofchargetoanyonewhowantsit.

Manyofthemythsaboutcommercialopensourcestemfromthemodel'sorigininthesharewareandFOSSworld.Intheearlydays,opensourcewasoftendesignedandbuiltforpersonalusebyindividualdevelopersratherthanaspartofanorganization'sITinfrastructure.

Becausemost

people

are

familiar

with

the

ideas

of

the

FOSS

model,

they

mis

apply

the

ideastotheCOSSmodel.Commercialopensourcecameaboutforadifferentreason.

CommercialopensourcevendorstrytomakemoneybyfillingthegapsintheFOSSmodel.

Enterpriseuseofopensourcewaschallengingintheearlydaysbecausetheproducts

lackedmuchofthefinishingworkcommercialproductsreceive.DocumentationinFOSSprojectsisoftenweak,qualityvariesandregularlyscheduledfixes,softwarereleasesandsupportareoftenlacking.


7/33


Commercialopensourceevolvedwithrecognitionthatcompaniesarewillingtopayfor

support,service,andotherlesstangibleitemslikeindemnificationorcertifyinginteroperability,forexampleofaBItoolwithagivenproprietarydatabase.

Acommercialopensourcevendorisjustlikeatraditionalsoftwarevendor,exceptthatthesourcecodeisnotshroudedinsecrecy.Thisenablesmoreanddeeperinteraction

between

customers

and

developers,

making

the

open

source

model

more

community

focusedthanthetraditionalmodel.

IncontrasttothemajorityofFOSSprojects,commercialopensourcevendorsemploy

mostoftheproject'sdevelopersandexpecttomakeaprofitwhiledoingso.Theyprovidethesameservicesandsupportthattraditionalvendorsdo,frequentlywithmore

flexibilityandlowercost.COSSvendorsuseelementsoftheproprietarymodelsuchasprovidingsupportcontractsorsellingnonopensourcecomponentsthatcanbe

purchasedinadditionto,orinplaceof,thefreeversionofthesoftware.

Thetwodifferentversionsofsoftware(communityandenterprise)cancauseconfusion.

Whenyouevaluatesoftwareitisimportanttonotewhetheryouarelookingatthefree

or

paid

version.

Somelessscrupuloussoftwarecompanieshaveobscuredthislineorarecalling

themselves"opensource"withoutanopensourcelicenseorwithsoftwarethatyoucangetonlyafterpayingthemforservices.Ifyoudon'thaveaccesstothesourcecodebut

theygiveyouasoftwareexecutableforfree,thentheyarereallyofferingafreetrialversion.Thetermsofyourusecanchangeatanytime.

UnlessthevendordeliverssoftwarewithsourcecodethatisunderanOSIcertifiedlicense,itisnotopensource.Thereisnoregulationoftermsorlabelssothese

"fauxpensource"vendorswillcontinuetooperateuntilthereisabacklash.

COSSvendorsarestillsoftwarecompanies.Ifyoupurchaseapaidenterpriseversionthen

you'll

find

that

the

experience

is

not

substantially

different

than

buying

software

fromaproprietaryvendor.ThekeydifferenceisthetransparencywithwhichCOSSvendorsoperate.

Asoneintervieweenoted,"Wecanseebugreportsandenhancementrequestsmadebyanyoneandhelpwithprioritizationbyvotingontheirimplementation.Thesameappliestofeaturesforupcomingreleases."Thisleveloftransparencyisnotoftenfoundinproprietaryvendors.

Open Source in the Business Intelligence and Data WarehouseMarket

Thereareopensourceprojectsavailableforeveryelementofthebusinessintelligenceanddatawarehousestack.Thisincludescoreproductssuchasdatabasesaswellasemerginganalyticdatabaseplatforms.

Theavailabilityofadvancedtechnologylikeanalyticdatabasessurprisesmanypeoplebecausethecommonassumptionisthatopensourceislowcost,lowfeaturesoftware.

Assuch,itappliestothecommoditymarketratherthaninnovativeoremergingtechnologies.


8/33


Commercialopensourcechangesthedynamicsofsoftwaredevelopmentbybringing

thesetechnologiestomarketasopensourcebeforetheyhaveachancetogothroughastandardproprietarygrowthphase.Thisaccelerationofthecommodityprocessisone

ofthebiggesteffectsopensourcebringstotheenterprisesoftwaremarket.

Thesurveyconductedforthisresearchaskedpeoplewhatopensourcesoftwarethey

are

using

or

evaluating

to

assess

the

popularity

of

projects

in

the

BI

market.

The

emergenceofcommercialopensourceaccelerateddevelopmentofthesoftwareand

opensourceadoptionoverthelastseveralyears.

Inthreeofthefoursoftwarecategoriesexamined,thetoprankedopensourcesoftwareisprovidedbycommercialopensourcevendors.Theholdoutcategoryisstillrelatively

newtomostorganizations,whetherusingopensourceornot.

FOSSandCOSSareavailableforeverypossibleBIapplication,fromtraditionalreporting

andOLAPtoolstoadvanceddataminingandstatistics.Evenmoreexotictoolslikeadvanceddatavisualization,simulationandwebbasedgeographicinformationsystems

areavailable.

Dataintegration

software

is

amore

recent

entrant

in

the

developer

tools

market,

with

ETL,dataqualityanddatafederationoptionsavailable.Thesurveyresultsshowedthat

thesetoolsarebeingappliedequallyinBIandtransactionprocessingenvironments.

Regardlessofwhatyouareseeking,itislikelythereisanopensourceprojecttofillyourneed.EnterprisecalibersoftwareisreadilyavailableforthecoredatawarehouseandBIcomponents.Theprimaryquestioniswhetherithasthefeaturesyouarelookingfor.

Softwaredevelopedasopensourceisnodifferentfromtraditionalcommercial

software.Thedifferenceliesinalicensethatgivesyoumorefreedomwiththecodethanaproprietarylicense.Thismeansyoushouldevaluateopensourcetoolsasyou

wouldanyothersoftware,byaskingwhetheritmeetsyourrequirementsatacompetitive

price.


9/33


Detailed Findings

Open Source is Maturing

Indications of Open Source Growth in the Bus iness Intelligence Market

Thesoftwareformostcategorieshasmatured,thanksinlargeparttoventurecapitalthatallowedcommercialopensourcevendorstoaddimportantfeaturesandfillin

majorgaps.Nowthattheproductsmeetmostorganizationsminimumfunctionalrequirements,thevendorshopetocapitalizeontheeconomicsbehindtheopensourcemethodofdeployinganddistributing.

Downloadstatisticsforthemostusedprojectshaveinmanycasessurpassedthemillion

mark.Whiledownloadsarenotagoodmeasureofproductionuse,theyareagoodindicatorofinterest.Ifevenafractionofapercentofdownloadsturnintoactiveusers,

theseproductswillhaveasmanyusersasthemajorvendors.Thedifferenceisinpaying

customers

approximatelyhalf

of

the

active

users

of

open

source

BI

and

data

warehousetoolsdidnotpurchaseanything.Onevendorclaimsthattheyhavemorethan300,000registeredusersofthesoftware.Whilethatclaimdoesnotindicateproductionuse,thesoftwareisfreelydownloadable

withoutregistration.Thismeanstheusersvoluntarilywentthroughtheprocessofregistration,soit'ssafetoassumethattheywereeitherdoingahandsonevaluationoractivelyusingtheproduct.Asingledigitconversionrateputsthemonparwithmanycompaniesthathavebeeninthismarketfortwiceaslong,aconsiderablegrowthrateforanystartupbusiness.

Thecurrentgrowthratelookslikeitwillcontinue.Roughly30%ofrespondentssaidthey

arecurrently

evaluating

or

piloting

open

source

in

one

of

the

four

software

categories

surveyed.About20%oftherespondentsindicatethattheyare"considering,"whichis

reallyjustanindicationofinterest.

WearestillinaveryearlystageofopensourceuseintheBImarket.43%oftherespondentstothesurveyarenotusinganyopensourceintheirBIenvironmentstoday.

Figure2:LengthofuseAlmosthalfofthosewhoareusingopensourceinaproductionsystemhavebeendoingsoforlessthanoneyear,asshowninFigure2.Thisindicatesthatmanyarestillontheinitiallearningcurve,andthepeoplerunninginproductiontodayshouldbeconsidered

earlyadopters.

46%

31%

23% Lessthan1year

1to3years

More than 3years


10/33


Anindicatoroffuturegrowthisthatexperiencewithopensourcebreedsnewadoption.

Thereisadirectcorrelationbetweentheyearsofexperiencewithopensourceandthenumberofdifferenttypesoftoolsinuse.

Noorganizationwithlessthanoneyearofuseisdeployingmorethanoneopensourceproduct.Somerespondentssaidtheywantedtoseehowwellthefirsttoolworked

before

deciding

whether

to

replace

any

of

their

other

tools.

Allorganizationswithmorethanthreeyearsofusehadatleastthreedifferentopensourcetoolsinplace.

Theuseofopensourcefordatadeliveredorobtainedoutsidetheorganizationisanotherareashowingstrongresponse.14%ofrespondentsaredeliveringinformationexternally.Inthisgroup,morethantwothirdsareusingopensourceBItoolsinsteadoftraditionalvendors'products.

Use is Mainly for New Projects

Onequestionposedwas,"Arepeopleusingopensourcetoreplaceexistingtools?"As

Figure3shows,

tool

replacement

and

the

retiring

of

custom

code

are

happening

at

arelativelylowrate,andthedominantuseisnewprojects.

Thisisnotgoodnewsfortraditionalvendorsinthemarketbecauseitmeansthattheyarebeingforcedtocompetefornewrevenue.Theseprojectsarethelongtermrevenue

streamanyvendorneedstosustainitself.Thisisgoodnewsforopensourceprojectsandvendorsbecauseitmeansthatopensourceisbeingadoptedinthegrowthareaof

themarket.

Figure3:CurrentorexpectedscenariofordeploymentbysoftwarecategoryOpensourcecouldbetakingnewcustomersfromtraditionalvendorsinthesmalltomidmarket.Thisisanareathey'vebeenlookingtoforgrowth,introducingnewsmallandmidmarketproductsandsalesprograms.Bylosingnewprojectsormidmarket

18%

10% 10%

13%

50%

14%15%

10%8%

53%

16%

11%14%

18%

41%

25%

18%

14%

7%

36%

Replacingproprietary

software

Replacinginternally

developed software

Supplementinga

systemwithsimilar

features

Addingnew

functionalitytoan

existingsystem

Usingaspartofa

newsystemor

project

Database DataInteg ration BI Adv.Analytics


11/33


companies,theylosealongtermsourceofrevenueinthefastestgrowingpartoftheBI

market.

Theotherpossibilityisthatopensourceisprovidingcapabilitiestoanunserved

segmentofthemarket.Traditionalvendors'productshavebeenpricedtoohighformostsmallandmediumsizedcompanies.Thissampledoesn'tprovideaclearanswer.It

appears

that

a

little

of

both

exists.

People are Using Open Source Across All Software Categories

UseisdistributedacrossalllayersoftheBIstackandfollowsapatternalignedwith

maturityofthesoftwareandthemostcommonuses.Fewerorganizationshaveboththesophisticationandtheneedforadvancedanalyticsorforembeddinginteractive

reportsintoanapplication,sotheseshouldshowsomewhatloweruse.

Thesoftwarewasdividedintofourcategoriesbasedonthelayersintraditionalbusinessintelligenceordatawarehousestacks:database,dataintegration,andreportingand

analyticstools.Becausetherearesomanydifferentfrontendtools,theywerefurther

separatedinto

business

intelligence

(reporting,

OLAP)

and

advanced

analytics

(data

mining,statistics,visualization).SoftwareusebycategoryisshowninFigure4.

Figure4:InterestanduseofopensourcebysoftwaretypeInterestinallcategoriesisstrongandgrowingasshownbythenumberoforganizations

inthecurrentlyevaluatingphase.Oneinterestingfindingfromthesurveyisthat

experiencewithopensourceleadstoincreasedadoptionofotheropensourcetoolsina

sortofvirtuouscycle.

Thismeans

we

should

expect

to

see

more

use

of

open

source

in

the

BI

and

data

warehousestackasmorecompaniesgainexperience.It'salsoasigntheproprietary

vendorswillfacemoreopensourcecompetitioninthefuture.

Thisremainderofthissectionexamineseachofthefoursoftwarecategoriesinmore

detail.

Figure7:Purchasesrelatedtoopensource

5%

14%

18%

18%

8%

8%

12%

13%

18%

22%

17%

18%

43%

37%

31%

29%

26%

19%

22%

22%

Advancedanalytics

Businessintelligence

Dataintegration andETL

Database

Inproduction Prototypeorp ilo t Ev alu at ing Co nside ring Noplans


12/33


Databases

Productionuseofopensourcedatabasesshowsthegreatestuseinthismarket.Opensourcedatabaseshavebeeninexistenceformanyyears,whilemanyprominent

projectsinothercategoriesarelessthanfiveyearsold.

Thenatureofanalyticworkloadsisholdingbackopensourcedatabaseadoption.Most

ofthe

engineering

effort

for

OSS

databases

is

focused

on

transaction

processing.

Analyticuserequiresbetterhandlingofcomplexqueries,largesinglequerydatavolumesandvariableuserconcurrency.Overall,interviewdatashowsthatopensourcedatabaseusewouldbehigherifitweren'tforpooranalyticsupportcapabilitiesandlowerqueryperformance.

How Much Data is Being Stored and Used?

ThechartinFigure5showstheresponsestothequestion,"Howmuchdataisbeing

storedoraccessed?"Thischartshowsthesizeregardlessofdatabasetype theconstraintisthatpeopleareusingopensourceinsomepartofthedatawarehouse

stack,soanopensourceBItoolaccessingaproprietarydatabasewouldbeincluded.

Figure5:Datavolumedistribution(includesopensourceandproprietarydatabases)Thedistributionisthesameaswhatweseeintheoveralldatawarehousemarketwhere

approximately60%ofdatabasesarelessthanoneterabyteinsize.Thissurveyshowsmost(67%)ofrespondentsareaccessinglessthanaterabyteofdataandaboutone

third(38%)lessthan100GB.

Whilemostofthesample(82%)isusingproprietarydatabasessuchasOracleorSQLServer,18%ofthesurveyrespondentsareusingopensourcedatabaseslikeMySQL

andPostgrestostoreandquerytheirdata.

Opensourcedatabasesarestoringlessdatathantheirproprietarycounterparts.Inthis

sample,76%oftheopensourcedatabasesrunninginproductionarelessthanaterabyteinsize.Evenso,36%aremorethan500GBinsize,whichisnotinsignificantfor

manyorganizationstoday.

24%

14% 14%

15%

13%

4%

1%

3%

Lessthan

50GB

50to


13/33


Databases and BI Performance

Databasesizeshaveincreasedsignificantlyacrossallindustriesduringthepastfewyears.Sizeandgrowtharecriticalfactorsaffectingqueryperformance,butquery

responsetimeatthesubterabytescaleisstillachallengeformanyorganizations.

Thissurveylookedatperformancebecauseonestatementaboutopensourceisthat

thesoftware

is

slow

when

compared

to

proprietary

counterparts.

Thesurveyresponsesshowthatroughly30%ofusersreportedproblemswithperformancerelatedtodatasizeandscalabilityorwithuserconcurrency.Overall,these

complaintsfellbelowotherproblemsdiscussedintheconcludingsections.Figure6showsthedistributionofperformancecomplaints.

Figure6:ComplaintsrelatedtoopensourceperformanceQueryperformancedominatescomplaints.Thisisnotasurprisebecausequeryisthe

mostvisibleelementandaffectsthelargestnumberofpeople.It'salsothemostdifficulttodiagnosebecausetherearemanydesignandtechnologyfactorsthatcanaffectquery

speed.

Gettingdataloadedturnsouttobelessproblematicformostpeople.ThattheseproblemsaresolowisasurprisebecausemeetingbatchETLwindowshashistorically

beenamajor

complaint

in

data

warehouse

and

data

mart

projects.

The

standard

solutionsfortheseproblemsapply:databaseandapplicationtuningandredesign.

Open Source Database Use

Thehighprofileanddifficultyofgettinggoodqueryperformanceisoneofthemajor

factorsdrivingITcustomerstolookatspecializedanalyticdatabases.ThisisparticularlytrueiftheyareusingstandardopensourcedatabaseslikeMySQLbecauseofthelackof

datawarehousespecificscalingandperformancefeatures.

Analyticdatabasesincludehorizontalscaleoutdatabasesthatgrowbyaddingmorecomputersratherthanresourcestoasinglelargecomputer,columnardatabasesand

specializeddatawarehouseappliancesthatmarryopensourcesoftwaretohardware.

SeveralofthenewopensourceanalyticdatabasesappearintheresultsshowninFigure7,whichlistsalloftheopensourcedatabasespeoplereportusinginproduction.MySQL'sdominanceisnotasurprisegiventhatitisthemostpopularopensource

databaseonthemarket.MySQL'spopularitybodeswellforMySQLcompatibleandMySQLengineproductsaimedatanalyticworkloads.

33%

33%

37%

69%

Poorbatchreporting performance

PoorETLordataintegrationperformance

Poorperformanceloadingdata

PoorinteractiveBIoranalyticsperformance


14/33


Figure7:OpensourcedatabasesinuseWhenfacedwithdatabaseperformanceproblems,ifthechoiceistomovefromMySQL

toanexpensivetraditionaldatabase,lessexpensiveanalyticdatabasesbecomemoreappealing.Thisisborneoutinthesurveydatawhichshowsthatpeoplearemorethan

twiceaslikelytomigrateadatawarehousetoananalyticdatabaseastheyaretoa

differentdatabase

in

the

same

class.

Becausemostopensourceanalyticdatabasesareaimedatdatabaseslessthan5terabytesinsize,theyalignwellwiththebulkofthedatawarehousemarketand

particularlywiththeopensourcedatabasemarket.

Data Integration

Asacategory,dataintegration(DI)toolsarealmostascommonlyusedasdatabases,

outrankingbusinessintelligencetools.Giventherelativeyouthoftheseprojects,itissurprisingthattheyareascommonastheyare.Thecommerciallysupportedopen

sourceintegrationtoolshavebeenavailableforamuchshortertimethanopensourceBItools.

Theinvestmentsincommercialopensourcetoolshavehadasignificantimpacton

productmaturity,makingmostofthesuitablefororganizationslookingfor"goodenough"tools.Theseproductssupportallthebasicsneededindataintegrationprojects.

2%

2%

2%

3%

3%

3%

3%

7%

7%

8%

10%11%

44%

75%

Bizgres

Kickfire

LucidDB

MonetDB

SQLite

CouchDB

Palo

Firebird

Ingres

BerkeleyDB

EnterpriseDBInfobright

Postgres

MySQL


15/33


Theirweaknessesareintheareasofadministration,teamsupportandadvanced

integrationfeaturessuchasdataqualityfunctionsdealingwithsemistructureddata.Theopensourcetoolsareprimarilysinglepurpose,liketheearlyETLtools,although

Talendhasbeenextendingitsproductlinewithdataqualityandmasterdatamanagementfeatures.

Figure8:UsesofopensourceintegrationsoftwareAfactorincreasingthepopularityofthiscategoryistheuseofthesetoolsforoperationalaswellasanalyticdataintegration,whichranksassecondmostpopularin

thelistshowninFigure8.

Therearetwodifferentwaystousedataintegrationtools:linkingtransactional

applicationsorfeedingdatatobusinessintelligencesystems.Theseusesaffecttheapproach,methods,featurerequirementsandbesttoolsforthejob.

BIsystemsaremostoftenloadedinbatchcyclesaccordingtoafixedschedule,bringing

datafrommanysystemstoonecentralrepository.Theyhaverelativelylargevolumesof

data

to

process

in

a

short

time,

but

have

little

concurrent

loading

activity.

Most

data

integrationproductswereoriginallydesignedtomeetthespecificneedsoftheanalyticdataintegrationmarket.

Mostapplicationintegrationprojectsneeddatafromoneortwoothersystems,notthemanysourcesandtablesfeedingadatawarehouse.Thescopeisusuallysmaller,with

lowerdatavolumesandnarrowersetsofdatabeingtransferredandminimaltransformationrequired.

Thesedifferencesinfrequencyofexecution,datavolume,latencyandscopeofsources

aretechnicalelementsthatdifferentiateoperationalandanalyticdataintegration.Dataintegrationisasmallelementofanapplicationproject,unlikeadatawarehousewhere

DI

may

consume

80%

of

the

project

budget

and

timeline.

Handcodingiscommoninapplicationprojectsbecausedataintegrationisthoughtofin

termsofapplicationglue.InBIprojects,handcodingismostoftenawaytosavemoneyonthehighcostofenterpriseETLproducts.

Communityopensourcedataintegrationtoolscanprovidethecostadvantagesofhand

codingwiththeproductivityadvantagesoftraditionaldataintegrationsoftware.ThemostpopularopensourcedataintegrationtoolsinthesurveyarePentahoDI/Kettle,

TalendandJitterbit,allcommerciallysupportedproducts.

8%

10%

15%

15%

21%

30%

LowlatencyETL foradatawarehouseormart

Masterdatamanagementefforts

Dataqualityefforts

Datamigrationefforts

Operationalintegration

BatchETL foradatawarehouseormart


16/33


PentahoandTalendmakeup75%oftheuseinthiscategory.Oneinterestingelementis

thebreakdownbyuse.WhilebothareusedforETL,Talendismorelikelytobeusedforoperationaldataintegrationthananyoftheothers.Thefulllistoftoolsandtheirlevel

ofuseisshowninFigure9.

Figure9:OpensourcedataintegrationtoolsinuseThesetoolsareestablishedinthedevelopermarketwhichhasbeenthetraditional

strongholdofopensourcesoftware.Expectopensourcetobeakeyelementofdataintegration(andespeciallyofoperationaldataintegration)inthefuture,similartoopen

sourceuseinapplicationdevelopmentenvironmentstoday.

Business Intelligence

AdetailedbreakdownoftheuseofBItoolsthatareinproductiontodayislistedin

Figure

10.

Traditional

reporting

and

dashboards

are

the

most

popular

uses.

This

mirrors

whatweseeninwithnonopensourceBItoolsinthemarket.

Figure10:BreakdownofproductionusefortheBIandreportingtoolscategoryApplicationreportingandembeddingarealmostaspopular,somethingyoudonotseewithmostofthetraditionalBItools.Thisispartlyduetothetechnicalproblem:

marryingfeaturestoanapplicationrequireseasycomponentintegrationor

14.6%

15.2%

15.9%

16.5%

17.1%

20.7%

OLAP

Reportsembedded inanapplicationorwebsite

Reportingagainstanapplicationdatabase

Enduserorinteractivereporting

Dashboardsorscorecards

Staticreports

2%

2%

2%

5%

5%

8%

13%

33%

42%

Clover

OpenDataQuality

OSDQ

Apatar

RedHat

DataCleaner

Jitterbit

Talend

PentahoDI

/Kettle


17/33


customizablecode.MostproprietaryBItoolsarebuilttobestandaloneapplications,

makingtheminappropriateforthistypeofuse.

Forsoftwareproviders,anotherintegrationneediscustomizingtheBIinterfacetomatchtheinterfaceoftheapplication.Softwareprovidersalsohavetoworryaboutthe

incrementalcosttotheproduct.Leveragingopensourcecanbeazeroorlowcost

alternativeto

using

BI

tools

from

one

of

the

proprietary

vendors.

Oneintervieweedeliveringsoftwareasaserviceapplicationsstatedthat"Without

opensourceBItools,Iwouldnothavebeenabletoprovidereportingcapabilitiesinmyapplications.MymarginsaretoonarrowtolicenseBItoolsfromoneofthebigvendors.They'realsomoredifficulttomanageinamultitenantenvironment."

SeveralrespondentsusingopensourceBItoolsembeddedintheirapplicationssaidthat

theychosethisalternativebecauseitofferedneutrality.BecauseoftheBImarketconsolidation,partneringwithoneofthemajorBIvendorscouldalienatecustomers

investedinacompetingvendor'sapplications.

PentahoandJaspersofttogetherareusedbythreequartersofthesurveyrespondents.Thenumbersdonotaddupto100%becausepeopleinmanyorganizationsareusing

morethanonetool,oftenforcomplementarypurposes.ThelistofprojectsisshowninFigure11.

2%

2%

5%

5%

9%

14%

19%

26%

28%

47%

OpenReports

Palo

MarvelIT

Openl

SpagoBI

Jfree

BIRT

Mondrian(aPentahoproject)Jaspersoft

Pentaho

Series1

Figure11:OpenSourceBItoolsinuseOneelementofopensourceBIthatdoesn'tappearinthesurveydataisthepeoplewho

chosetodeveloptheirownsoftwareusingopensourcecomponents.ThereareSQL

generators,userinterfacecomponents,graphinglibrariesandalltheotherelements

neededforadoityourselfmodel.


18/33


Thisistheapproachtakenbyanumberofwebbasedcompaniesandgovernment

agencieswherethenumberofusersisveryhighandtheinformationdelivery

capabilitiesarewelldefinedorconstrained.Whilethisisarelativelysmallpercentageof

thesample,it'susefultoknowbecause14%oftherespondentsmentiondeliveringdata

toexternaluserswithinthescopeoftheirdeployments.

Advanced Analytics

Thisadvancedanalyticscategoryisacombinationofdifferenttypesofsoftwarethatfall

outsidethenormal"queryandreporting"realm,includingstatistics,datamining,

visualization,modelingandsimulation.

Eachtypeofsoftwareisdifferent,buttheyallsharealowoveralladoptionrate(5%in

productioninoursample).Thelowadoptionrateismostlyduetothelowerapplicability

ofthesetoolsformanyorganizations,aswellasthelevelofanalyticalsophistication

foundinbusinessesratherthanthefactthatthetoolsareopensource.Themost

popularproductsinusebysurveyrespondentsareshowninFigure12.

2%

3%

4%

4%

7%

8%

8%

23%

42%

46%

Cytoscape

Taverna

Axiis

Processing

Orange

Graphviz

Knime

RapidMiner

Weka(aPentahoproject)R

Figure12:Opensourceanalyticstoolsinuse

TheRprojecthaslongbeenpopularforstatistics,soitsappearanceatthetopofthe

chartisnotasurprise.Wekadataminingsoftware(aPentahoprojectandpillarofits

stack)

has

also

been

in

existence

for

a

long

time

and

is

often

used

in

university

settings,

aidingvisibility.Unlikealltheothercategories,thetoptwotoolsarecommunityopen

sourceprojects.RapidMinerandKnimearecommercialopensourceproductswith

freelyavailablecommunityeditions.

Thereisatremendousamountofanalyticssoftwareavailableasopensource.The

challengeformostorganizationsisthatthetoolsareeithersinglepurpose,forexample


19/33


tiedtoaspecifictechnique,ortheyareavailableaslibrariesofcoderatherthantools.

Thisisthecasewithmostdatavisualizationsoftware.

Who is Using Open Source and How are They Using It?

Organization Size and the Use of Open Source

Onepersistentmythisthatsmallcompaniesaretheprimaryusersofopensource.Whiletherearemoresmallorganizationsusingopensourcetodaythanmidsizedor

large,asshowinFigure13,thedataalsoshowsthatmediumandlargeorganizationsaredoingmoreevaluations.Thisisachangefromanearliersurvey,wheresmallcompanies

wereleadinginbothareas.

ThechangeisinterestingbecauseitreflectsashiftinmidsizeandlargerorganizationsasusersofopensourceBIanddatawarehouseproducts.Whilethesmallcompanybase

isimportant,theproductsappeartobegoodenoughtomeetthemorestringentrequirementsoflargerorganizations.

Thisisgoodnewsforcommercialopensourcevendorsbecausethelargestgroupofproductionusersweretheleastlikelytopayforsupportorservices.Ascompanieswith

abetterabilitytopaymoveintothemarket,therevenuegrowthforopensourcevendorsshouldincreaseandwithitthequalityofserviceandsupport.

Figure13:OpensourceuseandevaluationbysizeoforganizationScope of Use

OnecommonbeliefaboutanalyticsandBIprojectsisthatopensourceismorelikelyto

be

used

by

departments

in

large

organizations

and

across

the

company

in

smaller

organizations.

Figure14showsthatsmallorganizationsaremorelikelythanmediumandlargetodo

companywidedeployments,andlargeorganizationsaredoingsmallerdeployments,supportingthisbelief.

Smallcompaniesanddepartmentsoflargeorganizationssharesimilarcharacteristics:theyareoftenconstrainedbybudget,theyhaveasmalleruserbaseandtheirusageis

moreuniform,makingsmallerdeploymentseasier.

38%

23%

41%

23%

37%

32%

Evaluating

UsingSmall

Medium

Large

Small

Medium

Large

Small

Medium

Large


20/33


Figure14:ScopeofopensourceuseDespitethisgeneralpattern,thereareenterprisewidedeploymentsofopensourceinlargeorganizations.40%oflargeorganizationsplantoorhavedeployedaBIorDW

applicationcorporatewidewithsomeopensourcecomponents,demonstratingalevelofsoftwarematurity.

What are Organizations Buying?

Morethan

half

of

the

organizations

did

not

purchase

anything.

They

are

using

the

free

andcommunitysupportedversionsofthesoftware.Thecurrentdatareversedone

conclusionfromapriorsurvey:smallerorganizationpurchaseataboutthesamerateaslargerorganizations.Whatthenewdatashowsisthatsmallerorganizationsarelesslike

topayforprofessionalorenterpriseversions.Figure15showsthepurchasesbrokendownbyorganizationsize.

Organizationsizeisreflectedinpurchasingprofiles.Largerorganizationsaremorelikelytopurchasetraining.Interviewswithsurveyrespondentsfoundthatlargerorganizations

havemorecomplexprojectswithmoreusers,placinganemphasisoninternalfirstlinesupport.Theyalsotendtodedicatemorebudgettostaffdevelopment.Differentforms

of

maintenance

and

support

appeal

to

different

buyers

as

well.

Mid

sized

and

large

companiesaremorethantwiceaslikelyassmallcompaniestopurchasesubscriptions.

Figure15:Purchasesbysizeoforganization

27%

38%

32%35%

40%

27%

Department orDiv ision Corporatewide

Small Medium Large

21%

31%

13%

6%

9%

14%

33%

31%

23%

24%

22%

29%

33%

28%

30%

52%

28%

36%

45%

38%

38%

58%

53%

54%

Large

Medium

Small

Nopurchasee

Maintenanceorsupportcontract

Training

Consultingorinstallationservices

Phone,emailoronsitesupportfromthevendor

Commerciallicense

Phone,emailoronsitesupportfromathirdparty

Subscriptiontovalueadded,enterprise features


21/33


Thisinformationshouldhelptobudgetforanopensourceimplementation.Basedon

thesizeofyourorganization,youcanseewhatthemostcommonpurchasesareandcheckthepricesontheseitems.

Use by Consultants and Systems Integrators

Manypeoplebelievethatconsultingfirmsandsystemsintegratorsaremorelikelyto

useopen

source

because

it

allows

them

to

be

more

competitive.

They

gain

an

edge

by

savingcustomersmoneyonsoftwarelicensesorbyhavingmorecustomizabletoolsforprojects,thuspricingthemselvesundercompetitorsorprovidingabetterfitwithclient

needs.Theotherhopeisthatbyfreeingprojectbudgetfromthesoftwarelicenses,thiscouldtranslateintomoremoneyspentonworkwiththeconsultants.Whilethesepointsareallvalid,thesurveydatadoesnotsupportthatbelief.ConsultantsarelesslikelythanITprofessionalstouseopensourcetoolsinthisspace

(10%forconsultantsto36%forIT).TheusagebyrespondentroleisshowninFigure16.

Figure16:OpensourceusebyITandconsultantsItisnotablethat49%oftheconsultantsandsystemsintegratorsareevaluatingopensourcesoftwaretoday,signalingapossibleshiftintheiruse.

Whatthedatasaysisthat,farfromleadingthetechnologymarket,systemsintegrators

(SIs)andconsultantsseemtotrailit,followingthemoneyratherthanleadingtheircustomersininnovation.Interviewssuggestthatmostlysmallerlocalorregional

consultingcompaniesareprovidingservicesusingopensourcesoftware.

Evenwiththesuddenriseisevaluation,consultantsandSIssignificantlytrailITdepartments.IfyouareinanITorganizationthatreliesonconsultantsforprojectwork

thenusing

open

source

tools

will

require

that

you

factor

the

availability

of

qualified

consultantsintoyourdecision.Giventhesestatistics,theyarelikelytoberarerthanyouexpect.

Rationale for Use

Costsavingsisthenumberonereasongivenforopensource,regardlessofmarket.Fortunatelyforthecompaniesinvolved,costsavingsisalsothenumberonereported

benefitafterdeployment.TheresultsshowthattheinitiallicensecostisabiggerobstacleinanalyticsandBIprojectsthanthetotalcostofownership(asreflectedin


22/33


ongoingmaintenancecosts).Figure17showsthetopreasonsgivenforchoosingopen

sourceproducts.

Reducedvendordependenceissurprisinglyhighinthelistofreasons.Thebenefits

anticipatedaremorethantheobviousavoidingofavendor'stechnologylockin,forexample,therequirementthatonerunWindowsandSQLServerinordertouse

Microsoft's

BI

tools.

Also

mentioned

were

more

options

to

resolve

problems,

communitysupportreducingtherequirementforvendoraidandusingopensourceto

offsettheeffectofvendoracquisitions.

Figure17:ReasonsforusingopensourceThe

business

intelligence

and

data

warehousing

market

has

seen

several

years

of

steady

consolidationacrossallsoftwarecategories.Thisconsolidationmakesitincreasinglylikelythataformerlymultivendorinstallationisnowentirelydependentonasingle

vendor.Manymanagersviewhavingalloftheirtechnologydecisionsinthehandsofasinglevendorasarisk.

Inlightofrecentpriceincreasesandrestrictionsimposedbyvendors,usingopensource

isprovingtobeawaytoreducedependenceandbalancetheriskofmorevendoracquisitionsorunilateralactionslikeraisingpricesonacaptivecustomerbase.

Otheradvantagesofopensourcesoftwarearetheeaseofadjustingdeployments,forexampleaddingordroppingenduserlicensesandcustomizingorextendingittofit

specificproject

circumstances.

Neither

of

these

is

as

simple

with

traditional

software.

Oneitemrelatedtorationaleiswhatinfluencedpeople'sdecisions.Theanswers(showninFigure18)reinforcethesocialandcommunityaspectsofopensourceandtheweb.

Productreviewsandpeerfeedbackwerebelievedtobethemostinfluentialitems.Onecaveattothedatashownisthatisaperilousquestiontoanalyzebecauseinfluenceis

oftenrationalizedafterthefactandpeopleareoftenunawareofwhatisreallyinfluencingtheirdecisions.

28%

28%

32%

32%

32%

33%

43%

44%

48%

66%

Accesstothesourcecode

Extensibility,customizabilityofsoftware

Opendevelopmentprocessandroad

Easiertoevaluateorprocure

Speedofinnovationofthesoftware

Flexibility

in

deployment

Lowermaintenancecosts

Reduceddependenceonavendor

Openstandards

Loweracquisitoncosts


23/33


Figure18:FactorsthatinfluencethechoiceofsoftwareFamiliarity

breeds

success

the

number

three

item

was

developers'

prior

success

with

opensourcesoftware.Opensourcevendorswerealsoratedastrustworthy,appearingaheadofconsultantsandindustryanalysts.

Oneitemthatwasnotlistedbutappearedfrequentlyinsurveycommentswasthehandsontestingevaluationandtestingpeopledidintheirowntime.Withopensource

tools,evaluationswereeasiertodothanwithproprietarysoftwarebecausetherewerenorestrictions.Itwasreportedtobemuchhardertoobtainevaluationcopiesof

softwarefromproprietaryvendorswithoutfirsttalkingtoasalesteam.

Problems Encountered When Adopting Open Source

Whilemostrespondentsprovidedfavorableinformationregardingopensource,itisnot

withoutproblems.Figure19showsthat47%ofrespondentsreportedatleastonefailedopensourceevaluation.Aftergoingintoproduction,42%reportnomajorproblems.

Figure19:Responsesto"Didanyopensourcesoftwarefailyourevaluation?"Whilethereweremanydifferentreasonscitedforthefailedevaluations,reasonsclusteredaroundseveralkeyissuesshowninFigure20.

Whiledocumentation

complaints

showed

up

at

the

bottom

of

the

list,

they

were

the

biggestwriteincomplaintinthesurvey,indicatingthatthisstronglyaffectssome

people.Documentationissomethingcommunitybasedopensourceprojectsoftenstrugglewith.It'soneofthegapsCOSSvendorsaretryingtofilltomakethesoftware

moreenterprisefriendly.

Yes 47.0% No 53.0%%respondents

21%

24%

27%

29%

31%

33%

34%

37%

47%

Informationfromfriends

OpinionsofITanalystfirms

Conmmunity sizeoractivity

OpinionsofindependentITanalysts

Recommendationsfromconsultants_systemsintegrators

Informationfrom

open

source

vendors

Priorsuccesswithopensourcesoftware inothersystems

Feedback fromprofessionalpeers

Productreviews


24/33


Difficultygettinganswerstoquestionsorproblemsrankedinthetopten.Thisruns

countertothemessagethatopensourcevendorsandcommunitiesaremoreresponsiveandquickertoanswerquestions.Wedidn'tcomparethisbetweenopensourceandproprietaryvendors,sowecan'tconcludethatit'sanyworsethanthesituationwithnonopensourcevendors.However,thisisanindicationthattherecanbe

challenges,and

that

there's

room

for

vendors

to

improve.

Figure20:ProblemsencounteredwithopensourceSoftwareinstallationisthebiggestsourceofproblems.Oftenthecausesstemfromthecomponentnatureoftheopensourceproducts.Therecanbemorediscreteelements

toconfigurewithintheenvironment.Traditionalsoftwarecomponentsareusuallypreintegrated.

Installation,configurationandreliabilityproblemsaredirectlyrelatedtothematurityofthesoftware,asisscalability,whichappearsintwodifferentplacesinthelist.

PerformancehasbeenaconstantcomplaintwithallBIanddatawarehouseprojects,so

welookedatwhatpeopleweredoingtoaddressperformanceproblems.Figure21liststhemostcommonpracticesfordealingwithperformanceproblemsinthisenvironment.

Thesurveyresponsereflectsthis,showingthemostcommonsolutionattemptedistuningthedatabase,followedbythrowinghardwareattheproblem.Thenumberthreeresultissurprising:changetoadifferentreportingorBItool.Giventhatthedatabasemaybethesourceoftheproblem,onewouldexpectdatabasetechnologychanges;

however,theseappearattheverybottomofthelist.

ThemostcommoncomplaintaboutperformanceinBIenvironmentsfocusesonthe

databasebecause

it

is

the

hub

of

most

activities.

5%

17%

21%

21%

24%

30%

31%

32%

41%

Poor/lackingdocumentation

Limiteduserconcurrency

Difficultygettinganswersthroughvendorchannels

Difficultygetting answersviathecommunity

Limiteddatascalability

Difficultyintegratingintoexistingtechnologyenvironment

Challengesfindingtraining oreducation

Crashing orotherreliabilityproblems

Installationorconfigurationproblems


25/33


Figure21:AddressingperformanceproblemsAlesssurprisingaspectofdatabasechangeisthatpeoplearemorethantwiceaslikely

tochangefromtheirdatabasetoananalyticdatabase.Ifyouarechangingthedatabasebecauseofperformance,goingtoasimilarproductmakeslesssensethantryinga

databasedesignedspecificallytosupportBIandanalysisworkloads.

Akeyelementtoperformanceisunderstandingthattoolsarenotusuallytherootcauseoftheproblems.OneITmanagersaid,"Toolsaretools.Thereisnocorrectingforbad

designbyreplacingonewithanother.Thedesignofthesystemisstillmoreimportant

than

what

any

one

product

can

do."

Wealsoaskedwhatpeoplefeltthekeyobstacleswerefortheirorganization'suseofopensource.TheresultsareshowninFigure22.Lackofinternalskillstoppedthelist,

whichmightinfluencespendingontraining,anitemthatwastrendingupinthepurchasedata.

4%

8%

10%

18%

18%

26%

30%

32%

32%

34%

38%

Migratetoadifferenttraditionaldatabase

Buyaspecializedaccellerator

Migratetoananalyticdatabase

Limitthenumberofusersaccessing thesystem

ChangeETLordataintegrationtools

RewritetheBIapplicationorreports

Limittheamountofdatastoredinthesystem

RedesigntheETLordataintegration

ChangeBI

or

analytics

tools

Buymorepowerfulhardware

Databaseorapplicationtuning


26/33


Figure22:BarrierstouseOverall,maturityofthesoftwareisstillatopconcern.Itemslikeproductsmissingneededfeatures,worryoverlongevity,qualityproblemsandlackofsupportareall

relatedtomaturityofthesoftwareandvendors.Onecontradictionwasfoundregardingcomplaintswaslongevity:severalcompanieswereevaluatingopensourcebecausethe

reportingproductstheywereusinghadbeendiscontinuedorthevendorhadbeenacquired.Itisalessonthatthereisnoguaranteeaproductwillbearound.

Ingeneral,procurementissues(corporatestandards,ITresistance,licenseandlegaldifficulties)arealmostnonexistentproblemstodayasorganizationshavebecomemore

familiarwithopensource.Anotherfactorwhichreducesthiscomplaintisthatmanycompaniespurchasecommerciallylicensedorsubscriptionversions.

Overall,opensourcesoftwareisgoodenoughtogetontotheshortlistformany

organizations.Theverylastitemofthetop20barrierswasthevendororprojectnotmeetingthecriteriatobeincludedinanevaluation.

Information Resources Found Useful

Aspartofthesurvey,weaskedpeopleabouttheinformationtheyusedandwhat

influencedthechoiceofprojectstoevaluate.Influenceswerementionedinaprior

sectioninFigure18.

18%

18%

20%

23%

26%

28%

28%

30%

30%

38%

Highercoststhananticipated

Noproven trackrecordinthismarket

Difficultyintegratingintocurrentenvironment

Lackoforinadequatesupport

InternalIT/organizationresistance

Softwarequalityproblems

Performanceorscalabilityproblems

Missingneededfeatures

Concernabout longevityofproject/company

Lackofinternal skillstomanageorintegratethesoftware


27/33


Onceintoanevaluationorproofofconcept,theresourceslistedinFigure23were

foundtobehelpful.

Figure23:RankingofusefulnessofopensourceresourcesOnlinecontentholdsthetopspots,withthirdpartywhitepapersrankedthird(theonly

resourceinthetopfivethatdirectlycoststhevendorsmoney).Communitydriven

elementsintheformofwikisandforumsaretwoofthetopfive.

Inablowtotheideaofthirdpartysupport,thiswasratednexttolast.Equallysurprising

wasthelowratinggiventothefeaturesfoundinenterpriseorprofessionalversionsof

thesoftware.Thismaybeduetothehighpercentofpeoplestillintheevaluationphase

inoursampleaswellasthe50%whodidn'tbuyanything.

Bundlingofcomponentsmadeadifferenceforsomerespondents.16%saidvirtual

machinesorpreintegratedsoftwarebundleswereusefulbecausetheyprovidedthe

abilitytogetupandrunningquickly.Thiswasdescribedbyseveralpeopleasa

differentiatortomostproprietarysoftwarewherequickevaluationswerenotpossible.

7%

14%

14%

16%

17 %

19%

20%

27 %

28%

29%

32%

37 %

37%

47%

47%

48 %

53 %

53 %

Internetrelaychat(IRC)

Supportfromathirdparty

Classroomtraining

Prebundledsoftware(e.g.adatabasepackaged withaBItool)

Softwarefeaturesinapaid"professional"versionofthesoftware

Outsideconsultantorsystemsintegrator

Vendorsupport,paidoraspartofasubscription

Thirdpartybooksordocumentation

Webbasedtraining

Printarticles

Vendorevaluation/trialsupport(free)

Blogs

Webseminarsorscreencasts

Communityforums

Onlinedemos

White papers

Online documentation/wikis

Onlinearticles


28/33


RecommendationsOpensourceinthebusinessintelligenceanddatawarehousingfieldisoutofthe

innovationstageandmovingintotheearlymainstream,buttherearestillchallenges.PeopleresponsibleforevaluatingBIanddatawarehousingtoolscanbenefitfromthe

followingsuggestions.

Don'tplantoreplaceexistingsoftwarewithopensource.Thesinglebiggestusagescenarioisfornewprojects,notasanattempttoreplaceothertools.

Oneoftheobstaclestothisasareplacementsolutionisthehighcostofredevelopingreportsorintegrationjobs.Ratherthanlookatsavingmoney

byreplacingsoftware,lookatgapsintheBIportfolioordatawarehousestackanduseopensourcetosupplementyoursystems.

Evaluateopensourcelikeanyothersoftware.Itdoesn'tmatterifthesoftwareisfreeifitdoesn'tdowhat'srequired.Opensourcesoftwareisstill

softwareandshouldbeevaluatedagainstthesamesetofcriteriayouwould

usewith

any

similar

application

from

atraditional

vendor.

In

evaluations,

be

suretofactorinsomeoftheopensourceelementspeoplelistedasbenefits.Opensourcetoolsmaynotbeasfeaturerichasproprietarysoftware,but

offerotherpotentialbenefitsliketimetomarket,deploymentflexibilityandcustomizabilitythatcanmakeupforthis.

Beclearonwhatyouareevaluating.Commercialopensourceusuallycomesintwoversions:afreecommunityeditionandapaidenterpriseedition.Peoplehaveevaluatedthecommunityeditionandruleditoutwithout

realizingthefeaturestheywantedwereavailableintheenterpriseedition.Thecommunityandenterprisefeaturescanbesignificantlydifferent,andthe

enterprise

version

is

often

a

fraction

of

the

cost

of

the

alternative

products.

Othersevaluatedtheenterpriseedition,assumingthecommunityedition

hadthesamefunctionsandwouldcostnothingtoimplement.

Factorconsultingintoyourplan.Findingskilledworkersthatareexperiencedinopensourcetoolsmaybedifficult.Mostconsultingcompanieshavenotbeentrainedonthesetools,soyouarepayingfortheir

genericexpertiseratherthantheiropensourceknowledge.Ifyourprojectispredicatedonexternalresources,opensourcetoolsmaynotbethebest

choiceunlessyoufirstverifythattheresourcesareavailable. Don'tfocussolelyoncostsavings.Whilecostisimportant,itisonlyone

factor.

The

other

top

ranked

benefits

are

reduced

dependence

on

vendors,

easeofintegrationanddeploymentflexibility.Ifyouarebuildingacasetojustifyanopensourcetool,itwillhelptoincludetheseotherfactors.People

oftendidnotmentiontheseasreasonsforinitialconsideration,butasbenefitstheydiscoveredlater.

Makeopensourcethedefaultoption.Wheninanenvironmentwithfewornotools,opensourceshouldbethepreferredalternative.Itisthesimplest,

fastestandlikelytheleastexpensiveroutewhencomparedtohandcoding


29/33


orpurchasingproprietaryproducts.Looktoproprietarytoolswhenopen

sourcetoolsdon'thavetherequiredfeatures,orwhenyouhaveproductsintheorganizationalreadyandexpandinglicensesisnotasbiganobstruction.

Developopensourcepolicies.Mostorganizationsareadoptingopensourceinanadhocfashion,projectbyproject.Whilethisworks,itcanalsoreduce

cost

savings

by

duplicating

evaluation

effort

and

maintenance

costs.

Open

sourcecanbypasstheprocurementprocess,leadingtosituationswhere

departmentsdeploytheirowntools,unawarethatsomeoneelsehasalreadydoneevaluationsordeployeddifferentsoftware.Therearealsosome

differenceswithopensourcelicensesthatcanputyourorganizationatriskifyourlegaldepartmenthasn'tdoneproperreview.

Whilethebiggestvalueofopensourcemayappeartobelicensecostsavings,an

importantoverlookedbenefitistimetomarket.Thefullpurchasecyclefortheenterpriseisusuallythreetosixmonths(orlonger).Inthetimeittakestobringina

vendor,meetthesalesteamandgetapprovaltodoaproofofconceptwiththesoftware,youcandownloadandprototypeyourentireapplicationinasimilaropen

sourcetool.

Theadvantagetodoingthisisthatyourhandsonexperiencewilltellyouwhetherthe

softwarewillworkforyou.Ifitworks,youcanextendanddeploythatprototype.Overtime,expecttraditionalvendorstobecomemoreflexiblebyallowingtrials,offering

subscriptionpricingandusingotherpracticesstartedbyopensourcevendors.

OpensourceuseisgrowingrapidlyintheBIanddatawarehousingfield.Aswemoveintotheearlymainstreamstage,thesoftwarewillbecomemorepolishedand

comparabletotraditionalproducts.Therearealreadysignsofashifttomainstreamadoptionasconsultingcompaniesandsystemsintegratorsbegintoevaluateopen

sourceforthemselvesandtheirclients.

Profile of Survey ParticipantsIndustryrepresentationisbroad,withatotalof399respondentsfrom16different

industrycategories.TheindustryandrespondentdistributionisshowninFigure24.

Computersoftwareandhardwarecompaniesmakeupthelargestportionat22%ofthe

total.Consultants,integratorsandsolutionprovidersarethenextlargestat15%.

Companiesinthesoftwareproductionindustriesweretheearliesttoadoptopensource

toolsinthiscategory.

Softwarecompanies

are

adding

reporting

and

business

intelligence

features

to

their

packageseitherbecauseitisrequiredbynewcustomersortoadddifferentiation.

Interviewsprovideevidencethatthesecompaniesareturningtoopensourcebecause

oflowerembeddingcosts,butmoreimportantlybecauseitiseasiertoincorporatethan

traditionallylicensedsoftware.


30/33


Severalcompaniesmentionedthattheywereabletoaddmissingfeaturestotheir

softwareandturnthecodeovertothevendorforinclusionintheopensourcepackage,

thussavingthevendorfromhavingtomaintainthecode.Thisisfarlesslikelytohappen

withtraditionalvendors.

Financialservicesisthelargestofthenoncomputerindustrycategoriesat11%whichis

notasurprise financialserviceswasanearlyadopterofothercategoriesofopen

sourcesoftware.Governmentinteresthaspickedupsignificantlyoverthelasttwo

years,nowfourthinthelist.

Figure24:Industriesanddistributionofsurveyrespondents1%

1%

2%

2%

2%

4%

4%

5%

6%

6%

6%6%

8%

11%

15%

22%

Aerospance,Defense

Agriculture,Mining,Construction,Architecture,Engineering

Transportation,Logistics

Travel,Hospitality,Recreation,Entertainment

Utilities,Petroleum,Oil,Energy

Other

Communications,Telecommunications,Cable

Manufacturing,Chemicals

Media,Publishing, Advertising,PR,Marketing

Education

Retail,ConsumerPackagedGoods,Distribution,Trade,Wholesale

Healthcare,Medical,

Pharmaceutical,

Biotech,

Biomed

Government(Federal,State,Local,Military)

Accounting,Banking,FinancialServices,Insurance,RealEstate,Legal

Consulting,BusinessIntegrator,VAR,SolutionProviders(ISV,ASP)

ComputerSoftware,Hardware,Services


31/33


ThebulkoftherespondentsarefromtheU.S.andEurope,representingatotalof81differentcountriesworldwide.ThedistributionacrossregionsisshowninFigure25.This

reflectsasimilardistributioninotherareasofopensourcesoftware.

Figure25:Geographicdistributionofsurveyrespondents

Thesizeoforganizationsrepresentedinthesurveyiswidelydistributed,fromGlobal100toverysmallcompanies.Thesizeoforganizationscanbemeasuredbyrevenueor

numberofemployees.Ratherthanusecompanyrevenue,thisreportusesemployeecountforthesizemetricbecauseitisapplicableacrossbothprivate,publicandnon

profitinstitutions,andbecauseitisamorereliablegaugeofthescaleofanorganization'suseofBIandanalyticsthantheirrevenues.

Forthis

survey,

small

organizations

are

considered

to

be

those

with

less

than

100

employees,midsizedorganizationsarebetween100and2,000employees,andlarge

organizationsarethosewithmorethan2,000employees.

ThesizedistributionoforganizationsinthissurveyisshowninFigure26byemployeecountandsmallmediumlargeclassification.

Figure26:Revenueandemployeesizeoforganizationsinthesurveysample

55%

29%

7%

5%3% 2%

NorthAmerica

Europe

Central /SouthAmerica

Asia

AfricaandMiddleEast

Oceania

14%

15%

14%

21%

37%

Morethan10,000employees

2,0001 10,000employees

501

2,000employees

100 500employees

Lessthan100employees

Small

37%

Medium

34%

Large

29%


32/33


About the Sponsors

Kickfireisanextgenerationdatawarehouseappliancethatdeliverstheindustrysbest

price/performanceandthefastesttimetoinsight.BasedonabreakthroughparallelprocessingSQLchip

thatpacksthepowerofdozensofCPUsintoanexceptionallysmall,lowpowerformfactor,Kickfire

deliversaquantumleapinperformanceefficiency avoidingthehardwarebuildout,poweranddata

centerspacerequiredoftoday'sdatawarehouseofferings.TheKickfireDataWarehouseAppliancealso

includesacolumnarengineandrunsMySQLEnterprise.Thecompanyisbackedbybluechipventure

capitalfirmsAccelPartners,GreylockPartners,TheMayfieldFundandPinnacleVentures.Formore

information,pleasevisitwww.kickfire.com.

JaspersoftsopensourcebusinessintelligencesuiteistheworldsmostwidelyusedBIsoftware,with

nearly9milliontotaldownloadsworldwideandmorethan10,000commercialcustomersin96countries.

ThecompanysJaspersoftBusinessIntelligenceSuiteprovidesawebbased,openandmodularapproach

totheevolvingbusinessintelligenceneedsoftheenterprise.Jaspersoftssoftwareisrapidlyupdatedbya

communityofmorethan100,000registeredmembersworkingonmorethan350projects,which

representstheworldslargestbusinessintelligencecommunity.Moreinformationisavailableat

www.jaspersoft.comandwww.jasperforge.org.

PentahoCorporationisthecommercialopensourcealternativeforbusinessintelligence(BI).PentahoBISuiteEnterpriseEditionprovidescomprehensivereporting,OLAPanalysis,dashboards,data

integration/ETL,dataminingandaBIplatformthathavemadeittheworldsleadingandmostwidely

deployedcommercialopensourceBIsuite.Pentahoprovidessupport,servicesandproduct

enhancementsviaanannualsubscriptionthatcanlowertotalcostofownershipby90%comparedto

traditional,proprietaryBIofferings.Sinceits2004foundingasthepioneerinopensourceBI,Pentaho's

productshavebeendownloadedmorethanfivemilliontimes,withproductiondeploymentsat

companiesrangingfromsmallorganizationstoTheGlobal2000.Formoreinformation,visit

www.pentaho.com.

Talend,therecognizedmarketleaderinopensourcedataintegration,leveragestheopensourcemodel

tomakedataintegrationavailabletoalltypesoforganizations,regardlessoftheirsize,levelofexpertise

orbudgetaryconstraints.Talendssolutionsconnecttoallsourceandtargetsystemsandtheycanbe

downloadedatnocost.Talendalsooffersdataqualitysolutions,fullycomplementarytoitsdata

integrationsolutions.Formoreinformation,pleasevisithttp://www.talend.com/.


33/33

About the Author

MARKMADSENispresidentofThirdNature,focusedoninformationmanagement,BIandanalytics.MarkisanawardwinningarchitectandformerCTOwhoseworkhasbeenfeaturedin

numerousindustrypublications.Heisaninternationalspeakerandmanagestheopensource

channelat

the

Business

Intelligence

Network.

For

more

information

or

to

contact

Mark,

visit

http://ThirdNature.net.

ThirdNatureisaresearchandconsultingfirmfocusedonnewpractices

andemergingtechnologyforbusinessintelligence,dataintegrationand

informationmanagement.

Ourgoalistohelpcompanieslearnhowtotakeadvantageofnewinformationdriven

managementpracticesandapplications.Weofferconsulting,educationandresearchservicesto

supportbusinessandITorganizationsaswellastechnologyvendors.

BeyeNetwork Open Source Research Report

Documents