Digital Image Processing Techniques for the Detection and Removal of Cracks in Digitized Painting

7/27/2019 Digital Image Processing Techniques for the Detection and Removal of Cracks in Digitized Painting

1/26

1

Digital imageprocessing techniques for thedetectionand removalofcracks indigitized

paintingsIoannisGiakoumis,NikosNikolaidis, IoannisPitas

Departmentof InformaticsAristotleUniversityofThessaloniki

54124Thessaloniki,Greecetel/fax:+302310996304

e-mail:{nikolaid,pitas}@zeus.csd.auth.gr

AbstractAnintegratedmethodologyforthedetectionandremovalofcracksondigitizedpaintingsispresentedinthis

paper.Thecracksaredetectedbythresholdingtheoutputofthemorphologicaltop-hattransform.Afterwards,thethindarkbrushstrokeswhichhavebeenmisidentifiedascracksareremovedusingeitheraMedianRadialBasisFunction(MRBF) neuralnetworkonhueandsaturationdataorasemi-automaticprocedurebasedonregiongrowing.Finally,crackfillingusingorderstatisticsfiltersorcontrolledanisotropicdiffusionisperformed.Themethodologyhasbeenshown toperformverywellondigitizedpaintingssuffering fromcracks.

I.INTRODUCTIONManypaintings,especiallyoldones,sufferfrombreaks inthesubstrate,thepaint,orthevarnish.Thesepatterns

areusuallycalledcracksorcraquelureandcanbecausedbyaging,drying,andmechanicalfactors.Agecrackscanresultfromnon-uniformcontractioninthecanvasorwood-panelsupportofthepainting,whichstressesthelayersofthepainting.Dryingcracksareusuallycausedbytheevaporationofvolatilepaintcomponentsandtheconsequentshrinkageofthepaint.Finally,mechanicalcracksresultfrompaintingdeformationsduetoexternalcauses,e.g.vibrationsand impacts.Theappearanceofcracksonpaintingsdeterioratestheperceivedimagequality.However,onecanusedigital

imageprocessing techniques todetectandeliminate thecracksondigitizedpaintings.Suchavirtual restorationcanprovidecluestoarthistorians,museumcuratorsandthegeneralpubliconhowthepaintingwould look like initsinitialstate,i.e.,withoutthecracks.Furthermore,itcanbeusedasanon-destructivetoolfortheplanningoftheactualrestoration.A systemthat iscapableoftrackingandinterpolatingcracks ispresentedin[1].Theusershouldmanuallyselectapointoneachcracktoberestored.Amethodforthedetectionofcracksusingmulti-orientedNovember30,2005 DRAFT


2/26

2

Gaborfiltersispresentedin[2].Crackdetectionandremovalbearscertainsimilaritieswithmethodsproposedforthedetectionandremovalofscratchesandotherartifactsfrommotionpicturefilms[3],[4],[5].However,suchmethodsrelyoninformationobtainedoverseveraladjacentframesforbothartifactdetectionandfillingandthusarenotdirectlyapplicableinthecaseofpaintingcracks.Otherresearchareasthatarecloselyrelatedtocrackremovalincludeimageinpaintingwhichdealswiththereconstructionofmissingordamagedimageareasbyfilling-in informationfromtheneighboringareas,anddisocclusion, i.e.,recoveryofobjectpartsthatarehiddenbehindotherobjectswithinanimage.Methodsdevelopedintheseareasassumethattheregionswhere informationhastobefilled-inareknown.Differentapproachesfor interpolating information instructured [6],[7],[8],[9],[10]and textured imageareas [11]havebeendeveloped.The formerareusuallybasedonpartialdifferentialequations(PDE)andonthecalculusofvariationswhereasthelatterrelyontexturesynthesisprinciples.Atechniquethatdecomposesthe imageto texturedandstructuredareasandusesappropriate interpolation techniquesdependingontheareawhere themissing information lieshasalsobeenproposed [12].Theresultsobtainedbythese techniquesare

very

good.

Amethodology

for

the

restoration

of

cracks

on

digitized

paintings,

which

adapts

and

integrates

a

numberofimageprocessingandanalysistoolsisproposedinthispaper.Themethodologyisanextensionofthecrack removal frameworkpresented in [13].The techniqueconsistsof the followingstages:

Crackdetection.Separationof the thindarkbrushstrokes,whichhavebeenmisidentifiedascracks.Crackfilling (interpolation).Acertaindegreeofuserinteraction,mostnotablyinthecrackdetectionstage,isrequiredforoptimalresults.

User interaction isratherunavoidablesincethe largevariationsobserved inthetypologyofcrackswould leadanyfullyautomaticalgorithmtofailure.However,allprocessingstepscanbeexecutedinrealtimeandthustheusercaninstantlyobservetheeffectofparametertuningontheimageunderstudyandselectinanintuitivewaythevaluesthatachievetheoptimalvisualresult.Needlesstosaythatonlysubjectiveoptimalitycriteriacanbeusedin thiscasesincenogroundtruthdataareavailable.Theopinionofrestorationexpertsthat inspectedthevirtuallyrestored imageswasverypositive.Thispaperisorganizedasfollows.SectionIIdescribesthecrackdetectionprocedure.Twomethodsforthe

separationofthebrushstrokeswhichhavebeenfalselyidentifiedascracksarepresentedinSectionIII.MethodsforfillingthecrackswithimagecontentfromneighboringpixelsareproposedinSectionIV.Conclusionsanddiscussion follow.

I I.DETECTIONOFCRACKSCracksusuallyhavelowluminanceandthuscanbeconsideredaslocalintensityminimawithratherelongated

structuralcharacteristics.Therefore,acrackdetectorcanbeappliedon the luminancecomponentofan imageandshouldbeableto identifysuchminima.Acrackdetectionprocedurebasedontheso-calledtop-hattransform [14]isproposed in thispaper.The top-hat transform isagrayscalemorphologicalfilterdefinedas follows:

y(x) = f(x) fnB(x) (1)November30,2005 DRAFT


3/26

3

wherefnB(x) istheopeningofthefunctionf(x) (inourcase,theluminancecomponentoftheimageunderstudy)with thestructuringsetnB ,definedas:

nB = BBB (ntimes) (2)Inthepreviousequationdenotesthedilationoperation.AsquareoracirclecanbeusedasstructuringelementB[15].ThefinalstructuringsetnB isevaluatedonlyonceusing(2)andisusedsubsequentlyintheopeningoperationof (1).TheopeningfnBofa function isa low-passnonlinearfilter thaterasesallpeaks (localmaxima) inwhichthestructuringelementnBcannotfit.Thus,theimagef fnBcontainsonlythosepeaksandnobackgroundatall.Sincecracksare localminimaratherthan localmaximathetop-hattransformshouldbeappliedonthenegatedluminanceimage.Alternatively,onecandetectcracksbyperformingclosingontheoriginalimagef(x)withthestructuringset nBand thensubtractingf(x) from the resultofclosingfnB(x):

y(x) = fnB(x) f(x) (3)Itcanbeeasilyshownthattheresultof(3)isidenticaltothatofapplying(1)onthenegatedimage.Useof(3)doesnot requirenegationoff(x) whichgrands itasmallbutnotnegligiblecomputationaladvantageover (1).Insituationswherethecrack-likeartifactsareofhighluminance,asinthecaseofscratchesonphotographs,

negationofthe luminancecomponentpriortothecrackdetectionisnotrequired, i.e.thecrackdetectionprocedurecanbeapplieddirectlyontheluminanceimage.Theusercancontroltheresultofthecrackdetectionprocedurebychoosingappropriatevalues for the followingparameters:

The typeof thestructuringelementB.Thesizeof thestructuringelementBand thenumbernofdilations in (2).

TheseparametersaffectthesizeofthefinalstructuringelementnB andmustbechosenaccordingtothethicknessofthecrackstobedetected.Itshouldbenotedhoweverthattheseparametersarenotverycriticalforthealgorithmperformanceduetothethresholdingoperationthatwillbedescribedinthenextparagraphandalsoduetotheexistenceofthebrushstroke/crackseparationprocedure(sectionIII),whichisabletoremovecrack-likebrushstrokesthathavebeenerroneouslyidentifiedascracks.Thefactthatalltheresultspresentedinthispaperhavebeenobtainedwiththesametop-hattransformparameterscomesasaclearindicationthattheabovestatementisindeedtrue.Theseparameterswere the following:

Structuringelement type:squareStructuringelementsize:3 3Numbernofdilations in (2):2Thetop-hattransformgeneratesagrayscaleoutputimaget(k, l) wherepixelswithalargegreyvaluearepotential

crackorcrack-likeelements.Therefore,a thresholdingoperationon t(k, l) isrequired toseparatecracks from therestoftheimage.ThethresholdTcanbechosenbyatrialanderrorprocedure,i.e.,byinspectingitseffectontheresultingcrackmap.The lowcomputationalcomplexityof the thresholdingoperationenables theuser toviewthecrackdetectionresultsinrealtimewhilechangingthethresholdvalue.e.g.,bymovingaslider.ThisfactNovember30,2005 DRAFT


4/26

4

makes interactive thresholdselectionveryeffectiveand intuitive.Alternatively, thresholdselectioncanbedonebyinspectingthehistogramoft(k, l)foralobeclosetothemaximumintensityvalue(whichwillmostprobablycorrespondtocrackorcrack-likepixels),andassigningitavaluethatseparatesthislobefromtherestoftheintensities.Theresultofthethresholdingisabinaryimageb(k, l)markingthepossiblecracklocations.Insteadofthisglobalthersholdingtechnique,morecomplexthresholdingschemes,whichuseaspatiallyvaryingthresholdcanbeused.Obviously,asthethresholdvalueincreasesthenumberofimagepixelsthatareidentifiedascracksdecreases.Thus,certaincracks,especiallyindarkimageareaswherethelocalminimumconditionmaynotbesatisfied,canremainundetected.Inprinciple,itismorepreferabletoselectthethresholdsothatsomecracksremainundetectedthantochooseathresholdthatwouldresultinthedetectionofallcracksbutwillalsofalselyidentifyascracks,andsubsequentlymodify,other imagestructures.Thethresholded (binary)outputofthetop-hattransformontheluminancecomponentofanimagecontainingcracks(Figure1)canbeseeninFigure2.Additionalexamplesofcracksdetectedusing thisapproachcanbeseen inFigures10-12.

I I I .SEPARATIONOFTHE BRUSHSTROKESFROMTH ECRACKSInsomepaintings,certainareasexistwherebrushstrokeshavealmostthesamethicknessandluminancefeatures

ascracks.Thehairofapersoninaportraitcouldbesuchanarea.Therefore,thetop-hattransformmightmisclassifythesedarkbrushstrokesascracks.Thus,inordertoavoidanyundesirablealterationstotheoriginalimage,itisveryimportanttoseparatethesebrushstrokesfromtheactualcracks,beforetheimplementationofthecrackfillingprocedure.Twomethods toachieve thisgoalaredescribed in the followingsubsections.A.Semi-automatic crack separation

Asimpleinteractiveapproachfortheseparationofcracksfrombrushstrokesistoapplyaregiongrowingalgorithmonthethresholdedoutputofthetop-hattransform,starting frompixels(seeds)ontheactualcracks.Thepixelsarechosenbytheuserinaninteractivemode.Atleastoneseedperconnectedcrackelementshouldbechosen.Alternatively,theusercanchoosetoapplythetechniqueonthebrushstrokes,ifthisismoreconvenient.Thegrowthmechanismthatwasusedimplementsthewell-knowngrassfirealgorithmthatchecksrecursivelyforunclassifiedpixelswithvalue1inthe8-neighborhoodofeachcrackpixel.Attheendofthisprocedure,thepixelsinthebinaryimage,whichcorrespondtobrushstrokesthatarenot8-connectedtocrackswillberemoved.Theaboveprocedurecanbeusedeitherinastand-alonemodeorappliedontheoutputoftheMRBFseparationproceduredescribed in thenextsection toeliminateany remainingbrushstrokes.B.Discrimination on the basisof hue and saturationHueH isassociatedwiththedominantwavelengthinamixtureoflightwavelengthsandrepresentsthedominant

color.IntheHSVcolormodel,hueisrepresentedastheanglearoundtheverticalaxis,withredat0,greenat120,andsoon.SaturationSreferstotheamountofwhite lightmixedwithacertainhue.Hueandsaturationare

November30,2005 DRAFT


5/26

5

definedsimilarly inotherrelatedcolordomains,e.g.inHueSaturation Intensity(HSI)orHueLightnessSaturation(HLS). Bystatisticalanalysisof47digitizedpaintings(24portablereligiousiconsfromtheByzantineeraand23

paintingsofvariousstylesandages),ithasbeenconcludedthatthehueofthecracksusuallyrangesfrom0to60.Onthecontrary,weobservedthatthehueofthedarkbrushstrokesvaries,asexpected,intheentiregamut[0,360].Furthermore,cracksaturationusuallyrangesfrom0.3to0.7,whilebrushstrokesaturationrangesfrom0to0.4.Thus,onthebasisoftheseobservations,agreatportionofthedarkbrushstrokes,falselydetectedbythetop-hattransform,canbeseparatedfromthecracks.ThisseparationcanbeachievedbyclassificationusingaMedianRadialBasisFunction(MRBF)neuralnetwork,whichisarobust,orderstatisticsbased,variationofRadialBasisFunction (RBF)networks [16].RBFsaretwo-layer feedforwardneuralnetworks [17],thatmodelamappingbetweenasetof inputvectorsand

asetofoutputs.Thenetworkarchitecture ispresentednFigure3.RBFs incorporatean intermediate,hidden layerwhere

each

hidden

unit

implements

akernel

function,

usually

aGaussian

function:

j(X) = exp[(j X)TS1(j X)],j= 1,...,L (4)

wherej,SjdenotethemeanvectorandthecovariancematrixforkerneljandLdenotesthenumberofunits(kernels) in thehidden layer.Eachoutputconsistsofaweightedsumofkernels. In typicalsituations that involvepatternclassification,thenumberofoutputsequalstothenumberofclasses. Insuchasetting,thecurrentvector isassignedtotheclassassociatedwiththeoutputunitexhibitingthemaximumactivation(winnertakesallapproach).Afterthelearningstage,thenetworkimplementstheinput-outputmappingruleandcangeneralizeittoinputvectorsnotbeingpartof the trainingset.Theparameters tobeestimated (learned) inaRBFnetworkarethecenter (mean)vectorjand thecovariance

matrixSjforeachGaussianfunctionandtheweightswk,jcorrespondingtotheconnectionsbetweenneuronsinthehidden layerandoutputnodes.Ahybridtechniquethathasbeenfrequentlyusedforthetrainingofsuchnetworks,hasbeenadoptedforthe learningstage.Accordingtothistechnique,training isperformed intwosuccessivesteps:thehiddenlayerparametersareestimatedusinganunsupervisedapproachand,afterwards,theoutputlayerweightsareupdated inasupervisedmanner,using the (nowfixed)hidden layerparametersevaluated in thepreviousstep.Intheclassicalversionoftheadoptedtrainingtechnique,avariationoftheLearningVectorQuantizer(LVQ)

algorithmisusedfortheunsupervisedhiddenlayerparameterupdating.EachinputvectorisassignedtotheGaussiankernelwhosecenter iscloser (in termsofeither theEuclideanor theMahalanobisdistance) to thisvector:

if Xi j= min Lk=1Xi k thenXiCj (5)where denoteseitherEuclideanorMahalanobisdistanceandCjdenotestheclassofinputvectorsassociatedwithkernelj.Subsequently,thealgorithmupdatesthecenterandcovariancematrixofthewinnerkernelusingrunningversionsoftheclassicalsamplemeanandsamplecovariancematrixformulas.Ontheotherhand,theMRBFalgorithmwhichhasbeenusedinourcaseisbasedonrobustestimation[16]ofthehiddenunitparameters.



6/26

j= marg na me an 0 j ,..., n1 j

0 ifX/C

6

ItemploystheMarginalMedianLVQ[18]thatselectsthewinnerkernelusing(5)andutilizesthemarginalmedianofthe inputvectorscurrentlyassignedtothiskernel fortheupdateofthecentervector (locationparameter)jofthekernel:

j= marginalmedian{X0,X1,...,Xn1}; (6)whereXn1isthelastvectorassignedtokernelj.Theupdateofthediagonalelementsofthecorrespondingcovariancematrixisperformedusingthemedianoftheabsolutedeviations(MAD)[19]oftheinputscurrentlyassigned to thiskernel:

0.6745 (7)Inthepreviousexpression,jdenotesthevectorcontainingthediagonalelementsofthecovariancematrixand|X|denotesthevectorobtainedbytakingtheabsolutevalueofeachcomponentofX.Theoff-diagonalcomponentsofthecovariancematrixarealsocalculatedbasedonrobuststatistics[16].Inordertoavoidexcessivecomputationstheaboveoperationscanbeappliedonasubsetofdataextractedthroughamovingwindowthatcontainsonlythelast Wdata samplesassigned to thehiddenunitj.Inthesupervisedpartofthe learningprocedure,theweightsoftheoutput layer,whichgrouptheclusters found

bythehiddenlayerintoclasses,areupdated.Theupdatemechanismfortheseweightsisdescribedbythefollowingexpression:

wk,j(t+ 1)= wk,j(t)+ nw(Fk(X) Yk(X))Yk(X)(1 Yk(X))j(X) (8)fork= 1,...,M,j= 1,...,L,anda learning factornw(0,1]. In theprevious formulaYk(X),Fk(X) denotetheactualanddesirednetworkoutput for inputvectorX.The latter isgivenby:

1 ifXCkFk(X) =

k (9)Theupdate formula (8)corresponds to thebackpropagation for theoutputweightsofaRBFnetworkwith respectto themeansquareerrorcost function.Inourimplementation,aMRBFnetworkwithtwooutputswasused.Thefirstoutputrepresentstheclassof

crackswhilethesecondonetheclassofbrushstrokes.Inputvectorsweretwo-dimensionalandconsistedofthehueandsaturationvaluesofpixelsidentifiedascracksbythetop-hattransform.Thenumberofclusters(hiddenunits)chosenforeachclassdependsontheoverlapbetweenthepopulationsofcracksandbrushstrokes.Ifthereisasubstantialoverlap,thenumbershouldbeincreased,inordertoreducetheclassificationerror.Inourimplementationthreehiddenunitshavebeenincorporated.Trainingwascarriedoutbypresentingthenetworkwithhueandsaturationvaluesforpixelscorrespondingtocracksandcrack-likebrushstrokes.Datafrom24digitizedportable religious icons from theByzantineerawereused for thispurpose.Thesystem trainedusing thisspecifictrainingsetcanbeconsideredtobeoptimizedforpaintingsofthisstyleanditsuseonpaintingsofotherstylemightresult insomewhatsuboptimalresults.However,appropriatelyselectedtrainingsetscanbeused totrainthesystemtoseparatecracksfrombrushstrokesonpaintingsofdifferentartisticstylesorcontent.InordertoselectNovember30,2005 DRAFT


7/26

7

pixelscorrespondingtocracksandcrack-likebrushstrokesthecrackdetectionalgorithmpresentedinsectionII wasappliedonthese images.Resultsweresubsequentlypost-processedbyanexpertusingthesemi-automaticapproachpresentedinsectionIII-A.Theaimofthispost-processingstepwastwofold:toremovepixelsthatareneithercracksnorcrack-likebrushstrokesandtoseparatecracksandcrack-likebrushstrokesforthesupervisedstepofthetrainingprocedure.Inthissupervisedtrainingstep,thenetworkwaspresentedwiththeselabelledinputs,i.e.,pairsofhue-saturationvaluesthatcorrespondedtoimagepixelsthathavebeenidentifiedasbelongingtocracksandcrack-likebrushstrokes.Afterthetrainingsession,theMRBFneuralnetworkwasabletoclassifypixelsidentifiedascracksbythetop

hattransformtocracksorbrushstrokes.Thetrainednetworkhasbeentestedon12imagesfromthetrainingsetand15 images(ofthesameartisticstyleandera)thatdidnotbelongtothetrainingset.Naturally,theperformanceofthecracks/brushstrokesseparationprocedurewasjudgedonly inasubjectivemanner(i.e.byvisualinspectionoftheresults),asgroundtruthdata(i.e.brushstrokes-freecrackimages)arenotavailable.Forthisreasontworestoration

experts

were

asked

toinspect

several

crack

images

before

and

after

the

application

of

the

separation

systemandconcludedthatintheprocessedcrack imagesthegreatmajorityofthebrushstrokeshasbeenremoved.Athresholdedtop-hattransformoutputcontainingmanybrushstrokese.g.hairisillustratedinFigure5.Agreatpartof thesebrushstrokes isseparatedby theMRBF,ascanbeseen inFigure6.Theoriginal imagecanbeseeninFigure4.

IV.CRACKFILLINGMETHODSAfteridentifyingcracksandseparatingmisclassifiedbrushstrokes,thefinaltaskistorestoretheimageusing

localimageinformation(i.e.,informationfromneighboringpixels)tofill(interpolate)thecracks.Twoclassesoftechniques,utilizingorderstatisticsfilteringandanisotropicdiffusionareproposedforthispurpose.BothareimplementedoneachRGBchannel independentlyandaffectonly thosepixelswhichbelong tocracks.Therefore,providedthattheidentifiedcrackpixelsareindeedcrackpixels,thefillingproceduredoesnotaffecttheusefulcontentoftheimage. Imageinpaintingtechniquesliketheonescited inSectionIcanalsobeusedforcrackfilling.Theperformanceofthecrackfillingmethodspresentedbelowwasjudgedbyvisualinspectionoftheresults.

Obviously,measuringtheperformanceofthesemethodsinanobjectivewayisinfeasiblesincegroundtruthdata(e.g.imagesdepictingthepaintingsinperfectcondition,i.e.,withoutcracks)arenotavailable.For theevaluationoftheresults, tworestorationexpertswereaskedto inspectseveral imagesrestoredusing thevariousmethodsandcomment,basedontheirexperience,onthequalityofthefillingresults,(i.e.,whethercracksweresufficientlyfilled),whether thecolorused forfillingwas thecorrectone,whetherfine imagedetailswere retained,etc.A.Crackfilling basedon order statisticsfilters

Aneffectivewaytointerpolatethecracksistoapplymedianorotherorderstatisticsfilters[15]intheirneighborhood.Allfiltersareselectivelyappliedonthecracks,i.e.,thecenterofthefilterwindowtraversesonlythecrackpixels.Ifthefilterwindowissufficientlylarge,thecrackpixelswithinthewindowwillbeoutliersandNovember30,2005 DRAFT


8/26

1 otherwise

8

willbe rejected.Thus, thecrackpixelwillbeassigned thevalueofoneof theneighboringnon-crackpixels.Thefollowingfilterscanbeused for thispurpose:

Medianfilter:yi= med(xi,...,xi,...,xi+) (10)

Recursivemedianfilter:yi= med(yi,...,yi1,xi,...,xi+) (11)

wheretheyi,...,yi1arethealreadycomputedmedianoutputsamples.Forboththerecursivemedianandthemedianfilter,thefilterwindow (consideringonlyrectangularwindows)shouldbeapproximately50%widerthanthewidest(thickest)crackappearingontheimage.Thisisnecessarytoguaranteethatthefilteroutput isselected tobe thevalueofanon-crackpixel.Smallerwindowswill result incracks thatwillnotbesufficientlyfilledwhereaswindowsthataremuchwider than thecrackswillcreate largehomogeneousareas,thusdistortingfine imagedetails.

Weightedmedianfilter:yi= med(wxi,...,wxi+) (12)

wherewxdenotesduplicationofx, wtimes.Forthisfilter,smallerfilterwindows(e.g.windowsthatareapproximately30%widerthanthewidestcrackappearingontheimage)canbeusedsincetheprobabilitythatacolorvaluecorrespondingtoacrack isselectedasthefilteroutput(afactthatwouldresult inthecrackpixelunder investigationnotbeingfilledeffectivelyby thefilter)canbe limitedbyusingsmallweights (e.g.1)forthepixelscentrallylocatedwithinthewindow(whichareusuallypartofthecrack)andbiggerones(e.g.2or3) for theotherpixels.

Avariationofthemodifiedtrimmedmean(MTM)filterwhichexcludesthesamplesxi+r,j+sinthefilterwindow,whichareconsiderablysmaller from the localmedianandaverages the remainingpixels:

yij= Arsxi+r,j+sThesummationscover theentirefilterwindowA .Thefiltercoefficientsarechosenas follows:

0 if med{xij} xi+r,j+sqrs= (14)

Theamountoftrimmingdependsonthepositiveparameterq.Dataofsmallvaluedeviatingstronglyfromthelocal

median

(which

correspond

usually

to

cracks)

are

trimmed

out.

Windows

used

along

with

this

variant

of

theMTMfiltercanalsobesmallerthanthoseusedforthemedianandrecursivemedianfilterssinceaportionof thecrackpixels isexpected tobe rejectedby the trimmingprocedure.

AnothervariationoftheMTMfilterthatperformsaveragingonlyonthosepixelsthatarenotpartofthecrack,i.e., itutilizes informationfromthebinaryoutput imageb(k, l) ofthetophattransform.Inthiscase,thefilter



9/26

Ii,j = Ii,j+ [cN DNI+ cS DSI+ cE DEI+ cWDWI]ti,j (17)

coefficients in (13)arechosenas follows:rs=

9

1if b(k, l) = 10otherwise (15)

Themedianoperatorcanbeusedinsteadofthearithmeticmeanin(13).ForthisvariantoftheMTMfilter,evensmallerfilterwindowscanbeused,sincecrackpixelsdonotcontributetothefilteroutput.Thus,itsuffices that thewindow is1pixelwider than thewidestcrack.

The resultof theapplicationof thesecondvariationof themodified trimmedmeanfilteron thepaintingdepictedinFigure1aisshown inFigure7(filtersize5 5).Anotherimagerestoredbythesamecrack-fillingapproachcanbeseen inFigure8(filtersize3 3).Extensiveexperimentationprovedthatthisfiltergivesthebestresultsamongallfilterspresentedaboveaccordingtosubjectiveevaluationbyrestorationexperts.Thesuperiorityofthisfiltercanbeattributedtothe factthatonlynon-crackpixelscontributeto itsoutput.Onthecontrary, forallotherfilterspresented in thissection theprobability thatcrackpixelswillcontribute to theoutput issmallbutnotnegligible.B.ControlledAnisotropicDiffusionAnisotropicdiffusion[20]isanimageenhancementmethodthatsuccessfullycombinessmoothingofslowly

varyingintensityregionsandedgeenhancement.Smoothingismodelledasadiffusionthatisallowedalonghomogeneousregionsandinhibitedbyregionboundaries.Anisotropicdiffusionisdescribedbythefollowingpartialdifferentialequation:

I (x, y, t) t = div(c(x, y,t)I(x, y,t)) = c(x, y,t)I(x, y,t) +c(x, y,t)I(x, y,t) (16)

wheredivdenotesthedivergenceoperatorand ,thegradientandLaplacianoperatorswithrespecttothespacevariablesx, y.Ateachpositionanditeration,diffusioniscontrolledbytheconduction(ordiffusion)coefficientsc(x, y,t).Sincediffusionshouldbeinhibitedacrossregionsseparatedbydiscontinuities,theconductioncoefficientsshouldobtainsmallvalues inpixelswith large intensitygradientmagnitude.AsimilarapproachnamedCurvatureDrivenDiffusion (CCD)hasbeenproposed in [9] for image inpaintingapplications.Inordertoobtainanumericalsolutiontothediffusionequation,discretizationofthespatialandtimecoordinates

andapproximationofthedifferentialoperatorsbyfinitedifferenceoperatorsshouldbeperformed in(16).Wehaveusedthesamediscretizationsasproposedin[20].Theiterative,discretesolutionto(16)isgovernedbythefollowingequation:

t twhere0 1/4fortheschemetobestable,N,S,E, WarethemnemonicsforNorth,South,East,Westand



10/26

ctNi,j= g( | DNIi,j |)ctSi,j= g( | DSIi,j |) ctEi,j= g( | DEIi,j |)

ctWi,j= g( | DWIi,j |) (19)

10

thesymbol Dindicatesnearest-neighbordifferences:DNIi,j Ii1,j Ii,jDSIi,j Ii+1,j Ii,jDEIi,j Ii,j+1 Ii,jDWIi,j Ii,j1 Ii,j (18)

Theconductioncoefficientsareevaluatedateveryiterationasafunctiong( )ofthemagnitudeoftheintensitygradient. Inour implementation, the followingapproximationwasused [20]:

tttt

The following functiong( ),proposed in [20],hasbeenused inourcase:g(I) = 1

1+( ITheconstantKwasmanuallyfixed.Inordertofillthecracks,theanisotropicdiffusionalgorithmwasapplied

selectively,inneighborhoodscenteredoncrackpixels.Allpixelswithintheseneighborhoodsparticipateinthediffusionprocess.However,only thevaluesof thecrackpixelsareupdated in theoutput image.Furtherimprovementswereobtainedbytakingintoaccountcrackorientation,i.e.,byapplyingtheoperationonly

inadirectionperpendiculartothecrackdirection.Forexample,ifthecrackishorizontal,onecanuseonlytheNorthandtheSouthneighbors,sincetheWestandtheEastneighborsbelongalsotothecrack. Inordertofindthedirectionsofthecracks,theHoughTransformwasapplied[21].TherestorationresultsofthisfiltercanbeseeninFigures9-12.Theorientation-sensitivecontrolledanisotropicdiffusionmethodgavethebestresultsamongallcrack-fillingmethodspresented inthispaper.The factthatthisfilterrequiresnowindowsizeselectiongrants itanadditionaladvantageover theorderstatisticsfilterspresented in theprevioussection.

V.CONCLUSIONS AND DISCUSSIONInthispaper,wehavepresentedanintegratedstrategyforcrackdetectionandfillingindigitizedpaintings.

Cracksaredetectedbyusingtop-hattransform,whereasthethindarkbrushstrokes,whicharemisidentifiedascracks,areseparatedeitherbyanautomatic technique (MRBFnetworks)orbyasemi-automaticapproach.Crackinterpolation isperformedbyappropriatelymodifiedorder statisticsfiltersor controlledanisotropicdiffusion.Themethodologyhasbeenappliedforthevirtualrestorationofimagesandwasfoundveryeffectivebyrestorationexperts.However,therearecertainaspectsoftheproposedmethodologythatcanbefurtherimproved.Forexample,



11/26

11

thecrackdetectionstageisnotveryefficientindetectingcrackslocatedonverydarkimageareas,sinceintheseareastheintensityofcrackpixelsisveryclosetotheintensityofthesurroundingregion.A possiblesolutiontothisshortcomingwouldbetoapplythecrackdetectionalgorithm locallyonthisareaandselecta low thresholdvalue.Anothersituationwherethesystem(moreparticularly,thecrackfillingstage)doesnotperformasefficientlyasexpectedisinthecaseofcracksthatcrosstheborderbetweenregionsofdifferentcolor.Insuchsituations,itmightbethecasethatpartofthecrackinoneareaisfilledwithcolorfromtheotherarea,resultinginsmallspursofcolorintheborderbetweenthetworegions.SuchasituationisdepictedinFigure11.Howeverthisphenomenonisratherseldomand, furthermore,theextentoftheseerroneouslyfilledareas isverysmall (2-3pixelsmaximum).A possiblesolutionwouldbetoperformedgedetectionorsegmentationontheimageandconfinethefillingofcracksthatcrossedgesorregionborderstopixelsfromthecorrespondingregion.Useof image inpaintingtechniques[6],[7],[8],[9],[10]couldalsoimproveresultsinthataspect.Anotherimprovementofthecrackfillingstagecouldaimatusingproperlyadaptedversionsofnonlinearmultichannelfilters(e.g.variantsofthevectormedianfilter)instead

of

processing

each

color

channel

independently.

These

improvements

will

be

the

topic

of

future

work

on

thissubject.

REFERENCES [1]M.Barni,F.Bartolini,V.Cappellini,Imageprocessingforvirtualrestorationofartworks,IEEEMultimedia,vol.7,no.2,pp.34-37,

April-June2000.[2]F.Abas,K.Martinez,CraquelureAnalysisforContent-basedRetrieval,inProc.14thInternationalConferenceonDigitalSignal

Processing,2002,vol1,pp.111-114.[3]L. Joyeux,O.Buisson,B. Besserer,S.Boukir,Detectionandremovaloflinescratchesinmotionpicturefilms,inProc.IEEEInternational

ConferenceonComputerVisionandPatternRecognition,1999,pp.548-553.[4]A.Kokaram,R. Morris,W.Fitzgerald,P.Rayner,Detectionofmissingdatainimagesequences,IEEETransactionsonImageProcessing,

vol.4,no.11,pp.1496-1508,November1995.[5]A.Kokaram,R.Morris,W.Fitzgerald,P.Rayner,Interpolationofmissingdatainimagesequences,IEEETransactionsonImage

Processing,vol.4,no.11,pp.1509-1519,November1995.[6]M.Bertalmio,G.Sapiro,V.Caselles,C.Ballester,Image Inpainting, inProc.SIGGRAPH2000,pp.417424,2000.[7]C.Ballester,M.Bertalmio,V.Caselles,G.Sapiro, J.Verdera,Filling-InbyJoint InterpolationofVectorFieldsandGrayLevels,IEEE

Transactionson ImageProcessing,vol.10,no.8,pp.12001211,August2001.[8]S.Masnou, J.M.Morel,LevelLinesBaseddisocclusion, inProc. IEEE ICIP98,vol. III,pp.259263,1998.[9]T.Chan,J.Shen,Non-textureinpaintingsbycurvature-drivendiffusions,JournalofVisualCommunicationandImageRepresentation,

vol.12,no.4,pp.436-449,2001.[10]S.Esedoglu,J.Shen,DigitalInpaintingBasedontheMumford-Shah-EulerImageModel,EuropeanJournalofAppliedMathematics,

vol.13,pp.353-370,2002.[11]A.Efros,T.Leung,TextureSynthesisbyNon-parametricSampling, inProc.1999 IEEE InternationalConferenceonComputerVision(ICCV),pp.10331038,1999.[12]M.Bertalmio,L.Vese,G.Sapiro,S.Osher,SimultaneousStructureandTextureImageInpainting,IEEETransactionsonImage

Processing,vol.12,no.8,pp.882-889,August2003.[13]I.Giakoumis, I. Pitas,DigitalRestorationofPaintingCracks, inProc. IEEE InternationalSymposiumonCircuitsandSystems,vol.4,

pp.269-272,1998.[14]F.Meyer,Iterative image transforms foranautomaticscreeningofcervicalsmears,J.Histoch.Cytochem.,vol.27,pp.128135,1979.[15]I.Pitas,A.N.Venetsanopoulos,NonlinearDigitalFilters,principlesandapplications,Norwell:KluwerAcademic,1990.November30,2005 DRAFT


12/26

12

[16]A.G.Bors,I.Pitas,MedianRadialBasisFunctionNeuralNetwork,IEEETransactionsonNeuralNetworks,vol.7,no.6,pp.1351-1364,November1996

[17]S.Haykin,NeuralNetworks,AComprehensiveFountation,2ndEdition,NewYork:PrenticeHall,1999.[18]I.Pitas,C.Kotropoulos,N.Nikolaidis,R.Yang,M.Gabbouj,OrderStatisticsLearningVectorQuantizer,IEEETransactionson Image

Processing,vol.5,no.6,pp.1048-1053, June1996.[19]G.Seber,MultivariateObservations,NewYork: JohnWiley,1986.[20]P.Perona,J.Malik,Scale-SpaceandEdgeDetectionusinganisotropicdiffusion,IEEETransactionsonPatternAnalysisandMachine

Intelligence,vol.12,no.7,pp.629639,July1990.[21]P.Hough, AMethodandMeans forRecognizingComplexPatterns,U.S.Patent,3069654,1962.



13/26

13

Fig.1.Originalpainting.



14/26

14

Fig.2.Thresholdedoutputof the tophat transform (thresholdvalue:23).



15/26

15

Fig.3.RadialBasisFunctionsneuralnetworkarchitecture.



16/26

16

Fig.4.Originalpainting.



17/26

17

Fig.5. Thresholdedoutputofthetop-hattransform(thresholdvalue:19).Anumberofbrushstrokes(hair,numberintheupperleftcorner)havebeenmisidentifiedascracks.



18/26

18

Fig.6.Theseparatedbrushstrokesafter theapplicationof theMRBF technique.



19/26

19

Fig.7.Crackfillingbyusing themodified trimmedmeanfilter(filtersize55).



20/26

20

Fig.8.(a)Originalpainting (detail). (b)Crackfillingbyusing themodified trimmedmeanfilter (filtersize33).



21/26

21

Fig.9.(a)Original imagewithcracks (detail), (b)crackfillingusing theorientation-sensitivecontrolledanisotropicdiffusion technique.



22/26

22

Fig.10.(a)Originalpainting(detail),(b)crackimage(thresholdvalue:17),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique.



23/26


24/26

24

Fig.12.(a)Originalpainting(detail),(b)crackimage(thresholdvalue:23),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique.



25/26

25

IoannisGiakoumisreceived theDiplomaof Informatics in1997.He iscurrentlywith IntracomS.A.

PLACEPHOTOHERE

NikosNikolaidisreceivedtheDiplomaofElectricalEngineeringin1991andthePhDdegreeinElectricalEngineeringin1997,bothfromtheAristotleUniversityofThessaloniki,Greece.From1992to1996heservedasteachingassistantintheDepartmentsofElectricalEngineeringandInformaticsatthesameUniversity.From1998to2002hewaspostdoctoralresearcherandteachingassistantattheDepartmentofInformatics,AristotleUniversityofThessaloniki.He iscurrentlyaLecturer inthesameDepartment.Dr.Nikolaidis istheco-authorofthebook3-DImageProcessingAlgorithms(J.Wiley,2000).Hehasco-authored2bookchapters,18journalpapersand46conferencepapers.Hisresearchinterestsincludecomputergraphics,imageandvideoprocessingandanalysis,copyrightprotectionof

multimediaand3-D imageprocessing.Dr.Nikolaidishasbeenascholarof theStateScholarshipFoundationofGreece.

PLACEPHOTOHERE

IoannisPitasreceivedtheDiplomaofElectricalEngineering in1980andthePhDdegreeinElectricalEngineering in1985bothfromtheAristotleUniversityofThessaloniki,Greece.Since1994,hehasbeenaProfessorattheDepartmentofInformatics,

Aristotle

University

ofThessaloniki.

From

1980

to1993

he

served

asScientific

Assistant,

Lecturer,

AssistantProfessor,andAssociateProfessorintheDepartmentofElectricalandComputerEngineeringatthesameUniversity.HeservedasaVisitingResearchAssociateattheUniversityofToronto,Canada,UniversityofErlangen-Nuernberg,Germany,TampereUniversityofTechnology,Finland,asVisitingAssistantProfessorattheUniversityofTorontoandasVisitingProfessorattheUniversityofBritishColumbia,Vancouver,Canada.Hewaslecturerin

shortcoursesforcontinuingeducation.Hehaspublished140journalpapers,350conferencepapersandcontributedin18booksinhisareasofinterest.Heistheco-authorofthebooksNonlinearDigitalFilters:PrinciplesandApplications(Kluwer,1990),3-DImageProcessingAlgorithms(J.Wiley,2000),NonlinearModel-BasedImage/VideoProcessingandAnalysis(J.Wiley,2001)andauthorofDigitalImageProcessingAlgorithmsandApplications(J.Wiley,2000).HeistheeditorofthebookParallelAlgorithmsandArchitecturesforDigitalImageProcessing,ComputerVisionandNeuralNetworks (Wiley,1993).Hehasalsobeenan invitedspeakerand/ormemberof theprogramcommitteeofseveralscientificconferencesandworkshops. In thepastheservedasAssociateEditorofthe IEEETransactionsonCircuitsandSystems,IEEE TransactionsonNeuralNetworks,IEEETransactionsonImageProcessing,EURASIPJournalonAppliedSignalProcessingandco-editorofMultidimensionalSystemsandSignalProcessing.Hewasgeneralchairofthe1995IEEEWorkshoponNonlinearSignalandImageProcessing(NSIP95),technicalchairofthe1998EuropeanSignalProcessingConferenceandgeneralchairofIEEEICIP2001.Hiscurrentinterestsareintheareasofdigitalimageandvideoprocessingandanalysis,multidimensionalsignalprocessing,watermarkingandcomputervision.



26/26

26

LIS TOFF IGURES1 Originalpainting............................................... 132 Thresholdedoutputof the tophat transform (thresholdvalue:23)..................... 143 RadialBasisFunctionsneuralnetworkarchitecture............................. 154 Originalpainting............................................... 165 Thresholdedoutputofthetop-hattransform(thresholdvalue:19).Anumberofbrushstrokes(hair,

number in theupper leftcorner)havebeenmisidentifiedascracks.................... 176 Theseparatedbrushstrokesafter theapplicationof theMRBF technique................ 187 Crackfillingbyusing themodified trimmedmeanfilter (filtersize5 5)................ 198 (a)Originalpainting(detail).(b)Crackfillingbyusingthemodifiedtrimmedmeanfilter(filtersize

3 3)..................................................... 209 (a)Originalimagewithcracks(detail),(b)crackfillingusingtheorientation-sensitivecontrolled

anisotropicdiffusion technique. ...................................... 2110 (a)Originalpainting(detail),(b)crackimage(thresholdvalue:17),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique............................. 22

11 (a)Originalpainting(detail),(b)crackimage(thresholdvalue:21),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique............................. 23

12 (a)Originalpainting(detail),(b)crackimage(thresholdvalue:23),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique............................. 24

Digital Image Processing Techniques for the Detection and Removal of Cracks in Digitized Painting

Documents