Digital Image Processing Techniques for the Detection and Removal of Cracks in Digitized Painting

7/27/2019 Digital Image Processing Techniques for the Detection and Removal of Cracks in Digitized Painting
1/26
1
Digital imageprocessing techniques for thedetectionand removalofcracks indigitized
paintingsIoannisGiakoumis,NikosNikolaidis, IoannisPitas
Departmentof InformaticsAristotleUniversityofThessaloniki
54124Thessaloniki,Greecetel/fax:+302310996304
e-mail:{nikolaid,pitas}@zeus.csd.auth.gr
AbstractAnintegratedmethodologyforthedetectionandremovalofcracksondigitizedpaintingsispresentedinthis
paper.Thecracksaredetectedbythresholdingtheoutputofthemorphologicaltop-hattransform.Afterwards,thethindarkbrushstrokeswhichhavebeenmisidentifiedascracksareremovedusingeitheraMedianRadialBasisFunction(MRBF) neuralnetworkonhueandsaturationdataorasemi-automaticprocedurebasedonregiongrowing.Finally,crackfillingusingorderstatisticsfiltersorcontrolledanisotropicdiffusionisperformed.Themethodologyhasbeenshown toperformverywellondigitizedpaintingssuffering fromcracks.
I.INTRODUCTIONManypaintings,especiallyoldones,sufferfrombreaks inthesubstrate,thepaint,orthevarnish.Thesepatterns
areusuallycalledcracksorcraquelureandcanbecausedbyaging,drying,andmechanicalfactors.Agecrackscanresultfromnon-uniformcontractioninthecanvasorwood-panelsupportofthepainting,whichstressesthelayersofthepainting.Dryingcracksareusuallycausedbytheevaporationofvolatilepaintcomponentsandtheconsequentshrinkageofthepaint.Finally,mechanicalcracksresultfrompaintingdeformationsduetoexternalcauses,e.g.vibrationsand impacts.Theappearanceofcracksonpaintingsdeterioratestheperceivedimagequality.However,onecanusedigital
imageprocessing techniques todetectandeliminate thecracksondigitizedpaintings.Suchavirtual restorationcanprovidecluestoarthistorians,museumcuratorsandthegeneralpubliconhowthepaintingwould look like initsinitialstate,i.e.,withoutthecracks.Furthermore,itcanbeusedasanon-destructivetoolfortheplanningoftheactualrestoration.A systemthat iscapableoftrackingandinterpolatingcracks ispresentedin[1].Theusershouldmanuallyselectapointoneachcracktoberestored.Amethodforthedetectionofcracksusingmulti-orientedNovember30,2005 DRAFT

2/26
2
Gaborfiltersispresentedin[2].Crackdetectionandremovalbearscertainsimilaritieswithmethodsproposedforthedetectionandremovalofscratchesandotherartifactsfrommotionpicturefilms[3],[4],[5].However,suchmethodsrelyoninformationobtainedoverseveraladjacentframesforbothartifactdetectionandfillingandthusarenotdirectlyapplicableinthecaseofpaintingcracks.Otherresearchareasthatarecloselyrelatedtocrackremovalincludeimageinpaintingwhichdealswiththereconstructionofmissingordamagedimageareasbyfilling-in informationfromtheneighboringareas,anddisocclusion, i.e.,recoveryofobjectpartsthatarehiddenbehindotherobjectswithinanimage.Methodsdevelopedintheseareasassumethattheregionswhere informationhastobefilled-inareknown.Differentapproachesfor interpolating information instructured [6],[7],[8],[9],[10]and textured imageareas [11]havebeendeveloped.The formerareusuallybasedonpartialdifferentialequations(PDE)andonthecalculusofvariationswhereasthelatterrelyontexturesynthesisprinciples.Atechniquethatdecomposesthe imageto texturedandstructuredareasandusesappropriate interpolation techniquesdependingontheareawhere themissing information lieshasalsobeenproposed [12].Theresultsobtainedbythese techniquesare
very
good.
Amethodology
for
the
restoration
of
cracks
on
digitized
paintings,
which
adapts
and
integrates
a
numberofimageprocessingandanalysistoolsisproposedinthispaper.Themethodologyisanextensionofthecrack removal frameworkpresented in [13].The techniqueconsistsof the followingstages:
Crackdetection.Separationof the thindarkbrushstrokes,whichhavebeenmisidentifiedascracks.Crackfilling (interpolation).Acertaindegreeofuserinteraction,mostnotablyinthecrackdetectionstage,isrequiredforoptimalresults.
User interaction isratherunavoidablesincethe largevariationsobserved inthetypologyofcrackswould leadanyfullyautomaticalgorithmtofailure.However,allprocessingstepscanbeexecutedinrealtimeandthustheusercaninstantlyobservetheeffectofparametertuningontheimageunderstudyandselectinanintuitivewaythevaluesthatachievetheoptimalvisualresult.Needlesstosaythatonlysubjectiveoptimalitycriteriacanbeusedin thiscasesincenogroundtruthdataareavailable.Theopinionofrestorationexpertsthat inspectedthevirtuallyrestored imageswasverypositive.Thispaperisorganizedasfollows.SectionIIdescribesthecrackdetectionprocedure.Twomethodsforthe
separationofthebrushstrokeswhichhavebeenfalselyidentifiedascracksarepresentedinSectionIII.MethodsforfillingthecrackswithimagecontentfromneighboringpixelsareproposedinSectionIV.Conclusionsanddiscussion follow.
I I.DETECTIONOFCRACKSCracksusuallyhavelowluminanceandthuscanbeconsideredaslocalintensityminimawithratherelongated
structuralcharacteristics.Therefore,acrackdetectorcanbeappliedon the luminancecomponentofan imageandshouldbeableto identifysuchminima.Acrackdetectionprocedurebasedontheso-calledtop-hattransform [14]isproposed in thispaper.The top-hat transform isagrayscalemorphologicalfilterdefinedas follows:
y(x) = f(x) fnB(x) (1)November30,2005 DRAFT

3/26
3
wherefnB(x) istheopeningofthefunctionf(x) (inourcase,theluminancecomponentoftheimageunderstudy)with thestructuringsetnB ,definedas:
nB = BBB (ntimes) (2)Inthepreviousequationdenotesthedilationoperation.AsquareoracirclecanbeusedasstructuringelementB[15].ThefinalstructuringsetnB isevaluatedonlyonceusing(2)andisusedsubsequentlyintheopeningoperationof (1).TheopeningfnBofa function isa low-passnonlinearfilter thaterasesallpeaks (localmaxima) inwhichthestructuringelementnBcannotfit.Thus,theimagef fnBcontainsonlythosepeaksandnobackgroundatall.Sincecracksare localminimaratherthan localmaximathetop-hattransformshouldbeappliedonthenegatedluminanceimage.Alternatively,onecandetectcracksbyperformingclosingontheoriginalimagef(x)withthestructuringset nBand thensubtractingf(x) from the resultofclosingfnB(x):
y(x) = fnB(x) f(x) (3)Itcanbeeasilyshownthattheresultof(3)isidenticaltothatofapplying(1)onthenegatedimage.Useof(3)doesnot requirenegationoff(x) whichgrands itasmallbutnotnegligiblecomputationaladvantageover (1).Insituationswherethecrack-likeartifactsareofhighluminance,asinthecaseofscratchesonphotographs,
negationofthe luminancecomponentpriortothecrackdetectionisnotrequired, i.e.thecrackdetectionprocedurecanbeapplieddirectlyontheluminanceimage.Theusercancontroltheresultofthecrackdetectionprocedurebychoosingappropriatevalues for the followingparameters:
The typeof thestructuringelementB.Thesizeof thestructuringelementBand thenumbernofdilations in (2).
TheseparametersaffectthesizeofthefinalstructuringelementnB andmustbechosenaccordingtothethicknessofthecrackstobedetected.Itshouldbenotedhoweverthattheseparametersarenotverycriticalforthealgorithmperformanceduetothethresholdingoperationthatwillbedescribedinthenextparagraphandalsoduetotheexistenceofthebrushstroke/crackseparationprocedure(sectionIII),whichisabletoremovecrack-likebrushstrokesthathavebeenerroneouslyidentifiedascracks.Thefactthatalltheresultspresentedinthispaperhavebeenobtainedwiththesametop-hattransformparameterscomesasaclearindicationthattheabovestatementisindeedtrue.Theseparameterswere the following:
Structuringelement type:squareStructuringelementsize:3 3Numbernofdilations in (2):2Thetop-hattransformgeneratesagrayscaleoutputimaget(k, l) wherepixelswithalargegreyvaluearepotential
crackorcrack-likeelements.Therefore,a thresholdingoperationon t(k, l) isrequired toseparatecracks from therestoftheimage.ThethresholdTcanbechosenbyatrialanderrorprocedure,i.e.,byinspectingitseffectontheresultingcrackmap.The lowcomputationalcomplexityof the thresholdingoperationenables theuser toviewthecrackdetectionresultsinrealtimewhilechangingthethresholdvalue.e.g.,bymovingaslider.ThisfactNovember30,2005 DRAFT

4/26
4
makes interactive thresholdselectionveryeffectiveand intuitive.Alternatively, thresholdselectioncanbedonebyinspectingthehistogramoft(k, l)foralobeclosetothemaximumintensityvalue(whichwillmostprobablycorrespondtocrackorcrack-likepixels),andassigningitavaluethatseparatesthislobefromtherestoftheintensities.Theresultofthethresholdingisabinaryimageb(k, l)markingthepossiblecracklocations.Insteadofthisglobalthersholdingtechnique,morecomplexthresholdingschemes,whichuseaspatiallyvaryingthresholdcanbeused.Obviously,asthethresholdvalueincreasesthenumberofimagepixelsthatareidentifiedascracksdecreases.Thus,certaincracks,especiallyindarkimageareaswherethelocalminimumconditionmaynotbesatisfied,canremainundetected.Inprinciple,itismorepreferabletoselectthethresholdsothatsomecracksremainundetectedthantochooseathresholdthatwouldresultinthedetectionofallcracksbutwillalsofalselyidentifyascracks,andsubsequentlymodify,other imagestructures.Thethresholded (binary)outputofthetop-hattransformontheluminancecomponentofanimagecontainingcracks(Figure1)canbeseeninFigure2.Additionalexamplesofcracksdetectedusing thisapproachcanbeseen inFigures10-12.
I I I .SEPARATIONOFTHE BRUSHSTROKESFROMTH ECRACKSInsomepaintings,certainareasexistwherebrushstrokeshavealmostthesamethicknessandluminancefeatures
ascracks.Thehairofapersoninaportraitcouldbesuchanarea.Therefore,thetop-hattransformmightmisclassifythesedarkbrushstrokesascracks.Thus,inordertoavoidanyundesirablealterationstotheoriginalimage,itisveryimportanttoseparatethesebrushstrokesfromtheactualcracks,beforetheimplementationofthecrackfillingprocedure.Twomethods toachieve thisgoalaredescribed in the followingsubsections.A.Semi-automatic crack separation
Asimpleinteractiveapproachfortheseparationofcracksfrombrushstrokesistoapplyaregiongrowingalgorithmonthethresholdedoutputofthetop-hattransform,starting frompixels(seeds)ontheactualcracks.Thepixelsarechosenbytheuserinaninteractivemode.Atleastoneseedperconnectedcrackelementshouldbechosen.Alternatively,theusercanchoosetoapplythetechniqueonthebrushstrokes,ifthisismoreconvenient.Thegrowthmechanismthatwasusedimplementsthewell-knowngrassfirealgorithmthatchecksrecursivelyforunclassifiedpixelswithvalue1inthe8-neighborhoodofeachcrackpixel.Attheendofthisprocedure,thepixelsinthebinaryimage,whichcorrespondtobrushstrokesthatarenot8-connectedtocrackswillberemoved.Theaboveprocedurecanbeusedeitherinastand-alonemodeorappliedontheoutputoftheMRBFseparationproceduredescribed in thenextsection toeliminateany remainingbrushstrokes.B.Discrimination on the basisof hue and saturationHueH isassociatedwiththedominantwavelengthinamixtureoflightwavelengthsandrepresentsthedominant
color.IntheHSVcolormodel,hueisrepresentedastheanglearoundtheverticalaxis,withredat0,greenat120,andsoon.SaturationSreferstotheamountofwhite lightmixedwithacertainhue.Hueandsaturationare
November30,2005 DRAFT

5/26
5
definedsimilarly inotherrelatedcolordomains,e.g.inHueSaturation Intensity(HSI)orHueLightnessSaturation(HLS). Bystatisticalanalysisof47digitizedpaintings(24portablereligiousiconsfromtheByzantineeraand23
paintingsofvariousstylesandages),ithasbeenconcludedthatthehueofthecracksusuallyrangesfrom0to60.Onthecontrary,weobservedthatthehueofthedarkbrushstrokesvaries,asexpected,intheentiregamut[0,360].Furthermore,cracksaturationusuallyrangesfrom0.3to0.7,whilebrushstrokesaturationrangesfrom0to0.4.Thus,onthebasisoftheseobservations,agreatportionofthedarkbrushstrokes,falselydetectedbythetop-hattransform,canbeseparatedfromthecracks.ThisseparationcanbeachievedbyclassificationusingaMedianRadialBasisFunction(MRBF)neuralnetwork,whichisarobust,orderstatisticsbased,variationofRadialBasisFunction (RBF)networks [16].RBFsaretwo-layer feedforwardneuralnetworks [17],thatmodelamappingbetweenasetof inputvectorsand
asetofoutputs.Thenetworkarchitecture ispresentednFigure3.RBFs incorporatean intermediate,hidden layerwhere
each
hidden
unit
implements
akernel
function,
usually
aGaussian
function:
j(X) = exp[(j X)TS1(j X)],j= 1,...,L (4)
wherej,SjdenotethemeanvectorandthecovariancematrixforkerneljandLdenotesthenumberofunits(kernels) in thehidden layer.Eachoutputconsistsofaweightedsumofkernels. In typicalsituations that involvepatternclassification,thenumberofoutputsequalstothenumberofclasses. Insuchasetting,thecurrentvector isassignedtotheclassassociatedwiththeoutputunitexhibitingthemaximumactivation(winnertakesallapproach).Afterthelearningstage,thenetworkimplementstheinput-outputmappingruleandcangeneralizeittoinputvectorsnotbeingpartof the trainingset.Theparameters tobeestimated (learned) inaRBFnetworkarethecenter (mean)vectorjand thecovariance
matrixSjforeachGaussianfunctionandtheweightswk,jcorrespondingtotheconnectionsbetweenneuronsinthehidden layerandoutputnodes.Ahybridtechniquethathasbeenfrequentlyusedforthetrainingofsuchnetworks,hasbeenadoptedforthe learningstage.Accordingtothistechnique,training isperformed intwosuccessivesteps:thehiddenlayerparametersareestimatedusinganunsupervisedapproachand,afterwards,theoutputlayerweightsareupdated inasupervisedmanner,using the (nowfixed)hidden layerparametersevaluated in thepreviousstep.Intheclassicalversionoftheadoptedtrainingtechnique,avariationoftheLearningVectorQuantizer(LVQ)
algorithmisusedfortheunsupervisedhiddenlayerparameterupdating.EachinputvectorisassignedtotheGaussiankernelwhosecenter iscloser (in termsofeither theEuclideanor theMahalanobisdistance) to thisvector:
if Xi j= min Lk=1Xi k thenXiCj (5)where denoteseitherEuclideanorMahalanobisdistanceandCjdenotestheclassofinputvectorsassociatedwithkernelj.Subsequently,thealgorithmupdatesthecenterandcovariancematrixofthewinnerkernelusingrunningversionsoftheclassicalsamplemeanandsamplecovariancematrixformulas.Ontheotherhand,theMRBFalgorithmwhichhasbeenusedinourcaseisbasedonrobustestimation[16]ofthehiddenunitparameters.

6/26
j= marg na me an 0 j ,..., n1 j
0 ifX/C
6
ItemploystheMarginalMedianLVQ[18]thatselectsthewinnerkernelusing(5)andutilizesthemarginalmedianofthe inputvectorscurrentlyassignedtothiskernel fortheupdateofthecentervector (locationparameter)jofthekernel:
j= marginalmedian{X0,X1,...,Xn1}; (6)whereXn1isthelastvectorassignedtokernelj.Theupdateofthediagonalelementsofthecorrespondingcovariancematrixisperformedusingthemedianoftheabsolutedeviations(MAD)[19]oftheinputscurrentlyassigned to thiskernel:
0.6745 (7)Inthepreviousexpression,jdenotesthevectorcontainingthediagonalelementsofthecovariancematrixand|X|denotesthevectorobtainedbytakingtheabsolutevalueofeachcomponentofX.Theoff-diagonalcomponentsofthecovariancematrixarealsocalculatedbasedonrobuststatistics[16].Inordertoavoidexcessivecomputationstheaboveoperationscanbeappliedonasubsetofdataextractedthroughamovingwindowthatcontainsonlythelast Wdata samplesassigned to thehiddenunitj.Inthesupervisedpartofthe learningprocedure,theweightsoftheoutput layer,whichgrouptheclusters found
bythehiddenlayerintoclasses,areupdated.Theupdatemechanismfortheseweightsisdescribedbythefollowingexpression:
wk,j(t+ 1)= wk,j(t)+ nw(Fk(X) Yk(X))Yk(X)(1 Yk(X))j(X) (8)fork= 1,...,M,j= 1,...,L,anda learning factornw(0,1]. In theprevious formulaYk(X),Fk(X) denotetheactualanddesirednetworkoutput for inputvectorX.The latter isgivenby:
1 ifXCkFk(X) =
k (9)Theupdate formula (8)corresponds to thebackpropagation for theoutputweightsofaRBFnetworkwith respectto themeansquareerrorcost function.Inourimplementation,aMRBFnetworkwithtwooutputswasused.Thefirstoutputrepresentstheclassof
crackswhilethesecondonetheclassofbrushstrokes.Inputvectorsweretwo-dimensionalandconsistedofthehueandsaturationvaluesofpixelsidentifiedascracksbythetop-hattransform.Thenumberofclusters(hiddenunits)chosenforeachclassdependsontheoverlapbetweenthepopulationsofcracksandbrushstrokes.Ifthereisasubstantialoverlap,thenumbershouldbeincreased,inordertoreducetheclassificationerror.Inourimplementationthreehiddenunitshavebeenincorporated.Trainingwascarriedoutbypresentingthenetworkwithhueandsaturationvaluesforpixelscorrespondingtocracksandcrack-likebrushstrokes.Datafrom24digitizedportable religious icons from theByzantineerawereused for thispurpose.Thesystem trainedusing thisspecifictrainingsetcanbeconsideredtobeoptimizedforpaintingsofthisstyleanditsuseonpaintingsofotherstylemightresult insomewhatsuboptimalresults.However,appropriatelyselectedtrainingsetscanbeused totrainthesystemtoseparatecracksfrombrushstrokesonpaintingsofdifferentartisticstylesorcontent.InordertoselectNovember30,2005 DRAFT

7/26
7
pixelscorrespondingtocracksandcrack-likebrushstrokesthecrackdetectionalgorithmpresentedinsectionII wasappliedonthese images.Resultsweresubsequentlypost-processedbyanexpertusingthesemi-automaticapproachpresentedinsectionIII-A.Theaimofthispost-processingstepwastwofold:toremovepixelsthatareneithercracksnorcrack-likebrushstrokesandtoseparatecracksandcrack-likebrushstrokesforthesupervisedstepofthetrainingprocedure.Inthissupervisedtrainingstep,thenetworkwaspresentedwiththeselabelledinputs,i.e.,pairsofhue-saturationvaluesthatcorrespondedtoimagepixelsthathavebeenidentifiedasbelongingtocracksandcrack-likebrushstrokes.Afterthetrainingsession,theMRBFneuralnetworkwasabletoclassifypixelsidentifiedascracksbythetop
hattransformtocracksorbrushstrokes.Thetrainednetworkhasbeentestedon12imagesfromthetrainingsetand15 images(ofthesameartisticstyleandera)thatdidnotbelongtothetrainingset.Naturally,theperformanceofthecracks/brushstrokesseparationprocedurewasjudgedonly inasubjectivemanner(i.e.byvisualinspectionoftheresults),asgroundtruthdata(i.e.brushstrokes-freecrackimages)arenotavailable.Forthisreasontworestoration
experts
were
asked
toinspect
several
crack
images
before
and
after
the
application
of
the
separation
systemandconcludedthatintheprocessedcrack imagesthegreatmajorityofthebrushstrokeshasbeenremoved.Athresholdedtop-hattransformoutputcontainingmanybrushstrokese.g.hairisillustratedinFigure5.Agreatpartof thesebrushstrokes isseparatedby theMRBF,ascanbeseen inFigure6.Theoriginal imagecanbeseeninFigure4.
IV.CRACKFILLINGMETHODSAfteridentifyingcracksandseparatingmisclassifiedbrushstrokes,thefinaltaskistorestoretheimageusing
localimageinformation(i.e.,informationfromneighboringpixels)tofill(interpolate)thecracks.Twoclassesoftechniques,utilizingorderstatisticsfilteringandanisotropicdiffusionareproposedforthispurpose.BothareimplementedoneachRGBchannel independentlyandaffectonly thosepixelswhichbelong tocracks.Therefore,providedthattheidentifiedcrackpixelsareindeedcrackpixels,thefillingproceduredoesnotaffecttheusefulcontentoftheimage. Imageinpaintingtechniquesliketheonescited inSectionIcanalsobeusedforcrackfilling.Theperformanceofthecrackfillingmethodspresentedbelowwasjudgedbyvisualinspectionoftheresults.
Obviously,measuringtheperformanceofthesemethodsinanobjectivewayisinfeasiblesincegroundtruthdata(e.g.imagesdepictingthepaintingsinperfectcondition,i.e.,withoutcracks)arenotavailable.For theevaluationoftheresults, tworestorationexpertswereaskedto inspectseveral imagesrestoredusing thevariousmethodsandcomment,basedontheirexperience,onthequalityofthefillingresults,(i.e.,whethercracksweresufficientlyfilled),whether thecolorused forfillingwas thecorrectone,whetherfine imagedetailswere retained,etc.A.Crackfilling basedon order statisticsfilters
Aneffectivewaytointerpolatethecracksistoapplymedianorotherorderstatisticsfilters[15]intheirneighborhood.Allfiltersareselectivelyappliedonthecracks,i.e.,thecenterofthefilterwindowtraversesonlythecrackpixels.Ifthefilterwindowissufficientlylarge,thecrackpixelswithinthewindowwillbeoutliersandNovember30,2005 DRAFT

8/26
1 otherwise
8
willbe rejected.Thus, thecrackpixelwillbeassigned thevalueofoneof theneighboringnon-crackpixels.Thefollowingfilterscanbeused for thispurpose:
Medianfilter:yi= med(xi,...,xi,...,xi+) (10)
Recursivemedianfilter:yi= med(yi,...,yi1,xi,...,xi+) (11)
wheretheyi,...,yi1arethealreadycomputedmedianoutputsamples.Forboththerecursivemedianandthemedianfilter,thefilterwindow (consideringonlyrectangularwindows)shouldbeapproximately50%widerthanthewidest(thickest)crackappearingontheimage.Thisisnecessarytoguaranteethatthefilteroutput isselected tobe thevalueofanon-crackpixel.Smallerwindowswill result incracks thatwillnotbesufficientlyfilledwhereaswindowsthataremuchwider than thecrackswillcreate largehomogeneousareas,thusdistortingfine imagedetails.
Weightedmedianfilter:yi= med(wxi,...,wxi+) (12)
wherewxdenotesduplicationofx, wtimes.Forthisfilter,smallerfilterwindows(e.g.windowsthatareapproximately30%widerthanthewidestcrackappearingontheimage)canbeusedsincetheprobabilitythatacolorvaluecorrespondingtoacrack isselectedasthefilteroutput(afactthatwouldresult inthecrackpixelunder investigationnotbeingfilledeffectivelyby thefilter)canbe limitedbyusingsmallweights (e.g.1)forthepixelscentrallylocatedwithinthewindow(whichareusuallypartofthecrack)andbiggerones(e.g.2or3) for theotherpixels.
Avariationofthemodifiedtrimmedmean(MTM)filterwhichexcludesthesamplesxi+r,j+sinthefilterwindow,whichareconsiderablysmaller from the localmedianandaverages the remainingpixels:
yij= Arsxi+r,j+sThesummationscover theentirefilterwindowA .Thefiltercoefficientsarechosenas follows:
0 if med{xij} xi+r,j+sqrs= (14)
Theamountoftrimmingdependsonthepositiveparameterq.Dataofsmallvaluedeviatingstronglyfromthelocal
median
(which
correspond
usually
to
cracks)
are
trimmed
out.
Windows
used
along
with
this
variant
of
theMTMfiltercanalsobesmallerthanthoseusedforthemedianandrecursivemedianfilterssinceaportionof thecrackpixels isexpected tobe rejectedby the trimmingprocedure.
AnothervariationoftheMTMfilterthatperformsaveragingonlyonthosepixelsthatarenotpartofthecrack,i.e., itutilizes informationfromthebinaryoutput imageb(k, l) ofthetophattransform.Inthiscase,thefilter

9/26
Ii,j = Ii,j+ [cN DNI+ cS DSI+ cE DEI+ cWDWI]ti,j (17)
coefficients in (13)arechosenas follows:rs=
9
1if b(k, l) = 10otherwise (15)
Themedianoperatorcanbeusedinsteadofthearithmeticmeanin(13).ForthisvariantoftheMTMfilter,evensmallerfilterwindowscanbeused,sincecrackpixelsdonotcontributetothefilteroutput.Thus,itsuffices that thewindow is1pixelwider than thewidestcrack.
The resultof theapplicationof thesecondvariationof themodified trimmedmeanfilteron thepaintingdepictedinFigure1aisshown inFigure7(filtersize5 5).Anotherimagerestoredbythesamecrack-fillingapproachcanbeseen inFigure8(filtersize3 3).Extensiveexperimentationprovedthatthisfiltergivesthebestresultsamongallfilterspresentedaboveaccordingtosubjectiveevaluationbyrestorationexperts.Thesuperiorityofthisfiltercanbeattributedtothe factthatonlynon-crackpixelscontributeto itsoutput.Onthecontrary, forallotherfilterspresented in thissection theprobability thatcrackpixelswillcontribute to theoutput issmallbutnotnegligible.B.ControlledAnisotropicDiffusionAnisotropicdiffusion[20]isanimageenhancementmethodthatsuccessfullycombinessmoothingofslowly
varyingintensityregionsandedgeenhancement.Smoothingismodelledasadiffusionthatisallowedalonghomogeneousregionsandinhibitedbyregionboundaries.Anisotropicdiffusionisdescribedbythefollowingpartialdifferentialequation:
I (x, y, t) t = div(c(x, y,t)I(x, y,t)) = c(x, y,t)I(x, y,t) +c(x, y,t)I(x, y,t) (16)
wheredivdenotesthedivergenceoperatorand ,thegradientandLaplacianoperatorswithrespecttothespacevariablesx, y.Ateachpositionanditeration,diffusioniscontrolledbytheconduction(ordiffusion)coefficientsc(x, y,t).Sincediffusionshouldbeinhibitedacrossregionsseparatedbydiscontinuities,theconductioncoefficientsshouldobtainsmallvalues inpixelswith large intensitygradientmagnitude.AsimilarapproachnamedCurvatureDrivenDiffusion (CCD)hasbeenproposed in [9] for image inpaintingapplications.Inordertoobtainanumericalsolutiontothediffusionequation,discretizationofthespatialandtimecoordinates
andapproximationofthedifferentialoperatorsbyfinitedifferenceoperatorsshouldbeperformed in(16).Wehaveusedthesamediscretizationsasproposedin[20].Theiterative,discretesolutionto(16)isgovernedbythefollowingequation:
t twhere0 1/4fortheschemetobestable,N,S,E, WarethemnemonicsforNorth,South,East,Westand

10/26
ctNi,j= g( | DNIi,j |)ctSi,j= g( | DSIi,j |) ctEi,j= g( | DEIi,j |)
ctWi,j= g( | DWIi,j |) (19)
10
thesymbol Dindicatesnearest-neighbordifferences:DNIi,j Ii1,j Ii,jDSIi,j Ii+1,j Ii,jDEIi,j Ii,j+1 Ii,jDWIi,j Ii,j1 Ii,j (18)
Theconductioncoefficientsareevaluatedateveryiterationasafunctiong( )ofthemagnitudeoftheintensitygradient. Inour implementation, the followingapproximationwasused [20]:
tttt
The following functiong( ),proposed in [20],hasbeenused inourcase:g(I) = 1
1+( ITheconstantKwasmanuallyfixed.Inordertofillthecracks,theanisotropicdiffusionalgorithmwasapplied
selectively,inneighborhoodscenteredoncrackpixels.Allpixelswithintheseneighborhoodsparticipateinthediffusionprocess.However,only thevaluesof thecrackpixelsareupdated in theoutput image.Furtherimprovementswereobtainedbytakingintoaccountcrackorientation,i.e.,byapplyingtheoperationonly
inadirectionperpendiculartothecrackdirection.Forexample,ifthecrackishorizontal,onecanuseonlytheNorthandtheSouthneighbors,sincetheWestandtheEastneighborsbelongalsotothecrack. Inordertofindthedirectionsofthecracks,theHoughTransformwasapplied[21].TherestorationresultsofthisfiltercanbeseeninFigures9-12.Theorientation-sensitivecontrolledanisotropicdiffusionmethodgavethebestresultsamongallcrack-fillingmethodspresented inthispaper.The factthatthisfilterrequiresnowindowsizeselectiongrants itanadditionaladvantageover theorderstatisticsfilterspresented in theprevioussection.
V.CONCLUSIONS AND DISCUSSIONInthispaper,wehavepresentedanintegratedstrategyforcrackdetectionandfillingindigitizedpaintings.
Cracksaredetectedbyusingtop-hattransform,whereasthethindarkbrushstrokes,whicharemisidentifiedascracks,areseparatedeitherbyanautomatic technique (MRBFnetworks)orbyasemi-automaticapproach.Crackinterpolation isperformedbyappropriatelymodifiedorder statisticsfiltersor controlledanisotropicdiffusion.Themethodologyhasbeenappliedforthevirtualrestorationofimagesandwasfoundveryeffectivebyrestorationexperts.However,therearecertainaspectsoftheproposedmethodologythatcanbefurtherimproved.Forexample,

11/26
11
thecrackdetectionstageisnotveryefficientindetectingcrackslocatedonverydarkimageareas,sinceintheseareastheintensityofcrackpixelsisveryclosetotheintensityofthesurroundingregion.A possiblesolutiontothisshortcomingwouldbetoapplythecrackdetectionalgorithm locallyonthisareaandselecta low thresholdvalue.Anothersituationwherethesystem(moreparticularly,thecrackfillingstage)doesnotperformasefficientlyasexpectedisinthecaseofcracksthatcrosstheborderbetweenregionsofdifferentcolor.Insuchsituations,itmightbethecasethatpartofthecrackinoneareaisfilledwithcolorfromtheotherarea,resultinginsmallspursofcolorintheborderbetweenthetworegions.SuchasituationisdepictedinFigure11.Howeverthisphenomenonisratherseldomand, furthermore,theextentoftheseerroneouslyfilledareas isverysmall (2-3pixelsmaximum).A possiblesolutionwouldbetoperformedgedetectionorsegmentationontheimageandconfinethefillingofcracksthatcrossedgesorregionborderstopixelsfromthecorrespondingregion.Useof image inpaintingtechniques[6],[7],[8],[9],[10]couldalsoimproveresultsinthataspect.Anotherimprovementofthecrackfillingstagecouldaimatusingproperlyadaptedversionsofnonlinearmultichannelfilters(e.g.variantsofthevectormedianfilter)instead
of
processing
each
color
channel
independently.
These
improvements
will
be
the
topic
of
future
work
on
thissubject.
REFERENCES [1]M.Barni,F.Bartolini,V.Cappellini,Imageprocessingforvirtualrestorationofartworks,IEEEMultimedia,vol.7,no.2,pp.34-37,
April-June2000.[2]F.Abas,K.Martinez,CraquelureAnalysisforContent-basedRetrieval,inProc.14thInternationalConferenceonDigitalSignal
Processing,2002,vol1,pp.111-114.[3]L. Joyeux,O.Buisson,B. Besserer,S.Boukir,Detectionandremovaloflinescratchesinmotionpicturefilms,inProc.IEEEInternational
ConferenceonComputerVisionandPatternRecognition,1999,pp.548-553.[4]A.Kokaram,R. Morris,W.Fitzgerald,P.Rayner,Detectionofmissingdatainimagesequences,IEEETransactionsonImageProcessing,
vol.4,no.11,pp.1496-1508,November1995.[5]A.Kokaram,R.Morris,W.Fitzgerald,P.Rayner,Interpolationofmissingdatainimagesequences,IEEETransactionsonImage
Processing,vol.4,no.11,pp.1509-1519,November1995.[6]M.Bertalmio,G.Sapiro,V.Caselles,C.Ballester,Image Inpainting, inProc.SIGGRAPH2000,pp.417424,2000.[7]C.Ballester,M.Bertalmio,V.Caselles,G.Sapiro, J.Verdera,Filling-InbyJoint InterpolationofVectorFieldsandGrayLevels,IEEE
Transactionson ImageProcessing,vol.10,no.8,pp.12001211,August2001.[8]S.Masnou, J.M.Morel,LevelLinesBaseddisocclusion, inProc. IEEE ICIP98,vol. III,pp.259263,1998.[9]T.Chan,J.Shen,Non-textureinpaintingsbycurvature-drivendiffusions,JournalofVisualCommunicationandImageRepresentation,
vol.12,no.4,pp.436-449,2001.[10]S.Esedoglu,J.Shen,DigitalInpaintingBasedontheMumford-Shah-EulerImageModel,EuropeanJournalofAppliedMathematics,
vol.13,pp.353-370,2002.[11]A.Efros,T.Leung,TextureSynthesisbyNon-parametricSampling, inProc.1999 IEEE InternationalConferenceonComputerVision(ICCV),pp.10331038,1999.[12]M.Bertalmio,L.Vese,G.Sapiro,S.Osher,SimultaneousStructureandTextureImageInpainting,IEEETransactionsonImage
Processing,vol.12,no.8,pp.882-889,August2003.[13]I.Giakoumis, I. Pitas,DigitalRestorationofPaintingCracks, inProc. IEEE InternationalSymposiumonCircuitsandSystems,vol.4,
pp.269-272,1998.[14]F.Meyer,Iterative image transforms foranautomaticscreeningofcervicalsmears,J.Histoch.Cytochem.,vol.27,pp.128135,1979.[15]I.Pitas,A.N.Venetsanopoulos,NonlinearDigitalFilters,principlesandapplications,Norwell:KluwerAcademic,1990.November30,2005 DRAFT

12/26
12
[16]A.G.Bors,I.Pitas,MedianRadialBasisFunctionNeuralNetwork,IEEETransactionsonNeuralNetworks,vol.7,no.6,pp.1351-1364,November1996
[17]S.Haykin,NeuralNetworks,AComprehensiveFountation,2ndEdition,NewYork:PrenticeHall,1999.[18]I.Pitas,C.Kotropoulos,N.Nikolaidis,R.Yang,M.Gabbouj,OrderStatisticsLearningVectorQuantizer,IEEETransactionson Image
Processing,vol.5,no.6,pp.1048-1053, June1996.[19]G.Seber,MultivariateObservations,NewYork: JohnWiley,1986.[20]P.Perona,J.Malik,Scale-SpaceandEdgeDetectionusinganisotropicdiffusion,IEEETransactionsonPatternAnalysisandMachine
Intelligence,vol.12,no.7,pp.629639,July1990.[21]P.Hough, AMethodandMeans forRecognizingComplexPatterns,U.S.Patent,3069654,1962.

13/26
13
Fig.1.Originalpainting.

14/26
14
Fig.2.Thresholdedoutputof the tophat transform (thresholdvalue:23).

15/26
15
Fig.3.RadialBasisFunctionsneuralnetworkarchitecture.

16/26
16
Fig.4.Originalpainting.

17/26
17
Fig.5. Thresholdedoutputofthetop-hattransform(thresholdvalue:19).Anumberofbrushstrokes(hair,numberintheupperleftcorner)havebeenmisidentifiedascracks.

18/26
18
Fig.6.Theseparatedbrushstrokesafter theapplicationof theMRBF technique.

19/26
19
Fig.7.Crackfillingbyusing themodified trimmedmeanfilter(filtersize55).

20/26
20
Fig.8.(a)Originalpainting (detail). (b)Crackfillingbyusing themodified trimmedmeanfilter (filtersize33).

21/26
21
Fig.9.(a)Original imagewithcracks (detail), (b)crackfillingusing theorientation-sensitivecontrolledanisotropicdiffusion technique.

22/26
22
Fig.10.(a)Originalpainting(detail),(b)crackimage(thresholdvalue:17),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique.

23/26

24/26
24
Fig.12.(a)Originalpainting(detail),(b)crackimage(thresholdvalue:23),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique.

25/26
25
IoannisGiakoumisreceived theDiplomaof Informatics in1997.He iscurrentlywith IntracomS.A.
PLACEPHOTOHERE
NikosNikolaidisreceivedtheDiplomaofElectricalEngineeringin1991andthePhDdegreeinElectricalEngineeringin1997,bothfromtheAristotleUniversityofThessaloniki,Greece.From1992to1996heservedasteachingassistantintheDepartmentsofElectricalEngineeringandInformaticsatthesameUniversity.From1998to2002hewaspostdoctoralresearcherandteachingassistantattheDepartmentofInformatics,AristotleUniversityofThessaloniki.He iscurrentlyaLecturer inthesameDepartment.Dr.Nikolaidis istheco-authorofthebook3-DImageProcessingAlgorithms(J.Wiley,2000).Hehasco-authored2bookchapters,18journalpapersand46conferencepapers.Hisresearchinterestsincludecomputergraphics,imageandvideoprocessingandanalysis,copyrightprotectionof
multimediaand3-D imageprocessing.Dr.Nikolaidishasbeenascholarof theStateScholarshipFoundationofGreece.
PLACEPHOTOHERE
IoannisPitasreceivedtheDiplomaofElectricalEngineering in1980andthePhDdegreeinElectricalEngineering in1985bothfromtheAristotleUniversityofThessaloniki,Greece.Since1994,hehasbeenaProfessorattheDepartmentofInformatics,
Aristotle
University
ofThessaloniki.
From
1980
to1993
he
served
asScientific
Assistant,
Lecturer,
AssistantProfessor,andAssociateProfessorintheDepartmentofElectricalandComputerEngineeringatthesameUniversity.HeservedasaVisitingResearchAssociateattheUniversityofToronto,Canada,UniversityofErlangen-Nuernberg,Germany,TampereUniversityofTechnology,Finland,asVisitingAssistantProfessorattheUniversityofTorontoandasVisitingProfessorattheUniversityofBritishColumbia,Vancouver,Canada.Hewaslecturerin
shortcoursesforcontinuingeducation.Hehaspublished140journalpapers,350conferencepapersandcontributedin18booksinhisareasofinterest.Heistheco-authorofthebooksNonlinearDigitalFilters:PrinciplesandApplications(Kluwer,1990),3-DImageProcessingAlgorithms(J.Wiley,2000),NonlinearModel-BasedImage/VideoProcessingandAnalysis(J.Wiley,2001)andauthorofDigitalImageProcessingAlgorithmsandApplications(J.Wiley,2000).HeistheeditorofthebookParallelAlgorithmsandArchitecturesforDigitalImageProcessing,ComputerVisionandNeuralNetworks (Wiley,1993).Hehasalsobeenan invitedspeakerand/ormemberof theprogramcommitteeofseveralscientificconferencesandworkshops. In thepastheservedasAssociateEditorofthe IEEETransactionsonCircuitsandSystems,IEEE TransactionsonNeuralNetworks,IEEETransactionsonImageProcessing,EURASIPJournalonAppliedSignalProcessingandco-editorofMultidimensionalSystemsandSignalProcessing.Hewasgeneralchairofthe1995IEEEWorkshoponNonlinearSignalandImageProcessing(NSIP95),technicalchairofthe1998EuropeanSignalProcessingConferenceandgeneralchairofIEEEICIP2001.Hiscurrentinterestsareintheareasofdigitalimageandvideoprocessingandanalysis,multidimensionalsignalprocessing,watermarkingandcomputervision.

26/26
26
LIS TOFF IGURES1 Originalpainting............................................... 132 Thresholdedoutputof the tophat transform (thresholdvalue:23)..................... 143 RadialBasisFunctionsneuralnetworkarchitecture............................. 154 Originalpainting............................................... 165 Thresholdedoutputofthetop-hattransform(thresholdvalue:19).Anumberofbrushstrokes(hair,
number in theupper leftcorner)havebeenmisidentifiedascracks.................... 176 Theseparatedbrushstrokesafter theapplicationof theMRBF technique................ 187 Crackfillingbyusing themodified trimmedmeanfilter (filtersize5 5)................ 198 (a)Originalpainting(detail).(b)Crackfillingbyusingthemodifiedtrimmedmeanfilter(filtersize
3 3)..................................................... 209 (a)Originalimagewithcracks(detail),(b)crackfillingusingtheorientation-sensitivecontrolled
anisotropicdiffusion technique. ...................................... 2110 (a)Originalpainting(detail),(b)crackimage(thresholdvalue:17),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique............................. 22
11 (a)Originalpainting(detail),(b)crackimage(thresholdvalue:21),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique............................. 23
12 (a)Originalpainting(detail),(b)crackimage(thresholdvalue:23),(c)crackfillingusingtheorientation-sensitivecontrolledanisotropicdiffusion technique............................. 24

Digital Image Processing Techniques for the Detection and Removal of Cracks in Digitized Painting

Documents

Assessing Load Transfer Across Joints and Cracks in Rigid...

Fatigue Tests of Hybrid Plate Girders Under Combined...

“A Digitized Enterprise needs a Digitized...

digitized by

Rendering Cracks in Batik - University of...

Visible cracks

Building Cracks

why concrete...

ISSN 2348 – 7968 Detection and Removal of Cracks in...

Digitized by Archive

The Str uctur al Conser va P anel Paintings - Getty ·...

Transverse Cracks Corner Cracks Pinholes Longitudinal...

Providing Turnkey Engineered Solutions - ESPRAT and welding....

Digitized for FRASER …...Digitized for FRASER Federal...

Los Cracks

Digitized Special Collections