Top Banner
SPATIALLY ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford GU2 7XH, United Kingdom. {g.forbin;t.vlachos;s.tredwell}@surrey.ac.uk. Keywords: Film archives, flicker, spatial correction, image restoration. Abstract We present an algorithm suitable for the compensation of flicker in archived film sequences. The proposed method is a substantial improvement over our previous work which adopted a nonlinear approach to flicker compensation motivated by fundamental principles of photographic image registration. The improvements include reliability weighting, spatial adaptation and motion compensated weighting. We present experimental evidence which suggests that our method compares favourably with competing state-of-the-art techniques for flicker correction. 1 Introduction The automatic restoration of film archives is a key enabling technology toward the successful exploitation of film and television archives [1] for a number of reasons. By improving basic picture quality and by reducing the perceptual impact of archive-related artefacts, it can meet viewers aesthetic expectations, improve the level of resolvable spatial and temporal detail, and ultimately enrich the viewing experience. Moreover, the suppression of such artefacts has a significant beneficial impact on the efficiency of video coding algorithms used in television and multimedia distribution chains such as MPEG-2 and MPEG-4. 1.1 Problem description Flicker refers to random temporal fluctuations in image intensity and is a common artefact in archived film sequences. The main contributing cause of flicker is inconsistent film exposure at the image acquisition stage. Other causes may include printing errors in film processing, film ageing, multiple copying, mould, and dust. Flicker is one of the most commonly encountered artefacts in archived film. It is immediately recognisable, even by non- expert viewers, as a signature artefact of old film sequences. Its perceptual impact can be significant as it interferes substantially with the viewing experience and has the potential of concealing essential details. Film flicker is very noticeable and can be quite unsettling to the viewer, especially in cases where film is displayed simultaneously with video or with electronically generated graphics and captions as is typically the case in modern-day television documentaries. It may also lead to considerable discomfort and eye fatigue after prolonged viewing. Camera and scene motion can partly mask film flicker and, as a consequence, the latter is much more noticeable in sequences consisting primarily of still frames or frames with low motion content. Flicker has often been categorised as a global artefact in the sense that it usually affects all the frames of a sequence in their entirety as opposed to so-called local artefacts such as dirt, dust, or scratches which affect a limited number of frames and are usually localised on the image plane. Nevertheless it is by no means constant within the boundaries of a single frame as explained in the next section and one of the main aims of this work is to address this issue. 1.2 Spatial variability Spatial variability of flicker can manifest itself in one of the following ways. First, when flicker affects approximately the same position of all frames in a sequence. This may occur directly during film shooting if the lighting is not properly synchronised with the shutter of the camera. For example, if part of the scene is illuminated with synchronised light while the rest is illuminated with natural light a localised flickering effect may occur. Flicker can also be due to fogging which is caused by the accidental exposure of film to incident light or the partial immersion of the film strip in the developer bath. In addition there can be other contributing causes, such as drying stains from chemical agents or vignetting. However, it is also possible that the position of flicker artefacts vary randomly. This is the case when the film strip ages badly and becomes affected by mould, or when it is charged with static electric charge generated from mechanical friction. The return to a normal state often produces static marks.
10

SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

Mar 27, 2018

Download

Documents

ledien
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

SPATIALLY ADAPTIVE FLICKER COMPENSATION FORARCHIVED FILM SEQUENCES USING A NONLINEAR

MODEL

Guillaume Forbin, Theodore Vlachos, Simon Tredwell,

Centre for Vision, Speech and Signal Processing (CVSSP),University of Surrey, GuildfordGU2 7XH, United Kingdom.

{g.forbin;t.vlachos;s.tredwell}@surrey.ac.uk.

Keywords: Film archives, flicker, spatial correction, imagerestoration.

Abstract

We present an algorithm suitable for the compensation offlicker in archived film sequences. The proposed methodis a substantial improvement over our previous work whichadopted a nonlinear approach to flicker compensationmotivated by fundamental principles of photographic imageregistration. The improvements include reliability weighting,spatial adaptation and motion compensated weighting. Wepresent experimental evidence which suggests that ourmethod compares favourably with competing state-of-the-arttechniques for flicker correction.

1 Introduction

The automatic restoration of film archives is a key enablingtechnology toward the successful exploitation of film andtelevision archives [1] for a number of reasons. By improvingbasic picture quality and by reducing the perceptual impactof archive-related artefacts, it can meet viewers aestheticexpectations, improve the level of resolvable spatial andtemporal detail, and ultimately enrich the viewing experience.Moreover, the suppression of such artefacts has a significantbeneficial impact on the efficiency of video coding algorithmsused in television and multimedia distribution chains such asMPEG-2 and MPEG-4.

1.1 Problem description

Flicker refers to random temporal fluctuations in imageintensity and is a common artefact in archived film sequences.The main contributing cause of flicker is inconsistent filmexposure at the image acquisition stage. Other causes mayinclude printing errors in film processing, film ageing, multiplecopying, mould, and dust.

Flicker is one of the most commonly encountered artefacts inarchived film. It is immediately recognisable, even by non-

expert viewers, as a signature artefact of old film sequences.Its perceptual impact can be significant as it interferessubstantially with the viewing experience and has the potentialof concealing essential details. Film flicker is very noticeableand can be quite unsettling to the viewer, especially in caseswhere film is displayed simultaneously with video or withelectronically generated graphics and captions as is typicallythe case in modern-day television documentaries. It mayalso lead to considerable discomfort and eye fatigue afterprolonged viewing. Camera and scene motion can partly maskfilm flicker and, as a consequence, the latter is much morenoticeable in sequences consisting primarily of still frames orframes with low motion content.

Flicker has often been categorised as a global artefact in thesense that it usually affects all the frames of a sequence intheir entirety as opposed to so-called local artefacts such as dirt,dust, or scratches which affect a limited number of frames andare usually localised on the image plane. Nevertheless it is byno means constant within the boundaries of a single frame asexplained in the next section and one of the main aims of thiswork is to address this issue.

1.2 Spatial variability

Spatial variability of flicker can manifest itself in one of thefollowing ways. First, when flicker affects approximately thesame position of all frames in a sequence. This may occurdirectly during film shooting if the lighting is not properlysynchronised with the shutter of the camera. For example, ifpart of the scene is illuminated with synchronised light whilethe rest is illuminated with natural light a localised flickeringeffect may occur. Flicker can also be due to fogging which iscaused by the accidental exposure of film to incident light orthe partial immersion of the film strip in the developer bath. Inaddition there can be other contributing causes, such as dryingstains from chemical agents or vignetting.

However, it is also possible that the position of flicker artefactsvary randomly. This is the case when the film strip ages badlyand becomes affected by mould, or when it is charged withstatic electric charge generated from mechanical friction. Thereturn to a normal state often produces static marks.

Page 2: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

Figure 1: Test sequence “boat” used to illustrate spatial variabilityof flicker measured at selected location

Figure 2: Evolution of the median intensity of the blocks

Figure 1 shows the first frame of the test sequence “boat”(taken from “Our Shrinking World” 1 (1946)). The cameralingers in the same position during the 93 frames of thesequence. There is some slight unsteadiness. Despite somelocal movement overall motion content is low. This sequenceis chosen to illustrate the spatial variation of flicker as no flickeris perceivable on the top-left part of the shot, while the bottom-left part changes from initially bright to dark later on. On theopposite side of the image, flicker is more noticable, with fastvariations of high amplitude. This is shown in Figure 2, wherethe median intensities of four blocks (16 × 16 pixels) locatedat different parts of the frame are plotted as a function of framenumber.

The selected blocks are motionless, low-textured and havepairwise similar grey-levels (A, B and C, D) at the start of thesequence. As the sequence evolves we can clearly observe that

1Our Shrinking World (1946) - Young America Films, Inc. - Sd, B&W.

each of the two blocks in each pair undergoes a substantiallydifferent level of flicker with respect to the other block

This paper is organised as follows. Section 2 contains abrief overview of the non-adaptive non-linear compensationalgorithm. Section 4, 5 and 6 describe respectively the newfeatures namely reliability weighting, spatial adaptation andmotion-compensated weighting. Finally Section 7 containsexperimental results and comparisons while conclusions aredrawn is Section 8.

2 Background

Previous research addressing flicker compensation hasfrequently lead to a linear model in which the correctedframe is obtained by a linear transformation of the originalpixel values. Initial effort formulated a global model whichassumed that the whole degraded frame was affected withthe same intensity. In [2], flicker was modelled as a globalintensity shift between a degraded frame and the mean levelof the shot to which this frame belongs. In [3] the flicker wasexpressed as a multiplicative constant relating the mean levelof a degraded frame to a reference frame. Both the additiveand multiplicative models above require the estimation ofa single parameter which although straightforward fails toaccount for spatial variability. In [4] it was observed thatarchive material typically has a limited dynamic range.Histogram stretching has been applied to individual framesallowing the available dynamic range to be used in its entirety.Despite the general improvement in picture quality the authorsadmitted that this technique was only moderately effective assignificant residual intensity variations remain. The conceptof histogram manipulation has been further explored in [2]where degradation due to flicker was modelled as a linear two-parameter grey-level transformation. The required parameterswere estimated under the constraint that the dynamic range ofthe corresponding non-degraded frames does not change withtime.

Recent work dealing with flicker correction has consideredthe incorporation of spatial variability into models like theabove. This is the case in [5] where a semi-global correctionis performed based on a block-partitioning of the degradedframe. Afterwards, each block is assumed to have undergone alinear-intensity transformation independent of all other blocks.A linear minimum mean-square error (LMMSE) estimatoris used to obtain an estimate of the required parameters.A block-based motion detector is used to prevent blockscontaining motion to contribute to the estimation process andthe missing parameters due to the motion are interpolated usinga successive over-relaxation technique. This smooth block-based sparse parameter field is bilinearly interpolated to yielda dense pixel-accurate correction field. Research carried outin [6, 7] has extended the global correction methods of [2, 3]by replacing the additive and multiplicative constants with 2nd

Page 3: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

Figure 3: Hurter-Driffield D(log E) characteristic (dashed) anddensity error (solid) due to exposure shown on the normal originalcharacteristic

order polynomials. In [6] a robust hierarchical frameworkwas proposed to estimate the polynomial functions, goingfrom zero-order to second-order polynomial. The parametersare obtained using M-estimators minimising a robust energycriterion while solutions from lower orders are used as theinitial estimate for higher ones. In [7] an alternative approachto the parameter estimation problem was proposed which triedto eliminate the bias related to the linear regression used in[6]. A method suitable for sequences without camera motionwas described in [8], based on spatio-temporal segmentation,the main idea being the isolation of a common backgroundfor the sequence and the moving objects. The backgroundis estimated through a regularisated average (preserving theedges) of the frames of the sequence, while the moving objectsare motion compensated, averaged and regularised to preservespatial continuities.

Work in [9, 10] approached the problem using histogramequalisation. A degraded frame is first histogram-equalised andthen inverse-histogram-equalised with respect to a referenceframe. Inverse equalisation is carried out in order for thedegraded frame to inherit the histogram profile of the reference.This technique is adapted and enhanced in [11] to deal withocclusions and spatial variations. The brightness distortion islocally estimated in several control points using a Maximum APosteriori technique and a dense correction function is obtainedusing interpolated splines. Our previous work [12] can alsobe viewed as a histogram manipulation approach. It does notprovide a mechanism for spatial adaptation and uses non-linearcompensation motivated by principles of photographic imageregistration. Its main features are summarised in the followingsection.

Figure 4: Intensity difference histograms Ht(50) and Ht(60) andtheir maxima for two consecutive frames of test sequence “caption”.

3 Nonlinear global compensation

3.1 Modelling

The Density versus log-Exposure characteristic D(log E)attributed to Hurter and Driffield [13] (Figure 3) can be used tocharacterise exposure inconsistencies and their correspondingdensity errors. The slope of the linear region is often referredto as gamma and defines the contrast characteristics of thephotosensitive material used for image acquisition. In [12] itwas shown that an observed image instensity I with underlyingdensity D and correponding errors ∆I and ∆D due to flickerare related via :

I → ∆I : exp(−D) → ∆D · exp(−D) (1)

As the Hurter-Driffield characteristic is usually film stockdependant and hence unknown, D and ∆D are difficult toobtain. Nevertheless it is possible to carry out measurementsusing available film samples as follows.

Page 4: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

Figure 5: Measured (solid) and approximated (dashed) intensity errorprofiles as a function of intensity (grey-levels) between frames 33 and34 of the test sequence “caption”.

3.2 Intensity error profile estimation

We aim to determine a suitable greyscale error profile ∆It(IR)between a reference and a degraded frame FR and Ft

respectively, where IR and It = ∆It + IR are grey-levels ofco-sited pixels in the reference and degraded frames and ∆It

is the effect of the flicker on grey-level It. For monochrome8-bits-per-pixel images, It, IR ∈ {0, 1, ..., 255}. This errorprofile allows the correction of Ft according to FR. In thisframework, FR is chosen arbitrarily, as a non-degraded frameis usually not available. To estimate ∆It(IR), pixel differencesbetween all pixels with intensity IR in the reference frame andtheir co-sited pixels in position ~p = (x, y) in the degradedframe are computed and their histogram Ht(IR) is compiled :

∀FR(~p) = IR : Ht(IR) = hist(FR(~p)− Ft(~p)) (2)

An example is shown in Figure 4 for the test sequence”caption” and two sample grey-levels. The correction value isgiven by:

∆It(IR) = arg max{Ht(IR)} (3)

The process is repeated for each intensity level IR to compilea correction profile for the entire greyscale. As the abovecomputation is obtained from real images, the profile ∆It(IR)is unlikely to be smooth and is likely to contain noisymeasurements. A cubic polynomial least-squares fitting,(capable of taking into account greyscale non-linearityassociated with telecine grading [12]) is applied to thecorrection profile (Figure 5):

~a = arg min∑R

(Pt(IR)−∆It(IR))2 (4)

with ~a = {a0, a1, a2, a3} and Pt(IR) =3∑

k=0

ak · IkR (5)

Figure 6: Measured and polynomial approximated (dashed:basicfitting - solid:weighted fitting) intensity error profiles as a functionof intensity between the first two frames of test sequence “caption”.The histogram below shows the normalised confidence values r foreach grey-level.

Finally the correction applied as for a pixel at position ~p is asfollows :

F′

t (~p) = Ft(~p)− Pt(Ft(~p)) (6)

4 Reliability weighting

The first important improvement over the baseline methoddescribed above is motivated by the observation that all grey-levels are incorrectly given equal weight during fitting, withoutregard of their frequency. In addition, measurements quality of∆It(IR) is likely to be different for different intensities. Figure6 illustrates this fact. Ht(50) is spread around 15 and even ifthe maximum is reached for 12, many pixels voted for anothercorrection value. On the other side, Ht(60) favours a moreunanimous verdict. Thus, the reliability of ∆It(IR) dependson the frequency of IR and also on the shape of Ht(IR). Aweighted polynomial least square fitting ([14]) is used and theweighting function chosen to estimate this reliability is:

rt(IR) = max{Ht(IR)} (7)

Indeed, if IR does not occur very frequently in FR, rt(IR) isclose to 0. On the other hand, the higher max{Ht(IR)} is,the more reliable ∆It(IR) can be assumed to be. The newpolynomial P ′

t parameters are obtained as a weighted least-

Page 5: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

Figure 7: Split of the first frame of “boat” using a 3 × 3 grid. Theconsidered pixel and the centre of each blocks are represented by blackand whites dots respectively. The black lines represent the Euclideandistances cb(p).

squares minimisation problem :

~a′ = arg min∑R

rt(IR) · (Pt(IR)−∆It(IR))2 (8)

The correction is still applied using (6). An illustration ofthe effect of reliability weighting is given in Figure 6. Thehistogram shows the confidence values rt(IR), and highlightsthe fact that the intensities error above 140 are not reliable (notoccuring frequently in FR). As a consequence the grey solidcurve corresponding to the weighted fitting is more faithful tothe measured values in the left half of the greyscale comparedto the unweighted dashed curve.

This enhancement has another important advantage which is todeal with compressed sequences such as MPEG material. Thequantisation used in compression may obliterate certain grey-levels. An absent grey-level IR implies that Ht(IR) = 0, thusrt(IR) = 0, which means that ∆It(IR) will not used in thefitting.

5 Spatial adaptation

The algorithm summarised above performs well if the degradedframe is globally affected with no spatial variations of flicker.However as illustrated in 1.2 this is not always the case.This issue can be addressed using a block-based approach.The degraded frame is split into rectangular blocks and thecorrection described in 3 and 4 is applied individually to eachblock. Boundary effects are avoided using weighted bilinearinterpolation as described below.

An example in shown in Figure 7. It is assumed initiallythat flicker is spatially invariant in each block. Let Nh, Nv

and N denote respectively the number of horizontal, vertical

Figure 8: Correction of the frame 20 of the test sequence “boat”.Frame 12 is the reference frame and the correction is appliedindependently on each block of a 3× 3 grid.

and total blocks. For each block an error profile is computedindependently, yielding values for FRb, Ftb, Htb, ∆Itb, P ′

tb

and rtb, b = [1;N ], b being the index of each block. Alocal correction at this stage can introduce blocking artefactsdue to neighbouring blocks being compensated with differentparameters. An example is shown in Figure 8.

This is avoided by applying bilinear interpolation of the errorvalues using the N values P ′

tb(Ft(~p)). This interpolationis based on the inverse of the Euclidean distance cb(~p) =√

(x− xb)2 + (y − yb)2, (Figure 7) :

db(~p) =1

cb(~p) + 1(9)

with (xb, yb) being the coordinates of the centre of the block b.According to (9) db(~p) = 1 for a pixel at the centre ~p of theblock b and decreases rapidly if a pixel is far from the centre.The corrected value F

t (~p) of the pixel in position ~p is then:

F ′t (~p) = Ft(~p)−

N∑b=1

db(~p) ·P ′tb(Ft(~p)) ;

N∑b=1

db(~p) = 1 (10)

This interpolation smooths the transition across blockboundaries as weighting by normalised distances db(~p)performs a weighted average between correction values givenby each block.

5.1 Incorporating reliability weighting

It is possible that compensation for a particular value in ablock is not reliable. The pixel in position ~p (black dot)in Figure 7 has the grey-level 180. This grey range is wellrepresented within its block while it occurs less frequently inthe neighbouring blocks. In this case it is worth giving more

Page 6: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

importance to its own error profile and less to the neighbouringone. This is done by introducing another weight depending ofthe confidence related to ∆Itb(IR). For a pixel in position ~pin block b, the reliability rtb(Ft(~p)) as discussed in Section 4is combined with (10) to yield the overall compensation ruleshown bellow:

F ′t (~p) = Ft(~p)−

N∑b=1

[db(~p) · rtb(Ft(~p))] · P ′tb(Ft(~p)) (11)

withN∑

b=1

[db(~p) · rtb(Ft(~p))] = 1

6 Motion compensated weighting

The framework described previously performs well withstatic or near-static sequences. Motion compensation can beemployed to relax this constraint allowing the computationof error profiles between a motion compensated referenceframe and the current frame. The reference frame is typicallythe first frame of the sequence and the Black and Anandandense motion estimator ([15]) is used. This algorithm is wellequiped to deal with violations of the brightness constancyassumption which is a defining feature of our aplication.Motion compensated prediction F c

R can be viewed as amapping T of the reference frame FR : F c

R = T (FR) withassociated predicion error Etb = F c

Rb − Ftb for each block.We use the framework developed earlier incorporating F c

Rb

into (2). A straightforward approach would involve thecomputation of :

∀F cRb(~p) = IR : Htb(IR) = hist(F c

Rb(~p)− Ftb(~p)) (12)

The disadvangate is that the motion estimation errors are nottaken into account, which means that an incorrectly estimatedpixel has the same influence than a correct one. Etb is takeninto account by compiling Htb(IR) by real-valued incrementsfor each pixel located at ~p using the following relationship :

etb(~p) = 1− |Etb(~p)|max{|Etb(~p)|}

(13)

which implies that if the motion estimation error Etb(~p) has asmall value, etb(~p) ∼ 1 while if Etb(~p) is high, etb(~p) ∼ 0.

7 Experimental results

In order to evaluate the proposed algorithms suitable testmaterial was chosen in the shape of three monochrome shorttest sequences as follows. The first of three sequences,“caption”, is from the Joanneum database [9] and shows astatic caption with only slight unsteadiness but substantialflicker. The second sequence, “boat”, was extracted from

MPEG-1-coded footage of “Our Shrinking World” (1946),(see section 1.2 for a description of this sequence). The finaltest sequence, “lumiere” 2 is also largely static except forobject motion towards the bottom of the image. It depictspeople walking in the Trocadero garden during the UniversalExposition of 1900. Sample frames from these sequences alongwith the block partitioning scheme can be seen in Figure 9.Each of these sequences represent historical footage and aretherefore susceptible to other archive-related artefacts (such asunsteadiness and dirt) in addition to the flicker which is thefocus of this work.

The proposed spatially adaptive algorithm is compared bothwith the global technique from which it is derived and LMMSE[5] which also accommodates for spatial variation. In order topresent an objective comparison the global correction schemewas modified to include the nonlinear weighting and themotion integration (Section 6) as used by the proposed spatialtechnique. In all cases the first frame of the sequence is used asa reference frame for all subsequent frames.

Traditionally flicker reduction algorithms are evaluated byexamining the variation of the mean frame intensity over time.However, this fails to address the spatial variation issue whichis addressed in this paper. Two new block-based visualisationmethods are proposed in order to highlight the differenceswhich occur within the frames of the sequences themselves.The first technique is similar to the traditional methods as itplots the mean frame intensity but on a block-by-block basisrather than for the entire frame. The second uses the standarddeviation of the mean block intensities. An effective correctionshould smooth the mean block intensity and decrease thestandard deviation of the static blocks. It should be noted thatblocks 2, 6 and 10 from the “boat” sequence and blocks 15and 16 from the “lumiere” sequence contain motion so aresubsequently not included in the assessment.

The results for the sequence “caption”, “boat” and “lumiere”are shown in Figures 10, 11 and 12 respectively. In eachcase the topmost was obtained using data from the original(degraded) sequence while the remaining plots depict resultsobtained using the global [12], LMMSE [5] and proposedtechniques. In addition Figure 13 summarises the results bydisplaying the standard deviation of the mean intensities of theblocks used in each sequence.

In the case of the original “caption” sequence the curves followa similar pattern along the block axis indicating that flickeraffects the entire frame at a similar level. As expected theglobal and proposed techniques both perform well with thelatter being only slightly better due to the global nature of thedefect in this sequence. It is also interesting to note that thenonlinear aspect of the flicker causes the LMMSE algorithm toperform less well in this case.

2Film Lumiere 1171 - “La tour Eiffel vue du Trocadero” - ExpositionUniverselle Paris 1900 - Copyright Association freres Lumiere.

Page 7: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

Figure 9: Frames 33-36, 48-51 and 45-48 of the test sequences “caption”, “boat” and “lumiere” respectively.

The results for the second sequence, “boat”, illustrate thespatial variability of the flicker within the frame. The curvesfor the degraded frame in Figure 11 are less uniform along theblock axis than those from “caption”. In this case the twospatially adaptive algorithms have a better overall performancethan the global method. The standard deviation for the blocksis reduced in all cases when comparing the proposed techniqueto the original global method.

Similar remarks can be made for the final sequence, “lumiere”,although this sequence appears more challenging for all theapproaches. Once again the spatially adaptive methods tendto outperform the global technique. The curves for the meanblock intensities produced by the proposed spatial method aresomewhat smoother than those produced by the LMMSE.

8 Conclusion

In this paper, improvements on our earlier non-linear algorithmfor flicker correction were introduced. The first was regardingthe estimation of a more reliable intensity error profile. Aweighted estimation solution was developed, taking into grey-level frequency of occurence. Additionally, spatial variabilityof flicker was addressed by applying the baseline algorithm in ablock-based fashion followed by bilinear interpolation to avoidblocking artefacts. Finally, motion compensated predictionwas used to prevent contamination of measurements due tomotion. This has also incorporated the use of the predictionerror as a reliability measure. Our results have shown that the

algorithm is very effective towards flicker compensation for avariety of test sequences and compares favourably to state-of-art techniques that feature in the literature.

Acknowledgements

The authors would like to thank Dr. Bernard Besserer of theUniversity of La Rochelle (France), for useful discussions andfor making available some of the test material. The sequence“lumiere” was available courtesy of the Centre National dela Cinematographie (CNC), Paris ([16]) while “caption” wasprovided by Joanneum Institute, Graz ([17]).

References

[1] J.H. Chenot, J.O. Drewery, and D. Lyon. Restoration ofarchived television programmes for digital broadcasting.In Proc. IBC, pages 26–31, Amsterdam, The Netherlands,1998.

[2] Y. Wu and D. Suter. Historical film processing. In Proc.SPIE, volume 2564, pages 289–300, San Diego, USA,1995.

[3] E. Decenciere Ferrandiere. Restauration automatique defilms anciens. PhD thesis, ENSMP, Paris, 1997.

Page 8: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

[4] P. Richardson and D. Suter. Restoration of historic filmfor digital compression : A case study. In Proc. ICIP,volume 2, pages 49–52, Washington DC, USA, 1995.

[5] P.M.B. Van Roosmalen, R.L. Lagendijk, and J. Biemond.Correction of intensity flicker in old film sequences. IEEETrans. Circ. Sys. Vid. Tech., 9(7):1013–1019, 1999.

[6] T. Ohuchi, T. Seto, T. Komatsu, and T. Saito. A robustmethod of image flicker correction for heavily-corruptedold film sequences. In Proc. ICIP, volume 2, pages 672–675, Vancouver BC, Canada, 2000.

[7] A. C. Kokaram, R. Dahyot, F. Pitie, and H. Denman.Simultaneous luminance and position stabilization forfilm and video. In Proc. SPIE, Visual Communicationsand Image Processing, San Jose, USA, 2003.

[8] J. Jung, M. Antonini, and M. Barlaud. Automaticrestoration of old movies with an object orientedapproach. In Proc. RFIA (Conference on PatternRecognition and Artificial Intelligence), Paris, France,2000.

[9] P. Schallauer, A. Pinz, and W. Haas. Automaticrestoration algorithms for 35mm film. Videre, 1(3):60–85, 1999.

[10] V. Naranjo and A. Albiol. Flicker reduction in old films.In Proc. ICIP, volume 2, pages 657–659, Vancouver BC,Canada, 2000.

[11] F. Pitie, R. Dahyot, F. Kelly, and Kokaram.A. C. A newrobust technique for stabilizing brightness fluctuations inimage sequences. In ECCV Workshop SMVP (StatisticalMethods in Video Processing), pages 153–164, Prague,Czech Republic, 2004.

[12] T. Vlachos. Flicker correction for archived film sequencesusing a non-linear model. IEEE Trans. on Circ. and Sys.Vid. Tech., 14(4):508–516, 2004.

[13] C.E.K. Mess. The Theory of the Photographic Process.New York : McMillan, 1954.

[14] Peter J. Huber. Robust Statistics. Wiley, 1981.

[15] Michael J. Black and P. Anandan. The robust estimationof multiple motions: Parametric and piecewise-smoothflow fields. Computer Vision and Image Understanding,63(1):75–104, 1996.

[16] CNC website. http://www.cnc.fr.

[17] Joanneum Institute website. http://www.joanneum.at.

Figure 10: Mean intensity of static blocks (indexed as shown inFigure 9) for frames 0-49 of test sequence “caption” and of global[12], LMMSE [5] and proposed correction results respectively.

Page 9: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

Figure 11: Mean intensity of static blocks for the frames 12-61 ofthe test sequence “boat” and of the global, LMMSE and proposedcorrection results respectively.

Figure 12: Mean intensity of static blocks for the frames 27-77 ofthe test sequence “lumiere” and of the global, LMMSE and proposedcorrection results respectively.

Page 10: SPATIALLY ADAPTIVE FLICKER COMPENSATION … ADAPTIVE FLICKER COMPENSATION FOR ARCHIVED FILM SEQUENCES USING A NONLINEAR MODEL Guillaume Forbin, Theodore Vlachos, Simon Tredwell, Centre

Figure 13: Standard deviation of mean intensity of static blocks forframes 0-49, 12-61 and 27-77 of test sequences “caption”, “boat”and “lumiere” respectively.