Key Frame Extraction On MPEG by using Threshold Algorithm CHAPTER 1 INTRODUCTION 1.1 LITERATURE REVIEW Recent years have witnessed an enormous increase in video data on the internet. This rapid increase demands efficient techniques for management and storage of video data. Video summarization is one of the commonly used mechanisms to build an efficient video archiving system. The video summarization methods generate summaries of the videos which are the sequences of stationary or moving images (Money and Agius, 2008). Key frame extraction is a widely used method for video summarization. The key frames are the characteristic frames of the video which render limited, but meaningful information about the contents of the video (Li et al., 2001). The researchers have attempted to exploit various features for the extraction of key frames in videos. These features have been utilized in a variety of different ways. Some of the low level features which are commonly used include color histogram, frame correlation, motion information and edge histogram etc. (Jiang et al., 2009). Zhang et al. (1997) used the color histogram difference between the current frame and the last extracted key frame to draw out key frames from the video. Gunsel and Tekalp (1998) compared the histogram of Department. of ECE, MRITS 1
93
Embed
Key Frame Extraction On MPEG by using Threshold Algorithm
Key Frame Extraction On MPEG by using Threshold Algorithm
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Key Frame Extraction On MPEG by using Threshold Algorithm
CHAPTER 1
INTRODUCTION
1.1 LITERATURE REVIEW
Recent years have witnessed an enormous increase in video data on the
internet. This rapid increase demands efficient techniques for management and
storage of video data. Video summarization is one of the commonly used mechanisms
to build an efficient video archiving system. The video summarization methods
generate summaries of the videos which are the sequences of stationary or moving
images (Money and Agius, 2008). Key frame extraction is a widely used method for
video summarization. The key frames are the characteristic frames of the video which
render limited, but meaningful information about the contents of the video (Li et al.,
2001).
The researchers have attempted to exploit various features for the extraction
of key frames in videos. These features have been utilized in a variety of different
ways. Some of the low level features which are commonly used include color
histogram, frame correlation, motion information and edge histogram etc. (Jiang et al.,
2009). Zhang et al. (1997) used the color histogram difference between the current
frame and the last extracted key frame to draw out key frames from the video. Gunsel
and Tekalp (1998) compared the histogram of current frame with the average color
histograms of the previous frames to compute the discontinuity value.
A thorough survey of existing techniques reveals that the researchers have
used many different visual features for the problem of key frame extraction. In our
project we dealt with the frame difference measures such as color histogram, frame
correlation and edge orientation histogram for the extraction of key frame.
1.2 OVERVIEW OF PROJECT
Efficient key frame extraction enables efficient cataloguing and retrieval with
large video collections. Video is rich in content and it results in a tremendous amount
of data to process. This can be made easier by only processing some frames, such as
the key frames of video. In general, a key frame extraction technique must be fully
automated in nature and must use the contents of the video to generate summary.
Department. of ECE, MRITS 1
Key Frame Extraction On MPEG by using Threshold Algorithm
Theoretically, key frames must be extracted using high level features such as
objects, actions and events. However, the key frame extraction based on high level
features is mostly specific to certain applications and usually low level features have
been employed. Some of the examples of low features that are commonly used are
colour histogram, correlation, moments, edges and motion features. These low level
features can then be employed to derive high level features to generate domain
specific applications.
A common methodology is to compare consecutive frames based on some
low level Frame Difference Measures (FDMs) and extract a key frame if this
difference satisfies a certain threshold value. The low level features used in our
project are
(1) Colour histogram
(2) Frame correlation
(3) Edge orientation histogram
The basic block diagram of Elicitation of key frames in sports video based on
multiple frame difference features is shown in Fig.1.1. It consists of Extraction of
frames, color histogram, correlation, and edge orientation histogram and threshold
logic modules. Extraction of frames module extract all the frames from the given
input video and the keyframes are identified based on color histogram, correlation and
edge orientation histogram methods by making use of threshold logic. In our work,
the results from these three methods are compared for sample video (Foot Ball),
Cricket video, Hockey video and Foot Ball Video.
Colour Histogram for the frames is calculated in HSV color space. HSV
stands for Hue, Saturation and Value. HSV colour model is based on how colors
appear to a human observer. From the colour histograms of these three channels
between the frames, colour histogram difference measure is calculated. This measure
lies between -64 to 0.
Frame correlation is done by using Pearson’s Distance. Pearson’s Distance is
defined as one minus Pearson’s correlation coefficient. Pearson's correlation
coefficient between two variables is defined as the covariance of the two variables
Key Frame Extraction On MPEG by using Threshold Algorithm
computations. On the other hand, the opposite of the gradient approximation that it
produces is relatively crude, in particular for high frequency variations in the image.
Mathematically, the operator uses two 3×3 kernels which are convolved with the
original image to calculate approximations of the derivatives - one for horizontal
changes, and one for vertical. If we define A as the source image, and Gx and Gy are
two images which at each point contain the horizontal and vertical derivative
approximations, the computations are as follows:
Where * here denotes the 2-dimensional convolution operation.
The x-coordinate is here defined as increasing in the "right"-direction, and the
y-coordinate is defined as increasing in the "down"-direction. At each point in the
image, the resulting Gradient approximations can be combined to give the gradient
magnitude, using
Using this information, we can also calculate the opposite of the gradient's direction:
Fig 5.2(b) shows the application of sobel operator for the original image shown in
Fig.5.2(a)
Department. of ECE, MRITS 29
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig. 5.2(a): Colour picture of a steam engine
Fig. 5.2(b): sobel operator applied to that image
The Fig 5.2(b) shows the application of sobel operator for the original image
shown in Fig.5.2 (a).
Department. of ECE, MRITS 30
Key Frame Extraction On MPEG by using Threshold Algorithm
5.4 FORMULATION
The purpose of edge detection in general is to significantly reduce the amount
of data in an image, while preserving the structural properties to be used for further
image processing. The edges are good under illumination changes. The edges are first
computed using horizontal and vertical Sobel operators which are then used to find
gradient and angle of edges. The angles are then used to build a histogram of edge
orientation. For simplicity, we defined only 72 bins for the angles. As in the case of
histograms, we compare histograms of corresponding sections of the two frames. The
edge Histogram difference “ED” between two frames i and j is calculated by taking
the average of the difference measure between each section. The formula for
calculating ED is
Department. of ECE, MRITS 31
Key Frame Extraction On MPEG by using Threshold Algorithm
CHAPTER 6
RESULTS6.1 FLOW CHART FOR THE EXTRACTION OF KEY FRAMES
Department. of ECE, MRITS 32
Video
Frames from input video
For n=0 n=n+10<n<total number of frames from the video
First frame?
Key frame data basek=0k=k+11<k<n
Current frame
Correlation difference (CD)
Colour histogram difference (HD)
Edge orientation histogram difference (ED)
Threshold
Discard frame
Key frame= current frame
Stop
False
True
Fig 6.1: Flow chart for the extraction of key frames
False
True
Last frame? FalseTrue
Start
Key Frame Extraction On MPEG by using Threshold Algorithm
6.2 ALGORITHM FOR EXTRACTING KEY FRAMES BASED ON
CORRELATION
The key frame extraction method is composed of the following steps
Step1: All the frames are extracted from the input sports video.
Step2: Consider first frame as a key frame.
Step3: Select the next subsequent frame from the extracted frames and divide frame
into a total of ‘Ts’ sections, each of size mxm (8x8).
Step4: Histogram Creation
Step4.1: Correlation Histogram Creation: The correlation values of each
section are then averaged. The correlation is measured for three color channel values
red, green and blue.
Step4.2: The correlation difference CDp,q,s,c of a color channel ‘c’ between
two corresponding sections ‘s’ of frame p and q is defined as:
Where s =1…T ; c =red, green, blue ;f= mean value of c channel of the frame.
Step4.3: The correlations of all sections of frame i and j are averaged to
Obtain the overall correlation CDi,j,c for a color channel.
Step4.4: Then, the overall correlation difference measure CDi,j between
frames i and j is obtained by averaging the value of each color channel.
Step4.5: CDi,j is compared with the threshold value to detect key frame. The
frames with higher CDi,j as compared to threshold are treated as key frame.
Department. of ECE, MRITS 33
Key Frame Extraction On MPEG by using Threshold Algorithm
Step5: To detect key frames based on correlation difference measure in entire video
repeat step3 & step4.
6.3 FLOW CHART FOR CORRELATION
Department. of ECE, MRITS 34
Current Frame
Key frame from the database
Division of each frame into Ts sections of size (m*m)
Correlation difference of two corresponding sections of current frame and previous frame (C1, C2 ...Cs) are calculated
Mean of correlation difference values
CD
Fig 6.2: Flow chart for correlation difference
Key Frame Extraction On MPEG by using Threshold Algorithm
6.4 ALGORITHM FOR EXTRACTING KEY FRAMES BASED ON
COLOUR DIFFERENCE MEASURE
The key frame extraction method is composed of the following steps
Step1: All the frames are extracted from the input sports video.
Step2: Consider first frame as a key frame.
Step3: select the next subsequent frame from the extracted frames and convert RGB
to HSV colour space then divide frame into a total of ‘Ts’ sections, each of size
mxm(8x8).
Step4: Histogram Creation
Step4.1: Colour Histogram Creation: A three dimensional colour histogram is
built by subdividing the HSV colour space into 8:2:4 bins.
Step4.2: The histogram difference HDi,j,s between two corresponding sections
‘s’ of histogram His of frame i and histogram Hjs of frame j is calculated by using the
formula
Step4.3: The histogram difference “HD” between two frames i and j is then
Calculated by taking the average of the difference measure between each section by
the formula
Step4.4: HDi,j is compared with the threshold value to detect key frame. The
frames with lower HDi,j as compared to threshold are treated as key frame.
Step5: To detect key frames based on colour difference measure in entire video
repeat step3 & step4.
Department. of ECE, MRITS 35
Key Frame Extraction On MPEG by using Threshold Algorithm
6.5 FLOW CHART FOR COLOUR HISTOGRAM
Department. of ECE, MRITS 36
Current Frame
Key frame from the database
Conversion of RGB to HSV
Colour histogram difference of two corresponding sections of current frame and previous frame (ch1, ch2 ...chs)
Mean of colour difference
values
HD
Division of each frame into Ts sections of size (m*m)
Fig 6.3: Flow chart of colour histogram difference
Key Frame Extraction On MPEG by using Threshold Algorithm
6.6 ALGORITHM FOR EXTRACTING KEY FRAMES BASED ON EDGE DIFFERENCE MEASURE
The key frame extraction method is composed of the following steps
Step1: All the frames are extracted from the input sports video.
Step2: Consider first frame as a key frame.
Step3: select the next subsequent frame from the extracted frames and convert RGB
to Gray image then divide frame into a total of ‘Ts’ sections, each of size mxm(8x8).
Step4: Histogram Creation Step4.1: Edge Histogram Creation: The edges are first computed using
horizontal and vertical Sobel operators which are then used to find gradient magnitude
and angle of edges. Gradient’s magnitude is given by
Gradient’s direction is given by
Step4.2: the angles are computed for only those pixels where value of gradient
is above a certain threshold (>3). The angles are then used to build a histogram of
edge orientation. We defined only 82 bins for the angles.
Step4.3: we compare histograms of corresponding sections of the two frames.
The edge histogram difference “ED” between two frames i and j is calculated by
taking the average of the difference measure between each section.
Step4.4: EDi,j is compared with the threshold value to detect key frame. The
frames with higher EDi,j as compared to threshold is treated as key frame.
Step5: To detect key frames based on edge difference measure in entire video repeat
step3 & step4.
Department. of ECE, MRITS 37
Key Frame Extraction On MPEG by using Threshold Algorithm
6.7 FLOW CHART FOR EDGE ORIENTATION HISTOGRAM
Department. of ECE, MRITS 38
Current Frame
Key frame from the database
RGB to gray conversion
Correlation difference of two corresponding sections of current frame and previous frame (e1, e2 ...es)
Mean of edge orientation difference values
Division of each frame into Ts sections of size (m*m)
Evaluate gradients magnitude
for all sections
If gradient magnitude <3
Evaluate gradient direction(ø=arc tan (Gy/Gx))
Eliminate edge False
True
Fig 6.4: Flow chart for edge orientation histogram difference
ED
Calculate gradients ( Gx & Gy )
Key Frame Extraction On MPEG by using Threshold Algorithm
6.8 COLOUR HISTOGRAM OUTPUT
Fig 6.5 : Reading the frames from the input video
Figure 6.5 indicates the reading of frames from the video as well as the comparisons of frame with the previous frame to find out the key frames.
Department. of ECE, MRITS 39
Key Frame Extraction On MPEG by using Threshold Algorithm
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
Fig 6.6: Frames extracted from the (sample) football video
Department. of ECE, MRITS 40
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.7: colour histogram difference values for the sample (football) video
Figure 6.7 indicates the colour histogram difference values of the current frame and previous frame. Total 19 colour histogram difference values are generated from 20 frames in the football video. The range of colour histogram difference values is -64 to 0.The absolute value of the colour histogram differences are compared with the set of threshold value to extract key frames based on colour histogram. In this the frames with colour histogram difference value greater than the threshold are discarded.
Department. of ECE, MRITS 41
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.8: output graph of colour histogram
The above Fig 6.8 shows the graph between frames and colour difference value.
Department. of ECE, MRITS 42
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.9 (a): key frames based on colour histogram for the sample (football) video with the threshold value as 35.
The above figure shows the number of key frames extracted based on colour histogram technique with the threshold value as 35.Total 8 frames are obtained with this threshold value.
Department. of ECE, MRITS 43
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.9 (b): set of key frames based on colour histogram for the sample (football) video with the threshold value as 35.
With 35 as the threshold value we obtained 8 frames as key frames based on colour histogram.
Department. of ECE, MRITS 44
Key Frame Extraction On MPEG by using Threshold Algorithm
Department. of ECE, MRITS 45
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.10 (a): key frames based on colour histogram for the sample (football) video with the threshold value as 45.
The above figure shows the number of key frames extracted based on colour histogram technique with the threshold value as 45. Total 12 frames are obtained with this threshold value.
Department. of ECE, MRITS 46
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.10(b): set of key frames based on colour histogram for the sample(football) video with the threshold value as 45.
With 45 as the threshold value we obtained 12 frames as key frames based on colour histogram.
Department. of ECE, MRITS 47
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.11 (a): key frames based on colour histogram for the sample (football) video with the threshold value as 55.
The above figure shows the number of key frames extracted based on colour histogram technique with the threshold value as 55. Total 13 frames are obtained with this threshold value.
Department. of ECE, MRITS 48
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.11(b): set of key frames based on colour histogram for the sample (football) video with the threshold value as 55.
With 55 as the threshold value we obtained 13 frames as key frames based on colour histogram.
Department. of ECE, MRITS 49
Key Frame Extraction On MPEG by using Threshold Algorithm
6.9 CORRELATION OUTPUT
Fig 6.12: Correlation difference values for the sample (football) video
Fig 6.12 indicates the correlation difference values of the current frame and previous frame. Total 19 correlation difference values are generated from 20 frames in the football video. The range of correlation difference values is 0 to 1. The absolute value of the correlation differences are compared with the set of threshold value to extract key frames based on correlation. In this the frames with correlation difference value lesser than the threshold are discarded.
Department. of ECE, MRITS 50
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.13: output graph of correlation
The above figure shows the graph between frames and correlation difference value.
Department. of ECE, MRITS 51
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.14 (a): key frames based on correlation for the sample (football) video with the threshold value as 0.4.
The above figure shows the number of key frames extracted based on correlation technique with the threshold value as 0.4. Total 4 frames are obtained with this threshold value.
Department. of ECE, MRITS 52
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.14(b): set of key frames based on correlation for the sample (football) video with the threshold value as 0.4.
With 0.4 as the threshold value we obtained 4 frames as key frames based on correlation.
Department. of ECE, MRITS 53
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.15 (a): key frames based on correlation for the sample (football) video with the threshold value as 0.6.
The above figure shows the number of key frames extracted based on correlation technique with the threshold value as 0.6. Total 2 frames are obtained with this threshold value.
Department. of ECE, MRITS 54
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.15(b) :set of key frames based on correlation for the sample (football) video with the threshold value as 0.6.
With 0.6 as the threshold value we obtained 2 frames as key frames based on correlation.
Department. of ECE, MRITS 55
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.16 (a): key frames based on correlation for the sample (football) video with the threshold value as 0.8.
The above figure shows the number of key frames extracted based on correlation technique with the threshold value as 0.8. Only one frame is obtained as a key frame.
Department. of ECE, MRITS 56
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.16(b): set of key frames based on correlation for the sample (football) video with the threshold value as 0.8.
With 0.8 as the threshold value we obtained 1 frame as key frames based on correlation.
Department. of ECE, MRITS 57
Key Frame Extraction On MPEG by using Threshold Algorithm
6.10 EDGE ORIENTATION HISTOGRAM OUTPUT
Fig 6.17: edge orientation histogram difference values for the sample (football) video
Figure 6.17 indicates the edge orientation histogram difference values of the current frame and previous frame. Total 19 edge orientation histogram difference values are generated from 20 frames in the football video. The range of edge orientation histogram difference values is 0 to 82.the absolute value of the edge orientation histogram differences are compared with the set of threshold value to extract key frames based on edge orientation histogram. In this the frames with edge orientation difference value lesser than the threshold are discarded.
Department. of ECE, MRITS 58
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.18: output graph of edge orientation histogram
The above figure shows the graph between frames and edge orientation difference value.
Department. of ECE, MRITS 59
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.19 (a): key frames based on edge orientation histogram for the sample (football) video with the threshold value as 40.
The above figure shows the number of key frames extracted based on edge orientation histogram technique with the threshold value as 40. Total 11 frames are obtained with this threshold value.
Department. of ECE, MRITS 60
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.19(b): set of key frames based on edge orientation histogram for the sample (football) video with the threshold value as 40.
With 40 as the threshold value we obtained 11 frames as key frames based on edge orientation histogram.
Department. of ECE, MRITS 61
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.20 (a): key frames based on edge orientation histogram for the sample (football) video with the threshold value as 50.
The above figure shows the number of key frames extracted based on edge orientation histogram technique with the threshold value as 50. Total 6 frames are obtained with this threshold value.
Department. of ECE, MRITS 62
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.20(b): set of key frames based on edge orientation histogram for the sample (football) video with the threshold value as 50
With 50 as the threshold value we obtained 6 frames as key frames based on edge orientation histogram.
Department. of ECE, MRITS 63
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.21 (a): key frames based on edge orientation histogram for the sample (football) video with the threshold value as 60.
The above figure shows the number of key frames extracted based on edge orientation histogram technique with the threshold value as 60. Total 4 frames are obtained with this threshold value.
Department. of ECE, MRITS 64
Key Frame Extraction On MPEG by using Threshold Algorithm
Fig 6.21(b): set of key frames based on edge orientation histogram for the sample (football) video with the threshold value as 60.
With 60 as the threshold value we obtained 4 frames as key frames based on edge orientation histogram.
Department. of ECE, MRITS 65
Key Frame Extraction On MPEG by using Threshold Algorithm
6.11 OUTPUT
For different sport videos the number of key frames for different threshold value based on colour, correlation and edge orientation techniques are shown below.