Automatic Disease Detection In Citrus Trees Using Machine Vision

Automatic Disease Detection In Automatic Disease Detection In Citrus Trees Using Machine VisionCitrus Trees Using Machine Vision

Rajesh PydipatiRajesh Pydipati

Research AssistantResearch Assistant

Agricultural Robotics & Mechatronics Group (ARMg)Agricultural Robotics & Mechatronics Group (ARMg)

Agricultural & Biological EngineeringAgricultural & Biological Engineering

IntroductionIntroduction

• Citrus industry is an important constituent Citrus industry is an important constituent of Florida’s overall agricultural economy of Florida’s overall agricultural economy

• Florida is the world’s leading producing Florida is the world’s leading producing region for grapefruit and second only to region for grapefruit and second only to Brazil in orange production Brazil in orange production

• The state produces over 80 percent of the The state produces over 80 percent of the United States’ supply of citrus United States’ supply of citrus

Research JustificationResearch Justification

• Citrus diseases cause economic loss in citrus Citrus diseases cause economic loss in citrus production due to long term tree damage and production due to long term tree damage and due to fruit defects that reduce crop size, quality due to fruit defects that reduce crop size, quality and marketability. and marketability.

• Early detection systems that might detect and Early detection systems that might detect and possibly treat citrus for observed diseases or possibly treat citrus for observed diseases or nutrient deficiency could significantly reduce nutrient deficiency could significantly reduce annual lossesannual losses..

ObjectivesObjectives

• Collect image data set of various common Collect image data set of various common citrus diseases. citrus diseases.

• Evaluate the Color Co-occurrence Method, for Evaluate the Color Co-occurrence Method, for disease detection in citrus trees.disease detection in citrus trees.

• Develop various strategies and algorithms for Develop various strategies and algorithms for classification of the citrus leaves based on the classification of the citrus leaves based on the features obtained from the color co-occurrence features obtained from the color co-occurrence method.method.

• Compare the classification accuracies from the Compare the classification accuracies from the algorithms.algorithms.

Vision based classificationVision based classification

Sample Collection and Image Acquisition

• Leaf sample sets were collected from a typical Florida grape fruit grove for three common citrus diseases and from normal leaves

• Specimens were separated according to classification in plastic ziploc bags and stored in a environmental chamber maintained at 10 degrees centigrade

• Forty digital RGB format images were collected for each classification and stored to disk in uncompressed JPEG format. Alternating image selection was used to build the test and training data sets

Leaf sample imagesLeaf sample images

Greasy spot diseased leaf Melanose diseased leaf

Leaf sample imagesLeaf sample images

Scab diseased leaf Normal leaf

Ambient vs Laboratory ConditionsAmbient vs Laboratory Conditions

• Initial tests were conducted in a laboratory Initial tests were conducted in a laboratory to minimize uncertainty created by to minimize uncertainty created by ambient lighting variation. ambient lighting variation.

• An effort was made to select an artificial An effort was made to select an artificial light source which would closely represent light source which would closely represent ambient light.ambient light.

• Leaf samples were analyzed individually to Leaf samples were analyzed individually to identify variations between leaf fronts and identify variations between leaf fronts and backs.backs.

Spectrum Comparison with Spectrum Comparison with NaturaLight FilterNaturaLight Filter

Image Acquisition SpecificationsImage Acquisition Specifications

• Four 16W Cool White Fluorescent bulbs (4500K) Four 16W Cool White Fluorescent bulbs (4500K) with NaturaLight filters and reflectors.with NaturaLight filters and reflectors.

• JAI MV90, 3 CCD Color Camera with 28-90 mm JAI MV90, 3 CCD Color Camera with 28-90 mm Zoom lens.Zoom lens.

• Coreco PC-RGB 24 bit color frame grabber with Coreco PC-RGB 24 bit color frame grabber with 480 by 640 pixels.480 by 640 pixels.

• MV Tools Image capture softwareMV Tools Image capture software• Matlab Image Processing ToolboxMatlab Image Processing Toolbox• SAS Statistical Analysis PackageSAS Statistical Analysis Package

Image Acquisition SystemImage Acquisition System

Camera CalibrationCamera Calibration• The camera was calibrated under the artificial light The camera was calibrated under the artificial light

source using a calibration grey-card.source using a calibration grey-card.

• An RGB digital image was taken of the grey-card and An RGB digital image was taken of the grey-card and each color channel was evaluated using histograms, each color channel was evaluated using histograms, mean and standard deviation statistics.mean and standard deviation statistics.

• Red and green channel gains were adjusted until the Red and green channel gains were adjusted until the grey-card images had similar means in R, G, and B grey-card images had similar means in R, G, and B equal to approximately 128, which is mid-range for a equal to approximately 128, which is mid-range for a scale from 0 to 255. Standard deviation of calibrated scale from 0 to 255. Standard deviation of calibrated pixel values were approximately equal to 3.0.pixel values were approximately equal to 3.0.

Image acquisition and classification Image acquisition and classification flow chartflow chart

Color cooccurence methodColor cooccurence method• Color Co-occurrence Method (CCM) uses HSI pixel

maps to generate three unique Spatial Gray-level Dependence Matrices (SGDM)

• Each sub-image was converted from RGB (red, green, blue) to HSI (hue, saturation, intensity) color format

• The SGDM is a measure of the probability that a given pixel at one particular gray-level will occur at a distinct distance and orientation angle from another pixel, given that pixel has a second particular gray-level

CCM TEXTURE STATISTICS

• CCM texture statistics were generated from the SGDM of each HSI color feature.

• Each of the three matrices is evaluated by thirteen texture statistic measures resulting in 39 texture features per image.

• CCM Texture statistics were used to build four data models. The data models used different combinations of the HSI color co-occurrence texture features. STEPDISC was used to reduce data models through a stepwise variable elimination procedure

Intensity Texture Features• I1 - Uniformity• I2 - Mean• I3 - Variance• I4 - Correlation• I5 - Product Moment• I6 - Inverse Difference• I7 - Entropy• I8 - Sum Entropy

• I9 - Difference Entropy• I10 - Information Correlation Measures #1• I11 - Information Correlation Measures #2• I12 - Contrast• I13 - Modus

Classification Models

All VariablesHSIFront4F

I2,I5,I4,S2,H11,S4,H4HSIFront3F

I2,I5,I4,I12,I13IFront2F

S2,H10,H6,H2,H8,S9,S4HSFront1F

All VariablesHSIBack4B

I2,S5,I10,H11,S1,I13,S13HSIBack3B

I2,I13,I8,I7,I6,I3IBack2B

S5,S2,H7,S6,S9,H8,H11,S12,H1,H12HSBack1B

STEPDISC Variable SetsColorLeafModel

All VariablesHSIFront4F

I2,I5,I4,S2,H11,S4,H4HSIFront3F

I2,I5,I4,I12,I13IFront2F

S2,H10,H6,H2,H8,S9,S4HSFront1F

All VariablesHSIBack4B

I2,S5,I10,H11,S1,I13,S13HSIBack3B

I2,I13,I8,I7,I6,I3IBack2B

S5,S2,H7,S6,S9,H8,H11,S12,H1,H12HSBack1B

STEPDISC Variable SetsColorLeafModel

Classifier based on Mahalanobis Classifier based on Mahalanobis distancedistance

• The Mahalanobis distance is a very useful The Mahalanobis distance is a very useful way of determining the similarity of a set of way of determining the similarity of a set of values from an unknown sample to a set of values from an unknown sample to a set of values measured from a collection of values measured from a collection of known samplesknown samples

• Mahalanobis distance method is very Mahalanobis distance method is very sensitive to inter-variable changes in the sensitive to inter-variable changes in the training data training data

Mahalanobis distance contd..Mahalanobis distance contd..

• Mahalanobis distance is measured in Mahalanobis distance is measured in terms of standard deviations from the terms of standard deviations from the mean of the training samples mean of the training samples

• The reported matching values give a The reported matching values give a statistical measure of how well the statistical measure of how well the spectrum of the unknown sample matches spectrum of the unknown sample matches (or does not match) the original training (or does not match) the original training spectraspectra

Formula for calculating the squared Mahalanobis Formula for calculating the squared Mahalanobis

distance metricdistance metric

µ)(µ)( 1T2 xxr

‘x’ is the N-dimensional test feature vector (N is the number of features )

‘µ’ is the N-dimensional mean vector for a particular class of leaves

‘∑’ is the N x N dimensional co-variance matrix for a particular class of leaves.

Minimum distance principleMinimum distance principle

• The squared Mahalanobis distance was The squared Mahalanobis distance was calculated from a test image to various classes calculated from a test image to various classes of leavesof leaves

• The minimum distance was used as the The minimum distance was used as the criterion to make classification decisionscriterion to make classification decisions

Neural networksNeural networks

“ “ A neural network is a system composed of many A neural network is a system composed of many simple processing elements operating in parallel simple processing elements operating in parallel whose function is determined by network whose function is determined by network structure, connection strengths, and the structure, connection strengths, and the processing performed at computing elements or processing performed at computing elements or nodes ”. nodes ”.

(According to the DARPA Neural Network Study (1988, AFCEA (According to the DARPA Neural Network Study (1988, AFCEA International Press, p. 60)International Press, p. 60)

Contd..Contd..

““A neural network is a massively parallel distributed A neural network is a massively parallel distributed processor that has a natural propensity for storing processor that has a natural propensity for storing experiential knowledge and making it available for use. It experiential knowledge and making it available for use. It resembles the brain in two respects: resembles the brain in two respects:

1. Knowledge is acquired by the network through a 1. Knowledge is acquired by the network through a learning process. learning process. 2. Inter-neuron connection strengths known as synaptic 2. Inter-neuron connection strengths known as synaptic weights are used to store the knowledge.weights are used to store the knowledge.

[According to Haykin, S. (1994), Neural Networks: A Comprehensive Foundation, NY: [According to Haykin, S. (1994), Neural Networks: A Comprehensive Foundation, NY: Macmillan, p. 2]Macmillan, p. 2]

A Basic NeuronA Basic Neuron

Multilayer Feed forward Neural Multilayer Feed forward Neural NetworkNetwork

Back propagationBack propagation

• In the MFNN shown earlier the input layer In the MFNN shown earlier the input layer of the BP network is generally fully of the BP network is generally fully connected to all nodes in the following connected to all nodes in the following hidden layerhidden layer

• Input is generally normalized to values Input is generally normalized to values between -1 and 1between -1 and 1

• Each node in the hidden layer acts as a Each node in the hidden layer acts as a summing node for all inputs as well as an summing node for all inputs as well as an activation functionactivation function

MFNN with Back propagationMFNN with Back propagation

• The hidden layer neuron first sums all the The hidden layer neuron first sums all the connection inputs and then sends this connection inputs and then sends this result to the activation function for output result to the activation function for output generation.generation.

• The outputs are propagated through all the The outputs are propagated through all the layers until final output is obtainedlayers until final output is obtained

Mathematical equationsMathematical equations

The governing equations are given below:The governing equations are given below:

Where x1,x2… are the input signals, Where x1,x2… are the input signals, w1,w2…. the synaptic weights, w1,w2…. the synaptic weights, uu is the activation potential is the activation potential of the neuron, of the neuron, is the threshold, is the threshold, yy is the output signal of the neuron, is the output signal of the neuron, and and f f (.) is the activation function(.) is the activation function..

N

iii xwu

1

)( ufy

Back propagationBack propagation

• The Back propagation algorithm is the The Back propagation algorithm is the most important algorithm for the most important algorithm for the supervised training of multilayer feed-supervised training of multilayer feed-forward ANNsforward ANNs

• The BP algorithm was originally developed The BP algorithm was originally developed using the gradient descent algorithm to using the gradient descent algorithm to train multi layered neural networks for train multi layered neural networks for performing desired tasks performing desired tasks

Back propagation algorithmBack propagation algorithm

• BP training process begins by selecting a BP training process begins by selecting a set of training input vectors along with set of training input vectors along with corresponding output vectors.corresponding output vectors.

• The outputs of the intermediate stages are The outputs of the intermediate stages are forward propagated until the output layer forward propagated until the output layer nodes are activated.nodes are activated.

• Actual outputs are compared with target Actual outputs are compared with target outputs using an error criterion.outputs using an error criterion.

Back propagationBack propagation

• The connection weights are updated using The connection weights are updated using the gradient descent approach by back the gradient descent approach by back propagating change in the network propagating change in the network weights from the output layer to the input weights from the output layer to the input layer.layer.

• The net changes to the network will be The net changes to the network will be accomplished at the end of one training accomplished at the end of one training cycle.cycle.

BP network architecture used in the BP network architecture used in the researchresearch

Network Architecture:2 hidden layers with 10 processing elements eachOutput layer consisting of 4 output neuronsAn input layer‘Tansig’ activation function used at all layers

Radial basis function networksRadial basis function networks

• A radial basis function network is a neural A radial basis function network is a neural network approached by viewing the design as a network approached by viewing the design as a curve-fitting (approximation) problem in a high curve-fitting (approximation) problem in a high dimensional space dimensional space

• Learning is equivalent to finding a Learning is equivalent to finding a multidimensional function that provides a best fit multidimensional function that provides a best fit to the training data to the training data

An RBF networkAn RBF network

RBF contd…RBF contd…

• The RBF front layer is the input layer The RBF front layer is the input layer where the input vector is applied to the where the input vector is applied to the networknetwork

• The hidden layer consist of radial basis The hidden layer consist of radial basis function neurons, which perform a fixed function neurons, which perform a fixed non-linear transformation mapping the non-linear transformation mapping the input space into a new space input space into a new space

• The output layer serves as a linear The output layer serves as a linear combiner for the new space.combiner for the new space.

RBF network used in the researchRBF network used in the research

Network architecture:

80 radial basis functions in the hidden layer2 outputs in the output layer

Data PreparationData Preparation

• 40 Images each, of the four classes of 40 Images each, of the four classes of leaves were taken.leaves were taken.

• The Images were divided into training and The Images were divided into training and test data sets sequentially for all the test data sets sequentially for all the classes.classes.

• The feature extraction was performed for The feature extraction was performed for all the images by following the CCM all the images by following the CCM method. method.

Data PreparationData Preparation

• Finally the data was divided in to two text files:Finally the data was divided in to two text files:

1)Training texture feature data ( with all 39 texture features) and 1)Training texture feature data ( with all 39 texture features) and

2)Test texture feature data ( with all 39 texture features)2)Test texture feature data ( with all 39 texture features)

• The files had 80 rows each, representing 20 The files had 80 rows each, representing 20 samples from each of the four classes of samples from each of the four classes of leaves as discussed earlier. Each row had 39 leaves as discussed earlier. Each row had 39 columns representing the 39 texture features columns representing the 39 texture features extracted for a particular sample image extracted for a particular sample image

Data preparationData preparation

• Each row had a unique number (1, 2, 3 or Each row had a unique number (1, 2, 3 or 4) which represented the class the 4) which represented the class the particular row of data belonged particular row of data belonged

• These basic files were used to select the These basic files were used to select the

appropriate input for various data models appropriate input for various data models based on SAS analysis.based on SAS analysis.

Experimental methodsExperimental methods

• The training data was used for training the The training data was used for training the various classifiers as discussed in the various classifiers as discussed in the earlier slides.earlier slides.

• Once training was complete the test data Once training was complete the test data was used to test the classification was used to test the classification accuracies.accuracies.

• Results for various classifiers are given in Results for various classifiers are given in the following slides.the following slides.

ResultsResults

ResultsResults

ResultsResults

ResultsResults

Comparison of various classifiers Comparison of various classifiers for Model 1Bfor Model 1B

ClassifierClassifier Greasy spotGreasy spot MelanoseMelanose Normal Normal ScabScab OverallOverall

SAS SAS 100100 100100 9090 9595 96.396.3

MahalanobisMahalanobis 100100 100100 100100 9595 98.7598.75

NNBPNNBP 100100 9090 9595 9595 9595

RBFRBF 100100 100100 8585 6060 86.2586.25

SummarySummary

• It is concluded that model 1B consisting of It is concluded that model 1B consisting of features from hue and saturation is the features from hue and saturation is the best model for the task of citrus leaf best model for the task of citrus leaf classification. classification.

• Elimination of intensity in texture feature Elimination of intensity in texture feature calculation is the major advantage. It calculation is the major advantage. It nullifies the effect of lighting variations in nullifies the effect of lighting variations in an outdoor environment an outdoor environment

ConclusionConclusion

• The research was a feasibility analysis to The research was a feasibility analysis to see whether the techniques investigated in see whether the techniques investigated in this research can be implemented in future this research can be implemented in future real time applications. real time applications.

• Results show a positive step in that Results show a positive step in that direction. Nevertheless, the real time direction. Nevertheless, the real time system involves some modifications and system involves some modifications and tradeoffs to make it practical for outdoor tradeoffs to make it practical for outdoor applicationsapplications

Thank YouThank You

May I answer any questions?May I answer any questions?