[figure]

Hybrid Learning Using Genetic Algorithms and Decision Trees for

Pattern Classification

J. Bala, J. Huang, H. Vafaie, K. DeJong, H. Wechsler George Mason University Presented by

Sushil J. Louis Department of Computer Science

University of Nevada

Reno, NV sushil@cs.unr.edu

http://gaslab.cs.unr.edu
http://www.cs.unr.edu/ sushil

ML-F99 S. J. Louis

[figure]

Outline

• Hybrid Approaches

• Genetic Algorithms

• Decision Trees

• Feature selection

• Facial image data and Satellite images • Results

ML-F99 S. J. Louis

[figure]

Pattern Classification

• Difficult but fundamental task

• Depends heavily on the particular choice of features used by classifier

Rank features according to some statistic Eliminate low-ranked features

⇝ Ranking is not enough: Non-linear interactions between features Does not measure effectiveness on the complete task

ML-F99 S. J. Louis

[figure]

Feature Selection

• First step in any classification task

• Feature Selection is crucial for classification performance

• Reduces amount of data

• Increases classification accuracy

• Very difficult: Number of features is very large, space of possible subsets is exponential in number of features (100features → 2^{100}subsets).

ML-F99 S. J. Louis

[figure]

Optimal Feature Selection

• Minimize number of features

• Improve classification rates

• GA + DT system: GA searches the space of all possible subsets of a large set of candidate discrimination features for an optimal feature subset

1. For a given subset, ID3 produces a DT

2. Accuracy on unseen data → fitness to GA 3. GA evolves better subsets

Stop when performance passes a threshold

ML-F99 S. J. Louis

[figure]

System Architecture

• Initial (large) feature set • Run GA+ID3

• Recommend best feature set

ML-F99 S. J. Louis

[figure]

GA Representation

• Fixed length binary string

• i^{th} position == 1 → i^{th} feature is included.

• Genetic Algorithm modifies a population of such strings.

• For each individual in the population – Run ID3 on training set containing only these features

– Evaluate resulting DT on test data

– Return accuracy to GA as fitness of individual

ML-F99 S. J. Louis

[figure]

\underline{Methodology}

ML-F99 S. J. Louis

[figure]

Methodology (cont’d)

• Standard GA, popsize 50, standard parameter values

• C4.5, standard parameter values

• 2-fold Cross-validation

• 40% training 60% testing

• 10 runs of the system with different seeds

• Best tree out of ten is applied to testing data • Compare

1. Full feature set

2. Same number of features but use ranking • Satellite data (36 features total)

• Facial data (105 features total)

ML-F99 S. J. Louis

[figure]

Satellite Data

• Predict central pixel classification based on 3x3 neighborhood multi-spectral values

• Nasa Landsat images

• 6 classes

• 4 spectral bands, 9 pixel values → 36 features • Feature/pixel values are 0 − 255

• Learning/training set = 4435 examples, test data = 2000 examples

ML-F99 S. J. Louis

[figure]

\underline{Results}

ML-F99 S. J. Louis

[figure]

\underline{Results}

ML-F99 S. J. Louis

[figure]

\underline{Facial Data}

• Detecting eyes very important for face recognition

• 102 eyes, 102 nose, 102 other facial region examples for training

• 52 eyes, 52 nose, 52 other facial region for testing • 105 features for each example

• Feature values between 0 − 255

• Three classes

ML-F99 S. J. Louis

[figure]

\underline{Results}

ML-F99 S. J. Louis

[figure]

Face Results

ML-F99 S. J. Louis

[figure]

\underline{Results}

• Better error rates

• Reduced tree complexity for Satellite

• 60% reduction in facial features, 52% in satellite

ML-F99 S. J. Louis

[figure]

\underline{Results}

ML-F99 S. J. Louis

[figure]

Conclusions and Future Work • These preliminary results are encouraging

• Larger more complex problems in image processing

• Use information other than classification accuracy as GA fitness

ML-F99 S. J. Louis