Will the candidate elimination algorithm converge to the correct Hypothesis? Yes, under the following assumptions: 1. There are NO errors in the training examples 2. There IS some hypothesis h in H that correctly describes the target concept Suppose there are errors in the training examples CE will converge to the empty set CE will REMOVE correct hypothesis from VS CE removes every hypothesis inconsistent with each training example What training example should the learner next request? An example that removes some competing hypotheses from the candidates in the VS. Optimally: an example that removes 1/2 the hypotheses Like playing 20 questions Suppose we have several hypotheses in the VS and no more training examples. Given a new instance, what should the learner do? 1. Vote 2. Refuse to classify -> I don't know. 3. Supply a probability -> Answer is positive with probability = number of hypotheses that positively classify --------------------------------------------- total number of hypotheses in VS 3. Supply a probability -> Answer is negative with probability = number of hypotheses that negatively classify --------------------------------------------- total number of hypotheses in VS INDUCTIVE BIAS Current space of Hypotheses is restricted to Conjunction of features Suppose we unrestrict this space to all possible Hypotheses given the number of instances we need to classify. How many instances? 4 * 3^5 = 96 Number of hypotheses = 2^(96) !!! Conjunctions, disjunctions, negations Let CE be allowed all these. Consider three possible instances x1, x2, x3 three negative instances x4, x5 S = x1 OR X2 OR X3 G = ~x4 AND ~x5 == ~(x4 OR x5) In general S is disjunction of all training instances G is negation of (disjunction of all training instances) Perfect classification of all training instances Cannot classify new instance, x!! Why? Exactly half of the hypotheses will classify positive Exactly half of the hypotheses will classify negative if h E H covers x and classifies positive then there exists h' E H covers x and classifies negative remember H is power set of all instances if h is in VS then h' must also be in VS because h correctly classifies all seen instances and h' correctly classifies all seen instances Futility of bias free learning. If you don't have a bias you do not have a rational basis for generalizing/learning EnjoySport bias ==> only considers conjunctions EnjoySport assumes ==> Target concept describable using conjunctions To learn you must have INDUCTIVE BIAS. Inductive bias: Set of assumptions needed to deductively prove that the classification assigned by a learner is correct. Assignment 2. Understand and apply ID3 to as2.train -- use your own definition of information gain or the book's. Construct your decision tree Show your decision tree and report accuracy on as2.test Extra credit. Write id3 in C/C++