In Decision Tree, the major challenge is the identification of the attribute for the root node in each level. This process is known as attribute selection. There are two popular attribute selection measures:
- Gini Index
- Information Gain
Gini Index calculates the amount of probability of a specific feature that is classified incorrectly when selected randomly. It means an attribute with a lower Gini index should be preferred.
Gini index varies between values 0 and 1, where 0 expresses the purity of classification and 1 indicates the random distribution of elements across various classes. The value of 0.5 of the Gini Index shows an equal distribution of elements over some classes.
The Gini Index is determined by deducting the sum of squared of probabilities of each class from one, mathematically, Gini Index can be expressed as:
Where Pi denotes the probability of an element being classified for a distinct class.
Information Gain: Information Gain is the entropy of the parent node minus the weighted entropy of the child nodes. The idea is to maximize information gain. Entropy measures the amount of randomness or chaos in the set. A set with equal distribution of classes has maximum entropy.