Decision Trees

The decision tree algorithm is one of the most widely used algorithms in Machine Learning. It is a supervised learning algorithm. A decision tree uses a tree-like model to make predictions. It resembles an upside-down tree.

A decision tree builds classification or regression models in the form of a tree structure. It breaks down a data set into smaller subsets and the final result is a tree with decision nodes and leaf nodes.

A decision tree typically starts with a single node (root node), which branches into possible outcomes. Each of those outcomes leads to additional nodes, which branch off into other possibilities. This gives it a tree-like shape.

The topmost decision node in a tree corresponds to the best predictor called the root node. The Leaf node represents a classification or decision. Decision trees can handle both categorical and numerical data.

The process of partitioning the data set into subsets is known as splitting. Splits are formed on a particular variable. If a test splits the data into two partitions, then it is called a binary decision tree.  If it is more than two partitions, then it is known as a multi-way decision tree.

There are some selection criteria to choose a particular attribute for splitting the data. We should try to split the nodes such that the resulting nodes are as homogenous as possible. More homogeneity will mean that most of the data points in the set belong to the same class label.

When dealing with datasets where different subsets exhibit distinct trends, using a decision tree model that can partition the data into subsets and apply independent linear regression models within each subset can lead to more accurate and flexible predictions compared to a single linear regression model applied to the entire dataset. This approach allows for better capturing the diverse relationships within the data.

The difference between decision tree classification and decision tree regression is that in regression, each leaf represents a linear regression model, as opposed to a class label.

Advantages of Decision Trees

  • Decision trees are easy to interpret.
  • Building a decision tree requires little data preparation from the user- there is no need to normalize data

Disadvantages of Decision Trees

  • Decision trees are likely to overfit noisy data. The probability of overfitting on noise increases as a tree gets deeper.


Decision trees in machine learning are like flowcharts that make predictions based on data. They’re easy to understand and use, but can sometimes be too specific to the data they’re trained on, which is called overfitting. If overfitting is a problem, we can use a team of decision trees, known as a Random Forest, to get better results.

Comments are closed.