Random forests

This ensemble method is specifically created for regression or classification trees. It is very similar to bagging since, here, each individual tree is trained on a bootstrap sample of the training dataset. The difference with bagging is that it makes the model very powerful, and on splitting a node from the tree, the split that is picked is the best among a random subset of the features. So, every individual predictor considers a random subset of the features. This has the effect of making each individual predictor slightly worse and more biased but, due to the correlation of the individual predictors, the overall ensemble is generally better than the individual predictors.