Random Forest Regression
Definition of Regression tree:
A regression tree is a decision-tree method for predicting a numerical target variable by recursively dividing the predictor data into subsets with increasingly homogeneous target values.
This is the common definition for Regression tree, other definitions can be discussed in the article
|
Definition of Random forest:
A random forest is a prediction method that combines the outcomes of many decision trees. Each tree is built from a different random sample of the training observations and considers a random selection of predictor variables at each node.
This is the common definition for Random forest, other definitions can be discussed in the article
|
Regression tree
|
Glossary |
Starting from the complete training dataset, a regression tree repeatedly divides the data into two subsets, as illustrated in Fig. 1. At each node, all possible splits of the predictor variables are examined and the split is selected that best separates the target values, commonly by minimizing the sum of squared deviations from the mean target value within the resulting subsets. Splitting continues until a stopping criterion is reached, for example when a group becomes too small to divide further. This final subset forms a leaf. The mean value of the leaf data provides the final prediction.
Note that the questions are chosen one at a time, each being the best choice at that step. The resulting tree is therefore not necessarily the best possible tree overall.
Random forest
A single regression tree is often highly sensitive to the particular training data and can therefore overfit the data. A random forest avoids this problem by building many trees and averaging their predictions[1]. Each tree is made different in two ways:
- Random selection of observations. Each tree is built from a random draw of observations taken from the training data. The draw contains as many observations as the original dataset, but each observation is drawn from the full set, like drawing a slip from a hat and putting it back each time. As a result, some observations are drawn two or more times, while others are not drawn at all; on average about one third are left out. This way of drawing samples is called bootstrapping.
- Random selection of predictor variables. At each node, the tree may only choose its question from a randomly selected subset of the predictor variables. A new subset is drawn at every node. This prevents all trees from starting with the same dominant variable and ending up nearly identical.
The individual trees are deliberately grown deep, so each one tends to overfit. However, because the trees differ, their prediction errors also differ. Averaging the predictions of many such trees reduces these errors. This averaging procedure is called bagging. The forest's prediction is more reliable than that of any single tree.
Random forest settings can be adjusted by choosing the number of trees, by choosing the number of variables considered at each node and by choosing the minimum number of observations in a leaf.
Because each tree is fitted to a bootstrap sample, some training observations are not used in constructing that tree. These “out-of-bag” observations provide a convenient internal estimate of prediction error. However, when observations are close together in space or time, as is common with coastal and remote-sensing data, this estimate can be too optimistic. A reliable test then requires independent data taken from other locations or periods than the training data[2].
Regression forests based on ordinary trees are fundamentally poor extrapolators because predictions are based on averages of observed target values. They cannot predict target values outside the range represented in the training data and are therefore primarily suited to interpolation rather than extrapolation. Predictions for environmental conditions not represented in the training data should be treated with particular caution. This limitation is important when a model trained on present-day observations is used to predict extremes or future conditions under a changed climate.
Random forests can also be used for classification. In that case, the target variable consists of categories, such as sediment types, rather than numerical values.
| Analysis technique | Strengths | Limitations | Application example |
|---|---|---|---|
| Prediction tool based on machine learning from training data | * Handles nonlinear relationships * Does classification and regression * Relatively robust to noise and outliers * Can efficiently handle large datasets and many predictor variables * Less prone to overfitting than a single tree * Provides measures of the predictive importance of predictor variables |
* Less readily interpretable than a single regression tree * Computing time and memory increase with the number of trees * Poor at extrapolation beyond conditions represented in the training data |
* Prediction of environmental variables from multiple observations, for example estimating water quality, sediment properties or bathymetry from field and remote-sensing data |
For more detailed explanations see:
- StatQuest: Regression Trees, Clearly Explained by Josh Starmer
- StatQuest: Random Forests Part 1 - Building, Using and Evaluating by Josh Starmer
- Wikipedia Machine learning decision tree
- Wikipedia Random forest
Related articles
- Data analysis techniques for the coastal zone
- Linear regression analysis of coastal data
- Wavelet analysis of coastal processes
- Analysis of coastal processes with Empirical Orthogonal Functions
- Artificial Neural Networks and coastal applications
- Support Vector Regression
- Data interpolation with Kriging
References
- ↑ Breiman, L. 2001. Random forests. Machine Learning 45: 5–32
- ↑ Roberts, D.R., Bahn, V., Ciuti, S., Boyce, M.S., Elith, J., Guillera-Arroita, G., Hauenstein, S., Lahoz-Monfort, J.J., Schröder, B., Thuiller, W., Warton, D.I., Wintle, B.A., Hartig, F. and Dormann, C.F. 2017. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography 40: 913–929
