XGBoost Explained: From Gradient Boosting to Weighted Quantile Sketch

XGBoost is a powerful tree-based algorithm that performs calculations significantly faster than traditional Gradient Boosting. It has gained popularity through its success in Kaggle competitions and its ability to handle sparse data and null values efficiently.

To understand XGBoost, it's essential to grasp the concept of boosting. In boosting, the mistakes of the first model are identified and fed to a subsequent model to correct them. This process is repeated, with each new model attempting to correct the errors of the previous one.

XGBoost improves upon the standard Gradient Boosting by optimizing an objective function that minimizes the difference between predicted and actual values. However, the objective function in XGBoost is non-optimizable in Euclidean space due to the varying decision tree structures. To address this, XGBoost uses the 2nd-order Taylor series expansion, which allows for optimization.

The algorithm estimates the optimal weights for each leaf node by solving the modified objective function. The split finding process involves calculating the gain for each potential split and selecting the one with the highest gain. XGBoost uses a greedy approach to evaluate split candidates, which can be computationally expensive for large datasets.

To address this, XGBoost employs an approximate split finding technique known as the weighted quantile sketch. This method represents feature values and their corresponding Hessians as a dataset, and then defines a weighted rank function to determine the fraction of total weight below a certain feature value. By setting an approximation factor (ε), XGBoost generates a manageable number of candidate split points. A smaller ε results in a finer approximation, allowing for more candidate split points.

Moreover, XGBoost handles missing values by assigning a default direction to each branch. During training, the algorithm determines the default direction that yields the best split score when encountering missing values during prediction. This approach ensures that missing values are appropriately handled without negatively impacting the model's performance.