What is XGBoost, and how does it improve upon standard GBM?

XGBoost is a modern implementation of Gradient Boosting Machine that works largely the same as the standard GBM. Like regular GBM, it fits to the residuals of previous trees and then predicts on a new observation by using a linear combination of the trees weighted by the learning rate, but it adds two additional regularization hyper-parameters that further aim to reduce prediction variance.

The first regularization parameter (lambda), ƛ, reduces the influence of individual observations on the structure of an XGBoost tree. Larger values of ƛ make it more difficult for the information gain of a split further down the tree to meet the criteria to be included in that tree, meaning increasing ƛ further controls the depth of trees created. Another hyperparameter (gamma), ɣ, also influences the degree of regularization by controlling the threshold at which to prune a tree at a given split. XGBoost also has the capability to be trained using GPUs, which can significantly reduce train time.