Skip to content

Commit

Permalink
fix bad references
Browse files Browse the repository at this point in the history
  • Loading branch information
jdemonasterio committed Oct 26, 2017
1 parent 4cd7f8d commit 8d50029
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 5 deletions.
7 changes: 4 additions & 3 deletions ch_ensemble_methods.tex
Original file line number Diff line number Diff line change
Expand Up @@ -761,16 +761,17 @@ \subsection{Notes on Gradient Tree Boosting}\label{subsection:boosting_other_not

Instead, applications of boosters rely on optimization heuristics which use first and second order approximations of the loss function at step $m$ to build on the next tree $m+1$.

To work with this in the adaboost formulation , smooth loss functions become very convenient for this procedure.
To work with this in the Adaboost formulation , smooth loss functions become very convenient for this procedure.
As an example, we explain in detail a \textit{boosting} optimization heuristic at \cref{appx:sec:boosting_optimization_heuristic}.

There are two additional heuristics commonly used to improve the booster's generalization performance.
And for these, the arguments in favor of their use are rather experimental than theoretical.
The authors in~\cite{hastie-elemstatslearn} and~\cite{bishop-patternecognition} mention them because of their overall contribution to the generalization error and because although they are intuitive to the idea of variance reduction.

The first idea to reduce the boosters's variance is to subsample the data.
The first idea to reduce the booster's variance is to subsample the data.
This means that at each iteration, only a bootstrapped sample of the dataset will be selected to build the new weak learner.
Samples from $\mathcal{T}$ which are not part of the bootstrapped sample are ignored when optimizing for the new learner at \cref{eq:boosting_iterative_minimization}.

Samples from $\mathcal{T}$ which are not part of the bootstrapped sample are ignored when optimizing for the new learner at \cref{eq:adaBoostIteration}.

The motivation behind this is the same that as in Random Forest, where reducing the overall of available data to fit the new weak learner will most likely reduce the variance of the method.
In practice, the rate of sampling will be supervised by a tuning parameter in the model.
Expand Down
2 changes: 1 addition & 1 deletion ch_model_selection.tex
Original file line number Diff line number Diff line change
Expand Up @@ -748,7 +748,7 @@ \subsection{Final experiments on model selection}\label{sub:final_model_selectio

Another interesting result is that in both experiments, the best CV scores were achieved for highly regularized models.
To further explore this, we looked at their CV mean scores for each hyperparameter setting.
\cref{fig:rocauc_logreg_cv_l1_regularized_comparison, fig:rocauc_logreg_cv_l2_regularized_comparison} both show a series resulting from each type of regularization fit.
\cref{fig:rocauc_logreg_cv_l1_regularized_comparison,fig:rocauc_logreg_cv_l2_regularized_comparison} both show a series resulting from each type of regularization fit.

From these figures it is clear that both models improve the more regularized they become.
The full extent to which this regularization would keep improving the score is not explored because both experiments' $C$ parameter had a minimum at $10^{-5}$ which corresponds to the highest score for both.
Expand Down
2 changes: 1 addition & 1 deletion ch_results.tex
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ \chapter{Summary of Results}\label{ch:results}
A similar argument can be made for the best-features calculated from the Gradient Boosting experiment that used only a filtered set of features, previously filtered by the Random Forest experiments.
There is a strong feature-target correlation and there is a leakage of target information within this experiment.

To look at a more thorough examination of these results and how they were selected, the reader can refer to \cref{tab:random_forest_big_experiment_best_features, tab:boosting_big_experiment_best_features}.
To look at a more thorough examination of these results and how they were selected, the reader can refer to \cref{tab:random_forest_big_experiment_best_features,tab:boosting_big_experiment_best_features}.

% \section{Missing parts Plan to add to this section }
% \begin{description}
Expand Down

0 comments on commit 8d50029

Please sign in to comment.