lambda parameter in sgd not working

[dustinvtran]

Not sure why, but in my experiments setting lambda to be anything other than zero leads to worse performance. Also the implicit case simply doesn't run.

[ptoulis]

Which experiment is this?

[dustinvtran]

I add the lambda parameter to the R code in MNIST, and partition Kaggle's training data into 75% train and 25% test (since the test data for Kaggle has no labels). Testing it on the 25% test can get horrific, up to 90ish percent misclassification per binary classifier.

[ptoulis]

Hmm is this line 73 in the sgd.R?
Do you see this for the implicit? Regularization could be funky here.

[dustinvtran]

Regularization for the explicit case and SVRG work now. The implicit seems to work on simulated data but it simply does not run for MNIST. After adding any non-zero lambda argument to line 44 in examples/exp_logistic_mnist.R:

source("examples/exp_logistic_mnist.R")
## [1] "Running AI-SGD for binary classifier on 0.."
## Error in uniroot(implicit.fn, interval = Bi) :
##   f() values at end points not of opposite sign

The update function fails after less than 10 iterations of the sgd loop.

This is possibly caused by xi multiplying the regularization term lambda*sqrt(sum((theta.old+ksi)^2)) after the implicit solve, c.f., lines 100 and 119 in sgd.R.

[ptoulis]

I think the implicit update is incorrect in the regularized version
i need to take a look