When I use glmnet and cv.glmnet in R, sometimes I get the warning message:
cv <- cv.glmnet(x = as.matrix(min_column_data), y = all_datasets[["X20"]],
family = poisson(link = "log"), type.measure = "mse", alpha = 1, nlambda = 100, nfolds = 5)
Warning messages:
1: Convergence for 76th lambda value not reached after maxit=1e+05 iterations; solutions for larger lambdas returned
2: Convergence for 65th lambda value not reached after maxit=1e+05 iterations; solutions for larger lambdas returned
3: Convergence for 84th lambda value not reached after maxit=1e+05 iterations; solutions for larger lambdas returned
4: Convergence for 77th lambda value not reached after maxit=1e+05 iterations; solutions for larger lambdas returned
5: Convergence for 84th lambda value not reached after maxit=1e+05 iterations; solutions for larger lambdas returned
6: Convergence for 59th lambda value not reached after maxit=1e+05 iterations; solutions for larger lambdas returned
Then I have a look at the outcome of the above.
cv$lambda
[1] 3702.426406 3373.513048 3073.819447 2800.749799 2551.938906 2325.231687 2118.664513 1930.448197 1758.952499
[10] 1602.692007 1460.313267 1330.583061 1212.377729 1104.673433 1006.537290 917.119291 835.644941 761.408547
[19] 693.767111 632.134753 575.977643 524.809377 478.186758 435.705965 396.999048 361.730746 329.595584
[28] 300.315221 273.636044 249.326972 227.177451 206.995633 188.606711 171.851411 156.584605 142.674060
[37] 129.999290 118.450511 107.927694 98.339695 89.603468 81.643343 74.390373 67.781738 61.760195
[46] 56.273589 51.274399 46.719322 42.568907 38.787202 35.341454 32.201817 29.341096 26.734514
[55] 24.359493 22.195463 20.223679 18.427063 16.790054 15.298472 13.939398 12.701060 11.572733
[64] 10.544643 9.607886 8.754348 7.976636 7.268014 6.622343 6.034033 5.497986 5.009560
[73] 4.564524 4.159025 3.789548
> cv$lambda.min
[1] 3702.426
But I think the last lambda is much better 3.789548. I guess lambda is on the way of convergence. Am I right?
See this also:
min_column_data
X14 X12 X1 X16 X8 X4 X2 X3 X18 X19 X17 X13 X7 X9
1 16 14 3 3 11 7 10 5 6 0 13 7 1 11
2 18 6 6 4 9 7 5 4 4 5 4 6 0 8
3 22 4 18 15 3 11 12 2 3 0 2 5 45 8
4 10 22 15 7 22 11 14 15 10 5 5 25 0 18
5 8 6 3 4 7 3 6 6 29 16 5 1 0 7
6 17 8 4 3 12 4 2 2 2 0 2 2 38 66
7 5 11 6 13 8 16 20 7 3 3 3 20 0 6
8 9 4 13 7 17 7 6 16 11 1 4 1 0 11
9 8 11 17 11 7 9 9 16 5 8 13 6 0 10
10 15 12 3 3 19 2 2 5 13 12 10 7 0 0
11 9 4 8 2 2 0 4 26 3 2 2 15 0 0
12 22 4 8 16 6 10 11 8 1 7 3 3 0 5
13 3 5 14 15 10 4 9 2 5 14 2 20 0 10
14 9 4 12 23 0 8 8 8 7 5 3 4 0 8
15 15 7 12 8 12 11 12 9 7 0 2 1 0 12
16 7 2 8 3 18 3 2 7 7 4 2 7 0 0
17 6 4 3 13 3 3 2 5 12 5 2 6 0 0
18 5 14 12 2 2 13 6 3 12 0 11 6 85 17
19 17 4 7 9 6 20 6 6 2 12 8 3 0 10
20 10 11 15 14 5 2 3 24 3 2 4 5 0 26
21 4 8 5 9 13 10 18 2 4 13 15 4 0 17
22 18 7 6 6 4 6 20 18 4 4 18 14 0 11
23 4 4 22 16 3 8 4 2 6 5 4 15 0 16
24 11 9 5 5 3 18 1 11 18 4 3 5 0 12
25 5 8 3 12 6 1 4 6 10 14 1 5 0 8
> cv <- cv.glmnet(x = as.matrix(min_column_data), y = all_datasets[["X20"]],
+ family = poisson(link = "log"), type.measure = "mse", alpha = 1, nlambda = 100, nfolds = 5)
Even all numbers all between 0 and 100, it still sometimes says that glmnet.fit: algorithm did not converge. Why it happens?