How to remove the models that failed convergence from a set of random questions?

Question

I want to include some random replications of model estimations (e.g., GARCH model) in the question. The code uses a different data series randomly. In this process, some GARCH estimations for some random data series may not achieve numerical convergence. Therefore, I need to code the question/problem in such a way that it has to remove the models that failed convergence from the set of questions. How can I code this when I use R-exams?

Answer 1

Basic idea

In general when using random data in the generation of exercises, there is a chance that sometimes something goes wrong, e.g., the solution does not fall into a desired range (i.e., becomes too large or too small), or the solution does not even exist due to mathematical intractability or numerical problems (as you point out) etc.

Of course, it is best to avoid such problems in the data-generating process so that they do not occur at all. However, it is not always possible to do so or not worth the effort because problems occur very rarely. In such situations I typically use a while() loop to re-generate the random data if necessary. As this might run potentially for several iterations it is important, though, to make the probably sufficiently small that it is needed.

Worked example

A worked example can be found in the fourfold exercise that ships with the package. It randomly generates a fourfold table with probabilities that should subsequently be reconstructed from partial information in the actual exercise. In order for the exercise to be well-defined all entries of the table must be (strictly) between 0 and 1 and they must sum up to 1. The simulation code actually tries to assure that but edge cases might occur. Rather than writing more code to avoid these edge cases, a simple while() loop tries to catch them and sample a new table if needed:

ok <- FALSE
while(!ok) {
  [...generate probabilities...]
  tab <- cbind(c(prob1, prob3), c(prob2, prob4))
  [...compute solutions...]
  ok <- sum(tab) == 1 & all(tab > 0) & all(tab < 1)
}

Application to catching errors

The same type of strategy could also be used for other problems such as the ones you describe. You can wrap the model estimation into a code like

fit <- try(mymodel(...), silent = TRUE)

and then use something like

ok <- !inherits(fit, "try-error")

In addition to not producing an error you might require, say that all coefficients are positive (or something like that). Then you would do:

ok <- !inherits(fit, "try-error") && all(coef(fit) > 0)

Analogously, you could check the convergence of the model etc.