Memory issue with ggpredict/predict_response of fixest object

[strengejacke]

Discussed in #603

^{Originally posted by alexandervossen November 12, 2024}
Hi all. Thanks for the great work! I just wanted to rerun some code that is about a year old where I successfully used ggpredict to create predictions for plotting a curvilinear effect with ggplot2. However, I am no longer able to do so. The fixest model is fairly large with about 2,000,000 observations from 6,000 entities over 200 weeks from 6 markets.

What I do in general looks simplified like:

m1 <- feols(DV ~ IV + I(IV^2) | product_id + time_id + market_id, data = df)
p1 <- ggpredict(m1, terms = c("IV[all]"))

While it used to work on my local MacBook (16 GB RAM) a year ago, it now does not anymore and runs out of memory with “vector memory limit of 16 GB reached.” I also tried it on a VM I set up with 180 GB RAM, but that also crashes after running out of RAM as well. I tried with the Grunfeld data, and it seems to work in general, so it might be the large number of observations. Stata’s margins also does the job even on the local 16 GB MacBook. And the results are comparable to the older ones I got with ggpredict.

What also surprises me is that even if I replace "all" with specific values like "0.9 0.99", it finishes, but I get a notification: “Could not compute variance-covariance matrix of predictions. No confidence intervals are returned.” It just returns an object with 1 observation of 2 variables, with "x" and "group" both set to 1. So the actual values of x I requested are not returned, nor are the predictions. Even if I specify ci.level = NA (indicating I don’t want CIs), I still get that error. I tried around with different clustering of the SEs in genrating the feols model but that did not make a difference. I also tried the new predict_response command but the problem remains.

Grateful for any suggestions that might give a hint to where the problem is on my side.
Thanks!
A

[strengejacke]

Not fixed yet, seems to be more complicated...