Getting NaN loss after some time with DeepAR and NegativeBinomial #833
Comments
|
maybe lower num_batches_per_epoch? I guess learning_rate is already low but the num_batches_per_epoch is high, so that during the first epoch training, arguments of distribution had numerical problems in calculating log prob. |
|
@samosun, my logic in increasing the num_batches_per_epoch is that I think each time-series should have a more or less equal chance of having a slice of it being part of each epoch. I mean, if I have a batch size of 32, 128976 / 32 = 4030,5. I'm actually using 4000 of num_batches_per_epoch right now (that number was because I had 300k+ time-series before and now I'm filtering some of them). Doesn't it make sense? Because I imagine if I was using a small num_batches_per_epoch, a given time-series could appear once in the first epoch and appear again several epochs later, which doesn't make sense to me. And I don't understand it would create numerical problems. Isn't the loss computed per batch, anyway? In my understanding, the gradient descendent doesn't even "know" about the concept of epochs. Only we know because it's when we take the avg_loss and is a possible point of stopping the training. |
yes, I think you are right. num_batches_per_epoch is not the root cause. so there may be other mysteries to be found if it's reproducible |
|
@samosun, I think I can confirm that it only happens with hybridize=True. I left it training with hybridize=False and it's in the 22th epoch now without NaN. Also, the model converges much faster with hybridize=False (0.147 at first epoch vs 0.28 at 4th epoch and then the NaN happened. There shouldn't be any difference besides speed with hybridize, but it seems that's something else happening. |
|
@fernandocamargoti are you using mxnet 1.6.0? And the It’s very weird indeed that this shows up with |
What’s the fraction of zero values, roughly? It would be nice if we could to reproduce the issue |
|
@lostella, yes, I'm using DeepAREstimator with mxnet 1.6.0. I'll get the fraction and get back to you today. Look at how faster the training with hybridize=False converges: Compared to the same training (every single hyperparameter is equal and I have seed in the random, numpy.random and mxnet) with hybridize=True: |
|
Hello, @lostella. Sorry for taking this long. I have 132131585 out of 143037878 (92.37%) of zeros. To give you more information, this dataset is from a retailer with many branches and many products. I have all the sales registered for each item in each branch. Some of these items might be discontinued (like smartphones) after some time. So, each time-series may start on a different date, but I've normalized all of them to finish on the same date. So, despite the fact that it's really normal for time-series to have must of its values equal to zero, there's also those that will never sell anything again, that contribute to the zeroes. |
|
Did you try switching back to mxnet-cu101mkl==1.4.1 - I think with that mxnet version you can leave out the hybridize False option. I just stumbled on this on my quest to get DeepAR converging somewhere in the M5 Kaggle challenge ... maybe it helps you @fernandocamargoti ? In case this works for you too something between mxnet 1.4.1 and 1.6 broke .... |
@fernandocamargoti as a side note: that that's not strictly needed, and may bias your predictions towards zero. Are you somehow indicating which values you have manually padded by adding some binary feature that indicates it? |
|
@fernandocamargoti also a minor note: the logs you pasted for hybridized=True suggest that you're using num_batches_per_epoch=4000, but you mentioned 10_000 (still, it seems to converge too slow even assuming that you used 10_000 for hybridized=False). |
|
@matthiasanderer, I'm running on IBM Power 9. If I'm not wrong, MKL is for Intel Processors. |
|
@lostella, thanks for your side note. But I also my predictions to indicate whether a time series will "die" (the product will lose all its demand). I haven't set up any binary feature to indicate it. If I had, how would I fill this feature in production? I mean, a product time series start dying, and at some point, no more sales are made. It would be hard to decide when to put set this flag to True in production. |
Yeah, it's another version where I filter time series with a minimum number of days with sales. Previously, I filtered with 10 and I increase to 30 to train faster. Then, I've updated the num_batches_per_epoch accordingly. Notice, though, that both logs have 4000. |
|
Oh I see, I hadn’t scrolled all the way to the right in the first log, sorry. I also agree on the zero padding you’re doing, since that’s a behavior you want to model 👍🏻 |
Sorry that was my copy paste of course only version 1.4.1 is the relevant part ... I just used that cuda10.1/mkl version... |
|
@lostella, I think I've got a way for you guys to reproduce the problem. The same thing happens on m5 dataset (https://www.kaggle.com/allunia/m5-uncertainty). To make it easier, I've already preprocessed the dataset, putting in the GluonTS format. It's available here: https://github.com/fernandocamargoti/m5_competition_gluonts_dataset The cardinality is [5, 12, 5, 9, 3051]. It always has +2 of the number of the categories because I leave the first index for unknown and the second for None (there are some aggregated time series where some categorical variables aren't defined). Also, I've been using my own implementation of Dataset instead of FileDataset from GluonTS. The reason is that the FileDataset was reading all the files to calculate the len() (which is called at the start of the training). It was taking a very long time. So, I wanted to start training and let the file be loaded only when necessary. To calculate the len(), I simply count the number of files, without reading them. So, here is my implementation: https://gist.github.com/fernandocamargoti/0f9c0b390ec44e0239a835cff91ae85c To separate the train and test set, I use: The SameSizeTransformedDataset is also provided in the gist above. It's only used to avoid reading the whole dataset to calculate the len(), allowing me to lazy load it during the training. |
|
As suggested by @matthiasanderer, I've tried MXNet 1.4.1 and this weird bug didn't happen. I'll also try 1.5.1 and report back. |
|
The bug doesn't happen with MXNet 1.5.1. So, it's something that happened when 1.6.0 was introduced. |
|
@fernandocamargoti thanks for looking into it, this is really cool (but also not cool that the bug is there). Did you observe that with 1.4 and 1.5 both NaN didn’t show up and convergence speed was unaffected by hybridize True/False? |
|
@lostella, both converge much faster and the NaN doesn't happen at all. But I've noticed that a weird bug happens with 1.5: I happens both when I try to run the evaluation and if I load a model I try to use it. So, I'm sticking with 1.4.1 for now. |
|
@fernandocamargoti the issue you’re facing with mxnet 1.5 is a known one on Linux: apache/mxnet#16135 This was fixed in 1.6 but the fix was not backported to 1.5 I believe. This is also why we skipped support for 1.5 in GluonTS, and directly jumped to 1.6 |
|
@fernandocamargoti did you mention you were running on GPU? In that case this may be related: #728 |
|
I'm wondering whether this still shows up with MXNet 1.7? (since that fixed the bug concerning #728) @fernandocamargoti not sure you're able to verify this, otherwise we'll try to reproduce with your instructions above |
|
Interesting, @GabrielDeza. My dynamic features are normalized as well. They're too important for my prediction for me to simply remove them, though. I also noticed that the resulting model that was trained after the NaNs, even though the logs say that the batch will be ignored, also result in NaNs in production. I didn't want to downgrade everything again after going through the hassle of upgrading it. Especially because MXNet 1.4.1 is very old now and requires an old version of numpy. But it seems that I, unfortunately, have no choice. It's very frustrating. |
|
Hello, @lostella. I have an update about this issue. I've used pytorch-ts, which is a library on top of GluonTS that reimplements DeepAR and other models in PyTorch and it works nicely. It might be good to give it a try as well, @GabrielDeza. |
|
@fernandocamargoti thanks for the update! The first thing that comes to mind is that pytorch-ts uses directly the distribution classes provided by PyTorch: negative binomial there uses a slightly different parametrization, which may be responsible for better numerical stability -- we should look into this. It also looks like MXNet's negative binomial uses the same parametrization. The best thing would be to rely directly on MXNet's distributions of course, as soon as they're out (v2.0 I believe). For reference:
|
|
Well, I've just got a NaN using the PyTorch implementation. When trying to use the model, I've got the following error: I know that it's from a different implementation, but it might be useful somehow. I've debugged the trained model and the I'll try to train again with torch.auto_grad.detect_anomaly() to try to discover what causes it. |
|
Well, all I got was: |
|
I don't know if it might be useful or not, but my loss starts increasing before the error: 18095it [31:13, 9.66it/s, avg_epoch_loss=0.0714, epoch=1] |
|
Actually, the loss might be rising because the pytorch-ts uses OneCycleLR. But it seems that the problem is simply exploding gradients. I'm trying to reduce the gradient clipping value and I'll also try to reduce the LR. |
|
Reducing the |
|
Same problem with MXNet implementation. I’ve reduced the learning rate and sometimes it help sometimes not. Maybe I’m not reducing it enough? Is there a way for me to know what clip_gradient to choose or is this a trial and error process? |
|
For reference, this issue also occurs on |
|
It might be quicker to investigate this issue with the |
|
I also encountered NaN loss when using DeepVAREstimator (mxnet version) on this toy dataset. This only happens when setting both hybridize=False and cell_type="lstm". When I try hybridize=True or change cell_type, the issue does not happen Hope this help! |
|
@fernandocamargoai Did you manage to find a fix for this issue ? I have the same problem when I try to use the NegativeBinomialOutput even with a strong gradient clipping. Thanks, |
Well, I've been using the PyTorch implementation. It still happens from time to time, depending on the hyperparameters. What seems to happen is that the parameters of the distribution might fall outside of the valid values depending on the hyperparameters. I noticed that some distributions are even more sensitive. But at least the PyTorch implementation seems better overall. |
|
Closed by mistake @fernandocamargoai I suspect that #1796 and this issue have a common root cause, which may have been fixed in #1893. If either of the issues gets closed because of the fix, maybe the other one can get closed too |
|
I am getting same exception what is the reason here? What is the meaning of this exception? |
Description
I have a dataset with 129k time series of demand for products within some branches of a retailer. I'm having trouble with NaN loss ever since I started using GluonTS. At first, I discovered some NaN and Infinity values within my FEAT_DYNAMIC_REAL. But even after solving those issues, I'm still getting NaN loss after some time. With the new release (which included some improvements on the NegativeBinomial, I'm getting a much lower loss value and it's taking longer for the problem to occur, but it still occurs.
I decided to put a conditional breakpoint in the log_prob() to stop whenever a NaN value is generated and used hybridize=False. It's super strange, but the error didn't happen yet (It's far past the point that it usually happens). I'll wait some more time, but it might be the case that it only happens with hybridize=True? If that's the case, it'd be very odd and problematic, since it runs much slower with hybridize=False.
To Reproduce
Unfortunately, I can't provide the dataset to reproduce the problem. But here are some information:
Error message or code output
Here is one example where it happened in the first epoch. But sometimes it happened later.
Environment
The text was updated successfully, but these errors were encountered: