You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've been having great fun with the auto tests over the past few days, where certain implementations
were managing to (I think) exhaust memory on the machine, which then freezes the machine. I'm not
quite sure how this is happening (there is an outer memory jail in place), and haven't been able
to diagnose it or strengthen or add internal memory jails enough to stop it happening. Another
possibility is that the GPU driver is somehow taking out the machine - this happens a fair amount on
consumer devices, but I haven't seen the AWS GPUs take out the machine before.
Anyway, I've fallen back on just adding logic to not run any implementation where a test run
has already started, then every time the machine freezes just start the script from just after
the place it left off. This may take a while, as sometimes the machine gets into a mode where
you can't stop it from the Amazon console (yay - watch the money burn on a machine you can't
log into!), so you have to terminate then rebuild.
The text was updated successfully, but these errors were encountered:
I've been having great fun with the auto tests over the past few days, where certain implementations
were managing to (I think) exhaust memory on the machine, which then freezes the machine. I'm not
quite sure how this is happening (there is an outer memory jail in place), and haven't been able
to diagnose it or strengthen or add internal memory jails enough to stop it happening. Another
possibility is that the GPU driver is somehow taking out the machine - this happens a fair amount on
consumer devices, but I haven't seen the AWS GPUs take out the machine before.
Anyway, I've fallen back on just adding logic to not run any implementation where a test run
has already started, then every time the machine freezes just start the script from just after
the place it left off. This may take a while, as sometimes the machine gets into a mode where
you can't stop it from the Amazon console (yay - watch the money burn on a machine you can't
log into!), so you have to terminate then rebuild.
The text was updated successfully, but these errors were encountered: