Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lack of compilation/performance runs #47

Closed
m8pple opened this issue Nov 22, 2017 · 0 comments
Closed

Lack of compilation/performance runs #47

m8pple opened this issue Nov 22, 2017 · 0 comments

Comments

@m8pple
Copy link
Contributor

m8pple commented Nov 22, 2017

I've been having great fun with the auto tests over the past few days, where certain implementations
were managing to (I think) exhaust memory on the machine, which then freezes the machine. I'm not
quite sure how this is happening (there is an outer memory jail in place), and haven't been able
to diagnose it or strengthen or add internal memory jails enough to stop it happening. Another
possibility is that the GPU driver is somehow taking out the machine - this happens a fair amount on
consumer devices, but I haven't seen the AWS GPUs take out the machine before.

Anyway, I've fallen back on just adding logic to not run any implementation where a test run
has already started, then every time the machine freezes just start the script from just after
the place it left off. This may take a while, as sometimes the machine gets into a mode where
you can't stop it from the Amazon console (yay - watch the money burn on a machine you can't
log into!), so you have to terminate then rebuild.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant