Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Recreating R14 error with minimal code and finding a fix #33

Open
wants to merge 124 commits into
base: main
Choose a base branch
from

Conversation

deepsp94
Copy link
Contributor

@deepsp94 deepsp94 commented Feb 15, 2022

Problem: Running the following script on Linux throws the R14 error (excessive memory use).

image

The graph shows that memory use increases continuously. At the 0.5 GB mark, Heroku (which runs on linux OS) periodically throws the R14 error and at the 1 GB mark it shuts down and restarts the whole process (dyno).
Script: https://github.com/overlay-market/overlay-risk/blob/heroku-r14/scripts/r14_error.py

Background:

  1. The script is using pystable, which is a python wrapper of the C library libstable. The excessive memory use is due to the use of pystable.
  2. Pystable is only usable on mac computers or linux OS with glibc versions 2.31/2.32 (heroku runs on linux with glibc 2.31)
  3. Running the same script on a mac computer (m1) doesn't result in monotonically increasing use of memory. I've observed this behaviour on Linux only.

Objective: The script should work on Linux without memory issues.

Resources to get started:
How to install pystable: https://pypi.org/project/pystable/
Pystable repo: https://github.com/overlay-market/pystable
Libstable resources: https://www.jstatsoft.org/article/view/v078i01
About R14 error in the context of ruby applications: https://devcenter.heroku.com/articles/ruby-memory-use

@deepsp94 deepsp94 temporarily deployed to overlay-risk-heroku-r14-2a2ymo February 15, 2022 16:38 Inactive
@deepsp94 deepsp94 temporarily deployed to overlay-risk-heroku-r14-2a2ymo February 15, 2022 16:41 Inactive
@deepsp94 deepsp94 temporarily deployed to overlay-risk-heroku-r14-2a2ymo February 15, 2022 16:57 Inactive
@deepsp94 deepsp94 temporarily deployed to overlay-risk-heroku-r14-2a2ymo February 15, 2022 18:05 Inactive
@deepsp94 deepsp94 temporarily deployed to overlay-risk-heroku-r14-2a2ymo February 15, 2022 18:46 Inactive
@deepsp94 deepsp94 temporarily deployed to overlay-risk-heroku-r14-2a2ymo February 15, 2022 20:34 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants