Keep the command's output when it's killed

[stsewd]

When a command is killed because of memory consumption we replace the output with a message

https://github.com/rtfd/readthedocs.org/blob/85d9a1fb236e2c2149358b1cf462159e523e80bb/readthedocs/doc_builder/environments.py#L274-L276

the output can be helpful to users to debug the build. I guess we should only show the error about the memory in the top? Or maybe just concatenated the output with that msg?

Raised in #4946 (comment)
Related to #4468

[mrava87]

This would be very useful :) right now I need to blindly reduce the size of my example, make another pull request, rerun the build and hope :)

[humitos]

I suppose the reason why we put this pre-defined output is that when the command is killed, we don't get any output in return from docker.

[stsewd]

Interesting, I'll try to reproduce that locally and see

[stsewd]

So, after some attempts, I was able to reproduce and catch locally, this is what I found

296  	            killed_in_output = 'Killed' in '\n'.join(self.output.splitlines()[-15:])
297  	            if self.exit_code == DOCKER_OOM_EXIT_CODE or (self.exit_code == 1 and killed_in_output):
298  	                from celery.contrib import rdb
299  	                rdb.set_trace()
300  ->	                self.output = _('Command killed due to excessive memory '
301  	                                'consumption\n')
302  	        except DockerAPIError:
303  	            self.exit_code = -1
304  	            if self.output is None or not self.output:
305  	                self.output = _('Command exited abnormally')
(Pdb) p self.output
'Collecting Pygments==2.2.0\n  Using cached https://files.pythonhosted.org/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl\nRequirement already up-to-date: setuptools<40 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nCollecting docutils==0.13.1\n  Using cached https://files.pythonhosted.org/packages/7c/30/8fb30d820c012a6f701a66618ce065b6d61d08ac0a77e47fc7808dbaee47/docutils-0.13.1-py3-none-any.whl\nCollecting mock==1.0.1\nRequirement already up-to-date: pillow==2.6.1 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nRequirement already up-to-date: alabaster!=0.7.5,<0.8,>=0.7 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nCollecting commonmark==0.5.4\nRequirement already up-to-date: recommonmark==0.4.0 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nCollecting sphinx<1.8\n  Using cached https://files.pythonhosted.org/packages/90/f9/a0babe32c78480994e4f1b93315558f5ed756104054a7029c672a8d77b72/Sphinx-1.7.9-py2.py3-none-any.whl\nRequirement already up-to-date: sphinx-rtd-theme<0.5 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nRequirement already up-to-date: readthedocs-sphinx-ext<0.6 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nRequirement already up-to-date: imagesize in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nRequirement already up-to-date: packaging in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nRequirement already up-to-date: babel!=2.0,>=1.3 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nRequirement already up-to-date: Jinja2>=2.3 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nCollecting requests>=2.0.0 (from sphinx<1.8)\n  Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl\nRequirement already up-to-date: sphinxcontrib-websupport in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nRequirement already up-to-date: snowballstemmer>=1.1 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nCollecting six>=1.5 (from sphinx<1.8)\n  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\nRequirement already up-to-date: pyparsing>=2.0.2 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from packaging->sphinx<1.8)\nRequirement already up-to-date: pytz>=0a in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from babel!=2.0,>=1.3->sphinx<1.8)\nRequirement already up-to-date: MarkupSafe>=0.23 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from Jinja2>=2.3->sphinx<1.8)\nCollecting idna<2.9,>=2.5 (from requests>=2.0.0->sphinx<1.8)\n  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl\nRequirement already up-to-date: certifi>=2017.4.17 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from requests>=2.0.0->sphinx<1.8)\nRequirement already up-to-date: urllib3<1.25,>=1.21.1 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from requests>=2.0.0->sphinx<1.8)\nRequirement already up-to-date: chardet<3.1.0,>=3.0.2 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from requests>=2.0.0->sphinx<1.8)\nInstalling collected packages: Pygments, docutils, mock, commonmark, idna, requests, six, sphinx\n  Found existing installation: Pygments 2.3.0\n    Uninstalling Pygments-2.3.0:\n      Successfully uninstalled Pygments-2.3.0\n  Found existing installation: docutils 0.14\n    Uninstalling docutils-0.14:\n      Successfully uninstalled docutils-0.14\n  Found existing installation: mock 2.0.0\n    Uninstalling mock-2.0.0:\n      Successfully uninstalled mock-2.0.0\n  Found existing installation: CommonMark 0.5.5\n    Uninstalling CommonMark-0.5.5:\n      Successfully uninstalled CommonMark-0.5.5\n  Found existing installation: idna 2.7\n    Uninstalling idna-2.7:\n      Successfully uninstalled idna-2.7\nKilled\n'
(Pdb) 

We have some output, it may be helpful for debugging, and there is a Killed at the end, so I think we can append the message or just rely on the top message to communicate the error.

I'm going to keep investigating if there are some side effects.

[mrava87]

Hi,
I still seem to get the initial error message with no hint on where the build breaks. See attachment and link (https://readthedocs.org/projects/pylops/builds/8330708/).

I thought this was merged?

[mrava87]

I have tried to even rebuild the stable version which points to a commit at tag v1.1.0. If this built at some point I don't see how it is possible that it is not building anymore due to 'excessive memory consumption'. Do you sometimes grab bigger nodes than other times for the built in docker?

[stsewd]

I still seem to get the initial error message with no hint on where the build breaks.

This isn't deployed yet

Do you sometimes grab bigger nodes than other times for the built in docker?

Sometimes the builders are very busy, so if your project needs a lot of resources it could fail.