Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bdist during installation of Python requirements #18745

Merged
merged 6 commits into from
Jun 8, 2020

Conversation

cachedout
Copy link
Contributor

What does this PR do?

This PR addresses an issue in the build system which produces errors similar to the following:

09:10:15  Building wheels for collected packages: autopep8, backports.ssl-match-hostname, dockerpty, docopt, ipaddress, MarkupSafe, nose-timer, PyYAML, termcolor, texttable, jsondiff, stomp.py, ordered-set
09:10:15    Running setup.py bdist_wheel for autopep8: started
09:10:15    Running setup.py bdist_wheel for autopep8: finished with status 'error'
09:10:15    Complete output from command /tmp/tmp.mtLXh51exX/python-env/build/ve/linux/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-a_wqse9r/autopep8/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpxy52gjolpip-wheel- --python-tag cp36:
09:10:15    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
09:10:15       or: -c --help [cmd1 cmd2 ...]
09:10:15       or: -c --help-commands
09:10:15       or: -c cmd --help
09:10:15    
09:10:15    error: invalid command 'bdist_wheel'
09:10:15    
09:10:15    ----------------------------------------
09:10:15    Failed building wheel for autopep8
09:10:15    Running setup.py clean for autopep8
09:10:15    Running setup.py bdist_wheel for backports.ssl-match-hostname: started
09:10:16    Running setup.py bdist_wheel for backports.ssl-match-hostname: finished with status 'error'
09:10:16    Complete output from command /tmp/tmp.mtLXh51exX/python-env/build/ve/linux/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-a_wqse9r/backports.ssl-match-hostname/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpzd52u8z1pip-wheel- --python-tag cp36:
09:10:16    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
09:10:16       or: -c --help [cmd1 cmd2 ...]
09:10:16       or: -c --help-commands
09:10:16       or: -c cmd --help
09:10:16    
09:10:16    error: invalid command 'bdist_wheel'

Note that these errors are from the APM CI where they have been discovered in builds for the APM Server project but the source of the errors appears to stem from libbeat.

A full discussion and background on this issue can be found here: elastic/apm-server#3661

Cause and resolution

I believe this may have been introduced during the migration to Python 3 but I don't have a test history that goes back far enough to confirm this.

This is a difficult-to-reproduce problem and I have yet to be able to do so locally. However, it seems clear that the issue is that the wheel Python package needs to be present in order for the wheels to be created. This fix attempts to simply simply tell pip to install wheel into the newly-created virtual environment before it proceeds with installing packages from the requirements file.

Why is it important?

Fixes errors found in the CI on package which depend on libbeat.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

I haven't found a way to just run the PythonVirtualenv target with mage but I did run mage pythonUnitTest which does seem to exercise this code path.

Related issues

@cachedout cachedout added the bug label May 26, 2020
@cachedout cachedout requested review from andrewkroh and a team May 26, 2020 13:37
@cachedout cachedout self-assigned this May 26, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 26, 2020
@cachedout cachedout changed the title Fix bdist Fix bdist during installation of Python requirements May 26, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 26, 2020

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18745 updated]

  • Reason: Aborted from #3

  • Start Time: 2020-05-26T14:02:33.156+0000

  • Duration: 56 min 22 sec

  • Commit: 5c8c27b

Test stats 🧪

Test Results
Failed 1
Passed 2608
Skipped 175
Total 2784

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat OSS Unit tests / [empty] – TEST-go-unit.xml

    • Age: 1
    • Duration: 0
    • Error Details:

Steps errors

Expand to view the steps failures

  • Name: Report to Codecov
    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-05-26T14:31:04.018+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-05-26T14:58:11.791Z] + FILE=heartbeat/build/coverage/full.cov
[2020-05-26T14:58:11.791Z] + '[' -f heartbeat/build/coverage/full.cov ']'
[2020-05-26T14:58:11.791Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-05-26T14:58:11.791Z] + FILE=libbeat/build/coverage/full.cov
[2020-05-26T14:58:11.791Z] + '[' -f libbeat/build/coverage/full.cov ']'
[2020-05-26T14:58:11.791Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-05-26T14:58:11.791Z] + FILE=metricbeat/build/coverage/full.cov
[2020-05-26T14:58:11.791Z] + '[' -f metricbeat/build/coverage/full.cov ']'
[2020-05-26T14:58:11.791Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-05-26T14:58:11.791Z] + FILE=packetbeat/build/coverage/full.cov
[2020-05-26T14:58:11.791Z] + '[' -f packetbeat/build/coverage/full.cov ']'
[2020-05-26T14:58:11.791Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-05-26T14:58:11.791Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-05-26T14:58:11.791Z] + '[' -f winlogbeat/build/coverage/full.cov ']'
[2020-05-26T14:58:11.791Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-05-26T14:58:11.791Z] + FILE=journalbeat/build/coverage/full.cov
[2020-05-26T14:58:11.791Z] + '[' -f journalbeat/build/coverage/full.cov ']'
[2020-05-26T14:58:13.144Z] Post stage
[2020-05-26T14:58:13.154Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats
[2020-05-26T14:58:13.940Z] + script/fix_permissions.sh /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745
[2020-05-26T14:58:13.940Z] Client:
[2020-05-26T14:58:13.940Z]  Version:           18.06.1-ce
[2020-05-26T14:58:13.940Z]  API version:       1.38
[2020-05-26T14:58:13.940Z]  Go version:        go1.10.3
[2020-05-26T14:58:13.940Z]  Git commit:        e68fc7a
[2020-05-26T14:58:13.940Z]  Built:             Tue Aug 21 17:21:31 2018
[2020-05-26T14:58:13.940Z]  OS/Arch:           darwin/amd64
[2020-05-26T14:58:13.940Z]  Experimental:      false
[2020-05-26T14:58:13.940Z] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[2020-05-26T14:58:13.940Z] It requires Docker daemon to be installed and running
[2020-05-26T14:58:29.900Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats
[2020-05-26T14:58:30.219Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-26T14:58:30.233Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Lint
[2020-05-26T14:58:30.322Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-05-26T14:58:30.395Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Winlogbeat-oss
[2020-05-26T14:58:30.469Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-05-26T14:58:30.543Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Dockerlogbeat
[2020-05-26T14:58:30.618Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-05-26T14:58:30.700Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Journalbeat-oss
[2020-05-26T14:58:30.776Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Libbeat-oss
[2020-05-26T14:58:30.849Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-05-26T14:58:30.921Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Packetbeat-oss
[2020-05-26T14:58:31.006Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-05-26T14:58:31.083Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-05-26T14:58:31.157Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Filebeat-oss
[2020-05-26T14:58:31.229Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Heartbeat-oss
[2020-05-26T14:58:31.306Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Auditbeat-Linux
[2020-05-26T14:58:31.382Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-05-26T14:58:31.453Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-05-26T14:58:31.532Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-05-26T14:58:31.616Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-05-26T14:58:31.690Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-05-26T14:58:31.770Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-05-26T14:58:31.843Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-05-26T14:58:31.917Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Libbeat-x-pack
[2020-05-26T14:58:31.989Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-05-26T14:58:32.062Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Filebeat-x-pack
[2020-05-26T14:58:32.135Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-05-26T14:58:32.207Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-05-26T14:58:32.294Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-05-26T14:58:32.661Z] + cat
[2020-05-26T14:58:32.661Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-26T14:58:32.661Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-26T14:58:40.835Z] runbld>>> runbld started
[2020-05-26T14:58:40.835Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-26T14:58:41.786Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18745' in order of occurrence in the config (last value wins).
[2020-05-26T14:58:43.179Z] runbld>>> Debug logging enabled.
[2020-05-26T14:58:43.180Z] runbld>>> Storing result
[2020-05-26T14:58:43.180Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-26T14:58:43.180Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200526145842-91298DDC
[2020-05-26T14:58:43.180Z] runbld>>> Adding system facts.
[2020-05-26T14:58:44.134Z] runbld>>> Adding vcs info for the latest commit:  709d13c260ba2991bbbda5569edaa2a20be35dfa
[2020-05-26T14:58:44.134Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-26T14:58:44.134Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-26T14:58:44.396Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-26T14:58:44.396Z] Processing JUnit reports with runbld...
[2020-05-26T14:58:44.659Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-26T14:58:44.660Z] runbld>>> DURATION: 17ms
[2020-05-26T14:58:44.660Z] runbld>>> STDOUT: 40 bytes
[2020-05-26T14:58:44.660Z] runbld>>> STDERR: 49 bytes
[2020-05-26T14:58:44.660Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-26T14:58:44.660Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats
[2020-05-26T14:58:46.050Z] runbld>>> Storing build metadata: 
[2020-05-26T14:58:46.050Z] runbld>>> Adding test report.
[2020-05-26T14:58:46.050Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745/src/github.com/elastic/beats
[2020-05-26T14:58:46.997Z] runbld>>> Found 13 test output files
[2020-05-26T14:58:47.949Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 2783 Skipped: 133
[2020-05-26T14:58:48.213Z] runbld>>> Storing result
[2020-05-26T14:58:48.213Z] runbld>>> FAILURES: 0
[2020-05-26T14:58:48.213Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-26T14:58:48.213Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200526145842-91298DDC
[2020-05-26T14:58:48.213Z] runbld>>> Email notification disabled by environment variable.
[2020-05-26T14:58:48.213Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-26T14:58:54.063Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18745
[2020-05-26T14:58:54.171Z] [INFO] getVaultSecret: Getting secrets
[2020-05-26T14:58:54.230Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-26T14:58:54.933Z] + chmod 755 generate-build-data.sh
[2020-05-26T14:58:54.933Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18745/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18745/runs/2 ABORTED 3381515
[2020-05-26T14:58:55.183Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18745/runs/2/steps/?limit=10000 -o steps-info.json
[2020-05-26T14:58:56.527Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18745/runs/2/tests/?status=FAILED -o tests-errors.json

@mdelapenya mdelapenya added the Team:Automation Label for the Observability productivity team label May 26, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 26, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 26, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18745 updated]

  • Start Time: 2020-06-05T07:28:04.263+0000

  • Duration: 89 min 57 sec

Test stats 🧪

Test Results
Failed 0
Passed 9317
Skipped 1574
Total 10891

@cachedout
Copy link
Contributor Author

Hi @andrewkroh . I was hoping this might be something you could take a look at. Any chance you could weigh in when you have a moment? Thanks!

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like a good change to me. Let's test it on CI and verify that it fixes the issues everywhere.

dev-tools/mage/pytest.go Outdated Show resolved Hide resolved
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
@cachedout
Copy link
Contributor Author

Sounds like a good change to me. Let's test it on CI and verify that it fixes the issues everywhere.

Hi @andrewkroh . I looked through the logs from this test run and couldn't detect any failures related to bdist or wheel.

The failure on APM server is somewhat transient so verifying this fix prior to merging it might prove challenging. Are you opposed to merging this as-is and seeing how things look on the APM Server CI job over the next 1-2 weeks?

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@cachedout cachedout merged commit 10a2ce9 into elastic:master Jun 8, 2020
marc-gr pushed a commit to marc-gr/beats that referenced this pull request Jul 14, 2020
* Fix bdist/wheel errors on build

* Use independent args

* Formatting

* Remove -r

* Collapse into single line

* Update dev-tools/mage/pytest.go

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 10a2ce9)
@marc-gr marc-gr added the v7.9.0 label Jul 14, 2020
andrewkroh pushed a commit that referenced this pull request Jul 14, 2020
* Fix bdist/wheel errors on build

* Use independent args

* Formatting

* Remove -r

* Collapse into single line

* Update dev-tools/mage/pytest.go

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 10a2ce9)

Co-authored-by: cachedout <mike.place@elastic.co>
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
* Fix bdist/wheel errors on build

* Use independent args

* Formatting

* Remove -r

* Collapse into single line

* Update dev-tools/mage/pytest.go

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Team:Automation Label for the Observability productivity team v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants