Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing or incorrect GitLab environment variables for parallelization #2343

Closed
bahmutov opened this issue Aug 14, 2018 · 6 comments · Fixed by #2345
Closed

Missing or incorrect GitLab environment variables for parallelization #2343

bahmutov opened this issue Aug 14, 2018 · 6 comments · Fixed by #2345
Assignees
@bahmutov

Comments

@bahmutov
Copy link
Contributor

Cypress 3.1.0

I created a GitLab project https://gitlab.com/bahmutov/kitchensink (branch 3.1.0) and ran it with --parallel flag

The config file defined 3 jobs

1x-electron:
  image: cypress/base:10
  stage: test

  # cache NPM modules using branch name
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
    - /root/.npm
    - /root/.cache

  before_script:
    - npm ci
    - npm run cy:verify

  script:
    - $(npm bin)/print-env CI
    - npm run start:ci &
    - npm run e2e:record -- --parallel --group 1x-electron

2x-electron-1:
  image: cypress/base:10
  stage: test

  # cache NPM modules using branch name
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
    - /root/.npm
    - /root/.cache

  before_script:
    - npm ci
    - npm run cy:verify

  script:
    - $(npm bin)/print-env CI
    - npm run start:ci &
    - npm run e2e:record -- --parallel --group 2x-electron

2x-electron-2:
  image: cypress/base:10
  stage: test

  # cache NPM modules using branch name
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
    - /root/.npm
    - /root/.cache

  before_script:
    - npm ci
    - npm run cy:verify

  script:
    - $(npm bin)/print-env CI
    - npm run start:ci &
    - npm run e2e:record -- --parallel --group 2x-electron

problem 1

Alls builds created separate runs in the dashboard

screen shot 2018-08-14 at 2 08 49 pm

problem 2

Also, the branch is missing, see https://dashboard.cypress.io/#/projects/4b7344/runs/2061/specs

screen shot 2018-08-14 at 2 04 55 pm

GitLab environment variables

GitLab injects the following environment variables during build: https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables

I printed all variables that start with CI_ and here they are from several jobs in the same pipeline https://gitlab.com/bahmutov/kitchensink/pipelines/27836952

screen shot 2018-08-14 at 2 11 33 pm

1ex-electron

https://gitlab.com/bahmutov/kitchensink/-/jobs/89078654

$ $(npm bin)/print-env CI
CI=true
CI_BUILD_BEFORE_SHA=32086b3a147d81e186c4131f6372844273c8e204
CI_BUILD_ID=89078654
CI_BUILD_NAME=1x-electron
CI_BUILD_REF=f8fc41d3e35203497326365d3a8e264fa207226d
CI_BUILD_REF_NAME=3.1.0
CI_BUILD_REF_SLUG=3-1-0
CI_BUILD_STAGE=test
CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_COMMIT_BEFORE_SHA=32086b3a147d81e186c4131f6372844273c8e204
CI_COMMIT_DESCRIPTION=
CI_COMMIT_MESSAGE=it is ci-build-id

CI_COMMIT_REF_NAME=3.1.0
CI_COMMIT_REF_SLUG=3-1-0
CI_COMMIT_SHA=f8fc41d3e35203497326365d3a8e264fa207226d
CI_COMMIT_TITLE=it is ci-build-id
CI_CONFIG_PATH=.gitlab-ci.yml
CI_DISPOSABLE_ENVIRONMENT=true
CI_JOB_ID=89078654
CI_JOB_NAME=1x-electron
CI_JOB_STAGE=test
CI_JOB_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_JOB_URL=https://gitlab.com/bahmutov/kitchensink/-/jobs/89078654
CI_PIPELINE_ID=27836952
CI_PIPELINE_IID=11
CI_PIPELINE_SOURCE=push
CI_PIPELINE_URL=https://gitlab.com/bahmutov/kitchensink/pipelines/27836952
CI_PROJECT_DIR=/builds/bahmutov/kitchensink
CI_PROJECT_ID=7930861
CI_PROJECT_NAME=kitchensink
CI_PROJECT_NAMESPACE=bahmutov
CI_PROJECT_PATH=bahmutov/kitchensink
CI_PROJECT_PATH_SLUG=bahmutov-kitchensink
CI_PROJECT_URL=https://gitlab.com/bahmutov/kitchensink
CI_PROJECT_VISIBILITY=public
CI_REGISTRY=registry.gitlab.com
CI_REGISTRY_IMAGE=registry.gitlab.com/bahmutov/kitchensink
CI_REGISTRY_PASSWORD=xxxxxxxxxxxxxxxxxxxx
CI_REGISTRY_USER=gitlab-ci-token
CI_REPOSITORY_URL=https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/bahmutov/kitchensink.git
CI_RUNNER_DESCRIPTION=shared-runners-manager-3.gitlab.com
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_RUNNER_ID=44028
CI_RUNNER_REVISION=081978aa
CI_RUNNER_TAGS=docker, east-c, gce, git-annex, linux, mongo, mysql, postgres, ruby, shared
CI_RUNNER_VERSION=11.1.0
CI_SERVER=yes
CI_SERVER_NAME=GitLab
CI_SERVER_REVISION=7996ef5
CI_SERVER_TLS_CA_FILE=/builds/bahmutov/kitchensink.tmp/CI_SERVER_TLS_CA_FILE
CI_SERVER_VERSION=11.2.0-rc2-ee

2x-electron-1

https://gitlab.com/bahmutov/kitchensink/-/jobs/89078655

CI=true
CI_BUILD_BEFORE_SHA=32086b3a147d81e186c4131f6372844273c8e204
CI_BUILD_ID=89078655
CI_BUILD_NAME=2x-electron-1
CI_BUILD_REF=f8fc41d3e35203497326365d3a8e264fa207226d
CI_BUILD_REF_NAME=3.1.0
CI_BUILD_REF_SLUG=3-1-0
CI_BUILD_STAGE=test
CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_COMMIT_BEFORE_SHA=32086b3a147d81e186c4131f6372844273c8e204
CI_COMMIT_DESCRIPTION=
CI_COMMIT_MESSAGE=it is ci-build-id

CI_COMMIT_REF_NAME=3.1.0
CI_COMMIT_REF_SLUG=3-1-0
CI_COMMIT_SHA=f8fc41d3e35203497326365d3a8e264fa207226d
CI_COMMIT_TITLE=it is ci-build-id
CI_CONFIG_PATH=.gitlab-ci.yml
CI_DISPOSABLE_ENVIRONMENT=true
CI_JOB_ID=89078655
CI_JOB_NAME=2x-electron-1
CI_JOB_STAGE=test
CI_JOB_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_JOB_URL=https://gitlab.com/bahmutov/kitchensink/-/jobs/89078655
CI_PIPELINE_ID=27836952
CI_PIPELINE_IID=11
CI_PIPELINE_SOURCE=push
CI_PIPELINE_URL=https://gitlab.com/bahmutov/kitchensink/pipelines/27836952
CI_PROJECT_DIR=/builds/bahmutov/kitchensink
CI_PROJECT_ID=7930861
CI_PROJECT_NAME=kitchensink
CI_PROJECT_NAMESPACE=bahmutov
CI_PROJECT_PATH=bahmutov/kitchensink
CI_PROJECT_PATH_SLUG=bahmutov-kitchensink
CI_PROJECT_URL=https://gitlab.com/bahmutov/kitchensink
CI_PROJECT_VISIBILITY=public
CI_REGISTRY=registry.gitlab.com
CI_REGISTRY_IMAGE=registry.gitlab.com/bahmutov/kitchensink
CI_REGISTRY_PASSWORD=xxxxxxxxxxxxxxxxxxxx
CI_REGISTRY_USER=gitlab-ci-token
CI_REPOSITORY_URL=https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/bahmutov/kitchensink.git
CI_RUNNER_DESCRIPTION=shared-runners-manager-5.gitlab.com
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_RUNNER_ID=380986
CI_RUNNER_REVISION=081978aa
CI_RUNNER_TAGS=gce, docker
CI_RUNNER_VERSION=11.1.0
CI_SERVER=yes
CI_SERVER_NAME=GitLab
CI_SERVER_REVISION=7996ef5
CI_SERVER_TLS_CA_FILE=/builds/bahmutov/kitchensink.tmp/CI_SERVER_TLS_CA_FILE
CI_SERVER_VERSION=11.2.0-rc2-ee

2x-electron-2

https://gitlab.com/bahmutov/kitchensink/-/jobs/89078656

CI=true
CI_BUILD_BEFORE_SHA=32086b3a147d81e186c4131f6372844273c8e204
CI_BUILD_ID=89078656
CI_BUILD_NAME=2x-electron-2
CI_BUILD_REF=f8fc41d3e35203497326365d3a8e264fa207226d
CI_BUILD_REF_NAME=3.1.0
CI_BUILD_REF_SLUG=3-1-0
CI_BUILD_STAGE=test
CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_COMMIT_BEFORE_SHA=32086b3a147d81e186c4131f6372844273c8e204
CI_COMMIT_DESCRIPTION=
CI_COMMIT_MESSAGE=it is ci-build-id

CI_COMMIT_REF_NAME=3.1.0
CI_COMMIT_REF_SLUG=3-1-0
CI_COMMIT_SHA=f8fc41d3e35203497326365d3a8e264fa207226d
CI_COMMIT_TITLE=it is ci-build-id
CI_CONFIG_PATH=.gitlab-ci.yml
CI_DISPOSABLE_ENVIRONMENT=true
CI_JOB_ID=89078656
CI_JOB_NAME=2x-electron-2
CI_JOB_STAGE=test
CI_JOB_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_JOB_URL=https://gitlab.com/bahmutov/kitchensink/-/jobs/89078656
CI_PIPELINE_ID=27836952
CI_PIPELINE_IID=11
CI_PIPELINE_SOURCE=push
CI_PIPELINE_URL=https://gitlab.com/bahmutov/kitchensink/pipelines/27836952
CI_PROJECT_DIR=/builds/bahmutov/kitchensink
CI_PROJECT_ID=7930861
CI_PROJECT_NAME=kitchensink
CI_PROJECT_NAMESPACE=bahmutov
CI_PROJECT_PATH=bahmutov/kitchensink
CI_PROJECT_PATH_SLUG=bahmutov-kitchensink
CI_PROJECT_URL=https://gitlab.com/bahmutov/kitchensink
CI_PROJECT_VISIBILITY=public
CI_REGISTRY=registry.gitlab.com
CI_REGISTRY_IMAGE=registry.gitlab.com/bahmutov/kitchensink
CI_REGISTRY_PASSWORD=xxxxxxxxxxxxxxxxxxxx
CI_REGISTRY_USER=gitlab-ci-token
CI_REPOSITORY_URL=https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/bahmutov/kitchensink.git
CI_RUNNER_DESCRIPTION=shared-runners-manager-5.gitlab.com
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_RUNNER_ID=380986
CI_RUNNER_REVISION=081978aa
CI_RUNNER_TAGS=gce, docker
CI_RUNNER_VERSION=11.1.0
CI_SERVER=yes
CI_SERVER_NAME=GitLab
CI_SERVER_REVISION=7996ef5
CI_SERVER_TLS_CA_FILE=/builds/bahmutov/kitchensink.tmp/CI_SERVER_TLS_CA_FILE
CI_SERVER_VERSION=11.2.0-rc2-ee

It seems that the common id is the CI_PIPELINE_ID, CI_PIPELINE_URL and for each job we have separate CI_JOB_ID and CI_JOB_URL
@bahmutov bahmutov self-assigned this Aug 14, 2018
bahmutov added a commit that referenced this issue Aug 14, 2018
@bahmutov
collect GitLab pipeline id and url for #2343
4fbafd2
@bahmutov bahmutov mentioned this issue Aug 14, 2018
Merged
3 tasks
@bahmutov
Copy link
Contributor Author

commit_branch is empty in the database

@bahmutov
Copy link
Contributor Author

Need to print good ci params!

cypress:server:api request to url: POST https://api.cypress.io/runs with params: { body: { ci: { params: [Object], provider: 'gitlab' }

@bahmutov
Copy link
Contributor Author

Also AppVeyor - if new version of commit info returns null for branch, use environment variable. Also add printing

cypress-io/commit-info#5

@bahmutov
Copy link
Contributor Author

ok, so we cannot get branch name on Appveyor using commit-info

commit-info commit-info in folder C:\projects\cypress-example-kitchensink +0ms
  commit-info could not find branch from CI variables +0ms
  commit-info using Git tool to find branch +0ms
  commit-info running git command: git rev-parse --abbrev-ref HEAD +0ms
  commit-info in folder C:\projects\cypress-example-kitchensink +0ms
  commit-info running git command: git show -s --pretty=%B +0ms
  commit-info in folder C:\projects\cypress-example-kitchensink +0ms
  commit-info running git command: git show -s --pretty=%ae +0ms
  commit-info in folder C:\projects\cypress-example-kitchensink +0ms
  commit-info running git command: git show -s --pretty=%an +0ms
  commit-info in folder C:\projects\cypress-example-kitchensink +0ms
  commit-info running git command: git show -s --pretty=%H +0ms
  commit-info in folder C:\projects\cypress-example-kitchensink +0ms
  commit-info running git command: git config --get remote.origin.url +0ms
  commit-info in folder C:\projects\cypress-example-kitchensink +0ms
  commit-info git stdout: Merge f8fc41d3e35203497326365d3a8e264fa207226d into c047d8541d90b0497f663fa7148c1271f32af3c8
 +94ms
  commit-info git stdout: HEAD +0ms
  commit-info git stdout: gleb.bahmutov@gmail.com +31ms
  commit-info git stdout: 83aa3788e9ce7700a90a786f7b10a40da0a35edd +0ms
  commit-info git stdout: Gleb Bahmutov +16ms
  commit-info git stdout: https://github.com/cypress-io/cypress-example-kitchensink.git +0ms

and thus we get empty string, which we keep and don't override with an environment variable

@jennifer-shehane jennifer-shehane added this to the 3.1.1 milestone Aug 15, 2018
brian-mann pushed a commit that referenced this issue Aug 16, 2018
@bahmutov @brian-mann
collect GitLab pipeline id and url for #2343 (#2345)
8f0a31a 
* collect GitLab pipeline id and url for #2343

* collect gitlab CI_COMMIT_REF_NAME for branch or tag name

* no need to send gitlab commit ref name

* print all API request objects without [object Object] shortcuts

* fix GitLab CI detection

* use explicit git and ci commit branch information

* update record test

* add appveyor ci branch name test
@chrisbreiding chrisbreiding removed this from the 3.1.1 milestone Sep 27, 2018
@philiiiiiipp
Copy link

I am still seeing that the branch is empty when using gitlab. Is this now on its way to be resolved? I am a bit confused that this is closed but also marked as "pending release".

@jennifer-shehane
Copy link
Member

The code for this is done, but this has yet to be released. We'll update this issue and reference the changelog when it's released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bahmutov bahmutov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

collect GitLab pipeline id and url for #2343 cypress-io/cypress
4 participants
@philiiiiiipp @chrisbreiding @jennifer-shehane @bahmutov