Not able to detect ciBuildId for Jenkins or run in --parallel #2346

drewbrend · 2018-08-14T20:32:56Z

This issue stemmed out of a conversion with @bahmutov on gitter

Current behavior:

Running my specs in Jenkins I get this error:

12:10:32 cypress-tests  | You passed the --group or --parallel flag but we could not automatically determine or generate a ciBuildId.
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | The --parallel flag you passed was: true
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | In order to use either of these features a ciBuildId must be determined.
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | The ciBuildId is automatically detected if you are running Cypress in any of the these CI providers:
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | - appveyor
12:10:32 cypress-tests  | - bamboo
12:10:32 cypress-tests  | - buildkite
12:10:32 cypress-tests  | - circle
12:10:32 cypress-tests  | - codeship
12:10:32 cypress-tests  | - drone
12:10:32 cypress-tests  | - gitlab
12:10:32 cypress-tests  | - jenkins
12:10:32 cypress-tests  | - semaphore
12:10:32 cypress-tests  | - shippable
12:10:32 cypress-tests  | - travis
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | Because the ciBuildId could not be auto-detected you must pass the --ci-build-id flag manually.
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | https://on.cypress.io/indeterminate-ci-build-id

If I add --ci-build-id ${BUILD_NUMBER} to the run command my test will run and report to the dashboard, however the values for Brach/Author/Commit/Message/CI are not populated in the dashboard, they show "unknown", and I am only running on one executor - not in parallel.

I am running inside a docker container, I think this may be part of the issues I am having and I may have to explicitly expose the environment variables that Cypress uses to my container. However, I do not know what environment variables Cypress needs to figure this all out.

Desired behavior:

Cypress should detect Jenkins and all environment variables automatically

Steps to reproduce:

Run Cypress specs in a docker container in Jenkins with the --parallel flag specified. I am on v1.642.4

Versions

Cypress 3.1.0
Jenkins 1.642.4
Electron 59

The text was updated successfully, but these errors were encountered:

John758364 · 2018-08-14T20:37:35Z

I have the exact same issue.

I'm thinking the issue is more in Docker

When I do

$(npm bin)/cypress run --parallel --record --key ci-key --ci-build-id test2

And have some docker images running

docker run -itd --env BUILD_NO=test2 harbor.bluestembrands.com/fulton/cypressio:3.1.0 tail -f /dev/null

I get no parallelization occuring

drewbrend · 2018-08-14T21:15:14Z

For the comment about the Branch/Author/Commit/Message/CI not being populated in the dashboard, this looks like the reason for author and message: https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/util/ci_provider.coffee#L241

drewbrend · 2018-08-14T21:40:11Z

Some more info.

From https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/util/ci_provider.coffee#L20 it looks like if I pass the JENKINS_URL env var into my docker container it should trigger isJekins to be true. I have done this and removed --ci-build-id ${BUILD_NUMBER} from my run command. It seems like it attempts to work with Jenkins but something goes wrong. Here is the error that's spit out

14:34:15 cypress-tests  | We encountered an unexpected error talking to our servers.
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | There is likely something wrong with the request.
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | The --parallel flag you passed was: true
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | The server's response was:
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | StatusCodeError: 422
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | {
14:34:15 cypress-tests  |   "code": "INDETERMINATE_CI_BUILD_ID",
14:34:15 cypress-tests  |   "message": "CI Build ID could not be generated and was not specified."
14:34:15 cypress-tests  | }

jennifer-shehane · 2018-08-14T22:07:03Z

@drewbrend Yeah - this is the error when we cannot get the CI Build id from the CI provider essentially. It looks like we are looking for these env vars from Jenkins to determine this ourselves: https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/util/ci_provider.coffee#L129

drewbrend · 2018-08-14T22:20:11Z

Ahhh thank you! I passed those into my container via the environment: section of my docker-compose file and it detects Jenkins and runs without me specifying the ci build id. The CI field is populated in the dashboard and the link works.

Some docs around what env vars Cypress uses would be very useful so we know which ones to provide to a docker container.

I still don't have Author, Commit, or Message populated in the dashboard, so that needs work.

And still need to work with someone to get parallelization working.

drewbrend · 2018-08-16T18:13:40Z

I spent a good deal of time on this but have parallelization working across multiple nodes in Jenkins, running in docker containers. Here's what I did for anyone who stumbles on this in the future. This assumes you already have cypress running successfully in a Jenkins job.

Sign up for the cypress dashboard, this is required for parallelization. Follow cypress docs on how to get your project reporting to the dashboard, it's easy, I won't explain it.

In Jenkins I created a cypress dispatcher job, which uses the Jenkins build-flow plugin to determine how many jobs to kick off in parallel and kicks off a cypress runner job for each of those inside of a parallel() block. The dispatcher waits for all of these to complete, then returns pass if they all pass, or a fail if any of those fail.

The runner job does basically what you already have set up to run serially, except it also passes these flags to the cypress run command: --parallel --ci-build-id ${CI_BUILD_ID}. The CI build ID is how cypress knows that all the jobs are supposed to run together, then does a bunch of magic behind the scenes to distribute specs and group results.

The dispatcher job passes the same CI_BUILD_ID parameter to the runner jobs, this needs to be a key that is unique to each run of the dispatcher, so I used the env variable BUILD_NUMBER.

drewbrend · 2018-08-16T18:14:57Z

So now that the parallelization is resolved, the only things outstanding for this ticket are the git details not being reported to the dashboard, which will require changes in Cypress:

-Author
-Commit
-Message

bahmutov · 2018-08-16T18:27:30Z

@drewbrend thanks so much for this! We are putting a fix for commit information here #2345 so that hopefully fills the missing fields

bahmutov · 2018-08-16T18:28:27Z

also @drewbrend can you list all environment variables from a few CI boxes that run in parallel (hiding or removing sensitive keys) - maybe there is something in common that Jenkins already puts for you?

drewbrend · 2018-08-16T18:47:37Z

Sure thing, these is the dump from the printenv command in the cypress runner job, with a liberal amount of stuff changed. Some of these will be params passed into the job, some are jenkins env variables. Any questions just ask.

Box#1:

10:36:53 CYPRESS_browser=electron
10:36:53 PARALLEL_LETTER=E
10:36:53 BUILD_URL=http://ci.OurCompany.com/job/cypress-runner/62/
10:36:53 HUDSON_SERVER_COOKIE=****
10:36:53 SHELL=/bin/bash
10:36:53 JENKINS_API_KEY=****
10:36:53 SSH_CLIENT=23.23.182.73 42456 22
10:36:53 BUILD_TAG=jenkins-cypress-runner-62
10:36:53 CI_BUILD_ID=40
10:36:53 GIT_PREVIOUS_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
10:36:53 ROOT_BUILD_CAUSE=MANUALTRIGGER
10:36:53 WORKSPACE=/var/lib/jenkins/workspace/cypress-runner
10:36:53 JOB_URL=http://ci.OurCompany.com/job/cypress-runner/
10:36:53 branch=QA-311
10:36:53 RUN_CHANGES_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/62/display/redirect?page=changes
10:36:53 USER=jenkins
10:36:53 CYPRESS_baseUrl=https://OurCompany-blue.OurDomain.net
10:36:53 GITHUB_STATUS_OAUTH=****
10:36:53 BUILD_CAUSE_UPSTREAMTRIGGER=true
10:36:53 SSH_AUTH_SOCK=/tmp/jenkins4683669874741720002.jnr
10:36:53 OurCompany_BUILD_TOOLS_PATH=/opt/OurCompany/build-tools
10:36:53 GIT_AUTHOR_NAME=FirstName LastName
10:36:53 CYPRESS_API=prod
10:36:53 browser=electron
10:36:53 spec=/**/*.js
10:36:53 GIT_COMMIT=b34641f414894587b27e528dba8202c3cfb4f9b2
10:36:53 JENKINS_HOME=/var/lib/jenkins
10:36:53 MAIL=/var/mail/jenkins
10:36:53 PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
10:36:53 RUN_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/62/display/redirect
10:36:53 GITHUB_OAUTH_TOKEN=****
10:36:53 CYPRESS_spec=/**/*.js
10:36:53 PWD=/var/lib/jenkins/workspace/cypress-runner
10:36:53 HUDSON_URL=http://ci.OurCompany.com/
10:36:53 LANG=en_US.UTF-8
10:36:53 JOB_NAME=cypress-runner
10:36:53 BUILD_DISPLAY_NAME=#62 - origin/QA-311 - E
10:36:53 BUILD_CAUSE=UPSTREAMTRIGGER
10:36:53 BUILD_ID=62
10:36:53 JENKINS_URL=http://ci.OurCompany.com/
10:36:53 OurCompany_DOCKER_REPO=****.dkr.ecr.us-east-1.amazonaws.com
10:36:53 GIT_PREVIOUS_SUCCESSFUL_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
10:36:53 HOME=/var/lib/jenkins
10:36:53 SHLVL=2
10:36:53 PG_CYPRESS_BRANCH=QA-311
10:36:53 GIT_BRANCH=origin/QA-311
10:36:53 EXECUTOR_NUMBER=0
10:36:53 JENKINS_SERVER_COOKIE=**********
10:36:53 GIT_URL=git@github.com:OurCompany/OurProduct
10:36:53 NODE_LABELS=scaled-dev scaled-slave-2
10:36:53 LOGNAME=jenkins
10:36:53 SSH_CONNECTION=23.23.182.73 42456 172.31.64.52 22
10:36:53 HUDSON_HOME=/var/lib/jenkins
10:36:53 NODE_NAME=scaled-slave-2
10:36:53 BUILD_NUMBER=62
10:36:53 JOB_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/display/redirect
10:36:53 ROOT_BUILD_CAUSE_MANUALTRIGGER=true
10:36:53 HUDSON_COOKIE=****
10:36:53 GIT_AUTHOR_EMAIL=FirstName.LastName@OurCompany.com
10:36:53 NEW_RELIC_INSIGHTS_KEY=****

Box#2:

10:36:48 CYPRESS_browser=electron
10:36:48 PARALLEL_LETTER=A
10:36:48 BUILD_URL=http://ci.OurCompany.com/job/cypress-runner/61/
10:36:48 HUDSON_SERVER_COOKIE=****
10:36:48 SHELL=/bin/bash
10:36:48 JENKINS_API_KEY=****
10:36:48 SSH_CLIENT=23.23.182.73 37475 22
10:36:48 BUILD_TAG=jenkins-cypress-runner-61
10:36:48 CI_BUILD_ID=40
10:36:48 GIT_PREVIOUS_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
10:36:48 ROOT_BUILD_CAUSE=MANUALTRIGGER
10:36:48 WORKSPACE=/var/lib/jenkins/workspace/cypress-runner
10:36:48 JOB_URL=http://ci.OurCompany.com/job/cypress-runner/
10:36:48 branch=QA-311
10:36:48 RUN_CHANGES_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/61/display/redirect?page=changes
10:36:48 USER=jenkins
10:36:48 CYPRESS_baseUrl=https://OurCompany-blue.OurDomain.net
10:36:48 GITHUB_STATUS_OAUTH=****
10:36:48 BUILD_CAUSE_UPSTREAMTRIGGER=true
10:36:48 SSH_AUTH_SOCK=/tmp/jenkins3675280086874796853.jnr
10:36:48 OurCompany_BUILD_TOOLS_PATH=/opt/OurCompany/build-tools
10:36:48 GIT_AUTHOR_NAME=FirstName LastName
10:36:48 CYPRESS_API=prod
10:36:48 browser=electron
10:36:48 spec=/**/*.js
10:36:48 GIT_COMMIT=b34641f414894587b27e528dba8202c3cfb4f9b2
10:36:48 JENKINS_HOME=/var/lib/jenkins
10:36:48 MAIL=/var/mail/jenkins
10:36:48 PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
10:36:48 RUN_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/61/display/redirect
10:36:48 GITHUB_OAUTH_TOKEN=****
10:36:48 CYPRESS_spec=/**/*.js
10:36:48 PWD=/var/lib/jenkins/workspace/cypress-runner
10:36:48 HUDSON_URL=http://ci.OurCompany.com/
10:36:48 LANG=en_US.UTF-8
10:36:48 JOB_NAME=cypress-runner
10:36:48 BUILD_DISPLAY_NAME=#61 - origin/QA-311 - A
10:36:48 BUILD_CAUSE=UPSTREAMTRIGGER
10:36:48 BUILD_ID=61
10:36:48 JENKINS_URL=http://ci.OurCompany.com/
10:36:48 OurCompany_DOCKER_REPO=****.dkr.ecr.us-east-1.amazonaws.com
10:36:48 GIT_PREVIOUS_SUCCESSFUL_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
10:36:48 HOME=/var/lib/jenkins
10:36:48 SHLVL=2
10:36:48 PG_CYPRESS_BRANCH=QA-311
10:36:48 GIT_BRANCH=origin/QA-311
10:36:48 EXECUTOR_NUMBER=0
10:36:48 JENKINS_SERVER_COOKIE=****
10:36:48 GIT_URL=git@github.com:OurCompany/OurProduct
10:36:48 NODE_LABELS=scaled-dev scaled-slave-8
10:36:48 LOGNAME=jenkins
10:36:48 SSH_CONNECTION=23.23.182.73 37475 172.31.65.176 22
10:36:48 HUDSON_HOME=/var/lib/jenkins
10:36:48 NODE_NAME=scaled-slave-8
10:36:48 BUILD_NUMBER=61
10:36:48 JOB_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/display/redirect
10:36:48 ROOT_BUILD_CAUSE_MANUALTRIGGER=true
10:36:48 HUDSON_COOKIE=****
10:36:48 GIT_AUTHOR_EMAIL=FirstName.LastName@OurCompany.com
10:36:48 NEW_RELIC_INSIGHTS_KEY=****

bahmutov · 2018-08-29T21:15:00Z

@drewbrend to get the git info, do you think this would work

authorEmail: GIT_AUTHOR_EMAIL
authorName: GIT_AUTHOR_NAME
sha: GIT_COMMIT
branch: GIT_BRANCH

The only one missing is message

from these vars

GIT_AUTHOR_EMAIL=FirstName.LastName@OurCompany.com
GIT_AUTHOR_NAME=FirstName LastName
GIT_BRANCH=origin/QA-311
GIT_COMMIT=b34641f414894587b27e528dba8202c3cfb4f9b2
GIT_PREVIOUS_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
GIT_PREVIOUS_SUCCESSFUL_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
GIT_URL=git@github.com:OurCompany/OurProduct

bahmutov · 2018-08-29T21:18:37Z

I see how CI_BUILD_ID stays the same 40 while BUILD_ID=61 and BUILD_ID=62 is different, and we can link to the individual builds using BUILD_URL=http://ci.OurCompany.com/job/cypress-runner/62/ but not tothe common CI build id 40.

drewbrend · 2018-08-29T21:26:43Z

@bahmutov The GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL env vars are created during the run of my cypress-runner job to try to get around the issues I was having when I created this ticket. GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL did not get picked up by cypress like I thought they would, so they are currently not doing anything. CI_BUILD_ID is passed from the cypress-dispatcher job to the cypress-runner job and then passed into the cypress run command via the --ci-build-id flag. So all of that is manual.

If you can do something similar to what I have below in cypress before you pass the infomation to the dashboard, I think that could work for the Git info. I imagine you could do the same to get message as well. GIT_COMMIT and GIT_BRANCH exist already, so those are an easy win.

This shell is in my cypress-runner job to create those variables

export GIT_AUTHOR_NAME=$(git --no-pager show -s --format='%an' $GIT_COMMIT)
export GIT_AUTHOR_EMAIL=$(git --no-pager show -s --format='%ae' $GIT_COMMIT)

if [ -z "$CI_BUILD_ID" ]; then
  export CI_BUILD_ID=$BUILD_NUMBER
fi

bahmutov · 2018-08-29T21:39:43Z

hmm, we do use https://github.com/cypress-io/commit-info to fetch Git things https://github.com/cypress-io/commit-info/blob/418002428b653ed75ccd82bff6966529c9009e24/src/git-api.js#L9 but we do NOT pass GIT_COMMIT, maybe that's why it fails?!!!

would git --no-pager show -s --format='%an' not work inside the job?

drewbrend · 2018-08-29T21:52:02Z

If you're not passing GIT_COMMIT, that sounds like that could be the issue.

I am running git --no-pager show -s --format='%an' inside the cypress-runner job. sorry, I will update my previous message - it said it was in the cypress-dispatcher job

bahmutov · 2018-08-29T22:06:07Z

could you add just git --no-pager show -s --format='%an' command inside each job just to see if it works without commit id, if this is the issue, we can update commit-info to be explicit if GIT_COMMIT is present

drewbrend · 2018-08-30T15:16:05Z

@bahmutov I changed export GIT_AUTHOR_NAME=$(git --no-pager show -s --format='%an' $GIT_COMMIT) to export GIT_AUTHOR_NAME=$(git --no-pager show -s --format='%an') and it still works, the GIT_AUTHOR_NAME env var is set.

bahmutov · 2018-08-30T17:32:22Z

Hmm, ok @drewbrend if you can have another env variable DEBUG=commit-info to see output from each individual git command from https://github.com/cypress-io/commit-info module. Then we could see what was going on when it tries to fetch commit information.

drewbrend · 2018-08-30T17:44:28Z

@bahmutov I set that, and the printenv command shows DEBUG=commit-info in the list of env vars, however I don't see any additional debug output. Just the usual cypress output about the tests that have ran.

bahmutov · 2018-08-30T17:55:06Z

@drewbrend ughh, we are hiding DEBUG output from stderr, that's unfortunate. I will need to think of the way to debug this

drewbrend · 2018-08-30T17:58:20Z

@bahmutov I'm happy to help, let me know what you need. I'll have time this evening to run a couple tests if you think of anything to try.

bahmutov · 2018-08-30T18:31:47Z

Great @drewbrend so one thing you might do is to look at the code in https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/util/ci_provider.coffee and enable debug logs for it to see the commit information in

cypress/packages/server/lib/util/ci_provider.coffee

Line 310 in 3b804cd

commitDefaults = (existingInfo) ->

which I believe should be printed if DEBUG=cypress:server is set

commitDefaults = (existingInfo) ->
  commitParamsObj = commitParams() or {}
  debug("commit params object")
  debug(commitParamsObj)

  ## based on the existingInfo properties
  ## merge in the commitParams if null or undefined
  ## defaulting back to null if all fails
  _.transform existingInfo, (memo, value, key) ->
    memo[key] = _.defaultTo(value ? commitParamsObj[key], null)

and here is where it is called

cypress/packages/server/lib/modes/record.coffee

Line 266 in 8f0a31a

commit = getCommitFromGitOrCi(git)

which is logged using DEBUG=cypress:server:record. So we can enable both logs using DEBUG=cypress:server,cypress:server:record

if we can see what each part does we could determine where things go wrong ...

drewbrend · 2018-08-31T13:10:10Z

@bahmutov I have DEBUG=cypress:server,cypress:server:record set and am not seeing any additional output to the console in Jenkins.

bahmutov · 2018-08-31T13:22:10Z

Hmm, we might have added more debug log output after 3.1.0 to actually debug this problem better

drewbrend · 2018-08-31T14:37:51Z

Okay no problem. It isn't a blocker for me, but would be nice to have. We have parallel working and with our setup we can pretty easily figure out the author and commit from the branch name if we need to.

bahmutov · 2018-08-31T15:03:46Z

Great, yeah, if you got it working - that is nice, and if this is not a blocker, than perfect, and next release hopefully will have all the information either fixed or able to debug. I have installed local Jenkins in Docker setup https://hub.docker.com/r/jenkinsci/blueocean/ --- and could not get Node plugin to actually add Node there :( https://medium.com/wearetheledger/jenkins-your-butler-for-continuous-integration-blue-ocean-docker-image-solution-to-node-not-ebf9ebfbfa78 Plus we really want a public Jenkins instance that can show to everyone the build results and pipeline for our cypress-example-kitchensink, so Jenkins CI demo is still work in progress.

jennifer-shehane · 2019-10-17T20:26:19Z

Closing as resolved. Please comment if you are still having this issue and we will consider reopening.

jennifer-shehane added the stage: needs investigating Someone from Cypress needs to look at this label Aug 14, 2018

jennifer-shehane closed this as completed Oct 17, 2019

jennifer-shehane removed the stage: needs investigating Someone from Cypress needs to look at this label Jan 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not able to detect ciBuildId for Jenkins or run in --parallel #2346

Not able to detect ciBuildId for Jenkins or run in --parallel #2346

drewbrend commented Aug 14, 2018 •

edited by jennifer-shehane

Loading

John758364 commented Aug 14, 2018

drewbrend commented Aug 14, 2018

drewbrend commented Aug 14, 2018

jennifer-shehane commented Aug 14, 2018

drewbrend commented Aug 14, 2018

drewbrend commented Aug 16, 2018

drewbrend commented Aug 16, 2018

bahmutov commented Aug 16, 2018

bahmutov commented Aug 16, 2018

drewbrend commented Aug 16, 2018 •

edited

Loading

bahmutov commented Aug 29, 2018

bahmutov commented Aug 29, 2018

drewbrend commented Aug 29, 2018 •

edited

Loading

bahmutov commented Aug 29, 2018

drewbrend commented Aug 29, 2018

bahmutov commented Aug 29, 2018

drewbrend commented Aug 30, 2018

bahmutov commented Aug 30, 2018

drewbrend commented Aug 30, 2018

bahmutov commented Aug 30, 2018

drewbrend commented Aug 30, 2018

bahmutov commented Aug 30, 2018

drewbrend commented Aug 31, 2018

bahmutov commented Aug 31, 2018

drewbrend commented Aug 31, 2018

bahmutov commented Aug 31, 2018

jennifer-shehane commented Oct 17, 2019

Not able to detect ciBuildId for Jenkins or run in --parallel #2346

Not able to detect ciBuildId for Jenkins or run in --parallel #2346

Comments

drewbrend commented Aug 14, 2018 • edited by jennifer-shehane Loading

Current behavior:

Desired behavior:

Steps to reproduce:

Versions

John758364 commented Aug 14, 2018

drewbrend commented Aug 14, 2018

drewbrend commented Aug 14, 2018

jennifer-shehane commented Aug 14, 2018

drewbrend commented Aug 14, 2018

drewbrend commented Aug 16, 2018

drewbrend commented Aug 16, 2018

bahmutov commented Aug 16, 2018

bahmutov commented Aug 16, 2018

drewbrend commented Aug 16, 2018 • edited Loading

bahmutov commented Aug 29, 2018

bahmutov commented Aug 29, 2018

drewbrend commented Aug 29, 2018 • edited Loading

bahmutov commented Aug 29, 2018

drewbrend commented Aug 29, 2018

bahmutov commented Aug 29, 2018

drewbrend commented Aug 30, 2018

bahmutov commented Aug 30, 2018

drewbrend commented Aug 30, 2018

bahmutov commented Aug 30, 2018

drewbrend commented Aug 30, 2018

bahmutov commented Aug 30, 2018

drewbrend commented Aug 31, 2018

bahmutov commented Aug 31, 2018

drewbrend commented Aug 31, 2018

bahmutov commented Aug 31, 2018

jennifer-shehane commented Oct 17, 2019

drewbrend commented Aug 14, 2018 •

edited by jennifer-shehane

Loading

drewbrend commented Aug 16, 2018 •

edited

Loading

drewbrend commented Aug 29, 2018 •

edited

Loading