PLT-7540: Automate build process + code-signing for Windows/Mac #676

MusikPolice · 2017-12-19T20:29:15Z

Before submitting, please confirm you've

read and understood our Contributing Guidelines
completed Mattermost Contributor Agreement

Summary
A Jenkins Pipeline that is capable of building and code signing the desktop application for Windows, MacOS, and Linux. The corresponding Jenkins job is here.

Once merged, the job configuration will be updated so that this runs on any change to any branch in the desktop repository. That way, the existing mattermost-desktop-cut-release can still be used to cut releases. It creates a new branch, which this job will pick up and build.

We may want to think about artifact naming before we start using this to build releases. Currently, this pipeline removes version numbers from all artifacts that it builds, but we might not want to do that, since it could lead to confusion when trying to download the latest artifacts for release testing.

Finally, sorry for all of the commits to this branch. There isn't a great way to test this stuff other than to commit it and run it on Jenkins, since setting up my own Jenkins instance with EC2 spot instances and AMIs and such is a pain. Just look at the final script. The interim commits don't matter, and they'll all be squashed down to one when we merge anyway.

Issue link
https://mattermost.atlassian.net/browse/PLT-7540

…application in a Docker container

…to Jenkinsfile

…d permissions issues

…ckage step

…uild

…ackaging succeeds

…eating directories in root. Wine wants to make stuff too, and keeping track of all the build folders is brittle

jasonblais · 2018-01-02T23:41:46Z

@MusikPolice Is there a rough ETA on when this PR might be ready for review?

Really excited for this PR :)

MusikPolice · 2018-01-03T13:41:54Z

@jasonblais There are three things left to do:

Create a macOS Jenkins slave and move the mac build step to it so that it succeeds (right now, the build always fails because we can't build a macOS release on Jenkins' linux-based slaves)
Add support for signing the Windows build. I almost have this working locally. More details in the linked JIRA ticket.
Add support for signing the macOS build. This should be as simple as setting some environment variables prior to running the existing build step, but I'm still investigating.

As for timeline, if we just do step 1, this is ready to be deployed, and we can use it to replace Circle CI with Jenkins. Steps 2 and 3 are an enhancement that remove the existing manual code signing step that we have to do with every release.

…on macOS

…packaging runs on a single executor

This reverts commit 7908fed.

This reverts commit 863b5b2.

This reverts commit 797b175.

yuya-oc · 2018-02-08T11:43:22Z

build/Jenkinsfile

+            steps {
+                echo 'Building Mattermost Desktop App'
+                sh 'rm -rf release'
+                sh 'npm install'


Please use yarn to respect yarn.lock.

Updated command to use yarn install, @yuya-oc can you give this another review?

yuya-oc · 2018-02-08T11:43:48Z

build/Jenkinsfile

+                echo 'Building Mattermost Desktop App'
+                sh 'rm -rf release'
+                sh 'rm -rf codesign-*'
+                sh 'npm install'


Please use yarn to respect yarn.lock.

yuya-oc · 2018-02-08T11:44:16Z

build/Jenkinsfile

+
+                  echo 'Building Mattermost Desktop App'
+                  sh 'rm -rf release'
+                  sh 'npm install'


Please use yarn to respect yarn.lock.

yuya-oc · 2018-02-08T11:47:48Z

build/Jenkinsfile

+                sh 'mv release/mattermost-desktop-*.tar.gz release/mattermost-desktop-linux-x64.tar.gz'
+                sh 'mv release/tmp-linux-ia32.tar.gz release/mattermost-desktop-linux-ia32.tar.gz'
+                sh 'mv release/mattermost-desktop_*_i386.deb release/mattermost-desktop-linux-i386.deb'
+                sh 'mv release/mattermost-desktop_*_amd64.deb  release/mattermost-desktop-linux-amd64.deb'


We would be able to use template feature of electron-builder to rename artifacts.
https://www.electron.build/configuration/configuration#artifact-file-name-template

#582 has some examples of template.

https://github.com/yuya-oc/desktop/blob/416edabdb18ba29e7c08732e02d22aa9e114b94a/electron-builder.json#L7

https://github.com/yuya-oc/desktop/blob/416edabdb18ba29e7c08732e02d22aa9e114b94a/electron-builder.json#L87

…n numbers in artifacts

…e declared in

jasonblais · 2018-02-14T14:48:50Z

@MusikPolice Is the plan to now only code sign the release branches, similar to what we do for the server?

MusikPolice · 2018-02-14T14:55:45Z

@jasonblais I think so. I'll have to look into how to actually make that secure though. I don't want to just blindly sign any branch that as a "release" prefix on it's name, because any developer can create one.

We may need to add an optional boolean parameter that controls whether or not the job will sign the builds. This parameter would default to false, but when the existing desktop-cut-release job is manually triggered, it would kick off this job with the parameter set to true. That way, only manually started jobs could actually code sign artifacts.

jasonblais · 2018-02-14T14:57:08Z

@MusikPolice Yeah, that would make sense.

It might also be worthwhile to check how the code signing is done for the server.

yuya-oc · 2018-02-19T12:26:06Z

@MusikPolice Just curious, and not required for now. electron-builder has a feature to do code-signing.

https://www.electron.build/code-signing

Are you considering to use it in future? I think that the feature would make scripts simple.

jasonblais · 2018-03-13T18:03:41Z

@AndersonWebStudio I believe you're taking over this PR now. If you have any questions, don't hesitate to let @yuya-oc or Joram know.

yuya-oc

Almost looks good to me. Now v4.1 uses different artifact name, so please check my comments.

yuya-oc · 2018-05-03T12:57:37Z

build/Jenkinsfile

+                sh '''#!/bin/bash
+                  WIN_IA32_SETUP_VERSION=$(awk -F' ' '{print $3}' <<< $(awk -F'-' '{print $2}' <<< $(ls release/win-ia32/Mattermost\\ Setup\\ *-ia32.exe)))
+                  echo "WIN_IA32_SETUP_VERSION is $WIN_IA32_SETUP_VERSION"
+                  mv release/win-ia32/Mattermost\\ Setup\\ $WIN_IA32_SETUP_VERSION-ia32.exe release/mattermost-setup-$WIN_IA32_SETUP_VERSION-win32.exe


release/win-ia32 -> release/squirrel-windows-ia32

yuya-oc · 2018-05-03T12:58:44Z

build/Jenkinsfile

+                  TEMP_VERSION=$(awk -F' ' '{print $3}' <<< $(ls release/win/Mattermost\\ Setup\\ *[^ia32].exe))
+                  WIN_64_SETUP_VERSION=${TEMP_VERSION%%.exe}
+                  echo "WIN_64_SETUP_VERSION is $WIN_64_SETUP_VERSION"
+                  mv release/win/Mattermost\\ Setup\\ $WIN_64_SETUP_VERSION.exe release/mattermost-setup-$WIN_64_SETUP_VERSION-win64.exe


release/win -> release/squirrel-windows

yuya-oc · 2018-05-03T13:00:07Z

build/Jenkinsfile

+                  echo 'Packaging for MacOS'
+                  sh '''#!/bin/bash
+                    npm run package:mac
+                    MACOS_VERSION=$(awk -F'-' '{print $2 }' <<< $(ls release/Mattermost-*-mac.tar.gz))


tar.gz -> zip

yuya-oc · 2018-05-03T13:00:13Z

build/Jenkinsfile

+                    npm run package:mac
+                    MACOS_VERSION=$(awk -F'-' '{print $2 }' <<< $(ls release/Mattermost-*-mac.tar.gz))
+                    echo "MACOS_VERSION is $MACOS_VERSION"
+                    mv release/Mattermost-*-mac.tar.gz release/mattermost-desktop-$MACOS_VERSION-macos.tar.gz


tar.gz -> zip

yuya-oc · 2018-05-03T13:00:20Z

build/Jenkinsfile

+                    echo "MACOS_VERSION is $MACOS_VERSION"
+                    mv release/Mattermost-*-mac.tar.gz release/mattermost-desktop-$MACOS_VERSION-macos.tar.gz
+                  '''
+                  archiveArtifacts artifacts: 'release/mattermost-desktop-*-macos.tar.gz'


tar.gz -> zip

yuya-oc · 2018-05-03T13:07:06Z

build/Jenkinsfile

@@ -0,0 +1,138 @@
+#!/usr/bin/env groovy


Just curious, is shebang needed? In the Jenkinsfile doc, shebang is not used.

https://jenkins.io/doc/book/pipeline/jenkinsfile/

yuya-oc · 2018-05-03T13:09:48Z

build/Jenkinsfile

+            }
+        }
+        stage('Build Windows') {
+            agent any


Just curious, can we use Windows slave in future? We would be able to use Node's native modules.

yuya-oc · 2018-05-03T13:13:24Z

build/Jenkinsfile

+            agent any
+            steps {
+                echo 'Building Mattermost Desktop App'
+                sh '''#!/bin/bash


I think that set -e is needed for each bash script in order to stop the pipeline as script error when an error occurred in a certain command.

yuya-oc · 2018-05-03T13:31:11Z

@AndersonWebStudio After we merge this, how do we cut new release? Is it enough just pushing a tag?

GoldUniform · 2018-05-16T12:20:45Z

We have moved this pipeline to the shared-pipelines repo, this pull request no longer needs to be merged.

Initial commit of a Jenkins Pipeline that is capable of building the …

655fbe6

…application in a Docker container

MusikPolice added the Work In Progress Not yet ready for review label Dec 19, 2017

MusikPolice self-assigned this Dec 19, 2017

Jonathan Fritz added 16 commits December 19, 2017 15:38

Backslashes are invalid characters in the Jenkinsfile format

3fee6e7

Jenkins expects Dockerfile to be in root of repository, not relative …

c2e902b

…to Jenkinsfile

Workaround so NPM doesn't require superuser privileges

2525af8

Switch from npm install to yarn to avoid superuser requirement

e48b6eb

Removed yarn in favour of npm, overrode npm working directory to avoi…

80da8f4

…d permissions issues

switching back to yarn for package management to see if that helps pa…

760c75c

…ckage step

Removing cache override. Suspect it's interfering with electron-builder

49cf468

Increased log level and created required directories before running b…

d6359a0

…uild

Removed unnecessary environment variables

89ad6bf

Moved macOS packaging step to end of build so we can see if Windows p…

aaabaf4

…ackaging succeeds

Removed unnecessary config dump

3e2359b

Deleting the release directory before building

dde84eb

Attempting to switch users to jenkins prior to building instead of cr…

639969c

…eating directories in root. Wine wants to make stuff too, and keeping track of all the build folders is brittle

Using gosu to execute build pipeline steps as jenkins instead of root

9f84f4b

User jenkins doesn't exist. User ubuntu should though

b4638ae

Try overriding /Users/jfritz instead of switching users

0d835fb

jasonblais added this to the v4.0.0 milestone Jan 5, 2018

jasonblais modified the milestones: v4.0.0, v4.1.0 Jan 18, 2018

Jonathan Fritz added 6 commits January 19, 2018 09:12

Merge branch 'master' into PLT-7540

3d543de

Force build/sign ops to run on macOS Jenkins node

1ce5fcc

Explicitly declared agents on all build stages to avoid using docker …

f924857

…on macOS

Adjusted workspace path

771a9be

Collapsing pipeline to two stages so that windows/linux building and …

8d5171c

…packaging runs on a single executor

Verifying that npm exists on the executor

174f5a2

Jonathan Fritz added 3 commits February 7, 2018 16:39

Revert "Got rid of double quotes inside of sh blocks"

65798de

This reverts commit 7908fed.

Revert "Missing single quote"

9874eb2

This reverts commit 863b5b2.

Revert "Preserving version numbers in all artifact file names"

51533d1

This reverts commit 797b175.

yuya-oc suggested changes Feb 8, 2018

View reviewed changes

Jonathan Fritz added 8 commits February 8, 2018 09:18

Changed sh blocks to use single quotes, attempting to preserve versio…

0105ce0

…n numbers in artifacts

Added missing single quote on stage name

44e43aa

Environment variables are only valid within the sh blocks that they'r…

f9b6847

…e declared in

Backslashes must be escaped

e2d36db

Adding debug logging

e22cc90

Less strict archive artifacts glob

27c4b60

Fixed environment variable name typo

e2ee30c

Forgot some 753

5f85626

GoldUniform approved these changes Feb 9, 2018

View reviewed changes

amyblais added the Priority/2 label Mar 19, 2018

jasonblais assigned GoldUniform and unassigned MusikPolice Mar 20, 2018

amyblais removed this from the v4.1.0 milestone Mar 21, 2018

amyblais removed the Priority/2 label Mar 21, 2018

jasonblais mentioned this pull request Mar 26, 2018

BASH-2 Bundle Mac Desktop App as DMG #588

Merged

3 tasks

switch to yarn install

9050e29

yuya-oc suggested changes May 3, 2018

View reviewed changes

GoldUniform closed this May 16, 2018

GoldUniform deleted the PLT-7540 branch May 16, 2018 12:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PLT-7540: Automate build process + code-signing for Windows/Mac #676

PLT-7540: Automate build process + code-signing for Windows/Mac #676

MusikPolice commented Dec 19, 2017 •

edited

Loading

jasonblais commented Jan 2, 2018

MusikPolice commented Jan 3, 2018

yuya-oc Feb 8, 2018

GoldUniform May 1, 2018

yuya-oc Feb 8, 2018

yuya-oc Feb 8, 2018

yuya-oc Feb 8, 2018

jasonblais commented Feb 14, 2018

MusikPolice commented Feb 14, 2018

jasonblais commented Feb 14, 2018

yuya-oc commented Feb 19, 2018

jasonblais commented Mar 13, 2018

yuya-oc left a comment

yuya-oc May 3, 2018

yuya-oc May 3, 2018

yuya-oc May 3, 2018

yuya-oc May 3, 2018

yuya-oc May 3, 2018

yuya-oc May 3, 2018

yuya-oc May 3, 2018

yuya-oc May 3, 2018

yuya-oc commented May 3, 2018

GoldUniform commented May 16, 2018

PLT-7540: Automate build process + code-signing for Windows/Mac #676

PLT-7540: Automate build process + code-signing for Windows/Mac #676

Conversation

MusikPolice commented Dec 19, 2017 • edited Loading

jasonblais commented Jan 2, 2018

MusikPolice commented Jan 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasonblais commented Feb 14, 2018

MusikPolice commented Feb 14, 2018

jasonblais commented Feb 14, 2018

yuya-oc commented Feb 19, 2018

jasonblais commented Mar 13, 2018

yuya-oc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuya-oc commented May 3, 2018

GoldUniform commented May 16, 2018

MusikPolice commented Dec 19, 2017 •

edited

Loading