documentation: Add CI/CD example

[tuananh]

Is there an example of using skaffold for CI/CD?

How can use skaffold to target different k8s namespace deployment eg: dev for dev and prod for production?

[jstrachan]

We're working on an OSS project called Jenkins X which is focussed on automating CI/CD for developers on kubernetes, automating the setup of a development environment with Jenkins/Nexus then Staging and Production etc. Here's a little demo to show you how previews and promotion work: etc.

So we're hoping ASAP to use skaffold as the engine for creating docker builds (local docker daemon or GCB etc) and hopefully updating the helm chart - then using skaffold in development (pre-commit) etc. I'll comment on this issue when I've a demo of Jenkins X using skaffold. I'm hopeful as soon as the next release of skaffold is out with the new envTemplate feature we'll switch Jenkins X over to being based on skaffold

[bhack]

It could be nice to see how the flow could be integrated with k8s spinnaker.
@jstrachan Any particular plan of Jenkins X with Spinnaker cause it already integrates with Jenkins.

[bhack]

Here we have investigated a little bit also the dev tools like this one from a CD point of view: argoproj/argo-cd#92

Instead is less clear to me how dev tools could interact with the Gitops approach like kubeflow/kubeflow#971. /cc @jlewi

[bhack]

/cc @jessesuen Any feedback from argo-cd? What is your point of view about dev tools like Skaffold and argo-cd and argo workflows management?

[bhack]

@jstrachan Any example for Jenkins X?

[jstrachan]

Yeah this example shows crating a new project & setting up CI/CD so that any merge to the master branch creates a new semantic version, artifacts, docker image, helm chart and then promoted through the environments via GitOps
https://jenkins-x.io/demos/create_spring/ which uses skaffold for all docker image generation. (It’d be nice to use skaffold for the chart version tweaking too).

Then the last half of this talk https://jenkins-x.io/demos/devoxx-uk-2018/ shows how to create pull requests and preview environments - again using skaffold for docker image creation - then doing incremental development in a DevPod so that any code change rebuilds the image and redeploys the helm chart with skaffold

[bhack]

@jstrachan Thank you for the update. So actually it will not support Skaffold for other deployers like i.e. Kustomize?

[jstrachan]

currently not but we should be able to modify the build packs to support kustomize if there is demand for it

[jessesuen]

/cc @jessesuen Any feedback from argo-cd? What is your point of view about dev tools like Skaffold and argo-cd and argo workflows management?

@bhack sorry for the late response. i'm pruning bugs and noticed a one of our bugs got linked here. I think Skaffold is very compatible with ArgoCD in that Skaffold handles the early container image development very well, and ArgoCD handles more of the finalized deployment of images very well, specifically from git.

One way I look at it is: Skaffold helps you rapidly develop your application to a point where you are comfortable releasing & giving your image a semvers. Once your image released, you will need configuration management (e.g. to integrate your app with other microservices with different environments). For this you use would use tools like kustomize, ksonnet which excel at this. Once you have configuration, you'll want to store it somewhere, which is naturally git. If you are storing manifests in git and treating it as your desired state, you can use tools like ArgoCD / Weave Flux to deploy those manifests to your cluster.

ArgoCD is agnostic to CI systems, as well as the tooling and everything that comes before storing your manifests in git. In terms of generating manifests from configs, we support helm charts, ksonnet apps, kustomize apps (in our next release), or just plain YAMLs.

[ahmetb]

Check out the https://github.com/GoogleCloudPlatform/microservices-demo/ app by Google. It uses skaffold to deploy a cloud-native app composed of 10+ services.

[anarcher]

Once you have configuration, you'll want to store it somewhere, which is naturally git. If you are storing manifests in git and treating it as your desired state, you can use tools like ArgoCD / Weave Flux to deploy those manifests to your cluster.

But, If I'm right, Skaffold modifies manifests during deployer works. It means that Skaffold doesn't keep image tag information in manifests in file and git. ArgoCD or Flux can't get the image tag information (modified in POD ) from Skaffold's modified. Is it right?

My small idea is to make ArgoCD deployer in Skaffold. Maybe this deploy do just customize manifests and then commit , push with git.. :-) But, Current Skaffold plugin model is just for building if I'm right [1].

Any ideas? I like both of Skaffold and ArgoCD. I would like to use both of :-)

[balopat]

@anarcher good news is, that we do have an issue open for that and we are planning to make this work: #2161

[iMerica]

Has anyone successfully setup Skaffold within Gitlab?

[iMerica]

Update

After many hours of trying, this is how I successfully configured Skaffold to build inside of Gitlab using DIND + the official Skaffold docker image:

services:
  - docker:dind

stages:
  - build

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""
  DOCKER_HOST: tcp://docker:2375
build:
  image:
    name: gcr.io/k8s-skaffold/skaffold:latest
  stage: build
  before_script:
    - docker info
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - skaffold run
  only:
    - master

[tvvignesh]

Here is a gist I made doing exactly the same - it works: https://gist.github.com/tvvignesh/e82bc33469320aa198c348336b4890fb

There is only one problem I am facing in this is since it is running as dind, I don't see any caching happening and all images are being pulled every time. My build takes 5 minutes every time since it has to download the base images and install all layers every time.

Wondering how to speed it up.

UPDATE: Was able to solve the speed issue by creating a dind service like mentioned in this link: https://gitlab.com/gitlab-org/gitlab-foss/issues/17861#note_87008532

Gist for the same here: https://gist.github.com/tvvignesh/25850fe4e0ce3bbdfac479f5ff020168

Had to then change the gitlab-ci file to use the dind service URL.

[adrien-barret]

@tvvignesh did you cover up a way to use skaffold+argoCD to get the pipeline view ?

[tvvignesh]

@Hkyx Nope. I don't use argoCD. Hence, did not try it out.

[tstromberg]

Anyone up for writing an example document for this?

[tvvignesh]

@tstromberg I can help if you are okay with me taking Gitlab CI as an example for explaining the docs.

[Djiit]

Hey, I'll be happy to help review any document on this. Using Skaffold on Gitlab CI right now and looking for optimizations.

[nkubala]

@tvvignesh yeah that sounds great

[tvvignesh]

@nkubala Sorry was held up with few other things. I will make a gist and share it with you this week. If that is good enough, you can use it for the docs.

[tvvignesh]

Hi. Have written a small doc about this trying to elaborate each step of the process. You can find it here: https://docs.google.com/document/d/1JLsgL2ooHNL_n2bufFVpnbMIkhJRtlf5F3V6Es98O8Q/edit?usp=sharing

Please feel free to use it if it makes sense. Haven't done much of formatting - just wanted to focus on the content.

Also, do let me know if you need edit access to the doc.

CC: @nkubala @tstromberg @tejal29 @briandealwis @dgageot

[briandealwis]

Thanks @tvvignesh, we'll give it a look soon!

[tejal29]

Thanks @tvvignesh for sharing your flow.

Were you able to use kaniko as a builder like you mentioned on slack?

[tvvignesh]

Thanks @tvvignesh for sharing your flow.

Were you able to use kaniko as a builder like you mentioned on slack?

@tejal29 The main issue with Kaniko is the privileges it needs on a cluster with restricted PSP. I have switched away from Docker to using Buildah - there are some bugs with buildah but they will be fixed soon. Wanted to use skaffold with Buildah and Helm in my CI pipeline. Have currently parked that exercise since I was going nuts on what to go for 😅 I will do it some other time. Currently, I have switched to tilt for development, and if all is well, I will use skaffold, Gitlab CI and buildah for prod.

There are just too many tools out there. I wish I could just have one tool which does all. Skaffold was one, but then it doesn't have a dashboard/UI like tilt, I cannot run my custom shell scripts with it, have to write my own custom builder for buildah and do not have a docker-compose like experience. I am still aligning to using Skaffold for prod but the devex needs to be a bit more better when working with multiple services - I end up switching a lot of terminals back and forth with skaffold but for CI/CD I don't think I have found anything better than skaffold and Gitlab yet and its awesome (just if it supports buildah out of the box - will have to write script for that when I find time).

I am yet to try out Buildkit. Will see that as well when I find time.

[tejal29]

Thanks @tvvignesh for this honest feedback. We will discuss this internally and update you.
Our goal is to provide one tool that developers use from dev to production and you bring up interesting points.

If not kaniko, skaffold should be able to integrate smoothly with buildah. I opened an investigation here #4905

Again thanks and stay tuned to hear more updates on this.