Codefresh installer of Kubernetes YAML onto the cluster
Version 1.x.x is is about to be released soon, read more about migration from older version here
Meanwhile 1.x.x is to release and makred as pre-release we will maintain 2 branches:
master
- the previous version (version < 1.0.0
)- we will keep maintaing if (bugs, security issues) - this version will be intalled when installing
venona
on MacOS using brew codefresh/venona:latest
will refer to this branch
- we will keep maintaing if (bugs, security issues) - this version will be intalled when installing
release-1.0
it the new release, which will be used when running Codefresh CLI to install the agent We highly suggest to use Codefresh official CLI to install the agent:
codefresh runner init
The last command will:
- Install the agent on the namespace
codefresh
(as you choose) - Install the runtime on the same namespace
- Attach the runtime to the agent
- Register cluster on codefresh platform
- Create and run demo pipeline
It is still possible, for advanced users to install all manually, for example: One process of Venona can manage multiple runtime environments NOTE: Please make sure that the process where Venona is installed there is a network connection to the clusters where the runtimes will be installed
# 1. Create namespace for the agent:
kubectl create namespace codefresh-agent
# 2. Install the agent on the namespace ( give your agent a unique):
# Print a token that the Venona process will be using.
codefresh create agent $NAME
codefresh install agent --token $TOKEN --kube-namespace codefresh-agent
# 3. Create namespace for the first runtime:
kubectl create namespace codefresh-runtime-1
# 4. Install the first runtime on the namespace
# 5. the runtime name is printed
codefresh install runtime --kube-namespace codefresh-runtime-1
# 6. Attach the first runtime to agent:
codefresh attach runtime --agent-name $AGENT_NAME --agent-kube-namespace codefresh-agent --runtime-name $RUNTIME_NAME --kube-namespace codefresh-runtime-1
# 7. Restart the venona pod in namespace `codefresh-agent`
kubectl delete pods $VENONA_POD
# 8. Create namespace for the second runtime
kubectl create namespace codefresh-runtime-2
# 9. Install the second runtime on the namespace
codefresh install runtime --kube-namespace codefresh-runtime-2
# 10. Attach the second runtime to agent and restart the Venoa pod automatically
codefresh attach runtime --agent-name $AGENT_NAME --agent-kube-namespace codefresh-agent --runtime-name $RUNTIME_NAME --runtime-kube-namespace codefresh-runtime-1 --restart-agent
Migrating from Venona < 1.x.x
to > 1.x.x
is not done automatically, please run the follwing
codefresh runner upgrade
- Kubernetes - Used to create resource in your K8S cluster
- Kube Version > 1.10:
- Instuction to install on cluster version < 1.10
- Disk size 50GB per node
- Kube Version > 1.10:
- Codefresh - Used to create resource in Codefresh
- Authenticated context exist under
$HOME/.cfconfig
or authenticate with Codefesh CLI
- Authenticated context exist under
- Download venona's binary
- With homebrew:
brew tap codefresh-io/venona
brew install venona
- With homebrew:
- Make sure the
PersistentLocalVolumes
feature gate is turned on - Venona's agent is trying to load avaliables apis using api
/openapi/v2
endpoint Add this endpoint to ClusterRolesystem:discovery
underrules[0].nonResourceURLs
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:discovery
rules:
- nonResourceURLs:
- ...other_resources
- /openapi
- /openapi/*
verbs:
- get
codefresh runner init --values <values.yaml> [parameters]
the values from values.yaml are applied to the templates in pkg/templates/kubernetes
See BuildValues() func in store.go for the format
Example with explanation is in values-example.yaml
- Make sure your user has
Kubernetes Engine Cluster Admin
role in google console - Bind your user with cluster-admin kubernetes clusterrole
kubectl create clusterrolebinding NAME --clusterrole cluster-admin --user YOUR_USER
Prerequisite: GKE custer with local SSD
Install venona for using GKE Local SSD:
codefresh install runtime [options] \
--set-value=Storage.LocalVolumeParentDir=/mnt/disks/ssd0/codefresh-volumes
--kube-node-selector=cloud.google.com/gke-local-ssd=true
Prerequisite: volume provisioner (dind-volume-provisioner) should have permissions to create/delete/get of google disks There are 3 options to provide cloud credentials on GCE:
- run venona dind-volume-provisioniner on node with iam role which is allowed to create/delete/get of google disks
- create Google Service Account with ComputeEngine.StorageAdmin, download its key and pass it to venona installed with
--set-file=Storage.GooogleServiceAccount=/path/to/google-service-account.json
- use Google Workload Identity to assign iam role to
volume-provisioner-venona
service account
Note: Builds will be running in single availability zone, so you must to specify AvailabilityZone params
Install venona for using GKE Disks:
codefresh install runtime [options] \
--set-value=Storage.Backend=gcedisk \
--set-value=Storage.AvailabilityZone=us-central1-a \
--kube-node-selector=failure-domain.beta.kubernetes.io/zone=us-central1-a \
[--set-file=Storage.GoogleServiceAccount=/path/to/google-service-account.json]
Prerequisite: volume provisioner (dind-volume-provisioner) should have permissions to create/delete/get of aws ebs Minimal iam policy for dind-volume-provisioner:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteSnapshot",
"ec2:DeleteTags",
"ec2:DeleteVolume",
"ec2:DescribeInstances",
"ec2:DescribeSnapshots",
"ec2:DescribeTags",
"ec2:DescribeVolumes",
"ec2:DetachVolume"
],
"Resource": "*"
}
]
}
There are 3 options to provide cloud credentials on AWS:
-
run venona dind-volume-provisioniner on node with the iam role - use
--set-value Storage.VolumeProvisioner.NodeSelector=node-label=value
option -
create AWS IAM User, assign it the permissions above and suppy aws credentials to venona installer
--set-value=Storage.AwsAccessKeyId=ABCDF --set-value=Storage.AwsSecretAccessKey=ZYXWV
-
use Aws Identity for Service Account to assign iam role to
volume-provisioner-venona
service account
Notes:
-
Builds will be running in single availability zone, so you must specify AvailabilityZone parameter
--set-value=Storage.AvailabilityZone=<aws-az>
and build-node-selector--build-node-selector=failure-domain.beta.kubernetes.io/zone=<aws-az>
in case of multizone cluster -
We support both in-tree ebs (
--set-value=Storage.Backend=ebs
) volumes and ebs-csi(https://github.com/kubernetes-sigs/aws-ebs-csi-driver) (--set-value=Storage.Backend=ebs-csi
)
Install Command to run pipelines on ebs volumes
codefresh install runtime [options] \
--set-value=Storage.Backend=ebs \
--set-value=Storage.AvailabilityZone=us-east-1d \
--kube-node-selector=failure-domain.beta.kubernetes.io/zone=us-east-1d \
[--set-value Storage.VolumeProvisioner.NodeSelector=kubernetes.io/role=master] \
[--set-value Storage.AwsAccessKeyId=ABCDF --set-value Storage.AwsSecretAccessKey=ZYXWV]\
[--set-value=Storage.Encrypted=true] \
[--set-value=Storage.KmsKeyId=<key id>]
Prerequisite: volume provisioner (dind-volume-provisioner) should have permissions to create/delete/get Auzure Disks
Minimal iam Role for dind-volume-provisioner - dind-volume-provisioner-role.json:
{
"Name": "CodefreshDindVolumeProvisioner",
"Description": "Perform create/delete/get disks",
"IsCustom": true,
"Actions": [
"Microsoft.Compute/disks/read",
"Microsoft.Compute/disks/write",
"Microsoft.Compute/disks/delete"
],
"AssignableScopes": ["/subscriptions/<your-subsripton_id>"]
}
If you use AKS with managed identities for node group, you can run the script below to assign CodefreshDindVolumeProvisioner role to aks node identity:
export ROLE_DEFINITIN_FILE=dind-volume-provisioner-role.json
export SUBSCRIPTION_ID=$(az account show --query "id" | xargs echo )
export RESOURCE_GROUP=codefresh-rt1
export AKS_NAME=codefresh-rt1
export LOCATION=$(az aks show -g $RESOURCE_GROUP -n $AKS_NAME --query location | xargs echo)
export NODES_RESOURCE_GROUP=MC_${RESOURCE_GROUP}_${AKS_NAME}_${LOCATION}
export NODE_SERVICE_PRINCIPAL=$(az aks show -g $RESOURCE_GROUP -n $AKS_NAME --query identityProfile.kubeletidentity.objectId | xargs echo)
az role definition create --role-definition @${ROLE_DEFINITIN_FILE}
az role assignment create --assignee $NODE_SERVICE_PRINCIPAL --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$NODES_RESOURCE_GROUP --role CodefreshDindVolumeProvisioner
Now create runner with --set-value Storage.Backend=azuredisk --set Storage.VolumeProvisioner.MountAzureJson=true
:
codefresh runner init --set-value Storage.Backend=azuredisk --set Storage.VolumeProvisioner.MountAzureJson=true
Or using runner-values.yaml file like below:
# CodefreshHost: https://g.codefresh.io
# Token: ******
# Namespace: default
# Context: codefresh-rt1
# RuntimeInCluster: true
Storage:
Backend: azuredisk
VolumeProvisioner:
MountAzureJson: true
codefresh runner init --values runner-values.yaml
Installation of Venona on Kubernetes cluster installing 2 groups of objects, Each one has own RBAC needs and therefore, created roles(and cluster-roles) The resource descriptors are avaliable here List of the resources that will be created
- Agent (grouped by
/.*.venona.yaml/
)service-account.re.yaml
- The service account that the Venona pod will use to create the resource on the runtime namespace(the resoucre installed on the runtime namespace)role.re.yaml
- Allow to GET, CREATE and DELETE pods and persistentvolumeclaimsrole-binding.re.yaml
- The agent is spinning up pods and pvc, this biniding bindsrole.venona.yaml
toservice-account.venona.yaml
cluster-role-binding.venona.yaml
- The agent discovering K8S apis by calling toopenapi/v2
, this ClusterRoleBinding binds bootstraped ClusterRole by Kubernetessystem:discovery
toservice-account.venona.yaml
. This role has only permissions to make a GET calls to non resources urls
- Runtime-environment (grouped by
/.*.re.yaml/
) Kubernetes controller that spins up all required resources to provide a good caching expirience during pipeline executionservice-account.dind-volume-provisioner.re.yaml
- The service account that the controller will usecluster-role.dind-volume-provisioner.re.yaml
Defines all the permission needed for the controller to operate correctlycluster-role-binding.dind-volume-provisioner.yaml
- Binds the ClusterRole toservice-account.dind-volume-provisioner.re.yaml