Merge pull request #1 from SAP/master

update codes from SAP/Infrabox

README.md

@@ -1,4 +1,4 @@
-# <a href="http://www.infrabox.net"><img src="src\dashboard-client\static\logo_compact_transparent.png" width="200"></a>
+# <img src="src\dashboard-client\static\logo_compact_transparent.png" width="200">
 [![Build Status](https://infraboxci.datahub.sapcloud.io/api/v1/projects/deb14c11-dcbe-41f0-ade8-9d26e87266c3/state.svg?branch=master)](https://infraboxci.datahub.sapcloud.io/dashboard/#/project/sap-infrabox)
 [![REUSE status](https://api.reuse.software/badge/github.com/SAP/InfraBox)](https://api.reuse.software/info/github.com/SAP/InfraBox)
 
@@ -17,16 +17,13 @@ Some of InfraBox's features are:
 - [Periodically schedule builds](docs/cronjobs.md)
 - [and many more, see our examples](https://github.com/SAP/infrabox-examples)
 
-## Want to see it in action?
-InfraBox is built on InfraBox. See all the builds [here](https://infrabox.ninja/dashboard/#/project/sap-infrabox).
 
 ## Documentation
 All our documentation can be found [here](docs/README.md). You can also look at our [example repository](https://github.com/SAP/infrabox-examples) on how to make use of the different features InfraBox provides.
 
 ## How to obtain support
 If you need help please post your questions to [Stack Overflow](https://stackoverflow.com/questions/tagged/infrabox).
 In case you found a bug please open a [Github Issue](https://github.com/SAP/infrabox/issues).
-Follow us on Twitter: [@Infra_Box](https://twitter.com/Infra_Box) or have look at our Slack channel [infrabox.slack.com](https://join.slack.com/t/infrabox/shared_invite/enQtNDM5OTUzNzg4NjU3LWFiNWY1MDYyZTA3MmUxNWNmZjgwNWEzZTI0NGYzN2U5OTFjYjQyNTBmNWE0NTYzOGJlOTdmZWI3NmQ3OTQ0MDA).
 
 ## Roadmap
 

deploy/infrabox/templates/_helpers.tpl

@@ -387,6 +387,20 @@ https://{{- required "host is required" .Values.host -}}:{{- .Values.port -}}
     value: {{ include "image_tag" . }}
 {{ end }}
 
+{{ define "github_access" }}
+-
+    name: INFRABOX_GITHUB_ENABLED
+    value: {{ .Values.github.enabled | quote }}
+{{- if .Values.github.enabled }}
+-
+    name: GITHUB_HOST
+    value: {{ .Values.github.host }}
+-
+    name: GITHUB_ENABLE_TOKEN_ACCESS
+    value: {{ .Values.github.token_access | quote}}
+{{ end }}
+{{ end }}
+
 {{ define "env_cluster" }}
 -
     name: INFRABOX_CLUSTER_NAME

deploy/infrabox/templates/function_crd.yaml

@@ -15,6 +15,8 @@ spec:
             memory: 1Gi
     env:
     {{ include "env_general" . | indent 4 }}
+    {{ include "env_cluster" . | indent 4 }}
+    {{ include "github_access" . | indent 4 }}
     -
         name: INFRABOX_JOB_STORAGE_DRIVER
         value: {{ .Values.job.storage_driver }}

deploy/infrabox/values.yaml

@@ -233,6 +233,11 @@ github:
     # Github API URL
     api_url: https://api.github.com
 
+    host: github.com
+
+    token_access: false
+
+
 # General Account configuration
 account:
     signup:

docs/job/dependencies.md

@@ -65,4 +65,4 @@ Suppose we have the following job definition:
 }
 ```
 
-If the the parent job would create the file `/infrabox/output/hello_world.txt` then the child job would be able to access it under `/infrabox/inputs/parent/hello_world.txt`.
+If the the parent job would create the file `/infrabox/output/hello_world.txt`, then the child job would be able to access it at run time (i.e. when the child job container is running) under `/infrabox/inputs/parent/hello_world.txt`. The file is also available at build time (i.e. when building a Docker image for the child job): Infrabox would add the file at `.infrabox/inputs/parent/hello_world.txt` on the repo root and it can be referenced on the child job's Dockerfile.

docs/job/docker.md

@@ -55,7 +55,7 @@ To set docker build arguments create an object with the names and values for `bu
         ...
         "build_arguments": {
             "ARG_1": "My value",
-            "ANOTHER_ARG": "some other value
+            "ANOTHER_ARG": "some other value"
         }
     }]
 }

docs/job/docker_image.md

@@ -26,10 +26,10 @@ You can also specify an already build image and run it as a job.
 
 | Name | Required | Type | Default | Description |
 |------|----------|------|---------|-------------|
-|type|true|string||Has to be "docker" to run a single Docker container|
+|type|true|string||Has to be "docker-image" to run a single Docker container|
 |name|true|string||Name of the job|
 |image|true|string||Image to use, i.e. `alpine:latest`|
-|command|true|string||The command in [exec form](https://docs.docker.com/engine/reference/builder/#cmd)|
+|command|false|string||The command in [exec form](https://docs.docker.com/engine/reference/builder/#cmd)|
 |resources|true|[Resource Configuration](/docs/job/resources.md)||Specify the required resources for your job|
 |build_context|false|string||Specify the docker build context. If not set the directory containing the `infrabox.json` file will be used.|
 |cache|false|[Cache Configuration](/docs/job/cache.md)|{}|Configure the caching behavior|

docs/job/vault.md

@@ -0,0 +1,37 @@
+# Vault(KV Engines only)
+InfraBox can fetch values as environment from vault service, so if your variable rotation regularly, you can configure it with vault in your projects. You just need to update the variable in Vault rather than update in your Infrabox project when the variable rotation. Login to the InfraBox Dashboard, select your project and go to the Settings tab. There you can create a vault with a name, a url, a namespace, a version, a token and a ca certificate.
+
+## Parameters explanation
+
+name: a DIY name (e.g. myvault)  
+
+url: the url of vault service (e.g. https://vault-service.com:1234)  
+
+namespace: the Vault's namespace, only enterprise edition enable namespace.  
+
+version：Vault provide version 1 or 2 for KV engine. just set it with 1 or 2.
+
+token: a token to access Vault.
+
+ca: provide ca certificate if using https. 
+
+
+
+## Using secrets as environment variable
+If you have created a [vault](#vault) you can make it available as a environment variable.
+
+```json
+{
+    "version": 1,
+    "jobs": [{
+        ...
+        "environment": {
+            "SOME_VALUE": {
+                "$vault": " the name of the vault",
+                "$vault_secret_path": "the secret path in vault's kv engine",
+                "$vault_secret_key": "the key of the vault secret"
+            },
+        }
+    }]
+}
+```

infrabox/generator/e2e.json

@@ -1,33 +1,6 @@
 {
     "version": 1,
     "jobs": [{
-        "type": "docker",
-        "name": "e2e-k8s-1-14",
-        "docker_file": "infrabox/test/e2e/Dockerfile",
-        "build_context": "../..",
-        "build_only": false,
-        "resources": {
-            "limits": { "cpu": 1, "memory": 2048 }
-        },
-        "cache": {
-            "image": true
-        },
-        "timeout": 1800,
-        "services": [{
-            "apiVersion": "gcp.service.infrabox.net/v1alpha1",
-            "kind": "GKECluster",
-            "metadata": {
-                "name": "e2e-cluster"
-            },
-            "spec": {
-                "machineType": "n1-standard-4",
-                "numNodes": 2,
-                "preemptible": false,
-                "zone": "us-east1-b",
-                "clusterVersion": "1.14"
-            }
-        }]
-    }, {
         "type": "docker",
         "name": "e2e-k8s-1-15",
         "docker_file": "infrabox/test/e2e/Dockerfile",
@@ -39,7 +12,7 @@
         "cache": {
             "image": true
         },
-        "timeout": 1800,
+        "timeout": 2700,
         "services": [{
             "apiVersion": "gcp.service.infrabox.net/v1alpha1",
             "kind": "GKECluster",

infrabox/test/e2e/Dockerfile

@@ -1,5 +1,5 @@
 FROM ubuntu:18.04
-ARG HELM_VERSION=2.12.3
+ARG HELM_VERSION=2.17.0
 
 RUN apt-get update -y && apt-get install -y \
     curl \

infrabox/test/e2e/entrypoint.sh

@@ -62,11 +62,14 @@ _getNginxIP() {
 
 _initHelm() {
     echo "## init helm"
+
     kubectl -n kube-system create sa tiller
     kubectl create clusterrolebinding tiller \
 		--clusterrole cluster-admin \
 		--serviceaccount=kube-system:tiller
     helm init --service-account tiller --wait
+
+    helm repo add stable https://charts.helm.sh/stable
 }
 
 _getPodNameImpl() {
@@ -109,6 +112,7 @@ _installPostgres() {
 
 _installMinio() {
     echo "## Install minio"
+
     helm install \
         --set serviceType=ClusterIP,replicas=1,persistence.enabled=false \
         -n infrabox-minio \

infrabox/test/github-review/test.py

@@ -123,7 +123,7 @@ def side_effect(_1, _2, _3):
 
         requests_post.assert_called_with(
             'status_url',
-            data='{"state": "pending", "target_url": "GITHUB_URL/dashboard/#/project/projectname/build/123/123/job/jobname", "description": "InfraBox", "context": "Job: jobname"}',
+            data='{"state": "pending", "target_url": "GITHUB_URL/dashboard/#/project/projectname/build/123/123/job/jobname", "description": "%s", "context": "Job: jobname"}' % ("project_id:%s job_id:%s" % (self.job_data['project_id'], self.job_data['id']) ),
             headers={'Authorization': 'token token', 'User-Agent': 'InfraBox'},
             timeout=10,
             verify=False)
@@ -149,7 +149,7 @@ def side_effect(_1, _2, _3):
 
         execute_sql.side_effect = side_effect
         handle_job_update(None, self.event)
-        data = '{"state": "%s", "target_url": "GITHUB_URL/dashboard/#/project/projectname/build/123/123/job/jobname", "description": "InfraBox", "context": "Job: jobname"}' % self.expected_github_status
+        data = '{"state": "%s", "target_url": "GITHUB_URL/dashboard/#/project/projectname/build/123/123/job/jobname", "description": "%s", "context": "Job: jobname"}' % (self.expected_github_status, "project_id:%s job_id:%s" % (self.job_data['project_id'], self.job_data['id']))
         requests_post.assert_called_with(
             'status_url',
             data=data,

src/api/handlers/account/account_ldap.py

@@ -7,6 +7,7 @@
 import ldap
 import bcrypt
 
+from ldap.filter import escape_filter_chars
 from pyinfraboxutils import get_logger
 from pyinfraboxutils.ibflask import OK
 from pyinfraboxutils.token import encode_user_token
@@ -34,7 +35,7 @@ def authenticate(email, password):
     ldap_password = os.environ['INFRABOX_ACCOUNT_LDAP_PASSWORD']
     ldap_base_dn = os.environ['INFRABOX_ACCOUNT_LDAP_BASE']
 
-    search_filter = "(mail=%s)" % str(email)
+    search_filter = "(mail=%s)" % escape_filter_chars(str(email))
     user_dn = None
 
     connect = ldap_conn(ldap_server)
@@ -97,7 +98,7 @@ def post(self):
             WHERE email = %s
         ''', [email])
 
-        if user and user['id'] == '00000000-0000-0000-0000-000000000000':
+        if user and user['password']:
             # Admin login
             if not bcrypt.checkpw(password.encode('utf8'), user['password'].encode('utf8')):
                 abort(400, 'Invalid email/password combination')

src/api/handlers/build.py

@@ -30,6 +30,7 @@ def get(self, project_id):
         sha = request.args.get('sha', None)
         branch = request.args.get('branch', None)
         cronjob = request.args.get('cronjob', None)
+        size = request.args.get('size', 10)
 
         if cronjob == "true":
             cronjob = True
@@ -44,6 +45,8 @@ def get(self, project_id):
         if build_to:
             build_to = int(build_to)
 
+        size = min(50, max(int(size), 0))
+
         if not build_to:
             r = g.db.execute_one_dict('''
                 SELECT max(build_number) as max
@@ -57,10 +60,10 @@ def get(self, project_id):
                 build_to = r['max'] + 1
 
         if not build_from:
-            build_from = max(build_to - 10, 0)
+            build_from = 0
 
-        if build_to - build_from > 200:
-            build_from = build_to - 200
+        #if build_to - build_from > 500:
+        #    build_from = max(build_to - 500, 0)
 
         p = g.db.execute_many_dict('''
             SELECT b.id, b.build_number, b.restart_counter, b.is_cronjob
@@ -74,13 +77,15 @@ def get(self, project_id):
             AND (%(branch)s IS NULL OR c.branch = %(branch)s)
             AND (%(cronjob)s IS NULL OR b.is_cronjob = %(cronjob)s)
             ORDER BY build_number DESC, restart_counter DESC
+            LIMIT %(size)s
         ''', {
             'pid': project_id,
             'from': build_from,
             'to': build_to,
             'sha': sha,
             'branch': branch,
             'cronjob': cronjob,
+            'size': size,
         })
 
         return p

src/api/handlers/job_api.py

@@ -5,6 +5,7 @@
 import copy
 import urllib
 import random
+import tempfile
 from datetime import datetime
 from io import BytesIO
 
@@ -806,18 +807,55 @@ def post(self):
                     value = job['environment'][ename]
 
                     if isinstance(value, dict):
-                        env_var_ref_name = value['$secret']
-                        result = g.db.execute_many("""
-                            SELECT value FROM secret WHERE name = %s and project_id = %s
-                        """, [env_var_ref_name, project_id])
-
-                        if not result:
-                            abort(400, "Secret '%s' not found" % env_var_ref_name)
-
-                        if not job['env_var_refs']:
-                            job['env_var_refs'] = {}
-
-                        job['env_var_refs'][ename] = env_var_ref_name
+                        if '$secret' in value:
+                            env_var_ref_name = value['$secret']
+                            result = g.db.execute_many("""
+                                SELECT value FROM secret WHERE name = %s and project_id = %s
+                            """, [env_var_ref_name, project_id])
+
+                            if not result:
+                                abort(400, "Secret '%s' not found" % env_var_ref_name)
+
+                            if not job['env_var_refs']:
+                                job['env_var_refs'] = {}
+
+                            job['env_var_refs'][ename] = env_var_ref_name
+
+                        if '$vault' in value and '$vault_secret_path' in value and "$vault_secret_key" in value:
+                            name = value['$vault']
+                            secret_path = value['$vault_secret_path']
+                            secret_key = value['$vault_secret_key']
+                            result = g.db.execute_one("""
+                                SELECT url,version,token,ca,namespace FROM vault WHERE name = %s and project_id = %s
+                            """, [name, project_id])
+
+                            if not result:
+                                abort(400, "Cannot get Vault '%s' in project '%s' " % (name, project_id))
+
+                            url, version, token, ca, namespace = result[0], result[1], result[2], result[3], result[4]
+                            if not namespace:
+                                namespace = ''
+                            if version == 'v1':
+                                url += '/v1/' + namespace + '/' + secret_path
+                            elif version == 'v2':
+                                paths = secret_path.split('/')
+                                url += '/v1/' + namespace + '/' + paths[0] + '/data/' + '/'.join(paths[1:])
+                            if not ca:
+                                res = requests.get(url=url, headers={'X-Vault-Token': token}, verify=False)
+                            else:
+                                with tempfile.NamedTemporaryFile(delete=False) as f:
+                                    f.write(ca)
+                                    f.flush()  # ensure all data written
+                                    res = requests.get(url=url, headers={'X-Vault-Token': token}, verify=f.name)
+                            if res.status_code == 200:
+                                json_res = json.loads(res.content)
+                                if json_res['data'].get('data') and isinstance(json_res['data'].get('data'), dict):
+                                    value = json_res['data'].get('data').get(secret_key)
+                                else:
+                                    value = json_res['data'].get(secret_key)
+                                job['env_vars'][ename] = value
+                            else:
+                                abort(400, "Getting value from vault error: url is '%s', token is '%s' " % (url, result))
                     else:
                         job['env_vars'][ename] = value
 

src/api/handlers/project.py

@@ -192,6 +192,7 @@ def get(self, project_id):
                                 ON b.id = j.build_id
                                     AND b.project_id = %s
                                     AND j.project_id = %s
+                                    AND j.name LIKE CONCAT(%s, '%%')
                                 INNER JOIN "commit" c
                                     ON c.id = b.commit_id
                                     AND c.project_id = b.project_id
@@ -201,7 +202,7 @@ def get(self, project_id):
                             )
                             AND j.name LIKE CONCAT(%s, '%%')
                     )
-            ''', [project_id, project_id, project_id, project_id, branch, job_name])
+            ''', [project_id, project_id, project_id, project_id, job_name, branch, job_name])
         elif build_number and build_restart_count:
             r = g.db.execute_one_dict('''
                 SELECT

src/api/handlers/projects/__init__.py

@@ -7,3 +7,4 @@
 import api.handlers.projects.tokens
 import api.handlers.projects.cronjobs
 import api.handlers.projects.sshkeys
+import api.handlers.projects.vault