Skip to content

Fix integration tests #7827

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 31, 2022
Merged

Fix integration tests #7827

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b6e4704
[test] Run tests via a vanilla pod
kylos101 Nov 12, 2021
6af99c5
[test] Avoid randomization which breaks tests
kylos101 Nov 12, 2021
f36b942
add which test is being executed
sagor999 Jan 22, 2022
d0c1544
[tests] fix integration tests
sagor999 Jan 23, 2022
ec61d39
[tests] run in core-dev again
kylos101 Jan 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions .werft/run-integration-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,19 +61,23 @@ pod:
set -Eeuo pipefail

echo "[prep] receiving config..."
mkdir /root/.config
cp -R /config/gcloud /root/.config/gcloud
export GOOGLE_APPLICATION_CREDENTIALS=/config/gcloud/legacy_credentials/gitpod-deployer@gitpod-core-dev.iam.gserviceaccount.com/adc.json
export GOOGLE_APPLICATION_CREDENTIALS="/config/gcloud/legacy_credentials/cd-gitpod-deployer@gitpod-core-dev.iam.gserviceaccount.com/adc.json"
echo "[prep] received config."

USERNAME="{{ .Annotations.username }}"
if [[ "$USERNAME" == "<no value>" ]]; then
USERNAME=""
fi
echo "[prep] using username: $USERNAME"

args=()
args+=( '-kubeconfig=/config/kubeconfig' )
args+=( "-namespace={{ .Annotations.namespace }}" )
[[ "$USERNAME" != "" ]] && args+=( "-username=$USERNAME" )
echo "[prep] args: ${args[@]}"
echo "[prep|DONE]"

/entrypoint.sh -kubeconfig=/config/kubeconfig -namespace={{ .Annotations.namespace }} -username=$USERNAME 2>&1 | ts "[int-tests] "
/entrypoint.sh "${args[@]}" 2>&1 | ts "[int-tests] "

RC=${PIPESTATUS[0]}
if [ $RC -eq 1 ]; then
Expand Down
41 changes: 32 additions & 9 deletions test/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,22 +29,45 @@ There is a [werft job](../.werft/run-integration-tests.yaml) that runs the integ

Example command:
```
werft job run github -j .werft/run-integration-tests.yaml -a namespace=staging-gpl-2658-int-tests -a version=gpl-2658-int-tests.57 -f
werft run github -j .werft/run-integration-tests.yaml -a namespace=staging-gpl-2658-int-tests -a version=gpl-2658-int-tests.57 -f
```

### Manually against a Kubernetes cluster
### Manually

You may want to run tests to assert whether your Gitpod installation is successfully integrated.
You may want to run tests to assert whether a Gitpod installation is successfully integrated.

To test your Gitpod installation:
#### Using a pod

1. Set your kubectl context to the cluster you want to test
2. Integrate the Gitpod installation with OAuth for Github and/or Gitlab, otherwise related tests may fail
3. Clone this repo, and `cd` to `./gitpod/test`
4. Run the tests like so
Best for when you want to validate an environment.

1. Update image name in `integration.yaml` for job `integration-job` to latest built by werft.
2. Optionally add your username in that job argument or any other additional params.
2. Apply yaml file that will add all necessary permissions and create a job that will run tests.
* [`kubectl apply -f ./integration.yaml`](./integration.yaml)
3. Check logs to inspect test results like so `kubectl logs -f jobs/integration-job`.
4. Tear down the integration user and job when testing is done.
* [`kubectl delete -f ./integration.yaml`](./integration.yaml)

#### Go test

Best for when you're actively developing Gitpod.
Test will work if images that they use are already cached by Gitpod instance. If not, they might fail if it takes too long to pull an image.
There are 4 different types of tests:
1. Enterprise specific, that require valid license to be installed. Run those with `-enterprise=true`
2. Tests that require correct user (user should have github OAuth integration setup with gitpod). Run those with `-username=<gitpod_username>`. Make sure to load https://github.com/gitpod-io/gitpod-test-repo and https://github.com/gitpod-io/gitpod workspaces inside your gitpod that you are testing to preload those images onto your node. Wait for it to finish pulling those image, this will ensure that test will not fail due to timeout while waiting to pull an image for the first time.
3. To test gitlab integration, add `-gitlab=true`
4. All other tests.

To run the tests:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you able to run the tests against a preview environment namespace? Like for this branch, as an example?

I know the goal was to get it running against workspace-preview, but it would be interesting to see how it fairs here, too.

Ideally we'd also be able to use these tests against core-dev (a set of namespaces in a single GKE cluster). Pretty soon core-dev will be replaced with Hertzner, where each preview environment (branch) gets a single node K3s cluster.

Copy link
Contributor Author

@sagor999 sagor999 Jan 27, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm. There is something with permissions when trying to run go tests in core-dev. when test calls kubectl it cannot access kubeconfig.

And also you cannot run it via docker container due to permissions:

kubectl apply -f integration.yaml 
serviceaccount/integration-svc created
job.batch/integration-job created
Error from server (Forbidden): error when creating "integration.yaml": roles.rbac.authorization.k8s.io is forbidden: User "pavel@gitpod.io" cannot create resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "staging-kyleb-installer-integration": requires one of ["container.roles.create"] permission(s).
Error from server (Forbidden): error when creating "integration.yaml": rolebindings.rbac.authorization.k8s.io is forbidden: User "pavel@gitpod.io" cannot create resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "staging-kyleb-installer-integration": requires one of ["container.roleBindings.create"] permission(s).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry I missed your response from a couple days ago! 🙏

I rebased with main and did ec61d39 to get the tests runnable in werft against core-dev. May I ask for you to review?

I just tried kubectl apply and had similar trouble in core-dev, which may be intentional (in hindsight) so that we do not "break" core-dev for other namespaces. In other words, kubectl apply is probably best for us (in hindsight) in single node clusters dedicated to ourselves (workspace preview, harvester, etc.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thank you!

1. Clone this repo (`git clone git@github.com:gitpod-io/gitpod.git`), and `cd` to `./gitpod/test`
2. Run the tests like so
```bash
go test -v ./... \
-kubeconfig=<path_to_kube_config_file> \
-namespace=<namespace_where_gitpod_is_installed> \
-username=<a_user_in_the_gitpod_database>
-username=<gitpod_user_with_oauth_setup> \
-enterprise=<true|false> \
-gitlab=<true|false>
```
3. Tests `TestUploadDownloadBlob` and `TestUploadDownloadBlobViaServer` will fail when testing locally, as they are trying to connect to cluster local resources directly. To test them use docker image instead that runs within the cluster.
4. If you want to run specific test, add `-run <test>` before `-kubeconfig` parameter.
23 changes: 21 additions & 2 deletions test/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,30 @@
# Licensed under the GNU Affero General Public License (AGPL).
# See License-AGPL.txt in the project root for license information.

set -ex
# exclude 'e' (exit on any error)
# there are many test binaries, each can have failures
set -x

export PATH=$PATH:/tests

FAILURE_COUNT=0
# shellcheck disable=SC2045
for i in $(find /tests/ -name "*.test" | sort -R); do
for i in $(find /tests/ -name "*.test" | sort); do
echo "running test: $i"
"$i" "$@";
TEST_STATUS=$?
if [ "$TEST_STATUS" -ne "0" ]; then
FAILURE_COUNT=$((FAILURE_COUNT+1))
echo "Test failed at $(date)"
else
echo "Test succeeded at $(date)"
fi;
done

if [ "$FAILURE_COUNT" -ne "0" ]; then
# exit with the number of test binaries that failed
echo "Test suite ended with failure at $(date)"
exit $FAILURE_COUNT
fi;

echo "Test suite ended with success at $(date)"
62 changes: 62 additions & 0 deletions test/integration.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: integration-svc
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: integration-role
rules:
- apiGroups:
- ''
resources:
- 'pods'
- 'secrets'
- 'services'
- 'configmaps'
- 'endpoints'
verbs:
- 'list'
- 'get'
- apiGroups:
- ''
resources:
- 'pods/portforward'
- 'pods/exec'
verbs:
- 'create'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: integration-rolebind
subjects:
- kind: ServiceAccount
name: integration-svc
roleRef:
kind: Role
name: integration-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: batch/v1
kind: Job
metadata:
name: integration-job
spec:
template:
spec:
serviceAccountName: integration-svc
containers:
- name: tests
image: eu.gcr.io/gitpod-core-dev/build/integration-tests:kyleb-installer-integration.24
imagePullPolicy: Always
#args: ["-username=sagor999"]
#args: ["-enterprise=true"]
#args: ["-gitlab=true"]
resources:
requests:
cpu: 2
memory: 4Gi
restartPolicy: Never
backoffLimit: 0
7 changes: 6 additions & 1 deletion test/leeway.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,16 @@ FROM alpine:3.15
RUN apk upgrade --no-cache \
&& apk add --no-cache \
ca-certificates \
coreutils
coreutils \
curl

# convenience scripting tools
RUN apk add --no-cache bash moreutils

# deps for tests to run
RUN curl -fsSL "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" -o /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl

COPY test--app/bin /tests
ENV PATH=$PATH:/tests
COPY entrypoint.sh /entrypoint.sh
Expand Down
24 changes: 13 additions & 11 deletions test/pkg/integration/apis.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,11 @@ import (
)

// API provides access to the individual component's API
func NewComponentAPI(ctx context.Context, namespace string, client klient.Client) *ComponentAPI {
func NewComponentAPI(ctx context.Context, namespace string, kubeconfig string, client klient.Client) *ComponentAPI {
return &ComponentAPI{
namespace: namespace,
client: client,
namespace: namespace,
kubeconfig: kubeconfig,
client: client,

closerMutex: sync.Mutex{},

Expand All @@ -71,8 +72,9 @@ type serverStatus struct {

// ComponentAPI provides access to the individual component's API
type ComponentAPI struct {
namespace string
client klient.Client
namespace string
kubeconfig string
client klient.Client

closer []func() error
closerMutex sync.Mutex
Expand Down Expand Up @@ -170,7 +172,7 @@ func (c *ComponentAPI) Supervisor(instanceID string) (grpc.ClientConnInterface,
}

ctx, cancel := context.WithCancel(context.Background())
ready, errc := common.ForwardPort(ctx, c.client.RESTConfig(), c.namespace, pod, fmt.Sprintf("%d:22999", localPort))
ready, errc := common.ForwardPort(ctx, c.kubeconfig, c.namespace, pod, fmt.Sprintf("%d:22999", localPort))
select {
case err = <-errc:
cancel()
Expand Down Expand Up @@ -483,7 +485,7 @@ func (c *ComponentAPI) WorkspaceManager() (wsmanapi.WorkspaceManagerClient, erro
}

ctx, cancel := context.WithCancel(context.Background())
ready, errc := common.ForwardPort(ctx, c.client.RESTConfig(), c.namespace, pod, fmt.Sprintf("%d:8080", localPort))
ready, errc := common.ForwardPort(ctx, c.kubeconfig, c.namespace, pod, fmt.Sprintf("%d:8080", localPort))
select {
case err := <-errc:
cancel()
Expand Down Expand Up @@ -556,7 +558,7 @@ func (c *ComponentAPI) BlobService() (csapi.BlobServiceClient, error) {
}

ctx, cancel := context.WithCancel(context.Background())
ready, errc := common.ForwardPort(ctx, c.client.RESTConfig(), c.namespace, pod, fmt.Sprintf("%d:8080", localPort))
ready, errc := common.ForwardPort(ctx, c.kubeconfig, c.namespace, pod, fmt.Sprintf("%d:8080", localPort))
select {
case err := <-errc:
cancel()
Expand Down Expand Up @@ -609,7 +611,7 @@ func (c *ComponentAPI) DB(options ...DBOpt) (*sql.DB, error) {
// if configured: setup local port-forward to DB pod
if config.ForwardPort != nil {
ctx, cancel := context.WithCancel(context.Background())
ready, errc := common.ForwardPort(ctx, c.client.RESTConfig(), c.namespace, config.ForwardPort.PodName, fmt.Sprintf("%d:%d", config.Port, config.ForwardPort.RemotePort))
ready, errc := common.ForwardPort(ctx, c.kubeconfig, c.namespace, config.ForwardPort.PodName, fmt.Sprintf("%d:%d", config.Port, config.ForwardPort.RemotePort))
select {
case err := <-errc:
cancel()
Expand Down Expand Up @@ -855,7 +857,7 @@ func (c *ComponentAPI) ImageBuilder(opts ...APIImageBuilderOpt) (imgbldr.ImageBu
}

ctx, cancel := context.WithCancel(context.Background())
ready, errc := common.ForwardPort(ctx, c.client.RESTConfig(), c.namespace, pod, fmt.Sprintf("%d:8080", localPort))
ready, errc := common.ForwardPort(ctx, c.kubeconfig, c.namespace, pod, fmt.Sprintf("%d:8080", localPort))
select {
case err = <-errc:
cancel()
Expand Down Expand Up @@ -904,7 +906,7 @@ func (c *ComponentAPI) ContentService() (ContentService, error) {
}

ctx, cancel := context.WithCancel(context.Background())
ready, errc := common.ForwardPort(ctx, c.client.RESTConfig(), c.namespace, pod, fmt.Sprintf("%d:8080", localPort))
ready, errc := common.ForwardPort(ctx, c.kubeconfig, c.namespace, pod, fmt.Sprintf("%d:8080", localPort))
select {
case err := <-errc:
cancel()
Expand Down
9 changes: 5 additions & 4 deletions test/pkg/integration/common/port_forward.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@ import (
"time"

"golang.org/x/xerrors"
"k8s.io/client-go/rest"
)

// ForwardPort establishes a TCP port forwarding to a Kubernetes pod
// Uses kubectl instead of Go to use a local process that can reproduce the same behavior outside the tests
func ForwardPort(ctx context.Context, config *rest.Config, namespace, pod, port string) (readychan chan struct{}, errchan chan error) {
// Since we are using kubectl directly we need to pass kubeconfig explicetly
func ForwardPort(ctx context.Context, kubeconfig string, namespace, pod, port string) (readychan chan struct{}, errchan chan error) {
errchan = make(chan error, 1)
readychan = make(chan struct{}, 1)

Expand All @@ -28,18 +28,19 @@ func ForwardPort(ctx context.Context, config *rest.Config, namespace, pod, port
"--address=0.0.0.0",
fmt.Sprintf("pod/%v", pod),
fmt.Sprintf("--namespace=%v", namespace),
fmt.Sprintf("--kubeconfig=%v", kubeconfig),
port,
}

command := exec.CommandContext(ctx, "kubectl", args...)
err := command.Start()
if err != nil {
errchan <- xerrors.Errorf("unexpected error starting port-forward: %w", err)
errchan <- xerrors.Errorf("unexpected error starting port-forward: %w, args: %v", err, args)
}

err = command.Wait()
if err != nil {
errchan <- xerrors.Errorf("unexpected error running port-forward: %w", err)
errchan <- xerrors.Errorf("unexpected error running port-forward: %w, args: %v", err, args)
}
}()

Expand Down
29 changes: 17 additions & 12 deletions test/pkg/integration/integration.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ func WithWorkspacekitLift(lift bool) InstrumentOption {
// If there isn't, we attempt to build `<agentName>_agent/main.go`.
// The binary is copied to the destination pod, started and port-forwarded. Then we
// create an RPC client.
func Instrument(component ComponentType, agentName string, namespace string, client klient.Client, opts ...InstrumentOption) (*rpc.Client, []func() error, error) {
func Instrument(component ComponentType, agentName string, namespace string, kubeconfig string, client klient.Client, opts ...InstrumentOption) (*rpc.Client, []func() error, error) {
var closer []func() error

options := instrumentOptions{
Expand Down Expand Up @@ -188,7 +188,7 @@ func Instrument(component ComponentType, agentName string, namespace string, cli
execErrs := make(chan error, 1)
go func() {
defer close(execErrs)
execErr := executeAgent(cmd, podName, containerName, namespace, client)
execErr := executeAgent(cmd, podName, containerName, namespace, kubeconfig, client)
if execErr != nil {
execErrs <- execErr
}
Expand All @@ -214,7 +214,7 @@ func Instrument(component ComponentType, agentName string, namespace string, cli
}
}()

fwdReady, fwdErr := common.ForwardPort(ctx, client.RESTConfig(), namespace, podName, strconv.Itoa(localAgentPort))
fwdReady, fwdErr := common.ForwardPort(ctx, kubeconfig, namespace, podName, strconv.Itoa(localAgentPort))
select {
case <-fwdReady:
case err := <-execErrs:
Expand Down Expand Up @@ -274,8 +274,9 @@ func getFreePort() (int, error) {
return result.Port, nil
}

func executeAgent(cmd []string, pod, container string, namespace string, client klient.Client) error {
args := []string{"exec", pod, fmt.Sprintf("--namespace=%v", namespace)}
func executeAgent(cmd []string, pod, container string, namespace string, kubeconfig string, client klient.Client) error {
// since we are self signing certs for gitpod components, pass insecure flag
args := []string{"exec", pod, fmt.Sprintf("--namespace=%v", namespace), "--insecure-skip-tls-verify=true", fmt.Sprintf("--kubeconfig=%v", kubeconfig)}
if len(container) > 0 {
args = append(args, fmt.Sprintf("--container=%s", container))
}
Expand All @@ -285,7 +286,7 @@ func executeAgent(cmd []string, pod, container string, namespace string, client
command := exec.Command("kubectl", args...)
out, err := command.CombinedOutput()
if err != nil {
return xerrors.Errorf("cannot run kubectl command: %w\n%v", err, string(out))
return xerrors.Errorf("cannot run kubectl command: %w\n%v\nargs:%v", err, string(out), args)
}

return nil
Expand Down Expand Up @@ -431,12 +432,16 @@ func GetServerConfig(namespace string, client klient.Client) (*ServerConfigParti
// ServerIDEConfigPartial is the subset of server IDE config we're using for integration tests.
// NOTE: keep in sync with chart/templates/server-ide-configmap.yaml
type ServerIDEConfigPartial struct {
IDEVersion string `json:"ideVersion"`
IDEImageRepo string `json:"ideImageRepo"`
IDEImageAliases struct {
Code string `json:"code"`
CodeLatest string `json:"code-latest"`
} `json:"ideImageAliases"`
IDEOptions struct {
Options struct {
Code struct {
Image string `json:"image"`
} `json:"code"`
CodeLatest struct {
Image string `json:"image"`
} `json:"code-latest"`
} `json:"options"`
} `json:"ideOptions"`
}

func GetServerIDEConfig(namespace string, client klient.Client) (*ServerIDEConfigPartial, error) {
Expand Down
Loading