Camel K and Amazon EKS (Elastic Kubernetes Service)

[abogatov]

I'm trying to deploy a simple camel-k configuration and make it work with Amazon EKS.
What I did:

1. created a new Kubernetes cluster based on Amazon EKS
   

2. installed Camel-K using the command
   kamel install --registry docker.io --organization <user id>--registry-secret <pass>
   
   After this, I have a camel-k active pod
   

3. tried to run follow sample integration

// camel-k: language=java
import org.apache.camel.builder.RouteBuilder;
public class Demo extends RouteBuilder {
  @Override
  public void configure() throws Exception {
      // Write your routes here, for example:
      from("timer:java?period=1000")
        .routeId("java")
        .setBody()
          .simple("Hello Camel K from ${routeId}")
        .to("log:info");
  }
}

using the command
kamel run Demo.java --dev

I have the next output after this command:

Progress: integration "demo" in phase Initialization
Progress: integration "demo" in phase Building Kit
Condition "IntegrationPlatformAvailable" is "True" for Integration demo: default/camel-k
Integration demo in phase "Initialization"
Integration demo in phase "Building Kit"
Condition "IntegrationKitAvailable" is "False" for Integration demo: creating a new integration kit
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Integration Kit) changed phase to "Build Submitted"
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Scheduling"
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Pending"
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Running"
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Integration Kit) changed phase to "Build Running"
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Failed"
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Initialization" (recovery 1 of 5)
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Scheduling" (recovery 1 of 5)
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Pending" (recovery 1 of 5)
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Running" (recovery 1 of 5)
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Failed" (recovery 1 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Initialization" (recovery 2 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Scheduling" (recovery 2 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Pending" (recovery 2 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Running" (recovery 2 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Failed" (recovery 2 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Initialization" (recovery 3 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Scheduling" (recovery 3 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Pending" (recovery 3 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Running" (recovery 3 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Failed" (recovery 3 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Initialization" (recovery 4 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Scheduling" (recovery 4 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Pending" (recovery 4 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Running" (recovery 4 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Failed" (recovery 4 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Initialization" (recovery 5 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Scheduling" (recovery 5 of 5)
(combined from similar events): Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Build) changed phase to "Failed" (recovery 5 of 5)
Integration demo subresource kit-c23tllgnm1jafvoms1t0 (Integration Kit) changed phase to "Error"
Progress: integration "demo" in phase Error
Error: integration "demo" deployment failed

As well I got logs of camel pod
camel-k-operator-784d84fdc9-nkc58.log

does this message in the log says something?

I could not find any information in the log that can help me to understand the real problem of this error.
Could someone please help how I can get the details of the error occurring?

And the main question is it possible to use Camel-K with Amazon Elastic Kubernetes Service?
I have not just found any information in the documentation.

[squakez]

EKS should be working too. I have the feeling there must be some problem with the registry as for what I can understand, the IntegrationKit is faling to push to the registry. Apparently the dockerhub registry settings needs different options: https://camel.apache.org/camel-k/latest/installation/registry/dockerhub.html. Ultimately you can try to run kubectl get integrationplatform -o yaml to verify all your registry setting is correct.

[abogatov]

Hey @squakez,
Good to know that it should work on EKS!

I tried both variants with --registry-auth-username (+ --registry-auth-password) parameter and with --registry-secret. In the last one case, I created the secret before it.

I did it according to the documentation https://camel.apache.org/camel-k/latest/installation/registry/dockerhub.html.

Which log line of camel-k-operator-784d84fdc9-nkc58.log indicates that there is a problem with pushing to the registry?
Maybe you can advise me how I can get the details of this error to make sure that the ploblem related this?

The response of the kubectl get integrationplatform -o yaml command:
integrationplatform.yaml.log

[squakez]

I the log I can see the following lines with the phase set to Failed:

spectrum - 2021/04/27 09:42:57 Pulling base image adoptopenjdk/openjdk11:slim (insecure=false)...
{"level":"info","ts":1619516578.8454387,"logger":"camel-k.controller.build","msg":"Build state transition","request-namespace":"default","request-name":"kit-c23tllgnm1jafvoms1t0","api-version":"camel.apache.org/v1","kind":"Build","ns":"default","name":"kit-c23tllgnm1jafvoms1t0","phase":"Failed"}

A successful image push would be like:

spectrum - 2021/04/30 10:30:17 Pulling base image adoptopenjdk/openjdk11:slim (insecure=false)...
spectrum - 2021/04/30 10:30:19 Composing layers...
spectrum - 2021/04/30 10:30:19 Pushing image .... (insecure=false)...
spectrum - 2021/04/30 10:30:20 pushed blob: sha256:ab643b9af80da2616f940dd74e8f43b7cd8a95c2eb8703f76fcf219bcce3e2ef
...
{"level":"info","ts":1619771433.4000874,"logger":"camel-k.controller.build","msg":"Build state transition","request-namespace":"operator-test","request-name":"kit-c25s059a6to37qoarfv0","api-version":"camel.apache.org/v1","kind":"Build","ns":"operator-test","name":"kit-c25s059a6to37qoarfv0","phase":"Succeeded"}

or maybe is the same pulling that is failing, if, by any chance there is no connectivity from/to the registry or any failure with that component.

[abogatov]

Do you think is it failed in the pulling base image moment?
Can you please tell me where the source code of this phase is started I mean where this log message (Pulling base image...) is generated? or is it sitting in another project, not camel-k? Just I would like to understand this process.

[squakez]

We do that through another project, Spectrum: https://github.com/container-tools/spectrum/blob/master/pkg/builder/build.go#L28

[abogatov]

Thanks, @squakez.
In case of any error that occurred during the pull, we should observe the "could not pull base image image" message in the log, right?

[squakez]

Thanks, @squakez.
In case of any error that occurred during the pull, we should observe the "could not pull base image image" message in the log, right?

Not necessarily, as I managed to replicate the same behavior by setting fake registries. I've also managed to make it work the registry procedure locally. My best guess is that there is some network issue when trying to reach out docker registry from the EKS nodes, assuming that the credentials are correct. You can try a very simple execution to confirm your EKS has no network limitation by running a simple pod based on a docker.io image, ie create a file named hello.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: hello-world
spec:
  containers:
  - image: docker.io/hello-world
    name: hello-world

$ kubectl apply -f hello-world.yaml
...
$ kubectl logs hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.
...

If that is working as shown here, then, we probably have some issue to be further analized.

[abogatov]

I've done. It looks fine

[squakez]

Thanks for checking. I'll keep having a look and revert when I found something, feel free to keep checking on your side too and report here if any news.

[abogatov]

thanks a lot!
sure, I'll try to connect to the camel pod directly and try to found some additional logs

[abogatov]

Hi @squakez, I'm just wondering is it possible to pull the adoptopenjdk/openjdk11 image manually from the camel-k-operator pod using bash?

[squakez]

Not sure if that alone would fix the problem. We should understand what exactly is failing during that operation in order to have a proper fix.

[abogatov]

I was just mentioning that if there is a way to pull image manually (not through kamel command) being inside the pod we could find out if there is a network problem (as example) of Amazon EKS environment or the actual problem related to the kamel.

[squakez]

We tested that with the hello-world Pod. As that worked, then, there must be something different than connectivity to the registry.

[squakez]

Hey @abogatov. I was trying to troubleshoot this problem and I've seen in the code that we store the error in the Build status without logging it. It would be nice if you can run:

$ k get builds
NAME                       PHASE     AGE     STARTED   DURATION        ATTEMPTS
kit-c299k1cmfkpd6fqa6ar0   Pending   3m26s   43s       43.734806319s   4
...
$ k get builds kit-c299k1cmfkpd6fqa6ar0 -o yaml
apiVersion: camel.apache.org/v1
kind: Build
...
  baseImage: adoptopenjdk/openjdk11:slim
...
  duration: 34.013935519s
  error: 'could not pull base image image adoptopenjdk/openjdk11:slim: GET https://auth.docker.io/token?scope=repository%3Aadoptopenjdk%2Fopenjdk11%3Apull&service=registry.docker.io:
    unsupported status code 401'
  failure:
    reason: 'could not pull base image image adoptopenjdk/openjdk11:slim: GET https://auth.docker.io/token?scope=repository%3Aadoptopenjdk%2Fopenjdk11%3Apull&service=registry.docker.io:
      unsupported status code 401'
    recovery:
      attempt: 4
      attemptMax: 5
      attemptTime: "2021-05-05T13:17:07Z"
    time: "2021-05-05T13:15:19Z"
  phase: Failed
  platform: camel-k
  startedAt: "2021-05-05T13:17:07Z"

In this example, I've simulated the error with a fake registry credentials. Let's look at your error, it may be finally tell us exactly what's going on.

[abogatov]

Hey @squakez,
here are the results

kubectl_get_builds_kit-c24lsc6hs6s6vdvm9jf0.log

[squakez]

Try to specify Spectrum as build strategy instead of Buildah: kamel install --build-publish-strategy=Spectrum ... plus the registry configuration as usual.

[abogatov]

I used this parameter as well.
I've re-installed camel-k (I've uninstalled it before re-install)

and again run kubectl get builds kit-c29nla863kg74t466aj0 -o yaml

so, the error details now:

error: 'could not pull base image image adoptopenjdk/openjdk11:slim: GET https://auth.docker.io/token?scope=repository%3Aadoptopenjdk%2Fopenjdk11%3Apull&service=registry.docker.io:
    unsupported status code 401'
  failure:
    reason: 'failure while building project: signal: killed'
    recovery:
      attempt: 5
      attemptMax: 5
      attemptTime: "2021-05-06T05:21:24Z"
    time: "2021-05-06T05:15:55Z"
  phase: Error
  platform: camel-k

this is the same as you have.
Next, I changed my password on the docker hub and it worked!

@squakez, thanks a lot for your support!

My next goals to try the Amason ECR repository instead docker hub and understand what is the camel-k-operator actually doing.
A couple of questions, if it's still possible :-)
The docker repository is used for deploying images that contain particular integration and environment to run it. And the main advantages of camel-k are the automatic creation and deployment of these images (I can pick up them and deploy them manually to the Kubernative), right?
Where can I read more details about camel-k-operator (what is it actually doing)?

[squakez]

I'm glad we sorted that out!

My next goals to try the Amason ECR repository instead docker hub and understand what is the camel-k-operator actually doing.

Yeah, I'd say it is more natural fit, and it would overcome the rate limitation you have on Docker Hub. If you are successful, feel free to contribute with some documentation, I'll be happy to support you with that too.

A couple of questions, if it's still possible :-)
The docker repository is used for deploying images that contain particular integration and environment to run it. And the main advantages of camel-k are the automatic creation and deployment of these images (I can pick up them and deploy them manually to the Kubernative), right?

The docker registry is used to store the base images used for the Integrations. The operator creates IntegrationKits which are mainly composed by the maven dependencies that will be needed by your Integration. The operator has the intelligence to understand what your Integration requires and to pick the best IntegrationKit or create one, if none is matching. Ideally you can use the base images kits manually, but I don't think it's something you really want/need to do.

Where can I read more details about camel-k-operator (what is it actually doing)?

We have documentation in the Camel community website. You can look the camel k operator part, specifically. In any case, feel free to contact us if you have any further question.

Also, if this problem has solved, feel free to close this issue :)

[abogatov]

Yes, this particular issue has been fixed, thanks again for the good effort!

[oscerd]

I think documentation about how to install on EKS would be useful on the website.

[murugenthirap-cv]

@abogatov @squakez Have anyone tried this with ECR. Is there any documentation/Issues? We are trying to get this working with ECR and Appreciate any information. If not I will raise a separate thread providing more information on the issue we are facing.

[squakez]

@abogatov @squakez Have anyone tried this with ECR. Is there any documentation/Issues? We are trying to get this working with ECR and Appreciate any information. If not I will raise a separate thread providing more information on the issue we are facing.

I cannot see any documentation related. Sure, feel free to open a new issue or contribute to the documentation with any insight you may find.