fix(#4361): Avoid errors when components not available in Camel catalog

[christophd]

• Allow integrations that uses components that are not listed in the Camel catalog
• Components may not be listed in the catalog for many reasons (e.g. custom components)
• Just log a warning message in case the component is missing in the catalog

[christophd]

Fixes #4361

[christophd]

@squakez I have tried to fix the failing E2E tests could you please rerun the workflows?

[squakez]

The PR change is good for me. We only need to change the warning output and let it to stderr, as it was done on purpose back in time to let dry run execution to get the output and be able to redirect it to any stream. I think we should add some line of doc in https://camel.apache.org/camel-k/next/configuration/dependencies.html to explain that and how the camel dependencies from catalog are managed.

[christophd]

@squakez E2E test did fail again 😵‍💫. this is the 2nd try to fix it. I also added some words to the dependencies page in the documentation

[squakez]

@squakez E2E test did fail again face_with_spiral_eyes. this is the 2nd try to fix it. I also added some words to the dependencies page in the documentation

Yes, [1] it's failing with:

❌ TestKamelCLIRun (16m26.42s)
      run_test.go:147: 
          Timed out after 900.000s.
          Expected
              <v1.PodPhase>: 
          to equal
              <v1.PodPhase>: Running

IIUC the test want to run an integration, even when a camel:missing-dep is not existing, is it? or the goal is to have an existing camel component that is not in the catalog (but which dependency exists?). In the latter case, you can check any camel component which is not supported by Camel Quarkus and use it instead of camel-main which may try to run the Integration as it was a Camel Main runtime.
[1] https://github.com/apache/camel-k/actions/runs/5054316425/jobs/9069142273

[christophd]

@squakez I have found another issue and fixed it in this PR.

The issue is related to Camel dependencies that are not available in Camel Quarkus extensions. These dependencies should not get resolved to camel-quarkus-xxx dependencies but rather fallback to arbitrary camel-xxx dependencies.

I noticed this in the newly created E2E tests where camel:resilience4j is not available as Camel Quarkus extension but got resolved to camel-quarkus-resilience4j which is simply not available. This was causing the integration to fail at build time with Maven dependency resolving errors. I have fixed that so now Camel dependencies get resolved to camel-xxx instead when there are no Camel Quarkus extensions for this specific dependency available.

[tadayosi]

LGTM.

One thing to note is that throwing some meaningful error or warning when specifying a non-existent or wrong component dependecy is also a request from another user (such as #3127). Please make sure this change also satisfies the request.

Also, as I remember it, the last discussion was that all valid dependencies for camel-k should be listed in the catalog. So if I'm not mistaken it seems to be rather a problem in the Camel catalog. I'm wondering why we can't fix the catalog in the first place.

[tadayosi]

I noticed this in the newly created E2E tests where camel:resilience4j is not available as Camel Quarkus extension but got resolved to camel-quarkus-resilience4j which is simply not available. This was causing the integration to fail at build time with Maven dependency resolving errors. I have fixed that so now Camel dependencies get resolved to camel-xxx instead when there are no Camel Quarkus extensions for this specific dependency available.

What does it mean to use a camel-xxx dependecy on a Quarkus runtime, since in Camel K 1.12.x Quarkus is still the default and only runtime? Doesn't trying to use camel-resilience4j directly mean running a library not yet ready for Quarkus, as the raw Camel dependency shouldn't yet go through stripping of possible runtime usages of reflection or dynamic loading which aren't allowed on Quarkus?

[christophd]

thanks @tadayosi for your comments.

all valid dependencies for camel-k should be listed in the catalog

+1 but recent history has shown that the catalog may be incomplete (see apache/camel-k-runtime#1029 which fixed that)

Doesn't trying to use camel-resilience4j directly mean running a library not yet ready for Quarkus

I think the raw Camel component is not optimized for Quarkus but still is able to run isn't it?

Of course using the Camel Quarkus extensions should be the first choice and the catalog makes sure to use those extensions once they are listed. But we should not fail when the user explicitly uses a non-listed Camel dependency. This would then prevent any customized Camel components and other things that are not listed. In the past I have also been requesting the error based on the catalog checks but today I think this is too restrictive.

[squakez]

I think that the usage of non "quarkified" dependencies/components should not be supported but neither restricted if the user, for any reason, want to use such dependency. I think the approach we are taking here is to tentatively run the Integration, reporting the warning. At this stage I am wondering if it makes sense to include an Integration condition to specify that. I think would be the neatest approach. Basically, if an integration fails, the user can quickly discover the reason of the failure (ie, non supported component) in the Integration conditions.

[squakez]

This was causing the integration to fail at build time with Maven dependency resolving errors. I have fixed that so now Camel dependencies get resolved to camel-xxx instead when there are no Camel Quarkus extensions for this specific dependency available.

Was it already committed? I can see the same error still in the checks.

[lburgazzoli]

thanks @tadayosi for your comments.

all valid dependencies for camel-k should be listed in the catalog

+1 but recent history has shown that the catalog may be incomplete (see apache/camel-k-runtime#1029 which fixed that)

Doesn't trying to use camel-resilience4j directly mean running a library not yet ready for Quarkus

I think the raw Camel component is not optimized for Quarkus but still is able to run isn't it?

Of course using the Camel Quarkus extensions should be the first choice and the catalog makes sure to use those extensions once they are listed. But we should not fail when the user explicitly uses a non-listed Camel dependency. This would then prevent any customized Camel components and other things that are not listed. In the past I have also been requesting the error based on the catalog checks but today I think this is too restrictive.

I think we should made a distinction between:

1. not optimized for the runtime or not known by the runtime
2. not supported by the runtime.

About case 1, we should allow the user to add such dependency as it could be a component/extension non part of the camel distribution but still a valid component/extensions. However in such case I think the user should define the component using a maven coordinate and the component should not use the ´camel.apache.org` as artifact group which should bypass the catalog check (at least this is what I recall).
About case 2, I think the catalog can also include a list of banned dependencies so if we know ahead of time that a specific dependency can cause harm to the runtime, then we can either fail or ad a warning to the integration CR.

Does this make any sense ?

[christophd]

@squakez yes, still same error in E2E test. I am diving deeper into this locally in order to provide a fix

[christophd]

@lburgazzoli sounds good. How do we identify those banned dependencies in the first place?

[christophd]

@squakez @lburgazzoli @tadayosi sorry, resolving to camel-xxx instead of camel-quarkus-xxx for dependencies not listed in the catalog is also not working because the raw Camel version is not resolvable in the generated Maven builder project. I see errors such as 'dependencies.dependency.version' for org.apache.camel:camel-resilience4j:jar is missing because there is no bom that resolves the raw Camel dependency.

So that makes me think that we should maybe go back to throwing errors when the Camel dependency is not in the catalog. In apache/camel-k-runtime#1029 we have fixed the issue that the catalog does not list miscellaneous Camel dependencies and this is the root cause for the initial issue reported in #4361

I think @lburgazzoli made a good point. When a user needs to use a custom Camel component it should not be using the Camel groupId and it should be added to the integration with the mvn: GAV dependency instead of camel:. Also in case the catalog is incomplete we need to fix it there as @tadayosi has mentioned.

So maybe I close this PR and we keep things as they have been with throwing errors once the Camel dependency is not resolvable via the catalog. Sorry folks for the back and forth. WDYT?

[lburgazzoli]

@lburgazzoli sounds good. How do we identify those banned dependencies in the first place?

I think we need to ask the quarkus team, @jamesnetherton @ppalaga any hint ?

[ppalaga]

I see several options:

1. Care that all Camel components have at least a JVM-only extension in Camel Quarkus. In their most basic form, the JVM-only extensions are 100% generated and have no tests (except for an autogenerated test that does nothing else than making sure that the component can be loaded at all - see an example here). We typically do not know about these basic JVM-only extensions whether they work, because they have no proper functional tests. We assume they work at least in JVM mode and through the fact that they are "thrown out to the wild" the folks using them can prove the opposite. JVM-only extensions are also listed in the Catalog and they are managed in the BOM. So Camel K would not need to get changed.

2. Use camel jars directly. As @christophd mentioned, they are not managed in Camel Quarkus BOM. You could perhaps auto-generate the missing entries in Camel K BOM by diffing Camel Quarkus Catalog against Camel Catalog at Camel K build time?

[claudio4j]

for dependencies not listed in the catalog is also not working because the raw Camel version is not resolvable in the generated Maven builder project. I see errors such as 'dependencies.dependency.version' for org.apache.camel:camel-resilience4j:jar is missing because there is no bom that resolves the raw Camel dependency.

In this case the <version> should be also added as part of the dependency, as the quarkus platform bom cannot resolve it.

[christophd]

@claudio4j yes, but what should be the proper version value? At the moment the operator will overwrite all user defined versions on Camel dependencies in order to force the versions coming from the bom. We would need to also manage the raw Camel version in the catalog to actually set the proper version

[christophd]

I think we can close this one in favor of using #4411 which keeps the logic to throw errors when a Camel dependency is not part of the catalog

[ppalaga]

what should be the proper version value?

For the record, the Camel version is present in Camel Quarkus catalog under metadata (which is a free form map):

{
  "component": {
    "kind": "component",
    "name": "amqp",
    "title": "AMQP",
...
    "metadata": { "quarkusVersion": "3.1.0.CR1", "camelArtifact": "org.apache.camel:camel-amqp", "camelVersion": "4.0.0-M3" },
...

Of course, there is no catalog entry for the missing components, but the camelVersion is the same for all components (at least in the community). So you could take the version from some foundation component, such as direct and use it for all missing camel jars.