feat(generator): Ingesting Custom User Camel Components

[mhempleman]

• Processes component.json files from /custom-components directories on the classpath to add component models to the catalog
• Example component.json: https://github.com/apache/camel/blob/main/components/camel-activemq/src/generated/resources/META-INF/org/apache/camel/component/activemq/activemq.json.

Work in progress as this will require an associated change in vscode-kaoto. Without a vscode plugin update, users will not be able to run integrations that include custom components via vscode/jbang. Creating this PR to start a discussion on how to close that gap.

We do not use the vscode plugin and I have not had a chance to dig into that codebase, but I'm guessing this will be an easy update. To run an integration that includes a custom component via jbang:

camel run foo.yaml --deps=com.foo.bar:custom-component:1.0

So, assuming the user has a maven settings file configured to access the custom component repo, this might be as simple as adding the deps command line arg to jbang calls targeting routes that contain custom components.

Also, I see you're in the process of adding camel-catalog tests. I'll create a test for this feature once that initial PR is merged.

[lhein]

cc @apupier

[apupier]

As long as it stays with JBang, so not yet on a complete Maven project, there are several ways to specify dependencies when launching with JBang from the VS Code extension:

• specify parameter in File -> Preferences -> Settings then Camel > Debug Adapter: Extra launch Parameter
• provide an application.properties to provide the dependencies with camel.jbang.dependencies property and potentially a specific Maven repository camel.jbang.repositories https://camel.apache.org/manual/camel-jbang.html#_using_3rd_party_maven_repositories
• mention the additional dependencies using modeline

(and when moved to Maven, well just need to provide classical maven dependencies)

[lordrip]

Following on @apupier's comment, I don't think it will break vscode-kaoto per se, considering that if the user adds a custom component, it needs to be made available through one of the methods that he mentioned.

Other than that, how can we provide custom components? Should we point to a folder where the .json files will be located? or simply adding the dependency into the classpath the generator will identify them?

[mhempleman]

Following on @apupier's comment, I don't think it will break vscode-kaoto per se, considering that if the user adds a custom component, it needs to be made available through one of the methods that he mentioned.

Other than that, how can we provide custom components? Should we point to a folder where the .json files will be located? or simply adding the dependency into the classpath the generator will identify them?

The current configuration will look for /custom-component directories on the classpath (similar to how it handles Kamelets). We have CICD that builds several customs and deploys a jar containing the component.json files, but of course you could also create a /custom-component directory in /src/main/resources or anywhere else on the generator classpath. It might make sense to also have the generator look for direct component dependencies on the classpath. I'd be happy to add that if you like.

[mhempleman]

As long as it stays with JBang, so not yet on a complete Maven project, there are several ways to specify dependencies when launching with JBang from the VS Code extension:

• specify parameter in File -> Preferences -> Settings then Camel > Debug Adapter: Extra launch Parameter
• provide an application.properties to provide the dependencies with camel.jbang.dependencies property and potentially a specific Maven repository camel.jbang.repositories https://camel.apache.org/manual/camel-jbang.html#_using_3rd_party_maven_repositories
• mention the additional dependencies using modeline

(and when moved to Maven, well just need to provide classical maven dependencies)

Makes sense. I should have read through the vscode docs before posting the PR. Thanks for the info.

[lordrip]

It might make sense to also have the generator look for direct component dependencies on the classpath. I'd be happy to add that if you like.

Interesting, do you mean scanning all classes inherited from a Camel abstract component class?

[mhempleman]

It might make sense to also have the generator look for direct component dependencies on the classpath. I'd be happy to add that if you like.

Interesting, do you mean scanning all classes inherited from a Camel abstract component class?

That's a possibility. I was actually thinking of checking jars on the classpath for the component metadata file in META_INF.

[lordrip]

That's a possibility. I was actually thinking of checking jars on the classpath for the component metadata file in META_INF.

Ok, got it, and how are you deploying the jars to make them available in the class path? (Sorry for the noob question, but I'm not a Java dev myself 😅 ) I'm curious because I think that would be a great item flr the generator doc

[mhempleman]

That's a possibility. I was actually thinking of checking jars on the classpath for the component metadata file in META_INF.

Ok, got it, and how are you deploying the jars to make them available in the class path? (Sorry for the noob question, but I'm not a Java dev myself 😅 ) I'm curious because I think that would be a great item flr the generator doc

Good question. There are a few options and it depends on what makes sense for the user. You could add the custom component dependency to the generator pom file. That might make sense for a development team if they plan to add a relatively small number of custom components. However, it would require a pom update for each new custom and I'm guessing most devs would prefer a more dynamic process. For example, we have a custom components build that packages the component.json files in a jar, which is then consumed by the generator. Any new component we add to the components build will automatically be added to the UI. We could implement a similar approach for component jars. Ex... recursively search the classpath for custom component jars, so the user could just add custom component project directories to the generator classpath. Of course, the component jars could also be manually placed in src/main/resources. On the other hand, this may be overkill. I see no real advantage to the component jar approach and I think devs would be ok using the component.json files.

[lordrip]

That's a possibility. I was actually thinking of checking jars on the classpath for the component metadata file in META_INF.

Ok, got it, and how are you deploying the jars to make them available in the class path? (Sorry for the noob question, but I'm not a Java dev myself 😅 ) I'm curious because I think that would be a great item flr the generator doc

Good question. There are a few options and it depends on what makes sense for the user. You could add the custom component dependency to the generator pom file. That might make sense for a development team if they plan to add a relatively small number of custom components. However, it would require a pom update for each new custom and I'm guessing most devs would prefer a more dynamic process. For example, we have a custom components build that packages the component.json files in a jar, which is then consumed by the generator. Any new component we add to the components build will automatically be added to the UI. We could implement a similar approach for component jars. Ex... recursively search the classpath for custom component jars, so the user could just add custom component project directories to the generator classpath. Of course, the component jars could also be manually placed in src/main/resources. On the other hand, this may be overkill. I see no real advantage to the component jar approach and I think devs would be ok using the component.json files.

My guess would be that if JSON files are available, then it would be useful to point the generator to that folder and take them from there, I'm imagining a flag like:

generator -customComponent ./my component folder

Especially if we want to package the generator in a distributable binary.

Another option might be adding the component dependency directly, maybe this is more useful, perhaps like:

generator --comp-dep org.acme.mydep --comp-dep org.acme.anotherdep

Maybe we could receive the entire maven locator so it's easier to handle versioning as well. I think this approach could be simplified as --dep and then it could serve for components and kamelets as well.

[lhein]

I am not a web developer so take it with a grain of salt please but isn't the browser forbidden to access local files? So how could you even parse the classpath at runtime from the browser? That sounds like impossible.

[lordrip]

I am not a web developer so take it with a grain of salt please but isn't the browser forbidden to access local files? So how could you even parse the classpath at runtime from the browser? That sounds like impossible.

You're right, the browser won't parse the jar at runtime, I was pointing the jar to the generator, so when the user creates the catalog, it can also pass additional components or kamelets, since we can dynamically fetch dependencies with the maven plugin.

In case you need to dynamically do this on the flight, it's very likely this needs to happen on a host process, being in a backend or a VSCode host process that regenerates the catalog when the UI adds a new dependency.

That being said, it's also very likely that despite this change, we still need to add logic to VSCode to be able to read kamelets from the filesystem as well, so we can edit them and show them in the catalog.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[mhempleman]

That's a possibility. I was actually thinking of checking jars on the classpath for the component metadata file in META_INF.

Ok, got it, and how are you deploying the jars to make them available in the class path? (Sorry for the noob question, but I'm not a Java dev myself 😅 ) I'm curious because I think that would be a great item flr the generator doc

Good question. There are a few options and it depends on what makes sense for the user. You could add the custom component dependency to the generator pom file. That might make sense for a development team if they plan to add a relatively small number of custom components. However, it would require a pom update for each new custom and I'm guessing most devs would prefer a more dynamic process. For example, we have a custom components build that packages the component.json files in a jar, which is then consumed by the generator. Any new component we add to the components build will automatically be added to the UI. We could implement a similar approach for component jars. Ex... recursively search the classpath for custom component jars, so the user could just add custom component project directories to the generator classpath. Of course, the component jars could also be manually placed in src/main/resources. On the other hand, this may be overkill. I see no real advantage to the component jar approach and I think devs would be ok using the component.json files.

My guess would be that if JSON files are available, then it would be useful to point the generator to that folder and take them from there, I'm imagining a flag like:

generator -customComponent ./my component folder

Especially if we want to package the generator in a distributable binary.

Another option might be adding the component dependency directly, maybe this is more useful, perhaps like:

generator --comp-dep org.acme.mydep --comp-dep org.acme.anotherdep

Maybe we could receive the entire maven locator so it's easier to handle versioning as well. I think this approach could be simplified as --dep and then it could serve for components and kamelets as well.

Sorry for the delay. I've been out of commission the last few days.

My guess would be that if JSON files are available, then it would be useful to point the generator to that folder and take them from there, I'm imagining a flag like:

Sure, makes sense. Our use case of pulling in JSON files via a dependency jar may not be the norm.

Another option might be adding the component dependency directly, maybe this is more useful, perhaps like:

My only concern here is that the dependency-based option could be cumbersome if a user wants to load several deps (although this might not matter if the cmd line is generated by vscode or something else). I definitely agree it makes sense to provide the full locator. Eg... org.acme.mydep:1.0.0.

I propose we implement the component folder option for the initial PR. In a subsequent PR we can introduce the dependency-based approach.

[mhempleman]

I am not a web developer so take it with a grain of salt please but isn't the browser forbidden to access local files? So how could you even parse the classpath at runtime from the browser? That sounds like impossible.

You're right, the browser won't parse the jar at runtime, I was pointing the jar to the generator, so when the user creates the catalog, it can also pass additional components or kamelets, since we can dynamically fetch dependencies with the maven plugin.

In case you need to dynamically do this on the flight, it's very likely this needs to happen on a host process, being in a backend or a VSCode host process that regenerates the catalog when the UI adds a new dependency.

That being said, it's also very likely that despite this change, we still need to add logic to VSCode to be able to read kamelets from the filesystem as well, so we can edit them and show them in the catalog.

Yes, since we're not using vscode, we've been discussing adding a light-weight backend to manage the catalog and provide filesystem info to the UI.

[lordrip]

Sorry for the delay. I've been out of commission the last few days.

Hi @mhempleman , sure no worries at all 👍

I propose we implement the component folder option for the initial PR. In a subsequent PR we can introduce the dependency-based approach.

Sure, but before we do it, and just to confirm, with your current approach in this PR, if someone wants to provide a custom component, how will they do it? if I understand correctly, you will provide the dependency in the classpath and that should do the trick, is that correct?

So, let's say I want to do it, I would need to add the custom component dependency to the pom.xml file of the catalog generator and then execute the build, is that?

If that's the case, then I think we're not too far from implementing the dependency topic, sure, we can do it in a subsequent PR, but I'm trying to understand where we are at the moment

Once again, thanks for your time handling this 🙏

[mhempleman]

Sorry for the delay. I've been out of commission the last few days.

Hi @mhempleman , sure no worries at all 👍

I propose we implement the component folder option for the initial PR. In a subsequent PR we can introduce the dependency-based approach.

Sure, but before we do it, and just to confirm, with your current approach in this PR, if someone wants to provide a custom component, how will they do it? if I understand correctly, you will provide the dependency in the classpath and that should do the trick, is that correct?

So, let's say I want to do it, I would need to add the custom component dependency to the pom.xml file of the catalog generator and then execute the build, is that?

If that's the case, then I think we're not too far from implementing the dependency topic, sure, we can do it in a subsequent PR, but I'm trying to understand where we are at the moment

Once again, thanks for your time handling this 🙏

Sure, no problem. The short answer is yes. I can update the implementation to add any components found on the classpath to the catalog.

I think we should also support the json file approach. The current implementation in this PR loads any component.json files found in any /custom-components directory on the classpath. That works for our use case because we bundle our component.json files in one jar, which is added as a dependency to the catalog generator. Incidentally, that's also how Kaoto handles custom Kamelets. It loads any .kamelet.yaml files found in any /kamelet directory on the classpath.

However, for core Kaoto, your suggestion might make more sense. Ex... I want to add several custom components to the catalog, so I put all the associated component.json files in a directory and point the generator at that directory:

generator -customComponent ./myComponentFolder

I'm happy to implement whatever you think makes sense for the general Kaoto user. If the dependency approach is sufficient for core Kaoto, I'll maintain the component.json approach downstream.

[lordrip]

Hi @mhempleman, I think I got it now. I think we can move forward with your current approach, this should put us in a position in which you can use the upstream Kaoto catalog generator if you want to, while also benefiting other users to add their custom components as well.

On the other hand, I'll try to check how the Camel maven plugin packages custom components, once that's clear, we can proceed to implement loading components from jar dependencies specified through the CLI, I think having this possibility, we won't need to implement reading from a folder in the filesystem.

That being said, for Kamelets, I think it is another story because while we want to load the standard Kamelets catalog, we also would like to read Kamelets from the filesystem (not implemented yet) and from dependencies (not implemented yet), so we can edit them while testing them with the Camel CLI.

[mhempleman]

Hi @mhempleman, I think I got it now. I think we can move forward with your current approach, this should put us in a position in which you can use the upstream Kaoto catalog generator if you want to, while also benefiting other users to add their custom components as well.

On the other hand, I'll try to check how the Camel maven plugin packages custom components, once that's clear, we can proceed to implement loading components from jar dependencies specified through the CLI, I think having this possibility, we won't need to implement reading from a folder in the filesystem.

That being said, for Kamelets, I think it is another story because while we want to load the standard Kamelets catalog, we also would like to read Kamelets from the filesystem (not implemented yet) and from dependencies (not implemented yet), so we can edit them while testing them with the Camel CLI.

Hi Lordip. I apologize for the wait. I have been focused on another project (also just back from vacation). I should be able to get to this next week. I'll be in touch.

[lordrip]

Hi @mhempleman, no worries at all, having some days to relax is always good.

There have been a few minor changes in the project, mostly a few bug fixes in the catalog generator and we brought the tests we had in the previous package plus I added other ones for the command. If you have the chance, it would be nice to sync with latest main.