Recomendation of not deriving a consumer from a producer is problematic for most users #931
Comments
|
Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request. |
|
this change you are referring to was added because of this issue -> #538 so it is all about the purpose of the AsyncAPI document. If you will create this document for your application with application generation in mind, you will make options are to:
what do you recommend? |
|
if this is just about the naming of the I find the following code perfectly fine to read and understand (which is what the other issue is worried about): You are not reacting to an
but even IF someone find that code confusing or difficult to understand, I would much rather to live with that than creating N different APIs for the same message because when you publish a DomainEvent/Message on Event-Driven Architectures there will be potentially many consumers. For instance if you have 5 consumers you will end up with 6 different API definitions for that message, 5 of them identicals... and linking to potentially protected resources.. And editing, dereferencing and generating code with protected resources is a pain... What worries me about this recomendation is its character of an That is my main worry. Maybe this calls for an article exploring both solutions pros and cons, I would be happy to contribute one of the sides... |
|
hi @fmvilas, tagging you to join the conversation.. |
|
What about "summary" and "description" fields? From the recommendation you linked:
And also the last paragraph:
Remember this is a recommendation. You don't have to follow strictly if that's not suiting your needs. We will anyway offer tools to derive one from another to make the process a bit easier to handle when needed. |
|
I think that naming (summary and description) is a minor issue comparing with ending up with N+1 (cuasi)duplicated APIs referencing each other. (For instance, following description fits both sides: And the infraestructure problem when a channel is read-only is a very corner case that can be dealt individually ("With this API you can not derive a consumer because..."). What troubles me is that RECOMENDATIONS in the official documentation will affect conversations on this topic about the convenience or not of "deriving consumer from producer" without mentioning the list of (IMO big) CONS:
What about creating a blog post or page explaining PROS/CONS of each option instead? |
|
by the way: by "deriving" I understand use the same API definition on the other side just inverting the meaning, not replicating it replacing "send" with "receive" |
|
I'm definitely up for clarifying stuff on blog posts or documentation. It is a recommendation because we don't want to enforce how people should do it. If you want to derive it and it's fine for you, go ahead and do it. We just noted a really bad experience when doing it in the past. It usually leads to poor documentation because you have to be careful not to say send/produce/receive/subscribe because that introduces confusion. In a distributed architecture, every node is different, and usually, a consumer of your API is not just a consumer of your API but also a consumer of other APIs and a producer of events itself as well. So actually having an AsyncAPI document per node makes total sense. That said, we don't want people to repeat themselves. We encourage you to have another AsyncAPI file but to use Maybe we should clarify that? |
|
Also just wanted to jump in, because we discussed this subject in some of the past v3 meetings. I agree that if you ONLY have two parties, where these two parties are the ONLY ones who interact with each other, then yes, it becomes slightly cumbersome for that use-case, i.e. the recommended way is to define two AsyncAPI documents. But it does not outweigh the benefits in my eyes. What's hard is if you are coming from using OpenAPI, where you have a single file, explaining both the server and the client behavior or rather what the client can perform on the server, it will be a bit of a learning curve. What makes this setup amazing is you can have a server exposing HTTP (or any other protocol that exposes an endpoint) endpoints, while it also interacts with Kafka as a producer (or consumer), all defined within a single AsyncAPI document, i.e. it defines the behavior of that server. Then you can have an AsyncAPI document explaining the external application and how it may interact with the HTTP endpoint in the above server. It of course does NOT need to know about the Kafka interaction, because it only interacts with the HTTP endpoint. Taking that a step further, if you have two users of your endpoint (admin and user for example), they can each have their own AsyncAPI document, explaining how they interact with the server. Starting to mix in deriving sets of actions from AsyncAPI documents that describe the behavior of someone else just creates the same set of confusion that publish/subscribe was, it's the whole reason v3 was put into the world (among other things of course). It's true that tooling MUST be there to support this setup, but I think it will be without a doubt. I am definitely not saying it's a silver bullet, but I personally would say this setup is better than what we had in v2. But people might also say I am slightly biased 😆 This is after I started integrating the v3 structure into my projects.
This has nothing to do with the structure changes in the spec? It's a general problem also for v2. v3 might just emphasize the problem even further. Tooling will get there, faster the more people help with providing tooling 🙂
They do not, channels can be reused (i.e. you can figure out the ownership of topics how you wish), only actions are application specific 😄
Just emphasizing the first section I wrote, I agree that if you ONLY have two parties, where these two parties are the ONLY ones who interact with each other with no exception, then yes, it becomes slightly cumbersome for that use-case to be described with AsyncAPI. But as @fmvilas said, no one is stopping you from deriving meaning as you see fit 🙂 |
|
What if you can describe Well, you can! External And what is troubling me about the official recomendation (which is written as NOT RECOMENDED in capital letter) is the following: -> It will make it harder to explain to unaware clients that you don't actually need external references at all... |
Can you clarify what you mean here? Cause to me this does not sound like something you can do with a single AsyncAPI document (regardless of v2 or 3), regardless of whether you derive the opposite behavior of an application. Some have an AsyncAPI document as a lookup of all common definitions, but that's the closest you come to this? Cause what exactly are you describing then? 😄 Or do you see it as you have a mono-AsyncAPI document, that contains the behavior of all underlying applications within your company/system? i.e. when you then generate code, it's basically a library every application should use, even though it contains far more operations then what a single application should be able to perform?
Definitely agree yea 👍 And tooling are not at all where it should be to support these use-cases (hopefully this year, lets see 😉) |
|
Yes, I would like to write a small blog post for this, but this is how it goes in a nutshell:
Now, for code generation, when an application wants to interact with 3rd party topics/messages it will ask the code generator to generate a Naming conventions:
(Async data request can be modeled as a Command+Event like doRequest/onResponse) And that's it. I implemented this on a quite large company since early 2020 and I believe it's still in production.. And this is my recommended aproach. |
|
I'm a bit lost now. Are you talking about a new format you invented on top of AsyncAPI? Can you come up with a simple example so that we're all aligned?
Well that's because we're following RFC2119 syntax. It's in the beginning of the spec:
That's why is in all caps, to make it visible that it's part of these special words. In particular, NOT RECOMMENDED is defined as follows:
Which I think it's perfect for that case. We're not saying you can't (for that we use MUST NOT), we're saying that we don't recommend it but you're free to still do it if it's the preferred way in your case.
I also agree but it's mainly a problem with tooling. It's tooling that should catch up, not the other way around. Can you imagine that someone suggests that the |
|
I'm talking about proper AsyncAPI definition files, nothing special on top.. One AsyncAPI file per application describing only the topics/messages this application is the When applications need to We got a code generator for SpringCloud Streams that can generate code following this perspective of roles, but nothing special about AsyncAPI definitions files, just propper and 100% compatible v2 or v3 asyncapi.yml. I would stay away from external $refs as much as posible because:
But more over because they are an unnecesary trouble. ZenWave SDK is the opensource succesor of that battle tested code generator. Will try to explay in the next comment a simple example about how this works... PS: I still see oportunity in v3 for external $refs for aggregating application channels from existing APIS, but that is another matter... |
|
(This is an example that uses Java/SpringBoot but the asyncapi.yml or the technique is not special about Java) Let's think we have a "Customers Service" that accepts commands for creating/updating/deleting customers and informs via domain events when a customer is created/updated/deleted. Two channels:
Customers service provides the following asyncapi: customers-asyncapi.yml (expand to see)asyncapi: 2.6.0
info:
title: Customers Service API
version: 0.0.1
defaultContentType: application/json
tags:
- name: "Customer"
channels:
customer.requests:
subscribe:
summary: Accepts Customer Commands to create/update/delete a Customer
operationId: doCustomerRequest
tags:
- name: Customer
message:
$ref: "#/components/messages/CustomerRequestMessage"
customer.events:
publish:
summary: Informs about Customer created/updated/deleted Domain Events
operationId: onCustomerEvent
tags:
- name: Customer
message:
$ref: "#/components/messages/CustomerEventMessage"
components:
messages:
CustomerRequestMessage:
messageId: CustomerRequestMessage
name: CustomerRequestMessage
title: Async Command/Request for a Customer
summary: Async Command/Request for a Customer
schemaFormat: application/vnd.aai.asyncapi;version=2.6.0
traits:
- $ref: '#/components/messageTraits/CommonHeaders'
payload:
$ref: "#/components/schemas/CustomerRequestPayload"
CustomerEventMessage:
name: CustomerEventMessage
messageId: CustomerEventMessage
title: Message for a Customer Event
summary: Message for a Customer Event
schemaFormat: application/vnd.aai.asyncapi;version=2.6.0
traits:
- $ref: '#/components/messageTraits/CommonHeaders'
payload:
$ref: "#/components/schemas/CustomerEventPayload"
messageTraits:
CommonHeaders:
headers:
type: object
properties:
kafka_messageKey:
type: string
description: This header value will be populated automatically at runtime
x-runtime-expression: $message.payload#/id
tracingId:
type: string
description: This header value will be populated automatically at runtime
x-runtime-expression: $tracingIdSupplier
schemas:
Customer:
type: "object"
x-business-entity: "Customer"
required:
- "username"
- "password"
- "email"
- "firstName"
- "lastName"
properties:
id:
type: "string"
username:
type: "string"
minLength: 3
maxLength: 250
password:
type: "string"
minLength: 3
maxLength: 250
email:
type: "string"
minLength: 3
maxLength: 250
firstName:
type: "string"
minLength: 3
maxLength: 250
lastName:
type: "string"
minLength: 3
maxLength: 250
CustomerRequestPayload:
type: object
properties:
requestType:
type: string
enum:
- "create"
- "update"
- "delete"
customer:
$ref: "#/components/schemas/Customer"
CustomerEventPayload:
type: object
properties:
id:
type: string
eventType:
type: string
enum:
- "created"
- "updated"
- "deleted"
customer:
$ref: "#/components/schemas/Customer"
Customers ServiceCode generation for Customer Service (see <!-- Generate PROVIDER -->
<inputSpec>${pom.basedir}/src/main/resources/model/customers-asyncapi.yml</inputSpec>
[...]
<execution>
<id>generate-asyncapi</id>
<phase>generate-sources</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<generatorName>spring-cloud-streams3</generatorName>
<configOptions>
<role>provider</role><!-- IMPORTANT ROLE -->
<style>imperative</style>
<transactionalOutbox>mongodb</transactionalOutbox>
<modelPackage>io.zenwave360.example.core.domain.events</modelPackage>
<producerApiPackage>io.zenwave360.example.core.outbound.events</producerApiPackage>
<consumerApiPackage>io.zenwave360.example.adapters.commands</consumerApiPackage>
</configOptions>
</configuration>
</execution>This will generate:
This is how it looks some of the generated code: // Autogenerated: you can @Autowire it in your code
public interface ICustomerEventsProducer {
// headers object omitted for brevity
/**
* Customer Domain Events
*/
boolean onCustomerEvent(CustomerEventPayload payload, CustomerEventPayloadHeaders headers);
}See ZenWave AsyncAPI Generated Code in a Nutshell for all the details about how generated code looks like and can be used. Other ApplicationsAny other application that wants to send "Customer Command" for creating/updating/deleting customers or want to be informed about changes in customer will do the following: Code generation for any other service that wants be a client of Customers Service: <!-- Generate CLIENT -->
<inputSpec>${pom.basedir}/src/main/resources/model/customers-asyncapi.yml</inputSpec>
[...]
<execution>
<id>generate-asyncapi</id>
<phase>generate-sources</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<generatorName>spring-cloud-streams3</generatorName>
<configOptions>
<role>client</role><!-- IMPORTANT ROLE -->
<style>imperative</style>
<transactionalOutbox>mongodb</transactionalOutbox>
<modelPackage>io.zenwave360.example.core.domain.events</modelPackage>
[...]
</configOptions>
</configuration>
</execution>This will generate:
Company WideYou can extend this model about how applications use each others APIs without duplication or external $refs This project has executable examples for most combinations, that I use for testing releases (so it may not be as clear as examples) |
As far as I can understand this is the root of this discussion, this is a design pattern that you implemented and prefer, and with reason to back it up. Where on the opposite side each application has an AsyncAPI document, that defines its behavior (this includes how the public can interact with your system). This seems to be the focus and direction AsyncAPI moves, and why I guess it grinds a little bit when it comes to the recommendation section for you 🙂 With your approach, you are forcing the need of With the recommended approach, you would not need the option in the code generator, and its straightforward what the operation keywords mean, instead of having this switch between what they mean and which context. I.e. creating confusion. This is the entire reason I love this approach, cause there is no confusion about the meanings of the operation keywords and other perspectives through implementation details. What you see is exactly what the application does, nothing more nothing less. You can of course still use your design pattern and the parameter in the code generator 👍 Maybe it makes more sense as a I personally don't think that design pattern unlocks AsyncAPI to its full potential, i.e. why I am also in favor of keeping that recommendation 🙂
In v3 a channel definition can be a reference to an external channel definition. Unless you mean something like inheriting channels from other applications? Nwm, feel free to start another discussion about that, let's not mix more stuff into the discussion 😆 |
|
This is not about the tooling, because zw can generate code for both aproaches in v2 and v3 already, it is about what we recommend AsyncAPI users (mostly new/unaware/unexperienced users). The root recomendation tries to prevent the following two problems for users:
But when linking to external resources, it introduces these two (imo major) problems:
May only be me but I think that the second two problems are way bigger for end users than the first ones... Let's do an exercise, let's say that there are 2 application "Customers Service" and another one that will use (produce and consume) "customer.request" and "customer.events"
https://gist.github.com/ivangsa/a1c7fa242ee9f38319deeb0e1dfd0991
Sounds fair? My take on this is to replace the current recomendation with a detailed explanation of both aproaches, probably linking to a different page or blog post with pros, cons and examples... With that I'm also happy stating that the offical recomendation is A, as long as B gets also explained... (My worry is trying to explain that "external $refs are trouble" to some users that will not look behind an official recomendation, not the recomendation on itself) |
|
Well at least now we got a shared understanding that it's possible to document all event-driven interactions without duplication or external $refs, right? I think this is something that we didn't have before.. |
To me it doesn't no, cause you have nothing that explains who interacts with your Customers Service, it's one big black box. It solves your immediate problem with not having a second AsyncAPI document but you limit yourself and start introducing other problems (such as discoverability, documentation, fine-graned control). I think there is definitely a reason to keep something like
Not exactly, and let me show you why 😄 Here you go: https://gist.github.com/jonaslagoni/66958036cb28af76dc1efe33dbf21d37. Please ignore spelling mistakes, and I bet there is some wording that's incorrect, so take it as an example 😄 Take notice namings of operations and descriptions and how straightforward it is to know what it means for the application itself.
All of these AsyncAPI documents are located within the same repository, where they have full access to relative files With this approach and design pattern, you achieve the full power of AsyncAPI:
As I bet you are thinking I skipped completely over your problem statement with references and authentication, but that's the underlying setup I have that allows me to do that. Problems with references and authentication are that it's mainly tool-specific, something that should be setup at runtime i.e. with environment variables, etc. You could include tokens etc as part of the URL, and even do a pre-action that replaces placeholders with environment variables, etc. For this, we definitely need better tooling. We 100% agree here 😉 Because if each repository had an AsyncAPI document, and needed to reference across GitHub repositories you would most probably run into a lot of issues 😄
The problem here is your implementation details start to creep into the spec itself, cause without ALL tooling implementations having |
|
Yes, you skiped all together the main concerns which is I'm the only one seeing this? Let's document both aproaches and let users decide for themselves.. |
|
I want to clarify that I also see these downsides. What I don't agree with is to remove this recommendation. I think the spec should recommend you to use it this way. IMHO, we should document the two approaches as part of the docs on the website but not as part of the spec. |
|
ok, but maybe we should warn users about the downsides with external refs preferably next to this recomentation on the spec.. |
|
anyway, did we get a shared understanding that you can document all interactions with AsyncAPI without duplication or references (v2 or v3) that we didn't have before, right? I implemented this approach succesfully in production for a quite large company for a number of years without problems about "naming" or "readonly topics" (I understand the official approach but I read the contents of the recomendation and it doesn't sound like the reason for the official recomendation) |
|
thanks to this conversation I learn a lot about my thought process and AsyncAPI publish/subscribe was never a confusion to me, that's why I manage to propose and succesfully adopt AsyncAPI without duplicaton or external $refs
Scenario: there are 100 application that each one publish a different event to a different topic, no one subscribes to anything. There will be 100 asyncapi.yml without crossed references (nothing confusing here). Application 101 kicks in and publishes one event but it also would like to subscribe to all other 100 messages.
(I preffer this thoughts not to be buried here, but until I get to write them somwhere else. This is just food for thought) |
I don't think so. The spec is full of things that can be done in multiple ways, each with pros and cons. The pros and cons in this case have more to do with the state of tooling than the spec itself. That warning that you want to put there doesn't belong in the spec but in the docs.
For sure. It's a cool approach that can serve multiple people and should be documented. Definitely not in the spec but in the docs.
Yeah, the thing is that using
I'm glad it was never a confusion for you but what we found since the very beginning of the spec is quite the opposite. We tried hard to educate people on that point of view but damn, the confusion was coming again and again for everyone. So yeah, it's a valid point of view but it's definitely not preferred by the majority. In fact, the world is now full of AsyncAPI files using publish and subscribe the wrong way 😅 As @jonaslagoni said above, this was the reason that triggered us to release v3 in the first place.
I'm glad you're going through all the thought process that we did during the last year. It's refreshing to see that people care about it. Here are some issues that are related to these thoughts:
Please 🙏 keep them coming. And make sure you're heard and it's properly tracked and discussed in its own issue. The spec is not set in stone and things can always evolve. |
|
This issue has been automatically marked as stale because it has not had recent activity 😴 It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation. There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model. Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here. Thank you for your patience ❤️ |
https://github.com/asyncapi/spec/blob/next-major-spec/spec/asyncapi.md?plain=1#L41
I think this official recommendation introduces more problems to users than it really solves.
This is a very corner case that doesn't apply to the great majority of cases and it will send most users to a rabbit hole of problems that tooling (IDEs, generators…) are not prepared to assist them with.
If you can not derive a consumer from a producer, who are you writing the producer API for? A producer writing an API just for himself doesn't sound very useful...
Then users will need to:
That is already some extra work, but linking to external resources is very problematic for two reasons:
(I expect most users are going to be linking to a git http interface)
Regarding naming, I think the example is backwards: in this example you can describe the producer and naming should be fine for consumers as well. Using the API to describe what the exchanged messages mean, not what actions consumers will perform with them.
I think consuming an event/message is analogous to reading data from a REST API, you don't describe what every consumer will do with that data, just what the data is (and its meaning).
How is asyncapi tooling prepared to assist users with this?
The text was updated successfully, but these errors were encountered: