Generate records manually

[derek1ee]

Goals

1. As a streaming data engineer, I can manually produce a message to a Kafka topic using the VS Code extension, so that I can test the Kafka consumer client I'm working on.
2. As a streaming data engineer, I can manually "re-produce" a message by selecting an existing message on the topic, so that I can test the change in consumer client with exactly the same payload.
3. As a streaming data engineer, the produce a message experience is schema-aware to help me craft the right message, so I can be more efficient and be sure of the message is correctly formatted.
4. As a streaming data engineer, I can manually produce messages from a collection of pre-authored messages, so that I can test my consumer client with different test cases.

Non-Goals

1. Automatically / programmatically produce messages, which is covered by Automated Data Generation #174
2. Unit/E2E test of clients.

Background

When authoring a Kafka consumer client, it's common for the developer to first make sure the client is properly configured and can receive messages from the Kafka topic. For a Kafka topic that's not yet configured with producers, this is done by manually producing a message.

As more logics are added to the consumer client, the developer will want to test the logic. If the code didn't work, then the developer would fix the code, then rerun the test the same message ("re-produce" the message).

As the code becomes more mature, the developer may build up a collection of messages the code should be tested against, e.g., messages with different payload for different code path, malformed messages for error handling, etc.

Manually producing messages to Kafka topic can be done today in a few different ways:

1. Write a producer client. While this can be done in the same IDE the user uses to author the consumer client, this require not insignificant work just to produce test messages. It can also cause false negatives - is the consumer client not receiving the message because it's not configured properly, or is it because the producer client isn't sending the messages correctly?
2. Produce a message dialog on Confluent Cloud's topic message viewer. This approach is much easier to use and requires little setup, but it requires the user to context-switch and use both the IDE and browser. It is also not available to users of Apache Kafka.
3. kafka-console-producer or Confluent CLI to produce message from the command line. This can be done in the terminal window of the same IDE, but requires the separate installation of CLI tools. To "re-produce" a message, while possible, will requires piping different commands.

Proposal

This proposal adds one or more additional ways to manually produce messages to a Kafka topic:

1. Integrated experience in topic message viewer.
   1. User can choose to produce a message from the topic message viewer, here we already have the context of which topic they want the message to go, and can also check if any schema is available.
   2. Since user also sees existing messages in this view, we can also allow them to select an existing message(s) to "re-produce" it.
2. Produce message as a command
   1. The command is what the topic message viewer but can also surface via the resource tree on topics, or CMD+Shift+P.
   2. When invoking command directly from CMD+Shift+P, we will need to know which topic user intend to produce the massage to, a search/filter experience would be nice here for users with a lot of resources.
3. File-based approach.
   1. While 1 & 2 can work well for producing a single message, it may not be optimized for sending various messages for different test cases. A file-baed approach can allow users to store multiple messages in the file, and selectively choose which one to use.
   2. A good reference on how we can support this with a file-based approach is REST Client, where user can have multiple REST API calls in a file marked type HTTP, and the extension is the file handler for such type and can overlay controls such as "send request", or "produce" in our case.

[rhauch]

Thanks for kicking off this discussion, @derek1ee. This is really a great start.

First, one question about this goal, as stated above:

2. As a streaming data engineer, I can manually "re-produce" a message by selecting an existing message on the topic, so that I can test the change in consumer client with exactly the same payload.

If a developer is working on and testing a consumer application, then if the topic already has messages can't the developer simply rerun their consumer to start from an earlier offset? Sure, the developer may want to manually produce atypical messages to test the consumer application's codepaths, but that seems more in line with the first goal. Plus, goal 1 seems to be the driving objective (at least initially).

Second, as you point out in the "Proposal" section, there are multiple places where the ability to manually produce one (or more) messages can be surfaced. I might argue that producing messages via a command is the underlying common mechanism that can be exposed in several ways, as you suggest: from command palette, on a particular topic, as an action in the message viewer, as a copy action on a message (or set of messages) followed by a paste action or "Send messages to..." action a topic, etc. One challenge is that those different calls to action will each have quite different contexts about the topic, schema, or even sample messages to start from.

Third, your mention of the value of a file-based approach and the REST Client extension is interesting. If we can define a file format that allows specifying the message contents (key, value, plus optionally schema refs, headers, timestamps, partition numbers) then we could start simple and expand. For example, if our first command was "send messages to topic..." and the input was a file of messages, then in a first release users could hand-craft these files (possibly checking them into their git repo) and easily produce messages. Of course hand-crafting is not ideal and doesn't quite solve goal 3, but it would achieve goal 1 and maybe 4. Plus, subsequent releases could make progress on goal 3 by adding other options, such as by copying messages from message viewer and pasting into a messages file. Or using a form to add more messages. Or generating "template" messages in a file based upon a schema. Or ....

The existing Kafka REST API supports producing records with an already-defined JSON structure for the produce REST request payloads. See our docs and the API reference. That API supports multiple records via the “Transfer-Encoding: chunked” header, which means we could have multiple records in each file. The structure already supports all of the optional attributes like partition number, timestamp, headers, etc. And the REST API request structure already accepts the JSON representations of Avro and Protobuf formatted records.

Having JSON files with one or more records would be useful, but we might also consider combining this pure JSON representation with the REST Client file's approach of including the HTTP method and URL. Could we just reuse the REST file format (which is used by multiple tools), but have our extension interpret the URL appropriately using the connection information?

###

POST https://<REST-endpoint>/kafka/v3/clusters/<cluster-id>/topics/<topic-name>/records HTTP/1.1
Content-Type: application/json
Transfer-Encoding: chunked
{
  // record 1  
},
{
  // record 2  
}

One big advantage is that other tooling could use the same already-in-use file formats, such as for testing.

Anyway, great start to the discussion!

[derek1ee]

If a developer is working on and testing a consumer application, then if the topic already has messages can't the developer simply rerun their consumer to start from an earlier offset? Sure, the developer may want to manually produce atypical messages to test the consumer application's codepaths, but that seems more in line with the first goal. Plus, goal 1 seems to be the driving objective (at least initially).

I agree, I think Goal #2 is nice-to-have, and lower priority than Goal #1. As we look at the design, I can also imagine Goal #2 may be accomplished via something like, working in conjunction with the file-based appraoch:

1. Go to message viewer
2. Double click on a message to open the message I want to reproduce.
3. The new tab we open for the selected message, is already in the format that I can just manually produce. This would then afford me the ability to either reproduce as-is, or tweak the data first.
4. Because it's already in the expected format, I can then also save it in my project for use later.

[derek1ee]

Notes from offline conversation:

• If topic has no schema, then any valid json can be produced to the topic.
• If topic has schema, then any valid json can be produced to the topic if choose “produce without schema”
• If topic has schema, and schema is selected, then payload must pass schema validation.
• If schema is not json, then json payload is serialized to the proper format, validated, then produced.
• If a schema is selected, a sample payload is generated as placeholder.

[derek1ee]

For file-based produce, another extension, Tools for Apache Kafka has a very similar approach to what's proposed here. I think it'd be worth taking a look and see if we want to be compatible with it.