Skip to content

Conduit connector able to generate sample records

License

Notifications You must be signed in to change notification settings

ConduitIO/conduit-connector-generator

Repository files navigation

Conduit Connector Generator

The generator connector is one of Conduit builtin plugins. It generates sample records using its source connector. It has no destination and trying to use that will result in an error.

How to build it

Run make.

Testing

Run make test to run all the unit tests.

Configuration

Important

Parameters starting with collections.* are used to configure the format and operations for a specific collection. The * in the parameter name should be replaced with the collection name.

Below is a list of all available configuration parameters:

Name Type Default Description

burst.generateTime

duration

1s

The amount of time the generator is generating records in a burst. Has an effect only if burst.sleepTime is set.

burst.sleepTime

duration

The time the generator "sleeps" between bursts.

collections.*.format.options.*

string

The options for the raw and structured format types. It accepts pairs of field names and field types, where the type can be one of: int, string, time, bool, duration.

collections.*.format.options.path

string

Path to the input file (only applicable if the format type is file).

collections.*.format.type

string

The format of the generated payload data (raw, structured, file).

collections.*.operations

string

create

Comma separated list of record operations to generate. Allowed values are "create", "update", "delete", "snapshot".

format.options.*

string

The options for the raw and structured format types. It accepts pairs of field names and field types, where the type can be one of: int, string, time, bool, duration.

format.options.path

string

Path to the input file (only applicable if the format type is file).

format.type

string

The format of the generated payload data (raw, structured, file).

operations

string

create

Comma separated list of record operations to generate. Allowed values are "create", "update", "delete", "snapshot".

rate

float

The maximum rate in records per second, at which records are generated (0 means no rate limit).

recordCount

int

Number of records to be generated (0 means infinite).

Examples

Bursts

The following configuration generates 100 records in bursts of 10 records each, with a 1 second sleep time between bursts.

Note

The generator currently has no concept of resuming work. For instance, below we have configured it to generate 100 records, but if we restart the pipeline (by stopping and starting the pipeline or by restarting Conduit), then it will start generating the 100 records from scratch.

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      - id: example
        type: source
        plugin: generator
        settings:
          # global settings
          rate: 10
          recordCount: 100
          burst.generateTime: 1s
          burst.sleepTime: 1s
          # default collection
          format.type: structured
          format.options.id: int
          format.options.name: string
          operations: create

Collections

The following configuration generates records forever with a steady rate of 1000 records per second. Records are generated in the users and orders collections. The generated records have a different format, depending on the collection they belong to.

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      - id: example
        type: source
        plugin: generator
        settings:
          # global settings
          rate: 1000
          # collection "users"
          collections.users.format.type: structured
          collections.users.format.options.id: int
          collections.users.format.options.name: string
          collections.users.operations: create
          # collection "orders"
          collections.orders.format.type: raw
          collections.orders.format.options.id: int
          collections.orders.format.options.product: string
          collections.orders.operations: create,update,delete