Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(new sink): aws sqs #2755

Closed

Conversation

francescop
Copy link
Contributor

This introduce AWS SQS as sink.
Closes #2138

Signed-off-by: palazzini francesco <palazzini.francesco@gmail.com>
Signed-off-by: palazzini francesco <palazzini.francesco@gmail.com>
Signed-off-by: palazzini francesco <palazzini.francesco@gmail.com>
Signed-off-by: palazzini francesco <palazzini.francesco@gmail.com>
@fanatid
Copy link
Contributor

fanatid commented Jun 7, 2020

Thank you for PR!

For fixing make generate you need add "sink: aws_sqs" to:

  • .meta/.schemas/guides.json
  • .meta/.schemas/highlights.json
Patch:

diff --git a/.meta/.schemas/guides.json b/.meta/.schemas/guides.json
index 7a7494f1..d2882a80 100644
--- a/.meta/.schemas/guides.json
+++ b/.meta/.schemas/guides.json
@@ -52,7 +52,7 @@
       "description": "Array of tags",
       "items": {
         "type": "string",
-        "enum": ["type: guide","type: post","type: tutorial","domain: buffers","domain: cli","domain: config","domain: dependencies","domain: networking","domain: observability","domain: operations","domain: platforms","domain: privacy","domain: security","domain: sinks","domain: sources","domain: tests","domain: topology","domain: transforms","domain: ux","domain: website","platform: arm64","platform: armv7","platform: amazon-linux","platform: centos","platform: debian","platform: docker","platform: dpkg","platform: homebrew","platform: macos","platform: msi","platform: nix","platform: nixos","platform: raspbian","platform: rhel","platform: rpm","platform: ubuntu","platform: windows","platform: x86_64","source: docker","source: file","source: generator","source: http","source: journald","source: kafka","source: logplex","source: prometheus","source: socket","source: splunk_hec","source: statsd","source: stdin","source: syslog","source: vector","transform: add_fields","transform: add_tags","transform: ansi_stripper","transform: aws_ec2_metadata","transform: coercer","transform: concat","transform: dedupe","transform: filter","transform: geoip","transform: grok_parser","transform: json_parser","transform: log_to_metric","transform: logfmt_parser","transform: lua","transform: merge","transform: regex_parser","transform: remove_fields","transform: remove_tags","transform: rename_fields","transform: sampler","transform: split","transform: swimlanes","transform: tag_cardinality_limit","transform: tokenizer","sink: aws_cloudwatch_logs","sink: aws_cloudwatch_metrics","sink: aws_kinesis_firehose","sink: aws_kinesis_streams","sink: aws_s3","sink: blackhole","sink: clickhouse","sink: console","sink: datadog_logs","sink: datadog_metrics","sink: elasticsearch","sink: file","sink: gcp_cloud_storage","sink: gcp_pubsub","sink: gcp_stackdriver_logs","sink: honeycomb","sink: http","sink: humio_logs","sink: influxdb_logs","sink: influxdb_metrics","sink: kafka","sink: logdna","sink: loki","sink: new_relic_logs","sink: papertrail","sink: prometheus","sink: pulsar","sink: sematext_logs","sink: socket","sink: splunk_hec","sink: statsd","sink: vector"]
+        "enum": ["type: guide","type: post","type: tutorial","domain: buffers","domain: cli","domain: config","domain: dependencies","domain: networking","domain: observability","domain: operations","domain: platforms","domain: privacy","domain: security","domain: sinks","domain: sources","domain: tests","domain: topology","domain: transforms","domain: ux","domain: website","platform: arm64","platform: armv7","platform: amazon-linux","platform: centos","platform: debian","platform: docker","platform: dpkg","platform: homebrew","platform: macos","platform: msi","platform: nix","platform: nixos","platform: raspbian","platform: rhel","platform: rpm","platform: ubuntu","platform: windows","platform: x86_64","source: docker","source: file","source: generator","source: http","source: journald","source: kafka","source: logplex","source: prometheus","source: socket","source: splunk_hec","source: statsd","source: stdin","source: syslog","source: vector","transform: add_fields","transform: add_tags","transform: ansi_stripper","transform: aws_ec2_metadata","transform: coercer","transform: concat","transform: dedupe","transform: filter","transform: geoip","transform: grok_parser","transform: json_parser","transform: log_to_metric","transform: logfmt_parser","transform: lua","transform: merge","transform: regex_parser","transform: remove_fields","transform: remove_tags","transform: rename_fields","transform: sampler","transform: split","transform: swimlanes","transform: tag_cardinality_limit","transform: tokenizer","sink: aws_cloudwatch_logs","sink: aws_cloudwatch_metrics","sink: aws_kinesis_firehose","sink: aws_kinesis_streams","sink: aws_s3","sink: aws_sqs","sink: blackhole","sink: clickhouse","sink: console","sink: datadog_logs","sink: datadog_metrics","sink: elasticsearch","sink: file","sink: gcp_cloud_storage","sink: gcp_pubsub","sink: gcp_stackdriver_logs","sink: honeycomb","sink: http","sink: humio_logs","sink: influxdb_logs","sink: influxdb_metrics","sink: kafka","sink: logdna","sink: loki","sink: new_relic_logs","sink: papertrail","sink: prometheus","sink: pulsar","sink: sematext_logs","sink: socket","sink: splunk_hec","sink: statsd","sink: vector"]
       }
     },
     "title": {
diff --git a/.meta/.schemas/highlights.json b/.meta/.schemas/highlights.json
index b478187a..88c16f4d 100644
--- a/.meta/.schemas/highlights.json
+++ b/.meta/.schemas/highlights.json
@@ -61,7 +61,7 @@
       "description": "Array of tags",
       "items": {
         "type": "string",
-        "enum": ["sink: tcp","source: tcp","source: udp","type: announcement","type: enhancement","type: new feature","type: performance","type: breaking change","domain: buffers","domain: cli","domain: config","domain: dependencies","domain: networking","domain: observability","domain: operations","domain: platforms","domain: privacy","domain: security","domain: sinks","domain: sources","domain: tests","domain: topology","domain: transforms","domain: ux","domain: website","event type: log","event type: metric","platform: arm64","platform: armv7","platform: amazon-linux","platform: centos","platform: debian","platform: docker","platform: dpkg","platform: homebrew","platform: macos","platform: msi","platform: nix","platform: nixos","platform: raspbian","platform: rhel","platform: rpm","platform: ubuntu","platform: windows","platform: x86_64","provider: aws","provider: confluent","provider: heroku","provider: splunk","provider: yandex","provider: datadog","provider: azure","provider: elastic","provider: gcp","provider: honeycomb","provider: humio","provider: influxdata","provider: logdna","provider: grafana","provider: new relic","provider: papertrail","provider: sematext","source: docker","source: file","source: generator","source: http","source: journald","source: kafka","source: logplex","source: prometheus","source: socket","source: splunk_hec","source: statsd","source: stdin","source: syslog","source: vector","source: kubernetes","transform: add_fields","transform: add_tags","transform: ansi_stripper","transform: aws_ec2_metadata","transform: coercer","transform: concat","transform: dedupe","transform: filter","transform: geoip","transform: grok_parser","transform: json_parser","transform: log_to_metric","transform: logfmt_parser","transform: lua","transform: merge","transform: regex_parser","transform: remove_fields","transform: remove_tags","transform: rename_fields","transform: sampler","transform: split","transform: swimlanes","transform: tag_cardinality_limit","transform: tokenizer","sink: aws_cloudwatch_logs","sink: aws_cloudwatch_metrics","sink: aws_kinesis_firehose","sink: aws_kinesis_streams","sink: aws_s3","sink: blackhole","sink: clickhouse","sink: console","sink: datadog_logs","sink: datadog_metrics","sink: elasticsearch","sink: file","sink: gcp_cloud_storage","sink: gcp_pubsub","sink: gcp_stackdriver_logs","sink: honeycomb","sink: http","sink: humio_logs","sink: influxdb_logs","sink: influxdb_metrics","sink: kafka","sink: logdna","sink: loki","sink: new_relic_logs","sink: papertrail","sink: prometheus","sink: pulsar","sink: sematext_logs","sink: socket","sink: splunk_hec","sink: statsd","sink: vector"]
+        "enum": ["sink: tcp","source: tcp","source: udp","type: announcement","type: enhancement","type: new feature","type: performance","type: breaking change","domain: buffers","domain: cli","domain: config","domain: dependencies","domain: networking","domain: observability","domain: operations","domain: platforms","domain: privacy","domain: security","domain: sinks","domain: sources","domain: tests","domain: topology","domain: transforms","domain: ux","domain: website","event type: log","event type: metric","platform: arm64","platform: armv7","platform: amazon-linux","platform: centos","platform: debian","platform: docker","platform: dpkg","platform: homebrew","platform: macos","platform: msi","platform: nix","platform: nixos","platform: raspbian","platform: rhel","platform: rpm","platform: ubuntu","platform: windows","platform: x86_64","provider: aws","provider: confluent","provider: heroku","provider: splunk","provider: yandex","provider: datadog","provider: azure","provider: elastic","provider: gcp","provider: honeycomb","provider: humio","provider: influxdata","provider: logdna","provider: grafana","provider: new relic","provider: papertrail","provider: sematext","source: docker","source: file","source: generator","source: http","source: journald","source: kafka","source: logplex","source: prometheus","source: socket","source: splunk_hec","source: statsd","source: stdin","source: syslog","source: vector","source: kubernetes","transform: add_fields","transform: add_tags","transform: ansi_stripper","transform: aws_ec2_metadata","transform: coercer","transform: concat","transform: dedupe","transform: filter","transform: geoip","transform: grok_parser","transform: json_parser","transform: log_to_metric","transform: logfmt_parser","transform: lua","transform: merge","transform: regex_parser","transform: remove_fields","transform: remove_tags","transform: rename_fields","transform: sampler","transform: split","transform: swimlanes","transform: tag_cardinality_limit","transform: tokenizer","sink: aws_cloudwatch_logs","sink: aws_cloudwatch_metrics","sink: aws_kinesis_firehose","sink: aws_kinesis_streams","sink: aws_s3","sink: aws_sqs","sink: blackhole","sink: clickhouse","sink: console","sink: datadog_logs","sink: datadog_metrics","sink: elasticsearch","sink: file","sink: gcp_cloud_storage","sink: gcp_pubsub","sink: gcp_stackdriver_logs","sink: honeycomb","sink: http","sink: humio_logs","sink: influxdb_logs","sink: influxdb_metrics","sink: kafka","sink: logdna","sink: loki","sink: new_relic_logs","sink: papertrail","sink: prometheus","sink: pulsar","sink: sematext_logs","sink: socket","sink: splunk_hec","sink: statsd","sink: vector"]
       }
     },
     "title": {

@fanatid
Copy link
Contributor

fanatid commented Jun 8, 2020

I honestly do not know how fix ruby code, but as workaround can propose add new line manually. make generate do not remove it later.

@francescop francescop force-pushed the feat/new-sinks/aws-sqs branch from 6034a2d to d8d5815 Compare June 8, 2020 13:15
Signed-off-by: palazzini francesco <palazzini.francesco@gmail.com>
@francescop francescop force-pushed the feat/new-sinks/aws-sqs branch from d8d5815 to 4809932 Compare June 8, 2020 13:28
@bruceg bruceg added domain: sinks Anything related to the Vector's sinks provider: aws Anything `aws` service provider related type: new feature labels Jun 8, 2020
@francescop francescop marked this pull request as ready for review June 8, 2020 15:52
@francescop francescop requested a review from binarylogic as a code owner June 8, 2020 15:52
@binarylogic binarylogic added type: feature A value-adding code addition that introduce new functionality. and removed type: new feature labels Jun 16, 2020
@binarylogic binarylogic self-assigned this Jul 21, 2020
@jamtur01 jamtur01 removed this from the 2020-09-14 - The Grid milestone Sep 28, 2020
@jamtur01 jamtur01 assigned juchiast and fanatid and unassigned binarylogic and juchiast Sep 29, 2020
@fanatid
Copy link
Contributor

fanatid commented Sep 30, 2020

@francescop sorry for the delay... from the last comment rusoto was upgraded to an async supported version and we introduced a lot of async changes. Would you able to update PR?

@binarylogic binarylogic added this to the 2020-10-12: Son of Flynn milestone Oct 11, 2020
fanatid added a commit that referenced this pull request Oct 21, 2020
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
fanatid added a commit that referenced this pull request Oct 21, 2020
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
@fanatid fanatid mentioned this pull request Oct 21, 2020
@fanatid
Copy link
Contributor

fanatid commented Oct 21, 2020

Moved to #4675

@fanatid fanatid closed this Oct 21, 2020
@fanatid fanatid added the sink: aws_sqs Anything `aws_sqs` sink related label Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: sinks Anything related to the Vector's sinks provider: aws Anything `aws` service provider related sink: aws_sqs Anything `aws_sqs` sink related type: feature A value-adding code addition that introduce new functionality.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New aws_sqs sink
6 participants