CC-29601, CC-29566, CC-29504: CVE Fixes #705

vbalani002 · 2024-10-20T13:43:28Z

CC-29504 : CVE-2024-47561: Bump kafka-connect-storage-common to update org.apacha.avro:avro version to v1.11.4
CC-29566 Bump kafka-connect-storage-common to update jetty-server to 9.4.56.v20240826
CC-29601: Bump velocity-engine-core to v2.4.0 as it declares commons-io as a dependency and results in a false positive for CVE-2024-47554

Testing with docker playground

>17:41:49 ℹ️ 🚦 containers have started!
17:41:49 ℹ️ 📊 JMX metrics are available locally on those ports:
17:41:49 ℹ️     - zookeeper       : 9999
17:41:49 ℹ️     - broker          : 10000
17:41:49 ℹ️     - schema-registry : 10001
17:41:49 ℹ️     - connect         : 10002
17:42:03 ℹ️ Creating HDFS Sink connector
17:42:11 ℹ️ 🛠️ Creating 🌎onprem connector hdfs-sink
17:42:12 ℹ️ 📋 🌎onprem connector config has been copied to the clipboard (disable with 'playground config set clipboard false')
17:42:13 ℹ️ ✅ 🌎onprem connector hdfs-sink was successfully created
17:42:16 ℹ️ 🧰 Current config for 🌎onprem connector hdfs-sink (using REST API /config endpoint)
playground connector create-or-update --connector hdfs-sink --no-clipboard << EOF
{
  "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
  "flush.size": "3",
  "hadoop.conf.dir": "/etc/hadoop/",
  "hive.database": "testhive",
  "hive.integration": "true",
  "hive.metastore.uris": "thrift://hive-metastore:9083",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "logs.dir": "/tmp",
  "name": "hdfs-sink",
  "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
  "rotate.interval.ms": "120000",
  "schema.compatibility": "BACKWARD",
  "store.url": "hdfs://namenode:8020",
  "tasks.max": "1",
  "topics": "test_hdfs",
  "value.converter": "io.confluent.connect.avro.AvroConverter",
  "value.converter.schema.registry.url": "http://schema-registry:8081"
}
EOF
17:42:22 ℹ️ 🔩 list of all available parameters for 🌎onprem connector hdfs-sink (org.apache.kafka.connect.mirror.MirrorSourceConnector) and version 7.6.1-ce (with default value when applicable)
    "allow.optional.map.keys": "false",
    "avro.codec": "",
    "connect.hdfs.keytab": "STRING",
    "connect.hdfs.principal": "STRING",
    "connect.meta.data": "true",
    "directory.delim": "/",
    "enhanced.avro.schema.support": "true",
    "file.delim": "+",
    "filename.offset.zero.pad.width": "10",
    "flush.size": "",
    "format.class": "io.confluent.connect.hdfs.avro.AvroFormat",
    "hadoop.conf.dir": "STRING",
    "hadoop.home": "STRING",
    "hdfs.authentication.kerberos": "false",
    "hdfs.namenode.principal": "STRING",
    "hdfs.url": "",
    "hive.conf.dir": "STRING",
    "hive.database": "default",
    "hive.home": "STRING",
    "hive.integration": "false",
    "hive.metastore.uris": "STRING",
    "hive.table.name": "${topic}",
    "kerberos.ticket.renew.period.ms": "3600000",
    "locale": "STRING",
    "logs.dir": "logs",
    "partition.duration.ms": "-1",
    "partition.field.name": "LIST",
    "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
    "path.format": "STRING",
    "retry.backoff.ms": "5000",
    "rotate.interval.ms": "-1",
    "rotate.schedule.interval.ms": "-1",
    "schema.compatibility": "NONE",
    "schemas.cache.config": "1000",
    "shutdown.timeout.ms": "3000",
    "storage.class": "io.confluent.connect.hdfs.storage.HdfsStorage",
    "store.url": "",
    "timestamp.extractor": "Wallclock",
    "timestamp.field": "timestamp",
    "timezone": "STRING",
    "topic.capture.groups.regex": "",
    "topics.dir": "topics",
17:42:23 ℹ️ 🥁 Waiting a few seconds to get new status
17:42:29 ℹ️ 🧩 Displaying status for 🌎onprem connector hdfs-sink
Name                           Status       Tasks                                                        Stack Trace
-------------------------------------------------------------------------------------------------------------
hdfs-sink                      ✅ RUNNING  0:🟢 RUNNING[connect]        -
-------------------------------------------------------------------------------------------------------------
17:42:31 ℹ️ 🌐 documentation for 🌎onprem connector kafka-connect-hdfs is available at:
https://docs.confluent.io/kafka-connect-hdfs/current/index.html
17:42:32 ℹ️ Sending messages to topic test_hdfs
17:42:36 ℹ️ 🔮 value schema was identified as avro
17:42:36 ℹ️ ✨ generating value data...
17:42:36 ℹ️ ☢️ --forced-value is set
17:42:36 ℹ️ ✨ 10 records were generated based on --forced-value  (only showing first 10), took: 0min 0sec
{"f1":"value1"}
{"f1":"value2"}
{"f1":"value3"}
{"f1":"value4"}
{"f1":"value5"}
{"f1":"value6"}
{"f1":"value7"}
{"f1":"value8"}
{"f1":"value9"}
{"f1":"value10"}
17:42:43 ℹ️ 📤 producing 10 records to topic test_hdfs
17:42:49 ℹ️ 📤 produced 10 records to topic test_hdfs, took: 0min 6sec
17:42:59 ℹ️ Listing content of /topics/test_hdfs/partition=0 in HDFS
Found 3 items
-rw-r--r--   3 appuser supergroup        213 2024-10-21 12:12 /topics/test_hdfs/partition=0/test_hdfs+0+0000000000+0000000002.avro
-rw-r--r--   3 appuser supergroup        213 2024-10-21 12:12 /topics/test_hdfs/partition=0/test_hdfs+0+0000000003+0000000005.avro
-rw-r--r--   3 appuser supergroup        213 2024-10-21 12:12 /topics/test_hdfs/partition=0/test_hdfs+0+0000000006+0000000008.avro
17:43:02 ℹ️ Getting one of the avro files locally and displaying content with avro-tools
Successfully copied 2.05kB to /tmp/
{"f1":"value1"}
{"f1":"value2"}
{"f1":"value3"}
17:43:07 ℹ️ Check data with beeline
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 2.3.2 by Apache Hive
beeline> !connect jdbc:hive2://hive-server:10000/testhive
Connecting to jdbc:hive2://hive-server:10000/testhive
Enter username for jdbc:hive2://hive-server:10000/testhive: hive
Enter password for jdbc:hive2://hive-server:10000/testhive: ****
Connected to: Apache Hive (version 2.3.2)
Driver: Hive JDBC (version 2.3.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://hive-server:10000/testhive> show create table test_hdfs;
+----------------------------------------------------+
|                   createtab_stmt                   |
+----------------------------------------------------+
| CREATE EXTERNAL TABLE `test_hdfs`(                 |
|   `f1` string COMMENT '')                          |
| PARTITIONED BY (                                   |
|   `partition` string COMMENT '')                   |
| ROW FORMAT SERDE                                   |
|   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'   |
| STORED AS INPUTFORMAT                              |
|   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'  |
| OUTPUTFORMAT                                       |
|   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' |
| LOCATION                                           |
|   'hdfs://namenode:8020/topics/test_hdfs'          |
| TBLPROPERTIES (                                    |
|   'avro.schema.literal'='{"type":"record","name":"ConnectDefault","namespace":"io.confluent.connect.avro","fields":[{"name":"f1","type":"string"}]}',  |
|   'transient_lastDdlTime'='1729512770')            |
+----------------------------------------------------+
15 rows selected (1.986 seconds)
0: jdbc:hive2://hive-server:10000/testhive> select * from test_hdfs;
+---------------+----------------------+
| test_hdfs.f1  | test_hdfs.partition  |
+---------------+----------------------+
| value1        | 0                    |
| value2        | 0                    |
| value3        | 0                    |
| value4        | 0                    |
| value5        | 0                    |
| value6        | 0                    |
| value7        | 0                    |
| value8        | 0                    |
| value9        | 0                    |
+---------------+----------------------+
9 rows selected (2.965 seconds)
0: jdbc:hive2://hive-server:10000/testhive> Closing: 0: jdbc:hive2://hive-server:10000/testhive
| value1        | 0                    |
17:43:16 ℹ️ ####################################################
17:43:16 ℹ️ ✅ RESULT: SUCCESS for hdfs2-sink.sh (took: 12min 44sec - )
17:43:16 ℹ️ ####################################################

17:43:23 ℹ️ 🧩 Displaying status for 🌎onprem connector hdfs-sink
Name                           Status       Tasks                                                        Stack Trace
-------------------------------------------------------------------------------------------------------------
hdfs-sink                      ✅ RUNNING  0:🟢 RUNNING[connect]        -
-------------------------------------------------------------------------------------------------------------
17:43:25 ℹ️ 🌐 documentation is available at:
https://docs.confluent.io/current/connect/kafka-connect-hdfs/index.html
17:43:27 ℹ️ 🎯 Version currently used for confluent platform

If yes, where?

Test Strategy

Testing done:

Unit tests
Integration tests
System tests
Manual tests

Release Plan

confluent-cla-assistant · 2024-10-20T13:43:40Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

sonarqube-confluent · 2024-10-21T11:59:41Z

Analysis Details

0 Issues

0 Bugs
0 Vulnerabilities
0 Code Smells

Coverage and Duplications

No coverage information (72.70% Estimated after merge)
No duplication information (1.50% Estimated after merge)

Project ID: kafka-connect-hdfs

View in SonarQube

bump deps versions

353f544

vbalani002 requested a review from a team as a code owner October 20, 2024 13:43

vbalani002 changed the base branch from master to 10.2.x October 20, 2024 13:43

This comment has been minimized.

Sign in to view

update parent pom

5cdcdcf

vbalani002 changed the title ~~CC-29601, CC-29504: CVE Fixes~~ CC-29601, CC-29566, CC-29504: CVE Fixes Oct 21, 2024

sumit-confluent approved these changes Oct 21, 2024

View reviewed changes

vbalani002 merged commit 0722591 into 10.2.x Oct 21, 2024
2 checks passed

vbalani002 deleted the cve-fixes-20241020 branch October 21, 2024 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CC-29601, CC-29566, CC-29504: CVE Fixes #705

CC-29601, CC-29566, CC-29504: CVE Fixes #705

vbalani002 commented Oct 20, 2024 •

edited

Loading

confluent-cla-assistant bot commented Oct 20, 2024

This comment has been minimized.

sonarqube-confluent bot commented Oct 21, 2024

CC-29601, CC-29566, CC-29504: CVE Fixes #705

CC-29601, CC-29566, CC-29504: CVE Fixes #705

Conversation

vbalani002 commented Oct 20, 2024 • edited Loading

Testing with docker playground

If yes, where?

Test Strategy

Testing done:

Release Plan

confluent-cla-assistant bot commented Oct 20, 2024

This comment has been minimized.

sonarqube-confluent bot commented Oct 21, 2024

Analysis Details

0 Issues

Coverage and Duplications

vbalani002 commented Oct 20, 2024 •

edited

Loading