Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OBSDATA-3156] Apply Confluent Patches on top of Druid 28.0.1 #171

Merged
merged 115 commits into from
Dec 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
115 commits
Select commit Hold shift + click to select a range
494f436
Bring dockerfile up to date
Jun 13, 2019
160b350
add opencensus extension
arrawatia May 17, 2019
fc03683
make checkstyle happy
xvrl Jul 26, 2019
40a4525
bump pom version for opencensus extension
xvrl Sep 5, 2019
f497af4
fix issues related to shading opencensus extension
xvrl Dec 10, 2019
1a07ece
METRICS-516: Adding Resource labels in OpenCensus Extension
apoorvmittal10 Feb 14, 2020
bb77616
bump extension version to match release
xvrl Mar 13, 2020
226f2b6
confluent-extensions with custom transform specs (#9)
xvrl Mar 24, 2020
8db469e
fix extraction transform serde (#10)
xvrl Mar 26, 2020
fe3a1f1
fix check-style build errors
hqin Mar 26, 2020
e69b515
setup semaphore build
xvrl Mar 26, 2020
f514bce
add checkstyle
xvrl Mar 26, 2020
78e523e
fix edge cases for internal topics
xvrl Mar 27, 2020
419828a
METRICS-1302: Added prefix support for resource labels. (#14)
apoorvmittal10 Apr 24, 2020
2ca99d7
Added getters for configs, labels for distribution metric. (#15)
apoorvmittal10 May 5, 2020
d810b3b
Default resource label prefix to blank - Backward Compatibility (#16)
apoorvmittal10 May 7, 2020
cf5fe23
update opencensus parent pom version
xvrl Aug 18, 2020
aee6405
update opencensus extensions for 0.19.x
xvrl Aug 18, 2020
473919f
update parent pom version for confluent-extensions
xvrl Aug 18, 2020
a3701f2
Add the capability to speed up S3 uploads using AWS transfer manager
hqin Sep 11, 2020
ace65fc
fix conflicting protobuf dependencies
xvrl Dec 21, 2020
b8e43ed
fix timestamp milliseconds in OpenCensusProtobufInputRowParser
xvrl Dec 16, 2020
3b4ebba
improve OpenCensusProtobufInputRowParser performance (#25)
xvrl Dec 22, 2020
d4b391a
deprecate OpenCensusInputRowParser in favor of OpenCensusProtobufInpu…
xvrl Jan 5, 2021
d23f01d
add default query context and update timeout to 30 sec
CodingParsley Feb 5, 2021
87b5b3e
Setting default query lane from druid console.
harinirajendran Feb 24, 2021
f0cb050
Giving more heap space for test jvm in semaphore config.
harinirajendran Apr 22, 2021
8321fc9
update parent pom version for Confluent extensions
harinirajendran Apr 27, 2021
0d3d3ac
Add Java 11 image build and remove unused MySQL images
xvrl Apr 27, 2021
bc2e92c
fix docker image build failure caused by #10506
xvrl Apr 27, 2021
4b77396
switch build to use Java 11 by default
xvrl Apr 27, 2021
d337dbe
Fixed forbiddenapi error
Sep 7, 2021
4be5076
Added phases before checks
Sep 8, 2021
042ed10
Fixed
Sep 14, 2021
0b08463
OpenTelemetry Emitter Extension (#47)
IvanVan Oct 23, 2021
357d13a
Add dependency check (#59)
IvanVan Dec 7, 2021
22958a5
METRICS-3663 OpenTelemetry Metrics InputFormat (#63)
marcusgreer Feb 4, 2022
6df767b
update parent pom version for Confluent extensions
harinirajendran Feb 22, 2022
005253e
Adding getRequiredColumns() in our custom transforms.
harinirajendran Feb 23, 2022
1045c94
Updating shade-plugin version in opentelemetry-emitter.
harinirajendran Feb 23, 2022
4b0bbff
Removing the unwanted maven-shade-plugin change.
harinirajendran Feb 24, 2022
5d0a038
Adding JDK version to DockerFile and removing unwanted executions fro…
harinirajendran Feb 25, 2022
0a44da2
Passing JDK_VERSION as build args to docker build. (#76)
harinirajendran Feb 26, 2022
0590763
Make the OpenTelemetry InputFormat More Flexible to Metric, Value and…
marcusgreer Feb 25, 2022
c57532c
Hybrid OpenCensusProtobufInputFormat in opencensus-extensions (#69)
freyayunfu Feb 28, 2022
151cea5
Modify the OpenTelemetry ProtobufReader's Handling of Attribute Types…
marcusgreer Mar 1, 2022
8d5a6d8
Fixing Opencensus extension build failures.
freyayunfu Mar 1, 2022
6ed684c
fix dependency check (#79)
marcusgreer Mar 1, 2022
08400b6
fix OpenTelemetry extension module service definition (#73) (#81)
marcusgreer Mar 3, 2022
469f36e
Setting default refresh value for task view as none. (#88)
lokesh-lingarajan Mar 18, 2022
1fa3096
go/codeowners: Generate CODEOWNERS [ci skip] (#87)
lyoung-confluent Mar 24, 2022
ba28327
fixes in pom.xml files
kkonstantine Sep 1, 2022
1872925
adapt to new input argument in ParseException
kkonstantine Sep 1, 2022
4c9a884
adapt to the new constructor for DimensionsSpec
kkonstantine Sep 1, 2022
ed78e9d
update obs-data team as codeownders (#98)
freyayunfu Jun 3, 2022
9ebd2f5
[OBSDATA-334] Patch opencensus/opentelemetry parse exception (#99)
CodingParsley Jul 11, 2022
e123f2e
[METRICS-4487] add obs-oncall as codeowners (#101)
freyayunfu Jul 14, 2022
89502f6
DP-8085 - Migrate to Sempahore self-hosted agent (#100)
nlou9 Jul 20, 2022
ba07c32
[OBSDATA-334] Patch opentelemetry IllegalStateException for unsupport…
CodingParsley Jul 27, 2022
a1c82a7
Fixing checkstyle issues in openncensus and opentelemetry extensions.…
harinirajendran Sep 16, 2022
f644e7a
Remove SNAPSHOT from versions in confluent pom files
kkonstantine Oct 26, 2022
4cd4223
Fixing CI/CD in 24.0.0 upgrade branch (#116)
harinirajendran Oct 31, 2022
35b47ad
OBSDATA-440 Adding SegmentMetadataEvent and publishing them via Kafka…
harinirajendran Nov 2, 2022
0268082
Change unsupported type message from WARN to TRACE (#119)
CodingParsley Nov 2, 2022
df617fc
Use place holder for logging invalid format (#120)
CodingParsley Nov 2, 2022
f6cf31d
DP-9370 - use cc-service-bot to manage Semaphore project (#118)
cchristous Nov 3, 2022
b7a178c
chore: update repo semaphore project
ConfluentJenkins Nov 23, 2022
10fc98a
DP-9632: remediate duplicate Semaphore workflows (#121)
cchristous Nov 30, 2022
ba98b82
chore: update repo semaphore project
ConfluentJenkins Dec 1, 2022
a736288
Bump version to 24.0.1 in confluent extensions after rebasing on top …
kkonstantine Dec 13, 2022
a2cbcce
Bump version to 24.0.2 in confluent extensions after rebasing on top …
kkonstantine Jan 3, 2023
9ac69f9
OBSDATA-483: Adapt OpenCensus and OpenTelemetry extensions to the int…
kkonstantine Jan 5, 2023
e3d5fc2
chore: update repo semaphore project (#124)
ConfluentTools Jan 26, 2023
e55570e
[Metrics-4776] OpenTelemetry Extensions - Upgrade otel-proto version …
kamal-narayan Feb 6, 2023
f94d116
Move to Java 17 (#128)
hampycapper Feb 13, 2023
d2d6b14
Trying to add required flags to run druid using java 17 (#130)
hampycapper Feb 24, 2023
3f72b50
Use apache-jar-resource-bundle:1.5 instead of 1.5-SNAPSHOT (#14054) (…
kkonstantine Apr 17, 2023
2977114
update parent pom version for Confluent extensions
m-ghazanfar May 29, 2023
15d0449
Fix CI/CD while upgrading to Druid 25.0.0
m-ghazanfar May 30, 2023
f16f72c
Fix jest and prettify checks
m-ghazanfar May 30, 2023
db8bfbf
Adding SegmentMetadataEvent and publishing them via KafkaEmitter (#14…
harinirajendran Jun 8, 2023
317b8cc
Downgrade busybox version to fix k8s IT (#14518) (#143)
pagrawal10 Jul 7, 2023
8ebbde2
Passing TARGETARCH in build_args to Docker build (#144)
pagrawal10 Jul 10, 2023
a6932fb
remove docker-maven-plugin and Dockerfile customizations
xvrl Jul 19, 2023
8a628e3
Revert "Trying to add required flags to run druid using java 17 (#130…
xvrl Jul 21, 2023
92d2402
removing use of semaphore cache as the public semaphore will not have…
xvrl Jul 24, 2023
30cf450
OBSDATA-1365: add support for debian based base images (#149)
hardikbajaj Jul 25, 2023
d5cf05b
Revert "fix KafkaInputFormat with nested columns by delegating to und…
hardikbajaj Aug 11, 2023
9aa75cf
Filter Out Metrics with NoRecordedValue Flag Set (#157)
Sep 27, 2023
90b7809
memcached cache: switch to AWS elasticache-java-cluster-client and ad…
pagrawal10 Oct 3, 2023
c1f42b9
PRSP-3603 Bump org.xerial.snappy:snappy-java to latest version to add…
KeerthanaSrikanth Nov 9, 2023
c0535a5
[backport] Upgrade Avro to latest version (#14440) (#162)
KeerthanaSrikanth Nov 14, 2023
02a00d7
Revert "PRSP-3603 Bump org.xerial.snappy:snappy-java to latest versio…
KeerthanaSrikanth Nov 15, 2023
50ba53c
Upgrade Avro to latest version to address CVEs (#167)
KeerthanaSrikanth Nov 15, 2023
17d52ea
OBSDATA-1697: Do not build extensions not loaded by cc-druid (#152)
janjwerner-confluent Nov 22, 2023
5fdd6b9
update parent pom version for Confluent extensions
pagrawal10 Nov 27, 2023
38bf7e2
Add value to child POMs
pagrawal10 Nov 28, 2023
dac8ac6
Upgrade dependencies to match upstream v28 & checkstyle fix
mustajibmk Nov 28, 2023
ee74d31
KafkaEmitter changes
pagrawal10 Dec 4, 2023
6103506
Modifying RowFunction interface
pagrawal10 Dec 4, 2023
b939580
Fix test cases
Pankaj260100 Dec 4, 2023
95efc4f
Fix test cases
Pankaj260100 Dec 6, 2023
d62e7c7
Fix test cases
Pankaj260100 Dec 6, 2023
21e5724
Fix test cases
Pankaj260100 Dec 6, 2023
a98c3ba
upgrade dependency as per druid 28
Pankaj260100 Dec 8, 2023
d97a5ca
Removing unnecessary change
Pankaj260100 Dec 10, 2023
a2cf23f
Change Maven repository URL
pagrawal10 Dec 12, 2023
13f48b2
Add Druid.xml
pagrawal10 Dec 12, 2023
d00dac5
Update tag name to match version
pagrawal10 Dec 12, 2023
f0b9016
Fix dist-used profile to use Hadoop compile version (#173)
KeerthanaSrikanth Dec 12, 2023
f29c646
Merge branch '28.0.1-confluent' into OBSDATA-3156-druid-28-with-cflt-…
pagrawal10 Dec 14, 2023
da6ef5f
Changes based on PR comments
pagrawal10 Dec 14, 2023
6fe02b4
Fix refreshButton
pagrawal10 Dec 14, 2023
e684a57
Use onRefresh only once
pagrawal10 Dec 14, 2023
0d8b0e0
Fix snapshot so that the test passes
pagrawal10 Dec 15, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* @confluentinc/obs-data @confluentinc/obs-oncall
42 changes: 42 additions & 0 deletions .semaphore/project.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# This file is managed by ServiceBot plugin - Semaphore. The content in this file is created using a common
# template and configurations in service.yml.
# Modifications in this file will be overwritten by generated content in the nightly run.
# For more information, please refer to the page:
# https://confluentinc.atlassian.net/wiki/spaces/Foundations/pages/2871296194/Add+SemaphoreCI
apiVersion: v1alpha
kind: Project
metadata:
name: druid
description: ""
spec:
visibility: private
repository:
url: git@github.com:confluentinc/druid.git
run_on:
- branches
- tags
- pull_requests
pipeline_file: .semaphore/semaphore.yml
integration_type: github_app
status:
pipeline_files:
- path: .semaphore/semaphore.yml
level: pipeline
whitelist:
branches:
- master
- /^.*-confluent$/
custom_permissions: true
debug_permissions:
- empty
- default_branch
- non_default_branch
- pull_request
- forked_pull_request
- tag
attach_permissions:
- default_branch
- non_default_branch
- pull_request
- forked_pull_request
- tag
143 changes: 143 additions & 0 deletions .semaphore/semaphore.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
version: v1.0
name: Apache Druid
agent:
machine:
type: s1-prod-ubuntu20-04-amd64-1
execution_time_limit:
hours: 3
blocks:
- name: "Install"
task:
env_vars: &env_vars
- name: MVN
value: "mvn -B"
- name: MAVEN_OPTS
value: "-Dmaven.repo.local=.m2"
- name: MAVEN_SKIP
value: >
-Danimal.sniffer.skip=true
-Dcheckstyle.skip=true
-Ddruid.console.skip=true
-Denforcer.skip=true
-Dforbiddenapis.skip=true
-Dmaven.javadoc.skip=true
-Dpmd.skip=true
-Dspotbugs.skip=true

- name: MAVEN_SKIP_TESTS
value: "-DskipTests -Djacoco.skip=true"
prologue:
commands:
- echo $SEMAPHORE_WORKFLOW_ID
- sem-version java 17
- checkout
jobs:
- name: "Install"
commands:
# This is a change meant to validate semaphore public builds
# so thus removing configurations for Confluent's internal CodeArtifact
- rm ~/.m2/settings.xml
- >
MAVEN_OPTS="${MAVEN_OPTS} -Xmx3000m" ${MVN} clean install
-q -ff ${MAVEN_SKIP} ${MAVEN_SKIP_TESTS} -T1C
# downstream tests depend on artifacts installed by mvn install into .m2
# also cache target to avoid the cost of recompiling tests
- tar zcf cache-post-install.tgz .m2 target
- artifact push workflow cache-post-install.tgz

- name: "Tests"
task:
env_vars: *env_vars
prologue:
commands:
- echo $SEMAPHORE_WORKFLOW_ID
- sem-version java 17
- checkout
- artifact pull workflow cache-post-install.tgz
- tar zxf cache-post-install.tgz
# This is a change meant to validate semaphore public builds
# so thus removing configurations for Confluent's internal CodeArtifact
- rm ~/.m2/settings.xml
jobs:
- name: "animal sniffer checks"
commands:
- ${MVN} test-compile ${MAVEN_SKIP} ${MAVEN_SKIP_TESTS}
- ${MVN} animal-sniffer:check --fail-at-end

- name: "checkstyle"
commands:
- ${MVN} checkstyle:checkstyle --fail-at-end

- name: "enforcer checks"
commands:
- ${MVN} enforcer:enforce --fail-at-end

- name: "forbidden api checks"
commands:
- ${MVN} test-compile ${MAVEN_SKIP} ${MAVEN_SKIP_TESTS}
- ${MVN} forbiddenapis:check forbiddenapis:testCheck --fail-at-end

- name: "pmd checks"
commands:
- ${MVN} pmd:check --fail-at-end # TODO: consider adding pmd:cpd-check

- name: "spotbugs checks"
commands:
- ${MVN} spotbugs:check --fail-at-end -pl '!benchmarks'

- name: "analyze dependencies"
commands:
- >
${MVN} ${MAVEN_SKIP} dependency:analyze -DoutputXML=true -DignoreNonCompile=true -DfailOnWarning=true --fail-at-end || { echo "

The dependency analysis has found a dependency that is either:
1) Used and undeclared: These are available as a transitive dependency but should be explicitly
added to the POM to ensure the dependency version. The XML to add the dependencies to the POM is
shown above.
2) Unused and declared: These are not needed and removing them from the POM will speed up the build
and reduce the artifact size. The dependencies to remove are shown above.
If there are false positive dependency analysis warnings, they can be suppressed:
https://maven.apache.org/plugins/maven-dependency-plugin/analyze-mojo.html#usedDependencies
https://maven.apache.org/plugins/maven-dependency-plugin/examples/exclude-dependencies-from-dependency-analysis.html
For more information, refer to:
https://maven.apache.org/plugins/maven-dependency-plugin/analyze-mojo.html
" && false; }

- name: "Confluent Extensions"
env_vars:
- name: MAVEN_PROJECTS
value: extensions-contrib/confluent-extensions
commands: &run_tests
- >
MAVEN_OPTS="${MAVEN_OPTS} -Xmx1g" ${MVN} test -pl ${MAVEN_PROJECTS}
${MAVEN_SKIP} -Dremoteresources.skip=true

- name: "Server"
env_vars:
- name: MAVEN_PROJECTS
value: server
commands: *run_tests

- name: "Processing"
env_vars:
- name: MAVEN_PROJECTS
value: processing
commands: *run_tests

- name: "Indexing Service"
env_vars:
- name: MAVEN_PROJECTS
value: indexing-service
commands: *run_tests

- name: "Kafka Indexing Service"
env_vars:
- name: MAVEN_PROJECTS
value: extensions-core/kafka-indexing-service
commands: *run_tests

- name: "Other Tests"
env_vars:
- name: MAVEN_PROJECTS
value: '!server,!processing,!indexing-service,!extensions-core/kafka-indexing-service,!extensions-contrib/confluent-extensions,!integration-tests-ex/cases'
commands: *run_tests
4 changes: 4 additions & 0 deletions codestyle/checkstyle-suppressions.xml
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,8 @@
<suppress checks="ImportOrder" files="[\\/]target[\\/]generated-test-sources[\\/]" />
<suppress checks="EmptyLineSeparator" files="[\\/]target[\\/]generated-test-sources[\\/]" />
<suppress checks="AvoidStaticImport" files="[\\/]src[\\/](test)[\\/]"/>

<!-- Confluent Extensions -->
<suppress checks="Header" files="[\\/]io[\\/]confluent[\\/]" />
<suppress checks="PackageName" files="[\\/]io[\\/]confluent[\\/]" />
</suppressions>
Loading