Provide shaded jar without classifier

[dolfinus]

Use case

I'm using PySpark with Clickhouse JDBC. ETL processes in my company use spark.jars.packages config option to download Maven packages - it is very convenient, users don't have to download .jar files manually or hardcode URLs with Maven Central. Repo URL can be changed any time by altering spark.jars.ivySettings.

For v0.3.2 this was as simple as:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("spark_app_onetl_demo")
    .config("spark.jars.packages", "ru.yandex.clickhouse:clickhouse-jdbc:0.3.2")
    .getOrCreate()
)

In 0.4.0 (#1134) some dependencies were made optional, and moved to new http and all classifiers. But Spark cannot use classifiers in package names, so this code breaks:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("spark_app_onetl_demo")
    .config("spark.jars.packages", "com.clickhouse:clickhouse-jdbc:http:0.4.0")
    .getOrCreate()
)

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Provided Maven Coordinates must be in the form 'groupId:artifactId:version'. The coordinate provided is: com.clickhouse:clickhouse-jdbc:all:0.4.0

For 0.4.0-0.7.2 I was able to use a workaround by manually downloading the same dependencies from http classifier:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("spark_app_onetl_demo")
    .config("spark.jars.packages", "com.clickhouse:clickhouse-jdbc:0.4.0:com.clickhouse:clickhouse-http-client:0.4.0")
    .getOrCreate()
)

spark = (
    SparkSession.builder
    .appName("spark_app_onetl_demo")
    .config("spark.jars.packages", "com.clickhouse:clickhouse-jdbc:0.7.2:com.clickhouse:clickhouse-http-client:0.7.2:org.apache.httpcomponents.client5:httpclient5:5.4.2")
    .getOrCreate()
)

But this doesn't work with 0.9.2. First of all, org.ow2.asm:asm (compile time dependency) cannot be downloaded for some reason (probably artifact name is incompatible with Ivy2):

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("spark_app_onetl_demo")
    .config("spark.jars.packages", "com.clickhouse:clickhouse-jdbc:0.9.2:com.clickhouse:clickhouse-http-client:0.9.2:org.apache.httpcomponents.client5:httpclient5:5.4.2")
    .getOrCreate()
)

Cannot download org.ow2.asm:asm:9.7

Ivy Default Cache set to: /home/maxim/.ivy2/cache
The jars for the packages stored in: /home/maxim/.ivy2/jars
com.clickhouse#clickhouse-jdbc added as a dependency
com.clickhouse#clickhouse-http-client added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-3bd0d1ff-033f-4a4a-95e0-6192ced0b746;1.0
        confs: [default]
        found com.clickhouse#clickhouse-jdbc;0.9.2 in central
        found com.clickhouse#clickhouse-client;0.9.2 in central
        found org.apache.commons#commons-compress;1.27.1 in central
        found commons-io#commons-io;2.16.1 in main
        found org.apache.commons#commons-lang3;3.18.0 in local-maven-cache
        found com.clickhouse#clickhouse-data;0.9.2 in central
        found com.clickhouse#clickhouse-http-client;0.9.2 in central
        found org.apache.httpcomponents.client5#httpclient5;5.4.4 in local-maven-cache
        found org.apache.httpcomponents.core5#httpcore5;5.3.4 in local-maven-cache
        found org.apache.httpcomponents.core5#httpcore5-h2;5.3.4 in local-maven-cache
        found com.clickhouse#jdbc-v2;0.9.2 in central
        found org.roaringbitmap#RoaringBitmap;0.9.47 in central
        found org.ow2.asm#asm;9.7 in local-maven-cache
        found com.google.guava#failureaccess;1.0.3 in local-maven-cache
        found com.google.guava#listenablefuture;9999.0-empty-to-avoid-conflict-with-guava in local-maven-cache
        found org.jspecify#jspecify;1.0.0 in local-maven-cache
        found com.google.errorprone#error_prone_annotations;2.36.0 in local-maven-cache
        found com.google.j2objc#j2objc-annotations;3.0.0 in local-maven-cache
        found org.roaringbitmap#shims;0.9.47 in central
        found com.clickhouse#client-v2;0.9.2 in central
        found com.google.guava#guava;33.4.6-jre in local-maven-cache
:: resolution report :: resolve 1014ms :: artifacts dl 29ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   21  |   0   |   0   |   0   ||   21  |   0   |
        ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
                [NOT FOUND  ] org.ow2.asm#asm;9.7!asm.jar (2ms)

        ==== local-maven-cache: tried

          file:///home/maxim/.m2/repository/org/ow2/asm/asm/9.7/asm-9.7.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::              FAILED DOWNLOADS            ::

                :: ^ see resolution messages for details  ^ ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: org.ow2.asm#asm;9.7!asm.jar

                ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: org.ow2.asm#asm;9.7!asm.jar]
        at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1608)
        at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:334)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:964)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Even if this dependency is excluded, JDBC driver doesn't properly work:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("spark_app_onetl_demo")
    .config("spark.jars.packages", "com.clickhouse:clickhouse-jdbc:0.9.2:com.clickhouse:clickhouse-http-client:0.9.2:org.apache.httpcomponents.client5:httpclient5:5.4.2")
    .config("spark.jars.excludes", "org.ow2.asm:asm")
    .getOrCreate()
)

antlr v4.12 version compatibility issue

: java.lang.ExceptionInInitializerError
        at com.clickhouse.jdbc.internal.SqlParser.walkSql(SqlParser.java:34)
        at com.clickhouse.jdbc.internal.SqlParser.parsePreparedStatement(SqlParser.java:28)
        at com.clickhouse.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:379)
        at com.clickhouse.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:171)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.getQueryOutputSchema(JDBCRDD.scala:65)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:58)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:241)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:37)
        at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:346)
        at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:229)
        at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:211)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:172)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.UnsupportedOperationException: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with version 4 (expected 3).
        at org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:187)
        at com.clickhouse.jdbc.internal.ClickHouseLexer.<clinit>(ClickHouseLexer.java:2098)
        ... 26 more
Caused by: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with version 4 (expected 3).
        ... 28 more

#2554 fixes that by shading antlr package, but this is done only in all classifier which cannot be used with PySpark spark.jars.packages.

Describe the solution you'd like

Please introduce new artifact clickhouse-jdbc-shaded:{VERSION} instead of/in combination of all classifier, so it can be used with spark.jars.packages.

Describe the alternatives you've considered

Someone tried to add support of Maven artifacts group:artifact:classifier:version to Spark several times since 2017, but with no success:

• https://issues.apache.org/jira/browse/SPARK-20075
• https://issues.apache.org/jira/browse/SPARK-22849
• https://issues.apache.org/jira/browse/SPARK-24287

Spark connector for Clickhouse is not suitable for my cases as well.

Additional context

[chernser]

Good day, @dolfinus!

Thank you for bringing this up!
I understand what you want and we will think how to implement it.

[dolfinus]

Thank you very much!

[chernser]

Good day, @dolfinus !
I've added clickhouse-jdbc-all and have pushed nightly build.
Would you please test before I do a release today?

To use nightly you need to use https://central.sonatype.com/repository/maven-snapshots/ as repository.

Thanks!

[dolfinus]

I've tried to use SNAPSHOT version, but it looks like .pom file has some errors:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("spark_app_onetl_demo")
    .config("spark.jars.packages", "com.clickhouse:clickhouse-jdbc-all:0.9.2-SNAPSHOT")
    .config("spark.jars.ivySettings", "ivysettings.xml")
    .getOrCreate()
)

ivysettings.xml

<ivysettings>
    <settings defaultResolver="main"/>
    <resolvers>
      <chain name="main" returnFirst="true">
        <!-- Use Maven cache -->
        <ibiblio name="local-maven-cache" m2compatible="true" root="file://${user.home}/.m2/repository"/>
        <!-- Use ~/.ivy2/jars/*.jar files -->
        <ibiblio name="local-ivy2-cache" m2compatible="false" root="file://${user.home}/.ivy2/jars"/>
        <!-- Download packages from Maven -->
        <ibiblio name="central" m2compatible="true" />
        <!-- Download packages from SparkPackages -->
        <ibiblio name="spark-packages" m2compatible="true" root="https://repos.spark-packages.org/" />
        <ibiblio name="maven-snapshots" m2compatible="true" root="https://central.sonatype.com/repository/maven-snapshots/" />
      </chain>
    </resolvers>
</ivysettings>

25/10/29 20:08:25 WARN Utils: Your hostname, msmartynov resolves to a loopback address: 127.0.1.1; using 192.168.1.16 instead (on interface wlo1)
25/10/29 20:08:25 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
:: loading settings :: file = ivysettings.xml
:: loading settings :: url = jar:file:/home/maxim/Repo/onetl/venv/lib/python3.12/site-packages/pyspark/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/maxim/.ivy2/cache
The jars for the packages stored in: /home/maxim/.ivy2/jars
com.clickhouse#clickhouse-jdbc-all added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-0b714e6b-9f54-41b0-a134-d799f38eb90d;1.0
        confs: [default]
:: resolution report :: resolve 7244ms :: artifacts dl 1ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   1   |   0   |   0   |   0   ||   0   |   0   |
        ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
        io problem while parsing ivy file: https://central.sonatype.com/repository/maven-snapshots/com/clickhouse/clickhouse-jdbc-all/0.9.2-SNAPSHOT/clickhouse-jdbc-all-0.9.2-20251029.182804-1.pom (java.io.IOException: Impossible to load parent for file:/home/maxim/.ivy2/cache/com.clickhouse/clickhouse-jdbc-all/ivy-0.9.2-SNAPSHOT.xml.original. Parent=com.clickhouse#clickhouse-java;${revision})

                module not found: com.clickhouse#clickhouse-jdbc-all;0.9.2-SNAPSHOT

        ==== local-maven-cache: tried

          file:///home/maxim/.m2/repository/com/clickhouse/clickhouse-java/${revision}/clickhouse-java-${revision}.pom

          -- artifact com.clickhouse#clickhouse-java;${revision}!clickhouse-java.jar:

          file:///home/maxim/.m2/repository/com/clickhouse/clickhouse-java/${revision}/clickhouse-java-${revision}.jar

        ==== local-ivy2-cache: tried

          -- artifact com.clickhouse#clickhouse-java;${revision}!clickhouse-java.jar:

          file:///home/maxim/.ivy2/jars/clickhouse-java/jars/clickhouse-java-${revision}.jar

        ==== central: tried

          https://repo1.maven.org/maven2/com/clickhouse/clickhouse-java/${revision}/clickhouse-java-${revision}.pom

          -- artifact com.clickhouse#clickhouse-java;${revision}!clickhouse-java.jar:

          https://repo1.maven.org/maven2/com/clickhouse/clickhouse-java/${revision}/clickhouse-java-${revision}.jar

        ==== spark-packages: tried

          https://repos.spark-packages.org/com/clickhouse/clickhouse-java/${revision}/clickhouse-java-${revision}.pom

          -- artifact com.clickhouse#clickhouse-java;${revision}!clickhouse-java.jar:

          https://repos.spark-packages.org/com/clickhouse/clickhouse-java/${revision}/clickhouse-java-${revision}.jar

        ==== maven-snapshots: tried

          https://central.sonatype.com/repository/maven-snapshots/com/clickhouse/clickhouse-java/${revision}/clickhouse-java-${revision}.pom

          -- artifact com.clickhouse#clickhouse-java;${revision}!clickhouse-java.jar:

          https://central.sonatype.com/repository/maven-snapshots/com/clickhouse/clickhouse-java/${revision}/clickhouse-java-${revision}.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::          UNRESOLVED DEPENDENCIES         ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: com.clickhouse#clickhouse-jdbc-all;0.9.2-SNAPSHOT: not found

                ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: com.clickhouse#clickhouse-jdbc-all;0.9.2-SNAPSHOT: not found]
        at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1613)
        at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:339)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:969)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
  File "/home/maxim/Repo/onetl/check_spark_clickhouse_jdbc.py", line 8, in <module>
    .getOrCreate()
     ^^^^^^^^^^^^^
  File "/home/maxim/Repo/onetl/venv/lib/python3.12/site-packages/pyspark/sql/session.py", line 497, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maxim/Repo/onetl/venv/lib/python3.12/site-packages/pyspark/context.py", line 515, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/home/maxim/Repo/onetl/venv/lib/python3.12/site-packages/pyspark/context.py", line 201, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/home/maxim/Repo/onetl/venv/lib/python3.12/site-packages/pyspark/context.py", line 436, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
                                       ^^^^^^^^^^^^^^^^^^^^
  File "/home/maxim/Repo/onetl/venv/lib/python3.12/site-packages/pyspark/java_gateway.py", line 107, in launch_gateway
    raise PySparkRuntimeError(
pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

clickhouse-jdbc-all-0.9.2-20251029.182804-1.pom

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <parent>
    <artifactId>clickhouse-java</artifactId>
    <groupId>com.clickhouse</groupId>
    <version>${revision}</version>
    <relativePath>../../pom.xml</relativePath>
  </parent>
  <modelVersion>4.0.0</modelVersion>
  <artifactId>clickhouse-jdbc-all</artifactId>
  <name>(Package) ClickHouse JDBC All Dependencies</name>
  <version>${revision}</version>
  <description>JDBC driver for ClickHouse</description>
  <url>https://github.com/ClickHouse/clickhouse-java/tree/main/clickhouse-jdbc-all</url>
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.2.4</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <shadedArtifactAttached>false</shadedArtifactAttached>
              <createDependencyReducedPom>true</createDependencyReducedPom>
              <createSourcesJar>true</createSourcesJar>
              <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
              <shadedClassifierName />
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

[chernser]

@dolfinus
I've fixed the pom - it should be flattened
I've tested locally with your sample code - works

I've pushed nightly - so you may test (in 20 minutes)

[dolfinus]

Thanks, it working on Java 11+. For Java 8 I found an issue #2629 not related to shaded jar