-
Notifications
You must be signed in to change notification settings - Fork 960
FAQ
The error looks something like this:
[ERROR] Failed to execute goal on project kafka-connect-jdbc: Could not resolve dependencies for project
io.confluent:kafka-connect-jdbc:jar:3.1.0-SNAPSHOT: Could not find artifact
org.apache.kafka:connect-api:jar:0.10.1.0-SNAPSHOT in confluent (http://packages.confluent.io/maven/) -> [Help 1]
During development of the JDBC connector we sometimes use a SNAPSHOT version of Kafka in order to build and test against new features. If you want to build a development version, you may need to build and install these dependencies to your local Maven repository in order to build the connector:
- Kafka - clone https://github.com/confluentinc/kafka.git and build with
./gradlew installAll
- Common - clone https://github.com/confluentinc/common and build with
mvn install
The connector sets the JDBC fetch size to be equal to batch.max.rows
. This provides a hint to the JDBC driver as to the number of rows to fetch. However, in practice, drivers may require this option to be set in conjunction with other settings to properly stream results and avoid OOM. Here are some dialects and configuration required gathered from Issue 34:
- MySQL: Attach to your JDBC connection string the parameter:
?useCursorFetch=true
- PostgreSQL: Should work out of the box
- MS SQL: Attach to connection string the parameter:
;selectMethod=cursor
Make sure to use Java 11 to compile and run tests, otherwise get errors like java.base does not "opens java.lang" to unnamed module
Running under WSL2 with source repository on Windows C
drive mounted into the WSL via /mnt/c
, 100% of the time, specific test will fail unexpectedly (it passes on Linux):
[ERROR] Failures:
[ERROR] JdbcSourceConnectorTest.testStartStop:148
Unexpected method calls:
CachedConnectionProvider.close()
Speculate the cause could be due to either very slow host filesystem access with the 9P server which is at least 10x slower then normal EXT4 access or some unimplemented filesystem attribute.
The only fix seems to be to move the source code under test into the WSL2 filesystem. This causes all tests to pass but means you now have "files you care about" inside of the WSL2 filesystem.
- Unit tests:
mvn test
- Integration tests:
mvn integration-test