Skip to content

Commit 8a05f13

Browse files
Copilotdevstress
Copilot
and
devstress
committed
Add ReleasePackagesTesting folder for validating release artifacts
- Created ReleasePackagesTesting solution with AppHost and IntegrationTests - AppHost uses Docker image (flinkdotnet/jobgateway) instead of project reference - IntegrationTests use FlinkDotnet NuGet package instead of project references - Copied test infrastructure from LocalTesting (connectors, Maven projects, configs) - Added test-release-packages.ps1 script for workflow integration - Tests validate Docker image and NuGet packages work before publishing Co-authored-by: devstress <30769729+devstress@users.noreply.github.com>
1 parent ab89b57 commit 8a05f13

30 files changed

+6689
-0
lines changed

‎ReleasePackagesTesting/NativeFlinkJob/README.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Native Flink Kafka Job - Infrastructure Validation
2+
3+
This is a standalone Apache Flink job using the official Flink Kafka connector to validate that the Aspire LocalTesting infrastructure is correctly configured.
4+
5+
## Purpose
6+
7+
Before debugging Gateway/IR issues, we need to prove the infrastructure works with a standard Flink job:
8+
- ✅ Aspire DCP correctly configures Flink cluster
9+
- ✅ Kafka is accessible from Flink containers at `kafka:9093`
10+
- ✅ Messages flow through: Kafka Input → Flink Transform → Kafka Output
11+
12+
## Build
13+
14+
```bash
15+
cd LocalTesting/NativeFlinkJob
16+
mvn clean package
17+
```
18+
19+
This creates: `target/native-flink-kafka-job-1.0.0.jar`
20+
21+
## Run via Flink REST API
22+
23+
```bash
24+
# Upload JAR
25+
curl -X POST -H "Expect:" -F "jarfile=@target/native-flink-kafka-job-1.0.0.jar" \
26+
http://localhost:8081/jars/upload
27+
28+
# Submit job (replace {jarId} with the ID from upload response)
29+
curl -X POST http://localhost:8081/jars/{jarId}/run \
30+
-H "Content-Type: application/json" \
31+
-d '{
32+
"entryClass": "com.flinkdotnet.NativeKafkaJob",
33+
"programArgsList": [
34+
"--bootstrap-servers", "kafka:9093",
35+
"--input-topic", "lt.native.input",
36+
"--output-topic", "lt.native.output",
37+
"--group-id", "native-test-consumer"
38+
],
39+
"parallelism": 1
40+
}'
41+
```
42+
43+
## Test with C#
44+
45+
The `FlinkNativeKafkaInfrastructureTest.cs` integration test:
46+
1. Starts Aspire infrastructure (Kafka + Flink)
47+
2. Builds and submits this native JAR
48+
3. Produces test messages
49+
4. Verifies messages are transformed and consumed
50+
51+
If this test **PASSES**: Infrastructure is correct, debug Gateway
52+
If this test **FAILS**: Fix infrastructure first
53+
54+
## Configuration
55+
56+
Default values (for LocalTesting environment):
57+
- **Bootstrap Servers**: `kafka:9093` (Aspire DCP internal listener)
58+
- **Input Topic**: `lt.native.input`
59+
- **Output Topic**: `lt.native.output`
60+
- **Group ID**: `native-flink-consumer`
61+
62+
Override with command-line args:
63+
```bash
64+
--bootstrap-servers kafka:9093
65+
--input-topic my-input
66+
--output-topic my-output
67+
--group-id my-consumer-group
68+
```
69+
70+
## Key Differences from FlinkJobRunner
71+
72+
1. **Uses official Flink Kafka Connector** (`flink-connector-kafka`) not raw Kafka clients
73+
2. **Proper dependency management** - connector packaged in fat JAR
74+
3. **Standard Flink APIs** - `KafkaSource` and `KafkaSink` builders
75+
4. **No IR/JSON** - direct Java code, no intermediate representation
76+
77+
## Troubleshooting
78+
79+
**Build fails with missing dependencies**:
80+
- Ensure Maven can reach Maven Central
81+
- Check Flink version compatibility (2.1.0)
82+
83+
**Job fails to start**:
84+
- Check Flink JobManager logs: `docker logs flink-jobmanager`
85+
- Verify bootstrap servers are accessible from Flink container
86+
87+
**No messages consumed**:
88+
- Check Kafka topics exist
89+
- Verify bootstrap servers (`kafka:9093` for containers, `localhost:{port}` for host)
90+
- Check Flink job is in RUNNING state
91+
- Look for exceptions in TaskManager logs
92+
93+
## Next Steps After Validation
94+
95+
Once this job works:
96+
1. Compare its Kafka configuration with Gateway's IR-generated config
97+
2. Identify what Gateway does differently
98+
3. Fix Gateway to match working configuration

‎ReleasePackagesTesting/NativeFlinkJob/pom.xml‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
5+
<modelVersion>4.0.0</modelVersion>
6+
7+
<groupId>com.flinkdotnet</groupId>
8+
<artifactId>native-flink-kafka-job</artifactId>
9+
<version>1.0.0</version>
10+
<packaging>jar</packaging>
11+
<name>Native Flink Kafka Job</name>
12+
<description>Native Apache Flink job to validate infrastructure setup</description>
13+
14+
<properties>
15+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
16+
<java.version>17</java.version>
17+
<maven.compiler.source>${java.version}</maven.compiler.source>
18+
<maven.compiler.target>${java.version}</maven.compiler.target>
19+
<flink.version>2.1.0</flink.version>
20+
<kafka.version>3.7.0</kafka.version>
21+
<slf4j.version>1.7.36</slf4j.version>
22+
</properties>
23+
24+
<dependencies>
25+
<!-- Flink Core Dependencies (provided by cluster) -->
26+
<dependency>
27+
<groupId>org.apache.flink</groupId>
28+
<artifactId>flink-streaming-java</artifactId>
29+
<version>${flink.version}</version>
30+
<scope>provided</scope>
31+
</dependency>
32+
33+
<dependency>
34+
<groupId>org.apache.flink</groupId>
35+
<artifactId>flink-clients</artifactId>
36+
<version>${flink.version}</version>
37+
<scope>provided</scope>
38+
</dependency>
39+
40+
<!-- Flink Connector Base (required for Flink 2.x) -->
41+
<dependency>
42+
<groupId>org.apache.flink</groupId>
43+
<artifactId>flink-connector-base</artifactId>
44+
<version>${flink.version}</version>
45+
<scope>provided</scope>
46+
</dependency>
47+
48+
<!-- Kafka Clients (bundled in JAR, same as FlinkJobRunner) -->
49+
<dependency>
50+
<groupId>org.apache.kafka</groupId>
51+
<artifactId>kafka-clients</artifactId>
52+
<version>3.9.1</version>
53+
</dependency>
54+
55+
<!-- Logging -->
56+
<dependency>
57+
<groupId>org.slf4j</groupId>
58+
<artifactId>slf4j-api</artifactId>
59+
<version>${slf4j.version}</version>
60+
<scope>provided</scope>
61+
</dependency>
62+
</dependencies>
63+
64+
<build>
65+
<plugins>
66+
<plugin>
67+
<groupId>org.apache.maven.plugins</groupId>
68+
<artifactId>maven-compiler-plugin</artifactId>
69+
<version>3.11.0</version>
70+
<configuration>
71+
<source>${java.version}</source>
72+
<target>${java.version}</target>
73+
</configuration>
74+
</plugin>
75+
76+
<!-- Shade plugin to create fat JAR with dependencies -->
77+
<plugin>
78+
<groupId>org.apache.maven.plugins</groupId>
79+
<artifactId>maven-shade-plugin</artifactId>
80+
<version>3.5.0</version>
81+
<executions>
82+
<execution>
83+
<phase>package</phase>
84+
<goals>
85+
<goal>shade</goal>
86+
</goals>
87+
<configuration>
88+
<!-- Suppress harmless warnings about overlapping resources -->
89+
<createDependencyReducedPom>true</createDependencyReducedPom>
90+
<shadedArtifactAttached>false</shadedArtifactAttached>
91+
<artifactSet>
92+
<excludes>
93+
<exclude>org.apache.flink:flink-shaded-force-shading</exclude>
94+
<exclude>com.google.code.findbugs:jsr305</exclude>
95+
<exclude>org.slf4j:*</exclude>
96+
<exclude>org.apache.logging.log4j:*</exclude>
97+
</excludes>
98+
</artifactSet>
99+
<filters>
100+
<filter>
101+
<artifact>*:*</artifact>
102+
<excludes>
103+
<exclude>META-INF/*.SF</exclude>
104+
<exclude>META-INF/*.DSA</exclude>
105+
<exclude>META-INF/*.RSA</exclude>
106+
<!-- Exclude module-info to avoid strong encapsulation warnings -->
107+
<exclude>module-info.class</exclude>
108+
<!-- Only keep one MANIFEST.MF (from this project) -->
109+
<exclude>META-INF/MANIFEST.MF</exclude>
110+
</excludes>
111+
</filter>
112+
</filters>
113+
<transformers>
114+
<!-- Create new MANIFEST.MF with main class -->
115+
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
116+
<mainClass>com.flinkdotnet.NativeKafkaJob</mainClass>
117+
</transformer>
118+
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
119+
</transformers>
120+
</configuration>
121+
</execution>
122+
</executions>
123+
</plugin>
124+
</plugins>
125+
</build>
126+
</project>

0 commit comments

Comments
 (0)