Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion R/run-tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ FAILED=0
LOGFILE=$FWDIR/unit-tests.out
rm -f $LOGFILE

SPARK_AVRO_JAR_PATH=$(find $FWDIR/../external/avro/ -name "spark-avro*jar" -print | egrep -v "tests.jar|test-sources.jar|sources.jar|javadoc.jar")
SPARK_AVRO_JAR_PATH=$(find $FWDIR/../connector/avro/ -name "spark-avro*jar" -print | egrep -v "tests.jar|test-sources.jar|sources.jar|javadoc.jar")

if [[ $(echo $SPARK_AVRO_JAR_PATH | wc -l) -eq 1 ]]; then
SPARK_JARS=$SPARK_AVRO_JAR_PATH
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
10 changes: 5 additions & 5 deletions docs/streaming-kinesis-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ A Kinesis stream can be set up at one of the valid Kinesis endpoints with 1 or m
.build()

See the [API docs](api/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.html)
and the [example]({{site.SPARK_GITHUB_URL}}/tree/master/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala). Refer to the [Running the Example](#running-the-example) subsection for instructions on how to run the example.
and the [example]({{site.SPARK_GITHUB_URL}}/tree/master/connector/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala). Refer to the [Running the Example](#running-the-example) subsection for instructions on how to run the example.

</div>
<div data-lang="java" markdown="1">
Expand All @@ -82,7 +82,7 @@ A Kinesis stream can be set up at one of the valid Kinesis endpoints with 1 or m
.build();

See the [API docs](api/java/index.html?org/apache/spark/streaming/kinesis/KinesisInputDStream.html)
and the [example]({{site.SPARK_GITHUB_URL}}/tree/master/external/kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java). Refer to the [Running the Example](#running-the-example) subsection for instructions to run the example.
and the [example]({{site.SPARK_GITHUB_URL}}/tree/master/connector/kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java). Refer to the [Running the Example](#running-the-example) subsection for instructions to run the example.

</div>
<div data-lang="python" markdown="1">
Expand All @@ -93,7 +93,7 @@ A Kinesis stream can be set up at one of the valid Kinesis endpoints with 1 or m
[region name], [initial position], [checkpoint interval], StorageLevel.MEMORY_AND_DISK_2)

See the [API docs](api/python/reference/pyspark.streaming.html#kinesis)
and the [example]({{site.SPARK_GITHUB_URL}}/tree/master/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py). Refer to the [Running the Example](#running-the-example) subsection for instructions to run the example.
and the [example]({{site.SPARK_GITHUB_URL}}/tree/master/connector/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py). Refer to the [Running the Example](#running-the-example) subsection for instructions to run the example.

</div>
</div>
Expand Down Expand Up @@ -246,8 +246,8 @@ To run the example,
</div>
<div data-lang="python" markdown="1">

./bin/spark-submit --jars 'external/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar' \
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py \
./bin/spark-submit --jars 'connector/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar' \
connector/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py \
[Kinesis app name] [Kinesis stream name] [endpoint URL] [region name]

</div>
Expand Down
18 changes: 9 additions & 9 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -101,11 +101,11 @@
<module>examples</module>
<module>repl</module>
<module>launcher</module>
<module>external/kafka-0-10-token-provider</module>
<module>external/kafka-0-10</module>
<module>external/kafka-0-10-assembly</module>
<module>external/kafka-0-10-sql</module>
<module>external/avro</module>
<module>connector/kafka-0-10-token-provider</module>
<module>connector/kafka-0-10</module>
<module>connector/kafka-0-10-assembly</module>
<module>connector/kafka-0-10-sql</module>
<module>connector/avro</module>
<!-- See additional modules enabled by profiles below -->
</modules>

Expand Down Expand Up @@ -3405,23 +3405,23 @@
<profile>
<id>spark-ganglia-lgpl</id>
<modules>
<module>external/spark-ganglia-lgpl</module>
<module>connector/spark-ganglia-lgpl</module>
</modules>
</profile>

<!-- Kinesis integration is not included by default due to ASL-licensed code -->
<profile>
<id>kinesis-asl</id>
<modules>
<module>external/kinesis-asl</module>
<module>external/kinesis-asl-assembly</module>
<module>connector/kinesis-asl</module>
<module>connector/kinesis-asl-assembly</module>
</modules>
</profile>

<profile>
<id>docker-integration-tests</id>
<modules>
<module>external/docker-integration-tests</module>
<module>connector/docker-integration-tests</module>
</modules>
</profile>

Expand Down