This Data Processing Library Scala example shows how to build a stateful processing compiler that extracts the cardinality of nodes in the "Road Topology & Geometry" layer of HERE Map Content catalog and how to use the output of the previous compilation as feedback input to count the number of times the node cardinalities changed for each partition.
The node cardinality refers to the number of segment references for every topology node. In this example we use the Road Topology and Geometry Layer that consists of Topology Geometry Partitions where each Partition contains a set of Nodes .
In this example, the DirectMToNCompiler
functional pattern is used,
since it allows to implement transformation partitions.
The compiler runs twice with different HERE Map Content catalog versions and does the following:
- takes HERE Map Content input data as well as data from a previous compilation as feedback
- finds the cardinality for each node in a partition
- if the input data has changed since the last compilation, the compiler updates the cardinality and increments the variable that counts the number of times the compiler has run
For more information about the DirectMToNCompiler
functional pattern, see the Data Processing Library Developer Guide
.
To run this example, you need two sets of credentials:
- Platform credentials: To get access to the platform data and resources, including HERE Map Content data for your pipeline input.
- Repository credentials: To download HERE Data SDK for Java & Scala libraries and Maven archetypes to your environment.
For more details on how to set up your credentials, see the Identity & Access Management Developer Guide.
For more details on how to verify that your platform credentials are configured correctly, see the Verify Your Credentials tutorial.
In the commands that follow, replace the variable placeholders with the following values:
$CATALOG_ID
is your output catalog's ID.$CATALOG_HRN
is your output catalog'sHRN
(returned by theolp catalog create
command).$PROJECT_HRN
is your project'sHRN
(returned by theolp project create
command).$CATALOG_RIB
is the HRN of the public HERE Map Content catalog in your pipeline configuration (HERE environment.
Note: We recommend that you set values to variables, so that you can easily copy and execute the following commands.
The catalog you create is used to store the cardinality of nodes and the number of times the compiler has run.
To run this compiler locally, use a local output catalog as described below. For more information about local catalogs, see the SDK tutorial about local development and testing and the OLP CLI documentation.
- Use the
olp local catalog create
command to create the local catalog.
olp local catalog create stateful-compiler-scala stateful-compiler-scala --summary "Stateful compiler example catalog" \
--description "Stateful compiler example catalog"
The local catalog will have the HRN hrn:local:data:::stateful-compiler-scala
.
- Use the
olp local catalog layer add
command to add twoversioned
layers to your catalog:
Layer ID | Partitioning | Zoom Level | Layer Type | Content Type | Schema | Content Encoding |
---|---|---|---|---|---|---|
nodecardinality-count | HEREtile | 12 | Versioned | application/json | None | uncompressed |
state | Generic | N.A. | Versioned | application/octet-stream | None | uncompressed |
olp local catalog layer add hrn:local:data:::stateful-compiler-scala nodecardinality-count nodecardinality-count --versioned --summary "nodecardinality count" \
--description "nodecardinality count" --partitioning heretile:12 \
--content-type application/json
olp local catalog layer add hrn:local:data:::stateful-compiler-scala state state --versioned --summary "state" --description "state" \
--partitioning Generic --content-type application/octet-stream
To build the compiler, run sbt package
in the stateful-nodecardinality-extraction
directory.
sbt package
To run the compiler locally, you will need to run the entry point to the compiler:
com.here.platform.data.processing.example.scala.feedback.Main
As arguments, you must provide the -Dspark.master
parameter with the address of the Spark server
master to connect to, and any configuration parameters you want to override. Alternatively, you can
add those parameters to the application.conf
file.
Additionally, you also need to specify the -Dpipeline-config.file
and -Dpipeline-job.file
parameters to define the location of a configuration file that contains the catalogs as well as
job-specific versions of the catalogs, to read and write to.
The example project provides two template job configurations, config/here/pipeline-job-first.conf
and
config/here/pipeline-job-second.conf
for the first and second run of the pipeline, respectively.
pipeline-job-first.conf
specifies in the line version = 1
that the version 1 of the input
catalog should be processed in the first run. You can change this version to any number between 0
and the most recent version of the HERE Map Content catalog. You can find the most recent version by
opening the HERE platform portal and navigating to the HERE Map Content
catalog, and viewing the current catalog's version in the Catalog info
section.
pipeline-job-second.conf
specifies in the line version = 2
that version 2 of the input
catalog should be processed in the second run. You can change this version to any number that is
less than or equal to the most recent version of the HERE Map Content catalog and greater than the version
specified in config/here/pipeline-job-first.conf
.
For local runs, a bounding box filter is provided in the
config/here/local-application.conf
to
limit the number of partitions to be processed. This speeds up the compilation process. In this
example, we use a bounding box around the cities of Berlin. You can edit the bounding box coordinates to compile a different
partition of HERE Map Content. Make sure you update the layer coverage to reflect the different
geographical region. In order to use this configuration file, you need to use the -Dconfig.file
parameter.
Set the environment variable $PATH_TO_CONFIG_FOLDER
to ./config/here
.
Finally, execute the following command in the
stateful-nodecardinality-extraction
directory to run The Stateful Processing Compiler.
For the HERE platform environment:
sbt run \
-Dpipeline-config.file=./config/here/local-pipeline-config.conf \
-Dpipeline-job.file=./config/here/pipeline-job-first.conf \
-Dconfig.file=./config/here/local-application.conf \
-Dspark.master=local[*]
To observe the behavior of the Stateful Processing Compiler, you have to run the compiler again using
pipeline-job-second.conf
as job configuration:
For the HERE platform environment:
sbt run \
-Dpipeline-config.file=./config/here/local-pipeline-config.conf \
-Dpipeline-job.file=./config/here/pipeline-job-second.conf \
-Dconfig.file=./config/here/local-application.conf \
-Dspark.master=local[*]
After the second run, in the HERE platform environment, you can inspect the local catalog with the OLP CLI:
olp local catalog inspect hrn:local:data:::stateful-compiler-scala
You should see the nodes with their cardinality, for example here:cm:node:1002845384: 3
, where here:cm:node:1002845384
is the node, and 3
- cardinality.
The field updatesCount
will be 1
for those partitions that did not change during the second run, otherwise it will be 2
.
To follow this example, you will need a project. A project is a collection of platform resources (catalogs, pipelines, and schemas) with controlled access. You can create a project through the HERE platform portal.
Alternatively, use the OLP CLI olp project create
command to create the project:
olp project create $PROJECT_ID $PROJECT_NAME
The command returns the HERE Resource Name (HRN) of your new project. Note down this HRN as you will need it later in this tutorial.
You do not have to provide a
--scope
parameter if your app has a default scope. For details on how to set a default project scope for an app, see the Specify a default Project for Apps chapter of the Identity & Access Management Developer Guide.
For more information on how to work with projects, see the Organize your work in projects tutorial.
As mentioned above, we will use the public HERE Map Content input catalog, however, we need to create our own output catalog to store the cardinality of nodes and the number of times the compiler has run.
- Use the
olp catalog create
command to create the catalog. Make sure to note down the HRN returned by the following command for later use:
olp catalog create $CATALOG_ID $CATALOG_ID --summary "Stateful compiler example catalog" \
--description "Stateful compiler example catalog" \
--scope $PROJECT_HRN
- Use the
olp catalog layer add
command to add twoversioned
layers to your catalog:
olp catalog layer add $CATALOG_HRN nodecardinality-count nodecardinality-count --versioned --summary "nodecardinality count" \
--description "nodecardinality count" --partitioning heretile:12 \
--content-type application/json --scope $PROJECT_HRN
olp catalog layer add $CATALOG_HRN state state --versioned --summary "state" --description "state" \
--partitioning Generic --content-type application/octet-stream \
--scope $PROJECT_HRN
If a billing tag is required in your realm, use the
--billing-tags: "YOUR_BILLING_TAG"
parameter.
- Use the
olp project resource link
command to link the HERE Map Content catalog to your project:
olp project resource link $PROJECT_HRN $CATALOG_RIB
- For more details on catalog commands, see Catalog Commands.
- For more details on layer commands, see Layer Commands.
- For more details on project commands, see Project Commands.
- For instructions on how to link a resource to a project, see Project Resource Link command.
From the SDK examples directory, open the data-processing/scala/stateful-nodecardinality-extraction
project in
your Integrated Development Environment (IDE).
The config/here/pipeline-config.conf
(for the HERE platform environment) file contains
the permanent configuration of the data sources for the compiler.
Pick the file that corresponds to your platform environemnt. For example, the pipeline configuration for the HERE platform environment looks like:
pipeline.config {
output-catalog {hrn = "YOUR_OUTPUT_CATALOG_HRN"}
input-catalogs {
rib {hrn = "hrn:here:data::olp-here:rib-2"}
}
}
Replace YOUR_OUTPUT_CATALOG_HRN
with the HRN of your nodecardinality catalog.
To find the HRN, in the HERE platform portal, navigate to your catalog. The HRN is displayed in the upper
left corner of the page.
The config/here/pipeline-job-first.conf
and config/here/pipeline-second.conf
files contain the compiler's run
configuration and point to two different versions of the HERE Map Content Catalog.
To find the latest version of the HERE Map Content catalog, in the HERE platform portal, navigate to the HERE Map Content catalog, and view the current catalog's version in the Catalog info section.
The remainder of the configuration is specified in the application.conf
file that can be found in the
src/main/resources
directory of the compiler project. However, you do not have to modify it unless
you want to change the behavior of the compiler.
Run the sbt assembly
command in the stateful-nodecardinality-extraction
directory to generate a fat JAR file to deploy the compiler to a pipeline.
sbt assembly
Once the previous command is finished, your JAR is then available at the target
directory, and you
can upload it using the HERE pipeline UI
or the OLP CLI.
You can use the OLP CLI to create pipeline components and activate the pipeline version with the following commands:
- Create pipeline components:
For this example, a bounding box filter is provided by --runtime-config
parameter to
limit the number of partitions to be processed. This speeds up the compilation process. In this
example, we use a bounding box around the cities of Berlin. You can edit the bounding box coordinates to compile a different
partition of HERE Map Content. Make sure you update the layer coverage to reflect the different
geographical region.
olp pipeline create $COMPONENT_NAME_Pipeline --scope $PROJECT_HRN
olp pipeline template create $COMPONENT_NAME_Template batch-4.0 $PATH_TO_JAR \
com.here.platform.data.processing.example.scala.feedback.Main \
--workers=4 --worker-units=3 --supervisor-units=2 --input-catalog-ids=rib \
--scope $PROJECT_HRN
olp pipeline version create $COMPONENT_NAME_version $PIPELINE_ID $PIPELINE_TEMPLATE_ID \
"$PATH_TO_CONFIG_FOLDER/pipeline-config.conf" \
--runtime-config here.platform.data-processing.executors.partitionKeyFilters.0.className=BoundingBoxFilter \
here.platform.data-processing.executors.partitionKeyFilters.0.param.boundingBox.north=52.67551 \
here.platform.data-processing.executors.partitionKeyFilters.0.param.boundingBox.south=52.338261 \
here.platform.data-processing.executors.partitionKeyFilters.0.param.boundingBox.east=13.76116 \
here.platform.data-processing.executors.partitionKeyFilters.0.param.boundingBox.west=13.08835 \
--scope $PROJECT_HRN
- Activate the pipeline version:
olp pipeline version activate $PIPELINE_ID $PIPELINE_VERSION_ID \
--input-catalogs "$PATH_TO_CONFIG_FOLDER/pipeline-job-first.conf" \
--scope $PROJECT_HRN
You do not have to specify the input catalog's version, unless you want to. The latest version will be automatically used.
In the HERE platform portal, navigate to your pipeline to see its status.
In the HERE platform portal, select the Data tab and find your catalog.
- Open the
nodecardinality-count
layer and select the Inspect tab. Verify that partitions with the JSON data are present, and you can view this data by selecting a partition. - Select any partition to look at its content. You should see the nodes with their cardinality, for example
here:cm:node:1002845384: 3
, wherehere:cm:node:1002845384
is the node, and3
- cardinality. The fieldupdatesCount
will be1
for those partitions that did not change during the second run. It will be2
otherwise. The JSON output of the compiler should be displayed on the right side as follows: