Skip to content

Commit

Permalink
Merge pull request #3841 from bamaer/3831
Browse files Browse the repository at this point in the history
  • Loading branch information
hansva authored Apr 20, 2024
2 parents 931e652 + bbf1cc8 commit 0ddbd69
Show file tree
Hide file tree
Showing 8 changed files with 18 additions and 15 deletions.
2 changes: 2 additions & 0 deletions docs/hop-user-manual/modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,7 @@ under the License.
*** xref:pipeline/transforms/splunkinput.adoc[Splunk Input]
*** xref:pipeline/transforms/sqlfileoutput.adoc[SQL File Output]
*** xref:pipeline/transforms/standardizephonenumber.adoc[Standardize Phone Number]
*** xref:pipeline/transforms/schemamapping.adoc[Static Schema Mapping]
*** xref:pipeline/transforms/streamlookup.adoc[Stream Lookup]
*** xref:pipeline/transforms/streamschemamerge.adoc[Stream Schema Merge]
*** xref:pipeline/transforms/stringcut.adoc[String cut]
Expand Down Expand Up @@ -429,6 +430,7 @@ under the License.
** xref:metadata-types/pipeline-unit-test.adoc[Pipeline Unit Test]
** xref:metadata-types/rdbms-connection.adoc[Relational Database Connection]
** xref:metadata-types/splunk-connection.adoc[Splunk Connection]
** xref:metadata-types/static-schema-definition.adoc[Static Schema Definition]
** xref:hop-server/web-service.adoc[Web Service]
** xref:metadata-types/workflow-log.adoc[Workflow Log]
** xref:metadata-types/workflow-run-config.adoc[Workflow Run Configuration]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,15 @@ under the License.
:page-pagination:
:description: A Schema File Definition describes a stream layout that can be applied to a selected set of input/output transforms. The Schema Definition is a way to define a recurrent stream layout that can be reused around multiple pipelines by making the things easier by nit requiring the user to redefine if multiple times.

= Schema Definition
= Static Schema Definition

== Description

image:icons/folder.svg[]

A Schema Definition describes a stream layout that can be applied to a selected set of input/output transforms. The Schema Definition is a way to define a recurrent stream layout, that can be reused in multiple pipelines. This simplify the development by making the things easier because it saves the user to redefine the same set of fields and its associated attributes multiple times in multiple pipelines.
A Static Schema Definition describes a stream layout that can be applied to a selected set of input/output transforms. The Schema Definition is a way to define a recurrent stream layout, that can be reused in multiple pipelines.

Being able to use the same schema definition in multiple pipelines simplifies development and reduces the risk of errors in manual field or layout specification.

== Related Plugins

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ under the License.
|
== Description

The CSV File Input transform reads data from a delimited file. You can choose to use a xref:metadata-types/schema-file-definition.adoc[Schema Definition] or to define the required fields' layout manually.
The CSV File Input transform reads data from a delimited file. You can choose to use a xref:metadata-types/static-schema-definition.adoc[Schema Definition] or to define the required fields' layout manually.

The CSV label for this transform is a misnomer because you can define whatever separator you want to use, such as pipes, tabs, and semicolons; you are not constrained to using commas.
Internal processing allows this transform to process data quickly.
Expand Down Expand Up @@ -76,7 +76,7 @@ When reading multiple files, the total size of all files is taken into considera
In that specific case, make sure that ALL transform copies receive all files that need to be read, otherwise, the parallel algorithm will not work correctly (for obvious reasons).
WARNING: For technical reasons, parallel reading of CSV files is only supported on files that don't have fields with line breaks or carriage returns in them.
|File Encoding|Specify the encoding of the file being read.
|Schema Definition|Name of the xref:metadata-types/schema-file-definition.adoc[Schema Definition] that we want to reference.
|Schema Definition|Name of the xref:metadata-types/static-schema-definition.adoc[Schema Definition] that we want to reference.
|Fields Table|This table contains an ordered list of fields to be read from the target file.
|Preview button|Click to preview the data coming from the target file.
|Get Fields button|Click to return a list of fields from the target file based on the current settings (i.e. Delimiter, Enclosure, etc.).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ The name of that file is <errorline dir>/filename.<date_time>.<errorline extensi

=== Fields tab

The fields tab is for specifying the fields that must be read from the Excel files. You can choose to use a xref:metadata-types/schema-file-definition.adoc[Schema Definition] or to define the required fields' layout manually.
The fields tab is for specifying the fields that must be read from the Excel files. You can choose to use a xref:metadata-types/static-schema-definition.adoc[Schema Definition] or to define the required fields' layout manually.

Use _Get fields from header_ row to fill in the available fields if the sheets have a header row automatically.

Expand All @@ -118,7 +118,7 @@ Note: In the case of Number to Date conversion (for example, 20051028--> October

|===
|Option|Description
|Schema Definition|Name of the xref:metadata-types/schema-file-definition.adoc[Schema Definition] that we want to reference.
|Schema Definition|Name of the xref:metadata-types/static-schema-definition.adoc[Schema Definition] that we want to reference.
|Name|The name of the field.
|Type|The field's data type; String, Date or Number.
|Length|The length option depends on the field type.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -119,14 +119,14 @@ Negative numbers may be useful if you need to append to a sheet, but still prese

*Fields section*

The fields section is for specifying the fields that must be written to the Excel file. You can choose to use a xref:metadata-types/schema-file-definition.adoc[Schema Definition] or to define the required fields' layout manually.
The fields section is for specifying the fields that must be written to the Excel file. You can choose to use a xref:metadata-types/static-schema-definition.adoc[Schema Definition] or to define the required fields' layout manually.

If you decide to define the fields layout by using a xref:metadata-types/schema-file-definition.adoc[Schema Definition], use the xref:pipeline/transforms/schemamapping.adoc[Schema Mapping] transform to adjust the incoming stream according to the choosen xref:metadata-types/schema-file-definition.adoc[Schema Definition]
If you decide to define the fields layout by using a xref:metadata-types/static-schema-definition.adoc[Schema Definition], use the xref:pipeline/transforms/schemamapping.adoc[Schema Mapping] transform to adjust the incoming stream according to the choosen xref:metadata-types/static-schema-definition.adoc[Schema Definition]

[options="header"]
|===
|Option|Description
|Schema Definition|Name of the xref:metadata-types/schema-file-definition.adoc[Schema Definition] that we want to reference.
|Schema Definition|Name of the xref:metadata-types/static-schema-definition.adoc[Schema Definition] that we want to reference.
|Name|The field to write
|Type|The type of data
|Format|The Excel format to use in the sheet.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,14 @@ The Schema Mapping transform maps fields from and incoming stream over a specifi
!===
|===

Schema mapping is a useful mapping transform to precede transforms like xref:pipeline/transforms/textfileoutput.adoc[Text File Output] or xref:pipeline/transforms/excelwriter.adoc[Excel Writer] when we decide to refer to a xref:metadata-types/schema-file-definition.adoc[Schema Definition] to map the output layout of the file. By preceding these transforms with the Schema Mapping, the incoming stream will be already setup according to the expected output layout definition.
Schema mapping is a useful mapping transform to precede transforms like xref:pipeline/transforms/textfileoutput.adoc[Text File Output] or xref:pipeline/transforms/excelwriter.adoc[Excel Writer] when we decide to refer to a xref:metadata-types/static-schema-definition.adoc[Schema Definition] to map the output layout of the file. By preceding these transforms with the Schema Mapping, the incoming stream will be already setup according to the expected output layout definition.

== Options

[options="header"]
|===
|Option|Description
|Schema Definition|Name of the xref:metadata-types/schema-file-definition.adoc[Schema Definition] that we are going to map to the incoming stream.
|Schema Definition|Name of the xref:metadata-types/static-schema-definition.adoc[Schema Definition] that we are going to map to the incoming stream.
|Schema Fields|The set of schema fields that are mapped to stream fields.
|Stream Fields name|The set of stream fields that are mapped to schema fields.
|===
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -185,14 +185,14 @@ If you specify a value below zero (0) here, the filter string is searched for in

=== Fields Tab

The fields tab allows you to specify the information about the name and format of the fields being read from the text file. You can choose to use a xref:metadata-types/schema-file-definition.adoc[Schema Definition] or to define the required fields' layout manually.
The fields tab allows you to specify the information about the name and format of the fields being read from the text file. You can choose to use a xref:metadata-types/static-schema-definition.adoc[Schema Definition] or to define the required fields' layout manually.

Available options include:

[options="header"]
|===
|Option|Description
|Schema Definition|Name of the xref:metadata-types/schema-file-definition.adoc[Schema Definition] that we want to reference.
|Schema Definition|Name of the xref:metadata-types/static-schema-definition.adoc[Schema Definition] that we want to reference.
|Name|Name of the field
|Type|Type of the field can be either String, Date or Number
|Format|See Number Formats for a complete description of format symbols.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ This is commonly used to generate Comma Separated Values (CSV files) that can be

It is also possible to generate fixed width files by setting lengths on the fields in the fields tab.

You can choose to use a xref:metadata-types/schema-file-definition.adoc[Schema Definition] or to define the required fields' layout manually. If you decide to define the fields layout by using a xref:metadata-types/schema-file-definition.adoc[Schema Definition], use the xref:pipeline/transforms/schemamapping.adoc[Schema Mapping] transform to adjust the incoming stream according to the choosen xref:metadata-types/schema-file-definition.adoc[Schema Definition]
You can choose to use a xref:metadata-types/static-schema-definition.adoc[Schema Definition] or to define the required fields' layout manually. If you decide to define the fields layout by using a xref:metadata-types/static-schema-definition.adoc[Schema Definition], use the xref:pipeline/transforms/schemamapping.adoc[Schema Mapping] transform to adjust the incoming stream according to the choosen xref:metadata-types/static-schema-definition.adoc[Schema Definition]


|
Expand Down Expand Up @@ -123,7 +123,7 @@ The table below describes each of the options for configuring the field properti
[options="header"]
|===
|Option|Description
|Schema Definition|Name of the xref:metadata-types/schema-file-definition.adoc[Schema Definition] that we want to reference.
|Schema Definition|Name of the xref:metadata-types/static-schema-definition.adoc[Schema Definition] that we want to reference.
|Name|The name of the field.
|Type|Type of the field can be either String, Date or Number.
|Format|The format mask to convert with.
Expand Down

0 comments on commit 0ddbd69

Please sign in to comment.