Acquirer Transaction Filter Batch

Component defining the batch process for filtering the input transaction records, based on a list of pan, recovered from a local directory or through a remote service.

Execution requirements

The artifact consists of an executable jar produced with spring-boot, therefore all the project dependencies are contained within the jar, together with the classes that contains the business logic. The artifact is completely autonomous and usable on any device that has a JVM

To install and run the batch, it's required:

• Java 1.8+
• batch-transaction-filter.jar artifact

For the application of PGP encryption to the result file, produced by the batch, there will also need a file containing the public key to be used, reported in Appendix 1 - PGP public key. For the application of decryption of the PGP pan file, it must provided a file containing the secret key to be applied for the operation. To produce the artifact from the source code it will be necessary to have an installation of Maven and a java compiler (jdk1.8+).

Bundle Distribution

The batch-transaction-filter.jar artifact will be provided in a bundle containing a folder with the configurations and the structures of the files, in order to allow an immediate execution with the default configuration, which point to path/folders listed in the bundle. The default service will poll to check for files to be processed every minute.

Inside the bundle there are also the public key file shown in the appendix, and some sample for a first test run.

Nota: The bundle contains a version potentially out of alignment with the implementation. In the default configuration attempts to connect to the REST services and the sftp comunication are disabled.

The bundle structure and the files is contains are described below:

• batch-transaction-filter.jar, the artifact containing the batch service
• /config, folder containing the configuration files
• /config/application.yml, file containing the configuration properties for the service
• /resources, folder containing the resources and folder for running the bundle under a default configuration
• /hpans, folder where to insert the files containing the pan list
• /keys, folder containing the keys for pgp encryption
• /transactions, folder where to insert the files containing the transactions to be processed
• /output, folder where the files produced by the service will be inserted
• /sample-files, folder containing test files for execution

Generating artifact from source

To obtain a version of the artifact produced directly from the source code, a Maven instance must be appropriately configured on the machine to use via the command line. Once the source has been downloaded, run the following command from the root directory:

mvn clean package <execution_options>

If the command is executed without other options, the artifact will be produced once the unit tests (declared in the module) have been performed. To perform the operation without waiting for the execution and validation of the tests, run the command in the following form:

mvn clean package -DskipTests

The artifact will be created into the target folder at root level

Database connection

Spring Batch uses a repository on which you can track the executions performed by the service. If there is no particular configuration: an in- memory instance will be executed to allow the batch to be executed. The configuration of the bundle uses this mode for greater immediacy of use. If you want to set it please refer to the properties in Appendix 2 - Configuration properties.

REST Services Connection

The Batch Acquirer is configurabile for contacting the salt recovery service, to be applied for the PAN hashing, and the pan list recovery service, to be used for filtering the transaction records. To enabled this services, the following configuration properties must be enabled: batchConfiguration.TransactionFilterBatch.saltRecovery.enabled and batchConfiguration.TransactionFilterBatch.hpanList.enabled.

Endpoint configurations are through the properties rest-client.hpan.base-url, for the base configuration , and the endpoint properties for the two services respectively rest-client.hpan.list.url and rest-client.hpan.salt.url.

If the client is to be configured for TLS/SSL protocol usage, the configuration property rest-client.hpan.mtls.enabled is to be used, and the keystore and trust-store files for the client, to be applied in the certificate exchange, respectively through the rest-client.hpan.key-store.file and rest-client.hpan.trust-store.file properties.

Other applicable configurations are those related to passwords to be applied for certificates, identified by the rest-client.hpan.key-store.password and rest-client.hpan.trust-store.password configurations. The type used for files containing certificates can also be defined, and the algorithm used for the encryption. By default the files are in Java's JKS format, using the standard implementation of the X509 algorithm. For dedicated configurations refer to properties listed in Appendix 2 - Configuration properties.

For references to the services displayed through Azure's API service, you can find the corresponding links in Appendix 3 - Authentication Services Acquirer.

Execution guidelines

• Install and configure the environment so that the Java 1.8+ version is available, as indicated in the prerequisites

• In case of execution of bundled version, extract artifact and resource in a position of your choice, if no additional
  configuration is required, refer to the execution step at the end of the paragraph. Consider whether to use the sample files contained in the transactions and hpan folders.

• If you are not using the bundled version, please produce a version of the artifact via source code, as indicated in the corresponding paragraph of the manual. Prepare a configuration application.yml file and, if needed, other files .yml or .properties to be used for the configuration properties.

• Place the batch-transaction-filter.jar artifact in a location of your choice

• Place in a location of your choice, the configuration files, supplied together with the artifact in the bundle, or your own.

• Place on the machine, the files of the public and/or private key for pgp, if one of the file encryption/decryption function is active.

• Configure the path to the file containing the public key, through the batchConfiguration.TransactionFilterBatch.transactionFilter.publicKeyPath property, or through the environment variable ACQ_BATCH_INPUT_PUBLIC_KEYPATH.

  Note: The configuration is strictly needed only if the encryption function of the produced files is enabled. In the case of configuration on file, the path must be preceded by the prefix file:/. for example:

  batchConfiguration.TransactionFilterBatch.transactionFilter.publicKeyPath = file:/C/:Development/keys/public.asc

• Configure the pointing to the file containing the private key, through the property batchConfiguration.TransactionFilterBatch.panList.secretKeyPath, or through the environment variable ACQ_BATCH_INPUT_SECRET_KEYPATH.

  Note: The configuration is strictly necessary only if the decryption function of the files containing the pan list is enabled. In the case of configuration on file, the path must be preceded by prefix file:/. for example::

  batchConfiguration.TransactionFilterBatch.panList.secretKeyPath = file:/C:/Development/keys/secret.asc

• Configure the passphrase to be applied if the secret key is enabled, through the batchConfiguration.TransactionFilterBatch.panList.passphrase property , or via the ACQ_BATCH_INPUT_SECRET_PASSPHRASE environment variable.

• Define a folder where the path files, to be processed, will be placed

• Configure the path to the transaction files to be processed, through the batchConfiguration.TransactionFilterBatch.transactionFilter.transactionDirectoryPath property, or through the environment variables ACQ_BATCH_TRX_INPUT_PATH for the folder, and ACQ_BATCH_INPUT_FILE_PATTERN, for the pattern of files to read.

  Note: In the case of file configuration, the path must be preceded by the prefix file:/. for example:

  batchConfiguration.TransactionFilterBatch.transactionFilter.transactionDirectoryPath = file:/C:/Development/transactions/*.csv

• Define a folder for the files containing the PAN list

• Configure the path to the files containing the pan list, through the batchConfiguration.TransactionFilterBatch.panList.hpanDirectoryPath property , or through the environment variables ACQ_BATCH_HPAN_INPUT_PATH for the folder, and ACQ_BATCH_HPAN_INPUT_FILE_PATTERN, for the pattern of files to read.

  Note: In the case of configuration on file, the path must be preceded by the prefix file:/. for example:

  batchConfiguration.TransactionFilterBatch.panList.hpanDirectoryPath = file:/C:/Development/hpan/*.pgp

• Define a folder for the output files

• Configure the pointing to the trace files to be processed, through the property batchConfiguration.TransactionFilterBatch.transactionFilter.outputDirectoryPath, or through the environment variable ACQ_BATCH_OUTPUT_PATH

  Note: In the case of configuration on file, the path must be preceded by the prefix file:/. for example:

  batchConfiguration.TransactionFilterBatch.transactionFilter.outputDirectoryPath = file:/C:/Development/output

• Configure the hashing application for the pan list, through the batchConfiguration.TransactionFilterBatch.panList.applyHashing property, or through the environment variable ACQ_BATCH_PAN_LIST_APPLY_HASHING

• Configure for decryption of the file containing the pan list, through the batchConfiguration.TransactionFilterBatch.panList.applyDecrypt property, or through the environment variable ACQ_BATCH_PAN_LIST_APPLY_DECRYPT

• Configure the hash application for transactions, through the batchConfiguration.TransactionFilterBatch.transactionFilter.applyHashing property, or through the environment variable ACQ_BATCH_TRX_LIST_APPLY_HASHING

• Configure for product encryption, through the batchConfiguration.TransactionFilterBatch.transactionFilter.applyEncrypt property, or through the environment variable ACQ_BATCH_TRX_LIST_APPLY_ENCRYPT

• Configure for the hash application in the transactions reported in the product file, through the batchConfiguration.TransactionFilterBatch.transactionFilter.saveHashing property, or through the environment variable ACQ_BATCH_TRX_LIST_HASHING_SAVE

• To send the product file on SFTP channel, the functionality must be enabled through batchConfiguration.TransactionFilterBatch.transactionSender.enabled properties, then the configurations related to the host, the user used and the authentication method, password-based, or through certificate must be reported. Configurations for sftp are listed under the batchConfiguration.TransactionFilterBatch.transactionFilter.sftp root in the configuration properties appendix.

• To enable the passages related to the jump recovery services, or the pan list through REST services, configure the properties following the definitions in the section Connecting to REST Services.

• Configure the scheduling configuration of the process, through a cron rule, through the batchConfiguration.TransactionFilterBatch.cron property, or through the environment variable ACQ_BATCH_INPUT_CRON

• Apply any other changes to the configuration parameters, the full list of properties is described in Appendix 2 - Configuration properties

• Run the batch. The batch can be started via the java command:

  java -jar --spring.config.location=

  Note: replace with the path to the proper configuration directory

  java -jar batch-transaction-filter.jar --spring.config.location=C:\Development\batch-transaction-file\property\

  For the bundle execution, referring to the structure already present, execute:

  java -jar batch-transaction-filter.jar --spring.config.location=file:config/

Appendix 1 - Public PGP Key

For any problem relating to the use of the public key and for the release of the specifications and / or updates relating to the public key to be used to encrypt the file, it is mandatory to contact the structure delegated by PagoPA (ref. SIA OPE Innovative Payments - sistemisti_bigdata@sia.eu).

Nota: The file filled with the key is included in the bundle containing the artifact for executing the batch.

Appendix 2 - Configuration properties

1. Logging properties

Key	Description	Default	Mandatory	Values
logging.file	Location where the log files will be written		NO
logging.level.root	Log level	INFO	NO	TRACE, DEBUG, INFO, WARN, ERROR, FATAL, OFF

2. Batch properties - General

Key	Description	Default	Mandatory	Values
batchConfiguration.TransactionFilterBatch.successArchivePath	Move initial csv to success path	file:/${ACQ_BATCH_SUCCESS_PATH:${ACQ_BATCH_TRX_INPUT_PATH:}/success}	YES
batchConfiguration.TransactionFilterBatch.errorArchivePath	Path where the files whose processing goes wrong are moved	file:/${ACQ_BATCH_ERROR_PATH:${ACQ_BATCH_TRX_INPUT_PATH:}/error}	YES
batchConfiguration.TransactionFilterBatch.cron	Batch scheduling	${ACQ_BATCH_INPUT_CRON:0 0/1 * 1/1 * ?}	YES
batchConfiguration.TransactionFilterBatch.partitionerMaxPoolSize	Batch max partitioner setting	${ACQ_BATCH_INPUT_PART_MAX_POOL_SIZE:5}	NO
batchConfiguration.TransactionFilterBatch.partitionerCorePoolSize	Batch partitioner pool setup	${ACQ_BATCH_INPUT_PART_CORE_POOL_SIZE:5}	NO
batchConfiguration.TransactionFilterBatch.readerMaxPoolSize	Maximum number of transaction csv file readers	${ACQ_BATCH_INPUT_PART_READ_MAX_POOL_SIZE:5}	NO
batchConfiguration.TransactionFilterBatch.readerCorePoolSize	Maximum number of transaction csv file readers	${ACQ_BATCH_INPUT_PART_READ_CORE_POOL_SIZE:5}	NO
batchConfiguration.TransactionFilterBatch.tablePrefix	Table prefix containing the metadata related to the execution of the batch, if active	${ACQ_BATCH_INPUT_TABLE_PREFIX:BATCH_}	NO

3. Batch properties - PAN List reading

Key	Description	Default	Mandatory	Values
batchConfiguration.TransactionFilterBatch.panList.hpanDirectoryPath	The path where you saved the file pgp containing HPAN	file:/${ACQ_BATCH_HPAN_INPUT_PATH:}/${ACQ_BATCH_INPUT_FILE_PATTERN:*.csv}	YES
batchConfiguration.TransactionFilterBatch.panList.secretKeyPath	Path where the private key is saved	file:/${ACQ_BATCH_INPUT_SECRET_KEYPATH:}	YES
batchConfiguration.TransactionFilterBatch.panList.passphrase	Passphrase for the private key	${ACQ_BATCH_INPUT_SECRET_PASSPHRASE:}	YES
batchConfiguration.TransactionFilterBatch.panList.partitionerSize	Size of the partitioner used to read the file	${ACQ_BATCH_INPUT_PARTITIONER_SIZE:1}	NO
batchConfiguration.TransactionFilterBatch.panList.chunkSize	Size of the chunks used for reading the file	${ACQ_BATCH_INPUT_PARTITIONER_SIZE:1}	NO
batchConfiguration.TransactionFilterBatch.panList.skipLimit	Maximum number of records discarded before execution is blocked	${ACQ_BATCH_INPUT_SKIP_LIMIT:0}	NO
batchConfiguration.TransactionFilterBatch.panList.applyDecrypt	Flag indicating whether or not to apply the decrypt at the hpan file	${ACQ_BATCH_PAN_LIST_APPLY_DECRYPT:true}	YES	TRUE FALSE

4. Batch properties - Transaction list reading

Key	Description	Default	Mandatory	Values
batchConfiguration.TransactionFilterBatch.transactionFilter.transactionDirectoryPath	Path where the transaction file to be processed is read	file:/${ACQ_BATCH_TRX_INPUT_PATH:}/${ACQ_BATCH_INPUT_FILE_PATTERN:*.csv}	YES
batchConfiguration.TransactionFilterBatch.transactionFilter.outputDirectoryPath	Path where the final file is writtene	file:/${ACQ_BATCH_OUTPUT_PATH:${ACQ_BATCH_TRX_INPUT_PATH:}/output}	YES
batchConfiguration.TransactionFilterBatch.transactionFilter.publicKeyPath	Path containing the public key with which to encrypt the result file	file:/${ACQ_BATCH_INPUT_PUBLIC_KEYPATH:}	YES
batchConfiguration.TransactionFilterBatch.transactionFilter.partitionerSize	Partitiner size for transaction files	${ACQ_BATCH_INPUT_PARTITIONER_SIZE:10}	NO
batchConfiguration.TransactionFilterBatch.transactionFilter.chunkSize	Chunck size for reading transaction files	${ACQ_BATCH_INPUT_CHUNK_SIZE:1000}	NO
batchConfiguration.TransactionFilterBatch.transactionFilter.skipLimit	Maximum number of records discarded before execution is blocked	${ACQ_BATCH_INPUT_SKIP_LIMIT:0}	NO
batchConfiguration.TransactionFilterBatch.transactionFilter.timestampPattern	Pattern relating to the transaction date	${ACQ_BATCH_INPUT_TIMESTAMP_PATTERN:MM/dd/yyyy HH:mm:ss}	NO
batchConfiguration.TransactionFilterBatch.transactionFilter.applyHashing	Flag that drives the hashing to the pan present in the transaction file	${ACQ_BATCH_TRX_LIST_APPLY_HASHING:false}	SI	TRUE FALSE
batchConfiguration.TransactionFilterBatch.transactionFilter.applyEncrypt	Flag to define whether to encrypt the result file	${ACQ_BATCH_TRX_LIST_APPLY_ENCRYPT:true}	YES	TRUE FALSE
batchConfiguration.TransactionFilterBatch.transactionFilter.saveHashing	Flag to define whether to save the hashing of the pan in the result file	${ACQ_BATCH_TRX_LIST_HASHING_SAVE:false}	YES	TRUE FALSE
batchConfiguration.TransactionFilterBatch.transactionFilter.linesToSkip	Number of lines to skip from the beginning of the file (e.g. to avoid the header )	${ACQ_BATCH_INPUT_LINES_TO_SKIP:0}	NO

5. Batch properties - SFTP

Key	Description	Default	Mandatory	Values
batchConfiguration.TransactionFilterBatch.transactionSender.enabled	Indicates whether the sending to the sftp channel is active or not	${ACQ_BATCH_TRX_SENDER_ENABLED:true}	SI	TRUE FALSE
batchConfiguration.TransactionFilterBatch.transactionFilter.sftp.localdirectory	Local directory from which to get the file to be sent on remote SFTP	${SFTP_LOCAL_DIR:}	SI
batchConfiguration.TransactionFilterBatch.transactionFilter.sftp.remotedirectory	Remote SFTP directory to copy the file to	${SFTP_REMOTE_DIR:}	SI
batchConfiguration.TransactionFilterBatch.transactionFilter.sftp.filenamepattern	Name / pattern of the file to be moved to remote SFTP	${SFTP_FILE_PATTERN:}	SI
batchConfiguration.TransactionFilterBatch.transactionFilter.sftp.filextension	File extension to copy to remote SFTP	${SFTP_FILE_EXTENSION:}	NO
batchConfiguration.TransactionFilterBatch.transactionFilter.sftp.host	SFTP Host	${SFTP_HOST:}	SI
batchConfiguration.TransactionFilterBatch.transactionFilter.sftp.port	SFTP Port	${SFTP_PORT:22}	SI
batchConfiguration.TransactionFilterBatch.transactionFilter.sftp.user	User for access to SFTP	${SFTP_USER:}	SI
batchConfiguration.TransactionFilterBatch.transactionFilter.sftp.password	Password for access to SFTP	${SFTP_PASSWORD:}	SI
batchConfiguration.TransactionFilterBatch.transactionFilter.sftp.timeout	Timeout related to connection with SFTP	${SFTP_SOCKET_TIMEOUT:0:}	SI
connectors.sftpConfigurations.connection.privateKey	Indicates the file for channel authentication will take place via a private key	file:/${SFTP_PRIVATE_KEY:}	NO
connectors.sftpConfigurations.connection.passphrase	Indicates the passphrase associated with the private key	${SFTP_PASSPHRASE:}	NO

6. Batch properties - REST services

Key	Description	Default	Mandatory	Values
batchConfiguration.TransactionFilterBatch.saltRecovery.enabled	Enable the recovery service for the salt	${ACQ_BATCH_SALT_RECOVERY_ENABLED:false}	NO
batchConfiguration.TransactionFilterBatch.hpanListRecovery.enabled	Enable the recovery service for the pan list	${ACQ_BATCH_HPAN_RECOVERY_ENABLED:true}	NO
batchConfiguration.TransactionFilterBatch.hpanListRecovery.directoryPath	Location where the file containing the list of files will be saved	${ACQ_BATCH_HPAN_INPUT_PATH:}	NO
batchConfiguration.TransactionFilterBatch.hpanListRecovery.filename	Name assigned to the recovered file	${CSV_TRX_BATCH_HPAN_LIST_FILENAME:}	NO
batchConfiguration.TransactionFilterBatch.hpanListRecovery.attemptExtract	Indication if the recovered file will be in the form of a compressed file with checksum	${ACQ_BATCH_HPAN_LIST_ATTEMPT_EXTRACT:false}	NO
batchConfiguration.TransactionFilterBatch.hpanListRecovery.checksumFilePattern	Pattern for the checksum file	${ACQ_BATCH_HPAN_LIST_CHECKSUM_FILE_PATTERN: .checksum. }	NO
batchConfiguration.TransactionFilterBatch.hpanListRecovery.listFilePattern	Pattern for the list containing the pan list	${CSV_TRX_BATCH_HPAN_LIST_CHECKSUM_FILE_PATTERN: .*\.csv }	NO
rest-client.hpan.base-url	Base url for REST services	${HPAN_SERVICE_URL}	NO
rest-client.hpan.list.url	Endpoint pan list service	/list	NO
rest-client.hpan.salt.url	Endpoint salt service	/salt	NO
rest-client.hpan.mtls.enabled	Enable MTLS for salt and pan list services	${HPAN_SERVICE_MTLS_ENABLED:true}	NO
rest-client.hpan.key-store.file	Path to key-store	file:/${HPAN_SERVICE_KEY_STORE_FILE:}	NO
rest-client.hpan.key-store.type	Key-store type	${HPAN_SERVICE_KEY_STORE_TYPE:#{null}}	NO
rest-client.hpan.key-store.algorithm	Key-store algorithm	${HPAN_SERVICE_KEY_STORE_ALGORITHM:#{null}}	NO
rest-client.hpan.key-store.password	Key-store password	${HPAN_SERVICE_KEY_STORE_PASSWORD:}	NO
rest-client.hpan.trust-store.file	Path to trust-store	file:/${HPAN_SERVICE_TRUST_STORE_FILE:}	NO
rest-client.hpan.trust-store.type	Trust-store type	${HPAN_SERVICE_TRUST_STORE_TYPE:#{null}}	NO
rest-client.hpan.trust-store.password	Trust-store password	${HPAN_SERVICE_TRUST_STORE_PASSWORD:}	NO

7. Batch properties - File handling

Key	Description	Default	Mandatory	Values
batchConfiguration.TransactionFilterBatch.transactionFilter.deleteLocalFiles	Enable deletion of locally generated files (all files related to batch computation)	${FLAG_DELETE_LOCAL_FILE:true}	SI	TRUE FALSE

8. Batch properties - Repository

Key	Description	Default	Mandatory	Values
spring.datasource.driver-class-name	Classname for the driver to user	${BATCH_DB_CLASS_NAME:}	SI
spring.datasource.url	Database connection url	${BATCH_DB_CONN_URL:}	SI
spring.datasource.username	Database username for login	${BATCH_DB_USERNAME:}	SI
spring.datasource.password	Database password for user login	${BATCH_DB_USERNAME:}	SI
spring.datasource.hikari.schema	Database schema	${BATCH_DB_SCHEMA:}	SI
spring.jpa.database-platform	Database dialect	${BATCH_DB_DIALECT:}	SI

Appendix 3 - Acquirer Services Authentication

The interactions for the Acquirer batch services use a mutual authentication mechanism over TLS/SSL protocol, through the exchange of public certificates, issued by a CA (the certifying authority), used for the verification by both actors compared to the keys in their possession. For this mechanism to be applicable it will therefore be necessary that:

The Client will have to be configured to send requests on TLS/SSL protocol, indicating a file containing the public certificate issued for the machine that will implement the requests, and will also need to be configured to receive a collection of keys to be used for verification of the certificates reported by the car contacted.

the API must be configured to accept requests on TLS/SSL protocol, it must be configured to use a collection of keys on which to apply the certificate verification, must be configured to provide an public certificate, used by the Client for authentication of the machine to which the request is directed.

Using the services provided on Azure, to enable the authentication process, the following must be entered certificates relating to the SOs (https://docs.microsoft.com/en-us/azure/api-management/api-management-howto-ca-certificates). The format of the certificates will in this case be ".cer".

The certificates used in the case of services displayed through Azure, must be included in the dedicated section, these must be in the ".pfx" format. (https://docs.microsoft.com/en-us/azure/api-management/api-management-howto-mutual-certificates).

The services displayed on Azure will allow the configuration of the backend services displayed so as to enable the mutual authentication process based on a given certificate. In the case of services used by Acquirer introduces a dedicated policy to allow the authentication process through multiple certificates, to allow the use of certificates for Acquirers (https://docs.microsoft.com/en-us/azure/api-management/api-management-howto-mutual-certificates-for-clients).