Commit 00663a6f authored by Sebastian Schüpbach's avatar Sebastian Schüpbach
Browse files

update README.md

parent 1912f6c6
Pipeline #25885 passed with stages
in 3 minutes and 54 seconds
### Import Process Bridge
# Import Process Bridge
The purpose of this service consists primarily in providing a "shortcut" between the first part of the import process (where data normalisation and aggregation happen) and the second part (where data is transformed to fit the needs of the respective endpoint). The reason for its existence lies in the high latency if data is intermediary pushed to the Fedora repository and read from there for post-processing, which has been the case before introducing this service.
As a central service in the import pipeline it has one other purpose: Keeping track of all the records being processed by the import process pipeline. This helps the among others the deletion process to identify records which match a certain pattern (see [Import Process Delete repository](https://gitlab.switch.ch/memoriav/memobase-2020/services/deletion-components/import-process-delete) for more information).
## Configuration
In order to work correctly, some environment variables have to be set:
* `APPLICATION_ID`: Id used by Kafka Streams application (see [Kafka documentation](https://kafka.apache.org/documentation/#streamsconfigs_application.id) for details)
* `TOPIC_IN`: Name of Kafka topic where messages are read from
* `TOPIC_OUT`: Name of Kafka topic where messages are written to (without environment postfix)
* `TOPIC_OUT_POSTFIX`: Environment postfix of Kafka topic where messages are written to
* `TOPIC_PROCESS`: Name of Kafka topic where status reports are written to
* `TOPIC_TRANSACTIONS_RECORDS`: Name of Kafka topic where record imports are saved
* `TOPIC_TRANSACTIONS_RECORD_SETS`: Name of Kafka topic where recordSet imports are saved
* `TOPIC_TRANSACTIONS_INSTITUTIONS`: Name of Kafka topic where institution imports are saved
Connects the import process with the post-processing workflow
\ No newline at end of file
......@@ -10,4 +10,4 @@ recordsTransactionsTopic: mb-di-transactions-records-prod
recordSetsTransactionsTopic: mb-gi-transactions-record-sets-prod
institutionsTransactionsTopic: mb-gi-transactions-institutions-prod
k8sEnvironment: prod
\ No newline at end of file
k8sEnvironment: prod
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment