Commit a7be3ac2 authored by Jonas Waeber's avatar Jonas Waeber
Browse files

Add LICENSE AND README to chart

parent cadd6880
Pipeline #31571 passed with stages
in 4 minutes and 46 seconds
This diff is collapsed.
# Import Process Bridge
The purpose of this service consists primarily in providing a "shortcut" between the first part of the import process (where data normalisation and aggregation happen) and the second part (where data is transformed to fit the needs of the respective endpoint). The reason for its existence lies in the high latency if data is intermediary pushed to the Fedora repository and read from there for post-processing, which has been the case before introducing this service.
As a central service in the import pipeline it has one other purpose: Keeping track of all the records being processed by the import process pipeline. This helps the among others the deletion process to identify records which match a certain pattern (see [Import Process Delete repository](https://gitlab.switch.ch/memoriav/memobase-2020/services/deletion-components/import-process-delete) for more information).
[Confluence Doku](https://ub-basel.atlassian.net/wiki/spaces/MEMOBASE/pages/468549883/Import+Process+Bridge)
## Configuration
In order to work correctly, some environment variables have to be set:
* `APPLICATION_ID`: Id used by Kafka Streams application (see [Kafka documentation](https://kafka.apache.org/documentation/#streamsconfigs_application.id) for details)
* `TOPIC_IN`: Name of Kafka topic where messages are read from
* `TOPIC_OUT`: Name of Kafka topic where messages are written to (without environment postfix)
* `TOPIC_OUT_POSTFIX`: Environment postfix of Kafka topic where messages are written to
* `TOPIC_PROCESS`: Name of Kafka topic where status reports are written to
* `TOPIC_TRANSACTIONS_RECORDS`: Name of Kafka topic where record imports are saved
* `TOPIC_TRANSACTIONS_RECORD_SETS`: Name of Kafka topic where recordSet imports are saved
* `TOPIC_TRANSACTIONS_INSTITUTIONS`: Name of Kafka topic where institution imports are saved
* `REPORTING_STEP_NAME`: Determines the step name in the reports.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment