GitLab

Merge branch 'master' of gitlab.switch.ch:memoriav/memobase-2020/services/import-process/fedora-ingest-service

Failures are now fatal reports.

Previously the RdfHandler only searched for locators starting with
the string "sftp:" in RDF resources of RICO type "digitalObject",
which resulted in binaries for thumbnails not being ingested into
fedora. This commit fixes this bug by searching for locators that
point to the sftp server in all instantiatons.

When the placeholder for an thumbnail, which is at a subpath
[INSTANTIATION_ID]/derived is created before the placeholder for the
instantiation itself, this leads to errors during the ingest, because
fedora creates a pairtree node for each part in a path. Pairtree nodes
can't hold metadata themselves.

Memo 669: improve mimetype handling

See merge request !11

Use RDFDataMgr interface instead of model writer.
Replace constants with variables.

Set required envvars

See merge request !10

MEMO-774: adapt to new import workflow

See merge request !9

MEMO-734: Update fedora client to version 0.6.2

See merge request !8

MEMO-740: Set max.poll.records to 10

See merge request !7

The IngestService needs more time if a lot of binary resources need to be ingested. It then can't commit the read messages on the next poll. By setting a lower limit for max.poll.records the next poll on the Kafka topic should occur before the default max.poll.interval.ms of 5 minutes has expired.