Memobase 2020 issueshttps://gitlab.switch.ch/groups/memoriav/memobase-2020/-/issues2021-10-29T15:34:03+02:00https://gitlab.switch.ch/memoriav/memobase-2020/services/postprocessing/mediametadatatodb/-/issues/1Verarbeitung von UPDATE und DELETE2021-10-29T15:34:03+02:00Sebastian SchüpbachVerarbeitung von UPDATE und DELETEFedora kann drei Arten von Events ausgeben:
* `CREATE`: Eine neue Ressource wurde erstellt => In MariaDB muss neuer Datensatz erstellt werden
* `UPDATE`: Eine Ressource wurde aktualisiert => In MariaDB müssen im betreffenden Datensatz d...Fedora kann drei Arten von Events ausgeben:
* `CREATE`: Eine neue Ressource wurde erstellt => In MariaDB muss neuer Datensatz erstellt werden
* `UPDATE`: Eine Ressource wurde aktualisiert => In MariaDB müssen im betreffenden Datensatz die Feldwerte aktualisiert werden
* `DELETE`: Eine Ressoure wurde gelöscht => In MariaDB muss der betreffende Datensatz gelöscht werden
Der `CREATE`-Fall ist bereits implementiert, `UPDATE` und `DELETE` müssten hingegen noch abgedeckt werden.
Der Eventtyp kann in der Message unter `@graph -> record -> eventType` gefunden werden.MatthiasMatthiashttps://gitlab.switch.ch/memoriav/memobase-2020/services/histogram/-/issues/2Error when reading from remote video files2020-07-24T11:53:49+02:00Sebastian SchüpbachError when reading from remote video filesDear @juergen.enge,
attempting to analyze a collection from ETHZ we get errors of the following kind:
```
2020-07-21T09:39:56.937 indexer::DoPanic [server.go:137] > ERROR - cannot create validateav of /tmp/indexer065602559: error execu...Dear @juergen.enge,
attempting to analyze a collection from ETHZ we get errors of the following kind:
```
2020-07-21T09:39:56.937 indexer::DoPanic [server.go:137] > ERROR - cannot create validateav of /tmp/indexer065602559: error executing (/usr/bin/ffmpeg [-v warning -i /tmp/indexer065602559 -f null -]): - : signal: killed
192.168.10.28 - - [21/Jul/2020:09:39:46 +0000] "GET /validateav/tmp/indexer065602559 HTTP/1.1" 500 218
192.168.10.28 - - [21/Jul/2020:09:39:56 +0000] "GET /exif/%2Ftmp%2Findexer065602559 HTTP/1.1" 301 0
2020-07-21T09:39:56.939 indexer::Exec [exif.go:72] > INFO - executing /usr/bin/exiftool [-json /tmp/indexer065602559]
192.168.10.28 - - [21/Jul/2020:09:39:56 +0000] "GET /exif/tmp/indexer065602559 HTTP/1.1" 200 1739
192.168.10.28 - - [21/Jul/2020:09:40:17 +0000] "GET /histogram/%2Ftmp%2Findexer073424722 HTTP/1.1" 301 0
192.168.10.28 - - [21/Jul/2020:09:40:17 +0000] "GET /histogram/tmp/indexer073424722 HTTP/1.1" 200 5
192.168.10.28 - - [21/Jul/2020:09:40:19 +0000] "GET /validateimage/%2Ftmp%2Findexer073424722 HTTP/1.1" 301 0
2020-07-21T09:40:19.704 indexer::Exec [image.go:57] > INFO - executing /usr/bin/identify [-verbose /tmp/indexer073424722]
2020-07-21T09:40:19.735 indexer::DoPanic [server.go:137] > ERROR - cannot create validateimage of /tmp/indexer073424722: error executing (/usr/bin/identify [-verbose /tmp/indexer073424722]): - identify: no decode delegate for this image format `' @ error/constitute.c/ReadImage/560.: exit status 1
192.168.10.28 - - [21/Jul/2020:09:40:19 +0000] "GET /validateimage/tmp/indexer073424722 HTTP/1.1" 500 296
192.168.10.28 - - [21/Jul/2020:09:40:19 +0000] "GET /validateav/%2Ftmp%2Findexer073424722 HTTP/1.1" 301 0
2020-07-21T09:40:19.737 indexer::Exec [av.go:57] > INFO - executing /usr/bin/ffmpeg [-v warning -i /tmp/indexer073424722 -f null -]
2020-07-21T09:40:29.751 indexer::DoPanic [server.go:137] > ERROR - cannot create validateav of /tmp/indexer073424722: error executing (/usr/bin/ffmpeg [-v warning -i /tmp/indexer073424722 -f null -]): - [mov,mp4,m4a,3gp,3g2,mj2 @ 0x558b53f888c0] st: 0 edit list: 1 Missing key frame while searching for timestamp: 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x558b53f888c0] st: 0 edit list 1 Cannot find an index entry before timestamp: 0.: signal: killed
192.168.10.28 - - [21/Jul/2020:09:40:19 +0000] "GET /validateav/tmp/indexer073424722 HTTP/1.1" 500 440
192.168.10.28 - - [21/Jul/2020:09:40:29 +0000] "GET /exif/%2Ftmp%2Findexer073424722 HTTP/1.1" 301 0
2020-07-21T09:40:29.767 indexer::Exec [exif.go:72] > INFO - executing /usr/bin/exiftool [-json /tmp/indexer073424722]
192.168.10.28 - - [21/Jul/2020:09:40:29 +0000] "GET /exif/tmp/indexer073424722 HTTP/1.1" 200 5509
192.168.10.28 - - [21/Jul/2020:09:40:59 +0000] "GET /histogram/%2Ftmp%2Findexer661691853 HTTP/1.1" 301 0
192.168.10.28 - - [21/Jul/2020:09:40:59 +0000] "GET /histogram/tmp/indexer661691853 HTTP/1.1" 200 5
192.168.10.28 - - [21/Jul/2020:09:41:00 +0000] "GET /validateimage/%2Ftmp%2Findexer661691853 HTTP/1.1" 301 0
2020-07-21T09:41:00.245 indexer::Exec [image.go:57] > INFO - executing /usr/bin/identify [-verbose /tmp/indexer661691853]
2020-07-21T09:41:00.267 indexer::DoPanic [server.go:137] > ERROR - cannot create validateimage of /tmp/indexer661691853: error executing (/usr/bin/identify [-verbose /tmp/indexer661691853]): - identify: no decode delegate for this image format `' @ error/constitute.c/ReadImage/560.: exit status 1
192.168.10.28 - - [21/Jul/2020:09:41:00 +0000] "GET /validateimage/tmp/indexer661691853 HTTP/1.1" 500 296
192.168.10.28 - - [21/Jul/2020:09:41:00 +0000] "GET /validateav/%2Ftmp%2Findexer661691853 HTTP/1.1" 301 0
2020-07-21T09:41:00.269 indexer::Exec [av.go:57] > INFO - executing /usr/bin/ffmpeg [-v warning -i /tmp/indexer661691853 -f null -]
2020-07-21T09:41:10.274 indexer::DoPanic [server.go:137] > ERROR - cannot create validateav of /tmp/indexer661691853: error executing (/usr/bin/ffmpeg [-v warning -i /tmp/indexer661691853 -f null -]): - : signal: killed
192.168.10.28 - - [21/Jul/2020:09:41:00 +0000] "GET /validateav/tmp/indexer661691853 HTTP/1.1" 500 218
192.168.10.28 - - [21/Jul/2020:09:41:10 +0000] "GET /exif/%2Ftmp%2Findexer661691853 HTTP/1.1" 301 0
2020-07-21T09:41:10.276 indexer::Exec [exif.go:72] > INFO - executing /usr/bin/exiftool [-json /tmp/indexer661691853]
192.168.10.28 - - [21/Jul/2020:09:41:10 +0000] "GET /exif/tmp/indexer661691853 HTTP/1.1" 200 1739
```
The files are often quite large (~1GB), so that could be a potential problem. However, the log of the Indexer shows no anomalies:
```
2020-07-21T11:47:14.602 indexer::getContent [server.go:127] > INFO - loading from http://digitalisate.afz.ethz.ch/?&guid=c94cc7b82aeb4d7c94be67c4a753f4b3
2020-07-21T11:47:20.023 indexer::getContent [server.go:165] > DEBUG - mimetype from server: video/mp4
2020-07-21T11:47:20.023 indexer::getContent [server.go:169] > INFO - full download of http://digitalisate.afz.ethz.ch/?&guid=c94cc7b82aeb4d7c94be67c4a753f4b3
192.168.9.210 - - [21/Jul/2020:11:47:14 +0000] "POST / HTTP/1.1" 200 262
2020-07-21T11:47:30.032 indexer::getContent [server.go:127] > INFO - loading from http://digitalisate.afz.ethz.ch/?&guid=14051e2784f94fb7ae7380297e22e613
2020-07-21T11:47:37.565 indexer::getContent [server.go:165] > DEBUG - mimetype from server: video/mp4
2020-07-21T11:47:37.565 indexer::getContent [server.go:169] > INFO - full download of http://digitalisate.afz.ethz.ch/?&guid=14051e2784f94fb7ae7380297e22e613
192.168.9.210 - - [21/Jul/2020:11:47:30 +0000] "POST / HTTP/1.1" 200 262
2020-07-21T11:47:47.575 indexer::getContent [server.go:127] > INFO - loading from http://digitalisate.afz.ethz.ch/?&guid=669364e0a48244cabfe0a8380cec8b0a
2020-07-21T11:47:54.017 indexer::getContent [server.go:165] > DEBUG - mimetype from server: video/mp4
2020-07-21T11:47:54.017 indexer::getContent [server.go:169] > INFO - full download of http://digitalisate.afz.ethz.ch/?&guid=669364e0a48244cabfe0a8380cec8b0a
```Jürgen EngeJürgen Engehttps://gitlab.switch.ch/memoriav/memobase-2020/services/externalapis/oai/-/issues/1Angabe eines nicht existenten Verbs für zu exception2020-07-15T10:58:39+02:00Günter HiplerAngabe eines nicht existenten Verbs für zu exceptionBeispiel
https://oai.memobase.k8s.unibas.ch/?verb=ListRecordBeispiel
https://oai.memobase.k8s.unibas.ch/?verb=ListRecordGünter HiplerGünter Hiplerhttps://gitlab.switch.ch/memoriav/memobase-2020/services/histogram/-/issues/1Use of go module2020-06-22T10:02:08+02:00Sebastian SchüpbachUse of go moduleDear Jürgen,
Is there any special reason why you don't use the Go module approach in this project like in the mediaserver project, for example? It would make the CI workflow much easier...
Thanks,
SebastianDear Jürgen,
Is there any special reason why you don't use the Go module approach in this project like in the mediaserver project, for example? It would make the CI workflow much easier...
Thanks,
SebastianJürgen EngeJürgen Engehttps://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/issues/7Mixed content error when connecting to imageserver in Kubernetes2020-06-12T08:22:44+02:00Sebastian SchüpbachMixed content error when connecting to imageserver in KubernetesDear Jürgen,
the mediaserver is up and running on Kubernetes: https://media.memobase.k8s.unibas.ch/.
The connection to the Cantaloupe instance works as well:
- https://media.memobase.k8s.unibas.ch/memo/sig-2/iiif/info.json
- https://me...Dear Jürgen,
the mediaserver is up and running on Kubernetes: https://media.memobase.k8s.unibas.ch/.
The connection to the Cantaloupe instance works as well:
- https://media.memobase.k8s.unibas.ch/memo/sig-2/iiif/info.json
- https://media.memobase.k8s.unibas.ch/iiif/sig-2/open/iiif/2/N_Eugen_Thierstein_206_12.jp2/full/full/0/default.jpg
However, it is not possible to view the image in the provided OpenSeadragon instance since we get a mixed content error:
```
Mixed Content: The page at 'https://media.memobase.k8s.unibas.ch/memo/sig-2/iiif/view' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://media.memobase.k8s.unibas.ch/memo/sig-2/iiif/info.json?auth='. This request has been blocked; the content must be served over HTTPS.
```
I assume that the error stems from this line: https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/blob/master/pkg/memostream/memoServerHandlerMain.go#L148.
I guess we can safely remove this switch since we use a proxy for the mediaserver which does not allow non-TLS connections.Jürgen EngeJürgen Engehttps://gitlab.switch.ch/memoriav/memobase-2020/services/postprocessing/media-converter/-/issues/1Tracking issue2020-08-10T10:59:21+02:00Sebastian SchüpbachTracking issueTODOs:
* [ ] [Definition of Fedora event stream message model](https://gitlab.switch.ch/memoriav/memobase-2020/services/postprocessing/fedora-media-extractor/snippets/180)
* [ ] How can we get the resource files from Fedora?
* [ ] Proc...TODOs:
* [ ] [Definition of Fedora event stream message model](https://gitlab.switch.ch/memoriav/memobase-2020/services/postprocessing/fedora-media-extractor/snippets/180)
* [ ] How can we get the resource files from Fedora?
* [ ] Processing of response from API
* [ ] Extraction of files considering that the service runs in Kubernetes
* [ ] There needs to be a shared folder with mediaserver
* [ ] The same goes for CantaloupeSebastian SchüpbachSebastian Schüpbachhttps://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/issues/6Job Failed #267482020-05-04T14:40:16+02:00Sebastian SchüpbachJob Failed #26748Should we fix this?
Job [#26748](https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/jobs/26748) failed for bd8ed287d14cbb580f89cc8efaaa28b6381ba304:Should we fix this?
Job [#26748](https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/jobs/26748) failed for bd8ed287d14cbb580f89cc8efaaa28b6381ba304:Jürgen EngeJürgen Engehttps://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/issues/5Job Failed #267522020-05-04T14:40:03+02:00Sebastian SchüpbachJob Failed #26752Should we fix this?
Job [#26752](https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/jobs/26752) failed for bd8ed287d14cbb580f89cc8efaaa28b6381ba304:Should we fix this?
Job [#26752](https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/jobs/26752) failed for bd8ed287d14cbb580f89cc8efaaa28b6381ba304:Jürgen EngeJürgen Engehttps://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/issues/4Job Failed #267442020-05-04T14:39:46+02:00Sebastian SchüpbachJob Failed #26744Should we fix this?
Job [#26744](https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/jobs/26744) failed for bd8ed287d14cbb580f89cc8efaaa28b6381ba304:Should we fix this?
Job [#26744](https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/jobs/26744) failed for bd8ed287d14cbb580f89cc8efaaa28b6381ba304:Jürgen EngeJürgen Engehttps://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/issues/3Tests failing2020-05-04T14:38:34+02:00Sebastian SchüpbachTests failingIf I run the tests for the media server, I get the following error:
```sh
go test gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/cmd/server
2020-05-04T11:15:48.249 memostream::ListenAndServe [memoServer.go:212] > INFO...If I run the tests for the media server, I get the following error:
```sh
go test gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/cmd/server
2020-05-04T11:15:48.249 memostream::ListenAndServe [memoServer.go:212] > INFO - starting HTTP memoServer at http://localhost:41937
--- FAIL: TestProxy (0.00s)
server_test.go:100: query http://localhost:41937/memo/sig-001 - proxy
server_test.go:108: error checking proxy: Get "http://localhost:41937/memo/sig-001": dial tcp [::1]:41937: connect: connection refused
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x80303a]
goroutine 15 [running]:
testing.tRunner.func1.1(0x86e140, 0xc8b270)
/usr/lib/go/src/testing/testing.go:940 +0x2f5
testing.tRunner.func1(0xc000124d80)
/usr/lib/go/src/testing/testing.go:943 +0x3f9
panic(0x86e140, 0xc8b270)
/usr/lib/go/src/runtime/panic.go:969 +0x166
gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/cmd/server.TestProxy(0xc000124d80)
/home/seb/.go/src/gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/cmd/server/server_test.go:110 +0x55a
testing.tRunner(0xc000124d80, 0x90e5c8)
/usr/lib/go/src/testing/testing.go:991 +0xdc
created by testing.(*T).Run
/usr/lib/go/src/testing/testing.go:1042 +0x357
FAIL gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/cmd/server 0.005s
FAIL
```
Are there further setup steps required?
This happens independently of running the tests locally or in the [CI workflow](https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/jobs/26745).
For the time being I've disabled the tests in the CI definition.Jürgen EngeJürgen Engehttps://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/issues/2Job Failed #264682020-05-04T11:36:01+02:00Sebastian SchüpbachJob Failed #26468Job [#26468](https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/jobs/26468) failed for 6e2df229caa21725a0db49d43c9b4590df7086b1:Job [#26468](https://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/jobs/26468) failed for 6e2df229caa21725a0db49d43c9b4590df7086b1:Sebastian SchüpbachSebastian Schüpbachhttps://gitlab.switch.ch/memoriav/memobase-2020/services/streaming-server/-/issues/1Structure of project2020-04-24T08:35:53+02:00Sebastian SchüpbachStructure of projectAs a Go language novice I have two "dummy" questions regarding the structure of the project.
- I would expect that I can run the `go build` command in the root directory of the project (and get the binary as a result in the same direct...As a Go language novice I have two "dummy" questions regarding the structure of the project.
- I would expect that I can run the `go build` command in the root directory of the project (and get the binary as a result in the same directory). However if I do so, I get an error along the lines of `can't load package: package .: no Go files in ...`. Obviously the error is caused by the absence of a Go file in the root dir. To that effect the same doesn't apply when running the command in the `main/` subdirectory where the main Go files reside. But this feels for me, as a person used to other build tools, as rather "awkward". What is your intention to not place the main package in the root directory?
- If I try to fetch the dependencies in the `main` directory, the commands exits with an error:
```sh
GIT_TERMINAL_PROMPT=1 go get
Username for 'https://gitlab.switch.ch': sschuepbach
Password for 'https://sschuepbach@gitlab.switch.ch':
# cd .; git clone -- https://gitlab.switch.ch/memoriav/memobase-2020.git /home/seb/.go/src/gitlab.switch.ch/memoriav/memobase-2020
Cloning into '/home/seb/.go/src/gitlab.switch.ch/memoriav/memobase-2020'...
remote: The project you were looking for could not be found.
fatal: repository 'https://gitlab.switch.ch/memoriav/memobase-2020.git/' not found
package gitlab.switch.ch/memoriav/memobase-2020/streaming-server/memostream: exit status 128
```
How can I get around this error - especially in the light of building the project in the CI process, which doesn't allow for interactivity? Wouldn't it be possible to use relative paths here (after setting the `GOPATH` variable), or choose the Go module approach?
Maybe it would help if you document in the README how you build the project.
Thanks in advance!Jürgen EngeJürgen Enge