Merge pull request 'release_1.1.11' (#15) from release_1.1.11 into ma…

…ster

Reviewed-on: https://gitea.intranda.com/intranda/goobi-vocabulary-server/pulls/15

.github/workflows/develop-build.yml

@@ -34,7 +34,7 @@ jobs:
       - name: Create ZIP archive for setup scripts
         uses: montudor/action-zip@v1
         with:
-          args: zip -qq -r setup-scripts.zip install
+          args: zip -qq -r vocabulary-init-script.zip install
       - name: Get current date
         id: date
         run: echo "::set-output name=date::$(date +'%Y-%m-%d %H:%M:%S %Z')"
@@ -53,5 +53,5 @@ jobs:
           files: |
             module-*/target/*.jar
             module-core/src/main/resources/application.properties
-            setup-scripts.zip
+            vocabulary-init-script.zip
             migration-tool.zip

.github/workflows/release-build.yml

@@ -34,7 +34,7 @@ jobs:
       - name: Create ZIP archive for setup scripts
         uses: montudor/action-zip@v1
         with:
-          args: zip -qq -r setup-scripts.zip install
+          args: zip -qq -r vocabulary-init-script.zip install
       - name: Release
         id: create_release
         uses: softprops/action-gh-release@v2
@@ -46,5 +46,5 @@ jobs:
           files: |
             module-*/target/*.jar
             module-core/src/main/resources/application.properties
-            setup-scripts.zip
+            vocabulary-init-script.zip
             migration-tool.zip

.gitignore

@@ -36,4 +36,5 @@ build/
 
 ### Python ###
 /migration/vmenv
+/migration/migration.csv
 __pycache__/

docs/de/migration.md

@@ -11,6 +11,7 @@ Erstellen Sie zunächst eine virtuelle Python-Umgebung, aktivieren Sie diese und
 ```bash
 python3 -m venv vmenv
 . vmenv/bin/activate
+pip install wheel # This needs to be done beforehand as a separate call
 pip install requests mysql-connector-python==8.4.0 alive_progress lxml
 ```
 

docs/de/setup.md

@@ -13,16 +13,21 @@ Diese Dokumentation beschreibt den Prozess der Installation und Ersteinrichtung
 ## Einrichtung von Goobi Workflow zur Kommunikation mit dem Vokabularserver
 - Goobi Workflow verwendet seit Version `24.07` den neuen Vokabularserver.
 - Konfigurieren Sie die Variablen `vocabularyServerHost`, `vocabularyServerPort` und `vocabularyServerToken` in der Datei `goobi_config.properties` entsprechend der Konfiguration Ihres Vokabularservers.
+- Alternativ zu `vocabularyServerHost` und `vocabularyServerPort` kann auch die Variable `vocabularyServerAddress` gesetzt werden (beispielsweise `vocabularyServerAddress=https://external.address.com/vocabulary`). Diese Variable erlaubt auch eine SSL Verbindung.
 - Starten Sie Goobi Workflow neu, damit die Änderungen wirksam werden.
 - Navigieren Sie zu `Administration` > `Vocabulare`, um zu überprüfen, ob alles funktioniert. Sie sollten eine Liste von Vokabularen sehen, wenn alles in Ordnung ist (nicht jetzt, sondern nachdem Sie einige Vokabulare erstellt oder die bestehenden migriert haben). Wenn etwas nicht funktioniert, wird eine rote Fehlermeldung angezeigt.
 
 ## Ersteinrichtung
-- Für den ordnungsgemäßen Betrieb benötigt der Vokabularserver einige Ausgangsdaten.
+- Im Falle einer Datenmigration benötigt der Vokabularserver einige Ausgangsdaten.
 - Diese Daten enthalten Sprachangaben (wenn mehrsprachige Vokabulare verwendet werden) und Feldtypdefinitionen. 
-- Sie können das folgende Skript verwenden, welches einige Beispielsprachen und Feldtypen installiert.
-- Laden Sie das [Initial Data Script](https://github.com/intranda/goobi-vocabulary-server/raw/develop/install/default_setup.sh) herunter.
-- Ändern Sie die Variablen `HOST` und `TOKEN` am Anfang entsprechend der Konfiguration des Vokabularservers, lassen Sie das Suffix `/api/v1` unverändert.
-- Führen Sie das Skript aus.
+- Diese Daten sind in einem Minimaldatensatz hinterlegt und können einfach mithilfe des dafür vorgesehenen Installationsscriptes installiert werden.
+- Laden Sie das [Vokabular-Initialisierungs-Tool](https://github.com/intranda/goobi-vocabulary-server/releases/latest/download/vocabulary-init-script.zip) herunter.
+- Falls nicht bereits geschehen, starten Sie den Vokabularserver.
+- Entpacken Sie das Archiv.
+- Passen Sie die Variablen `HOST`, `PORT` und `TOKEN` an Ihre Konfiguration an und starten Sie das Script wie folgt:
+```bash
+HOST=localhost PORT=8081 TOKEN=secret /path/to/the/script/install.sh minimal
+```
 
 ## Installationsskript
 Der Vokabularserver benötigt Java 17, der Systemd-Service geht davon aus, dass Java 17 der System-Default ist.
@@ -100,8 +105,9 @@ grep ^vocabularyServerToken= /opt/digiverso/goobi/config/goobi_config.properties
 sudo systemctl restart vocabulary.service & sudo journalctl -u vocabulary.service  -f -n 0 | grep -q "Started VocabularyServerApplication in"
 
 # initial set up
-wget https://github.com/intranda/goobi-vocabulary-server/raw/develop/install/default_setup.sh -O /tmp/default_setup.sh
-bash /tmp/default_setup.sh
+wget https://github.com/intranda/goobi-vocabulary-server/releases/latest/download/vocabulary-init-script.zip -O /tmp/vocabulary-init-script.zip
+sudo unzip /tmp/vocabulary-init-script.zip -d "${VOC_PATH}"
+HOST=localhost PORT=${VOC_PORT} TOKEN=${VOC_TOKEN} ${VOC_PATH}/vocabulary-init-script/install.sh minimal
 
 ## test
 curl -s http://localhost:${VOC_PORT}/api/v1/types --header "Authorization: Bearer $VOC_TOKEN" | jq -r '._embedded.fieldTypeList[] .name'
@@ -118,15 +124,8 @@ curl "http://localhost:${VOC_PORT:-8081}/api/v1/types" --header "Authorization:
 ```
 - Das Ergebnis sollte wie folgt aussehen:
 ```bash
+Any Text
 Anything
 Boolean
 Number
-Word
-skos:prefLabel
-skos:altLabel
-skos:definition
-skos:editorialNote
-skos:related
-skos:closeMatch
-skos:exactMatch
 ```

docs/en/migration.md

@@ -11,6 +11,7 @@ First, create a virtual Python environment, activate it and install all required
 ```bash
 python -m venv vmenv
 . vmenv/bin/activate
+pip install wheel # This needs to be done beforehand as a separate call
 pip install requests mysql-connector-python==8.4.0 alive_progress lxml
 ```
 
@@ -30,6 +31,23 @@ If you don't want to create any field types, you can start the data migration wi
 ```bash
 python vocabulary-migrator.py --vocabulary-server-host localhost --vocabulary-server-port 8081 --goobi-database-host localhost --goobi-database-port 3306 --goobi-database-name goobi --goobi-database-user goobi --goobi-database-password goobi --continue-on-error --fallback-language eng
 ```
+
+### Script 
+The above two points, the virtual Python environment and the migration of the vocabulary data in a typical installation, as root:
+```bash
+cd /opt/digiverso/vocabulary/migration
+python3 -m venv vmenv
+. vmenv/bin/activate
+pip install requests mysql-connector-python==8.4.0 alive_progress lxml
+VOC_PORT=$(sudo grep -oP '^server.port=\K.*' /opt/digiverso/vocabulary/application.properties)
+VOC_TOKEN=$(sudo grep -oP '^security.token=\K.*' /opt/digiverso/vocabulary/application.properties)
+DB_GOOBI_PW=$(sudo xmlstarlet sel -t -v '//Resource/@password' -n /etc/tomcat9/Catalina/localhost/goobi.xml)
+python vocabulary-migrator.py --vocabulary-server-host localhost --vocabulary-server-port "${VOC_PORT}" --vocabulary-server-token "${VOC_TOKEN}" --goobi-database-host localhost --goobi-database-port 3306 --goobi-database-name goobi --goobi-database-user goobi --goobi-database-password "${DB_GOOBI_PW}" --continue-on-error --fallback-language ger
+
+# Test
+curl -s http://localhost:8081/api/v1/vocabularies --header "Authorization: Bearer $VOC_TOKEN" | jq -r '._embedded.vocabularyList[] .name'
+```
+
 **Hint** Change the parameters according to your configuration. The `fallback-language` parameter defines the default language to be used for a multi-lingual vocabulary field for which no default language could be derived. The `continue-on-error` option prevents the migration tool to stop on data migration errors. These errors can occur if the data could not be inserted into the new vocabulary server. Possible reasons might be:
 - The vocabulary record is empty.
 - The vocabulary record contains data that is incompatible with some type restrictions.

docs/en/setup.md

@@ -7,63 +7,128 @@ This documentation describes the process of bootstrapping the vocabulary server.
 - Adapt configuration file properly and remove unmodified lines.
     - Database credentials and database name.
     - Base URL and port.
-- **TODO** *Install the `vocabulary-server.jar` and the `application.properties` configuration file both directly into a new folder (e. g. `/opt/digiverso/vocabulary/`)*
+    - Security token (this must also be configured identically in Goobi).
+- Create a Systemd service to start the service automatically. 
 
-## Start as systemd service
-- **TODO** *Create a systemd service unit for the vocabulary server (The application should be able to correctly shutdown on SIGTERM)*
-- **TODO** *Admin documentation here*
-- Run `java -jar vocabulary-server-VERSION.jar`.
-- If startup succeeds, you will see a line like this after a few seconds:
-```bash
-Started VocabularyServerApplication in 4.244 seconds (process running for 4.581)
-```
+## Setting up Goobi Workflow to communicate with the vocabulary server
 
-## Goobi Workflow Setup Communication
-- Goobi Workflow uses the new vocabulary server since version `24.07`.
-- Configure the `vocabularyServerHost` and `vocabularyServerPort` variables in `goobi_config.properties` according to your vocabulary server configuration.
+- Goobi Workflow has been using the new vocabulary server since version `24.07`.
+- Configure the variables `vocabularyServerHost`, `vocabularyServerPort` and `vocabularyServerToken` in the `goobi_config.properties` file according to the configuration of your vocabulary server.
+- As an alternative to `vocabularyServerHost` and `vocabularyServerPort`, the variable `vocabularyServerAddress` can also be set (e. g. `vocabularyServerAddress=https://external.address.com/vocabulary`). This variable also allows an SSL connection.
 - Restart Goobi Workflow for the changes to take effect.
-- Navigate to `Administration` > `Vocabularies` to check if everything works. You should see a list of vocabularies if everything is okay (not right now, but after you have created some vocabularies or migrated the existing ones). You will see a red error message if something doesn't work.
-
-## Initial Setup
-- For proper operation, the vocabulary server needs some initial data.
-- This data contains language specifications (if multi-language vocabularies are used) and field type definitions. 
-- You can use the following script that installs some sample languages and field types.
-- Download the [Initial Data Script](https://jenkins.intranda.com/job/intranda/job/vocabulary-server/job/develop/lastSuccessfulBuild/artifact/install/default_setup.sh).
-- Change the `HOST` variable at the top according to the vocabulary server configuration, leave the `/api/v1` suffix unchanged.
-- Run the script.
-
-## Security
-- You can setup Apache url restrictions in order to secure the vocabulary server from unauthorized access.
-- **TODO** *Admins, please find out what and how to do it in detail.*
-
-## Installation Test
-- For all commands, change host and port accordingly.
-- After the initial setup, check that types were created successfully:
+- Navigate to `Administration` > `Vocabularies` to check that everything is working. You should see a list of vocabularies if everything is OK (not now, but after you have created some vocabularies or migrated the existing ones). If something is not working, you will see a red error message.
+
+
+## Initial setup
+- In the case of data migration, the vocabulary server requires some initial data.
+- This data contains language information (if multilingual vocabularies are used) and field type definitions.
+- This data is stored in a minimal data set and can be easily installed using the installation script provided.
+- Download the [Vocabulary-Initialization-Tool](https://github.com/intranda/goobi-vocabulary-server/releases/latest/download/vocabulary-init-script.zip).
+- If you have not already done so, start the vocabulary server.
+- Unpack the archive.
+- Adapt the variables `HOST`, `PORT` and `TOKEN` to your configuration and start the script as follows:
 ```bash
-curl http://localhost:8081/api/v1/types | jq -r '._embedded.fieldTypeList[] .name'
+HOST=localhost PORT=8081 TOKEN=secret /path/to/the/script/install.sh minimal
 ```
-- The result should look like:
+
+## Installation script
+The vocabulary server requires Java 17, the systemd service assumes that Java 17 is the system default.
+
+For the above three points, under Ubuntu:
 ```bash
-Anything
-Boolean
-Number
-Word
-skos:prefLabel
-skos:altLabel
-skos:definition
-skos:editorialNote
-skos:related
-skos:closeMatch
-skos:exactMatch
+export VOC_PORT=8081
+export VOC_TOKEN=$(</dev/urandom tr -dc '[:alnum:]' | head -c17)
+export VOC_PATH=/opt/digiverso/vocabulary
+export VOC_USER=vocabulary
+export VOC_SQL_USER=${VOC_USER}
+export VOC_SQL_DB=${VOC_USER}
+export PW_SQL_VOC=$(</dev/urandom tr -dc '[:alnum:]' | head -c17)
+
+# create install folder
+sudo mkdir ${VOC_PATH}
+# download and link vocabulary server application file
+wget https://github.com/intranda/goobi-vocabulary-server/releases/latest/download/vocabulary-server-core.jar -O - | sudo tee ${VOC_PATH}/vocabulary-server-core.jar >/dev/null
+
+# create system user which will run the service
+sudo adduser --system --home ${VOC_PATH}/home --shell /usr/sbin/nologin --no-create-home --disabled-login ${VOC_USER}
+
+# download the vocabulary migration tools
+wget https://github.com/intranda/goobi-vocabulary-server/releases/latest/download/migration-tool.zip -O /tmp/migration-tool.zip
+sudo unzip /tmp/migration-tool.zip -d "${VOC_PATH}"
+
+# download and set up the config file
+wget https://github.com/intranda/goobi-vocabulary-server/releases/latest/download/application.properties -O - | sudo tee ${VOC_PATH}/application.properties >/dev/null
+sudo sed -re "s|^(server.port=).*|\1${VOC_PORT}|" \
+     -e "s|^#?(security.token=).*|\1${VOC_TOKEN}|" \
+     -e "s|^(spring.datasource.username=).*|\1${VOC_SQL_USER}|" \
+     -e "s|^(spring.datasource.password=).*|\1${PW_SQL_VOC}|" \
+     -e "s|^(spring.datasource.url=).*|\1jdbc:mariadb://localhost:3306/${VOC_SQL_DB}|" \
+     -i ${VOC_PATH}/application.properties
+sudo chown ${VOC_USER}: ${VOC_PATH}/application.properties
+sudo chmod 600 ${VOC_PATH}/application.properties
+
+# install a systemd service unit file
+cat << EOF | sudo tee /etc/systemd/system/vocabulary.service
+[Unit]
+Description=Goobi Vocabulary Server
+After=mysql.service remote-fs.target
+Requires=mysql.service remote-fs.target
+
+[Service]
+WorkingDirectory=${VOC_PATH}
+Restart=always
+RestartSec=20s
+StartLimitInterval=100s
+StartLimitBurst=4
+ExecStart=/usr/bin/java -jar vocabulary-server-core.jar
+User=${VOC_USER}
+NoNewPrivileges=true
+ProtectSystem=true
+PrivateTmp=yes
+
+[Install]
+WantedBy=default.target tomcat9.service
+EOF
+sudo systemctl daemon-reload
+sudo systemctl enable vocabulary.service
+
+# create and configure the database
+sudo mysql -e "CREATE DATABASE ${VOC_SQL_DB} CHARACTER SET = 'utf8mb4' COLLATE = 'utf8mb4_unicode_ci';
+               CREATE USER '${VOC_SQL_USER}'@'localhost' IDENTIFIED BY '${PW_SQL_VOC}';
+               GRANT ALL PRIVILEGES ON ${VOC_SQL_DB}.* TO '${VOC_SQL_USER}'@'localhost' WITH GRANT OPTION;
+               FLUSH PRIVILEGES;"
+
+# append vocabulary server address to the Goobi workflow config
+grep ^vocabularyServerHost= /opt/digiverso/goobi/config/goobi_config.properties || echo "vocabularyServerHost=localhost"   | sudo tee -a /opt/digiverso/goobi/config/goobi_config.properties
+grep ^vocabularyServerPort= /opt/digiverso/goobi/config/goobi_config.properties || echo "vocabularyServerPort=${VOC_PORT}" | sudo tee -a /opt/digiverso/goobi/config/goobi_config.properties
+grep ^vocabularyServerToken= /opt/digiverso/goobi/config/goobi_config.properties || echo "vocabularyServerToken=${VOC_TOKEN}" | sudo tee -a /opt/digiverso/goobi/config/goobi_config.properties
+
+# start the vocabulary server and wait for startup
+sudo systemctl restart vocabulary.service & sudo journalctl -u vocabulary.service  -f -n 0 | grep -q "Started VocabularyServerApplication in"
+
+# initial set up
+wget https://github.com/intranda/goobi-vocabulary-server/releases/latest/download/vocabulary-init-script.zip -O /tmp/vocabulary-init-script.zip
+sudo unzip /tmp/vocabulary-init-script.zip -d "${VOC_PATH}"
+HOST=localhost PORT=${VOC_PORT} TOKEN=${VOC_TOKEN} ${VOC_PATH}/vocabulary-init-script/install.sh minimal
+
+## test
+curl -s http://localhost:${VOC_PORT}/api/v1/types --header "Authorization: Bearer $VOC_TOKEN" | jq -r '._embedded.fieldTypeList[] .name'
 ```
-- If a data migration has been done, check that all vocabularies have been migrated:
+
+## Accessibility
+- You can make the vocabulary server accessible from outside by connecting a proxy with access control in front of it.
+
+## Installation test
+- Change the host and port accordingly for all commands.
+- After the initial setup, check whether the field types have been created successfully:
 ```bash
-curl http://localhost:8081/api/v1/vocabularies/all | jq -r '._embedded.vocabularyList[] .name'                
+curl ‘http://localhost:${VOC_PORT:-8081}/api/v1/types’ --header ‘Authorisation: Bearer $VOC_TOKEN’ | jq -r ‘._embedded.fieldTypeList[] .name’
 ```
-- Check that links resolve correctly (see configuration):
+
+- The result should look like this:
 ```bash
-curl http://localhost:8081/api/v1/records/1 | jq
+Any Text
+Anything
+Boolean
+Number
 ```
-The `_links` JSON element should contain references to other resources. 
-These URLs should be valid and resolvable. 
-The host part of these URLs is generated from the request.

install/.gitignore

@@ -1 +1 @@
-/cache.txt
+cache.txt