docker-compose: ESG installer updates (#775)

* docker-compose: add deployment for ESG * cleanups --------- Co-authored-by: An Phi <[email protected]>
finos · Nov 3, 2023 · 5252870 · 5252870
1 parent 65b63de
commit 5252870
Show file tree

Hide file tree

Showing 17 changed files with 1,125 additions and 0 deletions.
diff --git a/installers/esg/.env b/installers/esg/.env
@@ -0,0 +1,68 @@
+# Compose 
+COMPOSE_PROJECT_NAME=legend
+
+# MongoDB 
+MONGODB_IMAGE_VERSION=4.2
+MONGODB_NAME=legend
+MONGODB_USERNAME=admin
+MONGODB_PASSWORD=password
+MONGODB_PORT=27017
+MONGODB_URI=mongodb://admin:password@legend-mongodb:27017
+
+# Session management
+MONGO_SESSION_ENABLED=true 
+
+# Gitlab OAuth configuration
+GITLAB_HOST=gitlab.com
+GITLAB_APP_ID=${GITLAB_APP_ID:?err}
+GITLAB_APP_SECRET=${GITLAB_APP_SECRET:?err}
+
+# Common
+LEGEND_EXTERNAL_BASE_URL=https://d7nifimeibcde.cloudfront.net
+
+# Engine
+ENGINE_IMAGE_VERSION=4.33.0
+ENGINE_MAVEN_VERSION=3.15.3
+ENGINE_HOST=localhost
+ENGINE_PORT=6300
+METADATA_PURE_HOST=localhost
+METADATA_PURE_PORT=6301
+METADATA_ALLOY_HOST=localhost
+METADATA_ALLOY_PORT=6200
+TEMP_DB_PORT=6302
+
+# SDLC
+SDLC_IMAGE_VERSION=0.142.0
+SDLC_MAVEN_VERSION=0.91.1
+SDLC_HOST=localhost
+SDLC_PORT=6100
+SDLC_ADMIN_PORT=6101
+SDLC_REDIRECT_URI=http://localhost:6100/api/auth/callback
+SDLC_PROJECT_TAG=legend
+SDLC_PROJECT_ID_PREFIX=^org\.demo\.legend\..+
+
+# Studio 
+STUDIO_IMAGE_VERSION=10.23.0
+STUDIO_HOST=localhost
+STUDIO_PORT=9000
+
+# Depot store
+DEPOT_STORE_IMAGE_VERSION=1.5.3
+DEPOT_STORE_HOST=localhost
+DEPOT_STORE_PORT=6201
+DEPOT_STORE_ADMIN_USER=${DEPOT_STORE_ADMIN_USER:-nobody}
+
+# Depot
+DEPOT_IMAGE_VERSION=1.5.3
+DEPOT_HOST=localhost
+DEPOT_PORT=6200
+
+# Query 
+QUERY_IMAGE_VERSION=8.38.0
+QUERY_HOST=localhost
+QUERY_PORT=9001
+
+# Postgres
+POSTGRES_IMAGE_VERSION=alpine3.17
+
+SETUP_DIR=$PWD/setup
diff --git a/installers/esg/README.md b/installers/esg/README.md
@@ -0,0 +1,145 @@
+# Overview
+
+This directory contains a Docker Compose specification that can be used to spin up an instance of Legend Studio and Legend Query.
+
+# Assumptions
+
+## Localhost
+
+This installer assumes that localhost resolves to where the various Legend JVMs are running and localhost can be reached from a browser. 
+
+If this is not the case, for e.g, Legend containers are being run on a machine accessible only by an IP or other names, all references to localhost, both in the .env and Gitlab OAuth configuration has to be changed.
+
+## Gitlab.com
+
+Out of the box, Legend Studio uses Gitlab for model version control. The installer uses the public gitlab.com instance.
+
+If you do not wish to use gitlab.com, you can use any Gitlab instance. Make sure to change all the GITHUB variables in .env to point to your Gitlab instance.
+
+# Usage Instructions
+
+## Create a Gitlab.com account
+
+Legend uses Gitlab as the identity provider. Create a user account at https://gitlab.com
+
+## Create a Gitlab OAuth application
+
+Create an OAuth application as described here https://docs.gitlab.com/ee/integration/oauth_provider.html
+
+The OAuth application should be configured as follows :
+
+- Redirect URI:
+
+```
+http://localhost:6300/callback
+http://localhost:6100/api/auth/callback
+http://localhost:6100/api/pac4j/login/callback
+http://localhost:6201/depot-store/callback
+http://localhost:6200/depot/callback
+http://localhost:9000/studio/log.in/callback
+http://localhost:9001/query/log.in/callback
+```
+
+- Enable the "Confidential" check box
+- Enable these scopes: openid, profile, api
+
+Save the application and record the application id and secret.
+
+## Set the app id and secret of your Gitlab application
+
+```
+export GITLAB_APP_ID=<add your app id>
+export GITLAB_APP_SECRET=<add your app secret>
+```
+
+## Studio 
+
+Start Studio as follows. 
+
+```
+./docker-compose.sh --profile studio up -d
+```
+
+After a few minutes, the containers should pass their health checks and be marked as healthy.
+
+```
+
+./docker-compose.sh ps
+       Name                     Command                  State                Ports
+---------------------------------------------------------------------------------------------
+legend-engine        /bin/sh -c java -cp /app/b ...   Up (healthy)   0.0.0.0:6300->6300/tcp
+legend-mongodb       docker-entrypoint.sh --auth      Up             0.0.0.0:27017->27017/tcp
+legend-sdlc          /bin/sh -c java -cp /app/b ...   Up (healthy)   0.0.0.0:6100->6100/tcp
+legend-studio        /bin/sh -c java -cp /app/b ...   Up (healthy)   0.0.0.0:9000->9000/tcp
+setup                /setup/setup.sh                  Exit 0```
+```
+
+### Use Studio
+
+Once all containers are running, you should be able to access Studio at `http://localhost:9000/studio`
+
+When accessing Studio for the first time, you will see a URL redirect, redirecting you to Gitlab.com to authorize the Legend OAuth application.
+
+Once authorized, you should be able to start using Studio.
+
+
+### Open a sample project 
+
+In the opening page of Studio, select the "Legend Installer Demo" project or navigate directly to the project using this link http://localhost:9000/studio/setup/40061958
+
+## Query
+
+Run Query as follows :
+```
+./docker-compose.sh --profile query up -d
+```
+
+After a few minutes, the containers should pass their health checks and be marked as healthy.
+
+```
+./docker-compose.sh ps
+       Name                     Command                  State                Ports
+---------------------------------------------------------------------------------------------
+legend-depot         /app/entrypoint.sh               Up (healthy)   0.0.0.0:6200->6200/tcp
+legend-depot-store   /app/entrypoint.sh               Up (healthy)   0.0.0.0:6201->6201/tcp
+legend-engine        /bin/sh -c java -cp /app/b ...   Up (healthy)   0.0.0.0:6300->6300/tcp
+legend-mongodb       docker-entrypoint.sh --auth      Up             0.0.0.0:27017->27017/tcp
+legend-query         /bin/sh -c java -cp /app/b ...   Up (healthy)   0.0.0.0:9001->9001/tcp
+legend-sdlc          /bin/sh -c java -cp /app/b ...   Up (healthy)   0.0.0.0:6100->6100/tcp
+legend-studio        /bin/sh -c java -cp /app/b ...   Up (healthy)   0.0.0.0:9000->9000/tcp
+setup                /setup/setup.sh                  Exit 0
+
+```
+
+### Use Query
+
+Once all containers are running, you should be able to access Studio at `http://localhost:9001/query`
+
+When accessing Studio for the first time, you will see a URL redirect, redirecting you to Gitlab.com to authorize the Legend OAuth application.
+
+Once authorized, you should be able to start using Query.
+
+### Index Projects
+
+Query allows you to access projects that have been indexed by the depot store server.
+
+We have automatically indexed a project into the depot store. If this project is not visible in query, execute the following command.
+
+```
+curl -v -X GET "http://localhost:6201/depot-store/api/queue/PROD-1234/org.finos.legend.demo/legend-query/1.0.2?maxRetries=5" -H "accept: text/plain"
+```
+
+# Swagger 
+
+Each component exposes a Swagger endpoint that can be used to explore the component's API.
+
+| Component | Endpoint |
+| ------ | ---------|
+| Engine | http://localhost:6300/api/swagger# |
+| SDLC Server | http://localhost:6100/api/swagger# |
+| Depot Store Server | http://localhost:6201/depot-store/api/swagger# | 
+| Depot Server | http://localhost:6200/depot/api/swagger# |
+
+# Known Issues / Gotchas
+
+In some cases, navigating to the Studio/Query web page can return an "unauthorized" error. This is usually because of stale cookies. Clear browser cookies for localhost (or domain name/IP that you are using) and try again.
diff --git a/installers/esg/depot-store/config/authorisedIdentities.json b/installers/esg/depot-store/config/authorisedIdentities.json
@@ -0,0 +1,7 @@
+{
+  "Artifacts": ["${DEPOT_STORE_ADMIN_USER}"],
+  "Store Administration": ["${DEPOT_STORE_ADMIN_USER}"],
+  "Projects": ["${DEPOT_STORE_ADMIN_USER}"],
+  "Notifications": ["${DEPOT_STORE_ADMIN_USER}"],
+  "Entities": ["${DEPOT_STORE_ADMIN_USER}"]
+}
diff --git a/installers/esg/depot-store/config/config.json b/installers/esg/depot-store/config/config.json
@@ -0,0 +1,109 @@
+{
+  "applicationName": "Depot Store Manager API",
+  "deployment": "DEV",
+  "sessionCookie": "LEGEND_DEPOT_STORE_JSESSIONID",
+  "urlPattern": "/depot-store/api/*",
+  "server": {
+    "type": "simple",
+    "applicationContextPath": "/",
+    "adminContextPath": "/admin",
+    "connector": {
+      "type": "http",
+      "port": ${DEPOT_STORE_PORT},
+      "maxRequestHeaderSize": "32KiB"
+    },
+    "gzip": {
+      "includedMethods": [
+        "GET",
+        "POST"
+      ]
+    },
+    "requestLog": {
+      "appenders": [
+        {
+          "type": "console",
+          "filterFactories": [
+            {
+              "type": "healthcheck-filter-factory"
+            }
+          ]
+        }
+      ]
+    }
+  },
+  "mongo": {
+    "url": "${MONGODB_URI}",
+    "database": "depot"
+  },
+  "openTracing": {
+    "openTracingUri": "your URL here",
+    "serviceName": "legend-depot-store",
+    "enabled": false
+  },
+  "logging": {
+    "level": "INFO",
+    "loggers": {
+      "org.jboss.shrinkwrap.resolver": "off",
+      "org.eclipse.aether": "off",
+      "Legend Depot Store Manager": {
+        "level": "info",
+        "appenders": [
+          {
+            "type": "console",
+            "logFormat": "%msg\r\n"
+          }
+        ]
+      }
+    }
+  },
+  "swagger": {
+    "resourcePackage": "org.finos.legend.depot.store",
+    "title": "Legend Depot Store Manager",
+    "uriPrefix": "/depot-store/api"
+  },
+  "pac4j": {
+    "callbackPrefix": "/depot-store",
+    "bypassPaths": [
+      "/depot-store/api/info"
+    ],
+    "bypassBranches": [
+      "/depot-store/api/queue"
+    ],
+    "clients": [
+      {
+        "org.finos.legend.server.pac4j.gitlab.GitlabClient": {
+          "name": "gitlab",
+          "clientId": "${GITLAB_APP_ID}",
+          "secret": "${GITLAB_APP_SECRET}",
+          "discoveryUri": "https://${GITLAB_HOST}/.well-known/openid-configuration",
+          "scope": "openid profile api"
+        }
+      }
+    ],
+    "mongoUri": "${MONGODB_URI}",
+    "mongoDb": "${MONGODB_NAME}",
+    "mongoSession": {
+      "enabled": "${MONGO_SESSION_ENABLED}",
+      "collection": "userSessions"
+    }
+  },
+  "filterPriorities": {
+    "Username": 1,
+    "OpenTracingFilter": 2,
+    "org.pac4j.j2e.filter.SecurityFilter": 3,
+    "org.pac4j.j2e.filter.CallbackFilter": 4,
+    "CORS": 5
+  },
+  "artifactRepositoryProviderConfiguration": {
+
+    "org.finos.legend.depot.artifacts.repository.maven.impl.MavenArtifactRepositoryConfiguration": {
+      "settingsLocation": "/config/settings.xml",
+      "versionsUpdateIntervalInMillis": 3600000,
+      "latestUpdateIntervalInMillis": 3600000
+    }
+  },
+  "includeProjectPropertiesConfiguration": {
+    "properties": []
+  },
+  "queue-interval": 30
+}