add ability to do environment variables substitution (#14)

* add ability to do environment variables substitution * Update docs/configuration/chain.md Co-authored-by: Geraint Ballinger <[email protected]> Co-authored-by: Geraint Ballinger <[email protected]>
adevinta · Aug 7, 2020 · 6b48666 · 6b48666
1 parent 71111f6
commit 6b48666
Show file tree

Hide file tree

Showing 3 changed files with 42 additions and 1 deletion.
diff --git a/docs/configuration/chain.md b/docs/configuration/chain.md
@@ -14,3 +14,39 @@ The 2 first environment variables must contain the configuration (or a subset) i
 The final zoe configuration will be the result of merging all the above configuration values in the increasing level priority.
 
 The special `common.yml` configuration file is supposed to contain configuration values that are common to all the environments. Registered expressions or secrets providers are a good use case. 
+
+## Environment variables substitutions
+
+It is possible to use environment variables inside the zoe yaml or json configuration files and zoe will automatically substitute those variables at runtime from the environment.
+
+The following example shows the use of an environment variable inside a yaml configuraiton file:
+
+```yaml
+clusters:
+
+  my-cluster:
+    props:
+      bootstrap.servers: ${BOOTSTRAP_SERVER}
+      key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
+      value.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
+    registry: ${SCHEMA_REGISTRY:-localhost:8081}
+
+runners:
+    default: local
+```
+
+With the above configuration, zoe replaces the `${BOOTSTRAP_SERVER}` and `${SCHEMA_REGISTRY}` expressions with the value of the environment variables `BOOTSTRAP_SERVER` and `SCHEMA_REGISTRY` respectively. If the `SCHEMA_REGISTRY` is not found, the specified default value is used (`localhost:8081` in this case). If no default value is specified and no environment variable with the corresponding name exists, the expression is simply not replaced.
+
+The following command should make the previous statement clearer:
+
+```bash tab="command"
+BOOTSTRAP_SERVER=localhost:9092 zoe -o table config clusters list
+```
+
+```json tab="output"
+┌────────────┬────────────────┬────────────────┬────────┬────────┐
+│ cluster    │ brokers        │ registry       │ topics │ groups │
+├────────────┼────────────────┼────────────────┼────────┼────────┤
+│ my-cluster │ localhost:9092 │ localhost:8081 │        │        │
+└────────────┴────────────────┴────────────────┴────────┴────────┘
+```
diff --git a/zoe-cli/build.gradle.kts b/zoe-cli/build.gradle.kts
@@ -187,6 +187,7 @@ dependencies {
     implementation("org.jetbrains.kotlin:kotlin-reflect")
     implementation("org.eclipse.jgit:org.eclipse.jgit:5.7.0.202003110725-r")
 
+    implementation("org.apache.commons:commons-text:1.9")
     implementation("org.koin:koin-core:2.0.1")
     implementation("com.jakewharton.picnic:picnic:0.3.1")
     implementation("com.github.ajalt:clikt:2.8.0")

diff --git a/zoe-cli/src/config/urls.kt b/zoe-cli/src/config/urls.kt
@@ -17,6 +17,8 @@ import com.adevinta.oss.zoe.service.utils.userError
 import com.fasterxml.jackson.databind.JsonNode
 import com.fasterxml.jackson.databind.node.NullNode
 import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.commons.text.StringSubstitutor
+import org.apache.commons.text.lookup.StringLookupFactory
 import java.io.File
 import java.net.URL
 
@@ -80,7 +82,9 @@ class LocalConfigDirUrlProvider(private val directory: File) : ConfigUrlProvider
 
 private fun loadConfigFromUrl(url: URL): JsonNode {
     logger.info("loading config from url : $url")
-    val content = url.readBytes()
+    val content = with(StringSubstitutor(StringLookupFactory.INSTANCE.environmentVariableStringLookup())) {
+        replace(String(url.readBytes()))
+    }
     val parsed = if (url.path.endsWith(".yml")) yaml.readTree(content) else json.readTree(content)
     return parsed ?: NullNode.getInstance()
 }