Add support for environment variables in pipelines.yml

[fbaligand]

Actually, if I define LS_BASE environment variable and put path.config: "${LS_BASE}/pipeline-*.conf" in pipelines.yml, I get the following error :
[INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/current-directory/${LS_BASE}/pipeline-agg1*.conf"}

In comparison, with the same config in logstash.yml (main pipeline), it works fine.

=> It would be great to support environment variables injection in pipelines.yml to have the same behaviour/possibilities both with one main pipeline (logstash.yml) or several pipelines (pipelines.yml).

[bferg314]

I'd also really like to have this feature so I could migrate the pipelines.yml between environments, and only have to update the environmental variable.

Note: a workaround for this is to just always launch LogStash from the directory with all of your configs.

[vbohata]

For me it looks like a bug. It does not translate all ENV variables.

[githubmui]

Support of environment variables in pipline.yml would be great!

[jvosu2]

this would be very useful to not have to hard code paths to different environments. Expected to have this ability, the same as in logstash.yml

[robcowart]

I referenced this issue in a comment regarding #7989. However I also wanted to elaborate on the use-case here.

The use-case is running Logstash and associated configs in a docker container. In our case, the container could have up to 35 pipelines enabled. However not every instance of the container needs all of the pipelines (depending on the environment and scale requirements).

If the ability to specify environment variables in pipelines.yml was available (ideally combined with an option to specify pipeline.enable to enable/disable pipelines), it would allow configurations like this...

- pipeline.id: barracuda
  pipeline.enable: ${LS_BARRACUDA_ENABLE:true}
  pipeline.workers: ${LS_BARRACUDA_WORKERS:2}
  path.config: "/etc/logstash/pipelines/barracuda/*.conf"

- pipeline.id: forcepoint
  pipeline.enable: ${LS_FORCEPOINT_ENABLE:true}
  pipeline.workers: ${LS_FORCEPOINT_WORKERS:2}
  path.config: "/etc/logstash/pipelines/barracuda/*.conf"

NOTE: The example above also demonstrates usage similar to Logstash's support for specifying default values in the environment variable markup. This would further provide more flexibility in the use of environment variables.

The docker_compose.yml file which starts the container would then be able to provide control of which pipelines are started by defining environment variables. For example...

environment:
  LS_BARRACUDA_ENABLE: false

  LS_FORCEPOINT_ENABLE: true
  LS_FORCEPOINT_WORKERS: 4

This would eliminate the need to produce a container for every possible combination of pipelines.

[fbaligand]

Hi,

I just realized that this feature has already been implemented, since Logstash v7.4.0!
It has been fixed in this PR: #11081

So thanks @jsvd for this enhancement!

Finally, I think it should be documented here:
https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

[jsvd]

Thanks Fabien :)

we have this documented on the pipelines.yml documentation, that is linked from that doc. I noticed we only mention allowing env var interpolation and not keystore secrets (which also work), so this PR should fix this documentation gap.

[fbaligand]

Thanks to you @jsvd!