Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

environment variable parsing in http plugin #2923

Closed
fgeck opened this issue Apr 1, 2024 · 10 comments
Closed

environment variable parsing in http plugin #2923

fgeck opened this issue Apr 1, 2024 · 10 comments
Labels
kind/enhancement New feature or request needs/triage

Comments

@fgeck
Copy link

fgeck commented Apr 1, 2024

I store my crowdsec configuration in a github respository.
For notifications I use http plugin to push data to my telegram bot, but the issue relates to any http plugin which pushes data using credentials to authenticate.

In the format section I can already use sprig env function to substitue env vars (see below).

To not be forced to store my credentials in the notifications/http.yaml file I would like to use environment variables in the http plugin which will be resolved.

Example:

type: http # Don't change
name: http_default # Must match the registered plugin in the profile

# One of "trace", "debug", "info", "warn", "error", "off"
log_level: info

# group_wait: 30s # Time to wait collecting alerts before relaying a message to this plugin, eg "30s"
# group_threshold:    # Amount of alerts that triggers a message before <group_wait> has expired, eg "10"
# max_retry:          # Number of attempts to relay messages to plugins in case of error
# timeout:            # Time to wait for response from the plugin before considering the attempt a failure, eg "10s"

#-------------------------
# plugin-specific options

# The following template receives a list of models.Alert objects
# The output goes in the http request body

# Replace XXXXXXXXX with your Telegram chat ID
format: |
  {
   "chat_id": "{{ env "TELEGRAM_CHAT_ID" }}", 
   "text": "
     {{range . -}}  
     {{$alert := . -}}  
     {{range .Decisions -}}
     {{.Value}} will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}}.
     {{end -}}
     {{end -}}
   ",
   "reply_markup": {
      "inline_keyboard": [
          {{ $arrLength := len . -}}
          {{ range $i, $value := . -}}
          {{ $V := $value.Source.Value -}}
          [
              {
                  "text": "See {{ $V }} on shodan.io",
                  "url": "https://www.shodan.io/host/{{ $V -}}"
              },
              {
                  "text": "See {{ $V }} on crowdsec.net",
                  "url": "https://app.crowdsec.net/cti/{{ $V -}}"
              }
          ]{{if lt $i ( sub $arrLength 1) }},{{end }}
      {{end -}}
      ]
  }

url: https://api.telegram.org/{TELEGRAM_BOT_KEY}/sendMessage

method: POST
headers:
  Content-Type: "application/json"

Why is this needed?

  • Security improvement for users that store crowdsec configuration in a github repository as they can use environment variables to substitute credentials in their configuration yaml files
Copy link

github-actions bot commented Apr 1, 2024

@fgeck: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

Copy link

github-actions bot commented Apr 1, 2024

@fgeck: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

  • /kind feature
  • /kind enhancement
  • /kind bug
  • /kind packaging
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

@github-actions github-actions bot added kind/enhancement New feature or request and removed needs/kind labels Apr 1, 2024
@ynsta
Copy link

ynsta commented Nov 22, 2024

+1 would like the same thing.

@ynsta
Copy link

ynsta commented Nov 22, 2024

@LaurenceJJones
Copy link
Contributor

LaurenceJJones commented Nov 22, 2024

I tried and it works if you use ${VAR} cf https://docs.crowdsec.net/docs/configuration/crowdsec_configuration/#environment-variables

Just to confirm the ${VAR} even within "chat_id": "{{ env "TELEGRAM_CHAT_ID" }}", ? as it might since when we unmarshal the YAML is done at crowdsec level, and env sprig function is ran at notification level which doesnt get access to the same environment variables as crowdsec.

@instantdreams
Copy link

I tried and it works if you use ${VAR} cf https://docs.crowdsec.net/docs/configuration/crowdsec_configuration/#environment-variables

Can you paste your configuration so I can compare it to mine?

@LaurenceJJones
Copy link
Contributor

LaurenceJJones commented Dec 22, 2024

tldr; currently variables will work with configurations that are not the format key such as headers using the typical method $VAR or ${VAR} (but will not be shown in cscli notifications command but it does work from what I tested), we need to inform the broker about the expansion for it to work with the format key.

Test showing the current cscli working with variables outside the format key

TEST=1234 cscli notifications inspect http_default
 -            Type:            http
 -            Name:    http_default
 -         Timeout:              5s
 -          Format: {{ range . -}}
{{ $alert := . -}}
{
  "extras": {
    "client::display": {
    "contentType": "text/markdown"
  }
},
"priority": 3,
"title": "test from crowdsec! using $TEST",
"message": "message!"
}
{{ end -}}

 -       log_level:            info
 -             url: http://127.0.0.1:9090/
 -          method:            POST
 -         headers: map[   Content-Type:application/json   X-Test-Header:          $TEST]

Please note the X-Test-Header: $TEST

Connection from 127.0.0.1:54692
POST / HTTP/1.1
Host: 127.0.0.1:9090
User-Agent: Go-http-client/1.1
Content-Length: 166
Content-Type: application/json
X-Test-Header: 1234
Accept-Encoding: gzip

{
  "extras": {
    "client::display": {
    "contentType": "text/markdown"
  }
},
"priority": 3,
"title": "test from crowdsec! using $TEST",
"message": "message!"
}

As you can see the $TEST in the format is not replaced but the header X-Test-Header is replaced.

Note

Using env sprig command will never work as the plugin process is created with empty environment since a http plugin may have multiple configurations it needs to hold so we wouldnt want the plugin to hold the environments.

Add some debug information so we can make an informed choice on how to proceed:

It seems we already strictexpand the configuration when the configuration is sent to plugin via

data = []byte(csstring.StrictExpand(string(data), os.LookupEnv))

this means variables used outside the format scope should work for "headers" in this case, however, since the format is held in memory on the plugin broker side it means they are ignored and not expanded when we load the configuration.

An easy fix, would be to expand the yaml at load time instead of only sending the expanded data to the plugin so the broker has the information it needs within the format.

Example fix

func ParsePluginConfigFile(path string) ([]PluginConfig, error) {
	parsedConfigs := make([]PluginConfig, 0)
	yamlFile, err := os.Open(path)
	if err != nil {
		return nil, fmt.Errorf("while opening %s: %w", path, err)
	}
	dec := yaml.NewDecoder(yamlFile)
	dec.SetStrict(true)
	for {
		pc := PluginConfig{}
		err = dec.Decode(&pc)
		if err != nil {
			if errors.Is(err, io.EOF) {
				break
			}
			return nil, fmt.Errorf("while decoding %s got error %s", path, err)
		}
		// if the yaml document is empty, skip
		if reflect.DeepEqual(pc, PluginConfig{}) {
			continue
		}
		data, err := yaml.Marshal(pc)
		if err != nil {
			return nil, err
		}
		data = []byte(csstring.StrictExpand(string(data), os.LookupEnv))
		err = yaml.Unmarshal(data, &pc)
		if err != nil {
			return nil, fmt.Errorf("while unmarshalling %s got error %s", path, err)
		}
		parsedConfigs = append(parsedConfigs, pc)
	}
	return parsedConfigs, nil
}

and then removing the strict expand above since we already expanded above.

Security notice: there is a case where we could say "well this may impact security" but we already been expanding the information before it sent to the plugin anyways so this change will only allow the broker to know about the format expansion.

Edit: Add some testing I did with changes.

As you can see now the variable is modified within the format scope.

 ╭─loz repo: crowdsec/cmd/crowdsec-cli on  cscli_notifications_improvements [$!] via  v1.23.4 took 0s
 ╰─λ TEST=1234 ./cscli notifications inspect http_default
 -            Type:            http
 -            Name:    http_default
 -         Timeout:              5s
 -          Format: {{ range . -}}
{{ $alert := . -}}
{
  "extras": {
    "client::display": {
    "contentType": "text/markdown"
  }
},
"priority": 3,
"title": "test from crowdsec! using 1234",
"message": "message!"
}
{{ end -}}

 -       cert_path: /opt/domain.crt
 -         headers: map[   Content-Type:application/json]
 -        key_path: /opt/domain.u.key
 -       log_level:            info
 -          method:            POST
 - skip_tls_verification:            true
 -     unix_socket: /opt/test.s.sock
 -             url: https://_/message?token=AxHH7ZX1ttm-ZWc

 ╭─loz repo: crowdsec/cmd/crowdsec-cli on  cscli_notifications_improvements [$!] via  v1.23.4 took 0s
 ╰─λ ./cscli notifications inspect http_default
 -            Type:            http
 -            Name:    http_default
 -         Timeout:              5s
 -          Format: {{ range . -}}
{{ $alert := . -}}
{
  "extras": {
    "client::display": {
    "contentType": "text/markdown"
  }
},
"priority": 3,
"title": "test from crowdsec! using $TEST",
"message": "message!"
}
{{ end -}}

 -     unix_socket: /opt/test.s.sock
 -             url: https://_/message?token=AxHH7ZX1ttm-ZWc
 -       cert_path: /opt/domain.crt
 -         headers: map[   Content-Type:application/json]
 -        key_path: /opt/domain.u.key
 -       log_level:            info
 -          method:            POST
 - skip_tls_verification:            true

Important

A thing to note that cscli and crowdsec might have different environment variables as cscli runs under user scope so it may not have access to where ever you defined the environment variables to be. This is important thing to note as you may be surprised when inspect or test doesnt show the variable but crowdsec does parse it.

@LaurenceJJones
Copy link
Contributor

LaurenceJJones commented Dec 22, 2024

Okay following talking to the team it seems you can achieve this on the current crowdsec version i just had skill issues 😆

Add some more information env sprig command does work in the template, but again cscli notifications does not show this. So if you wish to use it for telegram for example you would have to set it like:

url: https://api.telegram.org/${TELEGRAM_BOT_KEY}/sendMessage
## URL only supports `${VAR}`
"chat_id": "{{ env "TELEGRAM_CHAT_ID" }}", 
## Format only supports `env` command call

Here is the config template:

type: http # Don't change
name: http_telegram # Must match the registered plugin in the profile

# One of "trace", "debug", "info", "warn", "error", "off"
log_level: info

# group_wait: 30s # Time to wait collecting alerts before relaying a message to this plugin, eg "30s"
# group_threshold:    # Amount of alerts that triggers a message before <group_wait> has expired, eg "10"
# max_retry:          # Number of attempts to relay messages to plugins in case of error
# timeout:            # Time to wait for response from the plugin before considering the attempt a failure, eg "10s"

#-------------------------
# plugin-specific options

# The following template receives a list of models.Alert objects
# The output goes in the http request body

# Replace XXXXXXXXX with your Telegram chat ID
format: |
  {
   "chat_id": "{{ env "TELEGRAM_CHAT_ID" }}", 
   "text": "
     {{range . -}}  
     {{$alert := . -}}  
     {{range .Decisions -}}
     {{.Value}} will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}}.
     {{end -}}
     {{end -}}
   ",
   "reply_markup": {
      "inline_keyboard": [
          {{ $arrLength := len . -}}
          {{ range $i, $value := . -}}
          {{ $V := $value.Source.Value -}}
          [
              {
                  "text": "See {{ $V }} on shodan.io",
                  "url": "https://www.shodan.io/host/{{ $V -}}"
              },
              {
                  "text": "See {{ $V }} on crowdsec.net",
                  "url": "https://app.crowdsec.net/cti/{{ $V -}}"
              }
          ]{{if lt $i ( sub $arrLength 1) }},{{end }}
      {{end -}}
      ]
  }

## Note I change to localhost change back to telegram
url: http://127.0.0.1:9090/${TELEGRAM_BOT_KEY}/sendMessage

method: POST
headers:
  Content-Type: "application/json"
$ sudo TELEGRAM_BOT_KEY=12345 TELEGRAM_CHAT_ID=56789 cscli notifications test http_telegram
Connection from 127.0.0.1:47338
POST /12345/sendMessage HTTP/1.1
Host: 127.0.0.1:9090
User-Agent: Go-http-client/1.1
Content-Length: 482
Content-Type: application/json
Accept-Encoding: gzip

{
 "chat_id": "56789",
 "text": "
   10.10.10.10 will get ban for next 4h for triggering test alert.
   ",
 "reply_markup": {
    "inline_keyboard": [
        [
            {
                "text": "See 10.10.10.10 on shodan.io",
                "url": "https://www.shodan.io/host/10.10.10.10"
            },
            {
                "text": "See 10.10.10.10 on crowdsec.net",
                "url": "https://app.crowdsec.net/cti/10.10.10.10"
            }
        ]
    ]
}

@instantdreams
Copy link

Thank you for the clarification! I adjusted my configuration:

$ docker exec crowdsec cscli notifications inspect http_telegram
 -            Type:            http
 -            Name:   http_telegram
 -         Timeout:              5s
 -          Format: {
 "chat_id": "{{ env "TELEGRAM_CHAT_ID" }}",
 "message_thread_id": {{ env "TELEGRAM_MESSAGE_THREAD_ID" }},
 "text": "
   {{range . -}}
   {{$alert := . -}}
   {{range .Decisions -}}
   {{.Value}} will get a {{.Type}} for the next {{.Duration}} for triggering {{.Scenario}}.
   {{end -}}
   {{end -}}
 ",
 "reply_markup": {
    "inline_keyboard": [
        {{ $arrLength := len . -}}
        {{ range $i, $value := . -}}
        {{ $V := $value.Source.Value -}}
        [
            {
                "text": "See {{ $V }} on shodan.io",
                "url": "https://www.shodan.io/host/{{ $V -}}"
            },
            {
                "text": "See {{ $V }} on crowdsec.net",
                "url": "https://app.crowdsec.net/cti/{{ $V -}}"
            }
        ]{{if lt $i ( sub $arrLength 1) }},{{end }}
    {{end -}}
    ]
}

 -       log_level:            warn
 -             url: https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage
 -          method:            POST
 -         headers: map[   Content-Type:application/json]

I tested the changes:

$ docker exec crowdsec cscli notifications test http_telegram
time="2024-12-22T16:02:51-07:00" level=debug msg="starting plugin" args="[/usr/local/lib/crowdsec/plugins/notification-http]" path=/usr/local/lib/crowdsec/plugins/notification-http
time="2024-12-22T16:02:51-07:00" level=debug msg="plugin started" path=/usr/local/lib/crowdsec/plugins/notification-http pid=582
time="2024-12-22T16:02:51-07:00" level=debug msg="waiting for RPC address" path=/usr/local/lib/crowdsec/plugins/notification-http
time="2024-12-22T16:02:51-07:00" level=debug msg="using plugin" version=1
time="2024-12-22T16:02:51-07:00" level=trace msg="waiting for stdio data"
level=info msg="registered plugin http_telegram"
level=info msg="pluginTomb dying"
level=info msg="killing all plugins"
time="2024-12-22T16:02:52-07:00" level=debug msg="received EOF, stopping recv loop" err="rpc error: code = Unavailable desc = error reading from server: EOF"
time="2024-12-22T16:02:52-07:00" level=info msg="plugin process exited" path=/usr/local/lib/crowdsec/plugins/notification-http pid=582
time="2024-12-22T16:02:52-07:00" level=debug msg="plugin exited"

Which worked:
image

Thank you for investigating this! All that remains is to update the documentation to identify the use of variables between the Go Templating and the notification template.

@LaurenceJJones
Copy link
Contributor

I have extended the environment part of the documentation about plugins.

If anyone has any comments to make on the clearness of what I wrote then please let me know on the PR itself linked above.

https://pr-702.d1to60jd2gb6y6.amplifyapp.com/docs/next/local_api/notification_plugins/intro#environment-variables

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement New feature or request needs/triage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants