Audit logs flooding the cloudwatch loggroup even if there is no activity

[srinivassambari1]

We have deployed vault docker container in a docker-compose along with aws-for-fluent-bit image where we would like to store vault audit logs in a cloudwatch log group.

we have enabled vault audit logs for a filepath and fluentbit tails this file and send logs to cloudwatch loggroup.

Initially after we enable audit logs, we only see the logs where there is an activity in vault like a user generating credentials for a db or updating any configuration or login and logout.

After few hours we observed that vault logs were streamed to cloudwatch even if there is no activity, streaming is continuous like a flood with a "display_name": "token-terraform".
we don't have any token issued with token-terraform or any user with token-terraform.

with this issue we were unable to search the real audit events logs.

Here is some info about images

[ec2-user@ip-xx-xx-xx-xx hashicorp]$ docker ps
CONTAINER ID   IMAGE                              COMMAND                  CREATED      STATUS        PORTS                                     NAMES
28e8b6eefcd6   vault:latest                       "docker-entrypoint.s…"   3 days ago   Up 26 hours   0.0.0.0:443->8200/tcp, :::443->8200/tcp   hashicorp-vault-1
e4f1acec507f   amazon/aws-for-fluent-bit:latest   "/bin/sh -c /entrypo…"   3 days ago   Up 26 hours   2020/tcp                                  hashicorp-fluent-bit-1
[ec2-user@ip-xx-xx-xx-xx hashicorp]$ docker images
REPOSITORY                  TAG       IMAGE ID       CREATED       SIZE
vault                       latest    7193130a202b   12 days ago   221MB
amazon/aws-for-fluent-bit   latest    d862dfcfa5b7   13 days ago   339MB
[ec2-user@ip-xx-xx-xx-xx  hashicorp]$ 


cat docker-compose.yml 
version: '3'
services:
    vault:
      image: vault:latest
      restart: always
      ports:
        - 443:8200
      volumes:
        - ./vault/config:/vault/config
        - ./vault/policies:/vault/policies
        - ./vault/data:/vault/data
        - ./vault/logs:/vault/logs
        - ./vault/file:/vault/file
      environment:
        - VAULT_ADDR=https://vault.example.com
        - VAULT_API_ADDR=https://vault.example.com
        - VAULT_ADDRESS=https://vault.example.com
      cap_add:
        - IPC_LOCK
      command: vault server -config=/vault/config/vault.json
    fluent-bit:
      image: amazon/aws-for-fluent-bit:latest
      restart: always
      volumes:
        - ./vault/logs:/var/log/vault/logs
        - ./fluent-bit/etc:/fluent-bit/etc/

Vault config:

cat vault.json 
{
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_cert_file": "/vault/data/certs/server.crt",
      "tls_key_file":  "/vault/data/certs/server.key"
    }
  },
  "backend": {
    "file": {
      "path": "/vault/file"
    }
  },
  "default_lease_ttl": "168h",
  "max_lease_ttl": "0h",
  "api_addr": "https://vault.example.com:8200",
  "ui": "true"
}




Fluent-bit config: 

cat fluent-bit.conf 
[SERVICE]
    flush                     5
    log_Level                 info

[INPUT]
    Name              tail
    Tag               audit
    Path              /var/log/vault/logs/audit.log
    Mem_Buf_Limit     8MB
    Refresh_Interval  30
    Parser            json

[OUTPUT]
    Name cloudwatch
    Match audit
    region eu-central-1
    log_group_name /aws/some_log_group_name/var/log
    log_stream_prefix vault_
    auto_create_group true
    log_retention_days 365
    log_key log