AWS authentication broken in fluent-bit 1.8.8

[ts3ng]

We have a configuration to send logging events to splunk that works up till fluent-bit 1.8.7. In 1.8.8 we are seeing failures using s3 output.

We used the following configurations to log to s3:

    [OUTPUT]
      # https://docs.fluentbit.io/manual/pipeline/outputs/s3
      Name s3
      Match_Regex ^(s3|s3splunk)$
      tls.Verify off 
      bucket ${LOGGING_S3_BUCKET}
      region ${LOGGING_S3_REGION}
      # File size of local file before upload to s3
      total_file_size ${LOGGING_S3_FILESIZE} 
      # Format string for keys in S3
      s3_key_format ${LOGGING_S3_KEY_FORMAT}
      # A series of characters which will be used to split the tag into 'parts' for use with the s3_key_format option
      s3_key_format_tag_delimiters ${LOGGING_S3_TAG_DELIMITERS}
      compression gzip
      use_put_object On

working in fluent-bit 1.8.7

Fluent Bit v1.8.7
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/12/02 00:37:05] [ info] [engine] started (pid=1)
[2021/12/02 00:37:05] [ info] [storage] version=1.1.1, initializing...
[2021/12/02 00:37:05] [ info] [storage] in-memory
[2021/12/02 00:37:05] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/12/02 00:37:05] [ info] [cmetrics] version=0.2.1
[2021/12/02 00:37:05] [ info] [fstore] created root path /tmp/fluent-bit/s3/tseng-s3-caas-test
[2021/12/02 00:37:05] [ info] [output:s3:s3.0] Using upload size 1000000 bytes
[2021/12/02 00:37:06] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/12/02 00:37:06] [ info] [sp] stream processor started
[2021/12/02 00:37:06] [ info] [input:tail:tail.0] inotify_fs_add(): inode=34408676 watch_fd=1 name=/logging-volume/bfbf9bb6e6df640aecddc3044de2022adb72cf25d2760f0ff9a8da16abaa9e12.log
[2021/12/02 00:47:11] [ info] [output:s3:s3.0] Successfully uploaded object /fluent-bit-logs/S3TEST/s3splunk/2021/12/02/00/37/10-objectospYeOi6
[2021/12/02 00:47:11] [ info] [output:s3:s3.0] Successfully uploaded object /fluent-bit-logs/S3TEST/s3/2021/12/02/00/37/10-object3xWxwmE7

seeing failures in fluent-bit 1.8.8

Fluent Bit v1.8.8
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/12/02 00:25:07] [ info] [engine] started (pid=1)
[2021/12/02 00:25:07] [ info] [storage] version=1.1.4, initializing...
[2021/12/02 00:25:07] [ info] [storage] in-memory
[2021/12/02 00:25:07] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/12/02 00:25:07] [ info] [cmetrics] version=0.2.2
[2021/12/02 00:25:07] [ info] [fstore] created root path /tmp/fluent-bit/s3/tseng-s3-caas-test
[2021/12/02 00:25:07] [ info] [output:s3:s3.0] Using upload size 1000000 bytes
[2021/12/02 00:25:07] [ warn] [imds] unable to evaluate IMDS version
[2021/12/02 00:25:07] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/12/02 00:25:07] [ info] [sp] stream processor started
[2021/12/02 00:25:07] [ info] [input:tail:tail.0] inotify_fs_add(): inode=34408405 watch_fd=1 name=/logging-volume/b99c378493c71020f0fa8eb19bee297098744454470acf5dce8b885f6ad1aa84.log
[2021/12/02 00:35:13] [error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
[2021/12/02 00:35:13] [error] [aws_credentials] Failed to retrieve credentials for AWS Profile default
[2021/12/02 00:35:13] [ warn] [imds] unable to evaluate IMDS version
[2021/12/02 00:35:13] [ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The current co-routine will retry.
[2021/12/02 00:35:13] [error] [signv4] Provider returned no credentials, service=s3
[2021/12/02 00:35:13] [error] [aws_client] could not sign request
[2021/12/02 00:35:13] [error] [output:s3:s3.0] PutObject request failed

In our kubernetes environment we use https://github.com/jtblin/kube2iam for our pods to inherit roles and allow access to s3 buckets without having to pass the role_arn. Seems like the 1.8.8 AWS lib changes this has broken the authentication functionality.

[ts3ng]

Tried using

[FILTER]
  Name aws
  Match *
  imds_version v1
  az false
  ec2_instance_id false

still getting the same errors:

Fluent Bit v1.8.8
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/12/02 20:45:42] [ info] [engine] started (pid=1)
[2021/12/02 20:45:42] [ info] [storage] version=1.1.4, initializing...
[2021/12/02 20:45:42] [ info] [storage] in-memory
[2021/12/02 20:45:42] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/12/02 20:45:42] [ info] [cmetrics] version=0.2.2
[2021/12/02 20:45:42] [ info] [fstore] created root path /tmp/fluent-bit/s3/tseng-s3-caas-test
[2021/12/02 20:45:42] [ info] [output:s3:s3.0] Using upload size 1000000 bytes
[2021/12/02 20:45:42] [ warn] [imds] unable to evaluate IMDS version
[2021/12/02 20:45:42] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/12/02 20:45:42] [ info] [sp] stream processor started
[2021/12/02 20:45:42] [ info] [input:tail:tail.0] inotify_fs_add(): inode=43262641 watch_fd=1 name=/logging-volume/151caa8cdb6389712b7bce1edf4562282c5d83cd59adc0ad67683a91347ae768.log
[2021/12/02 20:52:08] [error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
[2021/12/02 20:52:08] [error] [aws_credentials] Failed to retrieve credentials for AWS Profile default
[2021/12/02 20:52:08] [ warn] [imds] unable to evaluate IMDS version

[matthewfala]

Hi @ts3ng
Yes, unfortunately a patch adding IMDSv2 support in 1.8.8 does not handle fallbacks to IMDSv1 properly.

This issue is resolved in 1.8.9.
Please see aws-for-fluent-bit issue:
aws/aws-for-fluent-bit#259

If you would like to stick with 1.8.8, you may need to set hop count limit to 2 on you instance as described in the issue.

[agup006]

As this is solved in 1.8.9 I'm going to close this issue