Cloudflare HTTP Event Log Source Schema is incorrect for BotTags

[deeso]

There is a bug in the the cloudflare http event schema. The schema defines the cloudflare.http_event.bot.tag as a string here, but the actual value is an array of strings, see:

matano/data/managed/log_sources/cloudflare/tables/http_request.yml

Line 60 in b9975f5

- name: tag

When the VRL parses the log, the result is either null or an array of strings at this location:

matano/data/managed/log_sources/cloudflare/tables/http_request.yml

Line 457 in b9975f5

.cloudflare.http_request.bot.tag = del(.json.BotTags)

This causes any JSON log line containing a BotTags array to fail and be sidelined by the transform. The error creates the following error message in the CloudWatch logs for the Data Transformer lambda:

ERROR transformer: Line error: Line err: SchemaMismatchError, msg: Failed to resolve schema for due to schema mismatch for table cloudflare_http_request. (log source: tablename)

To fix this issue, this block snippet needs to be converted from:

         - name: bot
            type:
              type: struct
              fields:
              - name: score
                type:
                  type: struct
                  fields:
                  - name: src
                    type: string
                  - name: value
                    type: long
              - name: tag
                type: string

To:

         - name: bot
            type:
              type: struct
              fields:
              - name: score
                type:
                  type: struct
                  fields:
                  - name: src
                    type: string
                  - name: value
                    type: long
              - name: tag
                 type: list
                    element: string

[Samrose-Ahmed]

That looks correct, happy to accept a PR.

If you wish to continue using the existing table, you will have to manually drop or rename the column from your table via Spark or API since it's a breaking schema change (if you're testing you can just recreate it).