Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_parseable: Plugin for sending logs to Parseable #9622

Open
wants to merge 79 commits into
base: master
Choose a base branch
from

Conversation

AdheipSingh
Copy link

@AdheipSingh AdheipSingh commented Nov 20, 2024

Send kubernetes logs to https://www.parseable.com/


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@edsiper
Copy link
Member

edsiper commented Nov 21, 2024

can you please provide the link to the API spec ?

@AdheipSingh
Copy link
Author

  • Here is the API doc for ingesting into parseable.
https://www.postman.com/parseable/parseable/request/au1fn5d/send-logs-to-a-log-stream-with-http-header
  • Here is the sample config i am using.
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 10MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
         Name parseable
         Match kube.*
         P_Server parseable.parseable.svc.cluster.local
         P_Port 80
         P_Username admin
         P_Password admin
         P_Stream $NAMESPACE 

Copy link
Contributor

@patrick-stephens patrick-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this build for all targets and do we need any extra dependencies? We probably need some unit tests at least too.

I think at the moment you've not actually enabled it to build, only for the container but then we don't test or exercise that in unit tests or builds.

CMakeLists.txt Outdated Show resolved Hide resolved
@@ -75,6 +75,7 @@ RUN cmake -DFLB_RELEASE=On \
-DFLB_IN_SYSTEMD=On \
-DFLB_OUT_KAFKA=On \
-DFLB_OUT_PGSQL=On \
-DFLB_OUT_PARSEABLE=On \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this what should be used in the cmake config? It's already defaulting to on

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feel free to suggest the best practice here. I tried to build without it but was not able to run the plugin.

/* Configuration properties map */
static struct flb_config_map config_map[] = {
{
FLB_CONFIG_MAP_STR, "P_Server", NULL,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say we should use the standard options we already have for http output. I'm guessing the P_ prefix is for parseable but as it's already in the config for that plugin it seems redundant so I'd suggest removing and aligning with the usual names

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would need P_Server as that is standard convention followed within parseable.

@AdheipSingh
Copy link
Author

Does this build for all targets and do we need any extra dependencies? We probably need some unit tests at least too.

I think at the moment you've not actually enabled it to build, only for the container but then we don't test or exercise that in unit tests or builds.

Please suggest some examples on unit tests and what can be done on building the ideal way. I'll implement this. ( i am not an expert in C codebase, so whatever is the best practice, ill will try to adhere to it ).

edsiper and others added 9 commits December 5, 2024 00:49
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Signed-off-by: AdheipSingh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-required ok-package-test Run PR packaging tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.