feat(conf): support Nginx fine-grained debug levels

[hishamhm]

Summary

Expose seven additional Nginx log levels that are not normally exposed by OpenResty, which allows for finer-grained debugging with less noise in the debug-level logs: debug_core, debug_alloc, debug_mutex, debug_event, debug_http, debug_mail, debug_stream.

By setting a quieter value such as debug_mail, Kong Gateway debug logs are usable again, as the Kong (and WasmX) logs are still shown, but the other Nginx debug logs (memory allocations, etc.) are not shown.

Checklist

• The Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - updated docs are auto-generated from kong.conf.default

Support for dynamically setting these additional levels depends on this companion PR for lua-kong-nginx-module: Kong/lua-kong-nginx-module#87

[hishamhm]

If Kong/lua-kong-nginx-module#87 gets merged and the lua-kong-nginx-module is bumped, then the second commit from this branch (e8defcb) can be removed, and extended debug modes will work both statically and dynamically.

As this PR stands right now, these extra debug_* log levels now work only statically (e.g. export KONG_LOG_LEVEL=debug_mail, etc.); I've been using this extensively in the past weeks to get readable debug logs when working on WasmX running in Kong Gateway.

[bungle]

@hishamhm, freaky, what is DEBUG_MAIL is that something that is reserved when using nginx just with mail module, aka mail { } that we don't have?

[bungle]

Kong Gateway debug logs are usable again

@hishamhm I suggest you add:

build --//:debug=false

in HOME/.bazelrc (I think we should by default build without debug, it is so rarely usable - both on CI and local)

But in case you want that still, then perhaps the new debug levels make sense?

[bungle]

It is also strange that DEBUG_MAIL is quieter as it is 512 level, while normal DEBUG is 8 and ERR is 4. Are these also used as some kind of bit mask (I guess so)?

[hishamhm]

@hishamhm, freaky, what is DEBUG_MAIL is that something that is reserved when using nginx just with mail module, aka mail { } that we don't have?

@bungle Yes, so it never shows up and it quiets down the other C-level nginx debug logs, leaving only the debug logs we care about (OpenResty+Kong and WasmX).

build --//:debug=false

I'll try that out, thanks!! @locao, was that the local trick you were referring to?

(I think we should by default build without debug, it is so rarely usable - both on CI and local)
But in case you want that still, then perhaps the new debug levels make sense?

Kong packages/containers are already built like that right? So if I have a way to do that locally (and I can never see those gazillion malloc messages again :) ), then that bazel setting would address my issue (and yes, it should probably be the default, as the common case for Kong debugging is not debugging Nginx itself).

(True story: I forgot a Kong instance running idle on my local machine after a day of debugging and woke up the next day to find everything failing on my machine due to zero disk space... twice!)

[bungle]

Kong packages/containers are already built like that right?

Kong distributions (those that we ship) are build without debug aka with --//:debug=false, AFAIK (need a confirmation). But for some reason this debug thing is enabled on local builds and I think CI. There is a slack thread somewhere with arguments on it. The noise level has gone down a bit with --//:debug=false.

[locao]

@locao, was that the local trick you were referring to?

Yes, it will silence the C-level logs.

(After two months I am sure you already tested it, but maybe someone else can benefit from it 😁)

[hishamhm]

@locao Indeed, thanks! Given those tips serve as a usable alternative, I think we can retire this PR!

[thibaultcha]

But in case you want that still, then perhaps the new debug levels make sense?

Hmm too bad, this is precisely why those debug levels are painful to us. When working on an Nginx module embedded in the Gateway (be it kong-lua-module or ngx-wasm-module), we still want our module's debug logs be enabled, while still silencing the other Nginx debug logs that Hisham mentioned.