OTEL Dependency Panic #871

karmops · 2024-04-10T22:08:38Z

Migrating from 2.5.0 to 2.6.1

the issue also happens on: 2.6.0

targeting ARM machines

just ran the same deployment I had, updating the OTEL config to:

    "telemetry/opentelemetry": {
      "metric_reporting_period": 0,
      "exporters": {
        "prometheus": [
          {
            "port": 9091,
            "name": "krakend"
          }
        ]
      }
    }

Stack Trace

Parsing configuration file: /etc/krakend/krakend.json
2024/04/10 22:09:17 KRAKEND INFO: Starting KrakenD v2.6.1
2024/04/10 22:09:17 KRAKEND INFO: Working directory is /etc/krakend
2024/04/10 22:09:17 KRAKEND DEBUG: [SERVICE: Gin] Debug enabled
2024/04/10 22:09:17 KRAKEND DEBUG: [SERVICE: Gin][HTTPsecure] Successfuly loaded module
2024/04/10 22:09:17 KRAKEND DEBUG: [SERVICE: Gin][Botdetector] The bot detector has been registered successfully
2024/04/10 22:09:17 KRAKEND INFO: Starting the KrakenD instance

... for brevity, it was removed

2024/04/10 22:02:08 http2: panic serving 10.1.21.208:36858: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 131196 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x4027f6a510})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x400012a980}, 0x4008a63500)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x0?}, {0x3989540, 0x400012a980}, 0x4008a63500)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x4027f6a270?}, 0x402ab7e000?, {0x4000768c30?}}, {0x3989540, 0x400012a980}, 0x4008a63500)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x4b4?, 0x31590f5?, 0x16?, 0x4027f65eb0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c
2024/04/10 22:02:14 http2: panic serving 10.1.21.208:46688: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 139410 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x4027f6ad20})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x400012a9a0}, 0x4008a63e00)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x402a6c4d38?}, {0x3989540, 0x400012a9a0}, 0x4008a63e00)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x402adda420?}, 0x402266ce00?, {0x4000768c30?}}, {0x3989540, 0x400012a9a0}, 0x4008a63e00)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x0?, 0x11dbdf4?, 0x4000ef0380?, 0x0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c
2024/04/10 22:02:14 http2: panic serving 10.1.21.208:46690: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 139398 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x40288f9fb0})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x4000010910}, 0x400095fd00)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x402a6c1d38?}, {0x3989540, 0x4000010910}, 0x400095fd00)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x4027f6a600?}, 0x402266d180?, {0x4000768c30?}}, {0x3989540, 0x4000010910}, 0x400095fd00)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x402ab75560?, 0x402a6c1f88?, 0x4000648360?, 0x402a6c1f98?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c
2024/04/10 22:02:24 http2: panic serving 10.1.21.208:48160: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 129153 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x402addb2c0})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x400a1a8430}, 0x402addc800)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x40288fd1e0?}, {0x3989540, 0x400a1a8430}, 0x402addc800)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x402addab70?}, 0x4021a61500?, {0x4000768c30?}}, {0x3989540, 0x400a1a8430}, 0x402addc800)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x0?, 0x11dbdf4?, 0x40288f9d10?, 0x0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c
2024/04/10 22:02:24 http2: panic serving 10.1.21.208:48186: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 139402 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x402addb350})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x4000010920}, 0x402ae30200)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x402a6c1d38?}, {0x3989540, 0x4000010920}, 0x402ae30200)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x4027f6adb0?}, 0x402266d500?, {0x4000768c30?}}, {0x3989540, 0x4000010920}, 0x402ae30200)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x0?, 0x11dbdf4?, 0x402ae32000?, 0x0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c

karmops · 2024-04-10T22:20:28Z

now I am blind =D I can't see any metrics because I had to disable it

karmops · 2024-04-10T22:52:49Z

sounds like some package is using some different package version which it doesn't implement Hijacker interface

taik0 · 2024-04-11T12:17:49Z

Hi @karmops
Opentelemetry didn't exist in v2.5, only 2.6.
That configuration works fine for me, can you share instructions to generate that panic? I've tried on amd64 and arm64 hosts.

karmops · 2024-04-11T17:32:50Z

I didn't do anything too special. I just increased the version of my Krakend from 2.5.0 to 2.6.1 and replaced the settings as suggested in the documentation... nothing more.

my krakend-settings

{
  "$schema": "https://www.krakend.io/schema/v2.6/krakend.json",
  "version": 3,
  "name": "KrakenD Open Source API Gateway",
  "port": "{{ env "KRAKEND_PORT" }}",
  "host": ["{{ env "HOST" }}"],
  "debug_endpoint": true,
  "timeout": "6000ms",
  "tls": {
    "public_key": "/etc/krakend/ssl/certs/tls.crt",
    "private_key": "/etc/krakend/ssl/certs/tls.key",
    "min_version": "TLS12"
  },
  "endpoints": [
    {{ template "some-template.tmpl" }}
  ],
  "extra_config": {
    "router": {
      "return_error_msg": true
    },
    "security/cors": {
      "allow_origins": ["*"],
      "allow_methods": ["GET", "POST", "PATCH", "DELETE"],
      "allow_headers": [
        "Origin",
        "Authorization",
        "Content-Type",
        "Content-Length",
        "User-Agent",
        "X-Forwarded-For",
        "Accept-Language",
        "trackingId"
      ],
      "expose_headers": ["Content-Length", "Content-Type"],
      "allow_credentials": true,
      "debug": false,
      "max_age": "12h"
    },
    "security/bot-detector": {
      "empty_user_agent_is_bot": true,
      "cache_size": 1000
    },
    "security/http": {
      "frame_deny": true,
      "custom_frame_options_value": "DENY",
      "referrer_policy": "same-origin",
      "content_type_nosniff": true,
      "browser_xss_filter": true,
      "content_security_policy": "default-src 'self';",
      "is_development": false
    },
    "telemetry/opentelemetry": {
      "metric_reporting_period": 0,
      "exporters": {
        "prometheus": [
          {
            "port": 9091,
            "name": "krakend"
          }
        ]
      }
    }
  }
}

karmops · 2024-04-11T17:35:53Z

2024/04/10 22:02:08 http2: panic serving 10.1.21.208:36858: interface conversion: *http.http2responseWriter is not http.Hijacker: missing method Hijack
goroutine 131196 [running]:
net/http.(*http2serverConn).runHandler.func1()
/usr/local/go/src/net/http/h2_bundle.go:6104 +0x138
panic({0x2d73560, 0x4027f6a510})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/krakend/krakend-otel/http/server.newTrackingResponseWriter(...)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/response_writer.go:90
github.com/krakend/krakend-otel/http/server.(*trackingHandler).ServeHTTP(0x4000ee03c0, {0x3989540, 0x400012a980}, 0x4008a63500)
/go/pkg/mod/github.com/krakend/krakend-otel@v0.2.0/http/server/server.go:37 +0x418
net/http.serverHandler.ServeHTTP({0x0?}, {0x3989540, 0x400012a980}, 0x4008a63500)
/usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x398b520?, 0x4027f6a270?}, 0x402ab7e000?, {0x4000768c30?}}, {0x3989540, 0x400012a980}, 0x4008a63500)
/usr/local/go/src/net/http/server.go:3545 +0x1d0
net/http.(*http2serverConn).runHandler(0x4b4?, 0x31590f5?, 0x16?, 0x4027f65eb0?)
/usr/local/go/src/net/http/h2_bundle.go:6111 +0xc0
created by net/http.(*http2serverConn).scheduleHandler
/usr/local/go/src/net/http/h2_bundle.go:6046 +0x21c

but this sounds like some library version in some component should upgrade to a more recent version of golang.

alombarte · 2024-04-16T13:23:49Z

Fix: krakend/krakend-otel#14

alombarte · 2024-04-17T12:03:43Z

Hi @karmops,
This should be fixed now, please use v2.6.2

Thank you for reporting it

karmops · 2024-04-18T21:01:26Z

gonna try and let you know

karmops · 2024-04-19T20:06:25Z

@alombarte Thank you very much, it is fixed. no panic anymore and krakend is generating the open telemetry metrics.

alombarte added bug roadmap labels Apr 16, 2024

alombarte closed this as completed Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OTEL Dependency Panic #871

OTEL Dependency Panic #871

karmops commented Apr 10, 2024 •

edited

Loading

karmops commented Apr 10, 2024 •

edited

Loading

karmops commented Apr 10, 2024

taik0 commented Apr 11, 2024

karmops commented Apr 11, 2024 •

edited

Loading

karmops commented Apr 11, 2024 •

edited

Loading

alombarte commented Apr 16, 2024

alombarte commented Apr 17, 2024

karmops commented Apr 18, 2024

karmops commented Apr 19, 2024 •

edited

Loading

OTEL Dependency Panic #871

OTEL Dependency Panic #871

Comments

karmops commented Apr 10, 2024 • edited Loading

karmops commented Apr 10, 2024 • edited Loading

karmops commented Apr 10, 2024

taik0 commented Apr 11, 2024

karmops commented Apr 11, 2024 • edited Loading

karmops commented Apr 11, 2024 • edited Loading

alombarte commented Apr 16, 2024

alombarte commented Apr 17, 2024

karmops commented Apr 18, 2024

karmops commented Apr 19, 2024 • edited Loading

karmops commented Apr 10, 2024 •

edited

Loading

karmops commented Apr 10, 2024 •

edited

Loading

karmops commented Apr 11, 2024 •

edited

Loading

karmops commented Apr 11, 2024 •

edited

Loading

karmops commented Apr 19, 2024 •

edited

Loading