fail connecting apm-server to kibana

[zeihanaulia]

Ask a question...

I want to add apm-server to this stack, so I try to add a new service

  apm-server:
    image: docker.elastic.co/apm/apm-server:${ELK_VERSION}
    cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
    cap_drop: ["ALL"]
    ports:
      - 8200:8200
    command: >
       apm-server -e
         -E apm-server.rum.enabled=true
         -E setup.kibana.host=https://kibana:5601
         -E setup.template.settings.index.number_of_replicas=0
         -E apm-server.kibana.enabled=true
         -E apm-server.kibana.host=https://kibana:5601
         -E output.elasticsearch.hosts=["elasticsearch:9200"]
    depends_on:
      - kibana
    healthcheck:
      interval: 10s
      retries: 12
      test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/

all services running and apm-server is correctly setup.

the problem is there's no data in APM services or traces, this is my code

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"

	"github.com/gorilla/mux"
	"go.elastic.co/apm/module/apmgorilla"
	"go.elastic.co/apm/v2"
)

func helloHandler(w http.ResponseWriter, req *http.Request) {
	ctx := req.Context()
	span, ctx := apm.StartSpan(ctx, "helloHandler", "custom")
	defer span.End()

	fmt.Fprint(w, greet(ctx, mux.Vars(req)["name"]))
}

func greet(ctx context.Context, name string) string {
	span, _ := apm.StartSpan(ctx, "greet", "custom")
	defer span.End()

	return fmt.Sprintf("Hello, %s! from span handler\n", name)
}

func main() {
	r := mux.NewRouter()
	r.Use(apmgorilla.Middleware())
	r.HandleFunc("/hello/{name}", helloHandler)
	log.Fatal(http.ListenAndServe(":8000", r))
}

application apm debug

this logs error from apm-server

apm-server_1     | {"log.level":"error","@timestamp":"2022-07-19T21:23:08.098Z","log.logger":"kibana","log.origin":{"file.name":"kibana/connecting_client.go","file.line":79},"message":"failed to obtain connection to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana:5601/api/status fails: fail to execute the HTTP GET request: Get \"https://kibana:5601/api/status\": x509: certificate signed by unknown authority. Response: ","service.name":"apm-server","ecs.version":"1.6.0"}

apm-server_1     | {"log.level":"error","@timestamp":"2022-07-19T21:23:37.614Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: EOF","service.name":"apm-server","ecs.version":"1.6.0"}

so what I'm missing with this config, there's anyone who successfully adds an APM server?

[mostafahussein]

Usually this happens when you need to communicate with kibana using its CA certificate.
you can try the recent release of elastdocker which comes with apm-server.

and from the application side (your go application) you need to pass the CA certificate.

let me know if you ran into any issue while trying the recent version.

thank you