What you will learn
- How to build a customize fluentd container with the dynatrace plugin
- How to deploy fluentd in a kubernetes cluster using Configmap
- How to ingest metrics using the dynatrace output plugin
- How to chain fluentbit and fluentd
This repository showcase the usage of the fluend by using GKE with :
- the HipsterShop
- Prometheus
- Istio
- fluentd
- Dynatrace
The following tools need to be install on your machine :
- jq
- kubectl
- git
- gcloud ( if you are using GKE)
- Helm
If you don't have any dynatrace tenant , then let's start a trial on Dynatrace
PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}
ZONE=us-central1-b
gcloud container clusters create onlineboutique \
--project=${PROJECT_ID} --zone=${ZONE} \
--machine-type=e2-standard-2 --num-nodes=4
git clone https://github.com/isItObservable/Episode-10---FluentD-tutorial.git
cd Episode-10---FluentD-tutorial
- Create the various namespaces For the hipsterShop :
kubectl create namespace hipster-shop
kubectl -n hipster-shop create rolebinding default-view --clusterrole=view --serviceaccount=hipster-shop:default
- Download Istioctl
curl -L https://istio.io/downloadIstio | sh -
This command download the latest version of istio ( in our case iostio 1.10.2) compatible with our operating system. 2. Add istioctl to you PATH
cd istio-1.10.3
this directory contains samples with addons . We will refer to it later.
export PATH=$PWD/bin:$PATH
To enable Istio , you need to install istio with the following settings
istioctl install --set profile=demo -y
Then we want to instruct istio to automatically inject the envoy Proxy to all the pods of our Hipster-shop application so we will label the namesapce : hipster-shop
kubectl label namespace hipster-shop istio-injection=enabled
cd hipstershop
./setup.sh
kubectl edit svc istio-ingressgateway -n istio-system
Add the following ports :
- name: web
nodePort: 31770
port: 8080
protocol: TCP
targetPort: 8182
kubectl apply -f istio/hipstershop_gateway.yaml
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack
kubectl edit svc istio-ingressgateway -n istio-system
Add the following ports :
- name: grafana
nodePort: 31775
port: 8888
protocol: TCP
targetPort: 8888
kubectl edit svc istio-ingressgateway -n istio-system
Add the following ports :
- name: prometheus
nodePort: 31776
port: 9090
protocol: TCP
targetPort: 9090
Deploy the gateway and Virtual Services :
kubectl apply -f istio/Prometheus_Grafana_gateway.yaml
In order to deliver our tutorial we need a fluentd version having the following plugins preinstalled :
- the input plugin : forward ( to connect later on fluentbit with fluentd)
- the output plugin dynatrace
To combine both plugins we are going to build the new image based from fluentd-kubernetes-daemonset:v1.14.1-debian-forward-1.0
To build the image you will need to require to install docker on your laptop : docker desktop
cd /fluentd
docker build . -t fluentd-dyantrace:0.1
The dockerfile only add the installation of the the library with this commad :
RUN gem install fluent-plugin-dynatrace -v 0.1.5
RUN gem install fluent-plugin-kubernetes_metadata_filter -v 2.7.2
RUN gem install fluent-plugin-multi-format-parser
RUN gem install fluent-plugin-concat
In our tutorial i already have build the docker image and pushed it on docker hub.
We will use the following image : hrexed/fluentd-dyantrace:0.2
THe log ingest api of dynatrace is reachable only from the Active Gate. To deploy the active Gate, it would be required to generate a Paas Token: In dynatrace click :
- Settings
- Integration
- click on the button Generate
- Give a name and copy the value of the Paas Token
Follow the instruction described in dynatrace's documentation Make sure that the scope log ingest is enabled.
kubectl get namespace kube-system -o jsonpath='{.metadata.uid}'
- Create a service account and cluster role for accessing the Kubernetes API.
kubectl apply -f fluentd/service_account.yaml
Create a secret holding the environment URL and login credentials for this registry, making sure to replace.
export ENVIRONMENT_URL=<with your environment URL (without 'http'). Example: environment.live.dynatrace.com>
export CLUSTERID=<YOUR CLUSTER ID>
export PAAS_TOKEN=<YOUR PAAS TOKEN>
export API_TOKEN=<YOUR API TOKEN>
export ENVIRONMENT_ID=<YOUR environementid in your environment url>
kubectl create secret docker-registry tenant-docker-registry --docker-server=${ENVIRONMENT_URL} --docker-username=${ENVIRONMENT_ID} --docker-password=${PAAS_TOKEN} -n dynatrace
kubectl create secret docker-registry tenant-docker-registry --docker-server=${ENVIRONMENT_URL} --docker-username=${ENVIRONMENT_ID} --docker-password=${PAAS_TOKEN} -n dynatrace
kubectl create secret generic tokens --from-literal="log-ingest=${API_TOKEN}" -n dynatrace
Update the file named fluentd/fluentd-manifest.yaml and activegate.yaml, by running the following command :
sed -i "s,ENVIRONMENT_ID_TO_REPLACE,$ENVIRONMENT_ID," fluentd/fluentd-manifest.yaml
sed -i "s,CLUSTER_ID_TO_REPLACE,$CLUSTERID," fluentd/fluentd-manifest.yaml
sed -i "s,ENVIRONMENT_URL_TO_REPLACE,$ENVIRONMENT_URL," fluentd/activegate.yaml
kubectl apply -f fluentd/activegate.yaml
kubectl apply -f fluentd/fluentd-manifest.yaml
To get native Kubernetes metrics, you need to connect the Kubernetes API to Dynatrace.
Get the Kubernetes API URL.
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
Get the bearer token from the dynatrace-monitoring service account.
kubectl get secret $(kubectl get sa dynatrace-monitoring -o jsonpath='{.secrets[0].name}' -n dynatrace) -o jsonpath='{.data.token}' -n dynatrace | base64 --decode
In the Dynatrace menu, go to Settings > Cloud and virtualization > Kubernetes, and select Connect new cluster. Provide a Name, Kubernetes API URL, and the Bearer token for the Kubernetes cluster. Note: For Rancher distributions, you need the bearer token that was created in Rancher web UI, as described in Special instructions for Rancher distributions above. Once you connect your Kubernetes clusters to Dynatrace, you can get native Kubernetes metrics, like request limits, and differences in pods requested vs. running pods.
The current deployment of fluentd is collecting the logs from the kubernetes cluster using the input plugin tail :
<source>
@id in_tail_container_logs
@type tail
tag raw.kubernetes.*
path /var/log/containers/*.log
pos_file /var/log/fluentd.pos
read_from_head true
<parse>
@type multi_format
<pattern>
format json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</pattern>
<pattern>
format regexp
time_format %Y-%m-%dT%H:%M:%S.%N%:z
expression /^(?<time>.+)\b(?<stream>stdout|stderr)\b(?<log>.*)$/
</pattern>
</parse>
</source>
Let's have a look a the log ingested in Dynatrace. Open Dynatrace and click Logs on the left menu .
kubectl create namespace logging
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml
kubectl delete -f fluentd/fluentd-manifest.yaml
now let's use the fluentd deployment using the input forward plugin But we need to update all the information to connect to dynatrace. let's update the deployment :
sed -i "s,ENVIRONMENT_ID_TO_REPLACE,$ENVIRONMENT_ID," fluentbit/fluentd-manifest_with_fluentbit.yaml
sed -i "s,CLUSTER_ID_TO_REPLACE,$CLUSTERID," fluentbit/fluentd-manifest_with_fluentbit.yaml
Now we can deploy the new fluend log stream pipeline
kubectl apply -f fluentbit/fluentd-manifest_with_fluentbit.yaml
kubectl apply -f fluentbit/fluentbit_deployment.yaml