Feature/tilt | Dapr

README.md

@@ -51,12 +51,6 @@ Daher haben wir eine Roadmap entwickelt, was wir wann erreichen möchten: [Roadm
 
 ## Entwicklung & Werkzeuge
 
-[MeisterTask](https://www.meistertask.com/app/project/p7zXoSfq/demokratie)
-
-[Analysetool Dateistruktur](https://veniversum.me/git-visualizer/?owner=demokratie-live&repo=demokratie-app)
-
-[Analysetool Commits](http://ghv.artzub.com/#repo=demokratie-app&climit=10000&user=demokratie-live)
-
 [Malwerkzeug zur Planung](https://www.draw.io/)
 
 ## Open Source
@@ -149,11 +143,35 @@ docker buildx build \
 ```
 
 ## Contributing
+# Running the Dev Environment
+
+First clone this repository
+
+## Tilt (Kuberneetes)
+
+### Prerequirements
+
+- local kubernetes cluster
+  - [Docker Desktop](https://www.docker.com/products/docker-desktop/) - _the easiest way_
+  - [colima](https://github.com/abiosoft/colima) - _for Apple Silicon (M1)_
+  - or your favourite
+- [local docker registry](https://docs.docker.com/registry/deploying/#native-basic-auth)
+- [tilt.dev](https://docs.tilt.dev/install.html)
+
+### Start
+
+`tilt up`
+
+### Clean Up
+
+`tilt down`
+
+# Contributing
 
 Anyone and everyone is welcome to [contribute](https://github.com/demokratie-live/democracy-development/blob/master/CONTRIBUTE.md). Start by checking out the list of
 [open issues](https://github.com/demokratie-live/democracy-development/issues).
 
-## License
+# License
 
 Copyright © 2017-present DEMOCRACY Deutschland e.V.. This source code is licensed under the Apache 2.0 license found in the
 [LICENSE](https://github.com/demokratie-live/democracy-development/blob/master/LICENSE) file.

Tiltfile

@@ -0,0 +1,91 @@
+include('./infra/tilt/monitoring/Tiltfile')
+include('./infra/tilt/dapr/Tiltfile')
+
+k8s_yaml('./infra/kustomize/overlays/local/namespace.yaml')
+k8s_yaml(kustomize('./infra/kustomize/overlays/local'))
+
+k8s_resource(workload='democracy-app-depl', port_forwards='3000:3000', labels=["democracy"], resource_deps=['democracy-api-depl'])
+k8s_resource(
+    workload='democracy-api-depl',
+    port_forwards='3001:3000',
+    labels=["democracy"],
+    resource_deps=['bundestagio-depl']
+)
+
+
+k8s_resource(
+    workload='bundestagio-depl',
+    port_forwards='3101:3100',
+    labels=["bundestag"],
+    resource_deps=[
+        'dapr-dashboard',
+        'dapr-operator',
+        'dapr-sentry',
+        'dapr-sidecar-injector',
+        'dapr-placement-server',
+        'democracy-mongo-depl'
+    ])
+k8s_resource(workload='bundestagio-admin-depl', labels=["bundestag"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='bundestagio-dip-depl', port_forwards='3102:3101', labels=["bundestag"], resource_deps=['bundestagio-depl'])
+
+k8s_resource(workload='qr-code-handler-depl', labels=["services"], resource_deps=['bundestagio-depl'])
+
+k8s_resource(workload='cleanup-push-queue', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+k8s_resource(workload='democracy-sync-deputy-profiles-cronjob', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+k8s_resource(workload='democracy-sync-named-polls-cronjob', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+k8s_resource(workload='democracy-sync-procedures-cronjob', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+k8s_resource(workload='import-conference-week-details-cronjob', labels=["cronjobs"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='import-deputy-profiles-cronjob', labels=["cronjobs"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='import-deputy-profiles-period-18-cronjob', labels=["cronjobs"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='import-deputy-profiles-period-19-cronjob', labels=["cronjobs"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='import-named-poll-deputies-cronjob', labels=["cronjobs"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='import-named-polls-cronjob', labels=["cronjobs"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='import-plenary-minutes-cronjob', labels=["cronjobs"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='import-procedures-cronjob', labels=["cronjobs"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='index-sync-bundestagio-cronjob', labels=["cronjobs"], resource_deps=['bundestagio-depl'])
+k8s_resource(workload='index-sync-democracy-cronjob', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+k8s_resource(workload='push-send-queued-cronjob', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+k8s_resource(workload='queue-pushs-conference-week-cronjob', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+k8s_resource(workload='queue-pushs-vote-conference-week-cronjob', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+k8s_resource(workload='queue-pushs-vote-top-100-cronjob', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+k8s_resource(workload='shedule-bio-resync-cronjob', labels=["cronjobs"], resource_deps=['democracy-api-depl'])
+
+k8s_resource(workload='democracy-mongo-depl', labels=["third-paty"])
+k8s_resource(workload='gorush', labels=["third-paty"])
+k8s_resource(workload='nats-depl', labels=["third-paty"])
+k8s_resource(workload='redis', labels=["third-paty"])
+
+
+### democracy-api ##########################################################################################
+# docker_build(
+#     'democracy/democracy-server',
+#     context='./democracy/api',
+#     dockerfile='./democracy/api/Dockerfile',
+#     only=['.'],
+#     target='build_stage',
+#     entrypoint='yarn dev',
+#     live_update=[
+#         sync('./democracy/api/src/', '/app/src/'),
+#         run(
+#             'yarn install',
+#             trigger=['./yarn.lock']
+#         )
+#     ]
+# )
+
+### import-procedures-cronjob ##################################################
+docker_build(
+    'democracy/crawler',
+    context='./services/cron-jobs/crawler',
+    dockerfile='./services/cron-jobs/crawler/Dockerfile',
+    only=['.'],
+    target='build_stage',
+    entrypoint='yarn dev',
+    live_update=[
+        sync('./services/cron-jobs/crawler/src/', '/app/src/'),
+        run(
+            'yarn install',
+            trigger=['./yarn.lock']
+        )
+    ]
+)

infra/kustomize/overlays/local/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: democracy

infra/tilt/dapr/Tiltfile

@@ -0,0 +1,42 @@
+
+### DAPR #######################################################################
+load('ext://helm_remote', 'helm_remote')
+
+helm_remote('dapr',
+            repo_url='https://dapr.github.io/helm-charts/',
+            namespace='dapr-system',
+            create_namespace=True,
+            version='1.7.0',
+            values="./values.yaml"
+)
+
+k8s_resource(workload='dapr-dashboard', port_forwards='3300:8080', labels=["dapr"])
+k8s_resource(workload='dapr-operator', labels=["dapr"])
+k8s_resource(workload='dapr-sentry', labels=["dapr"])
+k8s_resource(workload='dapr-sidecar-injector', labels=["dapr"])
+k8s_resource(workload='dapr-placement-server', labels=["dapr"])
+
+k8s_yaml('./dapr-config.yaml', allow_duplicates=True)
+# k8s_yaml('./local-secret-store.yaml', allow_duplicates=True)
+
+### Zipkin  #######################################################################
+
+# k8s_yaml(helm('./zipkin-helm',
+#      namespace='dapr-system',
+# ))
+# # k8s_yaml('./dapr-tracing-zipkin-config.yaml')
+
+# k8s_resource(workload='chart-zipkin-ui', port_forwards='9411', labels=["dapr"], resource_deps=['chart-zipkin-cassandra'])
+# k8s_resource(workload='chart-zipkin-dependencies-gen', labels=["dapr"], resource_deps=['chart-zipkin-cassandra'])
+# k8s_resource(workload='chart-zipkin-cassandra', labels=["dapr"])
+# k8s_resource(workload='chart-zipkin-collector', labels=["dapr"], resource_deps=['chart-zipkin-cassandra'])
+
+### HASHICORP VAULT  #######################################################################
+# helm_remote('vault',
+#             repo_url='https://helm.releases.hashicorp.com',
+#             namespace='hashicorp-vault',
+#             create_namespace=True
+# )
+# k8s_resource(workload='vault-agent-injector', labels=["vault"])
+# k8s_resource(workload='vault', port_forwards='3301:8200', labels=["vault"])
+

infra/tilt/dapr/dapr-config.yaml

@@ -0,0 +1,12 @@
+apiVersion: dapr.io/v1alpha1
+kind: Configuration
+metadata:
+  name: daprsystem
+  namespace: dapr-system
+spec:
+  # tracing:
+  #   samplingRate: '1'
+  #   zipkin:
+  #     endpointAddress: 'http://zipkin:9411/api/v2/spans'
+  mtls:
+    enabled: false

infra/tilt/dapr/dapr-local-storage-binding.yaml

@@ -0,0 +1,11 @@
+apiVersion: dapr.io/v1alpha1
+kind: Component
+metadata:
+  name: local-storage
+  namespace: dapr-system
+spec:
+  type: bindings.localstorage
+  version: v1
+  metadata:
+    - name: rootPath
+      value: /tmp/dapr

infra/tilt/dapr/dapr-test-secrets.json

@@ -0,0 +1,3 @@
+{
+  "password": "geheim"
+}

infra/tilt/dapr/dapr-tracing-zipkin-config.yaml

@@ -0,0 +1,10 @@
+apiVersion: dapr.io/v1alpha1
+kind: Configuration
+metadata:
+  name: tracing
+  namespace: dapr-system
+spec:
+  tracing:
+    samplingRate: '1'
+    zipkin:
+      endpointAddress: 'http://zipkin:9411/api/v2/spans'

infra/tilt/dapr/local-secret-store.yaml

@@ -0,0 +1,11 @@
+apiVersion: dapr.io/v1alpha1
+kind: Component
+metadata:
+  name: localsecretstore
+  namespace: default
+spec:
+  type: secretstores.local.file
+  version: v1
+  metadata:
+    - name: secretsFile
+      value: /Users/manuelruck/Work/democracy/repos/democracy-development/infra/tilt/dapr/dapr-test-secrets.json

infra/tilt/dapr/values.yaml

@@ -0,0 +1,20 @@
+dapr_dashboard:
+  runAsNonRoot: false
+  logLevel: DEBUG
+dapr_placement:
+  runAsNonRoot: false
+  logLevel: DEBUG
+dapr_operator:
+  runAsNonRoot: false
+  logLevel: DEBUG
+dapr_sentry:
+  runAsNonRoot: false
+  logLevel: DEBUG
+dapr_sidecar_injector:
+  hostNetwork: true
+  runAsNonRoot: false
+  logLevel: DEBUG
+global:
+  logAsJson: true
+  mtls:
+    enabled: false

infra/tilt/dapr/zipkin-helm/.helmignore

@@ -0,0 +1,26 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+example/
+docs/
+LICENSE
+README.md
+scripts/

infra/tilt/dapr/zipkin-helm/Chart.yaml

@@ -0,0 +1,5 @@
+apiVersion: v1
+description: Zipkin distributed tracing
+name: zipkin-helm
+version: 0.6.2
+icon: https://avatars1.githubusercontent.com/u/11860887

infra/tilt/dapr/zipkin-helm/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Financial Times
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

infra/tilt/dapr/zipkin-helm/README.md

@@ -0,0 +1,71 @@
+# zipkin-helm
+A helm chart for zipkin
+
+## usage
+
+### Adding the repo 
+
+`helm repo add zipkin-helm https://financial-times.github.io/zipkin-helm/docs`
+
+### Installing
+
+`helm install -f my-cassandra-config.yaml https://financial-times.github.io/zipkin-helm/docs/zipkin-helm-0.1.1.tgz`
+
+## values
+
+Example
+
+```
+cassandra:
+  username: zipkinuser
+  password: my-super-secret-password
+  contactPoints: cassandra-1.me.com,cassandra-2.me.com
+
+ingress:
+  host: zipkin.example.com
+
+configmap:
+  localdc:
+    name: cassandra # use this notation if you want to reuse values from existing configmaps, this takes precedence over the .cassandra field
+    key: local.dc
+```
+
+## Releasing
+
+### Package the new version
+
+1. Change the version in the `Chart.yaml` file to the new version
+2. `helm package -d docs/ .`
+3. `helm repo index docs --url https://financial-times.github.io/zipkin-helm/docs/`
+4. Commit the changes
+5. Make a new release on github
+6. `git commit -m 'rebuild pages' --allow-empty && git push`
+
+## Gotchas
+
+#### collectors and ui readiness probe doesn't pass
+
+##### Problem
+
+My collector and ui pods are never coming into service as the readiness probes are not passing.
+
+e.g.
+```
+foiled-labradoodle-zipkin-cassandra-0                           1/1       Running                      0          8m
+foiled-labradoodle-zipkin-collector-6dff48b6df-crl4z            0/1       Running                      0          3m
+foiled-labradoodle-zipkin-collector-6dff48b6df-m4v8t            0/1       Running                      0          3m
+foiled-labradoodle-zipkin-collector-6dff48b6df-rrwdb            0/1       Running                      0          3m
+foiled-labradoodle-zipkin-ui-7bdf9c6c96-jh7vq                   0/1       Running                      0          3m
+```
+
+##### Explanation
+
+This can happen for some reason when cassandra comes up before any of the collectors/ui pods.
+
+##### Solution
+Try restarting the pods:
+
+```
+kubectl delete po -l app=zipkin-collector
+kubectl delete po -l app=zipkin-ui
+```