Skip to content

Commit

Permalink
DDEV Elastic Enterprise Search implementation
Browse files Browse the repository at this point in the history
  • Loading branch information
Marco Hermo committed Feb 4, 2024
1 parent ee20454 commit ad1b3b3
Show file tree
Hide file tree
Showing 7 changed files with 367 additions and 161 deletions.
152 changes: 92 additions & 60 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,93 +1,125 @@
[![tests](https://github.com/ddev/ddev-addon-template/actions/workflows/tests.yml/badge.svg)](https://github.com/ddev/ddev-addon-template/actions/workflows/tests.yml) ![project is maintained](https://img.shields.io/maintenance/yes/2024.svg)
[![tests](https://github.com/ddev/ddev-enterprise-search/actions/workflows/tests.yml/badge.svg)](https://github.com/ddev/ddev-enterprise-search/actions/workflows/tests.yml) ![project is maintained](https://img.shields.io/maintenance/yes/2024.svg)

# ddev-addon-template <!-- omit in toc -->
# ddev-enterprise-search <!-- omit in toc -->

* [What is ddev-addon-template?](#what-is-ddev-addon-template)
* [Components of the repository](#components-of-the-repository)
* [Getting started](#getting-started)
* [How to debug in Github Actions](#how-to-debug-tests-github-actions)
- [Introduction](#introduction)
- [Getting started](#getting-started)
- [How to debug in Github Actions](#how-to-debug-tests-github-actions)

## What is ddev-addon-template?
## Introduction

This repository is a template for providing [DDEV](https://ddev.readthedocs.io) add-ons and services.
ddev-enterprise-search is the un-official implementation of Elastic Enterprise Search service for DDEV based on their Docker guide\*.

In DDEV addons can be installed from the command line using the `ddev get` command, for example, `ddev get ddev/ddev-redis` or `ddev get ddev/ddev-solr`.
Enterprise Search is an additional Elastic service that adds APIs and UIs to those already provided by Elasticsearch and Kibana.

This repository is a quick way to get started. You can create a new repo from this one by clicking the template button in the top right corner of the page.
Currently sitting at version 8.12.0, part of the implementation as a service for DDEV includes Elastic Search and Kibana containers.
This means that to use this service, existing Elastic and Kibana servies needs to be uninstalled in your project.

![template button](images/template-button.png)
From your DDEV project, install this by running `ddev get ssmarco/ddev-enterprise-search` then `ddev restart`.
This will take up to 30 minutes or so due when downloading the required docker containers (Elastic Search, Kibana and Enterprise Search).

## Components of the repository

* The fundamental contents of the add-on service or other component. For example, in this template there is a [docker-compose.addon-template.yaml](docker-compose.addon-template.yaml) file.
* An [install.yaml](install.yaml) file that describes how to install the service or other component.
* A test suite in [test.bats](tests/test.bats) that makes sure the service continues to work as expected.
* [Github actions setup](.github/workflows/tests.yml) so that the tests run automatically when you push to the repository.
- [Reference](https://www.elastic.co/guide/en/enterprise-search/current/start.html)
- [Docker guide\*](https://www.elastic.co/guide/en/enterprise-search/current/docker.html)

## Getting started

1. Choose a good descriptive name for your add-on. It should probably start with "ddev-" and include the basic service or functionality. If it's particular to a specific CMS, perhaps `ddev-<CMS>-servicename`.
2. Create the new template repository by using the template button.
3. Globally replace "addon-template" with the name of your add-on.
4. Add the files that need to be added to a DDEV project to the repository. For example, you might replace `docker-compose.addon-template.yaml` with the `docker-compose.*.yaml` for your recipe.
5. Update the `install.yaml` to give the necessary instructions for installing the add-on:
1. In the DDEV project directory launch the command:

```
ddev get ssmarco/ddev-enterprise-search
```

2. Restart the DDEV instance:

```
ddev restart
```

3. Get the URL of the Kibana dashboard eg (https://your-project-name.ddev.site:5602):

```
ddev describe
```

4. Login with the username, `elastic` and password, `elastic`

## Configuring your framework

### Silverstripe

* The fundamental line is the `project_files` directive, a list of files to be copied from this repo into the project `.ddev` directory.
* You can optionally add files to the `global_files` directive as well, which will cause files to be placed in the global `.ddev` directory, `~/.ddev`.
* Finally, `pre_install_commands` and `post_install_commands` are supported. These can use the host-side environment variables documented [in DDEV docs](https://ddev.readthedocs.io/en/latest/users/extend/custom-commands/#environment-variables-provided).
1. Update your project's `.env` file. The API keys are found in the Enterprise Search section of Kibana dashboard.

6. Update `tests/test.bats` to provide a reasonable test for your repository. Tests are triggered either by manually executing `bats ./tests/test.bats`, automatically on every push to the repository, or periodically each night. Please make sure to attend to test failures when they happen. Others will be depending on you. Bats is a simple testing framework that just uses Bash. To run a Bats test locally, you have to [install bats-core](https://bats-core.readthedocs.io/en/stable/installation.html) first. Then you download your add-on, and finally run `bats ./tests/test.bats` within the root of the uncompressed directory. To learn more about Bats see the [documentation](https://bats-core.readthedocs.io/en/stable/).
7. When everything is working, including the tests, you can push the repository to GitHub.
8. Create a [release](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository) on GitHub.
9. Test manually with `ddev get <owner/repo>`.
10. You can test PRs with `ddev get https://github.com/<user>/<repo>/tarball/<branch>`
11. Update the `README.md` to describe the add-on, how to use it, and how to contribute. If there are any manual actions that have to be taken, please explain them. If it requires special configuration of the using project, please explain how to do those. Examples in [ddev/ddev-solr](https://github.com/ddev/ddev-solr), [ddev/ddev-memcached](https://github.com/ddev/ddev-memcached), and (advanced) [ddev-platformsh](https://github.com/ddev/ddev-platformsh).
12. Update the `README.md` header in Title Case format, for example, use `# DDEV Redis`, not `# ddev-redis`.
13. Add a good short description to your repo, and add the [topic](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/classifying-your-repository-with-topics) "ddev-get". It will immediately be added to the list provided by `ddev get --list --all`.
14. When it has matured you will hopefully want to have it become an "official" maintained add-on. Open an issue in the [DDEV queue](https://github.com/ddev/ddev/issues) for that.
```
ENTERPRISE_SEARCH_ENGINE_PREFIX="my-index"
ENTERPRISE_SEARCH_API_KEY="private-xxxxxxxxxxxx-change-this"
ENTERPRISE_SEARCH_API_SEARCH_KEY="search-xxxxxxxxxxxx-change-this"
ENTERPRISE_SEARCH_ENDPOINT="http://enterprisesearch:3002"
```

Add-ons were covered in [DDEV Add-ons: Creating, maintaining, testing](https://www.dropbox.com/scl/fi/bnvlv7zswxwm8ix1s5u4t/2023-11-07_DDEV_Add-ons.mp4?rlkey=5cma8s11pscxq0skawsoqrscp&dl=0) (part of the [DDEV Contributor Live Training](https://ddev.com/blog/contributor-training)).
2. The Enterprise Search endpoint is `http://enterprisesearch:3002`

Note that more advanced techniques are discussed in [DDEV docs](https://ddev.readthedocs.io/en/latest/users/extend/additional-services/#additional-service-configurations-and-add-ons-for-ddev).
## Troubleshooting

## How to debug tests (Github Actions)
1. Make sure all required containers are downloaded:

```
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.12.0
docker pull docker.elastic.co/kibana/kibana:8.12.0
docker pull docker.elastic.co/enterprise-search/enterprise-search:8.12.0
```

1. You need an SSH-key registered with GitHub. You either pick the key you have already used with `github.com` or you create a dedicated new one with `ssh-keygen -t ed25519 -a 64 -f tmate_ed25519 -C "$(date +'%d-%m-%Y')"` and add it at `https://github.com/settings/keys`.
2. Remove container volumes to restart from scratch.

2. Add the following snippet to `~/.ssh/config`:
List all existing volumes from your system:

```
Host *.tmate.io
User git
AddKeysToAgent yes
UseKeychain yes
PreferredAuthentications publickey
IdentitiesOnly yes
IdentityFile ~/.ssh/tmate_ed25519
docker volume ls
```
3. Go to `https://github.com/<user>/<repo>/actions/workflows/tests.yml`.

4. Click the `Run workflow` button and you will have the option to select the branch to run the workflow from and activate `tmate` by checking the `Debug with tmate` checkbox for this run.
This will show example output below:

![tmate](images/gh-tmate.jpg)
```
DRIVER VOLUME NAME
local ddev-your-project-name_elastic-certs
local ddev-your-project-name_elastic-data
local ddev-your-project-name_elastic-kibana
local ddev-your-project-name_enterprise-data
```

5. After the `workflow_dispatch` event was triggered, click the `All workflows` link in the sidebar and then click the `tests` action in progress workflow.
Delete the volumes by running:

7. Pick one of the jobs in progress in the sidebar.
```
docker volume rm ddev-your-project-name_elastic-certs \
ddev-your-project-name_elastic-data \
ddev-your-project-name_elastic-kibana \
ddev-your-project-name_enterprise-data
```

3. Restart by `ddev restart`

4. Check the status of the project by `ddev status`

5. Check the logs:

```
ddev logs -s elastic-config
ddev logs -s elasticsearch
ddev logs -s kibana
ddev logs -s enterprisesearch
```

8. Wait until the current task list reaches the `tmate debugging session` section and the output shows something like:
6. Check job health:

```
106 SSH: ssh [email protected]
107 or: ssh -i <path-to-private-SSH-key> [email protected]
108 SSH: ssh [email protected]
109 or: ssh -i <path-to-private-SSH-key> [email protected]
docker inspect --format "{{json .State.Health }}" ddev-your-project-name-enterprisesearch | jq
docker inspect --format "{{json .State.Health }}" ddev-your-project-name-kibana | jq
docker inspect --format "{{json .State.Health }}" ddev-your-project-name-elasticsearch | jq
```

9. Copy and execute the first option `ssh [email protected]` in the terminal and continue by pressing either <kbd>q</kbd> or <kbd>Ctrl</kbd> + <kbd>c</kbd>.
## Contribute

10. Start the Bats test with `bats ./tests/test.bats`.
- Anyone is welcome to submit a PR to this repo. See README.md at https://github.com/ddev/ddev-addon-template, the parent of this repo.

For a more detailed documentation about `tmate` see [Debug your GitHub Actions by using tmate](https://mxschmitt.github.io/action-tmate/).
## Maintainer

**Contributed and maintained by [@CONTRIBUTOR](https://github.com/CONTRIBUTOR)**
- Contributed and maintained by [Marco Hermo](https://github.com/ssmarco).
16 changes: 0 additions & 16 deletions docker-compose.addon-template.yaml
View file

This file was deleted.

180 changes: 180 additions & 0 deletions docker-compose.enterprise-search.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
#ddev-generated

services:
# mount certificate folder to website
web:
volumes:
- elastic-certs:/usr/share/elasticsearch/config/certs

elastic-config:
container_name: ddev-${DDEV_SITENAME}-elastic-config
hostname: ${DDEV_SITENAME}-elastic-config
image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
volumes:
- ".:/mnt/ddev_config"
- elastic-certs:/usr/share/elasticsearch/config/certs
environment:
- ELASTIC_PASSWORD=elastic
- KIBANA_PASSWORD=elastic
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: $DDEV_APPROOT
user: "0"
restart: "no"
command: >
bash -c '
if [ ! -f /usr/share/elasticsearch/config/certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f /usr/share/elasticsearch/config/certs/certs.zip ]; then
echo "Creating certs";
echo -ne \
"instances:\n"\
" - name: elasticsearch\n"\
" dns:\n"\
" - elasticsearch\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 755 \{\} \;;
find . -type f -exec chmod 644 \{\} \;;
echo "Listing certificates"
ls -la config/certs/ca/ca.crt
ls -la config/certs/elasticsearch/elasticsearch.crt
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt -u elastic:elastic https://elasticsearch:9200; do sleep 5 && date; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:elastic -H "Content-Type: application/json" https://elasticsearch:9200/_security/user/kibana_system/_password -d "{\"password\":\"elastic\"}" | grep -q "^{}"; do sleep 5 && date ; done;
echo "Testing kibana_system user";
until curl -s --cacert config/certs/ca/ca.crt -u kibana_system:elastic https://elasticsearch:9200; do sleep 5 && date; done;
echo "Exit in 5 minutes"
for i in {1..5}; do date && sleep 60; done
echo "All done!";
'
elasticsearch:
depends_on:
- elastic-config
# Name of container using standard ddev convention
container_name: ddev-${DDEV_SITENAME}-elasticsearch
hostname: ${DDEV_SITENAME}-elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
volumes:
- ".:/mnt/ddev_config"
- ./enterprise-search/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- elastic-certs:/usr/share/elasticsearch/config/certs
- elastic-data:/usr/share/elasticsearch/data
# These labels ensure this service is discoverable by ddev.
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: $DDEV_APPROOT
environment:
- node.name=elasticsearch
- cluster.name=elastic-cluster
- cluster.initial_master_nodes=elasticsearch
- ELASTIC_PASSWORD=elastic
restart: "no"
mem_limit: 1073741824
ulimits:
memlock:
soft: -1
hard: -1

kibana:
depends_on:
- elasticsearch
- elastic-config
# Name of container using standard ddev convention
container_name: ddev-${DDEV_SITENAME}-kibana
hostname: ${DDEV_SITENAME}-kibana
image: docker.elastic.co/kibana/kibana:8.12.0
volumes:
- ".:/mnt/ddev_config"
- ./enterprise-search/kibana.yml:/usr/share/kibana/config/kibana.yml
- elastic-certs:/usr/share/kibana/config/certs
- elastic-kibana:/usr/share/kibana/data
# These labels ensure this service is discoverable by ddev.
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: $DDEV_APPROOT
expose:
- "5601"
environment:
- VIRTUAL_HOST=$DDEV_HOSTNAME
- SERVERNAME=kibana
- KIBANA_PASSWORD=elastic
- HTTP_EXPOSE=5601:5601
- HTTPS_EXPOSE=5602:5601
mem_limit: 1073741824
healthcheck:
test:
[
"CMD-SHELL",
"curl -s -I http://kibana:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120

enterprisesearch:
depends_on:
- kibana
- elasticsearch
container_name: ddev-${DDEV_SITENAME}-enterprisesearch
hostname: ${DDEV_SITENAME}-enterprisesearch
image: docker.elastic.co/enterprise-search/enterprise-search:8.12.0
volumes:
- ".:/mnt/ddev_config"
- ./enterprise-search/config.yml:/usr/share/enterprise-search/config/enterprise-search.yml
- elastic-certs:/usr/share/enterprise-search/config/certs
- enterprise-data:/usr/share/enterprise-search/config
# These labels ensure this service is discoverable by ddev.
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: $DDEV_APPROOT
environment:
- VIRTUAL_HOST=$DDEV_HOSTNAME
- SERVERNAME=enterprisesearch
- ELASTIC_PASSWORD=elastic
- KIBANA_PASSWORD=elastic
- secret_management.encryption_keys=[thequickbrownfoxjumpsoverthelazydog1cozylummoxgivessmartsquidwho]
- allow_es_settings_modification=true
- elasticsearch.host=https://elasticsearch:9200
- elasticsearch.username=elastic
- elasticsearch.password=elastic
- elasticsearch.ssl.enabled=true
- elasticsearch.ssl.certificate_authority=/usr/share/enterprise-search/config/certs/ca/ca.crt
- kibana.external_url=http://kibana:5601
mem_limit: 1073741824
healthcheck:
test:
[
"CMD-SHELL",
"curl -s -I http://enterprisesearch:3002 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120

volumes:
elastic-certs:
elastic-data:
elastic-kibana:
enterprise-data:
Loading

0 comments on commit ad1b3b3

Please sign in to comment.