Skip to content

Commit

Permalink
Configure development environment with hot reloading
Browse files Browse the repository at this point in the history
- Add docs (CONTRIBUTING.md)
- Add command: 'make set-override'
- Add 'docker-compose.override.yml' to override sourced 'docker-compose.yml'
- Add 'watcher' command
- Change 'docker-entrypoint.sh' to:
  - (in Linux) Accept 'root' user, and swipe to the expected 'superset' user.
  - Serve the UI from 'webpack-dev-server' instead of 'flask'.

Signed-off-by: David Pordomingo <[email protected]>
  • Loading branch information
dpordomingo committed Jul 31, 2019
1 parent 345c08c commit 3110ec6
Show file tree
Hide file tree
Showing 6 changed files with 256 additions and 5 deletions.
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@

- New styling that follows the source{d} branding ([#139](https://github.com/src-d/sourced-ui/issues/139), [#142](https://github.com/src-d/sourced-ui/issues/142), [#204](https://github.com/src-d/sourced-ui/pull/204).)
- Improved method to export and import dashboards as JSON ([#165](https://github.com/src-d/sourced-ui/issues/165)).
- Added a development mode to run source{d} in a hot reloading way, so every change in `srcd` or `superset`
will trigger a refresh in the rowser [[see docs](./CONTRIBUTING.md#run-sourced-ce-for-development-with-hot-reloading)].


</details>

Expand Down
107 changes: 107 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Contribution Guidelines

As all source{d} projects, this project follows the
[source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md).


# Additional Contribution Guidelines

In addition to the [source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md),
this project follows the following guidelines.

**Content:**

- [Changelog](#changelog)
- [Run source{d} CE For Development With Hot Reloading](#run-source-d-ce-for-development-with-hot-reloading)


## Changelog

This project lists the important changes between releases in the
[`CHANGELOG.md`](CHANGELOG.md) file.

If you open a PR, you should also add a brief summary in the `CHANGELOG.md`
mentioning the new feature, change or bugfix that you proposed.


## Run source{d} CE For Development With Hot Reloading

Running **source{d} CE** in development mode will enable hot reloading at
http://127.0.0.1:8088, so every change you perform in `srcd` or `superset`
directories will trigger a refresh in your browser with the new code.

1. Install the latest version of [**source{d} CE**](https://github.com/src-d/sourced-ce/releases)
(see [installation guide](https://docs.sourced.tech/community-edition/quickstart/2-install-sourced)
if needed)

1. Run the watcher.

<details>
<summary><b>IMPORTANT note:</b> Requirements for the watcher</summary>

The watcher requires either [`inotify-tools`](https://github.com/rvoicilas/inotify-tools/wiki)
(for Linux), or [`fswatch`](https://github.com/emcrisostomo/fswatch)
(for Linux and MacOS)

- To install `inotify-tools`
- in Ubuntu you can run:
```shell
$ sudo apt-get install inotify-tools
```

- To install `fswatch`:
- in MacOS, you can [use brew to install `fswatch`](https://brewinstall.org/install-fswatch-on-mac-with-brew):
```shell
$ brew install fswatch
```
- in Ubuntu you can build and install [from `fswatch` sources](https://github.com/emcrisostomo/fswatch):
```shell
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install
$ sudo ldconfig
```
</details>

<details>
<summary><b>ALTERNATIVE:</b> If you can not use any watcher</summary>

- If you cannot use any of these watchers, you can just run this:

```shell
$ make set-override # to prepare the environment
$ make patch # needed EVERYTIME you change something in `srcd`
$ make clean # once you finish, to clean up everything.
```
</details>

To watch for changes in `srcd` directory, run the local watcher in one tab,
from your local `sourced-ui` root directory:

```shell
$ make dev-prepare
```

You can stop it pressing `Ctrl+C`. Once you do it, the patch will be
automatically cleaned.

1. Run `sourced` as usually (see [execution guide](https://docs.sourced.tech/community-edition/quickstart/3-init-sourced)
if needed):

```shell
# for repositories from GitHub organizations
$ sourced init orgs <orgs...> --token=<token>
# or, for repositories stored locally
$ sourced init local </path/to/repos>
```

The first time you launch it, it will take some time to build all the UI assets,
you can see the progress of the build from `sourced-ui` logs (see next step).

1. To see `sourced-ui` logs (with `npm` errors and such), run:

```shell
$ sourced logs -f sourced-ui
```
68 changes: 66 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ SUPERSET_DIR = superset
# Directory with custom code to copy into SUPERSET_DIR
PATCH_SOURCE_DIR = srcd

# Directory with template for docker-compose.override.yml
OVERRIDE_TEMPLATE_PATH := $(PATCH_SOURCE_DIR)/contrib/docker/docker-compose.override.yml
# Directory for the sourced global docker-compose.override.yml
OVERRIDE_OUTPUT_PATH := $(HOME)/.sourced/compose-files/__active__/docker-compose.override.yml
# Directory for the docker-compose.override.yml backup
OVERRIDE_BACKUP_PATH := $(OVERRIDE_OUTPUT_PATH).bak

# Name of the docker image to build
DOCKER_IMAGE_NAME ?= srcd/sourced-ui
# Docker registry where the docker image should be pushed to.
Expand All @@ -21,6 +28,29 @@ DOCKER_PASSWORD ?=
# will cause make docker-push to also push the latest tag.
DOCKER_PUSH_LATEST ?=

# Tools
STAT := stat -c


# Set the user as root inside of the container to have permissions to change
# internal `superset` UID and GUID, and make it match the host user.
# It will grant write access to the data from the volumes,
# inside of the container and on host file system (see #221)
LOGIN_USER := 0
ifeq ($(shell uname),Darwin)
STAT := stat -f
# In OSX there's no problem if the host user and the internal user are
# different when reading and accessing data trough docker volumes.
LOGIN_USER := superset
endif

SOURCED_UI_ABS_PATH := $(shell pwd)
LOCAL_USER := $(shell $(STAT) "%u" superset/superset)
export SOURCED_UI_ABS_PATH
export LOCAL_USER
export LOGIN_USER


# Build information
VERSION ?= latest
# Travis CI
Expand All @@ -33,9 +63,13 @@ IS_RELEASE := $(shell echo $(VERSION) | grep -q -E '^v[[:digit:]]+\.[[:digit:]]+

all: build

# Copy src-d files in the superset repository
# Clean, and copy src-d files in the superset repository
.PHONY: patch
patch: clean
patch: clean apply-patch

# Copy src-d files in the superset repository
.PHONY: apply-patch
apply-patch:
cp -r $(PATCH_SOURCE_DIR)/* $(SUPERSET_DIR)/

# Copy src-d files in the superset repository using symlinks. it's useful for development.
Expand All @@ -49,6 +83,30 @@ patch-dev: clean
done; \
ln -s "$(PWD)/$(PATCH_SOURCE_DIR)/superset/superset_config_dev.py" "$(SUPERSET_DIR)/superset_config.py"; \

# Start a watcher that will run 'make apply-patch' automatically when 'srcd' changes
# It will require either inotify or fswatch. More info in CONTRIBUTING.md
.PHONY: watch
watch:
@DIRECTORY_TO_OBSERVE=$(PATCH_SOURCE_DIR) bash watcher

# Writes the proper `docker-compose.override.yml` as the sourced global override file
.PHONY: set-override
set-override: $(OVERRIDE_BACKUP_PATH)
@awk '{ system("echo \""$$0"\"") }' $(OVERRIDE_TEMPLATE_PATH) > $(OVERRIDE_OUTPUT_PATH)

# Creates a backup of the sourced global override file if it exists
$(OVERRIDE_BACKUP_PATH):
@mkdir -p ~/.sourced/compose-files/__active__
@if [ -f "$(OVERRIDE_OUTPUT_PATH)" ]; then \
cp $(OVERRIDE_OUTPUT_PATH) $(OVERRIDE_BACKUP_PATH); \
else \
echo "\033[33mno docker-compose.override.yml to backup\033[0m"; \
fi;

# Prepares the development enviroment with hot reloading
.PHONY: dev-prepare
dev-prepare: set-override watch

# Create docker image
.PHONY: patch
build: patch
Expand Down Expand Up @@ -92,6 +150,12 @@ docker-push-latest-release:
clean:
rm -f "$(SUPERSET_DIR)/superset_config.py"
git clean -fd $(SUPERSET_DIR)
rm -f $(OVERRIDE_OUTPUT_PATH)
@if [ -f "$(OVERRIDE_BACKUP_PATH)" ]; then \
mv $(OVERRIDE_BACKUP_PATH) $(OVERRIDE_OUTPUT_PATH); \
else \
echo "\033[33mno docker-compose.override.yml.bak to restore\033[0m"; \
fi;

# Add superset upstream remote if doesn't exists
.PHONY: remote-add
Expand Down
13 changes: 13 additions & 0 deletions srcd/contrib/docker/docker-compose.override.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
version: '3.2'
services:
sourced-ui:
user: ${LOGIN_USER:-superset}:${LOGIN_USER:-superset}
volumes:
- ${SOURCED_UI_ABS_PATH}/superset/superset:/home/superset/superset
- ${SOURCED_UI_ABS_PATH}/superset/dashboards:/home/superset/dashboards
- ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/superset_config.py:/home/superset/superset/superset_config.py
- ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/bootstrap.py:/home/superset/bootstrap.py
- ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/docker-entrypoint.sh:/entrypoint.sh
environment:
SUPERSET_ENV: development
LOCAL_USER: ${LOCAL_USER:-}
22 changes: 19 additions & 3 deletions superset/contrib/docker/docker-entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,19 @@
#
set -ex

if [ "`whoami`" = "root" ] && [ -n "$LOCAL_USER" ]; then

# Change the UID and GUID of 'superset' user (matching host user).
# It will grant write access to the data from the volumes,
# inside of the container and on host file system (see #221)
find . -group superset -exec chgrp $LOCAL_USER '{}' \;
groupmod -g $LOCAL_USER superset
usermod -u $LOCAL_USER superset

su -c "/entrypoint.sh" superset
exit $?
fi

if [ "$SUPERSET_NO_DB_INIT" != "true" ]; then
python bootstrap.py
fi
Expand All @@ -25,10 +38,13 @@ if [ "$#" -ne 0 ]; then
exec "$@"
elif [ "$SUPERSET_ENV" = "development" ]; then
celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair &
# needed by superset runserver
# In development mode, the UI will be served by `webpack-dev-server` instead of by `Flask`
# `webpack-dev-server` will serve the UI from the port 8088, and it will proxy
# non-asset requests to `Flask`, wich is listening at the port 8081
# Doing so, updates to asset sources will be reflected in-browser without a refresh.
(cd superset/assets/ && npm ci)
(cd superset/assets/ && npm run dev) &
FLASK_ENV=development FLASK_APP=superset:app flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0
(cd superset/assets/ && npm run dev-server -- --host=0.0.0.0 --port=8088 --supersetPort=8081) &
FLASK_ENV=development FLASK_APP=superset:app flask run -p 8081 --with-threads --reload --debugger --host=0.0.0.0
elif [ "$SUPERSET_ENV" = "production" ]; then
celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair &
gunicorn --bind 0.0.0.0:8088 \
Expand Down
48 changes: 48 additions & 0 deletions watcher
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/bin/bash

trap ctrl_c SIGINT

function ctrl_c {
make --no-print-directory clean && \
echo -e "\033[1;92mPatch cleaned\033[0m" || \
echo -e "\033[1;33mCould not clean patch\033[0m"
exit $?
}

function watch_inotify {
inotifywait --recursive \
--event modify,move,create,delete \
$DIRECTORY_TO_OBSERVE
}

function watch_fswatch {
fswatch --recursive --one-event \
--event Created --event Updated --event Removed \
${DIRECTORY_TO_OBSERVE}
}


if command -v inotifywait > /dev/null 2>&1; then
whichWatcher=inotify
function watcher { watch_inotify; }
elif command -v fswatch > /dev/null 2>&1; then
whichWatcher=fswatch
function watcher { watch_fswatch; }
else
echo -e "\033[1;31m[ERROR] No watcher available\033[0m"
echo "Neither 'inotify' nor 'fswatch' are available."
echo "You can follow 'CONTRIBUTING.md' guidelines to install one of them,"
echo ""
echo -e "\033[1;33m[alternative]\033[0m follow these steps:"
echo "1. run 'make set-override' first,"
echo "2. then 'make patch' every time you change a file in 'srcd' dir"
echo "3. and 'make clean' once you finish developing"
echo ""
exit 1
fi

make --no-print-directory apply-patch
echo -e "\033[1;92mWatching for changes in 'srcd'; using '${whichWatcher}' ...\033[0m"
while watcher; do
make --no-print-directory apply-patch
done

0 comments on commit 3110ec6

Please sign in to comment.