From 3110ec6d6e2fc4cca33ab280020c8a6128632f60 Mon Sep 17 00:00:00 2001 From: David Pordomingo Date: Wed, 17 Jul 2019 12:17:09 +0200 Subject: [PATCH 1/4] Configure development environment with hot reloading - Add docs (CONTRIBUTING.md) - Add command: 'make set-override' - Add 'docker-compose.override.yml' to override sourced 'docker-compose.yml' - Add 'watcher' command - Change 'docker-entrypoint.sh' to: - (in Linux) Accept 'root' user, and swipe to the expected 'superset' user. - Serve the UI from 'webpack-dev-server' instead of 'flask'. Signed-off-by: David Pordomingo --- CHANGELOG.md | 3 + CONTRIBUTING.md | 107 ++++++++++++++++++ Makefile | 68 ++++++++++- .../docker/docker-compose.override.yml | 13 +++ superset/contrib/docker/docker-entrypoint.sh | 22 +++- watcher | 48 ++++++++ 6 files changed, 256 insertions(+), 5 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 srcd/contrib/docker/docker-compose.override.yml create mode 100755 watcher diff --git a/CHANGELOG.md b/CHANGELOG.md index 23bd6c92..2106853b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,9 @@ - New styling that follows the source{d} branding ([#139](https://github.com/src-d/sourced-ui/issues/139), [#142](https://github.com/src-d/sourced-ui/issues/142), [#204](https://github.com/src-d/sourced-ui/pull/204).) - Improved method to export and import dashboards as JSON ([#165](https://github.com/src-d/sourced-ui/issues/165)). +- Added a development mode to run source{d} in a hot reloading way, so every change in `srcd` or `superset` +will trigger a refresh in the rowser [[see docs](./CONTRIBUTING.md#run-sourced-ce-for-development-with-hot-reloading)]. + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..dd91f3ee --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,107 @@ +# Contribution Guidelines + +As all source{d} projects, this project follows the +[source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md). + + +# Additional Contribution Guidelines + +In addition to the [source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md), +this project follows the following guidelines. + +**Content:** + +- [Changelog](#changelog) +- [Run source{d} CE For Development With Hot Reloading](#run-source-d-ce-for-development-with-hot-reloading) + + +## Changelog + +This project lists the important changes between releases in the +[`CHANGELOG.md`](CHANGELOG.md) file. + +If you open a PR, you should also add a brief summary in the `CHANGELOG.md` +mentioning the new feature, change or bugfix that you proposed. + + +## Run source{d} CE For Development With Hot Reloading + +Running **source{d} CE** in development mode will enable hot reloading at +http://127.0.0.1:8088, so every change you perform in `srcd` or `superset` +directories will trigger a refresh in your browser with the new code. + +1. Install the latest version of [**source{d} CE**](https://github.com/src-d/sourced-ce/releases) +(see [installation guide](https://docs.sourced.tech/community-edition/quickstart/2-install-sourced) +if needed) + +1. Run the watcher. + +
+ IMPORTANT note: Requirements for the watcher + + The watcher requires either [`inotify-tools`](https://github.com/rvoicilas/inotify-tools/wiki) + (for Linux), or [`fswatch`](https://github.com/emcrisostomo/fswatch) + (for Linux and MacOS) + + - To install `inotify-tools` + - in Ubuntu you can run: + ```shell + $ sudo apt-get install inotify-tools + ``` + + - To install `fswatch`: + - in MacOS, you can [use brew to install `fswatch`](https://brewinstall.org/install-fswatch-on-mac-with-brew): + ```shell + $ brew install fswatch + ``` + - in Ubuntu you can build and install [from `fswatch` sources](https://github.com/emcrisostomo/fswatch): + ```shell + $ ./autogen.sh + $ ./configure + $ make + $ sudo make install + $ sudo ldconfig + ``` +
+ +
+ ALTERNATIVE: If you can not use any watcher + + - If you cannot use any of these watchers, you can just run this: + + ```shell + $ make set-override # to prepare the environment + $ make patch # needed EVERYTIME you change something in `srcd` + $ make clean # once you finish, to clean up everything. + ``` +
+ + To watch for changes in `srcd` directory, run the local watcher in one tab, + from your local `sourced-ui` root directory: + + ```shell + $ make dev-prepare + ``` + + You can stop it pressing `Ctrl+C`. Once you do it, the patch will be + automatically cleaned. + +1. Run `sourced` as usually (see [execution guide](https://docs.sourced.tech/community-edition/quickstart/3-init-sourced) + if needed): + + ```shell + # for repositories from GitHub organizations + $ sourced init orgs --token= + + # or, for repositories stored locally + $ sourced init local + ``` + + The first time you launch it, it will take some time to build all the UI assets, + you can see the progress of the build from `sourced-ui` logs (see next step). + +1. To see `sourced-ui` logs (with `npm` errors and such), run: + + ```shell + $ sourced logs -f sourced-ui + ``` diff --git a/Makefile b/Makefile index fbf78d9e..43f7d0af 100644 --- a/Makefile +++ b/Makefile @@ -7,6 +7,13 @@ SUPERSET_DIR = superset # Directory with custom code to copy into SUPERSET_DIR PATCH_SOURCE_DIR = srcd +# Directory with template for docker-compose.override.yml +OVERRIDE_TEMPLATE_PATH := $(PATCH_SOURCE_DIR)/contrib/docker/docker-compose.override.yml +# Directory for the sourced global docker-compose.override.yml +OVERRIDE_OUTPUT_PATH := $(HOME)/.sourced/compose-files/__active__/docker-compose.override.yml +# Directory for the docker-compose.override.yml backup +OVERRIDE_BACKUP_PATH := $(OVERRIDE_OUTPUT_PATH).bak + # Name of the docker image to build DOCKER_IMAGE_NAME ?= srcd/sourced-ui # Docker registry where the docker image should be pushed to. @@ -21,6 +28,29 @@ DOCKER_PASSWORD ?= # will cause make docker-push to also push the latest tag. DOCKER_PUSH_LATEST ?= +# Tools +STAT := stat -c + + +# Set the user as root inside of the container to have permissions to change +# internal `superset` UID and GUID, and make it match the host user. +# It will grant write access to the data from the volumes, +# inside of the container and on host file system (see #221) +LOGIN_USER := 0 +ifeq ($(shell uname),Darwin) + STAT := stat -f + # In OSX there's no problem if the host user and the internal user are + # different when reading and accessing data trough docker volumes. + LOGIN_USER := superset +endif + +SOURCED_UI_ABS_PATH := $(shell pwd) +LOCAL_USER := $(shell $(STAT) "%u" superset/superset) +export SOURCED_UI_ABS_PATH +export LOCAL_USER +export LOGIN_USER + + # Build information VERSION ?= latest # Travis CI @@ -33,9 +63,13 @@ IS_RELEASE := $(shell echo $(VERSION) | grep -q -E '^v[[:digit:]]+\.[[:digit:]]+ all: build -# Copy src-d files in the superset repository +# Clean, and copy src-d files in the superset repository .PHONY: patch -patch: clean +patch: clean apply-patch + +# Copy src-d files in the superset repository +.PHONY: apply-patch +apply-patch: cp -r $(PATCH_SOURCE_DIR)/* $(SUPERSET_DIR)/ # Copy src-d files in the superset repository using symlinks. it's useful for development. @@ -49,6 +83,30 @@ patch-dev: clean done; \ ln -s "$(PWD)/$(PATCH_SOURCE_DIR)/superset/superset_config_dev.py" "$(SUPERSET_DIR)/superset_config.py"; \ +# Start a watcher that will run 'make apply-patch' automatically when 'srcd' changes +# It will require either inotify or fswatch. More info in CONTRIBUTING.md +.PHONY: watch +watch: + @DIRECTORY_TO_OBSERVE=$(PATCH_SOURCE_DIR) bash watcher + +# Writes the proper `docker-compose.override.yml` as the sourced global override file +.PHONY: set-override +set-override: $(OVERRIDE_BACKUP_PATH) + @awk '{ system("echo \""$$0"\"") }' $(OVERRIDE_TEMPLATE_PATH) > $(OVERRIDE_OUTPUT_PATH) + +# Creates a backup of the sourced global override file if it exists +$(OVERRIDE_BACKUP_PATH): + @mkdir -p ~/.sourced/compose-files/__active__ + @if [ -f "$(OVERRIDE_OUTPUT_PATH)" ]; then \ + cp $(OVERRIDE_OUTPUT_PATH) $(OVERRIDE_BACKUP_PATH); \ + else \ + echo "\033[33mno docker-compose.override.yml to backup\033[0m"; \ + fi; + +# Prepares the development enviroment with hot reloading +.PHONY: dev-prepare +dev-prepare: set-override watch + # Create docker image .PHONY: patch build: patch @@ -92,6 +150,12 @@ docker-push-latest-release: clean: rm -f "$(SUPERSET_DIR)/superset_config.py" git clean -fd $(SUPERSET_DIR) + rm -f $(OVERRIDE_OUTPUT_PATH) + @if [ -f "$(OVERRIDE_BACKUP_PATH)" ]; then \ + mv $(OVERRIDE_BACKUP_PATH) $(OVERRIDE_OUTPUT_PATH); \ + else \ + echo "\033[33mno docker-compose.override.yml.bak to restore\033[0m"; \ + fi; # Add superset upstream remote if doesn't exists .PHONY: remote-add diff --git a/srcd/contrib/docker/docker-compose.override.yml b/srcd/contrib/docker/docker-compose.override.yml new file mode 100644 index 00000000..15a2855c --- /dev/null +++ b/srcd/contrib/docker/docker-compose.override.yml @@ -0,0 +1,13 @@ +version: '3.2' +services: + sourced-ui: + user: ${LOGIN_USER:-superset}:${LOGIN_USER:-superset} + volumes: + - ${SOURCED_UI_ABS_PATH}/superset/superset:/home/superset/superset + - ${SOURCED_UI_ABS_PATH}/superset/dashboards:/home/superset/dashboards + - ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/superset_config.py:/home/superset/superset/superset_config.py + - ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/bootstrap.py:/home/superset/bootstrap.py + - ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/docker-entrypoint.sh:/entrypoint.sh + environment: + SUPERSET_ENV: development + LOCAL_USER: ${LOCAL_USER:-} diff --git a/superset/contrib/docker/docker-entrypoint.sh b/superset/contrib/docker/docker-entrypoint.sh index 6e708cef..17ad1155 100755 --- a/superset/contrib/docker/docker-entrypoint.sh +++ b/superset/contrib/docker/docker-entrypoint.sh @@ -17,6 +17,19 @@ # set -ex +if [ "`whoami`" = "root" ] && [ -n "$LOCAL_USER" ]; then + + # Change the UID and GUID of 'superset' user (matching host user). + # It will grant write access to the data from the volumes, + # inside of the container and on host file system (see #221) + find . -group superset -exec chgrp $LOCAL_USER '{}' \; + groupmod -g $LOCAL_USER superset + usermod -u $LOCAL_USER superset + + su -c "/entrypoint.sh" superset + exit $? +fi + if [ "$SUPERSET_NO_DB_INIT" != "true" ]; then python bootstrap.py fi @@ -25,10 +38,13 @@ if [ "$#" -ne 0 ]; then exec "$@" elif [ "$SUPERSET_ENV" = "development" ]; then celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair & - # needed by superset runserver + # In development mode, the UI will be served by `webpack-dev-server` instead of by `Flask` + # `webpack-dev-server` will serve the UI from the port 8088, and it will proxy + # non-asset requests to `Flask`, wich is listening at the port 8081 + # Doing so, updates to asset sources will be reflected in-browser without a refresh. (cd superset/assets/ && npm ci) - (cd superset/assets/ && npm run dev) & - FLASK_ENV=development FLASK_APP=superset:app flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0 + (cd superset/assets/ && npm run dev-server -- --host=0.0.0.0 --port=8088 --supersetPort=8081) & + FLASK_ENV=development FLASK_APP=superset:app flask run -p 8081 --with-threads --reload --debugger --host=0.0.0.0 elif [ "$SUPERSET_ENV" = "production" ]; then celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair & gunicorn --bind 0.0.0.0:8088 \ diff --git a/watcher b/watcher new file mode 100755 index 00000000..6d6bd5b6 --- /dev/null +++ b/watcher @@ -0,0 +1,48 @@ +#!/bin/bash + +trap ctrl_c SIGINT + +function ctrl_c { + make --no-print-directory clean && \ + echo -e "\033[1;92mPatch cleaned\033[0m" || \ + echo -e "\033[1;33mCould not clean patch\033[0m" + exit $? +} + +function watch_inotify { + inotifywait --recursive \ + --event modify,move,create,delete \ + $DIRECTORY_TO_OBSERVE +} + +function watch_fswatch { + fswatch --recursive --one-event \ + --event Created --event Updated --event Removed \ + ${DIRECTORY_TO_OBSERVE} +} + + +if command -v inotifywait > /dev/null 2>&1; then + whichWatcher=inotify + function watcher { watch_inotify; } +elif command -v fswatch > /dev/null 2>&1; then + whichWatcher=fswatch + function watcher { watch_fswatch; } +else + echo -e "\033[1;31m[ERROR] No watcher available\033[0m" + echo "Neither 'inotify' nor 'fswatch' are available." + echo "You can follow 'CONTRIBUTING.md' guidelines to install one of them," + echo "" + echo -e "\033[1;33m[alternative]\033[0m follow these steps:" + echo "1. run 'make set-override' first," + echo "2. then 'make patch' every time you change a file in 'srcd' dir" + echo "3. and 'make clean' once you finish developing" + echo "" + exit 1 +fi + +make --no-print-directory apply-patch +echo -e "\033[1;92mWatching for changes in 'srcd'; using '${whichWatcher}' ...\033[0m" +while watcher; do + make --no-print-directory apply-patch +done From 26b55b1bd165958c3cd47d88138fefe3da5a34e0 Mon Sep 17 00:00:00 2001 From: David Pordomingo Date: Wed, 17 Jul 2019 12:18:55 +0200 Subject: [PATCH 2/4] Polish README.md, moving info to CONTRIBUTING.md Signed-off-by: David Pordomingo --- CONTRIBUTING.md | 38 ++++++++++++++++++++++++++++++++++++ README.md | 51 +------------------------------------------------ 2 files changed, 39 insertions(+), 50 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dd91f3ee..b378b95b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,6 +12,8 @@ this project follows the following guidelines. **Content:** - [Changelog](#changelog) +- [Work With Superset Upstream](#work-with-superset-upstream) +- [Build Docker Image](#build-docker-image) - [Run source{d} CE For Development With Hot Reloading](#run-source-d-ce-for-development-with-hot-reloading) @@ -24,6 +26,42 @@ If you open a PR, you should also add a brief summary in the `CHANGELOG.md` mentioning the new feature, change or bugfix that you proposed. +## Work With Superset Upstream + +Superset version which we are based on is defined in `Makefile`. + +To see which files are patched compare to upstream, run: + +```shell +$ make diff-stat +``` + +To see diff with upstream, run: + +```shell +$ make diff +``` + + +## Build Docker Image + +The official Docker images of sourced-ui, used by **source{d} CE** are released at +[hub.docker.com/r/srcd/sourced-ui](https://hub.docker.com/r/srcd/sourced-ui). You +can also build yours running: + +```shell +$ make build +``` + +The docker image name and tag are defined by the [`Makefile`](Makefile) and can +be overridden passing environment variables to the `build` target, example: + +```shell +$ DOCKER_IMAGE_NAME=my/sourced-ui VERSION=local make build +``` + +will locally build an image called `my/sourced-ui:local` + ## Run source{d} CE For Development With Hot Reloading Running **source{d} CE** in development mode will enable hot reloading at diff --git a/README.md b/README.md index e697793c..6ded4548 100644 --- a/README.md +++ b/README.md @@ -2,10 +2,6 @@ Web UI for [source{d} Community Edition (CE)](https://github.com/src-d/sourced-ce). -## Contents - -- [Description](#description) -- [Development](#development) ## Description @@ -22,6 +18,7 @@ This repository contains the code for the [`srcd/sourced-ui`](https://hub.docker ### Environment Variables + You can configure the Docker image using the following environment variables: | Environment Variable | Description | @@ -54,52 +51,6 @@ You can configure the Docker image using the following environment variables: | `METADATA_PASSWORD` | Password for metadata DB (when `SYNC_MODE` is set to `true`) | | `METADATA_DB` | Database name for metadata (when `SYNC_MODE` is set to `true`) | -## Development - -### Setup local environment - -Download the `docker-compose.yml` file from [`src-d/sourced-ce`](https://github.com/src-d/sourced-ce), and run the dependencies: -``` -docker-compose up gitbase bblfsh-web -``` - -Update superset directory: - -``` -make patch-dev -``` - -Enter into `superset` directory: -``` -cd superset -``` - -Follow original superset instructions for [Flask server](https://github.com/apache/incubator-superset/blob/release--0.32/CONTRIBUTING.md#flask-server) and [Frontend assets](https://github.com/apache/incubator-superset/blob/release--0.32/CONTRIBUTING.md#frontend-assets) - - -### Build docker image - -``` -make build -``` - -The image name is defined in the `Makefile`. - -### Work with superset upstream - -Superset version which we are based on is defined in `Makefile`. - -To see which files are patched compare to upstream, run: - -``` -make diff-stat -``` - -To see diff with upstream, run: - -``` -make diff -``` ## Contribute From d88f81f206dd3da38fcf7ed6dc45a39125097413 Mon Sep 17 00:00:00 2001 From: David Pordomingo Date: Wed, 17 Jul 2019 12:19:52 +0200 Subject: [PATCH 3/4] Polish Makefile removing unused target 'patch-dev' Since 'make patch-dev' is no longer needed, we can get rid of it, and also delete 'srcd/superset/superset_config_dev.py' Signed-off-by: David Pordomingo --- Makefile | 11 ----------- srcd/superset/superset_config_dev.py | 11 ----------- 2 files changed, 22 deletions(-) delete mode 100644 srcd/superset/superset_config_dev.py diff --git a/Makefile b/Makefile index 43f7d0af..4a0cf328 100644 --- a/Makefile +++ b/Makefile @@ -72,17 +72,6 @@ patch: clean apply-patch apply-patch: cp -r $(PATCH_SOURCE_DIR)/* $(SUPERSET_DIR)/ -# Copy src-d files in the superset repository using symlinks. it's useful for development. -# Allows to run flask locally and work only inside superset directory. -.PHONY: patch-dev -patch-dev: clean - @diff=`diff -r $(PATCH_SOURCE_DIR) $(SUPERSET_DIR) | grep "$(PATCH_SOURCE_DIR)" | awk '{gsub(/: /,"/");print $$3}'`; \ - for file in $${diff}; do \ - to=`echo $${file} | cut -d'/' -f2-`; \ - ln -s "$(PWD)/$${file}" "$(SUPERSET_DIR)/$${to}"; \ - done; \ - ln -s "$(PWD)/$(PATCH_SOURCE_DIR)/superset/superset_config_dev.py" "$(SUPERSET_DIR)/superset_config.py"; \ - # Start a watcher that will run 'make apply-patch' automatically when 'srcd' changes # It will require either inotify or fswatch. More info in CONTRIBUTING.md .PHONY: watch diff --git a/srcd/superset/superset_config_dev.py b/srcd/superset/superset_config_dev.py deleted file mode 100644 index 401c86dc..00000000 --- a/srcd/superset/superset_config_dev.py +++ /dev/null @@ -1,11 +0,0 @@ -GITBASE_DATABASE_URI = 'mysql://root@127.0.0.1:3306/gitbase' -BBLFSH_WEB_ADDRESS = 'http://127.0.0.1:9999' -WTF_CSRF_EXEMPT_LIST = ['superset.bblfsh.views.api'] - - -def mutator(): - # pylint: disable=unused-variable - from superset.bblfsh import views # noqa - - -FLASK_APP_MUTATOR = mutator From ab8d628290f131f3013b6b3e3a07b30759f18501 Mon Sep 17 00:00:00 2001 From: David Pordomingo Date: Fri, 5 Jul 2019 11:59:39 +0200 Subject: [PATCH 4/4] Restore contrib/docker/docker-init.sh to its initial state Signed-off-by: David Pordomingo --- superset/contrib/docker/docker-init.sh | 49 +++++--------------------- 1 file changed, 9 insertions(+), 40 deletions(-) diff --git a/superset/contrib/docker/docker-init.sh b/superset/contrib/docker/docker-init.sh index d918eb56..0e8b87d1 100755 --- a/superset/contrib/docker/docker-init.sh +++ b/superset/contrib/docker/docker-init.sh @@ -17,47 +17,16 @@ # set -ex -# always run migrations -superset db upgrade - -# always gitbase script to update datasource if it was changed in env var -python add_gitbase.py - -# add metadata data source only in sync mode -if [ ! -z "$SYNC_MODE" ]; then - python add_metadata_db.py -fi +# Create an admin user (you will be prompted to set username, first and last name before setting a password) +fabmanager create-admin --app superset -# initialize database if empty -if ! fabmanager list-users --app superset | grep -q $ADMIN_LOGIN; then - # Create an admin user - fabmanager create-admin \ - --app superset \ - --username $ADMIN_LOGIN \ - --firstname $ADMIN_FIRST_NAME \ - --lastname $ADMIN_LAST_NAME \ - --email $ADMIN_EMAIL \ - --password $ADMIN_PASSWORD - - # Create default roles and permissions - superset init - - # Add dashboards - superset import_dashboards --path /home/superset/dashboards/gitbase/overview.json - - # Add local or organization dashboards - if [ -z "$SYNC_MODE" ]; then - sleep 2s - superset import_dashboards --path /home/superset/dashboards/gitbase/welcome.json - else - sleep 2s - superset import_dashboards --path /home/superset/dashboards/metadata/welcome.json - - sleep 2s - superset import_dashboards --path /home/superset/dashboards/metadata/collaboration.json - fi +# Initialize the database +superset db upgrade - # set welcome dashboard as a default - python set_default_dashboard.py +if [ "$SUPERSET_LOAD_EXAMPLES" = "yes" ]; then + # Load some data to play with + superset load_examples fi +# Create default roles and permissions +superset init