diff --git a/CHANGELOG.md b/CHANGELOG.md
index 23bd6c92..2106853b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,9 @@
- New styling that follows the source{d} branding ([#139](https://github.com/src-d/sourced-ui/issues/139), [#142](https://github.com/src-d/sourced-ui/issues/142), [#204](https://github.com/src-d/sourced-ui/pull/204).)
- Improved method to export and import dashboards as JSON ([#165](https://github.com/src-d/sourced-ui/issues/165)).
+- Added a development mode to run source{d} in a hot reloading way, so every change in `srcd` or `superset`
+will trigger a refresh in the rowser [[see docs](./CONTRIBUTING.md#run-sourced-ce-for-development-with-hot-reloading)].
+
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 00000000..b378b95b
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,145 @@
+# Contribution Guidelines
+
+As all source{d} projects, this project follows the
+[source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md).
+
+
+# Additional Contribution Guidelines
+
+In addition to the [source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md),
+this project follows the following guidelines.
+
+**Content:**
+
+- [Changelog](#changelog)
+- [Work With Superset Upstream](#work-with-superset-upstream)
+- [Build Docker Image](#build-docker-image)
+- [Run source{d} CE For Development With Hot Reloading](#run-source-d-ce-for-development-with-hot-reloading)
+
+
+## Changelog
+
+This project lists the important changes between releases in the
+[`CHANGELOG.md`](CHANGELOG.md) file.
+
+If you open a PR, you should also add a brief summary in the `CHANGELOG.md`
+mentioning the new feature, change or bugfix that you proposed.
+
+
+## Work With Superset Upstream
+
+Superset version which we are based on is defined in `Makefile`.
+
+To see which files are patched compare to upstream, run:
+
+```shell
+$ make diff-stat
+```
+
+To see diff with upstream, run:
+
+```shell
+$ make diff
+```
+
+
+## Build Docker Image
+
+The official Docker images of sourced-ui, used by **source{d} CE** are released at
+[hub.docker.com/r/srcd/sourced-ui](https://hub.docker.com/r/srcd/sourced-ui). You
+can also build yours running:
+
+```shell
+$ make build
+```
+
+The docker image name and tag are defined by the [`Makefile`](Makefile) and can
+be overridden passing environment variables to the `build` target, example:
+
+```shell
+$ DOCKER_IMAGE_NAME=my/sourced-ui VERSION=local make build
+```
+
+will locally build an image called `my/sourced-ui:local`
+
+## Run source{d} CE For Development With Hot Reloading
+
+Running **source{d} CE** in development mode will enable hot reloading at
+http://127.0.0.1:8088, so every change you perform in `srcd` or `superset`
+directories will trigger a refresh in your browser with the new code.
+
+1. Install the latest version of [**source{d} CE**](https://github.com/src-d/sourced-ce/releases)
+(see [installation guide](https://docs.sourced.tech/community-edition/quickstart/2-install-sourced)
+if needed)
+
+1. Run the watcher.
+
+
+ IMPORTANT note: Requirements for the watcher
+
+ The watcher requires either [`inotify-tools`](https://github.com/rvoicilas/inotify-tools/wiki)
+ (for Linux), or [`fswatch`](https://github.com/emcrisostomo/fswatch)
+ (for Linux and MacOS)
+
+ - To install `inotify-tools`
+ - in Ubuntu you can run:
+ ```shell
+ $ sudo apt-get install inotify-tools
+ ```
+
+ - To install `fswatch`:
+ - in MacOS, you can [use brew to install `fswatch`](https://brewinstall.org/install-fswatch-on-mac-with-brew):
+ ```shell
+ $ brew install fswatch
+ ```
+ - in Ubuntu you can build and install [from `fswatch` sources](https://github.com/emcrisostomo/fswatch):
+ ```shell
+ $ ./autogen.sh
+ $ ./configure
+ $ make
+ $ sudo make install
+ $ sudo ldconfig
+ ```
+
+
+
+ ALTERNATIVE: If you can not use any watcher
+
+ - If you cannot use any of these watchers, you can just run this:
+
+ ```shell
+ $ make set-override # to prepare the environment
+ $ make patch # needed EVERYTIME you change something in `srcd`
+ $ make clean # once you finish, to clean up everything.
+ ```
+
+
+ To watch for changes in `srcd` directory, run the local watcher in one tab,
+ from your local `sourced-ui` root directory:
+
+ ```shell
+ $ make dev-prepare
+ ```
+
+ You can stop it pressing `Ctrl+C`. Once you do it, the patch will be
+ automatically cleaned.
+
+1. Run `sourced` as usually (see [execution guide](https://docs.sourced.tech/community-edition/quickstart/3-init-sourced)
+ if needed):
+
+ ```shell
+ # for repositories from GitHub organizations
+ $ sourced init orgs --token=
+
+ # or, for repositories stored locally
+ $ sourced init local
+ ```
+
+ The first time you launch it, it will take some time to build all the UI assets,
+ you can see the progress of the build from `sourced-ui` logs (see next step).
+
+1. To see `sourced-ui` logs (with `npm` errors and such), run:
+
+ ```shell
+ $ sourced logs -f sourced-ui
+ ```
diff --git a/Makefile b/Makefile
index fbf78d9e..4a0cf328 100644
--- a/Makefile
+++ b/Makefile
@@ -7,6 +7,13 @@ SUPERSET_DIR = superset
# Directory with custom code to copy into SUPERSET_DIR
PATCH_SOURCE_DIR = srcd
+# Directory with template for docker-compose.override.yml
+OVERRIDE_TEMPLATE_PATH := $(PATCH_SOURCE_DIR)/contrib/docker/docker-compose.override.yml
+# Directory for the sourced global docker-compose.override.yml
+OVERRIDE_OUTPUT_PATH := $(HOME)/.sourced/compose-files/__active__/docker-compose.override.yml
+# Directory for the docker-compose.override.yml backup
+OVERRIDE_BACKUP_PATH := $(OVERRIDE_OUTPUT_PATH).bak
+
# Name of the docker image to build
DOCKER_IMAGE_NAME ?= srcd/sourced-ui
# Docker registry where the docker image should be pushed to.
@@ -21,6 +28,29 @@ DOCKER_PASSWORD ?=
# will cause make docker-push to also push the latest tag.
DOCKER_PUSH_LATEST ?=
+# Tools
+STAT := stat -c
+
+
+# Set the user as root inside of the container to have permissions to change
+# internal `superset` UID and GUID, and make it match the host user.
+# It will grant write access to the data from the volumes,
+# inside of the container and on host file system (see #221)
+LOGIN_USER := 0
+ifeq ($(shell uname),Darwin)
+ STAT := stat -f
+ # In OSX there's no problem if the host user and the internal user are
+ # different when reading and accessing data trough docker volumes.
+ LOGIN_USER := superset
+endif
+
+SOURCED_UI_ABS_PATH := $(shell pwd)
+LOCAL_USER := $(shell $(STAT) "%u" superset/superset)
+export SOURCED_UI_ABS_PATH
+export LOCAL_USER
+export LOGIN_USER
+
+
# Build information
VERSION ?= latest
# Travis CI
@@ -33,21 +63,38 @@ IS_RELEASE := $(shell echo $(VERSION) | grep -q -E '^v[[:digit:]]+\.[[:digit:]]+
all: build
-# Copy src-d files in the superset repository
+# Clean, and copy src-d files in the superset repository
.PHONY: patch
-patch: clean
+patch: clean apply-patch
+
+# Copy src-d files in the superset repository
+.PHONY: apply-patch
+apply-patch:
cp -r $(PATCH_SOURCE_DIR)/* $(SUPERSET_DIR)/
-# Copy src-d files in the superset repository using symlinks. it's useful for development.
-# Allows to run flask locally and work only inside superset directory.
-.PHONY: patch-dev
-patch-dev: clean
- @diff=`diff -r $(PATCH_SOURCE_DIR) $(SUPERSET_DIR) | grep "$(PATCH_SOURCE_DIR)" | awk '{gsub(/: /,"/");print $$3}'`; \
- for file in $${diff}; do \
- to=`echo $${file} | cut -d'/' -f2-`; \
- ln -s "$(PWD)/$${file}" "$(SUPERSET_DIR)/$${to}"; \
- done; \
- ln -s "$(PWD)/$(PATCH_SOURCE_DIR)/superset/superset_config_dev.py" "$(SUPERSET_DIR)/superset_config.py"; \
+# Start a watcher that will run 'make apply-patch' automatically when 'srcd' changes
+# It will require either inotify or fswatch. More info in CONTRIBUTING.md
+.PHONY: watch
+watch:
+ @DIRECTORY_TO_OBSERVE=$(PATCH_SOURCE_DIR) bash watcher
+
+# Writes the proper `docker-compose.override.yml` as the sourced global override file
+.PHONY: set-override
+set-override: $(OVERRIDE_BACKUP_PATH)
+ @awk '{ system("echo \""$$0"\"") }' $(OVERRIDE_TEMPLATE_PATH) > $(OVERRIDE_OUTPUT_PATH)
+
+# Creates a backup of the sourced global override file if it exists
+$(OVERRIDE_BACKUP_PATH):
+ @mkdir -p ~/.sourced/compose-files/__active__
+ @if [ -f "$(OVERRIDE_OUTPUT_PATH)" ]; then \
+ cp $(OVERRIDE_OUTPUT_PATH) $(OVERRIDE_BACKUP_PATH); \
+ else \
+ echo "\033[33mno docker-compose.override.yml to backup\033[0m"; \
+ fi;
+
+# Prepares the development enviroment with hot reloading
+.PHONY: dev-prepare
+dev-prepare: set-override watch
# Create docker image
.PHONY: patch
@@ -92,6 +139,12 @@ docker-push-latest-release:
clean:
rm -f "$(SUPERSET_DIR)/superset_config.py"
git clean -fd $(SUPERSET_DIR)
+ rm -f $(OVERRIDE_OUTPUT_PATH)
+ @if [ -f "$(OVERRIDE_BACKUP_PATH)" ]; then \
+ mv $(OVERRIDE_BACKUP_PATH) $(OVERRIDE_OUTPUT_PATH); \
+ else \
+ echo "\033[33mno docker-compose.override.yml.bak to restore\033[0m"; \
+ fi;
# Add superset upstream remote if doesn't exists
.PHONY: remote-add
diff --git a/README.md b/README.md
index e697793c..6ded4548 100644
--- a/README.md
+++ b/README.md
@@ -2,10 +2,6 @@
Web UI for [source{d} Community Edition (CE)](https://github.com/src-d/sourced-ce).
-## Contents
-
-- [Description](#description)
-- [Development](#development)
## Description
@@ -22,6 +18,7 @@ This repository contains the code for the [`srcd/sourced-ui`](https://hub.docker
### Environment Variables
+
You can configure the Docker image using the following environment variables:
| Environment Variable | Description |
@@ -54,52 +51,6 @@ You can configure the Docker image using the following environment variables:
| `METADATA_PASSWORD` | Password for metadata DB (when `SYNC_MODE` is set to `true`) |
| `METADATA_DB` | Database name for metadata (when `SYNC_MODE` is set to `true`) |
-## Development
-
-### Setup local environment
-
-Download the `docker-compose.yml` file from [`src-d/sourced-ce`](https://github.com/src-d/sourced-ce), and run the dependencies:
-```
-docker-compose up gitbase bblfsh-web
-```
-
-Update superset directory:
-
-```
-make patch-dev
-```
-
-Enter into `superset` directory:
-```
-cd superset
-```
-
-Follow original superset instructions for [Flask server](https://github.com/apache/incubator-superset/blob/release--0.32/CONTRIBUTING.md#flask-server) and [Frontend assets](https://github.com/apache/incubator-superset/blob/release--0.32/CONTRIBUTING.md#frontend-assets)
-
-
-### Build docker image
-
-```
-make build
-```
-
-The image name is defined in the `Makefile`.
-
-### Work with superset upstream
-
-Superset version which we are based on is defined in `Makefile`.
-
-To see which files are patched compare to upstream, run:
-
-```
-make diff-stat
-```
-
-To see diff with upstream, run:
-
-```
-make diff
-```
## Contribute
diff --git a/srcd/contrib/docker/docker-compose.override.yml b/srcd/contrib/docker/docker-compose.override.yml
new file mode 100644
index 00000000..15a2855c
--- /dev/null
+++ b/srcd/contrib/docker/docker-compose.override.yml
@@ -0,0 +1,13 @@
+version: '3.2'
+services:
+ sourced-ui:
+ user: ${LOGIN_USER:-superset}:${LOGIN_USER:-superset}
+ volumes:
+ - ${SOURCED_UI_ABS_PATH}/superset/superset:/home/superset/superset
+ - ${SOURCED_UI_ABS_PATH}/superset/dashboards:/home/superset/dashboards
+ - ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/superset_config.py:/home/superset/superset/superset_config.py
+ - ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/bootstrap.py:/home/superset/bootstrap.py
+ - ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/docker-entrypoint.sh:/entrypoint.sh
+ environment:
+ SUPERSET_ENV: development
+ LOCAL_USER: ${LOCAL_USER:-}
diff --git a/srcd/superset/superset_config_dev.py b/srcd/superset/superset_config_dev.py
deleted file mode 100644
index 401c86dc..00000000
--- a/srcd/superset/superset_config_dev.py
+++ /dev/null
@@ -1,11 +0,0 @@
-GITBASE_DATABASE_URI = 'mysql://root@127.0.0.1:3306/gitbase'
-BBLFSH_WEB_ADDRESS = 'http://127.0.0.1:9999'
-WTF_CSRF_EXEMPT_LIST = ['superset.bblfsh.views.api']
-
-
-def mutator():
- # pylint: disable=unused-variable
- from superset.bblfsh import views # noqa
-
-
-FLASK_APP_MUTATOR = mutator
diff --git a/superset/contrib/docker/docker-entrypoint.sh b/superset/contrib/docker/docker-entrypoint.sh
index 6e708cef..17ad1155 100755
--- a/superset/contrib/docker/docker-entrypoint.sh
+++ b/superset/contrib/docker/docker-entrypoint.sh
@@ -17,6 +17,19 @@
#
set -ex
+if [ "`whoami`" = "root" ] && [ -n "$LOCAL_USER" ]; then
+
+ # Change the UID and GUID of 'superset' user (matching host user).
+ # It will grant write access to the data from the volumes,
+ # inside of the container and on host file system (see #221)
+ find . -group superset -exec chgrp $LOCAL_USER '{}' \;
+ groupmod -g $LOCAL_USER superset
+ usermod -u $LOCAL_USER superset
+
+ su -c "/entrypoint.sh" superset
+ exit $?
+fi
+
if [ "$SUPERSET_NO_DB_INIT" != "true" ]; then
python bootstrap.py
fi
@@ -25,10 +38,13 @@ if [ "$#" -ne 0 ]; then
exec "$@"
elif [ "$SUPERSET_ENV" = "development" ]; then
celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair &
- # needed by superset runserver
+ # In development mode, the UI will be served by `webpack-dev-server` instead of by `Flask`
+ # `webpack-dev-server` will serve the UI from the port 8088, and it will proxy
+ # non-asset requests to `Flask`, wich is listening at the port 8081
+ # Doing so, updates to asset sources will be reflected in-browser without a refresh.
(cd superset/assets/ && npm ci)
- (cd superset/assets/ && npm run dev) &
- FLASK_ENV=development FLASK_APP=superset:app flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0
+ (cd superset/assets/ && npm run dev-server -- --host=0.0.0.0 --port=8088 --supersetPort=8081) &
+ FLASK_ENV=development FLASK_APP=superset:app flask run -p 8081 --with-threads --reload --debugger --host=0.0.0.0
elif [ "$SUPERSET_ENV" = "production" ]; then
celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair &
gunicorn --bind 0.0.0.0:8088 \
diff --git a/superset/contrib/docker/docker-init.sh b/superset/contrib/docker/docker-init.sh
index d918eb56..0e8b87d1 100755
--- a/superset/contrib/docker/docker-init.sh
+++ b/superset/contrib/docker/docker-init.sh
@@ -17,47 +17,16 @@
#
set -ex
-# always run migrations
-superset db upgrade
-
-# always gitbase script to update datasource if it was changed in env var
-python add_gitbase.py
-
-# add metadata data source only in sync mode
-if [ ! -z "$SYNC_MODE" ]; then
- python add_metadata_db.py
-fi
+# Create an admin user (you will be prompted to set username, first and last name before setting a password)
+fabmanager create-admin --app superset
-# initialize database if empty
-if ! fabmanager list-users --app superset | grep -q $ADMIN_LOGIN; then
- # Create an admin user
- fabmanager create-admin \
- --app superset \
- --username $ADMIN_LOGIN \
- --firstname $ADMIN_FIRST_NAME \
- --lastname $ADMIN_LAST_NAME \
- --email $ADMIN_EMAIL \
- --password $ADMIN_PASSWORD
-
- # Create default roles and permissions
- superset init
-
- # Add dashboards
- superset import_dashboards --path /home/superset/dashboards/gitbase/overview.json
-
- # Add local or organization dashboards
- if [ -z "$SYNC_MODE" ]; then
- sleep 2s
- superset import_dashboards --path /home/superset/dashboards/gitbase/welcome.json
- else
- sleep 2s
- superset import_dashboards --path /home/superset/dashboards/metadata/welcome.json
-
- sleep 2s
- superset import_dashboards --path /home/superset/dashboards/metadata/collaboration.json
- fi
+# Initialize the database
+superset db upgrade
- # set welcome dashboard as a default
- python set_default_dashboard.py
+if [ "$SUPERSET_LOAD_EXAMPLES" = "yes" ]; then
+ # Load some data to play with
+ superset load_examples
fi
+# Create default roles and permissions
+superset init
diff --git a/watcher b/watcher
new file mode 100755
index 00000000..6d6bd5b6
--- /dev/null
+++ b/watcher
@@ -0,0 +1,48 @@
+#!/bin/bash
+
+trap ctrl_c SIGINT
+
+function ctrl_c {
+ make --no-print-directory clean && \
+ echo -e "\033[1;92mPatch cleaned\033[0m" || \
+ echo -e "\033[1;33mCould not clean patch\033[0m"
+ exit $?
+}
+
+function watch_inotify {
+ inotifywait --recursive \
+ --event modify,move,create,delete \
+ $DIRECTORY_TO_OBSERVE
+}
+
+function watch_fswatch {
+ fswatch --recursive --one-event \
+ --event Created --event Updated --event Removed \
+ ${DIRECTORY_TO_OBSERVE}
+}
+
+
+if command -v inotifywait > /dev/null 2>&1; then
+ whichWatcher=inotify
+ function watcher { watch_inotify; }
+elif command -v fswatch > /dev/null 2>&1; then
+ whichWatcher=fswatch
+ function watcher { watch_fswatch; }
+else
+ echo -e "\033[1;31m[ERROR] No watcher available\033[0m"
+ echo "Neither 'inotify' nor 'fswatch' are available."
+ echo "You can follow 'CONTRIBUTING.md' guidelines to install one of them,"
+ echo ""
+ echo -e "\033[1;33m[alternative]\033[0m follow these steps:"
+ echo "1. run 'make set-override' first,"
+ echo "2. then 'make patch' every time you change a file in 'srcd' dir"
+ echo "3. and 'make clean' once you finish developing"
+ echo ""
+ exit 1
+fi
+
+make --no-print-directory apply-patch
+echo -e "\033[1;92mWatching for changes in 'srcd'; using '${whichWatcher}' ...\033[0m"
+while watcher; do
+ make --no-print-directory apply-patch
+done