Development environment with hot reloading

CHANGELOG.md

@@ -11,6 +11,9 @@
 
 - New styling that follows the source{d} branding ([#139](https://github.com/src-d/sourced-ui/issues/139), [#142](https://github.com/src-d/sourced-ui/issues/142), [#204](https://github.com/src-d/sourced-ui/pull/204).)
 - Improved method to export and import dashboards as JSON ([#165](https://github.com/src-d/sourced-ui/issues/165)).
+- Added a development mode to run source{d} in a hot reloading way, so every change in `srcd` or `superset`
+will trigger a refresh in the rowser [[see docs](./CONTRIBUTING.md#run-sourced-ce-for-development-with-hot-reloading)].
+
 
 </details>
 

CONTRIBUTING.md

@@ -0,0 +1,145 @@
+# Contribution Guidelines
+
+As all source{d} projects, this project follows the
+[source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md).
+
+
+# Additional Contribution Guidelines
+
+In addition to the [source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md),
+this project follows the following guidelines.
+
+**Content:**
+
+- [Changelog](#changelog)
+- [Work With Superset Upstream](#work-with-superset-upstream)
+- [Build Docker Image](#build-docker-image)
+- [Run source{d} CE For Development With Hot Reloading](#run-source-d-ce-for-development-with-hot-reloading)
+
+
+## Changelog
+
+This project lists the important changes between releases in the
+[`CHANGELOG.md`](CHANGELOG.md) file.
+
+If you open a PR, you should also add a brief summary in the `CHANGELOG.md`
+mentioning the new feature, change or bugfix that you proposed.
+
+
+## Work With Superset Upstream
+
+Superset version which we are based on is defined in `Makefile`.
+
+To see which files are patched compare to upstream, run:
+
+```shell
+$ make diff-stat
+```
+
+To see diff with upstream, run:
+
+```shell
+$ make diff
+```
+
+
+## Build Docker Image
+
+The official Docker images of sourced-ui, used by **source{d} CE** are released at
+[hub.docker.com/r/srcd/sourced-ui](https://hub.docker.com/r/srcd/sourced-ui). You
+can also build yours running:
+
+```shell
+$ make build
+```
+
+The docker image name and tag are defined by the [`Makefile`](Makefile) and can
+be overridden passing environment variables to the `build` target, example:
+
+```shell
+$ DOCKER_IMAGE_NAME=my/sourced-ui VERSION=local make build
+```
+
+will locally build an image called `my/sourced-ui:local`
+
+## Run source{d} CE For Development With Hot Reloading
+
+Running **source{d} CE** in development mode will enable hot reloading at
+http://127.0.0.1:8088, so every change you perform in `srcd` or `superset`
+directories will trigger a refresh in your browser with the new code.
+
+1. Install the latest version of [**source{d} CE**](https://github.com/src-d/sourced-ce/releases)
+(see [installation guide](https://docs.sourced.tech/community-edition/quickstart/2-install-sourced)
+if needed)
+
+1. Run the watcher.
+
+    <details>
+        <summary><b>IMPORTANT note:</b> Requirements for the watcher</summary>
+
+    The watcher requires either [`inotify-tools`](https://github.com/rvoicilas/inotify-tools/wiki)
+    (for Linux), or [`fswatch`](https://github.com/emcrisostomo/fswatch)
+    (for Linux and MacOS)
+
+    - To install `inotify-tools`
+        - in Ubuntu you can run:
+            ```shell
+            $ sudo apt-get install inotify-tools
+            ```
+
+    - To install `fswatch`:
+        - in MacOS, you can [use brew to install `fswatch`](https://brewinstall.org/install-fswatch-on-mac-with-brew):
+            ```shell
+            $ brew install fswatch
+            ```
+        - in Ubuntu you can build and install [from `fswatch` sources](https://github.com/emcrisostomo/fswatch):
+            ```shell
+            $ ./autogen.sh
+            $ ./configure
+            $ make
+            $ sudo make install
+            $ sudo ldconfig
+            ```
+    </details>
+
+    <details>
+        <summary><b>ALTERNATIVE:</b> If you can not use any watcher</summary>
+
+    - If you cannot use any of these watchers, you can just run this:
+
+        ```shell
+        $ make set-override # to prepare the environment
+        $ make patch # needed EVERYTIME you change something in `srcd`
+        $ make clean # once you finish, to clean up everything.
+        ```
+    </details>
+
+    To watch for changes in `srcd` directory, run the local watcher in one tab,
+    from your local `sourced-ui` root directory:
+
+    ```shell
+    $ make dev-prepare
+    ```
+
+    You can stop it pressing `Ctrl+C`. Once you do it, the patch will be
+    automatically cleaned.
+
+1. Run `sourced` as usually (see [execution guide](https://docs.sourced.tech/community-edition/quickstart/3-init-sourced)
+    if needed):
+
+    ```shell
+    # for repositories from GitHub organizations
+    $ sourced init orgs <orgs...> --token=<token>
+
+    # or, for repositories stored locally
+    $ sourced init local </path/to/repos>
+    ```
+
+    The first time you launch it, it will take some time to build all the UI assets,
+    you can see the progress of the build from `sourced-ui` logs (see next step).
+
+1. To see `sourced-ui` logs (with `npm` errors and such), run:
+
+    ```shell
+    $ sourced logs -f sourced-ui
+    ```

Makefile

@@ -7,6 +7,13 @@ SUPERSET_DIR = superset
 # Directory with custom code to copy into SUPERSET_DIR
 PATCH_SOURCE_DIR = srcd
 
+# Directory with template for docker-compose.override.yml
+OVERRIDE_TEMPLATE_PATH := $(PATCH_SOURCE_DIR)/contrib/docker/docker-compose.override.yml
+# Directory for the sourced global docker-compose.override.yml
+OVERRIDE_OUTPUT_PATH := $(HOME)/.sourced/compose-files/__active__/docker-compose.override.yml
+# Directory for the docker-compose.override.yml backup
+OVERRIDE_BACKUP_PATH := $(OVERRIDE_OUTPUT_PATH).bak
+
 # Name of the docker image to build
 DOCKER_IMAGE_NAME ?= srcd/sourced-ui
 # Docker registry where the docker image should be pushed to.
@@ -21,6 +28,29 @@ DOCKER_PASSWORD ?=
 # will cause make docker-push to also push the latest tag.
 DOCKER_PUSH_LATEST ?=
 
+# Tools
+STAT := stat -c
+
+
+# Set the user as root inside of the container to have permissions to change
+# internal `superset` UID and GUID, and make it match the host user.
+# It will grant write access to the data from the volumes,
+# inside of the container and on host file system (see #221)
+LOGIN_USER := 0
+ifeq ($(shell uname),Darwin)
+	STAT := stat -f
+	# In OSX there's no problem if the host user and the internal user are
+	# different when reading and accessing data trough docker volumes.
+	LOGIN_USER := superset
+endif
+
+SOURCED_UI_ABS_PATH := $(shell pwd)
+LOCAL_USER := $(shell $(STAT) "%u" superset/superset)
+export SOURCED_UI_ABS_PATH
+export LOCAL_USER
+export LOGIN_USER
+
+
 # Build information
 VERSION ?= latest
 # Travis CI
@@ -33,21 +63,38 @@ IS_RELEASE := $(shell echo $(VERSION) | grep -q -E '^v[[:digit:]]+\.[[:digit:]]+
 
 all: build
 
-# Copy src-d files in the superset repository
+# Clean, and copy src-d files in the superset repository
 .PHONY: patch
-patch: clean
+patch: clean apply-patch
+
+# Copy src-d files in the superset repository
+.PHONY: apply-patch
+apply-patch:
 	cp -r $(PATCH_SOURCE_DIR)/* $(SUPERSET_DIR)/
 
-# Copy src-d files in the superset repository using symlinks. it's useful for development.
-# Allows to run flask locally and work only inside superset directory.
-.PHONY: patch-dev
-patch-dev: clean
-	@diff=`diff -r $(PATCH_SOURCE_DIR) $(SUPERSET_DIR) | grep "$(PATCH_SOURCE_DIR)" | awk '{gsub(/: /,"/");print $$3}'`; \
-	for file in $${diff}; do \
-		to=`echo $${file} | cut -d'/' -f2-`; \
-		ln -s "$(PWD)/$${file}" "$(SUPERSET_DIR)/$${to}"; \
-	done; \
-	ln -s "$(PWD)/$(PATCH_SOURCE_DIR)/superset/superset_config_dev.py" "$(SUPERSET_DIR)/superset_config.py"; \
+# Start a watcher that will run 'make apply-patch' automatically when 'srcd' changes
+# It will require either inotify or fswatch. More info in CONTRIBUTING.md
+.PHONY: watch
+watch:
+	@DIRECTORY_TO_OBSERVE=$(PATCH_SOURCE_DIR) bash watcher
+
+# Writes the proper `docker-compose.override.yml` as the sourced global override file
+.PHONY: set-override
+set-override: $(OVERRIDE_BACKUP_PATH)
+	@awk '{ system("echo \""$$0"\"") }' $(OVERRIDE_TEMPLATE_PATH) > $(OVERRIDE_OUTPUT_PATH)
+
+# Creates a backup of the sourced global override file if it exists
+$(OVERRIDE_BACKUP_PATH):
+	@mkdir -p ~/.sourced/compose-files/__active__
+	@if [ -f "$(OVERRIDE_OUTPUT_PATH)" ]; then \
+		cp $(OVERRIDE_OUTPUT_PATH) $(OVERRIDE_BACKUP_PATH); \
+	else \
+		echo "\033[33mno docker-compose.override.yml to backup\033[0m"; \
+	fi;
+
+# Prepares the development enviroment with hot reloading
+.PHONY: dev-prepare
+dev-prepare: set-override watch
 
 # Create docker image
 .PHONY: patch
@@ -92,6 +139,12 @@ docker-push-latest-release:
 clean:
 	rm -f "$(SUPERSET_DIR)/superset_config.py"
 	git clean -fd $(SUPERSET_DIR)
+	rm -f $(OVERRIDE_OUTPUT_PATH)
+	@if [ -f "$(OVERRIDE_BACKUP_PATH)" ]; then \
+		mv $(OVERRIDE_BACKUP_PATH) $(OVERRIDE_OUTPUT_PATH); \
+	else \
+		echo "\033[33mno docker-compose.override.yml.bak to restore\033[0m"; \
+	fi;
 
 # Add superset upstream remote if doesn't exists
 .PHONY: remote-add

README.md

@@ -2,10 +2,6 @@
 
 Web UI for [source{d} Community Edition (CE)](https://github.com/src-d/sourced-ce).
 
-## Contents
-
-- [Description](#description)
-- [Development](#development)
 
 ## Description
 
@@ -22,6 +18,7 @@ This repository contains the code for the [`srcd/sourced-ui`](https://hub.docker
 
 
 ### Environment Variables
+
 You can configure the Docker image using the following environment variables:
 
 | Environment Variable  | Description                                                     |
@@ -54,52 +51,6 @@ You can configure the Docker image using the following environment variables:
 | `METADATA_PASSWORD`   | Password for metadata DB (when `SYNC_MODE` is set to `true`)    |
 | `METADATA_DB`         | Database name for metadata (when `SYNC_MODE` is set to `true`)  |
 
-## Development
-
-### Setup local environment
-
-Download the `docker-compose.yml` file from [`src-d/sourced-ce`](https://github.com/src-d/sourced-ce), and run the dependencies:
-```
-docker-compose up gitbase bblfsh-web
-```
-
-Update superset directory:
-
-```
-make patch-dev
-```
-
-Enter into `superset` directory:
-```
-cd superset
-```
-
-Follow original superset instructions for [Flask server](https://github.com/apache/incubator-superset/blob/release--0.32/CONTRIBUTING.md#flask-server) and [Frontend assets](https://github.com/apache/incubator-superset/blob/release--0.32/CONTRIBUTING.md#frontend-assets)
-
-
-### Build docker image
-
-```
-make build
-```
-
-The image name is defined in the `Makefile`.
-
-### Work with superset upstream
-
-Superset version which we are based on is defined in `Makefile`.
-
-To see which files are patched compare to upstream, run:
-
-```
-make diff-stat
-```
-
-To see diff with upstream, run:
-
-```
-make diff
-```
 
 ## Contribute
 

srcd/contrib/docker/docker-compose.override.yml

@@ -0,0 +1,13 @@
+version: '3.2'
+services:
+  sourced-ui:
+    user: ${LOGIN_USER:-superset}:${LOGIN_USER:-superset}
+    volumes:
+      - ${SOURCED_UI_ABS_PATH}/superset/superset:/home/superset/superset
+      - ${SOURCED_UI_ABS_PATH}/superset/dashboards:/home/superset/dashboards
+      - ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/superset_config.py:/home/superset/superset/superset_config.py
+      - ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/bootstrap.py:/home/superset/bootstrap.py
+      - ${SOURCED_UI_ABS_PATH}/superset/contrib/docker/docker-entrypoint.sh:/entrypoint.sh
+    environment:
+      SUPERSET_ENV: development
+      LOCAL_USER: ${LOCAL_USER:-}

superset/contrib/docker/docker-entrypoint.sh

@@ -17,6 +17,19 @@
 #
 set -ex
 
+if [ "`whoami`" = "root" ] && [ -n "$LOCAL_USER" ]; then
+
+    # Change the UID and GUID of 'superset' user (matching host user).
+    # It will grant write access to the data from the volumes,
+    # inside of the container and on host file system (see #221)
+    find . -group superset -exec chgrp $LOCAL_USER '{}' \;
+    groupmod -g $LOCAL_USER superset
+    usermod -u $LOCAL_USER superset
+
+    su -c "/entrypoint.sh" superset
+    exit $?
+fi
+
 if [ "$SUPERSET_NO_DB_INIT" != "true" ]; then
     python bootstrap.py
 fi
@@ -25,10 +38,13 @@ if [ "$#" -ne 0 ]; then
     exec "$@"
 elif [ "$SUPERSET_ENV" = "development" ]; then
     celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair &
-    # needed by superset runserver
+    # In development mode, the UI will be served by `webpack-dev-server` instead of by `Flask`
+    # `webpack-dev-server` will serve the UI from the port 8088, and it will proxy
+    # non-asset requests to `Flask`, wich is listening at the port 8081
+    # Doing so, updates to asset sources will be reflected in-browser without a refresh.
     (cd superset/assets/ && npm ci)
-    (cd superset/assets/ && npm run dev) &
-    FLASK_ENV=development FLASK_APP=superset:app flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0
+    (cd superset/assets/ && npm run dev-server -- --host=0.0.0.0 --port=8088 --supersetPort=8081) &
+    FLASK_ENV=development FLASK_APP=superset:app flask run -p 8081 --with-threads --reload --debugger --host=0.0.0.0
 elif [ "$SUPERSET_ENV" = "production" ]; then
     celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair &
     gunicorn --bind  0.0.0.0:8088 \