Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: full edx-platform setup with tutor dev launch -m ... #813

Merged
merged 3 commits into from
Mar 15, 2023
Merged

feat: full edx-platform setup with tutor dev launch -m ... #813

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4fbe896
feat: full edx-platform setup with `tutor dev launch -m ...`
kdmccormick Jan 11, 2023
540e33f
docs: update dev.rst based on code review
kdmccormick Mar 14, 2023
fd626f5
feat: mount themes & requirements in dev job containers
kdmccormick Mar 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/20230313_110919_kdmc_mounted_init.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- [Improvement] Running ``tutor dev launch --mount=edx-platform`` now performs all necessary setup for a local edx-platform development. This includes running setup.py, installing node modules, and building assets; previously, those steps had to be run explicitly after bind-mounting a local copy of edx-platform (by @kdmccormick).
106 changes: 53 additions & 53 deletions docs/dev.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,23 +5,29 @@ Open edX development

In addition to running Open edX in production, Tutor can be used for local development of Open edX. This means that it is possible to hack on Open edX without setting up a Virtual Machine. Essentially, this replaces the devstack provided by edX.

.. _edx_platform_dev_env:

First-time setup
----------------

First, ensure you have already :ref:`installed Tutor <install>` (for development against the named releases of Open edX) or :ref:`Tutor Nightly <nightly>` (for development against Open edX's master branches).
Firstly, either :ref:`install Tutor <install>` (for development against the named releases of Open edX) or :ref:`install Tutor Nightly <nightly>` (for development against Open edX's master branches).

Then, launch the developer platform setup process::
Then, run one of the following in order to launch the developer platform setup process::

# To use the edx-platform repository that is built into the image, run:
tutor dev launch

This will perform several tasks for you. It will:
# To bind-mount and run a local clone of edx-platform, replace
# './edx-platform' with the path to the local clone and run:
tutor dev launch --mount=./edx-platform

This will perform several tasks. It will:

* stop any existing locally-running Tutor containers,

* disable HTTPS,

* set your ``LMS_HOST`` to `local.overhang.io <http://local.overhang.io>`_ (a convenience domain that simply `points at 127.0.0.1 <https://dnschecker.org/#A/local.overhang.io>`_),
* set ``LMS_HOST`` to `local.overhang.io <http://local.overhang.io>`_ (a convenience domain that simply `points at 127.0.0.1 <https://dnschecker.org/#A/local.overhang.io>`_),

* prompt for a platform details (with suitable defaults),

Expand All @@ -33,52 +39,82 @@ This will perform several tasks for you. It will:

* run service initialization scripts, such as service user creation and Waffle configuration.

Additionally, when a local clone of edx-platform is bind-mounted, it will:

* re-run setup.py,
kdmccormick marked this conversation as resolved.
Show resolved Hide resolved
* clean-reinstall Node modules, and
* regenerate static assets.

Once setup is complete, the platform will be running in the background:

* LMS will be accessible at `http://local.overhang.io:8000 <http://local.overhang.io:8000>`_.
* CMS will be accessible at `http://studio.local.overhang.io:8001 <http://studio.local.overhang.io:8001>`_.
* Plugged-in services should be accessible at their documented URLs.

Now, use the ``tutor dev ...`` command-line interface to manage the development environment. Some common commands are described below.

.. note::

Wherever the ``[--mount=./edx-platform]`` option is present, either:

* omit it when running of the edx-platform repository built into the image, or
* substitute it with ``--mount=<path/to/edx-platform>``.

Read more about bind-mounts :ref:`below <bind_mounts>`.

Stopping the platform
---------------------

To bring down your platform's containers, simply run::
To bring down the platform's containers, simply run::

tutor dev stop


Starting the platform back up
-----------------------------

Once you have used ``launch`` once, you can start the platform in the future with the lighter-weight ``start`` command, which brings up containers but does not perform any initialization tasks::
Once first-time setup has been performed with ``launch``, the platform can be started going forward with the lighter-weight ``start -d`` command, which brings up containers *detached* (that is: in the background), but does not perform any initialization tasks::

tutor dev start -d [--mount=./edx-platform]

Or, to start with platform with containers *attached* (that is: in the foreground, the current terminal), omit the ``-d`` flag::

tutor dev start # Run platform in the same terminal ("attached")
tutor dev start -d # Or, run platform the in the background ("detached")
tutor dev start [--mount=./edx-platform]

Nonetheless, ``launch`` is idempotent, so it is always safe to run it again in the future without risk to your data. In fact, you may find it useful to use this command as a one-stop-shop for pulling images, running migrations, initializing new plugins you have enabled, and/or executing any new initialization steps that may have been introduced since you set up Tutor::
When running containers attached, stop the platform with ``Ctrl+c``, or switch to detached mode using ``Ctrl+z``.

tutor dev launch --pullimages
Finally, the platform can also be started back up with ``launch``. It will take longer than ``start``, but it will ensure that config is applied, databases are provisioned & migrated, plugins are fully initialized, and (if applicable) the bind-mounted edx-platform is set up. Notably, ``launch`` is idempotent, so it is always safe to run it again without risk to data. Including the ``--pullimages`` flag will also ensure that container images are up-to-date::

tutor dev launch [--mount=./edx-platform] --pullimages

Debugging with breakpoints
--------------------------

To debug a local edx-platform repository, add a `python breakpoint <https://docs.python.org/3/library/functions.html#breakpoint>`__ with ``breakpoint()`` anywhere in the code. Then, attach to the applicable service's container by running ``start`` (without ``-d``) followed by the service's name::

# Debugging LMS:
tutor dev start [--mount=./edx-platform] lms

# Or, debugging CMS:
tutor dev start [--mount=./edx-platform] cms

Running arbitrary commands
--------------------------

To run any command inside one of the containers, run ``tutor dev run [OPTIONS] SERVICE [COMMAND] [ARGS]...``. For instance, to open a bash shell in the LMS or CMS containers::

tutor dev run lms bash
tutor dev run cms bash
tutor dev run [--mount=./edx-platform] lms bash
tutor dev run [--mount=./edx-platform] cms bash

To open a python shell in the LMS or CMS, run::

tutor dev run lms ./manage.py lms shell
tutor dev run cms ./manage.py cms shell
tutor dev run [--mount=./edx-platform] lms ./manage.py lms shell
tutor dev run [--mount=./edx-platform] cms ./manage.py cms shell

You can then import edx-platform and django modules and execute python code.

To collect assets, you can use the ``openedx-assets`` command that ships with Tutor::
To rebuild assets, you can use the ``openedx-assets`` command that ships with Tutor::

tutor dev run lms openedx-assets build --env=dev
tutor dev run [--mount=./edx-platform] lms openedx-assets build --env=dev


.. _specialized for developer usage:
Expand Down Expand Up @@ -193,42 +229,6 @@ This override file will be loaded when running any ``tutor dev ..`` command. The
Common tasks
------------

.. _edx_platform_dev_env:

Setting up a development environment for edx-platform
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Following the instructions :ref:`above <bind_mounts>` on how to bind-mount directories from the host above, you may mount your own `edx-platform <https://github.com/openedx/edx-platform/>`__ fork in your containers by running::

tutor dev start -d --mount=/path/to/edx-platform lms

But to achieve that, you will have to make sure that your fork works with Tutor.

First of all, you should make sure that you are working off the latest release tag (unless you are running the Tutor :ref:`nightly <nightly>` branch). See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.

Then, you should run the following commands::

# Run bash in the lms container
tutor dev run --mount=/path/to/edx-platform lms bash

# Compile local python requirements
pip install --requirement requirements/edx/development.txt

# Install nodejs packages in node_modules/
npm clean-install

# Rebuild static assets
openedx-assets build --env=dev

After running all these commands, your edx-platform repository will be ready for local development. To debug a local edx-platform repository, you can then add a `python breakpoint <https://docs.python.org/3/library/functions.html#breakpoint>`__ with ``breakpoint()`` anywhere in your code and run::

tutor dev start --mount=/path/to/edx-platform lms

The default debugger is ``ipdb.set_trace``. ``PYTHONBREAKPOINT`` can be modified by setting an environment variable in the Docker imamge.

If LMS isn't running, this will start it in your terminal. If an LMS container is already running background, this command will stop it, recreate it, and attach your terminal to it. Later, to detach your terminal without stopping the container, just hit ``Ctrl+z``.


XBlock and edx-platform plugin development
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
10 changes: 10 additions & 0 deletions tutor/commands/jobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

from tutor import config as tutor_config
from tutor import env, fmt, hooks
from tutor.hooks import priorities


class DoGroup(click.Group):
Expand Down Expand Up @@ -40,6 +41,15 @@ def _add_core_init_tasks() -> None:
("mysql", env.read_core_template_file("jobs", "init", "mysql.sh"))
)
with hooks.Contexts.APP("lms").enter():
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
kdmccormick marked this conversation as resolved.
Show resolved Hide resolved
(
"lms",
env.read_core_template_file("jobs", "init", "mounted-edx-platform.sh"),
),
# If edx-platform is mounted, then we may need to perform some setup
# before other initialization scripts can be run.
priority=priorities.HIGH,
)
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
("lms", env.read_core_template_file("jobs", "init", "lms.sh"))
)
Expand Down
7 changes: 7 additions & 0 deletions tutor/templates/build/openedx/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,13 @@ RUN openedx-assets themes \
# Create a data directory, which might be used (or not)
RUN mkdir /openedx/data

# If this "canary" file is missing from a container, then that indicates that a
# local edx-platform was bind-mounted into that container, thus overwriting the
# canary. This information is useful during edx-platform initialisation.
RUN echo \
"This copy of edx-platform was built into a Docker image." \
> bindmount-canary
kdmccormick marked this conversation as resolved.
Show resolved Hide resolved

# service variant is "lms" or "cms"
ENV SERVICE_VARIANT lms
ENV DJANGO_SETTINGS_MODULE lms.envs.tutor.production
Expand Down
9 changes: 9 additions & 0 deletions tutor/templates/dev/docker-compose.jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,15 @@ x-openedx-job-service:
args:
# Note that we never build the openedx-dev image with root user ID, as it would simply fail.
APP_USER_ID: "{{ HOST_USER_ID or 1000 }}"
volumes:
# Settings & config
- ../apps/openedx/settings/lms:/openedx/edx-platform/lms/envs/tutor:ro
- ../apps/openedx/settings/cms:/openedx/edx-platform/cms/envs/tutor:ro
- ../apps/openedx/config:/openedx/config:ro
# theme files
- ../build/openedx/themes:/openedx/themes
# editable requirements
- ../build/openedx/requirements:/openedx/requirements

services:

Expand Down
26 changes: 26 additions & 0 deletions tutor/templates/jobs/init/mounted-edx-platform.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# When a new local copy of edx-platform is bind-mounted, certain build
# artifacts from the openedx image's edx-platform directory are lost.
# We regenerate them here.

if [ -f /openedx/edx-platform/bindmount-canary ] ; then
# If this file exists, then edx-platform has not been bind-mounted,
# so no build artifacts need to be regenerated.
echo "Using edx-platform from image (not bind-mount)."
echo "No extra setup is required."
exit
fi

echo "Performing additional setup for bind-mounted edx-platform."
set -x # Echo out executed lines

# Regenerate Open_edX.egg-info
pip install -e .

# Regenerate node_modules
npm clean-install

regisb marked this conversation as resolved.
Show resolved Hide resolved
# Regenerate static assets.
openedx-assets build --env=dev
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These latter two steps will be performed every time we run launch with a bind-mount. I don't have a better alternative, so let's keep it that way, but we should keep it in mind.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something we could do, eventually, would be to use a hash to ensure that these steps are not run extraneously.

  • node_modules would be easy: upon running npm install, create a node_modules/.package_lock_hash file, containing a hash of package-lock.json. If package-lock.json matches node_modules/.package_lock_hash, then we can skip node_modules installation.
  • assets would be more complicated, but still doable. The scripts/build-assets.sh has the means to a compile a list of all asset input files. Saving the combined hash of those somewhere would allow us to skip asset compilation when assets are unchanged.

Thing is, this is just reimplementing what Docker already does. That's why I'd rather move node_modules and assets outside of the bind-mount if at all possible.


set -x
echo "Done setting up bind-mounted edx-platform."