Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation cleanup #3720

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Documentation cleanup #3720

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
045182a
docs cleanup
astro-friedel Dec 10, 2024
c03a33c
bug fixes
astro-friedel Dec 10, 2024
d646aaa
Merge remote-tracking branch 'upstream/master'
astro-friedel Dec 10, 2024
35fc2f5
Merge branch 'master' into cleanUpDocs
astro-friedel Dec 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,7 @@ ENV/

# emacs buffers
\#*

docs/stubs/

docs/1-parsl-introduction.ipynb
156 changes: 89 additions & 67 deletions CONTRIBUTING.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,20 @@
Where to start
--------------

We eagerly welcome contributions of any type (e.g., bug fixes, new features, reporting issues, documentation, etc). If you're looking for a good place to get started you might like to peruse our current Git issues (those marked with `help wanted <https://github.com/Parsl/parsl/labels/help%20wanted>`_ are a good place to start).
We eagerly welcome contributions of any type (e.g., bug fixes, new features, reporting issues,
documentation, etc). If you're looking for a good place to get started you might like to peruse our
current Git issues (those marked with
`help wanted <https://github.com/Parsl/parsl/labels/help%20wanted>`_ are a good place to start).

Please be aware of `Parsl's Code of Conduct <https://github.com/Parsl/parsl/blob/master/CODE_OF_CONDUCT.md>`_.
Please be aware of `Parsl's Code of Conduct <https://github.com/Parsl/parsl/blob/master/CODE_OF_CONDUCT.md>`_.

If you are not familiar with GitHub pull requests, the main mechanism to contribute changes to our code, there is `documentation available <https://opensource.com/article/19/7/create-pull-request-github>`_.
If you are not familiar with GitHub pull requests, the main mechanism to contribute changes to our
code, there is `documentation available <https://opensource.com/article/19/7/create-pull-request-github>`_.

The best places to ask questions or discuss development activities are:

* in our Slack's `#parsl-hackers channel <https://parsl-project.slack.com/archives/C02P57G6NCB>`_. You can `join our Slack here <https://join.slack.com/t/parsl-project/shared_invite/zt-4xbquc5t-Ur65ZeVtUOX51Ts~GRN6_g>`_.
* in our Slack's `#parsl-hackers channel <https://parsl-project.slack.com/archives/C02P57G6NCB>`_.
You can `join our Slack here <https://join.slack.com/t/parsl-project/shared_invite/zt-4xbquc5t-Ur65ZeVtUOX51Ts~GRN6_g>`_.

* using `GitHub issues <https://github.com/Parsl/parsl/issues>`_.

Expand All @@ -20,111 +25,119 @@ Coding conventions
Formatting conventions
======================

Parsl code should adhere to Python `PEP-8 <https://peps.python.org/pep-0008/>`_. This is enforced in CI (with some exceptions). You can also run this test yourself using ``make flake8``.
Parsl code should adhere to Python `PEP-8 <https://peps.python.org/pep-0008/>`_. This is enforced in
CI (with some exceptions). You can also run this test yourself using ``make flake8``.


Naming conventions
==================

The following convention should be followed: ClassName, ExceptionName, GLOBAL_CONSTANT_NAME, and lowercase_with_underscores for everything else.
The following convention should be followed: ClassName, ExceptionName, GLOBAL_CONSTANT_NAME, and
lowercase_with_underscores for everything else.


Version increments
==================

Parsl follows the `calendar versioning scheme <https://calver.org/#scheme>`_ with ``YYYY.MM.DD`` numbering scheme for versions.
This scheme was chosen following a switch from ad-hoc versioning and manual release processes to an automated weekly process.
Releases are pushed from github actions to PyPI and will be picked up automatically by Conda.
Manual packaging instructions are included in the
Parsl follows the `calendar versioning scheme <https://calver.org/#scheme>`_ with ``YYYY.MM.DD``
numbering scheme for versions. This scheme was chosen following a switch from ad-hoc versioning and
manual release processes to an automated weekly process.Releases are pushed from github actions to
PyPI and will be picked up automatically by Conda. Manual packaging instructions are included in the
`packaging docs <http://parsl.readthedocs.io/en/latest/devguide/packaging.html>`_


Documentation
==================
=============

Classes should be documented following the `NumPy/SciPy <https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt>`_
style. A concise summary is available `here <http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html>`_. User and developer documentation is auto-generated and made available on
Classes should be documented following the
`NumPy/SciPy <https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt>`_
style. A concise summary is available
`here <http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html>`_. User and
developer documentation is auto-generated and made available on
`ReadTheDocs <https://parsl.readthedocs.io>`_.


Testing
=======

Parsl uses ``pytest`` to run most tests. All tests should be placed in
the ``parsl/tests`` directory.
Parsl uses ``pytest`` to run most tests. All tests should be placed in the ``parsl/tests`` directory.

There are two broad groups of tests: those which must run with a
specific configuration, and those which should work with any
configuration.
There are two broad groups of tests: those which must run with a specific configuration, and those
which should work with any configuration.

Tests which should run with with any configuration live under
themed directories ``parsl/tests/test*/`` and should be named ``test*.py``.
They can be run with any configuration, by specifying ``--config CONFIGPATH``
where CONFIGPATH is a path to a ``.py`` file exporting a parsl configuration
object named ``config``. The parsl-specific test fixtures will ensure
a suitable DFK is loaded with that configuration for each test.
Tests which should run with with any configuration live under themed directories
``parsl/tests/test*/`` and should be named ``test*.py``. They can be run with any configuration, by
specifying ``--config CONFIGPATH`` where CONFIGPATH is a path to a ``.py`` file exporting a parsl
configuration object named ``config``. The parsl-specific test fixtures will ensure a suitable DFK
is loaded with that configuration for each test.

Tests which require their own specially configured DFK, or no DFK at all,
should be labelled with ``@pytest.mark.local`` and can be run with
``--config local``.
Provide the special configuration creating a ``local_config`` function
that returns the required configuration in that test file.
Or, provide both a ``local_setup`` function that loads the proper configuration
and ``local_teardown`` that stops parsl.
Tests which require their own specially configured DFK, or no DFK at all, should be labelled with
``@pytest.mark.local`` and can be run with ``--config local``. Provide the special configuration
creating a ``local_config`` function that returns the required configuration in that test file. Or,
provide both a ``local_setup`` function that loads the proper configuration and ``local_teardown``
that stops parsl.

There is more fine-grained enabling and disabling of tests within the
above categories:
There is more fine-grained enabling and disabling of tests within the above categories:

A pytest marker of ``cleannet`` (for clean network) can be used to select
or deselect tests which need a very clean network (for example, for tests
making FTP transfers). When the test environment (github actions) does not
provide a sufficiently clean network, run all tests with ``-k "not cleannet"`` to
disable those tests.
A pytest marker of ``cleannet`` (for clean network) can be used to select or deselect tests which
need a very clean network (for example, for tests making FTP transfers). When the test environment
(github actions) does not provide a sufficiently clean network, run all tests with
``-k "not cleannet"`` to disable those tests.

Some other markers are available but unused in testing;
see ``pytest --markers parsl/tests/`` for more details.
Some other markers are available but unused in testing; see ``pytest --markers parsl/tests/`` for
more details.

A specific test in a specific file can be run like this:::

$ pytest test_python_apps/test_basic.py::test_simple

A timeout can be added to test runs using a pytest parameter such as
``--timeout=60``
A timeout can be added to test runs using a pytest parameter such as ``--timeout=60``

Many tests are marked with ``@pytest.mark.skip`` for reasons usually
specified directly in the annotation - generally because they are broken
in one way or another.
Many tests are marked with ``@pytest.mark.skip`` for reasons usually specified directly in the
annotation - generally because they are broken in one way or another.


Coverage testing
================

There is also some coverage testing available. The CI by default records
coverage for most of the tests that it runs and outputs a brief report
at the end of each CI run. This is purely informational and a Lack of
coverage won't produce a CI failure.
There is also some coverage testing available. The CI by default records coverage for most of the
tests that it runs and outputs a brief report at the end of each CI run. This is purely
informational and a Lack of coverage won't produce a CI failure.

It is possible to produce a more detailed coverage report on your development machine: make sure you
have no `.coverage` file, run the test commands as shown in `.github/workflows/ci.yaml`, and then
run `coverage report` to produce the summary as seen in CI, or run `coverage html` to produce
annotated source code in the `htmlcov/` subdirectory. This will show, line by line, if each line of
parsl source code was executed during the coverage test.

It is possible to produce a more detailed coverage report on your
development machine: make sure you have no `.coverage` file, run the
test commands as shown in `.github/workflows/ci.yaml`, and then run
`coverage report` to produce the summary as seen in CI, or run
`coverage html` to produce annotated source code in the `htmlcov/`
subdirectory. This will show, line by line, if each line of parsl
source code was executed during the coverage test.

Development Process
-------------------

If you are a contributor to Parsl at large, we recommend forking the repository and submitting pull requests from your fork.
The `Parsl development team <https://github.com/orgs/Parsl/teams>`_ has the additional privilege of creating development branches on the repository.
Parsl development follows a common pull request-based workflow similar to `GitHub flow <http://scottchacon.com/2011/08/31/github-flow.html>`_. That is:
If you are a contributor to Parsl at large, we recommend forking the repository and submitting pull
requests from your fork. The `Parsl development team <https://github.com/orgs/Parsl/teams>`_ has the
additional privilege of creating development branches on the repository. Parsl development follows a
common pull request-based workflow similar to `GitHub flow <http://scottchacon.com/2011/08/31/github-flow.html>`_.
That is:

* every development activity (except very minor changes, which can be discussed in the PR) should have a related GitHub issue
* all development occurs in branches (named with a short descriptive name, for example, `add-globus-transfer-#1`)
* every development activity (except very minor changes, which can be discussed in the PR) should
have a related GitHub issue
* all development occurs in branches (named with a short descriptive name, for example,
`add-globus-transfer-#1`)
* the master branch is always stable
* development branches should include tests for added features
* development branches should be tested after being brought up-to-date with the master (in this way, what is being tested is what is actually going into the code; otherwise unexpected issues from merging may come up)
* development branches should be tested after being brought up-to-date with the master (in this way,
what is being tested is what is actually going into the code; otherwise unexpected issues from
merging may come up)
* branches what have been successfully tested are merged via pull requests (PRs)
* PRs should be used for review and discussion
* PRs should be reviewed in a timely manner, to reduce effort keeping them synced with other changes happening on the master branch
* PRs should be reviewed in a timely manner, to reduce effort keeping them synced with other changes
happening on the master branch

Git commit messages should include a single summary sentence followed by a more explanatory
paragraph. Note: all commit messages should reference the GitHub issue to which they relate. A nice
discussion on the topic can be found `here <https://chris.beams.io/posts/git-commit/>`_.

Git commit messages should include a single summary sentence followed by a more explanatory paragraph. Note: all commit messages should reference the GitHub issue to which they relate. A nice discussion on the topic can be found `here <https://chris.beams.io/posts/git-commit/>`_.
::
Implemented Globus data staging support

Expand All @@ -134,22 +147,31 @@ Git commit messages should include a single summary sentence followed by a more
destination Parsl will use the Globus transfer service to move data to the
compute host. Fixes #-1.


Git hooks
---------

Developers may find it useful to setup a pre-commit git hook to automatically lint and run tests. This is a script which is run before each commit. For example::
Developers may find it useful to setup a pre-commit git hook to automatically lint and run tests.
This is a script which is run before each commit. For example::

$ cat ~/parsl/.git/hooks/pre-commit
#!/bin/sh

make lint flake8 mypy local_thread_test


Project documentation
---------------------

All project documentation is written in reStructuredText. `Sphinx <http://sphinx-doc.org/>`_ is used to generate the HTML documentation from the rst documentation and structured docstrings in Parsl code. Project documentation is built automatically and added to the `Parsl documentation <https://parsl.readthedocs.io>`_.
All project documentation is written in reStructuredText. `Sphinx <http://sphinx-doc.org/>`_ is used
to generate the HTML documentation from the rst documentation and structured docstrings in Parsl
code. Project documentation is built automatically and added to the
`Parsl documentation <https://parsl.readthedocs.io>`_.


Credit and Contributions
------------------------

Parsl wants to make sure that all contributors get credit for their contributions. When you make your first contribution, it should include updating the codemeta.json file to include yourself as a contributor to the project.
Parsl wants to make sure that all contributors get credit for their contributions. When you make
your first contribution, it should include updating the codemeta.json file to include yourself as a
contributor to the project.
34 changes: 22 additions & 12 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ Parsl extends parallelism in Python beyond a single computer.

You can use Parsl
`just like Python's parallel executors <https://parsl.readthedocs.io/en/stable/userguide/workflow.html#parallel-workflows-with-loops>`_
but across *multiple cores and nodes*.
However, the real power of Parsl is in expressing multi-step workflows of functions.
Parsl lets you chain functions together and will launch each function as inputs and computing resources are available.
but across *multiple cores and nodes*. However, the real power of Parsl is in expressing multi-step
workflows of functions. Parsl lets you chain functions together and will launch each function as
inputs and computing resources are available.

.. code-block:: python

Expand Down Expand Up @@ -37,8 +37,10 @@ Parsl lets you chain functions together and will launch each function as inputs
assert future.result() == 7


Start with the `configuration quickstart <https://parsl.readthedocs.io/en/stable/quickstart.html#getting-started>`_ to learn how to tell Parsl how to use your computing resource,
then explore the `parallel computing patterns <https://parsl.readthedocs.io/en/stable/userguide/workflow.html>`_ to determine how to use parallelism best in your application.
Start with the `configuration quickstart <https://parsl.readthedocs.io/en/stable/quickstart.html#getting-started>`_
to learn how to tell Parsl how to use your computing resource, then explore the
`parallel computing patterns <https://parsl.readthedocs.io/en/stable/userguide/workflow.html>`_ to
determine how to use parallelism best in your application.

.. |licence| image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg
:target: https://github.com/Parsl/parsl/blob/master/LICENSE
Expand Down Expand Up @@ -68,7 +70,7 @@ then explore the `parallel computing patterns <https://parsl.readthedocs.io/en/s
:target: https://numfocus.org
:alt: Powered by NumFOCUS


Quickstart
==========

Expand All @@ -80,16 +82,20 @@ To run the Parsl tutorial notebooks you will need to install Jupyter::

$ pip3 install jupyter

Detailed information about setting up Jupyter with Python is available `here <https://jupyter.readthedocs.io/en/latest/install.html>`_
Detailed information about setting up Jupyter with Python is available
`here <https://jupyter.readthedocs.io/en/latest/install.html>`_

Note: Parsl uses an opt-in model to collect usage statistics for reporting and improvement purposes. To understand what stats are collected and enable collection please refer to the `usage tracking guide <http://parsl.readthedocs.io/en/stable/userguide/usage_tracking.html>`__
Note: Parsl uses an opt-in model to collect usage statistics for reporting and improvement purposes.
To understand what stats are collected and enable collection please refer to the
`usage tracking guide <http://parsl.readthedocs.io/en/stable/userguide/usage_tracking.html>`__

Documentation
=============

The complete parsl documentation is hosted `here <http://parsl.readthedocs.io/en/stable/>`_.

The Parsl tutorial is hosted on live Jupyter notebooks `here <https://mybinder.org/v2/gh/Parsl/parsl-tutorial/master>`_
The Parsl tutorial is hosted on live Jupyter notebooks
`here <https://mybinder.org/v2/gh/Parsl/parsl-tutorial/master>`_


For Developers
Expand Down Expand Up @@ -120,14 +126,18 @@ For Developers
Requirements
============

Parsl is supported in Python 3.9+. Requirements can be found `here <requirements.txt>`_. Requirements for running tests can be found `here <test-requirements.txt>`_.
Parsl is supported in Python 3.9+. Requirements can be found `here <requirements.txt>`_.
Requirements for running tests can be found `here <test-requirements.txt>`_.

Code of Conduct
===============

Parsl seeks to foster an open and welcoming environment - Please see the `Parsl Code of Conduct <https://github.com/Parsl/parsl?tab=coc-ov-file#parsl-code-of-conduct>`_ for more details.
Parsl seeks to foster an open and welcoming environment - Please see the
`Parsl Code of Conduct <https://github.com/Parsl/parsl?tab=coc-ov-file#parsl-code-of-conduct>`_ for
more details.

Contributing
============

We welcome contributions from the community. Please see our `contributing guide <https://github.com/Parsl/parsl/blob/master/CONTRIBUTING.rst>`_.
We welcome contributions from the community. Please see our
`contributing guide <https://github.com/Parsl/parsl/blob/master/CONTRIBUTING.rst>`_.
4 changes: 2 additions & 2 deletions docs/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ help:
@echo " epub to make an epub"
@echo " epub3 to make an epub3"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
@echo " latexpdf to make LaTeX files and run them through pdflatex (currently does not work)"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx (currently does not work)"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
Expand Down
4 changes: 2 additions & 2 deletions docs/devguide/packaging.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ Steps to release
* ``parsl/README.rst``

3. Commit and push the changes to github
4. Run the ``tag_and_release.sh`` script. This script will verify that
version number matches the version specified.
4. Run the ``tag_and_release.sh`` script. This script will verify that version number matches the
version specified.

.. code:: bash

Expand Down
Loading
Loading