Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "docs: modernize py dependencies docs and example" #32382

Merged
merged 1 commit into from
Aug 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@

import pytest

from apache_beam.examples.complete.juliaset.src.juliaset import juliaset
from apache_beam.examples.complete.juliaset.juliaset import juliaset
from apache_beam.testing.util import open_shards


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
import pytest
from hamcrest.core.core.allof import all_of

from apache_beam.examples.complete.juliaset.src.juliaset import juliaset
from apache_beam.examples.complete.juliaset.juliaset import juliaset
from apache_beam.io.filesystems import FileSystems
from apache_beam.runners.runner import PipelineState
from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
Expand Down
33 changes: 0 additions & 33 deletions sdks/python/apache_beam/examples/complete/juliaset/pyproject.toml

This file was deleted.

26 changes: 18 additions & 8 deletions sdks/python/apache_beam/examples/complete/juliaset/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,14 @@
# limitations under the License.
#

"""setup.py module for the pipeline package.
"""Setup.py module for the workflow's worker utilities.

In this example, the pipeline code is gathered in a package that can be built
as source distribution and installed on the workers. The package is defined
in the pyproject.toml file. You can use setup.py file for defining
configuration that needs to be determined programatically, for example,
custom commands to run when a package is installed.
All the workflow related code is gathered in a package that will be built as a
source distribution, staged in the staging area for the workflow being run and
then installed in the workers when they start running.

You can install this package into the workers at runtime by using
the --setup_file pipeline option.
This behavior is triggered by specifying the --setup_file command line option
when running the workflow for remote execution.
"""

# pytype: skip-file
Expand Down Expand Up @@ -109,7 +107,19 @@ def run(self):
self.RunCustomCommand(command)


# Configure the required packages and scripts to install.
# Note that the Python Dataflow containers come with numpy already installed
# so this dependency will not trigger anything to be installed unless a version
# restriction is specified.
REQUIRED_PACKAGES = [
'numpy',
]

setuptools.setup(
name='juliaset',
version='0.0.1',
description='Julia set workflow package.',
install_requires=REQUIRED_PACKAGES,
packages=setuptools.find_packages(),
cmdclass={
# Command class instantiated and run during pip install scenarios.
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -95,53 +95,43 @@ If your pipeline uses packages that are not available publicly (e.g. packages th

Often, your pipeline code spans multiple files. To run your project remotely, you must group these files as a Python package and specify the package when you run your pipeline. When the remote workers start, they will install your package. To group your files as a Python package and make it available remotely, perform the following steps:

1. Create a [pyproject.toml](https://packaging.python.org/en/latest/tutorials/packaging-projects/) file for your project. The following is a very basic `pyproject.toml` file.
1. Create a [setup.py](https://pythonhosted.org/an_example_pypi_project/setuptools.html) file for your project. The following is a very basic `setup.py` file.

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "PACKAGE-NAME"
version = "PACKAGE-VERSION"
dependencies = [
# List Python packages your pipeline depends on.
]

2. If your package requires if some programmatic configuration, or you need to use the `--setup_file` pipeline option, create a setup.py file for your project.

# Note that the package can be completely defined by pyproject.toml.
# This file is optional.
import setuptools
setuptools.setup()

3. Structure your project so that the root directory contains the `pyproject.toml`, the `setup.py` file, and a `src/` directory with the rest of the files. For example:
setuptools.setup(
name='PACKAGE-NAME',
version='PACKAGE-VERSION',
install_requires=[
# List Python packages your pipeline depends on.
],
packages=setuptools.find_packages(),
)

2. Structure your project so that the root directory contains the `setup.py` file, the main workflow file, and a directory with the rest of the files, for example:

root_dir/
pyproject.toml
setup.py
src/
main.py
my_package/
my_pipeline_launcher.py
my_custom_dofns_and_transforms.py
other_utils_and_helpers.py
main.py
my_package/
my_pipeline_launcher.py
my_custom_dofns_and_transforms.py
other_utils_and_helpers.py

See [Juliaset](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/juliaset) for an example that follows this project structure.

4. Install your package in the submission environment, for example by using the following command:
3. Install your package in the submission environment, for example by using the following command:

pip install -e .

5. If you use a [custom container](#custom-containers), copy and install the package in the container as well.

6. Run your pipeline with the following command-line option:
4. Run your pipeline with the following command-line option:

--setup_file /path/to/setup.py

**Note:** It is not necessary to supply the `--requirements_file` [option](#pypi-dependencies) if the dependencies of your package are defined in the
`dependencies` field of the `pyproject.toml` file (see step 1). However unlike with the `--requirements_file` option, when you use the `--setup_file` option, Beam doesn't stage the dependent packages to the runner.
Only the pipeline package is staged. If they aren't already provided in the runtime environment, the package dependencies are installed from PyPI at runtime.
**Note:** It is not necessary to supply the `--requirements_file` [option](#pypi-dependencies) if the dependencies of your package are defined in the `install_requires` field of the `setup.py` file (see step 1).
However unlike with the `--requirements_file` option, when you use the `--setup_file` option, Beam doesn't stage the dependent packages to the runner.
Only the pipeline package is staged. If they aren't already provided in the runtime environment,
the package dependencies are installed from PyPI at runtime.


## Non-Python Dependencies or PyPI Dependencies with Non-Python Dependencies {#nonpython}
Expand Down
Loading