Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve error message for bad looper config #515

Open
nsheff opened this issue Aug 22, 2024 · 6 comments
Open

Improve error message for bad looper config #515

nsheff opened this issue Aug 22, 2024 · 6 comments
Assignees
Milestone

Comments

@nsheff
Copy link
Contributor

nsheff commented Aug 22, 2024

The looper init constructed a bad looper config (see #514), which looks like this.

cat .looper.yaml 
pep_config: metadata/sample_table.csv
output_dir: results
pipeline_interfaces:
- - pipeline/pipeline_interface.yaml

When I try to run a looper command, like looper run, I would expect it would say something like "Looper config is not formatted correctly" so that I know where to look for the error. Instead, I get this error:

looper run
Looper version: 2.0.0a1
Command: run
Traceback (most recent call last):
  File "/home/nsheff/.local/bin/looper", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/cli_pydantic.py", line 352, in main
    return run_looper(args, parser, test_args=test_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/cli_pydantic.py", line 158, in run_looper
    looper_config_dict = read_looper_dotfile()
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/utils.py", line 527, in read_looper_dotfile
    return read_looper_config_file(looper_config_path=dot_file_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/utils.py", line 807, in read_looper_config_file
    pipeline_types, piface_path = determine_pipeline_type(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/utils.py", line 733, in determine_pipeline_type
    piface_path = expandpath(piface_path)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/ubiquerg/paths.py", line 21, in expandpath
    return os.path.expandvars(os.path.expanduser(path))
                              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 259, in expanduser
TypeError: expected str, bytes or os.PathLike object, not list
@nsheff
Copy link
Contributor Author

nsheff commented Aug 22, 2024

Can you also check the messaging for other possible common mistakes, like:

  • - what if the pipeline interface is malformed?
  • - what if pipeline interface is missing?
  • - what if the PEP is malformed?
  • - what if PEP is missing?
  • - what if looper config file is not found?
  • - what if the data file is not found, (path is wrong) and the pipeline fails?
  • - any other common mistake you could think

To test these, just mess up the file intentionally and then run looper run, and see what the error message is. You probably just need to do a better job catching errors and giving better messages in these common cases.

@nsheff
Copy link
Contributor Author

nsheff commented Aug 22, 2024

here's an error I'm getting if the pipeline interface is malformed:

Using looper config (.looper.yaml).
Traceback (most recent call last):
  File "/home/nsheff/.local/bin/looper", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/cli_pydantic.py", line 352, in main
    return run_looper(args, parser, test_args=test_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/cli_pydantic.py", line 201, in run_looper
    p = Project(
        ^^^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/project.py", line 142, in __init__
    self._samples_by_interface = self._samples_by_piface(self.piface_key)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/project.py", line 687, in _samples_by_piface
    PipelineInterface(
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/pipeline_interface.py", line 59, in __init__
    self._expand_paths(["compute", "dynamic_variables_script_path"])
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/pipeline_interface.py", line 321, in _expand_paths
    raw_path = _get_from_dict(self, keys)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsheff/.local/lib/python3.12/site-packages/looper/pipeline_interface.py", line 299, in _get_from_dict
    map = map[a]
          ~~~^^^
TypeError: string indices must be integers, not 'str'

@donaldcampbelljr
Copy link
Contributor

donaldcampbelljr commented Aug 23, 2024

Currently if the pipeline interface is malformed or the path doesn't exist looper marks it as so:

1 unique reasons for submission failure: No pipeline interfaces defined

Summary of failures:
No pipeline interfaces defined: frog_1, frog_2

Should we change this to more specifically state that it could not find the given interface and show the user provided path?

Update: I added some better warnings for the above case and if the path is missing.

@donaldcampbelljr
Copy link
Contributor

I've added better warnings for certain scenarios regarding looper config, pifaces, and PEPs.

@donaldcampbelljr
Copy link
Contributor

For the data path, the user will be given info that something is wrong at time of job submission by the invoked pipeline.

Some simple examples in the case of a missing or malformed path to data:

## [1 of 2] sample: frog_1; pipeline: count_lines
Writing script to /home/drc/GITHUB/hello_looper/hello_looper/minimal/results/submission/count_lines_frog_1.sub
Job script (n=1; 0.00Gb): /home/drc/GITHUB/hello_looper/hello_looper/minimal/results/submission/count_lines_frog_1.sub
wc: data/frog_1.txt: No such file or directory
## [2 of 2] sample: frog_2; pipeline: count_lines
Writing script to /home/drc/GITHUB/hello_looper/hello_looper/minimal/results/submission/count_lines_frog_2.sub
Job script (n=1; 0.00Gb): /home/drc/GITHUB/hello_looper/hello_looper/minimal/results/submission/count_lines_frog_2.sub
/bin/sh: 7: /home/drc/GITHUB/hello_looper/hello_looper/minimal/results/submission/count_lines_frog_2.sub: Syntax error: "(" unexpected (expecting "}")

sample annotation for reference (where data folder does not exist):

sample_name,library,file,toggle
frog_1,anySampleType,data/frog_1.txt,1
frog_2,anySampleType,@(data/frog_2.txt,1

@nsheff nsheff added this to the v2.0.0 milestone Aug 27, 2024
@donaldcampbelljr
Copy link
Contributor

I've addressed these items directly by adding better handling with the exception of the missing data path (which is handled by the pipeline in the above examples). I'll mark this as likely solved for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants