Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

e13 Preprocessing transcriptomic library encounter pipefail #68

Open
guijie2015 opened this issue Oct 23, 2024 · 4 comments
Open

e13 Preprocessing transcriptomic library encounter pipefail #68

guijie2015 opened this issue Oct 23, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@guijie2015
Copy link

Hi,when I`m running example data on the Preprocessing transcriptomic library step, I encounter the problem.

98adec7c81a075a584e7f534f283f83

Here is the log file.
log.log

I understand you're busy, but if you have some time, I would really appreciate your help with this issue.

Steps to reproduce the bug

# Sample code to reproduce the bug
ref_path=/home/data/hejunjie/16.openST_20241014/ref
spacemake init --dropseq_tools /home/data/hejunjie/16.openST_20241014/1.e13_mouse_head_test_20241014/1.spacemake/openst_e13_mouse_head_demo/bins/Drop-seq_tools-2.5.4

spacemake config add_species \
   --name mouse \
   --reference genome \
   --sequence ${ref_path}/GRCm39vM30.genome.fa \
   --annotation ${ref_path}/gencodevM30.annotation.gtf

spacemake config add_species \
   --name mouse \
   --reference rRNA \
   --sequence ${ref_path}/mouse.rRNA.fa

spacemake config add_species \
   --name mouse \
   --reference phiX \
   --sequence ${ref_path}/phiX.fa

spacemake projects add_sample --project_id openst_demo \
    --sample_id openst_demo_e13_mouse_head \
    --R1 e13_mouse_head_R1_001.fastq.gz e13_mouse_head_reseq_R1_001.fastq.gz \
    --R2 e13_mouse_head_R2_001.fastq.gz e13_mouse_head_reseq_R2_001.fastq.gz \
    --species mouse \
    --puck openst \
    --run_mode openst \
    --barcode_flavor openst \
    --puck_barcode_file e13_mouse_head_tiles/*.txt.gz \
    --map_strategy "bowtie2:phiX->bowtie2:rRNA->STAR:genome:final"

spacemake run --cores 32 --keep-going

Environment info

  • Platform:Ubuntu
  • Python version:Python 3.9.19
  • Java version:openjdk 11.0.15-internal 2022-04-19
  • python packages:Package Version
    aiobotocore 2.5.4
    aioeasywebdav 2.4.0
    aiohappyeyeballs 2.4.3
    aiohttp 3.10.10
    aioitertools 0.12.0
    aiosignal 1.3.1
    alabaster 0.7.16
    amply 0.1.6
    anndata 0.10.9
    annotated-types 0.7.0
    appdirs 1.4.4
    array_api_compat 1.9
    asciitree 0.3.3
    asttokens 2.4.1
    async-timeout 4.0.3
    attmap 0.13.2
    attrs 24.2.0
    babel 2.16.0
    bcrypt 4.2.0
    beautifulsoup4 4.12.3
    bio 1.6.2
    biopython 1.84
    biothings-client 0.3.1
    bleach 6.1.0
    boto3 1.35.39
    botocore 1.31.17
    Brotli 1.1.0
    cachetools 5.5.0
    cellpose 3.0.11
    certifi 2024.8.30
    cffi 1.17.1
    charset-normalizer 3.4.0
    click 8.1.7
    cloudpickle 3.1.0
    colorama 0.4.6
    colorcet 3.1.0
    coloredlogs 15.0.1
    comm 0.2.2
    commonmark 0.9.1
    ConfigArgParse 1.7
    connection-pool 0.0.3
    contourpy 1.3.0
    cryptography 43.0.1
    cycler 0.12.1
    dask 2024.8.0
    dask-expr 1.1.10
    dask-image 2024.5.3
    datashader 0.16.3
    datrie 0.8.2
    debugpy 1.8.7
    decorator 5.1.1
    defusedxml 0.7.1
    distributed 2024.8.0
    dnaio 1.2.2
    docrep 0.3.2
    docutils 0.21.2
    dpath 2.2.0
    dropbox 12.0.2
    eido 0.2.2
    exceptiongroup 1.2.2
    executing 2.1.0
    fasteners 0.19
    fastjsonschema 2.20.0
    fastremap 1.15.0
    filechunkio 1.8
    filelock 3.16.1
    fonttools 4.54.1
    frozenlist 1.4.1
    fsspec 2023.6.0
    ftputil 5.1.0
    geopandas 1.0.1
    get-annotations 0.1.2
    ghp-import 2.1.0
    gitdb 4.0.11
    GitPython 3.1.43
    google-api-core 2.21.0
    google-api-python-client 2.149.0
    google-auth 2.35.0
    google-auth-httplib2 0.2.0
    google-cloud-core 2.4.1
    google-cloud-storage 2.18.2
    google-crc32c 1.1.2
    google-resumable-media 2.7.2
    googleapis-common-protos 1.65.0
    gprofiler-official 1.0.0
    grpcio 1.46.3
    h5py 3.12.1
    httplib2 0.22.0
    humanfriendly 10.0
    idna 3.10
    igraph 0.11.6
    imagecodecs 2024.9.22
    imageio 2.35.1
    imagesize 1.4.1
    importlib_metadata 8.4.0
    importlib_resources 6.4.5
    inflect 7.4.0
    iniconfig 2.0.0
    ipykernel 6.29.5
    ipython 8.18.1
    isal 1.7.1
    jedi 0.19.1
    Jinja2 3.1.4
    jmespath 1.0.1
    joblib 1.4.2
    jsonschema 4.23.0
    jsonschema-specifications 2024.10.1
    jupyter_client 8.6.3
    jupyter_core 5.7.2
    jupyterlab_pygments 0.3.0
    kiwisolver 1.4.7
    kornia 0.7.3
    kornia_rs 0.1.5
    lazy_loader 0.4
    legacy-api-wrap 1.4
    leidenalg 0.10.2
    llvmlite 0.43.0
    locket 1.0.0
    logmuse 0.2.8
    Markdown 3.7
    markdown-it-py 3.0.0
    MarkupSafe 3.0.1
    matplotlib 3.6.0
    matplotlib-inline 0.1.7
    matplotlib-scalebar 0.8.1
    mdurl 0.1.2
    mergedeep 1.3.4
    mistune 3.0.2
    mkdocs 1.6.1
    mkdocs-autorefs 1.2.0
    mkdocs-get-deps 0.2.0
    mkdocs-material 9.5.40
    mkdocs-material-extensions 1.3.1
    mknotebooks 0.8.0
    more-itertools 10.5.0
    mpmath 1.3.0
    msgpack 1.1.0
    multidict 6.1.0
    multipledispatch 1.0.0
    multiscale_spatial_image 1.0.1
    munkres 1.1.4
    mygene 3.2.2
    natsort 8.4.0
    nbclient 0.10.0
    nbconvert 7.16.4
    nbformat 5.10.4
    nest-asyncio 1.6.0
    networkx 3.2.1
    novosparc 0.4.4
    numba 0.60.0
    numcodecs 0.12.1
    numpy 1.26.4
    nvidia-cublas-cu12 12.1.3.1
    nvidia-cuda-cupti-cu12 12.1.105
    nvidia-cuda-nvrtc-cu12 12.1.105
    nvidia-cuda-runtime-cu12 12.1.105
    nvidia-cudnn-cu12 9.1.0.70
    nvidia-cufft-cu12 11.0.2.54
    nvidia-curand-cu12 10.3.2.106
    nvidia-cusolver-cu12 11.4.5.107
    nvidia-cusparse-cu12 12.1.0.106
    nvidia-nccl-cu12 2.20.5
    nvidia-nvjitlink-cu12 12.6.77
    nvidia-nvtx-cu12 12.1.105
    oauth2client 4.1.3
    ome-zarr 0.9.0
    omnipath 1.0.8
    opencv-python 4.10.0.84
    opencv-python-headless 4.10.0.84
    openst 0.2.1 /home/data/hejunjie/software/openst
    packaging 24.1
    paginate 0.5.7
    pandas 2.2.3
    pandocfilters 1.5.1
    param 2.1.1
    paramiko 3.5.0
    parso 0.8.4
    partd 1.4.2
    pathspec 0.12.1
    patsy 0.5.6
    pephubclient 0.4.4
    peppy 0.40.7
    pexpect 4.9.0
    Pillow 10.0.1
    PIMS 0.7
    pip 24.2
    pkgutil_resolve_name 1.3.10
    plac 1.4.3
    platformdirs 4.3.6
    pluggy 1.5.0
    ply 3.11
    pooch 1.8.2
    POT 0.9.4
    prettytable 3.11.0
    prompt_toolkit 3.0.48
    propcache 0.2.0
    proto-plus 1.24.0
    protobuf 5.28.2
    psutil 6.0.0
    ptyprocess 0.7.0
    PuLP 2.7.0
    pure_eval 0.2.3
    pyarrow 17.0.0
    pyasn1 0.6.1
    pyasn1_modules 0.4.1
    pycparser 2.22
    pyct 0.5.0
    pydantic 2.9.2
    pydantic_core 2.23.4
    Pygments 2.18.0
    pygraphviz 1.11
    pymdown-extensions 10.11.2
    PyNaCl 1.5.0
    pynndescent 0.5.13
    pyogrio 0.10.0
    pyOpenSSL 24.2.1
    pyparsing 3.2.0
    pyproj 3.6.1
    PyQt5 5.15.9
    PyQt5-Qt5 5.15.2
    PyQt5-sip 12.12.2
    pyqtgraph 0.13.7
    pysam 0.22.1
    pysftp 0.2.9
    PySocks 1.7.1
    pytest 8.3.3
    python-dateutil 2.9.0
    python-irodsclient 2.1.0
    pytz 2024.1
    pyu2f 0.1.5
    PyWavelets 1.6.0
    PyYAML 6.0.2
    pyyaml_env_tag 0.1
    pyzmq 26.2.0
    ratelimiter 1.2.0
    recommonmark 0.7.1
    referencing 0.35.1
    regex 2024.9.11
    requests 2.32.3
    reretry 0.11.8
    rich 13.9.2
    roifile 2024.9.15
    rpds-py 0.20.0
    rsa 4.9
    s3fs 2023.6.0
    s3transfer 0.10.3
    scanpy 1.10.3
    scikit-image 0.19.3
    scikit-learn 1.5.2
    scipy 1.13.1
    seaborn 0.13.2
    session_info 1.0.0
    setuptools 75.1.0
    shapely 2.0.6
    shellingham 1.5.4
    six 1.16.0
    slacker 0.14.0
    slicerator 1.1.0
    smart_open 7.0.5
    smmap 5.0.0
    snakemake 6.3.0
    snowballstemmer 2.2.0
    sortedcontainers 2.4.0
    soupsieve 2.6
    spacemake 0.7.9
    spatial_image 1.1.0
    spatialdata 0.2.3
    Sphinx 7.4.7
    sphinxcontrib-applehelp 2.0.0
    sphinxcontrib-devhelp 2.0.0
    sphinxcontrib-htmlhelp 2.1.0
    sphinxcontrib-jsmath 1.0.1
    sphinxcontrib-qthelp 2.0.0
    sphinxcontrib-serializinghtml 2.0.0
    squidpy 1.2.1
    stack-data 0.6.3
    statsmodels 0.14.4
    stdlib-list 0.10.0
    stone 3.3.2
    stopit 1.1.2
    sympy 1.13.3
    tabulate 0.9.0
    tblib 3.0.0
    texttable 1.7.0
    threadpoolctl 3.5.0
    throttler 1.2.2
    tifffile 2024.8.30
    tinycss2 1.3.0
    tomli 2.0.2
    toolz 1.0.0
    toposort 1.10
    torch 2.4.1
    tornado 6.4.1
    tqdm 4.66.5
    traitlets 5.14.3
    triton 3.0.0
    typeguard 4.3.0
    typer 0.12.5
    typing_extensions 4.12.2
    tzdata 2024.2
    ubiquerg 0.8.0
    umap-learn 0.5.6
    unicodedata2 15.1.0
    uritemplate 4.1.1
    urllib3 1.26.19
    validators 0.34.0
    veracitools 0.1.3
    watchdog 5.0.3
    wcwidth 0.2.13
    webencodings 0.5.1
    wheel 0.44.0
    wrapt 1.16.0
    xarray 2024.7.0
    xarray-dataclasses 1.8.0
    xarray-datatree 0.0.14
    xarray-schema 0.0.3
    xarray-spatial 0.4.0
    xopen 2.0.2
    yarl 1.14.0
    yte 1.5.4
    zarr 2.18.2
    zict 3.0.0
    zipp 3.20.2
    zlib-ng 0.5.1
@guijie2015 guijie2015 added the bug Something isn't working label Oct 23, 2024
@danilexn
Copy link
Member

Hi, and thanks for the very detailed report.

We have noticed this issue in the spacemake automated_analysis, and a bugfix is in progress.

For the rest of the Open-ST workflow, these reports are not really necessary. You can make sure the rest of the pipeline runs by using --keep_going flag in spacemake.

I hope this helps, and sorry for the inconvenience!

Dani

@guijie2015
Copy link
Author

Hi, and thanks for the very detailed report.

We have noticed this issue in the spacemake automated_analysis, and a bugfix is in progress.

For the rest of the Open-ST workflow, these reports are not really necessary. You can make sure the rest of the pipeline runs by using --keep_going flag in spacemake.

I hope this helps, and sorry for the inconvenience!

Dani

Very thanksful for your reply. But I am very sorry fot that when I am trying using '--keep_going' and then running the next step Pairwise alignment. It told me I miss the dge.all.polyA_adapter_trimmed.mm_included.spatial_beads_puck_collection.h5ad file.
图片

How can I create this dge.all.polyA_adapter_trimmed.mm_included.spatial_beads_puck_collection.h5ad.

Here is my Pairwise alignment code.
openst from_spacemake --project-id openst_demo --sample-id openst_demo_e13_mouse_head merge_modalities

@danilexn
Copy link
Member

That probably means you used the default spacemake mode that runs meshing before stitching the puck_collection, therefore that file will not be available. You have to generate it using openst from_spacemake ... spatial_stitch, take a look here:

https://rajewsky-lab.github.io/openst/latest/computational/pairwise_alignment/#tissue-transcriptome

Under "My run_mode and/or puck do not look like that"

@guijie2015
Copy link
Author

spatial_stitch

Very thanksful for your reply.
I try running these code
openst from_spacemake --project-id openst_demo\ --sample-id openst_demo_e13_mouse_head\ spatial_stitch\ --tile-coordinates fc_1_coordinate_system.csv
then get this error.
图片

under my dge directory，I have these files. I can`t find "fc_1_tile_2267.h5ad dge.fc_1_tile_2268.h5ad", which is running spatial_stitch in manual on the example
图片
图片

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@guijie2015 @danilexn