Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebuild for jpegturbo #303

Conversation

regro-cf-autotick-bot
Copy link
Contributor

@regro-cf-autotick-bot regro-cf-autotick-bot commented Mar 11, 2023

Build checklist

It was decided that absl and grpc migration are out of scope here.

This PR has been triggered in an effort to update jpeg_to_libjpeg_turbo.

Notes and instructions for merging this PR:

  1. Please merge the PR only after the tests have passed.
  2. Feel free to push to the bot's branch to update this PR if needed.

Please note that if you close this PR we presume that the feedstock has been rebuilt, so if you are going to perform the rebuild yourself don't close this PR until the your rebuild has been merged.

If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase @conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. Feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/cf-scripts/actions/runs/4389141418, please use this URL for debugging.

@conda-forge-webservices
Copy link
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

  • It looks like the 'tensorflow-base' output doesn't have any tests.

@hmaarrfk
Copy link
Contributor

@conda-forge-admin please rerender

@hmaarrfk hmaarrfk changed the title Rebuild for jpegturbo migration Rebuild for jpegturbo and libgrpc migration Mar 11, 2023
@hmaarrfk hmaarrfk force-pushed the rebuild-jpeg_to_libjpeg_turbo-0-3_h5ee88f branch from bd11818 to a95b3e9 Compare March 11, 2023 14:15
@hmaarrfk
Copy link
Contributor

@h-vetinari it seems the migrator is stuck with an unsolvable environment here. I manually added the migration and can recreate. Any idea?

conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {MatchSpec("libabseil==20230125.0[build=cxx17*]"), MatchSpec("libgrpc==1.52.0=hbc085c6_0")}

@hmaarrfk hmaarrfk changed the title Rebuild for jpegturbo and libgrpc migration Rebuild for jpegturbo and libgrpc 1.52 and libabseil20230125 migration Mar 11, 2023
@hmaarrfk hmaarrfk force-pushed the rebuild-jpeg_to_libjpeg_turbo-0-3_h5ee88f branch from a95b3e9 to c7ac4d0 Compare March 11, 2023 14:29
@hmaarrfk hmaarrfk mentioned this pull request Mar 11, 2023
5 tasks
@h-vetinari
Copy link
Member

Please include the libgrpc migration as well :)

@h-vetinari
Copy link
Member

Ah, just saw you did that already. Once it's got jpeg, abseil and grpc, we should be good IMO

@hmaarrfk
Copy link
Contributor

Thanks for checking in h-vetinari, @ngam if you can build OSX, i can start to build linux.

@hmaarrfk
Copy link
Contributor

hmm i'm getting a failure like:

/home/conda/feedstock_root/build_artifacts/tensorflow-split_1678570433796/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/11.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: debug_service.grpc.pb.cc:(.text._ZN4grpc8internal30ClientCallbackReaderWriterImplIN10tensorflow5EventENS2_10EventReplyEE9StartCallEv[_ZN4grpc8internal30ClientCallbackReaderWriterImplIN10tensorflow5EventENS2_10EventReplyEE9StartCallEv]+0xf7): undefined reference to `absl::lts_20230125::Mutex::Unlock()'
/home/conda/feedstock_root/build_artifacts/tensorflow-split_1678570433796/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/11.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: bazel-out/k8-opt/bin/tensorflow/core/distributed_runtime/rpc/libgrpc_master_service_impl.pic.a(grpc_master_service_impl.pic.o): in function `grpc::CompletionQueue::CompletionQueue(grpc_completion_queue_attributes const&)':
grpc_master_service_impl.cc:(.text._ZN4grpc15CompletionQueueC2ERK32grpc_completion_queue_attributes[_ZN4grpc15CompletionQueueC5ERK32grpc_completion_queue_attributes]+0xaa): undefined reference to `absl::lts_20230125::Mutex::~Mutex()'

linux_64_c_compiler_version11cuda_compiler_versionNonecudnnundefinedcxx_compiler_version11numpy1.20python3.8.____cpython-log.txt

@ngam
Copy link
Contributor

ngam commented Mar 12, 2023

hmmm... any idea what the error means exactly? I will start building soon on osx

@hmaarrfk
Copy link
Contributor

I think we are not linking correctly on abseil for some new feature. but i couldn't find a relevant issue on tensorflow or a relevant release note. i'm hoping h-vetinari has an idea ^_^

@Tobias-Fischer
Copy link
Contributor

It seems that as there is no osx_arm64 CI, the mapping osx_arm64: osx_arm64 does not generate the yaml files: conda-forge/conda-smithy#1546

I'm not sure what the alternative is for local native builds.

@hmaarrfk
Copy link
Contributor

Maybe try to manually edit the ci files? We could likely add a local provider to smithy. But "minor" edits to the ci files is likely ok for this exceptional feedstock.

@Tobias-Fischer
Copy link
Contributor

Tobias-Fischer commented Mar 20, 2023

It seems like this did the job:

diff --git a/recipe/meta.yaml b/recipe/meta.yaml
index 899c02c..33b7f9f 100644
--- a/recipe/meta.yaml
+++ b/recipe/meta.yaml
@@ -41,7 +41,7 @@ requirements:
     - {{ compiler('cxx') }}
     - {{ compiler('cuda') }}                 # [cuda_compiler_version != "None"]
     - bazel >=5.2.0,<6
-    - libgrpc
+    - libgrpc {{ libgrpc }}
     - libprotobuf
     - nasm
     - sed

I think the pinning from the ci_support files only applies to host, but not build. In this case, this leads to a conflict as per #303 (comment)

@hmaarrfk
Copy link
Contributor

This is really strange. It used to apply to both in the past. I think..... This is what we used to do to ensure that openssl was correctly pinned in the past.

Did you get through a full build?

@Tobias-Fischer
Copy link
Contributor

My M1 isn't that fast, I'll keep you posted :)

@ngam
Copy link
Contributor

ngam commented Mar 20, 2023

My M1 isn't that fast, I'll keep you posted :)

I have two super-spec’ed M1 Max macs, I can give this a try soon.

@Tobias-Fischer does it work for you natively or only emulated?

@Tobias-Fischer
Copy link
Contributor

Go for it, would be nice if you try. I build emulated; haven't tried natively.

@ngam
Copy link
Contributor

ngam commented Mar 20, 2023

Natively never worked for me :( but @xhochy used to build natively

@ngam
Copy link
Contributor

ngam commented Mar 20, 2023

Testing in #306

@ngam
Copy link
Contributor

ngam commented Mar 21, 2023

Unfortunately, the patch in #306 failed. Maybe we could attempt to build with hardcoded pins on all build/host/run?

@Tobias-Fischer
Copy link
Contributor

What was the error you got?

@ngam
Copy link
Contributor

ngam commented Mar 21, 2023

Same as above:

The reported errors are:
- Encountered problems while solving:
-   - package tensorflow-base-2.11.0-cpu_py310h760b059_0 requires libgrpc >=1.51.1,<1.52.0a0, but none of the providers can be installed
- 

@ngam
Copy link
Contributor

ngam commented Mar 21, 2023

Btw @hmaarrfk could you please look at this closely?

Am I seeing it correctly that py310 is mixed up with py38? This is supposed to be building py38.

The following NEW packages will be INSTALLED:

    absl-py:                 1.4.0-pyhd8ed1ab_0           conda-forge
    aiohttp:                 3.8.4-py38hb991d35_0         conda-forge
    aiosignal:               1.3.1-pyhd8ed1ab_0           conda-forge
    astunparse:              1.6.3-pyhd8ed1ab_0           conda-forge
    async-timeout:           4.0.2-pyhd8ed1ab_0           conda-forge
    attrs:                   22.2.0-pyh71513ae_0          conda-forge
    blinker:                 1.5-pyhd8ed1ab_0             conda-forge
    brotlipy:                0.7.0-py38hb991d35_1005      conda-forge
    bzip2:                   1.0.8-h3422bc3_4             conda-forge
    c-ares:                  1.18.1-h3422bc3_0            conda-forge
    ca-certificates:         2022.12.7-h4653dfc_0         conda-forge
    cached-property:         1.5.2-hd8ed1ab_1             conda-forge
    cached_property:         1.5.2-pyha770c72_1           conda-forge
    cachetools:              5.3.0-pyhd8ed1ab_0           conda-forge
    certifi:                 2022.12.7-pyhd8ed1ab_0       conda-forge
    cffi:                    1.15.1-py38ha45ccd6_3        conda-forge
    charset-normalizer:      2.1.1-pyhd8ed1ab_0           conda-forge
    click:                   8.1.3-unix_pyhd8ed1ab_2      conda-forge
    cryptography:            39.0.2-py38h23f6d3d_0        conda-forge
    flatbuffers:             23.3.3-hb7217d7_0            conda-forge
    frozenlist:              1.3.3-py38hb991d35_0         conda-forge
    gast:                    0.4.0-pyh9f0ad1d_0           conda-forge
    giflib:                  5.2.1-h1a8c8d9_3             conda-forge
    google-auth:             2.16.2-pyh1a96a4e_0          conda-forge
    google-auth-oauthlib:    0.4.6-pyhd8ed1ab_0           conda-forge
    google-pasta:            0.2.0-pyh8c360ce_0           conda-forge
    grpcio:                  1.51.1-py38h171e7b7_1        conda-forge
    h5py:                    3.8.0-nompi_py38hd3ff8df_101 conda-forge
    hdf5:                    1.14.0-nompi_h6b85c65_103    conda-forge
    icu:                     70.1-h6b3803e_0              conda-forge
    idna:                    3.4-pyhd8ed1ab_0             conda-forge
    importlib-metadata:      6.1.0-pyha770c72_0           conda-forge
    keras:                   2.11.0-pyhd8ed1ab_0          conda-forge
    keras-preprocessing:     1.1.2-pyhd8ed1ab_0           conda-forge
    krb5:                    1.20.1-h69eda48_0            conda-forge
    libabseil:               20220623.0-cxx17_h28b99d4_6  conda-forge
    libaec:                  1.0.6-hb7217d7_1             conda-forge
    libblas:                 3.9.0-16_osxarm64_openblas   conda-forge
    libcblas:                3.9.0-16_osxarm64_openblas   conda-forge
    libcurl:                 7.88.1-h9049daf_1            conda-forge
    libcxx:                  15.0.7-h75e25f2_0            conda-forge
    libedit:                 3.1.20191231-hc8eb9b7_2      conda-forge
    libev:                   4.33-h642e427_1              conda-forge
    libffi:                  3.4.2-h3422bc3_5             conda-forge
    libgfortran:             5.0.0-12_2_0_hd922786_31     conda-forge
    libgfortran5:            12.2.0-h0eea778_31           conda-forge
    libgrpc:                 1.51.1-hb15be72_1            conda-forge
    libjpeg-turbo:           2.1.5.1-h1a8c8d9_0           conda-forge
    liblapack:               3.9.0-16_osxarm64_openblas   conda-forge
    libnghttp2:              1.52.0-hae82a92_0            conda-forge
    libopenblas:             0.3.21-openmp_hc731615_3     conda-forge
    libpng:                  1.6.39-h76d750c_0            conda-forge
    libprotobuf:             3.21.12-hb5ab8b9_0           conda-forge
    libsqlite:               3.40.0-h76d750c_0            conda-forge
    libssh2:                 1.10.0-h7a5bd25_3            conda-forge
    libzlib:                 1.2.13-h03a7124_4            conda-forge
    llvm-openmp:             16.0.0-h7cfbb63_0            conda-forge
    markdown:                3.4.1-pyhd8ed1ab_0           conda-forge
    markupsafe:              2.1.2-py38hb991d35_0         conda-forge
    multidict:               6.0.4-py38hb991d35_0         conda-forge
    ncurses:                 6.3-h07bb92c_1               conda-forge
    numpy:                   1.24.2-py38hac8ee1c_0        conda-forge
    oauthlib:                3.2.2-pyhd8ed1ab_0           conda-forge
    openssl:                 3.1.0-h03a7124_0             conda-forge
    opt_einsum:              3.3.0-pyhd8ed1ab_1           conda-forge
    packaging:               23.0-pyhd8ed1ab_0            conda-forge
    pip:                     23.0.1-pyhd8ed1ab_0          conda-forge
    platformdirs:            3.1.1-pyhd8ed1ab_0           conda-forge
    pooch:                   1.7.0-pyhd8ed1ab_0           conda-forge
    protobuf:                4.21.12-py38h2b1e499_0       conda-forge
    pyasn1:                  0.4.8-py_0                   conda-forge
    pyasn1-modules:          0.2.7-py_0                   conda-forge
    pycparser:               2.21-pyhd8ed1ab_0            conda-forge
    pyjwt:                   2.6.0-pyhd8ed1ab_0           conda-forge
    pyopenssl:               23.0.0-pyhd8ed1ab_0          conda-forge
    pysocks:                 1.7.1-pyha2e5f31_6           conda-forge
    python:                  3.8.16-h3ba56d0_1_cpython    conda-forge
    python-flatbuffers:      23.1.21-pyhd8ed1ab_0         conda-forge
    python_abi:              3.8-3_cp38                   conda-forge
    pyu2f:                   0.1.5-pyhd8ed1ab_0           conda-forge
    re2:                     2023.02.01-hb7217d7_0        conda-forge
    readline:                8.1.2-h46ed386_0             conda-forge
    requests:                2.28.2-pyhd8ed1ab_0          conda-forge
    requests-oauthlib:       1.3.1-pyhd8ed1ab_0           conda-forge
    rsa:                     4.9-pyhd8ed1ab_0             conda-forge
    scipy:                   1.10.1-py38h7b4f323_0        conda-forge
    setuptools:              67.6.0-pyhd8ed1ab_0          conda-forge
    six:                     1.16.0-pyh6c4a22f_0          conda-forge
    snappy:                  1.1.10-h17c5cce_0            conda-forge
    tensorboard:             2.11.2-pyhd8ed1ab_0          conda-forge
    tensorboard-data-server: 0.6.1-py38h23f6d3d_4         conda-forge
    tensorboard-plugin-wit:  1.8.1-pyhd8ed1ab_0           conda-forge
    tensorflow-base:         2.11.0-cpu_py38h1ad6885_1    local      
    termcolor:               2.2.0-pyhd8ed1ab_0           conda-forge
    tk:                      8.6.12-he1e0b03_0            conda-forge
    typing-extensions:       4.5.0-hd8ed1ab_0             conda-forge
    typing_extensions:       4.5.0-pyha770c72_0           conda-forge
    urllib3:                 1.26.15-pyhd8ed1ab_0         conda-forge
    werkzeug:                2.2.3-pyhd8ed1ab_0           conda-forge
    wheel:                   0.40.0-pyhd8ed1ab_0          conda-forge
    wrapt:                   1.15.0-py38hb991d35_0        conda-forge
    xz:                      5.2.6-h57fd34a_0             conda-forge
    yarl:                    1.8.2-py38hb991d35_0         conda-forge
    zipp:                    3.15.0-pyhd8ed1ab_0          conda-forge
    zlib:                    1.2.13-h03a7124_4            conda-forge
    zstd:                    1.5.2-hf913c23_6             conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Reloading output folder: 
/Users/ngam/tensorflow-feedstock-osx_arm64_numpy1.20python3.8.____cpython/minifo
rge3/conda-bld
Mamba failed to solve:
 - cctools_osx-64 973.0.1.* h48a5a9d_12
 - bazel 5.4.0.* h4865ccd_2
 - libzlib 1.2.13.* hfd90126_4
 - gettext 0.21.1.* h8a4c099_0
 - icu 70.1.* h96cf925_0
 - libxml2 2.10.3.* h201ad9d_4
 - libgrpc 1.52.1.* h5bc3d57_1
 - zstd 1.5.2.* hbc0c0cd_6
 - sed 4.7.* h3efe00b_1000
 - ld64_osx-64 609.* hfd63004_12
 - sigtool 0.1.3.* h88f4db0_0
 - clang 14.0.6.* h694c41f_0
 - libabseil 20230125.0.* cxx17_hf0c8a7f_1
 - llvm-tools 14.0.6.* h5b596cc_1
 - rsync 3.2.7.* h30d983d_0
 - clang_osx-arm64 14.0.6.* he41d1a2_5
 - python 3.8.16.* hf9b03c3_1_cpython
 - openssl 3.1.0.* hfd90126_0
 - ca-certificates 2022.12.7.* h033912b_0
 - libsqlite 3.40.0.* ha978bb4_0
 - crossenv 1.4.0.* pyhd8ed1ab_0
 - popt 1.16.* h7b079dc_2002
 - c-ares 1.18.1.* h0d85af4_0
 - tensorflow-base >=2.11.0,<3.0a0
 - ld64_osx-arm64 609.* h566df06_12
 - clangxx 14.0.6.* default_h55ffa42_0
 - libcxx 15.0.7.* h71dddab_0
 - libllvm14 14.0.6.* h5b596cc_1
 - libclang-cpp14 14.0.6.* default_h55ffa42_0
 - openjdk 11.0.15.* hbc0c0cd_3
 - libffi 3.4.2.* h0d85af4_5
 - clangxx_osx-arm64 14.0.6.* h748fd88_5
 - libllvm15 15.0.7.* h7001e86_1
 - compiler-rt 14.0.6.* h613da45_0
 - xxhash 0.8.0.* h35c211d_3
 - compiler-rt_osx-64 14.0.6.* hab78ec2_0
 - libiconv 1.17.* hac89ed1_0
 - xz 5.2.6.* h775f41a_0
 - tapi 1100.0.11.* h9ce4665_0
 - lz4-c 1.9.4.* hf0c8a7f_0
 - bzip2 1.0.8.* h0d85af4_4
 - cross-python_osx-arm64 3.8.* 37_cpython
 - ncurses 6.3.* h96cf925_1
 - re2 2023.02.02.* hf0c8a7f_0
 - clang-14 14.0.6.* default_h55ffa42_0
 - libprotobuf 3.21.12.* hbc0c0cd_0
 - tk 8.6.12.* h5dbffcc_0
 - readline 8.1.2.* h3899abd_0
 - cctools_osx-arm64 973.0.1.* hd9287c6_12
 - zlib 1.2.13.* hfd90126_4

with channels:

The reported errors are:
- Encountered problems while solving:
-   - package tensorflow-base-2.11.0-cpu_py310h760b059_0 requires libgrpc >=1.51.1,<1.52.0a0, but none of the providers can be installed
- 

@ngam
Copy link
Contributor

ngam commented Mar 21, 2023

Also, this is the older tensorflow... (build 0) tensorflow-base-2.11.0-cpu_py310h760b059_0

Why is this happening in the cross-compilation case? We almost never build more than the 0th build, so that's why we never faced it!

@ngam
Copy link
Contributor

ngam commented Mar 21, 2023

We need {{ pin_subpackage('tensorflow*', exact=True) }} in more places...

@hmaarrfk
Copy link
Contributor

yes, i am thinking the same about the pinnings.

I'm trying to add a few more constraints in:
#309

You will also need to add a channel that can actually provide the native osx packages if you are cross compiling.

@ngam
Copy link
Contributor

ngam commented Mar 21, 2023

You will also need to add a channel that can actually provide the native osx packages if you are cross compiling.

Okay, this could be why! I will fix ~soon

@hmaarrfk
Copy link
Contributor

I actually think that when i'm cross compiling, it is trying to test things.... fd019c8#diff-d10439659fe44156d9eac347f9e9eb6992e5b516d0f7e86df061c8273397fd7dR58

@hmaarrfk
Copy link
Contributor

adding ICU, abseil, and grpc helped. I also skipped tests on emulated.

Fingers crossed. In my .condarc file, I have:

channels:
  - mark.harfouche/label/cfep03
  - conda-forge

@hmaarrfk
Copy link
Contributor

ok, 3.10 built for arm. Now off to 3.9 then 3.8.

Then we can do the ICU migration lol!

@ngam
Copy link
Contributor

ngam commented Mar 22, 2023

Is it the testing or the pins, you think? Or the channel?

@ngam
Copy link
Contributor

ngam commented Mar 22, 2023

We should try to focus our resources on getting the abseil, icu, etc in the next release… that may prove more productive and easier

@hmaarrfk
Copy link
Contributor

Both i think.

I am hopeful that the ICU and 2.11.1 are easy to migrate. I can likely start the compilation job and come back in 3 days when it is done (now that we spent 3 weeks troubleshooting this)

I am less hopeful that the abseil migration. It might require more thinking

@hmaarrfk
Copy link
Contributor

for package in tensorflow-base tensorflow tensorflow-estimator libtensorflow libtensorflow_cc tensorflow-cpu tensorflow-gpu; do
  anaconda copy --from-label cfep03 --to-label main --to-owner conda-forge mark.harfouche/${package}/2.11.0
done

@hmaarrfk hmaarrfk merged commit 72ecde5 into conda-forge:main Mar 23, 2023
@regro-cf-autotick-bot regro-cf-autotick-bot deleted the rebuild-jpeg_to_libjpeg_turbo-0-3_h5ee88f branch March 23, 2023 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants