cleaner: bump 2.8.0; rebase patches; new numpy pins #205

ngam · 2022-02-06T19:09:40Z

Closes #202
Closes #203
Closes #204
Closes #212

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

conda-forge-linter · 2022-02-06T19:09:44Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

It looks like the 'tensorflow-base' output doesn't have any tests.

README.md

…nda-forge-pinning 2022.02.06.08.21.47

ngam · 2022-02-06T19:32:19Z

Okay, everything broke for some reason! I will guess it is the 7f295b5 updating grpc_cpp to '1.43' since this is the only change from the other PR. grpc_cpp 1.43 wrecks mayhem basically: downgrading bazel, messing with cuda stuff, etc., so reverting to grpc_cpp 1.42 for now.

…nda-forge-pinning 2022.02.06.08.21.47

recipe/conda_build_config.yaml

ngam · 2022-02-06T20:23:57Z

Is this looking good for you, @hmaarrfk? Should we basically lock this in for production?

~~We can assess the tensorflow-io thing later. Easiest path is pip install wheel but this one has a nasty matrix, cp37--310 for different arches, and no osx-arm64.~~ (per #203 (comment) this is not feasible)

hmaarrfk · 2022-02-06T21:10:16Z

We really should fix

conda-forge/bazel-feedstock#90

recipe/patches/customize-setup.patch

conda-forge-linter · 2022-02-07T02:06:13Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

Old-style Python selectors (py27, py35, etc) are only available for Python 2.7, 3.4, 3.5, and 3.6. Please use explicit comparisons with the integer py, e.g. # [py==37] or # [py>=37]. See lines [85, 86, 160, 161]

For recipe:

It looks like the 'tensorflow-base' output doesn't have any tests.

conda-forge-linter · 2022-02-07T02:11:22Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

It looks like the 'tensorflow-base' output doesn't have any tests.

ngam · 2022-04-10T01:02:33Z

@conda-forge-admin, please rerender

…nda-forge-pinning 2022.04.09.19.01.16

ngam · 2022-04-10T15:25:12Z

@xhochy @hmaarrfk any idea why this has regressed significantly? Could you take a quick look and see if the errors look familiar?

hmaarrfk · 2022-04-26T02:13:08Z

You might want to update the pinnings and take a few patches from:
#218

They seem to be included in the 2.9.0 milestone.

hmaarrfk · 2022-04-26T13:02:38Z

I got stuck at the fact that grpc updated abseil/absl and that is causing compilation errors with CUDA builds.

We might want to do all the migrations except for GRPC

ngam · 2022-05-12T17:17:35Z

@conda-forge-admin, please rerender

…nda-forge-pinning 2022.05.12.13.19.50

ngam · 2022-05-12T17:31:15Z

@hmaarrfk @xhochy looks like you two did some magic in that last PR! Thanks a lot. This seems fine for now, so we could probably push forward. With the new singularity framework, we will hopefully have an easier time building going forward :)

Let me know if you know of any issues that I should be aware of, looks smooth to me so far, granted only ~10 mins...

xhochy · 2022-05-12T18:20:46Z

I will give this a spin locally and can build out all packages (osx+linux) if it passes.

ngam · 2022-05-12T19:21:47Z

These are gonna be slightly longer builds than 2.7.x, looking at 23000 vs 17000 (compilation stuff). I should have one build ready soon-ish if it doesn't fail... Will post top-line results here

ngam · 2022-05-12T19:30:58Z

Okay, I think we are ready for this! @hmaarrfk @xhochy 🥳

Would love a quick review and then we can get this out of the way!

cuda 11.2 py38:

INFO: Analyzed 3 targets (441 packages loaded, 31257 targets configured).
INFO: Found 3 targets...
[0 / 40] [Prepa] Expanding template tensorflow/lite/tools/visualize
[599 / 5,648] Compiling tensorflow/python/util/determinism.cc; 3s local ... (128 actions, 65 running)
[849 / 5,880] Compiling llvm/utils/TableGen/CodeGenDAGPatterns.cpp; 6s local ... (128 actions, 65 running)
[5,329 / 10,076] Compiling llvm/utils/TableGen/GlobalISelEmitter.cpp; 9s local ... (128 actions, 65 running)
[5,957 / 14,512] Compiling third_party/icu/data/conversion_data.c; 7s local ... (128 actions, 65 running)
[6,558 / 15,731] Compiling third_party/icu/data/conversion_data.c; 14s local ... (128 actions, 65 running)
[7,649 / 18,934] Compiling tensorflow/lite/kernels/conv.cc; 8s local ... (128 actions, 65 running)
[8,082 / 20,176] Compiling tensorflow/lite/kernels/conv.cc; 18s local ... (128 actions, 65 running)
[8,438 / 21,142] Compiling src/cpu/cpu_convolution_list.cpp; 18s local ... (128 actions, 65 running)
[9,568 / 23,290] Compiling src/cpu/cpu_rnn_list.cpp; 29s local ... (128 actions, 65 running)
[10,163 / 23,290] Compiling src/cpu/rnn/ref_rnn.cpp; 37s local ... (128 actions, 65 running)
[10,444 / 23,290] Compiling llvm/lib/Transforms/Utils/SimplifyCFG.cpp; 9s local ... (128 actions, 64 running)
[10,746 / 23,290] Compiling llvm/lib/Passes/PassBuilder.cpp; 14s local ... (128 actions, 64 running)
[11,024 / 23,290] Compiling llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp; 22s local ... (128 actions, 64 running)
[11,699 / 23,290] Compiling llvm/lib/Target/X86/X86ISelLowering.cpp; 32s local ... (128 actions, 64 running)
[12,157 / 23,290] Compiling llvm/lib/Transforms/IPO/AttributorAttributes.cpp; 14s local ... (127 actions, 64 running)
[12,606 / 23,290] Compiling llvm/lib/Target/X86/X86ISelLowering.cpp; 16s local ... (128 actions, 65 running)
[14,614 / 23,290] Compiling mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp; 14s local ... (128 actions, 64 running)
[15,673 / 23,290] Compiling mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp; 27s local ... (128 actions, 65 running)
[16,636 / 23,290] Compiling tensorflow/compiler/mlir/hlo/lib/Dialect/mhlo/IR/hlo_ops.cc; 46s local ... (128 actions, 65 running)
[17,590 / 23,290] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops_a_m.cc; 63s local ... (128 actions, 65 running)
[18,162 / 23,290] Compiling tensorflow/core/kernels/rnn/lstm_ops_gpu.cu.cc; 111s local ... (128 actions, 65 running)
[18,987 / 23,290] Compiling tensorflow/core/kernels/sparse_fill_empty_rows_op_gpu.cu.cc; 187s local ... (128 actions, 65 running)
[19,067 / 23,290] Compiling tensorflow/core/kernels/training_ops_gpu.cu.cc; 270s local ... (128 actions, 65 running)
[19,336 / 23,290] Compiling tensorflow/core/kernels/cwise_op_gpu_mul.cu.cc; 209s local ... (128 actions, 65 running)
[19,482 / 23,290] Compiling tensorflow/core/kernels/cwise_op_gpu_random_grad.cu.cc; 328s local ... (128 actions, 65 running)
[20,720 / 23,291] Compiling tensorflow/core/kernels/cwise_op_gpu_random_grad.cu.cc; 467s local ... (128 actions, 65 running)
[21,193 / 23,291] Compiling tensorflow/core/kernels/cwise_op_gpu_igammas_float.cu.cc; 624s local ... (128 actions, 64 running)
[23,265 / 23,335] Compiling tensorflow/core/kernels/segment_reduction_ops_gpu_1.cu.cc; 671s local ... (7 actions running)
[23,273 / 23,335] Compiling tensorflow/core/kernels/pad_op_gpu.cu.cc; 567s local ... (2 actions running)
INFO: Elapsed time: 1838.318s, Critical Path: 947.64s
INFO: 23335 processes: 5923 internal, 17412 local.
INFO: Build completed successfully, 23335 total actions
INFO: Build completed successfully, 23335 total actions
...
####################################################################################
Resource usage summary:

Total time: 0:51:50.6
CPU usage: sys=0:00:30.3, user=0:01:11.8
Maximum memory usage observed: 1.2G
Total disk usage observed (not including envs): 2.2G
Validating outputs
+ validate_recipe_outputs tensorflow-feedstock
validation results:
{
  "linux-64/libtensorflow-2.8.0-cuda112h98e1010_0.tar.bz2": true,
  "linux-64/libtensorflow_cc-2.8.0-cuda112h98e1010_0.tar.bz2": true,
  "linux-64/tensorflow-base-2.8.0-cuda112py38h6a3b174_0.tar.bz2": true,
  "linux-64/tensorflow-estimator-2.8.0-cuda112py38hf5dcc89_0.tar.bz2": true,
  "linux-64/tensorflow-2.8.0-cuda112py38hded6998_0.tar.bz2": true,
  "linux-64/tensorflow-gpu-2.8.0-cuda112py38h0bbbad9_0.tar.bz2": true
}

xhochy · 2022-05-13T11:33:36Z

The OSX builds don't pass. I know what to do and will tackle that later.

+ test -f /Users/uwe/mambaforge/conda-bld/tensorflow-split_1652379672357/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libtensorflow.dylib
+ ./test_libtensorflow.sh
+ arm64-apple-darwin20.0.0-clang -o test_c -L/Users/uwe/mambaforge/conda-bld/tensorflow-split_1652379672357/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/ -ltensorflow -I/Users/uwe/mambaforge/conda-bld/tensorflow-split_1652379672357/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include/ test_c.c
+ ./test_c
dyld[2693]: Library not loaded: @rpath/libtensorflow.2.dylib
  Referenced from: /Users/uwe/mambaforge/conda-bld/tensorflow-split_1652379672357/test_tmp/test_c
  Reason: tried: '/usr/local/lib/libtensorflow.2.dylib' (no such file), '/usr/lib/libtensorflow.2.dylib' (no such file)
./test_libtensorflow.sh: line 6:  2693 Abort trap: 6           ./test_c
Tests failed for libtensorflow-2.8.0-cpu_hb019c61_0.tar.bz2 - moving package to /Users/uwe/mambaforge/conda-bld/broken
WARNING:conda_build.build:Tests failed for libtensorflow-2.8.0-cpu_hb019c61_0.tar.bz2 - moving package to /Users/uwe/mambaforge/conda-bld/broken
TESTS FAILED: libtensorflow-2.8.0-cpu_hb019c61_0.tar.bz2

ngam · 2022-05-13T23:28:56Z

Looking good, I am happy to split the builds with you, so let me know. I am on standby for today and tomorrow (Saturday). After that, I will be mostly absent for ~10 days. I also have more stuff for you 😜 (cuda-full jaxlib + a preliminary PR for 2.8.1)

xhochy · 2022-05-14T18:36:01Z

Enjoy being absent. There are still some small issues. I can iron them out and build things in the next days. Currently the osx-arm64 fail locally because MACOSX_SDK_VERSION is not set when running the local builds. Once all OSX builds have passed, I'll do the Linux builds.

ngam · 2022-05-15T02:44:10Z

Sounds good! Always here to help :) thanks a lot, you're a star 🌟

xhochy · 2022-05-17T18:14:12Z

@hmaarrfk Builds are ready in https://anaconda.org/uwe.korn/repo for osx-* and https://anaconda.org/uwe.korn-tf-gpu/repo for linux-* Can you have a look and approve? I would then copy over (or feel free to do so yourself)

Logs:

https://gist.github.com/xhochy/d5419c15045d5d2c0507310456cf3255 linux_64_c_compiler_version7cuda_compiler_version10.2cudnn7cxx_compiler_version7python3.8.____cpython.log
https://gist.github.com/xhochy/6f602aacdc2f258a77fe78203a7e8a27 linux_64_c_compiler_version9cuda_compiler_version11.0cudnn8cxx_compiler_version9python3.9.____cpython.log
https://gist.github.com/xhochy/8148b65a3f3118bdb63ebafb7b283dd2 linux_64_c_compiler_version9cuda_compiler_version11.0cudnn8cxx_compiler_version9python3.7.____cpython.log
https://gist.github.com/xhochy/a6928522bd77c6c4de93940d719720b0 linux_64_c_compiler_version7cuda_compiler_version10.2cudnn7cxx_compiler_version7python3.7.____cpython.log
https://gist.github.com/xhochy/046214128e01be1efbda6bd6fa4f78fd linux_64_c_compiler_version9cuda_compiler_version11.0cudnn8cxx_compiler_version9python3.8.____cpython.log
https://gist.github.com/xhochy/72f9eade96f3424199a676e0ac37c9c1 linux_64_c_compiler_version9cuda_compiler_version11.0cudnn8cxx_compiler_version9python3.10.____cpython.log
https://gist.github.com/xhochy/f27792e27b47dd0b1b76476e0e09c04f linux_64_c_compiler_version7cuda_compiler_version10.2cudnn7cxx_compiler_version7python3.9.____cpython.log
https://gist.github.com/xhochy/d52ca8d43996a68915f5848cad515e4e linux_64_c_compiler_version7cuda_compiler_version10.2cudnn7cxx_compiler_version7python3.10.____cpython.log
https://gist.github.com/xhochy/3ca2e01bc85d6a3c5a9cbc60a9b47ec8 linux_64_c_compiler_version10cuda_compiler_version11.2cudnn8cxx_compiler_version10python3.9.____cpython.log
https://gist.github.com/xhochy/ca5140ca5177380ffb017e3d8b248257 linux_64_c_compiler_version10cuda_compiler_version11.2cudnn8cxx_compiler_version10python3.7.____cpython.log
https://gist.github.com/xhochy/6eab7c33cb1f57d8a237d7e8bec848d6 linux_64_c_compiler_version10cuda_compiler_versionNonecudnnundefinedcxx_compiler_version10python3.7.____cpython.log
https://gist.github.com/xhochy/35fdd676ea47e7af7fcd196b5edfdc6e linux_64_c_compiler_version10cuda_compiler_version11.2cudnn8cxx_compiler_version10python3.10.____cpython.log
https://gist.github.com/xhochy/3c63cef4b9087b244bcecd4b3b1d144e linux_64_c_compiler_version10cuda_compiler_version11.2cudnn8cxx_compiler_version10python3.8.____cpython.log
https://gist.github.com/xhochy/b114dd937555d1aff3fdec5638f66050 linux_64_c_compiler_version10cuda_compiler_version11.1cudnn8cxx_compiler_version10python3.10.____cpython.log
https://gist.github.com/xhochy/67091ff176f66aabda345df56e5b9508 linux_64_c_compiler_version10cuda_compiler_version11.1cudnn8cxx_compiler_version10python3.9.____cpython.log
https://gist.github.com/xhochy/80e6cd732a2b837fe60eb7d02fd3d01b linux_64_c_compiler_version10cuda_compiler_versionNonecudnnundefinedcxx_compiler_version10python3.8.____cpython.log
https://gist.github.com/xhochy/4e97c3fe1d7452536b56557d5b20c193 linux_64_c_compiler_version10cuda_compiler_version11.1cudnn8cxx_compiler_version10python3.8.____cpython.log
https://gist.github.com/xhochy/b9eab5fb69d71129d09935954dd01602 linux_64_c_compiler_version10cuda_compiler_version11.1cudnn8cxx_compiler_version10python3.7.____cpython.log
https://gist.github.com/xhochy/99f24dca0bc3dafcc1d218de178e4c46 linux_64_c_compiler_version10cuda_compiler_versionNonecudnnundefinedcxx_compiler_version10python3.10.____cpython.log
https://gist.github.com/xhochy/fdcd6e47f9f638a80dcfa5a2b341fe88 linux_64_c_compiler_version10cuda_compiler_versionNonecudnnundefinedcxx_compiler_version10python3.9.____cpython.log
https://gist.github.com/xhochy/4a1f02af11b14a0d6844a7f938740807 osx_64_python3.10.____cpython.log
https://gist.github.com/xhochy/e0528cd03e65301b7af30212b703f4dd osx_64_python3.9.____cpython.log
https://gist.github.com/xhochy/945b54b74b5812be1fca4c6e7df2ab74 osx_64_python3.8.____cpython.log
https://gist.github.com/xhochy/ef366d5c9771bd5e8b4404d6bba07440 osx_64_python3.7.____cpython.log
https://gist.github.com/xhochy/4ab2c8c388451160c014b84cd9d54d6a osx_arm64_python3.10.____cpython.log
https://gist.github.com/xhochy/e49cd5c5e96c5e6d3a26c34351db1bfd osx_arm64_python3.9.____cpython.log
https://gist.github.com/xhochy/0734f33e507293e26bdf156ea3feb291 osx_arm64_python3.8.____cpython.log

xhochy · 2022-05-17T18:14:26Z

OSX is still uploading …

xhochy · 2022-05-17T18:54:08Z

@hmaarrfk all uploaded, so ready for review/copy.

hmaarrfk · 2022-05-18T02:02:05Z

Thanks!

for package in  tensorflow-estimator tensorflow-cpu tensorflow-base tensorflow  libtensorflow_cc libtensorflow; do
anaconda copy --from-label main --to-label main --to-owner conda-forge uwe.korn/${package}/2.8.0
done
for package in  tensorflow-estimator tensorflow-cpu tensorflow-gpu tensorflow-base tensorflow  libtensorflow_cc libtensorflow; do
anaconda copy --from-label main --to-label main --to-owner conda-forge uwe.korn-tf-gpu/${package}/2.8.0
done

h-vetinari · 2022-05-18T08:36:55Z

Thanks everyone for sticking through this! 🥳

ngam requested review from FarhanTejani, ghego, h-vetinari, hajapy, hmaarrfk, jschueller, njzjz, waitingkuo, wolfv and xhochy as code owners February 6, 2022 19:09

ngam mentioned this pull request Feb 6, 2022

Bump 2.8.0; remove patches #203

Closed

12 tasks

ngam commented Feb 6, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

ngam added 2 commits February 6, 2022 14:15

bump 2.8.0; remove patches

8395a27

MNT: Re-rendered with conda-build 3.21.8, conda-smithy 3.16.2, and co…

7f295b5

…nda-forge-pinning 2022.02.06.08.21.47

ngam force-pushed the v2.8.0 branch from 2bb43db to 7f295b5 Compare February 6, 2022 19:15

ngam added 2 commits February 6, 2022 14:38

force grpc_cpp to 1.42

52a503b

MNT: Re-rendered with conda-build 3.21.8, conda-smithy 3.16.2, and co…

2ac43a0

…nda-forge-pinning 2022.02.06.08.21.47

hmaarrfk reviewed Feb 6, 2022

View reviewed changes

recipe/conda_build_config.yaml Outdated Show resolved Hide resolved

h-vetinari reviewed Feb 6, 2022

View reviewed changes

recipe/patches/customize-setup.patch Outdated Show resolved Hide resolved

h-vetinari reviewed Feb 6, 2022

View reviewed changes

recipe/patches/customize-setup.patch Outdated Show resolved Hide resolved

Update conda_build_config.yaml

664d955

pin numpy to 1.20/1.21 and rerender

feada08

ngam force-pushed the v2.8.0 branch from 9975629 to feada08 Compare February 7, 2022 02:25

ngam changed the title ~~cleaner --- bump 2.8.0; remove patches~~ cleaner --- bump 2.8.0; remove patches; new numpy pins Feb 7, 2022

MNT: Re-rendered with conda-build 3.21.8, conda-smithy 3.19.0, and co…

106b1ac

…nda-forge-pinning 2022.04.09.19.01.16

limit flatbuffers

bdab3c9

rebase meta

a8e2681

MNT: Re-rendered with conda-build 3.21.8, conda-smithy 3.19.0, and co…

0f1e912

…nda-forge-pinning 2022.05.12.13.19.50

xhochy added 4 commits May 13, 2022 13:39

Fix build of libtensorflow on OSX

245e7db

Add abseil-cpp pinning to libtensorflow*

228683b

Enable cross-compilation

e47e79e

Add cross-compilation dependencies

1bb14b3

Allow for empty MACOSX_SDK_VERSION

cc1eca4

xhochy mentioned this pull request May 16, 2022

Protobuf320 and GRPC migration #236

Closed

5 tasks

hmaarrfk merged commit 51db7f8 into conda-forge:main May 18, 2022

ngam deleted the v2.8.0 branch June 2, 2022 20:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cleaner: bump 2.8.0; rebase patches; new numpy pins #205

cleaner: bump 2.8.0; rebase patches; new numpy pins #205

ngam commented Feb 6, 2022 •

edited by hmaarrfk

Loading

conda-forge-linter commented Feb 6, 2022

ngam commented Feb 6, 2022 •

edited

Loading

ngam commented Feb 6, 2022 •

edited

Loading

hmaarrfk commented Feb 6, 2022

conda-forge-linter commented Feb 7, 2022

conda-forge-linter commented Feb 7, 2022

ngam commented Apr 10, 2022

ngam commented Apr 10, 2022

hmaarrfk commented Apr 26, 2022

hmaarrfk commented Apr 26, 2022

ngam commented May 12, 2022

ngam commented May 12, 2022

xhochy commented May 12, 2022

ngam commented May 12, 2022 •

edited

Loading

ngam commented May 12, 2022 •

edited

Loading

xhochy commented May 13, 2022

ngam commented May 13, 2022

xhochy commented May 14, 2022

ngam commented May 15, 2022

xhochy commented May 17, 2022

xhochy commented May 17, 2022

xhochy commented May 17, 2022

hmaarrfk commented May 18, 2022

h-vetinari commented May 18, 2022

cleaner: bump 2.8.0; rebase patches; new numpy pins #205

cleaner: bump 2.8.0; rebase patches; new numpy pins #205

Conversation

ngam commented Feb 6, 2022 • edited by hmaarrfk Loading

conda-forge-linter commented Feb 6, 2022

ngam commented Feb 6, 2022 • edited Loading

ngam commented Feb 6, 2022 • edited Loading

hmaarrfk commented Feb 6, 2022

conda-forge-linter commented Feb 7, 2022

conda-forge-linter commented Feb 7, 2022

ngam commented Apr 10, 2022

ngam commented Apr 10, 2022

hmaarrfk commented Apr 26, 2022

hmaarrfk commented Apr 26, 2022

ngam commented May 12, 2022

ngam commented May 12, 2022

xhochy commented May 12, 2022

ngam commented May 12, 2022 • edited Loading

ngam commented May 12, 2022 • edited Loading

xhochy commented May 13, 2022

ngam commented May 13, 2022

xhochy commented May 14, 2022

ngam commented May 15, 2022

xhochy commented May 17, 2022

xhochy commented May 17, 2022

xhochy commented May 17, 2022

hmaarrfk commented May 18, 2022

h-vetinari commented May 18, 2022

ngam commented Feb 6, 2022 •

edited by hmaarrfk

Loading

ngam commented Feb 6, 2022 •

edited

Loading

ngam commented Feb 6, 2022 •

edited

Loading

ngam commented May 12, 2022 •

edited

Loading

ngam commented May 12, 2022 •

edited

Loading