WIP: Bump to 2.12 + add aarch64 #301

Tobias-Fischer · 2023-02-13T19:40:30Z

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

conda-forge-webservices · 2023-02-13T19:40:36Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

It looks like the 'tensorflow-base' output doesn't have any tests.

Tobias-Fischer · 2023-02-13T19:40:54Z

@conda-forge-admin, please rerender.

…nda-forge-pinning 2023.02.13.17.41.30

Tobias-Fischer · 2023-02-13T20:46:16Z

Seems like we'll need to wait until the release of tensorflow 2.12 (as we only have tensorboard 2.12 for aarch64). This is scheduled to happen at the end of this month: https://discuss.tensorflow.org/t/when-will-tensorflow-2-12-be-released/14717

ngam · 2023-02-13T22:37:28Z

Can you target the r2.12 branch? And for the sake of resources, let's try to turn off everything else? Do you want me to do it?

ngam · 2023-02-13T22:42:34Z

something like this maybe:

source:
- git_url: https://github.com/tensorflow/tensorflow/
  git_tag: r2.12
- git_url: https://github.com/tensorflow/estimator/
  git_tag: r2.12
  folder: tensorflow-estimator
 #
 # - url: https://github.com/tensorflow/tensorflow/archive/refs/tags/v{{ version.replace(".rc", "-rc") }}.tar.gz
 #   sha256: 99c732b92b1b37fc243a559e02f9aef5671771e272758aa4aec7f34dc92dac48
 #   patches:
 #     - patches/0001-loosen-requirements.patch
#
 # - url: https://github.com/tensorflow/estimator/archive/refs/tags/v{{ estimator_version.replace(".rc", "-rc") }}.tar.gz
 #   sha256: 922f9187de79e8e7f7d7a5b2d6d3aabc81bbbd6ba5f12a4f52967dd302214a43
 #   folder: tensorflow-estimator

like we do in pytorch: https://github.com/conda-forge/pytorch-cpu-feedstock/blob/main/recipe/meta.yaml

Tobias-Fischer · 2023-02-13T23:20:33Z

Yes please, go for it!

Tobias-Fischer · 2023-02-13T23:22:16Z

Note that we'll need to update keras as well ..

ngam · 2023-02-13T23:47:46Z

Yeah, but keras is just a run dep, what we are interested in here is catching serious issues in the compilation. In the first 10 hours of the build or so 🤣

ngam · 2023-02-13T23:49:27Z

Oops, I forgot we have keras in host… well…

ngam · 2023-02-14T01:38:43Z

Oops, I forgot we have keras in host… well…

Tobias-Fischer · 2023-02-14T02:29:11Z

Good point actually - afaik keras sits on top of tensorflow, so we might want to remove the keras host dep (at least to play around on this PR)?

ngam · 2023-02-14T02:59:28Z

I usually wait until they tag something, https://github.com/tensorflow/tensorflow/tags. They usually do three RCs. It seems they're currently testing rc0 internally but haven't tagged it yet...

conda-forge.yml

ngam · 2023-02-14T22:58:42Z

Hm. Let's try! But that may be over the lite only. Let's see how this goes.

some recs:

their ci_build folder: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/ci_build/
our custom_toolchain folder: https://github.com/Tobias-Fischer/tensorflow-feedstock/tree/patch-1/recipe/custom_toolchain

ngam · 2023-02-17T03:54:48Z

Time to switch to the prerelease and please feel free to include some of the other arches since we are interested in seeing how that plays out as well https://github.com/tensorflow/tensorflow/releases/tag/v2.12.0-rc0

…nda-forge-pinning 2023.02.16.23.42.32

Tobias-Fischer · 2023-02-17T04:25:26Z

This weird protobuf error now shows up everywhere - this commit introduced it: tensorflow/tensorflow@84f4092

Tobias-Fischer · 2023-02-17T04:29:14Z

I don't understand bazel, but this looks like it could be helpful: envoyproxy/envoy#23345

ngam · 2023-02-17T04:32:17Z

We are not supposed to be vendoring protobuf though, https://github.com/Tobias-Fischer/tensorflow-feedstock/blob/022daa6a1f2701be2426057b7e8395e771249f2c/recipe/build.sh#L43

I am sure @h-vetinari would be intrigued by this failure and the lovely commit upstream tensorflow/tensorflow@84f4092

ngam · 2023-02-17T04:33:06Z

we could try to vendor it to see if it gets us any further. Need to take it out from here https://github.com/Tobias-Fischer/tensorflow-feedstock/blob/022daa6a1f2701be2426057b7e8395e771249f2c/recipe/build.sh#L43

Tobias-Fischer · 2023-02-17T06:14:15Z

Feel free to play around here @ngam :)

ngam · 2023-02-20T02:19:30Z

This weird protobuf error now shows up everywhere

Any other places you've seen? Could you link them so that we can have a look?

ngam · 2023-02-20T02:21:09Z

The current error is

ERROR: /home/conda/feedstock_root/build_artifacts/tensorflow-split_1676859266213/work/tensorflow/distribute/experimental/rpc/proto/BUILD:16:17: no such target '@com_google_protobuf//:well_known_types_py_pb2_genproto': target 'well_known_types_py_pb2_genproto' not declared in package '' defined by /home/conda/feedstock_root/build_artifacts/tensorflow-split_1676859266213/_build_env/share/bazel/8c55eb693c369964777d51780f646c97/external/com_google_protobuf/BUILD.bazel and referenced by '//tensorflow/distribute/experimental/rpc/proto:tf_rpc_service_proto_py_genproto'

which is definitely related to the commit you referenced tensorflow/tensorflow@84f4092. We can change that one piece since we rely on our own protobuf. Let me think

ngam · 2023-02-20T02:50:35Z

Moved forward only very little:

WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/10939d1d580b9d3c9c2f3539c6bdb39f408179c0.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
Analyzing: 3 targets (3 packages loaded, 0 targets configured)
Analyzing: 3 targets (254 packages loaded, 4302 targets configured)
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/081771d4a0e9d7d3aa0eed2ef389fa4700dfb23e.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/659147817805d17c7be2d60bd7bbca7e780f9c82.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
Analyzing: 3 targets (560 packages loaded, 24640 targets configured)
INFO: Analyzed 3 targets (581 packages loaded, 30216 targets configured).
INFO: Found 3 targets...
[0 / 4] [Prepa] Creating source manifest for //tensorflow/lite/python:tflite_convert
ERROR: /home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/work/tensorflow/c/BUILD:359:11: Compiling tensorflow/c/tf_status.cc failed: (Exit 1): x86_64-conda-linux-gnu-cc failed: error executing command 
  (cd /home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/_build_env/share/bazel/609be7cbd037399b3ae88bd273c2898a/execroot/org_tensorflow && \
  exec env - \
    PATH=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/work:/home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/_build_env/bin:/home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin:/opt/conda/condabin:/home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/_build_env/bin:/home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin:/opt/conda/bin:/opt/conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/conda/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin/python \
    PYTHON_LIB_PATH=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages \
    TF2_BEHAVIOR=1 \
    TF_SYSTEM_LIBS=absl_py,astor_archive,astunparse_archive,boringssl,com_github_googlecloudplatform_google_cloud_cpp,com_github_grpc_grpc,com_google_protobuf,curl,cython,dill_archive,flatbuffers,gast_archive,gif,icu,libjpeg_turbo,org_sqlite,png,pybind11,snappy,zlib \
  /home/conda/feedstock_root/build_artifacts/tensorflow-split_1676860792782/_build_env/bin/x86_64-conda-linux-gnu-cc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/tensorflow/c/_objs/tf_status/tf_status.pic.d '-frandom-seed=bazel-out/k8-opt/bin/tensorflow/c/_objs/tf_status/tf_status.pic.o' -fPIC -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/double_conversion -iquote bazel-out/k8-opt/bin/external/double_conversion -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/gif -iquote bazel-out/k8-opt/bin/external/gif -iquote external/libjpeg_turbo -iquote bazel-out/k8-opt/bin/external/libjpeg_turbo -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/k8-opt/bin/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/k8-opt/bin/external/fft2d -iquote external/highwayhash -iquote bazel-out/k8-opt/bin/external/highwayhash -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -isystem third_party/eigen3/mkl_include -isystem bazel-out/k8-opt/bin/third_party/eigen3/mkl_include -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem external/farmhash_archive/src -isystem bazel-out/k8-opt/bin/external/farmhash_archive/src -Wno-all -Wno-extra -Wno-deprecated -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-array-bounds -Wunused-result '-Werror=unused-result' -Wswitch '-Werror=switch' '-Wno-error=unused-but-set-variable' -DAUTOLOAD_DYNAMIC_KERNELS '-std=c++17' -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare '-ftemplate-depth=900' -fno-exceptions '-DTENSORFLOW_USE_XLA=1' -DINTEL_MKL -DAMD_ZENDNN -msse3 -pthread -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c tensorflow/c/tf_status.cc -o bazel-out/k8-opt/bin/tensorflow/c/_objs/tf_status/tf_status.pic.o)
# Configuration: 057794c2c7a443d665e0dee42dee9ec3d98b8fb04857114cb23dc222a477a11c
# Execution platform: @local_execution_config_platform//:platform
In file included from ./tensorflow/tsl/platform/status.h:37,
                 from ./tensorflow/tsl/c/tsl_status_internal.h:19,
                 from ./tensorflow/c/tf_status_internal.h:19,
                 from tensorflow/c/tf_status.cc:18:
bazel-out/k8-opt/bin/tensorflow/tsl/protobuf/error_codes.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory
   10 | #include <google/protobuf/port_def.inc>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
INFO: Elapsed time: 78.134s, Critical Path: 0.77s
INFO: 61 processes: 7 internal, 54 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Now hitting this: tensorflow/tensorflow#27697

conda-forge-webservices · 2023-03-07T20:34:42Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

For recipe:

It looks like the 'tensorflow-base' output doesn't have any tests.

Tobias-Fischer · 2023-03-07T21:52:21Z

Same problem still in rc1:

# Configuration: 3f443e42aa2f8201efa7964106f941d560d07bafca668c8eca9133ef11e55c03
# Execution platform: @local_execution_config_platform//:platform
In file included from bazel-out/k8-opt/bin/tensorflow/core/framework/step_stats.pb.cc:4:
bazel-out/k8-opt/bin/tensorflow/core/framework/step_stats.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory
   10 | #include <google/protobuf/port_def.inc>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

hmaarrfk · 2023-03-11T14:47:07Z

Any objections to moving forward with #303 while some issues with 2.12 are resolved?

ngam · 2023-03-11T18:53:43Z

No objections :)

ngam · 2023-03-11T19:03:47Z

Let me know when everything is ready and I can potentially help build this week

Tobias-Fischer · 2023-03-11T20:16:10Z

Go for it :)

Add aarch64

f52141a

Tobias-Fischer requested review from FarhanTejani, ghego, h-vetinari, hajapy, hmaarrfk, jschueller, ngam, njzjz, waitingkuo, wolfv and xhochy as code owners February 13, 2023 19:40

MNT: Re-rendered with conda-build 3.23.3, conda-smithy 3.22.1, and co…

7d92818

…nda-forge-pinning 2023.02.13.17.41.30

Tobias-Fischer mentioned this pull request Feb 13, 2023

Linux-aarch64 support? #136

Open

1 task

hmaarrfk reviewed Feb 14, 2023

View reviewed changes

conda-forge.yml Outdated Show resolved Hide resolved

Tobias-Fischer added 3 commits February 15, 2023 06:40

Skip non-aarch and non-py39

dbf5851

No new conda-smithy yet

2708646

Temp: Use git_branch for testing

8b46d2a

ngam and others added 3 commits February 16, 2023 22:59

use RCs

590de0a

only build py310

6e281e0

MNT: Re-rendered with conda-build 3.23.3, conda-smithy 3.22.1, and co…

022daa6

…nda-forge-pinning 2023.02.16.23.42.32

Tobias-Fischer and others added 2 commits February 18, 2023 20:55

Vendor protobuf for testing

49cf92c

use our own protobuf

93a3c72

ngam added 4 commits February 19, 2023 21:25

add patch

c3316d9

try patch

3d5dbbb

argparse patch absl

ad43bb8

Create disable-argparse-in-absl.patch

518fb98

ngam mentioned this pull request Feb 28, 2023

merge tensorflow-base and libtensorflow_cc? #302

Open

Bump

023da31

Update 0001-loosen-requirements.patch

423b222

Tobias-Fischer closed this Mar 27, 2023

Tobias-Fischer mentioned this pull request Mar 27, 2023

Bump to 2.12 #316

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Bump to 2.12 + add aarch64 #301

WIP: Bump to 2.12 + add aarch64 #301

Tobias-Fischer commented Feb 13, 2023 •

edited

Loading

conda-forge-webservices bot commented Feb 13, 2023

Tobias-Fischer commented Feb 13, 2023 •

edited by ngam

Loading

Tobias-Fischer commented Feb 13, 2023

ngam commented Feb 13, 2023

ngam commented Feb 13, 2023 •

edited

Loading

Tobias-Fischer commented Feb 13, 2023

Tobias-Fischer commented Feb 13, 2023

ngam commented Feb 13, 2023

ngam commented Feb 13, 2023

ngam commented Feb 14, 2023

Tobias-Fischer commented Feb 14, 2023 •

edited

Loading

ngam commented Feb 14, 2023

ngam commented Feb 14, 2023

ngam commented Feb 17, 2023

Tobias-Fischer commented Feb 17, 2023

Tobias-Fischer commented Feb 17, 2023

ngam commented Feb 17, 2023

ngam commented Feb 17, 2023

Tobias-Fischer commented Feb 17, 2023

ngam commented Feb 20, 2023

ngam commented Feb 20, 2023

ngam commented Feb 20, 2023

conda-forge-webservices bot commented Mar 7, 2023

Tobias-Fischer commented Mar 7, 2023

hmaarrfk commented Mar 11, 2023

ngam commented Mar 11, 2023

ngam commented Mar 11, 2023

Tobias-Fischer commented Mar 11, 2023

WIP: Bump to 2.12 + add aarch64 #301

WIP: Bump to 2.12 + add aarch64 #301

Conversation

Tobias-Fischer commented Feb 13, 2023 • edited Loading

conda-forge-webservices bot commented Feb 13, 2023

Tobias-Fischer commented Feb 13, 2023 • edited by ngam Loading

Tobias-Fischer commented Feb 13, 2023

ngam commented Feb 13, 2023

ngam commented Feb 13, 2023 • edited Loading

Tobias-Fischer commented Feb 13, 2023

Tobias-Fischer commented Feb 13, 2023

ngam commented Feb 13, 2023

ngam commented Feb 13, 2023

ngam commented Feb 14, 2023

Tobias-Fischer commented Feb 14, 2023 • edited Loading

ngam commented Feb 14, 2023

ngam commented Feb 14, 2023

ngam commented Feb 17, 2023

Tobias-Fischer commented Feb 17, 2023

Tobias-Fischer commented Feb 17, 2023

ngam commented Feb 17, 2023

ngam commented Feb 17, 2023

Tobias-Fischer commented Feb 17, 2023

ngam commented Feb 20, 2023

ngam commented Feb 20, 2023

ngam commented Feb 20, 2023

conda-forge-webservices bot commented Mar 7, 2023

Tobias-Fischer commented Mar 7, 2023

hmaarrfk commented Mar 11, 2023

ngam commented Mar 11, 2023

ngam commented Mar 11, 2023

Tobias-Fischer commented Mar 11, 2023

Tobias-Fischer commented Feb 13, 2023 •

edited

Loading

Tobias-Fischer commented Feb 13, 2023 •

edited by ngam

Loading

ngam commented Feb 13, 2023 •

edited

Loading

Tobias-Fischer commented Feb 14, 2023 •

edited

Loading