{lib}[foss/2022a] TensorFlow v2.9.1 w/ Python 3.10.4 #17092

Flamefire · 2023-01-12T15:45:31Z

(created using eb --new-pr)

Based on #16008 by @alinelena and #16620 by @VRehnberg

Requires new Easyblock fix TensorFlow easyblock for new versions of Bazel & TensorFlow easybuild-easyblocks#2854

Note that there will be

WARNING: 1 TensorFlow dependencies have not been resolved by EasyBuild. Check the log for details.

This is due to Abseil being a possible $TF_SYSTEM_LIBS since 2.9 but it looks like it got broken between when that PR was opened and 2.9 released: tensorflow/tensorflow#53765 (comment)

Might be worth resolving in the EasyBlock

jfgrimm · 2023-01-12T17:51:00Z

Test report by @jfgrimm
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2854
FAILED
Build succeeded for 8 out of 10 (2 easyconfigs in total)
himem04.pri.viking.alces.network - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz (cascadelake), Python 3.6.8
See https://gist.github.com/c33aaf092e9972d5516ecaa5d2536ffe for a full test report.

Flamefire · 2023-01-13T09:18:17Z

@jfgrimm I'm a bit surprised of this failure:

NodeFileWriterTest.test_simple
...
/python/framework/node_file_writer_test.py", line 142, in test_simple
    self.assertEqual(node_def1.op, 'MatMul')
AssertionError: 
- _MklMatMul
? ----
+ MatMul

Also in the source of the failing testGetMemoryInfoCPU I see a skip if MKL is enabled (but isn't triggered).

Are you using any modifications to the toolchain or so which pulls in MKL? Or have mkl-dnn somewhere loaded? Maybe attach the full log.

jfgrimm · 2023-01-13T10:54:34Z

@Flamefire my site doesn't modify toolchains, and I really shouldn't have any MKL stuff loaded
TensorFlow-2.9.1-20230112-log.tar.gz

Flamefire · 2023-01-13T13:15:58Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2854
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
taurusi8019 - Linux CentOS Linux 7.9.2009, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 470.57.02, Python 2.7.5
See https://gist.github.com/a80859d5c82a34691118b7cda69da3d3 for a full test report.

Flamefire · 2023-01-16T10:22:57Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2854
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
taurusi8021 - Linux CentOS Linux 7.9.2009, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 470.57.02, Python 2.7.5
See https://gist.github.com/5454c70c2d5d7d4422a73c8e2fe1dac4 for a full test report.

Flamefire · 2023-01-16T16:55:58Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2854
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
taurusml30 - Linux RHEL 7.6, POWER, 8335-GTX (power9le), 6 x NVIDIA Tesla V100-SXM2-32GB, 440.64.00, Python 2.7.5
See https://gist.github.com/e13895b3515f85c08ef1bdfc9dbe277e for a full test report.

Flamefire · 2023-01-16T19:59:48Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2854
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
taurusa12 - Linux CentOS Linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), 3 x NVIDIA GeForce GTX 1080 Ti, 460.32.03, Python 2.7.5
See https://gist.github.com/4010143d07d606a185997ae46c937929 for a full test report.

VRehnberg · 2023-01-19T13:21:17Z

I'm a bit surprised of this failure:
NodeFileWriterTest.test_simple
...
/python/framework/node_file_writer_test.py", line 142, in test_simple
    self.assertEqual(node_def1.op, 'MatMul')
AssertionError: 
- _MklMatMul
? ----
+ MatMul
Also in the source of the failing testGetMemoryInfoCPU I see a skip if MKL is enabled (but isn't triggered).

Are you using any modifications to the toolchain or so which pulls in MKL? Or have mkl-dnn somewhere loaded? Maybe attach the full log.

I'd just like to second that I'm seeing this same Failure on our system as well.

Flamefire · 2023-01-19T22:36:43Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2854
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
taurusa12 - Linux CentOS Linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), 3 x NVIDIA GeForce GTX 1080 Ti, 460.32.03, Python 2.7.5
See https://gist.github.com/d5c1a8c8c0eee0548ab1c8eb626d771e for a full test report.

SebastianAchilles · 2023-01-23T13:54:44Z

Test report by @SebastianAchilles
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2854
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
skl-rockylinux-87 - Linux Rocky Linux 8.7, x86_64, Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz (skylake), 1 x NVIDIA NVIDIA RTX A4000, 525.60.13, Python 3.6.8
See https://gist.github.com/6044f743a8e88aea0e2a9698d7707385 for a full test report.

Flamefire · 2023-02-01T13:19:07Z

I'd just like to second that I'm seeing this same Failure on our system as well.

I worked with @jfgrimm a bit trying to catch a log of the failing tests but it suddenly succeeded (i.e. no error anymore, all tests green).

So if anyone (@VRehnberg ?) is seeing those MKL-related failures consistently, maybe he can run a modified version with
TF2.9_logging.patch.txt and send me the logs to have a look at.

I also opened a PR for TF 2.10.1 and TF 2.11.0

The upstream issue I reported is tensorflow/tensorflow#59252

jfgrimm · 2023-02-02T13:58:08Z

yeah no idea what changed, but my builds (both CPU and CUDA) are happy now 🤷

VRehnberg · 2023-02-06T15:02:48Z

So if anyone (@VRehnberg ?) is seeing those MKL-related failures consistently, maybe he can run a modified version with
TF2.9_logging.patch.txt and send me the logs to have a look at.

I'm rerunning the builds now. Should be finished tomorrow and will try to remember to upload them then.

VRehnberg · 2023-02-07T10:50:11Z

Still getting MKL test failure:
easybuild-TensorFlow-2.9.1-20230206.161046.aaEEN.log.gz

Tried to include the logging patch with --amend, not sure if it worked.

Flamefire · 2023-02-07T14:14:36Z

@VRehnberg

Tried to include the logging patch with --amend, not sure if it worked.

Nope that didn't work. I guess yet another instance of easybuilders/easybuild-framework#3358 / easybuilders/easybuild-framework#2222

Best is to download the files and modify it directly. E.g. via eb --from-pr 17092 --copy-ec /tmp/foo --stop (places all files from the PR into /tmp/foo)
Not much hope I can figure out why it doesn't work for you though. But worth a shot.

VRehnberg · 2023-02-08T10:45:59Z

Second attempt at including patch went better, thanks for the tip for easily fetching the relevant files.
easybuild-TensorFlow-2.9.1-20230207.161810.rDwiP.log.gz

…foss-2022a-CUDA-11.7.0.eb and patches: TensorFlow-2.9.1_fix-PPC-Eigen-build.patch, TensorFlow-2.9.1_remove-duplicate-gpu-tests.patch, TensorFlow-2.9.1_remove-libclang-and-io-gcs-deps.patch, TensorFlow-2.9.1_support_flatbuffers_2.0.patch, TensorFlow-2.8.4_exclude-xnnpack-on-ppc.patch, TensorFlow-2.8.4_fix-PPC-JIT.patch, TensorFlow-2.8.4_resolve-gcc-symlinks.patch

Flamefire · 2023-05-11T12:57:57Z

@VRehnberg I think I found the issue: They were inconsistent checking for MKL and some place was using a runtime check for CPU features while another place was only checking for defines. This was fixed in TF 2.11 by tensorflow/tensorflow@5ec3d2e

I included the patch in the new commit, so that version should now build for you. Can you verify?

VRehnberg · 2023-05-11T13:21:14Z

A test report will hopefully come in a few hours with the results.

boegelbot · 2023-05-11T14:26:03Z

@Flamefire: Tests failed in GitHub Actions, see https://github.com/easybuilders/easybuild-easyconfigs/actions/runs/4948100242
Output from first failing test suite run:

FAIL: test_dep_versions_per_toolchain_generation (test.easyconfigs.easyconfigs.EasyConfigTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/easyconfigs/easyconfigs.py", line 884, in test_dep_versions_per_toolchain_generation
    self.assertFalse(multi_dep_vars, error_msg)
AssertionError: No multi-variant deps found for '^.*-(?P<tc_gen>20(1[89]|[2-9][0-9])[ab]).*\.eb$' easyconfigs:

found 2 variants of 'flatbuffers' dependency in easyconfigs using '2022a' toolchain generation
* version: 2.0.0; versionsuffix:  as dep for set(['TensorFlow-2.9.1-foss-2022a-CUDA-11.7.0.eb', 'TensorFlow-2.9.1-foss-2022a.eb'])
* version: 2.0.7; versionsuffix:  as dep for set(['AlphaFold-2.3.1-foss-2022a.eb', 'AlphaFold-2.3.4-foss-2022a-ColabFold.eb', 'ColabFold-1.5.2-foss-2022a-CUDA-11.7.0.eb', 'TensorFlow-2.11.0-foss-2022a-CUDA-11.7.0.eb', 'ColabFold-1.5.2-foss-2022a.eb', 'pod5-file-format-0.1.8-foss-2022a.eb', 'AlphaFold-2.3.4-foss-2022a-CUDA-11.7.0-ColabFold.eb', 'TensorFlow-2.11.0-foss-2022a.eb', 'M3GNet-0.2.4-foss-2022a.eb', 'AlphaFold-2.3.1-foss-2022a-CUDA-11.7.0.eb'])


----------------------------------------------------------------------
Ran 17002 tests in 748.353s

FAILED (failures=1)
ERROR: Not all tests were successful

bleep, bloop, I'm just a bot (boegelbot v20200716.01)
Please talk to my owner @boegel if you notice me acting stupid),
or submit a pull request to https://github.com/boegel/boegelbot fix the problem.

VRehnberg · 2023-05-11T17:52:48Z

Test report by @VRehnberg
FAILED
Build succeeded for 1 out of 2 (2 easyconfigs in total)
alvis-c1 - Linux Rocky Linux 8.6, x86_64, Intel Xeon Processor (Skylake), Python 3.6.8
See https://gist.github.com/VRehnberg/274b5ff0ebe9db379fb9dc8ac8465cb8 for a full test report.

Flamefire · 2023-05-12T06:56:36Z

@VRehnberg

fatal error: google/protobuf/port_def.inc: No such file or directory

That again? Are you using the latest easyblock? I.e. Easybuild 4.7.1 or easybuilders/easybuild-easyblocks#2854

Flamefire · 2023-05-12T11:56:02Z

It looks like some other ECs were merged already using flatbuffers 2.0.6 while TF 2.9 barely worked with 2.0.0. However I was able to backport the changes so it should work now.

Flamefire · 2023-05-12T13:32:48Z

Test report by @Flamefire
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
taurusi7005 - Linux CentOS Linux 7.7.1908, x86_64, AMD EPYC 7702 64-Core Processor (zen2), Python 2.7.5
See https://gist.github.com/Flamefire/45148c23fc982e579e8d154a16f55d52 for a full test report.

VRehnberg · 2023-05-25T14:52:38Z

I'm still seeing the MKL failure on this one. Though, TensorFlow-2.11.0 I don't think had this problem. At least I've installed the CUDA version of 2.11.0 on our systems since some time ago.

Log for latest failure:
easybuild-TensorFlow-2.9.1-20230522.170143.PAJqr.log.gz

Flamefire · 2023-06-20T10:44:44Z

@VRehnberg Looks like this PR wasn't merged yet and your log file looks like you didn't build from this PR so the required patch for the MKL failure isn't included.

TF 2.11 already includes the patch (upstream) but here my backport-patch is required

@boegel Can this be merged?

SebastianAchilles · 2023-06-22T01:09:10Z

Test report by @SebastianAchilles
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
skl-rockylinux-88 - Linux Rocky Linux 8.8, x86_64, Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz (skylake), 1 x NVIDIA NVIDIA RTX A4000, 530.30.02, Python 3.6.8
See https://gist.github.com/SebastianAchilles/11e00f03f70b02af5a0707d2b2a499ee for a full test report.

SebastianAchilles

lgtm

SebastianAchilles · 2023-06-22T06:50:40Z

Going in, thanks @Flamefire!

Failed with HTTP Error

…asyconfigs into 20230112164524_new_pr_TensorFlow291

SebastianAchilles · 2023-06-22T07:18:24Z

Going in, thanks @Flamefire!

boegel added the update label Jan 12, 2023

boegel added this to the next release (4.7.1?) milestone Jan 12, 2023

Flamefire mentioned this pull request Jan 15, 2023

fix build of TensorFlow 2.5+ on aarch64 #17101

Merged

Flamefire mentioned this pull request Jan 20, 2023

{lib}[foss/2022a] TensorFlow 2.9.1 w/ Bazel 5.0.0 #17169

Closed

Flamefire mentioned this pull request Feb 7, 2023

{lib}[foss/2022a] TensorFlow v2.11.0 w/ Python 3.10.4 (+ CUDA 11.7.0) #17241

Merged

2 tasks

Flamefire force-pushed the 20230112164524_new_pr_TensorFlow291 branch from dacfb36 to af2b983 Compare February 10, 2023 11:24

VRehnberg mentioned this pull request Feb 15, 2023

{lib}[foss/2022a] TensorFlow v2.9.1 w/ Python 3.10.4 + CUDA 11.7.0 #16620

Closed

easybuilders deleted a comment from boegelbot Mar 2, 2023

boegel mentioned this pull request Mar 3, 2023

AlphaFold (2.3.x) vscentrum/vsc-software-stack#100

Closed

boegel modified the milestones: 4.7.1, release after 4.7.1 Mar 17, 2023

boegel modified the milestones: next release (4.7.2), release after 4.7.2 Apr 12, 2023

Fix MKL related test failures

3317a0d

Flamefire force-pushed the 20230112164524_new_pr_TensorFlow291 branch from b0583d6 to 3317a0d Compare May 11, 2023 12:46

Update flatbuffers dependency

7a40ad6

Add patch for updated flatbuffers

95a9189

Flamefire mentioned this pull request Jun 20, 2023

{lib}[foss/2022a] TensorFlow v2.10.1 w/ Python 3.10.4 #17168

Closed

SebastianAchilles approved these changes Jun 22, 2023

View reviewed changes

Merge branch 'develop' of https://github.com/easybuilders/easybuild-e…

b29771d

…asyconfigs into 20230112164524_new_pr_TensorFlow291

SebastianAchilles merged commit 7d99e5f into easybuilders:develop Jun 22, 2023
5 checks passed

Flamefire deleted the 20230112164524_new_pr_TensorFlow291 branch June 22, 2023 10:32

branfosj mentioned this pull request Jun 29, 2023

{lib}[foss/2022a] Tensorflow v2.9.1 #16008

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{lib}[foss/2022a] TensorFlow v2.9.1 w/ Python 3.10.4 #17092

{lib}[foss/2022a] TensorFlow v2.9.1 w/ Python 3.10.4 #17092

Flamefire commented Jan 12, 2023 •

edited

Loading

jfgrimm commented Jan 12, 2023

Flamefire commented Jan 13, 2023

jfgrimm commented Jan 13, 2023

Flamefire commented Jan 13, 2023

Flamefire commented Jan 16, 2023

Flamefire commented Jan 16, 2023

Flamefire commented Jan 16, 2023

VRehnberg commented Jan 19, 2023

Flamefire commented Jan 19, 2023

SebastianAchilles commented Jan 23, 2023

Flamefire commented Feb 1, 2023

jfgrimm commented Feb 2, 2023

VRehnberg commented Feb 6, 2023

VRehnberg commented Feb 7, 2023

Flamefire commented Feb 7, 2023

VRehnberg commented Feb 8, 2023

Flamefire commented May 11, 2023

VRehnberg commented May 11, 2023

boegelbot commented May 11, 2023

VRehnberg commented May 11, 2023

Flamefire commented May 12, 2023 •

edited

Loading

Flamefire commented May 12, 2023

Flamefire commented May 12, 2023

VRehnberg commented May 25, 2023 •

edited

Loading

Flamefire commented Jun 20, 2023

SebastianAchilles commented Jun 22, 2023

SebastianAchilles left a comment

SebastianAchilles commented Jun 22, 2023 •

edited

Loading

SebastianAchilles commented Jun 22, 2023

{lib}[foss/2022a] TensorFlow v2.9.1 w/ Python 3.10.4 #17092

{lib}[foss/2022a] TensorFlow v2.9.1 w/ Python 3.10.4 #17092

Conversation

Flamefire commented Jan 12, 2023 • edited Loading

jfgrimm commented Jan 12, 2023

Flamefire commented Jan 13, 2023

jfgrimm commented Jan 13, 2023

Flamefire commented Jan 13, 2023

Flamefire commented Jan 16, 2023

Flamefire commented Jan 16, 2023

Flamefire commented Jan 16, 2023

VRehnberg commented Jan 19, 2023

Flamefire commented Jan 19, 2023

SebastianAchilles commented Jan 23, 2023

Flamefire commented Feb 1, 2023

jfgrimm commented Feb 2, 2023

VRehnberg commented Feb 6, 2023

VRehnberg commented Feb 7, 2023

Flamefire commented Feb 7, 2023

VRehnberg commented Feb 8, 2023

Flamefire commented May 11, 2023

VRehnberg commented May 11, 2023

boegelbot commented May 11, 2023

VRehnberg commented May 11, 2023

Flamefire commented May 12, 2023 • edited Loading

Flamefire commented May 12, 2023

Flamefire commented May 12, 2023

VRehnberg commented May 25, 2023 • edited Loading

Flamefire commented Jun 20, 2023

SebastianAchilles commented Jun 22, 2023

SebastianAchilles left a comment

Choose a reason for hiding this comment

SebastianAchilles commented Jun 22, 2023 • edited Loading

SebastianAchilles commented Jun 22, 2023

Flamefire commented Jan 12, 2023 •

edited

Loading

Flamefire commented May 12, 2023 •

edited

Loading

VRehnberg commented May 25, 2023 •

edited

Loading

SebastianAchilles commented Jun 22, 2023 •

edited

Loading