Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated build recipe and patches to ray-1.3.0 #14

Merged
merged 8 commits into from
Jun 22, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 22 additions & 22 deletions recipe/meta.yaml
Original file line number Diff line number Diff line change
@@ -1,31 +1,30 @@
{% set version = "1.2.0" %}
{% set version = "1.3.0" %}

package:
name: ray-packages
version: {{ version }}

source:
url: https://github.com/ray-project/ray/archive/ray-{{ version }}.tar.gz
sha256: e4c9aa0cbfecf4d1112a43de473606a81f6fbba5c3f0b3013644268b366e2b75
sha256: 66c65a6b7f5838726c908ebcc9d6c985f59ddbb0f22a700730422dde997d48d2
patches:
- patches/0001-Do-not-force-pickle5-in-sys.path.patch
- patches/0002-Fix-redis-build-for-non-default-compiler-drivers.patch
- patches/0003-Fix-plasma-linking-for-ancient-glibc.patch
- patches/0004-Redis-deps-now-build-but-do-not-link.patch
- patches/0005-Disable-making-non-core-entry-scripts.patch
- patches/0006-Ignore-warnings-for-one-file.patch
- patches/0007-Contain-bazel-root-and-output-dir-right-above-SRC_DI.patch
- patches/0008-Do-not-crash-if-BAZEL_SH-not-set-on-Windows.patch
- patches/0009-Convert-symlinks-to-junctions-on-Windows-before-buil.patch
- patches/0010-Add-workaround-for-os.path.isdir-on-Windows.patch
- patches/0011-Include-process.h-for-getpid-explicitly-on-Windows.patch
- patches/0012-Empty-install-requirements-installed-by-conda-build.patch
- patches/0013-permit-python-3.9.patch
- patches/0014-Update-redis-patch-for-redis-6.0.9.patch
- patches/0015-Ignore-warnings-in-logging.cc.patch
- patches/0006-Contain-bazel-root-and-output-dir-right-above-SRC_DI.patch
- patches/0007-Do-not-crash-if-BAZEL_SH-not-set-on-Windows.patch
- patches/0008-Convert-symlinks-to-junctions-on-Windows-before-buil.patch
- patches/0009-Add-workaround-for-os.path.isdir-on-Windows.patch
- patches/0010-Include-process.h-for-getpid-explicitly-on-Windows.patch
- patches/0011-Empty-install-requirements-installed-by-conda-build.patch
- patches/0012-permit-python-3.9.patch
- patches/0013-Ignore-warnings-in-event.cc-and-logging.cc.patch
- patches/0014-Disable-runfiles-on-windows.patch

build:
number: 1
number: 0
skip: true # [py<36]
# skip on MacOS as there's some weird compilation issue and I have no MacOS to develop on
skip: true # [osx]
Expand All @@ -35,8 +34,7 @@ requirements:
build:
- {{ compiler('c') }}
- {{ compiler('cxx') }}
- bazel <=3.4.1 # [not win]
- bazel <3.7 # [win]
- bazel <=3.7.2
- cython >=0.29
- curl
- make
Expand Down Expand Up @@ -91,6 +89,7 @@ outputs:
- click >=7.0
- colorama
- colorful
- dataclasses # [py<37]
- filelock
# gpustat-0.6.0 has a dependency which does not exist on Windows;
# skip it there until gpustat is fixed as it is optional
Expand All @@ -102,7 +101,7 @@ outputs:
- opencensus
- pickle5 # [py<38]
- prometheus_client >=0.7.1
- protobuf >=3.8.0
- protobuf >=3.15.3
- psutil
- pyyaml
- requests
Expand All @@ -120,7 +119,7 @@ outputs:
# init-code seemingly depends on platform or other ambient things;
# the following doesn't get triggered in CI, but was a problem in
# https://github.com/conda-forge/ray-packages-feedstock/issues/16
- ray.metrics_agent
- ray._private.metrics_agent
commands:
- python -c "import ray; ray.init()"

Expand Down Expand Up @@ -219,18 +218,20 @@ outputs:
- rllib --help

- name: ray-serve
build:
entry_points:
- serve = ray.serve.scripts:cli
requirements:
host:
- python
run:
- python
- {{ pin_subpackage('ray-core', exact=True) }}
- dataclasses # [py<37]
- flask
- pydantic <1.7
- uvicorn
- requests
- pydantic >=1.8
- starlette
- uvicorn
- fastapi
test:
imports:
- ray.serve
Expand All @@ -245,7 +246,6 @@ outputs:
run:
- python
- {{ pin_subpackage('ray-core', exact=True) }}
- dataclasses # [py<37]
- pandas
- tabulate
- tensorboardX
Expand Down
8 changes: 4 additions & 4 deletions recipe/patches/0001-Do-not-force-pickle5-in-sys.path.patch
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
From 2a2d816d4232d3f921a71d6edfe2f1cd04e2e3dc Mon Sep 17 00:00:00 2001
From 8a560449e6668c9aeea9d1a1efbfaa63c2639b08 Mon Sep 17 00:00:00 2001
From: Vasily Litvinov <vasilij.n.litvinov@intel.com>
Date: Thu, 5 Nov 2020 12:04:25 +0300
Subject: [PATCH 01/15] Do not force pickle5 in sys.path
Subject: [PATCH 01/14] Do not force pickle5 in sys.path

Signed-off-by: Vasily Litvinov <vasilij.n.litvinov@intel.com>
---
python/ray/__init__.py | 6 ------
1 file changed, 6 deletions(-)

diff --git a/python/ray/__init__.py b/python/ray/__init__.py
index f1be50dce..712626501 100644
index 18315b2da..facc91451 100644
--- a/python/ray/__init__.py
+++ b/python/ray/__init__.py
@@ -31,12 +31,6 @@ if "OMP_NUM_THREADS" not in os.environ:
Expand All @@ -26,5 +26,5 @@ index f1be50dce..712626501 100644
thirdparty_files = os.path.join(
os.path.abspath(os.path.dirname(__file__)), "thirdparty_files")
--
2.11.0
2.28.0

Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From aec3509a5681e75cd674ac8ca7f0af3a698ca47f Mon Sep 17 00:00:00 2001
From 1ed5ff57c5367303479f117dea72eabefefa1946 Mon Sep 17 00:00:00 2001
From: Vasily Litvinov <vasilij.n.litvinov@intel.com>
Date: Thu, 5 Nov 2020 17:05:44 +0300
Subject: [PATCH 02/15] Fix redis build for non-default compiler drivers
Subject: [PATCH 02/14] Fix redis build for non-default compiler drivers

Signed-off-by: Vasily Litvinov <vasilij.n.litvinov@intel.com>
---
Expand Down Expand Up @@ -29,5 +29,5 @@ index f428f7b36..9ec69d433 100644

# This library is for internal hiredis use, because hiredis assumes a
--
2.11.0
2.28.0

Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
From 6ea89bff1595d3d48c285c0569abae42966473f0 Mon Sep 17 00:00:00 2001
From 092468d20a20d4ee8da4d256ab3aa2b53645c4ca Mon Sep 17 00:00:00 2001
From: Vasily Litvinov <vasilij.n.litvinov@intel.com>
Date: Thu, 5 Nov 2020 17:09:46 +0300
Subject: [PATCH 03/15] Fix plasma linking for ancient glibc
Subject: [PATCH 03/14] Fix plasma linking for ancient glibc

Signed-off-by: Vasily Litvinov <vasilij.n.litvinov@intel.com>
---
BUILD.bazel | 2 ++
1 file changed, 2 insertions(+)

diff --git a/BUILD.bazel b/BUILD.bazel
index c1745e468..fbe66124e 100644
index 879633444..0463f7c9d 100644
--- a/BUILD.bazel
+++ b/BUILD.bazel
@@ -223,6 +223,7 @@ PLASMA_LINKOPTS = [] + select({
Expand All @@ -29,5 +29,5 @@ index c1745e468..fbe66124e 100644
deps = [
":plasma_store_server_lib",
--
2.11.0
2.28.0

87 changes: 45 additions & 42 deletions recipe/patches/0004-Redis-deps-now-build-but-do-not-link.patch
Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
From 6e739a9dc46bae93dc71b4d61f949a15ff9a3af4 Mon Sep 17 00:00:00 2001
From 7214b9b51a29acc08e6d1d0cffaa155209d68f7e Mon Sep 17 00:00:00 2001
From: Vasily Litvinov <vasilij.n.litvinov@intel.com>
Date: Fri, 6 Nov 2020 00:55:05 +0300
Subject: [PATCH 04/15] Redis deps now build but do not link
Subject: [PATCH 04/14] Redis deps now build but do not link

Includes update for redis-6.0.9

Signed-off-by: Vasily Litvinov <vasilij.n.litvinov@intel.com>
Signed-off-by: Gregory Shimansky <gregory.shimansky@intel.com>
---
bazel/BUILD.redis | 12 +++++-
bazel/BUILD.redis | 12 ++++-
bazel/ray_deps_setup.bzl | 1 +
thirdparty/patches/redis-deps-ar.patch | 74 ++++++++++++++++++++++++++++++++++
thirdparty/patches/redis-deps-ar.patch | 74 ++++++++++++++++++++++++++
3 files changed, 85 insertions(+), 2 deletions(-)
create mode 100644 thirdparty/patches/redis-deps-ar.patch

Expand Down Expand Up @@ -47,11 +50,11 @@ index 9ec69d433..06d79b72a 100644

# This library is for internal hiredis use, because hiredis assumes a
diff --git a/bazel/ray_deps_setup.bzl b/bazel/ray_deps_setup.bzl
index c398dd15d..9b77e4f39 100644
index fcef5edee..d7ffa20a0 100644
--- a/bazel/ray_deps_setup.bzl
+++ b/bazel/ray_deps_setup.bzl
@@ -84,6 +84,7 @@ def ray_deps_setup():
sha256 = "2819b6d9c56be1f25cd157b9cb6b7c2733edcb46f4f6bcb1b79cefe639a2853b",
sha256 = "900cb82227bac58242c9b7668e7113cd952253b256fe04bbdab1b78979cf255a",
patches = [
"//thirdparty/patches:redis-quiet.patch",
+ "//thirdparty/patches:redis-deps-ar.patch",
Expand All @@ -60,76 +63,73 @@ index c398dd15d..9b77e4f39 100644

diff --git a/thirdparty/patches/redis-deps-ar.patch b/thirdparty/patches/redis-deps-ar.patch
new file mode 100644
index 000000000..15acb3165
index 000000000..36497c4b6
--- /dev/null
+++ b/thirdparty/patches/redis-deps-ar.patch
@@ -0,0 +1,74 @@
+From 3b634ca40c7876d58d571a6d99780558c791d5e3 Mon Sep 17 00:00:00 2001
+From: Vasily Litvinov <vasilij.n.litvinov@intel.com>
+Date: Thu, 11 Mar 2021 13:09:26 +0300
+Subject: [PATCH] Enable overriding AR and RANLIB
+
+Signed-off-by: Vasily Litvinov <vasilij.n.litvinov@intel.com>
+---
+ deps/Makefile | 9 +++++----
+ src/Makefile | 5 ++++-
+ 2 files changed, 9 insertions(+), 5 deletions(-)
+
+diff --git deps/Makefile deps/Makefile
+index 6865ee655..3c3f18d68 100644
+index 9952b80d8..b8dea748f 100644
+--- deps/Makefile
++++ deps/Makefile
+@@ -8,7 +8,7 @@ SRCCOLOR="\033[33m"
+@@ -8,6 +8,9 @@ SRCCOLOR="\033[33m"
+ BINCOLOR="\033[37;1m"
+ MAKECOLOR="\033[32;1m"
+ ENDCOLOR="\033[0m"
+-
++AR=ar
++ARFLAGS=rcu
++RANLIB=ranlib
+
+ default:
+ @echo "Explicit target required"
+
+@@ -43,7 +43,7 @@ distclean:
+@@ -47,7 +50,7 @@ endif
+
+ hiredis: .make-prerequisites
+ # @printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
+- cd hiredis && $(MAKE) static
++ cd hiredis && $(MAKE) static AR="$(AR)"
+ #@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
+- cd hiredis && $(MAKE) static $(HIREDIS_MAKE_FLAGS)
++ cd hiredis && $(MAKE) static $(HIREDIS_MAKE_FLAGS) AR="$(AR)"
+
+ .PHONY: hiredis
+
+@@ -63,12 +63,12 @@ LUA_LDFLAGS+= $(LDFLAGS)
+@@ -67,12 +70,10 @@ LUA_LDFLAGS+= $(LDFLAGS)
+ # lua's Makefile defines AR="ar rcu", which is unusual, and makes it more
+ # challenging to cross-compile lua (and redis). These defines make it easier
+ # to fit redis into cross-compilation environments, which typically set AR.
+-AR=ar
++RANLIB=ranlib
+ ARFLAGS=rc
+-ARFLAGS=rcu
+
+ lua: .make-prerequisites
+ # @printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
+ #@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
+- cd lua/src && $(MAKE) all CFLAGS="$(LUA_CFLAGS)" MYLDFLAGS="$(LUA_LDFLAGS)" AR="$(AR) $(ARFLAGS)"
++ cd lua/src && $(MAKE) all CFLAGS="$(LUA_CFLAGS)" MYLDFLAGS="$(LUA_LDFLAGS)" AR="$(AR) $(ARFLAGS)" RANLIB="$(RANLIB)"
+
+ .PHONY: lua
+
+diff --git deps/hiredis/Makefile deps/hiredis/Makefile
+index 9a4de8360..1f8427f75 100644
+--- deps/hiredis/Makefile
++++ deps/hiredis/Makefile
+@@ -51,7 +51,8 @@ DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR)
+ DYLIBNAME=$(LIBNAME).$(DYLIBSUFFIX)
+ DYLIB_MAKE_CMD=$(CC) -shared -Wl,-soname,$(DYLIB_MINOR_NAME) -o $(DYLIBNAME) $(LDFLAGS)
+ STLIBNAME=$(LIBNAME).$(STLIBSUFFIX)
+-STLIB_MAKE_CMD=ar rcs $(STLIBNAME)
++AR=ar
++STLIB_MAKE_CMD=$(AR) rcs $(STLIBNAME)
+
+ # Platform-specific overrides
+ uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
+diff --git src/Makefile src/Makefile
+index 0e97757c4..68d9acb2b 100644
+index c06214105..1e0158b56 100644
+--- src/Makefile
++++ src/Makefile
+@@ -154,6 +154,9 @@ ifeq ($(MALLOC),jemalloc)
+ FINAL_LIBS := ../deps/jemalloc/lib/libjemalloc.a $(FINAL_LIBS)
+ endif
+@@ -237,6 +237,9 @@ BINCOLOR="\033[37;1m"
+ MAKECOLOR="\033[32;1m"
+ ENDCOLOR="\033[0m"
+
++AR=ar
++RANLIB=ranlib
++
+ REDIS_CC=$(QUIET_CC)$(CC) $(FINAL_CFLAGS)
+ REDIS_LD=$(QUIET_LINK)$(CC) $(FINAL_LDFLAGS)
+ REDIS_INSTALL=$(QUIET_INSTALL)$(INSTALL)
+@@ -206,7 +209,7 @@ persist-settings: distclean
+ ifndef V
+ QUIET_CC = @printf ' %b %b\n' $(CCCOLOR)CC$(ENDCOLOR) $(SRCCOLOR)$@$(ENDCOLOR) 1>&2;
+ QUIET_LINK = @printf ' %b %b\n' $(LINKCOLOR)LINK$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR) 1>&2;
+@@ -280,7 +283,7 @@ persist-settings: distclean
+ echo REDIS_LDFLAGS=$(REDIS_LDFLAGS) >> .make-settings
+ echo PREV_FINAL_CFLAGS=$(FINAL_CFLAGS) >> .make-settings
+ echo PREV_FINAL_LDFLAGS=$(FINAL_LDFLAGS) >> .make-settings
Expand All @@ -138,6 +138,9 @@ index 000000000..15acb3165
+
+ .PHONY: persist-settings
+
+--
+2.11.0
+
--
2.11.0
2.28.0

20 changes: 10 additions & 10 deletions recipe/patches/0005-Disable-making-non-core-entry-scripts.patch
Original file line number Diff line number Diff line change
@@ -1,31 +1,31 @@
From 63830f8977e09fde3e70b42f01b0acc2c446a7dd Mon Sep 17 00:00:00 2001
From 82817f12571058d235ca813e77a96cf2fc24f411 Mon Sep 17 00:00:00 2001
From: Vasily Litvinov <vasilij.n.litvinov@intel.com>
Date: Tue, 10 Nov 2020 23:26:35 +0300
Subject: [PATCH 05/15] Disable making non-core entry scripts
Subject: [PATCH 05/14] Disable making non-core entry scripts

---
python/setup.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/python/setup.py b/python/setup.py
index a1542a7a2..5f68ae765 100644
index 0575ad3a1..8ed7e07a9 100644
--- a/python/setup.py
+++ b/python/setup.py
@@ -447,10 +447,10 @@ setuptools.setup(
@@ -442,10 +442,10 @@ setuptools.setup(
entry_points={
"console_scripts": [
"ray=ray.scripts.scripts:main",
- "rllib=ray.rllib.scripts:cli [rllib]",
- "tune=ray.tune.scripts:cli",
- "ray-operator=ray.operator.operator:main",
- "ray-operator=ray.ray_operator.operator:main",
- "serve=ray.serve.scripts:cli",
+ # "rllib=ray.rllib.scripts:cli [rllib]",
+ # "tune=ray.tune.scripts:cli",
+ # "ray-operator=ray.operator.operator:main",
+ # "serve=ray.serve.scripts:cli",
+ #"rllib=ray.rllib.scripts:cli [rllib]",
+ #"tune=ray.tune.scripts:cli",
+ #"ray-operator=ray.ray_operator.operator:main",
+ #"serve=ray.serve.scripts:cli",
Comment on lines +22 to +25
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel we should be setting those entry points in the appropriate sub-packages.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either that (if they need special requirements and those subpackages provide those) or allow them be in ray-core

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch is taken verbatim from ray-1.2.0, no changes done to how these subpackages were handled in version 1.2.0
https://github.com/conda-forge/ray-packages-feedstock/blob/master/recipe/patches/0005-Disable-making-non-core-entry-scripts.patch

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added serve entry point to ray-serve. I suggest keeping ray-operator removed for now as it's unclear to which package it should belong. We can wait for someone coming and raising an issue I think :)

]
},
include_package_data=True,
--
2.11.0
2.28.0

Loading