Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Stop compiling embedded go components #3520

Merged
merged 7 commits into from
Mar 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 0 additions & 54 deletions .github/workflows/build_wheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -72,16 +72,6 @@ jobs:
CIBW_BUILD: "cp3*_x86_64"
CIBW_SKIP: "cp36-* cp37-* *-musllinux_x86_64 cp310-macosx_x86_64"
CIBW_ARCHS: "native"
CIBW_ENVIRONMENT: >
COMPILE_GO=True PATH=$PATH:/usr/local/go/bin
CIBW_BEFORE_ALL_LINUX: |
curl -o go.tar.gz https://dl.google.com/go/go1.18.2.linux-amd64.tar.gz
tar -C /usr/local -xzf go.tar.gz
go version
yum -y update &&
yum install -y epel-release || yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1).noarch.rpm &&
yum install -y https://apache.jfrog.io/artifactory/arrow/centos/$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1)/apache-arrow-release-latest.rpm &&
yum install -y --enablerepo=epel arrow-devel # For C++
CIBW_BEFORE_ALL_MACOS: |
brew install apache-arrow
brew install pkg-config
Expand All @@ -90,8 +80,6 @@ jobs:
# There's a `git restore` in here because `make install-go-ci-dependencies` is actually messing up go.mod & go.sum.
CIBW_BEFORE_BUILD: |
make install-protoc-dependencies
make install-go-proto-dependencies
make install-go-ci-dependencies
git status
git restore go.mod go.sum
git restore sdk/python/feast/ui/yarn.lock
Expand Down Expand Up @@ -139,8 +127,6 @@ jobs:
run: |
pip install -U pip setuptools wheel twine
make install-protoc-dependencies
make install-go-proto-dependencies
make install-go-ci-dependencies
make build-ui
git status
git restore go.mod go.sum
Expand Down Expand Up @@ -203,9 +189,6 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- uses: actions/setup-go@v3
with:
go-version: '>=1.17.0'
- uses: actions/download-artifact@v2
with:
name: wheels
Expand All @@ -217,33 +200,6 @@ jobs:
cd dist/
pip install wheel
for f in *.whl; do pip install $f || true; done
- name: Install apache-arrow on ubuntu
if: ${{ matrix.from-source && matrix.os == 'ubuntu-latest' }}
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
- name: Install apache-arrow on macos
if: ${{ matrix.from-source && matrix.os == 'macos-10.15' && matrix.python-version != '3.10' }}
run: |
brew install apache-arrow
brew install pkg-config
- name: Install dist with go
if: ${{ matrix.from-source && (matrix.python-version != '3.10' || matrix.os == 'ubuntu-latest')}}
env:
COMPILE_GO: "True"
run: |
pip install 'grpcio-tools==1.47.0' 'pybindgen==0.22.0'
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26.0
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1.0
pip install dist/*tar.gz
# py3.10 on MacOS does not work with Go so we have to install separately. Issue is tracked here: https://github.com/feast-dev/feast/issues/2881
- name: Install dist w/o go
if: ${{ matrix.from-source && matrix.python-version == '3.10' && matrix.os == 'macos-10.15'}}
run: pip install dist/*tar.gz
- name: Install OS X dependencies
if: matrix.os == 'macos-10.15'
run: brew install coreutils
Expand All @@ -269,13 +225,3 @@ jobs:
echo "$TEST_SCRIPT" > run-and-wait.sh
bash run-and-wait.sh feast serve
bash run-and-wait.sh feast ui
# We disable this test for the Python 3.10 binary since it does not include Go.
- name: Smoke test with go
if: matrix.python-version != '3.10' || matrix.os == 'ubuntu-latest'
run: |
cd test_repo/feature_repo
feast apply
echo "$TEST_SCRIPT" > run-and-wait.sh
pip install cffi
printf "\ngo_feature_serving: True" >> feature_store.yaml
bash run-and-wait.sh feast serve
14 changes: 1 addition & 13 deletions .github/workflows/java_master_only.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,6 @@ jobs:
with:
python-version: 3.8
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.18.0
- name: Upgrade pip version
run: |
pip install --upgrade pip
Expand All @@ -136,14 +131,7 @@ jobs:
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev

- name: Install Python dependencies
run: make install-python-ci-dependencies
- uses: actions/cache@v2
Expand Down
13 changes: 0 additions & 13 deletions .github/workflows/java_pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -148,11 +148,6 @@ jobs:
with:
python-version: 3.8
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.18.0
- name: Upgrade pip version
run: |
pip install --upgrade pip
Expand All @@ -172,14 +167,6 @@ jobs:
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
- name: Install Python dependencies
run: make install-python-ci-dependencies
- name: Run integration tests
Expand Down
42 changes: 0 additions & 42 deletions .github/workflows/linter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,6 @@ jobs:
with:
python-version: "3.8"
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.18.0
- name: Upgrade pip version
run: |
pip install --upgrade pip
Expand All @@ -39,45 +34,8 @@ jobs:
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
- name: Install dependencies
run: |
make compile-protos-go
make install-python-ci-dependencies
- name: Lint python
run: make lint-python

lint-go:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.18.0
- name: Setup Python
id: setup-python
uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: Upgrade pip version
run: |
pip install --upgrade pip
- name: Install apache-arrow on ubuntu
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
- name: Lint go
run: make lint-go
22 changes: 1 addition & 21 deletions .github/workflows/master_only.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,15 +58,14 @@ jobs:
docker push $ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }}
outputs:
DOCKER_IMAGE_TAG: ${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }}
integration-test-python-and-go:
integration-test-python:
if: github.repository == 'feast-dev/feast'
needs: build-lambda-docker-image
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: [ "3.8", "3.9", "3.10" ]
go-version: [ 1.17.0 ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand All @@ -89,11 +88,6 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go-version }}
- name: Authenticate to Google Cloud
uses: 'google-github-actions/auth@v1'
with:
Expand Down Expand Up @@ -131,20 +125,6 @@ jobs:
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
- name: Install apache-arrow on macos
if: matrix.os == 'macOS-latest'
run: |
brew install apache-arrow
brew install pkg-config
- name: Install dependencies
run: make install-python-ci-dependencies
- name: Setup Redis Cluster
Expand Down
19 changes: 0 additions & 19 deletions .github/workflows/pr_integration_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,11 +115,6 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.18.0
- name: Authenticate to Google Cloud
uses: 'google-github-actions/auth@v1'
with:
Expand Down Expand Up @@ -157,20 +152,6 @@ jobs:
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
- name: Install apache-arrow on macos
if: matrix.os == 'macOS-latest'
run: |
brew install apache-arrow
brew install pkg-config
- name: Install dependencies
run: make install-python-ci-dependencies
- name: Setup Redis Cluster
Expand Down
9 changes: 0 additions & 9 deletions .github/workflows/pr_local_integration_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,6 @@ jobs:
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
- name: Install dependencies
run: make install-python-ci-dependencies
- name: Test local integration tests
Expand Down
47 changes: 0 additions & 47 deletions .github/workflows/unit_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,6 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.18.0
- name: Install mysql on macOS
if: startsWith(matrix.os, 'macOS')
run: |
Expand All @@ -54,53 +49,11 @@ jobs:
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
- name: Install apache-arrow on macos
if: matrix.os == 'macOS-latest'
run: |
brew install apache-arrow
brew install pkg-config
- name: Install dependencies
run: make install-python-ci-dependencies
- name: Test Python
run: pytest -n 8 --cov=./ --cov-report=xml --color=yes sdk/python/tests

unit-test-go:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Setup Python
id: setup-python
uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=22.1,<23"
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.18.0
- name: Install apache-arrow on ubuntu
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
sudo apt install -y -V pkg-config
- name: Test
run: make test-go

unit-test-ui:
runs-on: ubuntu-latest
Expand Down
Loading