Skip to content

Commit 07cd8b4

Browse files
jamesx-improvingalexr-bqalexey-temnikov
authored
Go: Add MUSL support (#4476)
* Go: Switch to MUSL binary Signed-off-by: James Xin <james.xin@improving.com> Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * rustup add musl targets Signed-off-by: James Xin <james.xin@improving.com> Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * lint Signed-off-by: James Xin <james.xin@improving.com> Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * rustup add musl targets in Makefile Signed-off-by: James Xin <james.xin@improving.com> Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Add pathing Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Trigger CodeQL on main Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update CI Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Fix typo Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Fix typo Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Modify CD Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Modify CD Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Modify CD Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Modify CD Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update docs Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Format Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Fix Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Fix Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Address feedback Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> * Update doc Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> --------- Signed-off-by: James Xin <james.xin@improving.com> Signed-off-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> Co-authored-by: Alex Rehnby-Martin <alex.rehnby-martin@improving.com> Co-authored-by: Alexey Temnikov <alexey.temnikov@improving.com>
1 parent 36ec725 commit 07cd8b4

File tree

9 files changed

+89
-27
lines changed

9 files changed

+89
-27
lines changed

‎.github/json_matrices/build-matrix.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@
4949
"CD_RUNNER": "ubuntu-24.04-arm",
5050
"IMAGE": "node:lts-alpine",
5151
"CONTAINER_OPTIONS": "--user root --privileged --rm",
52-
"PACKAGE_MANAGERS": ["npm", "maven"],
52+
"PACKAGE_MANAGERS": ["npm", "maven", "pkg_go_dev"],
5353
"languages": []
5454
},
5555
{
@@ -61,8 +61,8 @@
6161
"RUNNER": "ubuntu-latest",
6262
"IMAGE": "node:lts-alpine",
6363
"CONTAINER_OPTIONS": "--user root --privileged",
64-
"PACKAGE_MANAGERS": ["npm", "maven"],
65-
"languages": ["node", "java"]
64+
"PACKAGE_MANAGERS": ["npm", "maven", "pkg_go_dev"],
65+
"languages": ["node", "java", "go"]
6666
},
6767
{
6868
"OS": "amazon-linux",

‎.github/workflows/go-cd.yml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -116,25 +116,29 @@ jobs:
116116
uses: ./.github/workflows/install-shared-dependencies
117117
with:
118118
os: ${{ matrix.host.OS }}
119-
target: ${{ matrix.host.TARGET }}
119+
target: ${{ matrix.host.CD_TARGET || matrix.host.TARGET }}
120120
github-token: ${{ secrets.GITHUB_TOKEN }}
121121
- name: Build Go client
122122
working-directory: go
123123
env:
124124
RELEASE_VERSION: ${{ needs.validate-release-version.outputs.RELEASE_VERSION }}
125125
run: |
126126
make install-build-tools
127-
make build GLIDE_VERSION="${RELEASE_VERSION}"
127+
make build GLIDE_VERSION="${RELEASE_VERSION}" TARGET_TRIPLET="${{ matrix.host.TARGET }}"
128128
- name: Move FFI artifacts on linux
129129
if: ${{ contains(matrix.host.TARGET, 'linux-gnu') }}
130130
run: |
131131
mkdir -p $GITHUB_WORKSPACE/ffi/target/release
132132
cp ffi/target/*/release/libglide_ffi.a $GITHUB_WORKSPACE/ffi/target/release/
133+
- name: Set artifact name
134+
id: set-artifact-name
135+
run: |
136+
echo "ARTIFACT_NAME=${{ matrix.host.TARGET }}" >> $GITHUB_OUTPUT
133137
- name: Upload artifacts
134138
continue-on-error: true
135139
uses: actions/upload-artifact@v4
136140
with:
137-
name: ${{ matrix.host.TARGET }}
141+
name: ${{ steps.set-artifact-name.outputs.artifact-name }}
138142
path: |
139143
ffi/target/release/libglide_ffi.a
140144
go/lib.h

‎.github/workflows/go.yml‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -255,10 +255,17 @@ jobs:
255255
steps:
256256
- name: Install git
257257
run: |
258-
yum update
259-
yum install -y git tar
260-
git config --global --add safe.directory "$GITHUB_WORKSPACE"
261-
echo IMAGE=amazonlinux:latest | sed -r 's/:/-/g' >> $GITHUB_ENV
258+
if [[ "${{ matrix.host.OS }}" == "amazon-linux" ]]; then
259+
yum update
260+
yum install -y git tar
261+
git config --global --add safe.directory "$GITHUB_WORKSPACE"
262+
echo IMAGE=amazonlinux:latest | sed -r 's/:/-/g' >> $GITHUB_ENV
263+
elif [[ "${{ matrix.host.TARGET }}" == *"musl"* ]]; then
264+
apk update
265+
apk add git bash tar
266+
git config --global --add safe.directory "$GITHUB_WORKSPACE"
267+
fi
268+
262269
# Replace `:` in the variable otherwise it can't be used in `upload-artifact`
263270
- uses: actions/checkout@v4
264271
with:
@@ -278,6 +285,19 @@ jobs:
278285
github-token: ${{ secrets.GITHUB_TOKEN }}
279286
engine-version: ${{ matrix.engine.version }}
280287

288+
- name: Install Musl Dependencies
289+
if: ${{ contains(matrix.host.TARGET, 'musl') }}
290+
run: |
291+
apk add clang lld compiler-rt
292+
# Ensure Rust is in PATH for container environments
293+
echo "$HOME/.cargo/bin:$PATH" >> $GITHUB_PATH
294+
295+
- name: Install zig
296+
if: ${{ contains(matrix.host.TARGET, 'musl') }}
297+
uses: ./.github/workflows/install-zig
298+
with:
299+
target: ${{ matrix.host.TARGET }}
300+
281301
- uses: actions/cache@v4
282302
with:
283303
path: |

‎.github/workflows/install-zig/action.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,14 @@ runs:
1717
run: |
1818
if [[ `cat /etc/os-release | grep "Amazon Linux"` ]]; then
1919
yum install -y python3-pip
20+
pip3 install ziglang
21+
elif [[ `cat /etc/os-release | grep "Alpine"` ]]; then
22+
apk add py3-pip
23+
pip3 install ziglang --break-system-packages
2024
else
2125
sudo apt install -y python3-pip
26+
pip3 install ziglang
2227
fi
23-
pip3 install ziglang
2428
cargo install --locked cargo-zigbuild
2529
2630
# Set environment variable to prevent cargo-zigbuild from auto-detecting malformed targets

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ Looking at the existing issues is a great way to find something to contribute on
4141
- [Java](./java/DEVELOPER.md)
4242
- [Node](./node/DEVELOPER.md)
4343
- [Python](./python/DEVELOPER.md)
44+
- [Go](./go/DEVELOPER.md)
4445

4546
## Code of Conduct
4647
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).

‎go/DEVELOPER.md‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -141,15 +141,7 @@ Before starting this step, make sure you've installed all software requirements.
141141
make build
142142
```
143143

144-
4. Run tests:
145-
1. Ensure that you have installed valkey-server and valkey-cli on your host. You can find the Valkey installation guide at the following link: [Valkey Installation Guide](https://valkey.io/topics/installation/).
146-
2. Execute the following command from the go folder:
147-
148-
```bash
149-
go test -race ./...
150-
```
151-
152-
5. Install Go development tools with:
144+
4. Install Go development tools with:
153145

154146
```bash
155147
make install-dev-tools

‎go/Makefile‎

Lines changed: 37 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,27 +10,39 @@ TARGET_DIR := $(GLIDE_FFI_PATH)/target
1010
# Determine the target folder based on OS and architecture
1111
UNAME := $(shell uname)
1212
ARCH := $(shell uname -m)
13+
IS_ALPINE := $(shell test -f /etc/alpine-release && echo true || echo false)
1314

1415
# Set default values for GLIDE_NAME and GLIDE_VERSION
1516
# GLIDE_VERSION is automatically set during the deployment workflow based on the value defined in go-cd.yml.
1617
# For local builds, you can manually specify the version using `GLIDE_VERSION=<version> make build`
1718
GLIDE_NAME = GlideGo
1819
GLIDE_VERSION ?= unknown
1920

21+
ifndef TARGET_TRIPLET
2022
ifeq ($(UNAME), Darwin)
2123
TARGET_TRIPLET := $(if $(filter arm64,$(ARCH)),aarch64-apple-darwin,x86_64-apple-darwin)
24+
else ifeq ($(UNAME), Linux)
25+
ifeq ($(IS_ALPINE), true)
26+
TARGET_TRIPLET := $(if $(filter arm64 aarch64,$(ARCH)),aarch64-unknown-linux-musl,x86_64-unknown-linux-musl)
27+
else
28+
TARGET_TRIPLET := $(if $(filter arm64 aarch64,$(ARCH)),aarch64-unknown-linux-gnu,x86_64-unknown-linux-gnu)
29+
endif
30+
else
31+
$(error Unsupported platform: $(UNAME) $(ARCH))
32+
endif
33+
endif
34+
35+
ifeq ($(UNAME), Darwin)
2236
# https://github.com/rust-lang/rust/issues/51009#issuecomment-2274649980
2337
BUILD_CMD := rustc --crate-type staticlib
2438
CARGO_FIX_CMD := : # no-op
2539
CARGO_POSTFIX_CMD := : # no-op
2640
STRIP_CMD := strip -x
2741
else ifeq ($(UNAME), Linux)
28-
# TODO: musl
29-
TARGET_TRIPLET := $(if $(filter arm64 aarch64,$(ARCH)),aarch64-unknown-linux-gnu,x86_64-unknown-linux-gnu)
3042
# zigbuild https://github.com/rust-cross/cargo-zigbuild
31-
BUILD_CMD := zigbuild --target $(TARGET_TRIPLET).2.17
43+
BUILD_CMD := zigbuild --target $(TARGET_TRIPLET)
3244
# workaround for https://github.com/rust-cross/cargo-zigbuild/issues/337
33-
CARGO_FIX_CMD := sed -i 's/crate-type.*/crate-type = ["staticlib"]/g' Cargo.toml
45+
CARGO_FIX_CMD := sed -i 's/crate-type.*/crate-type = ["staticlib"]/g' Cargo.toml && rustup target add $(TARGET_TRIPLET)
3446
CARGO_POSTFIX_CMD := git restore Cargo.toml
3547
STRIP_CMD := strip --strip-unneeded
3648
TARGET_DIR := $(TARGET_DIR)/$(TARGET_TRIPLET)
@@ -41,6 +53,15 @@ endif
4153
# Path where compiled binary is copied to and therefore used by go
4254
DEST_PATH := rustbin/$(TARGET_TRIPLET)
4355

56+
# Determine if we're using musl
57+
IS_MUSL := $(if $(filter true,$(IS_ALPINE)),true,$(if $(findstring musl,$(TARGET_TRIPLET)),true,false))
58+
59+
# Set GOFLAGS for musl to use musl tag
60+
ifeq ($(IS_MUSL), true)
61+
export GOFLAGS := -tags=musl
62+
endif
63+
64+
4465
install-build-tools:
4566
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.33.0
4667
cargo install cbindgen
@@ -171,10 +192,21 @@ long-timeout-test: __it
171192
opentelemetry-test: export TEST_FILTER = -skip "TestGlideTestSuite/TestModule" -otel-test $(if $(test-filter), -testify.m $(test-filter), -testify.m TestOpenTelemetry)
172193
opentelemetry-test: __it
173194

195+
# Test command for integration tests
196+
ifeq ($(IS_MUSL), true)
197+
# musl gcc does not support -fsanitize=address, so we use clang instead
198+
# TODO - Investigate musl + address sanitizer issue - https://github.com/valkey-io/valkey-glide/issues/4671
199+
# Disabled for now
200+
# MUSL_TEST_CMD := CC=clang go test -asan -v ./integTest/...
201+
TEST_CMD := CC="gcc" go test -v ./integTest/...
202+
else
203+
TEST_CMD := CC="gcc -fsanitize=address" go test -v ./integTest/...
204+
endif
205+
174206
__it:
175207
mkdir -p reports
176208
set -o pipefail; \
177-
CC="gcc -fsanitize=address" go test -v ./integTest/... \
209+
$(TEST_CMD) \
178210
$(TEST_FILTER) \
179211
$(if $(filter true, $(tls)), --tls,) \
180212
$(if $(standalone-endpoints), --standalone-endpoints=$(standalone-endpoints)) \

‎go/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,13 @@ To install Valkey GLIDE in your Go project, follow these steps:
5050
```
5151
3. After installation, you can start up a Valkey server and run one of the examples in [Basic Examples](#basic-examples).
5252

53+
### Alpine Linux / MUSL
54+
55+
If you are running on Alpine Linux or otherwise require a MUSL-based build, you must add the 'musl' tag to your build.
56+
57+
```
58+
export GOFLAGS := -tags=musl
59+
```
5360

5461
## Basic Examples
5562

‎go/base_client.go‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,10 @@ package glide
66
// #cgo !windows LDFLAGS: -lm
77
// #cgo darwin LDFLAGS: -framework Security
88
// #cgo darwin,amd64 LDFLAGS: -framework CoreFoundation
9-
// #cgo linux,amd64 LDFLAGS: -L${SRCDIR}/rustbin/x86_64-unknown-linux-gnu
10-
// #cgo linux,arm64 LDFLAGS: -L${SRCDIR}/rustbin/aarch64-unknown-linux-gnu
9+
// #cgo linux,amd64,!musl LDFLAGS: -L${SRCDIR}/rustbin/x86_64-unknown-linux-gnu
10+
// #cgo linux,amd64,musl LDFLAGS: -L${SRCDIR}/rustbin/x86_64-unknown-linux-musl
11+
// #cgo linux,arm64,!musl LDFLAGS: -L${SRCDIR}/rustbin/aarch64-unknown-linux-gnu
12+
// #cgo linux,arm64,musl LDFLAGS: -L${SRCDIR}/rustbin/aarch64-unknown-linux-musl
1113
// #cgo darwin,arm64 LDFLAGS: -L${SRCDIR}/rustbin/aarch64-apple-darwin
1214
// #cgo darwin,amd64 LDFLAGS: -L${SRCDIR}/rustbin/x86_64-apple-darwin
1315
// #include "lib.h"

0 commit comments

Comments
 (0)