GH-46909: [CI][Dev] Fix shellcheck errors in the ci/scripts/install_sccache.sh #46910

hiroyuki-sato · 2025-06-26T00:41:42Z

Rationale for this change

This is the sub issue #44748.

shellcheck ci/scripts/install_sccache.sh

In ci/scripts/install_sccache.sh line 22:
if [  "$#" -lt 1 -o "$#" -gt 3 ]; then
                 ^-- SC2166 (warning): Prefer [ p ] || [ q ] as [ p -o q ] is not well defined.


In ci/scripts/install_sccache.sh line 42:
curl -L $SCCACHE_URL --output $SCCACHE_ARCHIVE
        ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
curl -L "$SCCACHE_URL" --output $SCCACHE_ARCHIVE


In ci/scripts/install_sccache.sh line 43:
curl -L $SCCACHE_URL.sha256 --output $SCCACHE_ARCHIVE.sha256
        ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
curl -L "$SCCACHE_URL".sha256 --output $SCCACHE_ARCHIVE.sha256


In ci/scripts/install_sccache.sh line 53:
sha256sum $SHA_ARGS $SCCACHE_ARCHIVE.sha256
          ^-------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
sha256sum "$SHA_ARGS" $SCCACHE_ARCHIVE.sha256


In ci/scripts/install_sccache.sh line 55:
if [ ! -d $PREFIX ]; then
          ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
if [ ! -d "$PREFIX" ]; then


In ci/scripts/install_sccache.sh line 56:
    mkdir -p $PREFIX
             ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
    mkdir -p "$PREFIX"


In ci/scripts/install_sccache.sh line 61:
tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory $PREFIX --exclude="sccache*/*E*E*"
                                                           ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory "$PREFIX" --exclude="sccache*/*E*E*"


In ci/scripts/install_sccache.sh line 62:
chmod a+x $PREFIX/sccache
          ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
chmod a+x "$PREFIX"/sccache


In ci/scripts/install_sccache.sh line 65:
    echo "$PREFIX" >> $GITHUB_PATH
                      ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
    echo "$PREFIX" >> "$GITHUB_PATH"


In ci/scripts/install_sccache.sh line 67:
    echo "SCCACHE_PATH=$PREFIX/sccache.exe" >> $GITHUB_ENV
                                               ^---------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
    echo "SCCACHE_PATH=$PREFIX/sccache.exe" >> "$GITHUB_ENV"

For more information:
  https://www.shellcheck.net/wiki/SC2166 -- Prefer [ p ] || [ q ] as [ p -o q...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...

What changes are included in this PR?

SC2086: Use multiple tests.
SC2086: Quoting variables.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

GitHub Issue: [CI][Dev] Fix shellcheck errors in the ci/scripts/install_sccache.sh #46909

…tall_sccache.sh

github-actions · 2025-06-26T00:42:06Z

⚠️ GitHub issue #46909 has been automatically assigned in GitHub to PR creator.

hiroyuki-sato · 2025-06-26T01:19:44Z

Do we need set default $PREFIX value?

This script allows one argument like ci/scripts/install_sccache.sh unknown-linux-musl
not, ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

But In this case, $PREFIX doesn't set any value.
If PREFIX is not set, it appears to behave unexpectedly.

arrow/ci/scripts/install_sccache.sh

Lines 29 to 57 in d97e7ab

    
           PREFIX=$2 
        
           VERSION=${3:-0.3.0} 
        
           ARCH=$(uname -m) 
        
           if [ "${ARCH}" != x86_64 ] && [ "${ARCH}" != aarch64 ]; then 
        
               echo "Skipped sccache installation on unsupported arch: ${ARCH}" 
        
               exit 0 
        
           fi 
        
           SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v$VERSION/sccache-v$VERSION-$ARCH-$BUILD.tar.gz" 
        
           SCCACHE_ARCHIVE=sccache.tar.gz 
        
           # Download archive and checksum 
        
           curl -L $SCCACHE_URL --output $SCCACHE_ARCHIVE 
        
           curl -L $SCCACHE_URL.sha256 --output $SCCACHE_ARCHIVE.sha256 
        
           echo "  $SCCACHE_ARCHIVE" >> $SCCACHE_ARCHIVE.sha256 
        
           SHA_ARGS="--check --status" 
        
           # Busybox sha256sum uses different flags 
        
           if sha256sum --version 2>&1 | grep -q BusyBox; then 
        
             SHA_ARGS="-sc" 
        
           fi 
        
           sha256sum $SHA_ARGS $SCCACHE_ARCHIVE.sha256 
        
           if [ ! -d $PREFIX ]; then 
        
               mkdir -p $PREFIX 
        
           fi

raulcd

I would make PREFIX mandatory and require 2 arguments instead of 1.
We could also default to /usr/local/bin but that would be incorrect on Windows, as we already always call with 2 arguments we should make it mandatory in my opinion.

hiroyuki-sato · 2025-06-26T08:08:58Z

@raulcd Thank you for your comment.

The second argument has been made mandatory.
Currently, all shells seem to specify two arguments, so I believe the behavior remains unchanged.

rg 'install_sccache'
ci/docker/ubuntu-24.04-cpp.dockerfile
167:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
168:RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ci/docker/ubuntu-22.04-cpp.dockerfile
182:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
183:RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ci/docker/ubuntu-22.04-cpp-minimal.dockerfile
81:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
82:RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ci/docker/fedora-39-cpp.dockerfile
76:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
77:RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ci/docker/conda-cpp.dockerfile
56:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
57:RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ci/docker/linux-r.dockerfile
51:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
52:RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ci/docker/ubuntu-24.04-cpp-minimal.dockerfile
78:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
79:RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ci/docker/debian-12-cpp.dockerfile
109:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
110:RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ci/docker/debian-experimental-cpp.dockerfile
108:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
109:RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ci/docker/centos-7-cpp.dockerfile
55:COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
56:RUN bash /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

docker-compose.yml
769:        sudo /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin &&

dev/tasks/matlab/github.yml
105:        run: arrow/ci/scripts/install_sccache.sh pc-windows-msvc $(pwd)/sccache

dev/tasks/r/github.macos-linux.local.yml
57:          arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

dev/tasks/r/github.packages.yml
184:        run: arrow/ci/scripts/install_sccache.sh pc-windows-msvc $(pwd)/sccache
402:          arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

dev/tasks/r/github.linux.offline.build.yml
70:          arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

dev/tasks/r/github.linux.arrow.version.back.compat.yml
46:        run: arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

hiroyuki-sato · 2025-07-02T06:17:30Z

@raulcd Please take a look when you get a chance.
Change to two arguments mandatory

assignUser

Thanks for the continues effort to fix up our scripts!

assignUser · 2025-07-02T12:10:56Z

ci/scripts/install_sccache.sh

-SHA_ARGS="--check --status"
+SHA_ARGS=(--check --status)

 # Busybox sha256sum uses different flags
 if sha256sum --version 2>&1 | grep -q BusyBox; then
-  SHA_ARGS="-sc"
+  SHA_ARGS+=(-sc)


Oh this would have just overwritten each other before, nice catch!

Hmm. I think that we need to overwrite --check --status with -sc because BusyBox based sha256sum doesn't accept --check --status.

@assignUser @kou Thank you for your comment. I changed to SHA_ARGS="-sc".

assignUser · 2025-07-02T12:13:15Z

@github-actions crossbow submit -g cpp
Just running some extended tests where the script is used.

github-actions · 2025-07-02T12:15:57Z

Revision: 106aebc

Submitted crossbow builds: ursacomputing/crossbow @ actions-402ea01d3f

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

hiroyuki-sato · 2025-07-03T01:30:20Z

@github-actions crossbow submit -g cpp

github-actions · 2025-07-03T01:33:03Z

Revision: 61da45e

Submitted crossbow builds: ursacomputing/crossbow @ actions-da7a3516e9

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

kou · 2025-07-03T01:47:10Z

ci/scripts/install_sccache.sh


 # Busybox sha256sum uses different flags
 if sha256sum --version 2>&1 | grep -q BusyBox; then
  SHA_ARGS="-sc"


Suggested change

SHA_ARGS="-sc"

SHA_ARGS=("-sc")

Thanks.

What do you think SHA_ARGS=(-sc)? Because The previous part SHA_ARGS=(--check --status) doesn't use double quote. If we use quote in this place, It seems better to change SHA_ARGS=("--check" "--status")

(-sc) is fine here.

Thanks. I changed to (-sc).

hiroyuki-sato · 2025-07-03T06:04:43Z

@github-actions crossbow submit -g cpp

github-actions · 2025-07-03T06:07:23Z

Revision: 2b91057

Submitted crossbow builds: ursacomputing/crossbow @ actions-b8a648159c

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

hiroyuki-sato · 2025-07-03T08:16:13Z

It seems that CI failure unrelates to this PR.

  Connection #0 to host www.apache.org left intact

  Issue another request to this URL:
  'https://archive.apache.org/dist/thrift/0.20.0/thrift-0.20.0.tar.gz'

  Host archive.apache.org:443 was resolved.

  IPv6: 2a01:4f9:1a:a084::2

  IPv4: 65.108.204.189

    Trying 65.108.204.189:443...
    Trying [2a01:4f9:1a:a084::2]:443...

  Immediate connect fail for 2a01:4f9:1a:a084::2: Network is unreachable

  connect to 65.108.204.189 port 443 from 172.18.0.2 port 50800 failed:
  Connection timed out

  Failed to connect to archive.apache.org port 443 after 132393 ms: Couldn't
  connect to server

  Closing connection

hiroyuki-sato · 2025-07-07T05:22:39Z

@github-actions crossbow submit example-cpp-tutorial test-ubuntu-22.04-cpp-bundled test-ubuntu-24.04-cpp-minimal-with-formats

github-actions · 2025-07-07T05:24:57Z

Revision: 2b91057

Submitted crossbow builds: ursacomputing/crossbow @ actions-95e52a850d

Task	Status
example-cpp-tutorial
test-ubuntu-22.04-cpp-bundled
test-ubuntu-24.04-cpp-minimal-with-formats

hiroyuki-sato · 2025-07-07T07:18:16Z

@github-actions crossbow submit test-ubuntu-22.04-cpp-bundled

github-actions · 2025-07-07T07:20:25Z

Revision: 2b91057

Submitted crossbow builds: ursacomputing/crossbow @ actions-1c22e82ca8

Task	Status
test-ubuntu-22.04-cpp-bundled

hiroyuki-sato · 2025-07-07T08:10:45Z

@kou All CIs passed. Please take a look when you get a chance.

kou

+1

conbench-apache-arrow · 2025-07-07T15:51:21Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 3ebe7ee.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 54 possible false positives for unstable benchmarks that are known to sometimes produce them.

apacheGH-46909: [CI][Dev] Fix shellcheck errors in the ci/scripts/ins…

9c8fe08

…tall_sccache.sh

hiroyuki-sato requested review from assignUser, jonkeane, kou and raulcd as code owners June 26, 2025 00:41

github-actions bot added the awaiting review Awaiting review label Jun 26, 2025

Use array arguments

d111442

raulcd reviewed Jun 26, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Jun 26, 2025

Change to two arguments mandatory

106aebc

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 26, 2025

assignUser approved these changes Jul 2, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jul 3, 2025

Override SHA option

61da45e

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 3, 2025

kou reviewed Jul 3, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jul 3, 2025

Use array

2b91057

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 3, 2025

kou approved these changes Jul 7, 2025

View reviewed changes

kou merged commit 3ebe7ee into apache:main Jul 7, 2025
15 checks passed

kou removed the awaiting change review Awaiting change review label Jul 7, 2025

kou mentioned this pull request Jul 7, 2025

[CI][Dev] Fix shellcheck errors in the ci/scripts/install_sccache.sh #46909

Closed

github-actions bot added the awaiting merge Awaiting merge label Jul 7, 2025

hiroyuki-sato deleted the topic/shellcheck-install_sccache branch July 7, 2025 11:10

hiroyuki-sato restored the topic/shellcheck-install_sccache branch July 12, 2025 08:04

hiroyuki-sato deleted the topic/shellcheck-install_sccache branch July 12, 2025 08:04

GH-46909: [CI][Dev] Fix shellcheck errors in the ci/scripts/install_sccache.sh #46910

GH-46909: [CI][Dev] Fix shellcheck errors in the ci/scripts/install_sccache.sh #46910

Uh oh!

Conversation

hiroyuki-sato commented Jun 26, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

hiroyuki-sato commented Jun 26, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

hiroyuki-sato commented Jun 26, 2025

Uh oh!

hiroyuki-sato commented Jul 2, 2025

Uh oh!

assignUser left a comment

Choose a reason for hiding this comment

Uh oh!

assignUser Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

kou Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

hiroyuki-sato Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

assignUser commented Jul 2, 2025

Uh oh!

github-actions bot commented Jul 2, 2025

Uh oh!

hiroyuki-sato commented Jul 3, 2025

Uh oh!

github-actions bot commented Jul 3, 2025

Uh oh!

kou Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

hiroyuki-sato Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

kou Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

hiroyuki-sato Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

hiroyuki-sato commented Jul 3, 2025

Uh oh!

github-actions bot commented Jul 3, 2025

Uh oh!

hiroyuki-sato commented Jul 3, 2025

Uh oh!

hiroyuki-sato commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

hiroyuki-sato commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

hiroyuki-sato commented Jul 7, 2025

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

conbench-apache-arrow bot commented Jul 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

hiroyuki-sato commented Jun 26, 2025 •

edited by github-actions bot

Loading