ci: add shellcheck linter #6428

kwanhur · 2022-02-23T10:28:42Z

What this PR does / why we need it:

use ShellCheck to analysis shell scripts and gives warnings and suggestions.

intention: import shell script's quality

test case SC2148

.shellcheckrc to enable/disable shellcheck code(equals rule), content now comes from existed shell scripts.

cause all of them had run successfully, so a lot of rules disable

#6320 (comment)

Pre-submission checklist:

Did you explain what problem does this PR solve? Or what new features have been added?
Have you added corresponding test cases?
Have you modified the corresponding document?
Is this PR backward compatible? If it is not backward compatible, please discuss on the mailing list first

Signed-off-by: kwanhur <huang_hua2012@163.com>

spacewander

Good, let's add #6320 to this PR.

leslie-tsang · 2022-02-24T01:43:07Z

.github/workflows/code-lint.yml

+      - name: Shellcheck code
+        run: |
+          scversion="latest"
+          wget -O- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" | tar -xJv


Suggested change

wget -O- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" | tar -xJv

wget -O- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" | tar -xJv

Why use -J here ?

Can it replace with -z ?

Because it's format is tar.xz not tar.gz

-J filter the archive through xz

-z filter the archive through gzip

leslie-tsang · 2022-02-24T01:46:48Z

.github/workflows/code-lint.yml

+          scversion="latest"
+          wget -O- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" | tar -xJv
+          cp -av "shellcheck-${scversion}/shellcheck" /usr/local/bin/


can we use apt to install spellcheck ?
like

apt install -y shellcheck

This way is ok to install shellcheck, but I'm not sure its repository updates to latest version.

latest version may add new check rules and it covers the before.

different developers use different version, the latest covers them.

The package must uniformly install from repo unless no supported, I'll improve installation.

LGTM, let keep it this way.

Thanks for replying.

kwanhur · 2022-02-24T13:15:56Z

Good, let's add #6320 to this PR.

Had linked to here.

spacewander · 2022-02-25T02:35:46Z

I mean squashing the modification in #6320 to this PR. It is bad to split a tiny change into multiple PRs. And if you want to solve more shellcheck warnings, please do it in this PR.

Don't bombard with other shellcheck relative PR anymore.

kwanhur · 2022-02-25T05:58:38Z

mentioned PRs differ goal:

ci(apisix test): grep search [error] from file error.log #6320 modify search error logicality, reduce one command.
ci: add shellcheck linter #6428 add code(shell script) limitation.

I think they're should stay with two, one PR reaches one goal. Of cause, still need they squash into one, I'll take it.

Signed-off-by: kwanhur <huang_hua2012@163.com>

kwanhur · 2022-02-25T06:27:49Z

Anyway, they had been done.

spacewander · 2022-02-27T12:02:43Z

ci/common.sh

@@ -15,6 +15,8 @@
 # limitations under the License.
 #

+# shellcheck disable=SC2148


I think it is quite strange to add a warning and disable it later. Let's remove it.

SC2148(Add a shebang) rule has enabled, remove it ci code-lint will run failed.
The key point is that, all the scripts have the shebang but ci/common.sh and t/cli/common.sh.

disable in script file, just scope the file, no influences on the other.

To fix the check failure, three solutions to choose:

enable SC2148 and add shellcheck disable declaration(just like now), test case SC2148.

enable SC2148 and add a shebang in common.sh, but it seems not so suitable, like ci(action): add shellcheck linter #6340 (comment).

disable SC2148, this will not find it when add a script without shebang.

Summary, solution 1 could be the better.
Hope your voice.

Sure, but why use

enable=SC2148

in https://github.com/apache/apisix/pull/6428/files#diff-c71502ae2fb6ee770263dbf93031c6bb1549ef6b2970b862daf24d01a1ded277R22?

Could we keep it disabled? Look like the check is enabled by you.

Had disable it.

Actually, all the rules enable by default. Here, I mark it obviously.

Signed-off-by: kwanhur <huang_hua2012@163.com>

kwanhur · 2022-02-28T08:11:43Z

build (ubuntu-18.04, linux_openresty_1_17, t/plugin) runs failed, it's not related to shellcheck, please trigger re-run the job.

from the logs, seems that grpc server launch abnormally.

spacewander · 2022-03-01T03:08:33Z

.github/workflows/code-lint.yml

+          wget -O- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" | tar -xJv
+          cp -av "shellcheck-${scversion}/shellcheck" /usr/local/bin/
+          shellcheck --version
+          shellcheck **/*.sh


Let's limit the scope. We don't want to lint the code from git submodule.

Ok, I got the meaning that checkout repo without submodule, so it won't lint the code from git submodule.

Let's limit the scope. We don't want to lint the code from git submodule.

Done.

shellcheck **/*.sh

? It is still linting all the shell files in this repo, including the one from submodule...

Solution 1: not checkout submodule, then this won't lint shell files in submodule

steps: - name: Checkout code uses: actions/checkout@v2.4.0

Solution 2: specify shellcheck paths, exclude submodule, so it won't lint in submodule

From this tip, I think you mean the solution 2.

shellcheck **/*.sh

? It is still linting all the shell files in this repo, including the one from submodule...

Specify file and dirs to lint had been done.

Signed-off-by: kwanhur <huang_hua2012@163.com>

spacewander · 2022-03-03T01:40:10Z

.github/workflows/code-lint.yml

+          wget -O- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" | tar -xJv
+          cp -av "shellcheck-${scversion}/shellcheck" /usr/local/bin/
+          shellcheck --version
+          shellcheck benchmark/*.sh bin/apisix ci/*.sh ci/pod/nacos/healthcheck/*.sh t/cli/*.sh t/plugin/grpc-web/*.sh utils/*.sh


Would be better to use git ls-files | grep "\.sh$" | xargs

Yep, it's better. xargs -t shellcheck echo the command to be executed, it'll show which files to check.

Signed-off-by: kwanhur <huang_hua2012@163.com>

ci: add shellcheck linter

c734b16

Signed-off-by: kwanhur <huang_hua2012@163.com>

kwanhur mentioned this pull request Feb 23, 2022

ci(action): add shellcheck linter #6340

Closed

4 tasks

spacewander reviewed Feb 24, 2022

View reviewed changes

leslie-tsang reviewed Feb 24, 2022

View reviewed changes

kwanhur mentioned this pull request Feb 24, 2022

ci(apisix test): grep search [error] from file error.log #6320

Closed

4 tasks

kwanhur added 2 commits February 25, 2022 14:25

ci(apisix test): grep search [error] from file error.log

56b6700

Signed-off-by: kwanhur <huang_hua2012@163.com>

ci(apisix test): abandon shellcheck directive

8f27197

Signed-off-by: kwanhur <huang_hua2012@163.com>

spacewander reviewed Feb 27, 2022

View reviewed changes

ci(shellcheck): disable SC2148 add a shebang

fa549ac

Signed-off-by: kwanhur <huang_hua2012@163.com>

spacewander reviewed Mar 1, 2022

View reviewed changes

kwanhur added 3 commits March 1, 2022 17:45

ci(shellcheck): no lint on submodule's codes

38c7d0e

Signed-off-by: kwanhur <huang_hua2012@163.com>

ci(shellcheck): specify file and directories

7c935af

Signed-off-by: kwanhur <huang_hua2012@163.com>

ci(shellcheck): specify accurate directory pattern

92f3d06

Signed-off-by: kwanhur <huang_hua2012@163.com>

spacewander previously approved these changes Mar 3, 2022

View reviewed changes

ci(shellcheck): git list files

1c191ca

Signed-off-by: kwanhur <huang_hua2012@163.com>

kwanhur dismissed spacewander’s stale review via 1c191ca March 3, 2022 07:41

spacewander approved these changes Mar 4, 2022

View reviewed changes

kwanhur mentioned this pull request Mar 4, 2022

request help: github ci jobs run failed frequently #6501

Closed

tzssangglass approved these changes Mar 4, 2022

View reviewed changes

spacewander merged commit f31bef4 into apache:master Mar 4, 2022

kwanhur deleted the ci-shellcheck branch March 4, 2022 07:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: add shellcheck linter #6428

ci: add shellcheck linter #6428

kwanhur commented Feb 23, 2022

spacewander left a comment

leslie-tsang Feb 24, 2022

kwanhur Feb 24, 2022

leslie-tsang Feb 24, 2022

kwanhur Feb 24, 2022

leslie-tsang Feb 24, 2022

kwanhur commented Feb 24, 2022

spacewander commented Feb 25, 2022 •

edited

Loading

kwanhur commented Feb 25, 2022

kwanhur commented Feb 25, 2022

spacewander Feb 27, 2022

kwanhur Feb 28, 2022

spacewander Feb 28, 2022

kwanhur Feb 28, 2022

kwanhur commented Feb 28, 2022

spacewander Mar 1, 2022

kwanhur Mar 1, 2022

kwanhur Mar 1, 2022

spacewander Mar 2, 2022

kwanhur Mar 2, 2022

kwanhur Mar 2, 2022

spacewander Mar 3, 2022

kwanhur Mar 3, 2022

	wget -O- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" \| tar -xJv
	wget -O- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" \| tar -xJv

ci: add shellcheck linter #6428

ci: add shellcheck linter #6428

Conversation

kwanhur commented Feb 23, 2022

What this PR does / why we need it:

Pre-submission checklist:

spacewander left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kwanhur commented Feb 24, 2022

spacewander commented Feb 25, 2022 • edited Loading

kwanhur commented Feb 25, 2022

kwanhur commented Feb 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kwanhur commented Feb 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spacewander commented Feb 25, 2022 •

edited

Loading