-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: implement shellcheck lints in command blocks (#191) #264
base: main
Are you sure you want to change the base?
feat: implement shellcheck lints in command blocks (#191) #264
Conversation
There's still a lot of testing and polishing to do, which will undoubtedly reveal some issues, but I wanted to open for comments before I went any further in case I'm way off from what's desired. |
const SHELLCHECK_BIN: &str = "shellcheck"; | ||
|
||
/// shellcheck lints that we want to suppress | ||
const SHELLCHECK_SUPPRESS: &[&str] = &[ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a copy-paste of miniwdl's list. I think we can remove most of these since most were added to work around their use of dummy literals, which we're not (currently) doing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think this is good for now. It would be nice to make these configurable in the future. Perhaps you could add a TODO
comment above it and/or an issue when this is merged?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not perfect, but users can add #shellcheck disable=
directives right into the comment of a command section. Or use SHELLCHECK_OPTS
to globally disable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this is on the right track! Didn't go through it with a fine-tooth comb yet, but left some comments/Qs on things that jumped out at me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking really good!
const SHELLCHECK_BIN: &str = "shellcheck"; | ||
|
||
/// shellcheck lints that we want to suppress | ||
const SHELLCHECK_SUPPRESS: &[&str] = &[ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think this is good for now. It would be nice to make these configurable in the future. Perhaps you could add a TODO
comment above it and/or an issue when this is merged?
wdl-lint/tests/lints/deprecated-placeholder-options-v1.1/source.errors
Outdated
Show resolved
Hide resolved
958e222
to
8474ebd
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking great so far. Really excited to get this feature merged in 😎
wdl-lint/src/rules/shellcheck.rs
Outdated
"ShellCheck (https://shellcheck.net) is a static analysis tool and linter for sh / bash. \ | ||
The lints provided by ShellCheck help prevent common errors and \ | ||
pitfalls in your scripts. Following its recommendations will increase \ | ||
the robustness of your command sections." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just want to say good job writing this explanation()
text. I think it's spot on 👍
wdl-lint/src/rules/shellcheck.rs
Outdated
state.exceptable_add( | ||
Diagnostic::error("running `shellcheck` on command section") | ||
.with_label( | ||
"could not find `shellcheck` executable.", | ||
command_keyword.text_range().to_span(), | ||
) | ||
.with_rule(ID) | ||
.with_fix("install shellcheck or disable this lint."), | ||
SyntaxElement::from(section.syntax().clone()), | ||
&self.exceptable_nodes(), | ||
); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Making a comment to open the floor for discussion.
Is this how we want to handle the case that SC isn't installed?
If we want to create a diagnostic that ShellCheck
couldn't be run, it should be a note
I think. Not an error
. Errors should be reserved for broken WDL, and this isn't that. I could maybe see this at warning
level? But I'd prefer note
personally.
Putting myself in the shoes of a naive user, if I've never heard of ShellCheck I'm just disabling this lint going forward. After all, we are asking folks to install another dependency without offering them any guidance on how to do that. And it would potentially annoy me seeing this every time I forget to disable the shellcheck rule. I think we need some more here. A link to shellcheck.net? And a one sentence sales pitch on the benefit of the SC?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought we decided that ShellCheck was only going to be an opt-in rule? I think the behavior depends largely on that decision. I like making it opt-in because it delineates the behavior between silent failure because of the missing tool and just being turned off.
- If the rule is enabled by default, I agree that it should be silent so that every user doesn't have to disable it. I don't think this is what we should do though.
- If the rule is disabled by default and is opt-in only, then I think it should alert you with a warning that you are missing the required binary for this rule (it's probably not a note because, if you explicitly enabled this rule, you're probably expecting it to be installed).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought we decided that ShellCheck was only going to be an opt-in rule? I think the behavior depends largely on that decision. I like making it opt-in because it delineates the behavior between silent failure because of the missing tool and just being turned off.
Maybe I missed something then. But the complication with making it opt-in is that this would be the first rule like that, and would probably require some tinkering of the architecture. We don't have a config system yet, so how exactly do we "opt-in"? Is that OOS for this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, I was thinking that this wouldn't be in the default set, and we would just add something in Sprocket that injects this rule in on a flag or similar. But, that being said, I'm fine with keeping it in the default set with no diagnostic on missing, so long as the TODO is marked in an issue (and, I would ask that, if we're close to having that working, we include it in the commit just commented out for now so it's easy to enable when we're ready).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll go ahead and leave it in the default set but remove the diagnostic message on missing. I'll just comment out the diagnostic as requested (will fix up the wording a bit based on other feedback).
759c7f4
to
6f792a2
Compare
b76e234
to
f2506be
Compare
Rebased. Gauntlet in arena mode is giving me the following:
so I'll look into that tomorrow. |
@thatRichman do you think you can figure out how to install Feel free to ping me on Zulip if that poses any issues! I imagine the windows runner could pose a challenge 🤔 |
@a-frantz Sure, shouldn't be too bad. Shellcheck provides a pre-built windows binary that we can pull from their releases. Might just take a little fiddling. |
@a-frantz Here's a link to an example of a new install-shellcheck action running on my fork. One thing to note is Ubuntu being Ubuntu only has 0.8.0 by default. macOS and Windows will always run latest. Maybe a good thing to be testing across versions. But if we want to forego the apt caching we could just pull from GH the same as Windows. edit: it's also suspicious that Arena passed on Windows when the duplicate diagnostic bug was still present. Makes me think it probably isn't finding the executable. |
wdl-lint/src/util.rs
Outdated
@@ -24,6 +34,22 @@ pub fn is_inline_comment(token: &Comment) -> bool { | |||
false | |||
} | |||
|
|||
/// Determines if a string containing embedded quotes is properly quoted. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// Determines if a string containing embedded quotes is properly quoted. | |
/// Determines whether or not a string containing embedded quotes is properly quoted. |
My two cents: let's just always pull from latest on everything (including Mac). It makes it a bit more straightforward to maintain the Github action associated with that, and it also ensures that every platform always works with the bleeding edge Shellcheck. |
I'm starting to see why the miniwdl folks went with literals instead of variables / expansions when substituting placeholders. The common false positives I'm seeing from Arena are:
caused by:
because we're replacing both of these placeholders with double-quoted expansions. and SC2001
caused by:
because with a bash variable you can use the suggested syntax but that obviously doesn't apply to WDL placeholders. as well as SC2066:
caused by:
Though, this one is also basically identical to one that miniwdl has to suppress. I've found (what I think) is an elegant solution to determine whether to use a literal or expansion, which reduces but does not eliminate false positives. That being said, I think miniwdl's approach of guessing the placeholder's type and using an appropriate literal would be overly complicated here. So I think it comes down to the severity and frequency of the lints we end up needing to suppress. I still think (with the commit I'm about to push) that we're better off than always using literals. We could probably spend weeks working to get the false positives to an absolute minimum without suppressing anything, but that feels a bit bikesheddy to me. Of course I will defer. |
So am I. However with I'm thinking we shouldn't expend a large amount of effort eliminating these false positives with the current AST-based implementation. @thatRichman can you remove this rule from the default set of rules (as Clay originally suggested) so that this is opt-in only? We should re-architect |
Sounds good. I'll disable the shellcheck-install action (once I get it working on all OSes) so that it be re-enabled at a later date. I know the intent is to support non-default rules in sprocket eventually, but should I put any work into supporting non-default rules in the dev CLI in this PR? Seems OOS, so happy to throw in an issue if there isn't one already. Also, I think I will wait for the resolution of #268 before making any final changes to this PR, as it throws off diagnostic placements. |
Fixed! Thanks for logging that issue. |
a7781eb
to
76ed909
Compare
@a-frantz This should be ready for final review. One question I have is whether or not to run gauntlet with shellcheck enabled (I added this option), since this rule is in 'beta' and thus gauntlet will report missing lints if not run with --shellcheck (if it has previously been blessed while using --shellcheck). Also, I fully intend to squash these 39 commits into 1. I can do that before or after final review, whatever you prefer. |
Sweetness! Will give this a review in the morning. I also added the rest of the team as reviewers in case I miss something.
IMO we leave it out of gauntlet/arena for now. I imagine it is going to add a boatload of diagnostics, and we like to review each one. Considering we know upfront some portion will be false positives, I don't think it's worth the effort.
We squash as part of the merge, so you shouldn't need to do anything to deal with it. |
let label = format!( | ||
"SC{}[{}]: {}", | ||
diagnostic.code, diagnostic.level, diagnostic.message | ||
); | ||
Diagnostic::note(&diagnostic.message) | ||
.with_rule(ID) | ||
.with_label(label, span) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This repeats the diagnostic.message
twice. Is that desirable? Should we omit it from the label
as being redundant, or do we like it appearing twice?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm conflicted. Pulling an example from below, I think the initial instance by the note
is fine. The second provides the additional info link, which could be helpful. I do notice that the span isn't highlighted correctly. I also think the fix
needs to be something else.
note[ShellCheck]: You need a space before the ].
┌─ tests/lints/shellcheck-error/source.wdl:18:25
│
18 │ if [ -f "$broken"]
│ ^
│
│ SC1020[error]: You need a space before the ].
│ more info: https://www.shellcheck.net/wiki/SC1020
│
= fix: address the diagnostic as recommended in the message
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@adthrasher Could you clarify what you mean about the span not being highlighted correctly? Shellcheck outputs this diagnostic as a single position one character ahead of the closing bracket.
#!/usr/bin/env bash
# the below line has 11 columns of text
[ -f "foo"]
yields:
{
"file": "test.sh",
"line": 3,
"endLine": 3,
"column": 12,
"endColumn": 12,
"level": "error",
"code": 1072,
"message": "Missing space before ]. Fix any mentioned problems and try again.",
"fix": null
}
(note that shellcheck 1-indexes columns)
so I think it's 1:1 with how shellcheck reports it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re: the message
@peterhuene and I discussed this some in the Zulip. This initially came about because Gauntlet requires all diagnostic messages to be unique for a given line + position. The solution I came up with was to include the SC###[] in the message, but only the shellcheck message in the label. But happy to rework. Agree the fix could be much more useful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So it's the underlying shellcheck
that returns the column position after ]
. That's good to know. It still seems like the highlight should be one position earlier, so that it flags the ]
.
@thatRichman can you add a test that |
format!("more info: {}/SC{}", &SHELLCHECK_WIKI, diagnostic.code), | ||
span, | ||
) | ||
.with_fix("address the diagnostic as recommended in the message") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@a-frantz - I know you just added/updated our fix
messages. What do you think of this one? The generic message isn't helpful to users, but the alternative would presumably involve parsing the warnings from shellcheck and writing a fix for each type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be perfectly honest, yall didn't really convince me that it's the right move for every diagnostic to include a fix
. I was just outvoted and didn't feel strongly enough to put up a fight 😅
So my frank opinion is that this fix
should be dropped along with some of our other "unhelpful" fix messages. But we've set the precedent that every diagnostic should include some kind of fix
, and I'm not sure how we can improve this one without some rather intense work 🤷
@thatRichman I see here that shellcheck
provides it's own fix
message. Maybe we should just copy that? I haven't seen what they look like, so maybe they aren't of good quality 🤔
Can you share some examples of the shellcheck fix messages?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fix messages describe the structural changes required to address the lint, and are not always available.
Example:
echo [ -z $foo ]
"fix": {
"replacements": [
{
"line": 4,
"endLine": 4,
"precedence": 7,
"insertionPoint": "afterEnd",
"column": 11,
"replacement": "\"",
"endColumn": 11
},
{
"line": 4,
"endLine": 4,
"precedence": 7,
"insertionPoint": "beforeStart",
"column": 15,
"replacement": "\"",
"endColumn": 15
}
]
We could convert these into a human-readable message, but I can't say if we want to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@a-frantz - This could be used to get toward my goal for fix
. I would want to use that fix
to print the correct line.
echo [ -z "$foo" ]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oo. That's kind of messy... At the very least it wouldn't be trivial to convert that to something I would want us to print.
@adthrasher I agree that we should work towards having every fix
message print the "corrected" code where possible. But that's not very easy to implement robustly. This one in particular looks tough to parse when it's present.
My two cents on this issue is we leave the fix
message as Spencer has written it, but we can make a new issue to revisit this specific fix
(and many of our other fixes for that matter).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@a-frantz - That particular example doesn't look too bad to convert to text. You've just got the line, column, replacement text, and how to place it. The bigger issue, to me, is that it isn't always present (though some errors probably aren't straightforward like this one). But I agree, we can leave it as-is and have a new issue to revisit.
@@ -150,3 +152,32 @@ pub fn rules() -> Vec<Box<dyn Rule>> { | |||
|
|||
rules | |||
} | |||
|
|||
/// Gets the optional rule set. | |||
pub fn optional_rules() -> Vec<Box<dyn Rule>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is probably overkill since this will (hopefully) be a temporary home for shellcheck.
We're pretty dang close here @thatRichman . Please get the CI all green, then I think this will have my approval The general rule we follow is to require approval from two core team members for a merge. |
@a-frantz Sounds good. One thing to note is that I did end up needing to re-enable shellcheck install in the CI. This is because if shellcheck isn't present, the first lint test that is run will end up with the 'shellcheck not found' warning, and the source.errors for the shellcheck-* lints will be empty. I thought about modifying the lints test script to selectively skip the shellcheck lints, but this would not address the 'missing shellcheck' lint. Here is an example of the modified CI going green on my repo but I don't think there's any way to reflect that here unless we merge the actions changes to main in a separate PR first. |
This pull request adds a new rule to
wdl-lint
.ShellCheckRule
Describe the rules you have implemented and link to any relevant issues.
I have implemented a lint rule that runs shellcheck against command sections and reports the comments as diagnostics. This is to address #191 .
Before submitting this PR, please make sure:
CHANGELOG.md
(see["keep a changelog"] for more information).
Rule specific checks:
RULES.md
.rules()
function inwdl-lint/src/lib.rs
.wdl-lint/tests/lints
that covers everypossible diagnostic emitted for the rule within the file where the rule
is implemented.
Visitor
callback, you have alsooverridden that callback method for the special
Validator
(
wdl-ast/src/validation.rs
) andLintVisitor
(
wdl-lint/src/visitor.rs
) visitors. These are required to ensure the newvisitor callback will execute.
wdl-gauntlet --refresh
to ensure that there are nounintended changes to the baseline configuration file (
Gauntlet.toml
).wdl-gauntlet --refresh --arena
to ensure that all of therules added/removed are now reflected in the baseline configuration file
(
Arena.toml
).