Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: RuleTester supports processor #31

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

mysticatea
Copy link
Member

Link to rendered RFC

Summary

This RFC makes RuleTester class supporting processor option.

Related Issues

@mysticatea mysticatea added the Initial Commenting This RFC is in the initial feedback stage label Jul 15, 2019

## Drawbacks

It's a pretty rare case of a rule depends on the processor's behavior. I'm not sure if this feature is needed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced that this is needed either. Unit tests are supposed to test smallest unit of code (unit). So I think it's perfectly fine to feed some pre-processed code to the tests to check if the rule is working fine, and then create separate unit tests to verify that processor is working correctly as well. I think adding processor to the ruleTester will encourage bad practice.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the sake of discussion, can you elaborate on what bad practices this might encourage?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also be curious about the bad practices you are referring to and in general I'm not sure I follow this logic - integration tests are far more valuable than unit tests, you could definitely have processor unit tests and rule unit tests individually passing but not have a working solution for your users.

Additionally, you could argue that RuleTester is already not a true unit test for rules, given they operate on ASTs, not on source code which is what you provide to RuleTester.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaicataldo I'm talking about bad practice of increasing the scope of a "unit".
@JamesHenry I don't want to argue about which types of tests are more valuable. They are supposed to test different things, and each one is valuable in their own right. You are correct about RuleTester not operating on AST, which makes it none pure unit testing library. But in my opinion it's a convenience part of the library. It would be very inconvenient to test directly on AST, so library is providing convenience feature to make it easier. But the fact of the matter is that the library is called "RuleTester", and processors are not related to rules at all. It's also quite easy to test rules and processors separately. It's also reasonable easy to create integration tests outside of RuleTester to test the whole flow.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I have written vue/comment-directive rule in the motivation section, I have a rule that requires a specific processor. That rule is to implement <!-- eslint-disable --> directive comments in HTML templates of Vue.js. That rule makes messages at every directive comment and vue.postprocess() function filters those messages and the messages which are disabled by the directive comments. That rule requires processor option for their unittest.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mysticatea I don't want to derail this RFC, but it almost seems to me that what you need in your case is ESLint to support AST extensions to mark certain tokens as disable comments, which ESLint would then handle as if they were disable comments in JavaScript. Then you would be able to decorate your comment tokens in the Vue parser and ESLint would just apply its own disable logic the way it normally would. But maybe I'm missing something.

@btmills
Copy link
Member

btmills commented Jul 29, 2021

@mysticatea if you have a moment to take a look at this: I'll be updating this RFC soon so we can use it to solve eslint/eslint#14800. Before I do, were there any additional changes you had in mind for this that I should incorporate?

@mdjermanovic mdjermanovic changed the title New: RuleTester supports processor feat: RuleTester supports processor Oct 26, 2021
@eslint-github-bot

This comment has been minimized.

@linux-foundation-easycla

This comment has been minimized.

@btmills
Copy link
Member

btmills commented Nov 4, 2021

Since the original commit, 4af9338, was written under the old CLA, it is already covered. The CLA bot will report an error, but we can ignore it for that commit because all commits are covered by a CLA.

This is ready for review.


This is a definition object of a processor that the "[Processors in Plugins](https://eslint.org/docs/developer-guide/working-with-plugins#processors-in-plugins)" section describes.

If a processor is given, the tester passes its `preprocess`, `postprocess`, and optional `supportsAutofix` properties as part of `Linter#verify()`'s `filenameOrOptions` options object.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!processor.supportsAutofix value will be passed as filenameOrOptions.disableFixes?

This comment was marked as off-topic.


## Backwards Compatibility Analysis

`RuleTester` currently accepts the `processor` key as a string in valid and invalid test case objects, but it does nothing with that string.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am struggling with this point. Fundamentally, it seems wrong to put this functionality into RuleTester, as it does seem beyond the scope of what testing a rule should do. Rules only run on JS code, so I’m not sure it makes sense to be able to pass non-JS code into a test.

I’m wondering if maybe we should provide a separate utility that applies processors to any text?

Maybe it’s worth taking a step back and thinking through what it is, exactly, we are trying to achieve here. Do we need a ProcessorTester?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reference, we picked this RFC back up as a solution to eslint/eslint#14800, and the meeting notes for that discussion are at https://github.com/eslint/tsc-meetings/blob/main/notes/2021/2021-07-15.md#add-physicalfilename-option-into-rule-tester.

Stepping back a bit first, are we still in agreement that we ought to provide a way to test rules that care about physicalFilename?

From an implementation perspective, RuleTester uses the Linter API, so the approach in this RFC takes advantage of existing API. If we were to allow setting physicalFilename from RuleTester directly, we'd need to add additional API surface to Linter that is only used by tests, which feels icky.

Making RuleTester aware of processors doesn't bother me too much because the focus is still on testing a rule, and the physicalFilename that the rule has access to is processor-related context. If we had an easy way to test rules with fully accurate context that didn't require running the processor (and perhaps even Linter), that might be preferable. Maybe that would look like replicating a minimal bit of parsing and traversal logic inside RuleTester, which would call rules directly? Then your idea of a utility to run a processor on arbitrary text could slot in nicely.

Can you elaborate more on what a ProcessorTester might do? It sounds like it might be useful for processors, though potentially without solving the physicalFilename in rules use case this RFC addresses.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t have good answers for your questions right now, I’m just concerned that we are going down a path that has led to pain the past, which is when we start adding more and more responsibility to an object to the point where it becomes unwieldy (see Linter) rather than taking a step back and thinking about how we might rethink the solution given what we know now.

So yes, the easiest solution is to add processor support into RuleTester — I just think that maybe there’s a more fundamental issue, like perhaps we need to rethink how Linter deals with physical file names to enable better testing.

(I’ll think more on this when I’m less tired.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably isn’t feasible, but throwing it out anyway: as you pointed out, RuleTester already does a lot internally, some of which parallels logic elsewhere in the engine. But its API surface is relatively small, which gives us a lot of freedom. What if we refactored RuleTester to use the ESLint API instead of Linter, then allow consumers to replace the default by passing in a pre-configured ESLint instance to the constructor? That second step might only work with flat config so that it doesn’t depend on modules being present in the file system. Then, if this works at all, if someone wanted to test a rule in a processor context, they could configure ESLint with a processor, then use the test case’s existing name option to trigger the processor.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not sure that changes the equation here. My concern is that I’m not sure RuleTester should know about processors at all; changing from Linter to ESLint just changes how processors might be implemented.

If the main concern is access to context.getPhysicalFilename(), it just seems like overkill to load a processor for such a narrow use case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not sure RuleTester should know about processors at all

Ah, now I understand your objection. Thank you. In theory, I'm inclined to agree. But we may have already crossed this bridge, so let's see if this counterargument is compelling:

Rules themselves already know about processors in the form of getFilename() vs getPhysicalFilename(). Therefore shouldn't RuleTester also know about processors?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the difference is that rules know about processor meta data, but not about processors themselves. In theory, the filename and physical filename could be populated by something other than a processor, and I’m more inclined to feel like that’s a better direction than running an actual processor to fill in those values.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Initial Commenting This RFC is in the initial feedback stage
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants