Tracking issue for moving ui tests to subdirectories

[pnkfelix]

NOTE: see #73494 (comment) for mentoring instructions.

Looking at https://github.com/rust-lang/rust/tree/master/src/test/ui, github truncates the list to just the first 1000 entries (of 1564 total entries).

This is especially frustrating since github also sorts by name, not by kind, and so its truncation is cutting off whole subdirectories from the view.

TODO:

• We should restructure the test suite so that no one directory has more than 1000 entries
• We should add code to tidy to ensure that future changes do not break the above invariant. (done by Check the number of entries in UI test on tidy #79959)

[lcnr]

The same problem exists for https://github.com/rust-lang/rust/tree/master/src/test/ui/issues with 1,839 omitted entries.

That could be solved by adding using a directory every 10000 issue ids, i.e. ui/issues/10000-19999, ui/issues/20000-29999 and so on

[workingjubilee]

Also see #76268: many of these are run pass tests that do not particularly need to be UI tests.

[Havvy]

Is the compiler team okay with PRs that move tests into subdirectories?

[Nadrieril]

That could be solved by adding using a directory every 10000 issue ids, i.e. ui/issues/10000-19999, ui/issues/20000-29999 and so on

That would technically work, but I'd much rather see some kind of thematic folders. I've been doing that for the bit of the compiler I'm hacking on, and it's been tremendously helpful to have most relevant tests be caught by --test-args pattern. For that same reason I like having the corresponding run-pass/check-pass tests in the same folders too. I don't find the ui/non-ui distinction all that helpful when I'm trying to check if my latest changes broke anything.

[lcnr]

yeah, i agree that it would be better to group tests by what they are testing, but that's a lot of work and probably won't be done in the near future so imo it would be great to start by grouping them by issue id which is fairly easy and then continue sorting them afterwards

[petrochenkov]

A simple machine learning model could assist this task well.
Putting a test into one of N subdirectories is a typical classification task.

My knowledge in this area is outdated by about 10 years, but here's one suggested solution

• Each test (including stderr/stdout) is represented as a bag of tokens + their bigrams and trigrams.
• A classifier is trained on existing subdirectories using e.g. https://www.csie.ntu.edu.tw/~cjlin/liblinear/ (or substitute some modern analogue).
• Tests in src/test/ui root and src/test/ui/issues are classified, high confidence tests are moved, low confidence tests are left for manual classification.

(We just need someone who will be able to implement this quickly, that is probably any ML student.)

[petrochenkov]

cc @Eh2406 on the previous comment.

[Eh2406]

Not an ML student, nor familiar with the compiler. Was I who you meant to ping?
Code coverage results for each test would probably make for great features, if we can get it to work. (Did this test run this line of code.)
I'd guess that a simple pca decomposition of that can bring it to a workable number of dimensions, and a k-means can make some clusters. (Keep it low teck until we prove we need more.) But someone with NLP experience may know more about what the best tools are these days.

[petrochenkov]

(@Eh2406 I just remember you've been talking about being familiar with statistics, not suggesting you to actually implement this.)

[matklad]

Seems like there's a bunch of discussion here and on #79718. I feel rather strongly about this topic, but I also don't have a large stake here, as I don't work with rust-lang/rust test suite on a daily basis.

So I've tried creating a summary document here:

https://hackmd.io/@matklad/Hypu-hosv/edit

[petrochenkov]

I've made a classifier tool as described in #73494 (comment) (https://github.com/petrochenkov/classifui), trained it on the existing subdirectories, and then run on tests in ui/issues and ui root.
The results can be found in https://gist.github.com/petrochenkov/3130672b9d5e6c4253a8fe2f34405703, each line is a test name + three most likely subdirectories with their "confidence scores" (the higher the better - 1.0 good, -1.0 bad).

I don't plan to work on this tool further, but it would be great if someone adopted it and made it conveniently usable by people for suggesting subdirectories for newly added tests.

Some further tasks for the tool:

• Improve user interface to make the tool usable by other people than me, possibly online. Complexity: easy/medium.
• Look with human eyes at features with maximum weights or non-zero weights, what are they? Look at the classification results too. Measure quality of the classification by splitting the training set into two parts. Think about better feature construction using this information. Complexity: unknown.
• Try using coverage-based features instead of (or in addition to) "bag of tokens" features. Complexity: medium/hard.
• Find out what NLP and related ML achieved in the last ten years and rewrite the classifier using modern simpler and better tools. Complexity: depends on your background.

[petrochenkov]

Some subdirectories besides issues that were found to be "special":

• error-codes - very specialized, but can contain anything (not used during training).
• bad, did_you_mean, rfcs, span, suggestions - meaningless, need to be dismantled (not used during training).
• binding - has too many misclassified tests, needs an audit.

[jyn514]

This work seems to have stalled a bit. Here is my understanding of the current status:

rust/src/tools/tidy/src/ui_tests.rs

Lines 10 to 13 in 8d321f7

	const ENTRY_LIMIT: usize = 1000;
	// FIXME: The following limits should be reduced eventually.
	const ROOT_ENTRY_LIMIT: usize = 881;
	const ISSUES_ENTRY_LIMIT: usize = 1978;

• The tests/ui subdirectory is small enough. 881 entries is still large, but small enough it doesn't break github.
• All other directories are small enough except for the ui/issues subdirectory, which still needs organizing.

---

Mentoring instructions: For each test in tests/ui/issues you want to move,

• Find a reasonable directory to move it to. You can either "eyeball" the contents of the test file, look at the name of the issue it's referring to, or try to use Petrochenkov's ML tool: Tracking issue for moving ui tests to subdirectories #73494 (comment)
• Run git mv tests/ui/issues/issue-NNNN.rs tests/ui/new-subdirectory. Preferably also give the test a more helpful name, but just moving it is ok too.
• Run x test tests/ui/new-subdirectory/issue-NNNN.rs to make sure it passes after your change. I suggest enabling download-rustc = true in config.toml so you don't have to build the compiler from source.

[c410-f3r]

I will get back to this issue in the next few days.

IIUC, the thing about keeping tests/ui small is to help in a future organization where each test is in an more appropriated and descriptive directory.

[petrochenkov]

Find out what NLP and related ML achieved in the last ten years and rewrite the classifier using modern simpler and better tools.

@cinemere did some research on Rust code classification (https://github.com/cinemere/code-classification), and applied the results to this task.
The improvements from using modern CodeParrot models are about 71% -> 74% in the accuracy of classification, at cost of larger required training and classification time/resources.

So from practical point of view (e.g. running on CI) classifui (based on sparse linear SVM) may be a good enough tradeoff.

[seqre]

I understand the need to categorize the issues into more descriptive directories for better organization in the long run and to reduce the number of files in the directory. But this is something to be done on an issue level and seems to be a more difficult project.

If we wanted to reduce the number of files significantly on a file level, we could group all test files for a single issue into a single directory. This would quickly reduce the number of files in the issues/ directory from 1854 to 1227. The categorization at an issue level would still have to happen, but at least the effort required to reduce the number of files in the directory would be significantly reduced.

For quick&dirty POC, here's the snippet that shows the idea:

for i in $(ls issue*.* | cut -d'-' -f2 | cut -d'.' -f 1 | grep -v "[^0-9]" | sort -n | uniq -c |grep -v "      1" | cut -d ' ' -f 8); do mkdir issue-$i; git mv issue-$i*.* issue-$i; done

If there's a consensus to implement that, I could submit a PR.

[petrochenkov]

and to reduce the number of files in the directory

I don't think simply reducing this number is a goal.

The classifui tool is working pretty well, and @c410-f3r has already moved hundreds of tests to "semantically appropriate" subdirectories using it.
This is just some work that needs to happen in the background, not even intensively, just consistently.

[c410-f3r]

Time to get back to this topic. Hope it ends this year 🤞