cmd ref: status: ignoring cache that is not in local cache nor in remote

[nik123]

UPDATE: Jump to #1701 (comment)

---

Command reference for status should probably describe situation when target is missing both in local cache and remote.

Target missing both in cache and remote may happen if Alice executes dvc add data.csv, git add data.csv, git commit and git push but doesn't execute dvc push and Bob executes dvc status -c data.csv on another machine.

For context see treeverse/dvc#4383

UPDATE: Jump to #1701 (comment) for summary.

[nik123]

It should be probably described in "Examples" section but I was unable to come up with nice and simple example therefore it's only issue and not PR.

[jorgeorpinel]

Good idea for a note or example. We could also clarify the bullet list in the Description: some bullets are confusing e.g. "new" inside "changed deps or changed outs", and the explanation of "not in cache" seems contradictory.

[nik123]

@jorgeorpinel by buletted list do you mean list in "Local worskpace status"? It probably should be updated but I'm not sure it's part of this issue. After a while I think it's would be the right thing to add one more bullet after "new" and "deleted" in "Comparison against remote storage" section. The miminum required text is:

• deleted means that the file/directory doesn't exist neither in the cache, neither in remote storage.

What do you think about it?

Though proposed text is technically correct it's not much of a help. Users who will face "missing" message will not get any answer how it actually happened. An explanation of reasons which caused this situation was my original concern. Probably a longer message would do a better job:

• deleted means that the file/directory doesn't exist neither in the cache, neither in remote storage. Probably the other user on another machine didn't execute dvc push after adding data.

[jorgeorpinel]

It probably should be updated but I'm not sure it's part of this issue.

It's closely related. I'm just saying we can do both things in the same PR. Not required but would be nice.

would be the right thing to add one more bullet after "new" and "deleted" in "Comparison against remote storage" section

Yes, that's a good place to add this note.

deleted means that the file/directory doesn't exist neither in the cache, neither in remote storage.
What do you think about it?

That's not what deleted means though. It means "doesn't exist in the cache, but exists in remote storage" as explained in https://dvc.org/doc/command-reference/status#comparison-against-remote-storage.

When the data is not cached or in a remote, you get:

WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: data.csv, md5: 009520053b00386d1173f3988c55d192
...

So we can add an example or expand an existing one to cover this scenario (on top of the Description note) so that we can present possible explanations and solutions (e.g. to repro, or pushing from the original project).

[nik123]

That's not what deleted means though. It means "doesn't exist in the cache, but exists in remote storage" as explained in https://dvc.org/doc/command-reference/status#comparison-against-remote-storage.

deleted is a typo. I meant missing. Sorry, my bad.

[jorgeorpinel]

Gotcha. It would be nice to have that state! But it would need to be implemented in the core repo first. For now it doesn't exist but we do get that WARNING so we can mention it as a note in the section (not a "missing" bullet though). Are you planning to try contributing this change BTW @nik123? 🙂

[nik123]

@jorgeorpinel I'm not sure what you mean, probably there is a little misunderstanding: either I've already added "missing", either we talk about different things.

I've added "missing" as a part of PR treeverse/dvc#4398. This PR introduces several changes for dvc status -c/-r when there are files missing both in cache and remote:

1. These files are listed as "missing" in status report.
2. No "Data and pipelines are up to date" message.
3. If executed with -q, dvc exits with non-zero code.

Here as an example output for dvc status -c dvc 1.6.0+adce62 the missing state looks like this:

$ dvc status -c -R .
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: data-in-cache2-not-in-remote.txt, md5: 599b9eea7a2c4f97be18c077d2df4789
	new:                data-in-cache-1-not-in-remote.txt
	missing:            data-in-cache-2-not-in-remote.txt

P.S.: I guess neither original post for this issue, neither PR and issue in the dvc repo described it good enough. I guess it's because issue raised from Q&A in Discord. Next time I'll go in more details when creating issues on dvc.org repo.