modules: git-ignored files are inappropriately published with the rest of a module

[ptr1120]

Is your feature request related to a problem? Please describe.
I tried out the new cue mod support, which is great. But I had a problem during cue mod publish where the zip process was not successful due to a file lock in a subdirectory, that was not relevant for the module. The file lock was in the .idea folder that contains internals for my Jetbrains ide and is typically ignored by gitignore, dockerignore,...

Describe the solution you'd like
I would like to have the possibillity to ignore files and folders when working with cue modules. These should not be processed and also not packaged/shipped by cue modules.

Describe alternatives you've considered
I was nevertheless able to publish the module by just closing my Jetbrains ide, which resolved the fille lock. But the .idea folder was in the end part of the cue module, which should not be case, since it contains potential sensitive and at least not relevant data.

[myitcv]

Thanks for raising, @ptr1120. I'm not entirely convinced that there isn't a bug here. Because we should not be including any files/directories that begin with _ or ..

Assigning to @rogpeppe to take a look.

[ptr1120]

Thank you @myitcv . Additional info for debugging:
I had the frostyconfig example open in the ide Webstorm, with GitHub copilot plugin activated. I opened a separate commandline in the root of frostyconfig and entered the command cue mod publish v0.0.3.
The result was this error message:

create zip <truncated-by-me>\Desktop\frostyconfig: read <truncated-by-me>\Desktop\frostyconfig\.idea\copilot\chatSessions\xd.lck: The process cannot access the file because another process has locked a portion of the file.

Consuming the module (after successfully pushing it with the close ide workaround) shows that the complete .idea folder has been added to the cue module when you look into the module cache at %LOCALAPPDATA%\cue\glacial-tech.example\frostyconfig@v0.0.3\.idea

[myitcv]

Because we should not be including any files/directories that begin with _ or ..

I'm wrong here. I'm mixing up concepts in my head. We're going to reply with a more correct response shortly :)

[myitcv]

Joining various dots here. Thanks again @ptr1120 for raising this issue because it has given rise to some important (indeed critical) conversation about the security implications of what goes into a module.

#3017 has been created as a proposal. The stack of changes under https://review.gerrithub.io/c/cue-lang/cue/+/1193116 (with a couple more to follow) implements that proposal.

As a starting point we have support for two sources of module files:

• self - the files on disk
• git - the files that form current commit

The design is such that other sources can be added at a later date: e.g. support for other VCS systems, an explicit manifest file, etc.

With the building blocks of a general solution therefore hopefully in place, @ptr1120 returning to your use-case here. Can you confirm whether you are publishing VCS-based CUE, and hence whether you would be able to use git as a source?

(noting that the design we have proposed is also compatible with some notion of .cueignore if we wanted to add that... but we started with well-established modes and can go from there).

We will be discussing this topic in much greater detail tomorrow on the modules update call: #3020

[ptr1120]

Thank you, @myitcv. Yes, I am using Git as the source. I have read the proposal and it seems very well thought out. Regarding .cueignore, I am still a proponent of the ability to explicitly ignore items that are in Git but should not be part of the module. For example, there can be various settings related to IDEs, VCS, and other tools that are version-controlled but would not be appropriate for inclusion in a module. Additionally, I often have shell scripts, CUE tools, and possibly some tests written in CUE which I hope can be excluded.

[rogpeppe]

Although we have solved the original problem described, it would seem wrong to close the issue as we haven't implemented what the title says. So for now, I'm going to retitle the issue (because all the discussion has been here) and raise a new issue specifically oriented towards .cueignore so we can track that feature request there.

I'll mark this as a fixed by https://cuelang.org/cl/1192907