buf build xyz.tar fails if a file in this archive has an extended attribute

[oliversun9]

Files created in vscode, and potentially other apps, have an extended attribute. Making a tape archive (.tar) and invoke buf build on such an archive will fail.

Steps to reproduce:
Run the following in the terminal built in to mac (not the one from vscode)

% mkdir dir1 dir2                              
% mv somewhere/vscode.proto dir1/vscode.proto #### assuming such a file exists and is created by vscode
% echo "message Foo {}" > dir1/vscode.proto    
% echo "message Bar {}" > dir2/terminal.proto  
% ls -al@ dir1/vscode.proto                    
-rw-r--r--@ 1 oliversun  staff  15 18 Aug 16:46 dir1/vscode.proto
	com.apple.provenance	11 
% ls -al@ dir2/terminal.proto                  
-rw-r--r--  1 oliversun  staff  15 18 Aug 16:46 dir2/terminal.proto
% tar -cvf t1.tar dir1                         
a dir1
a dir1/vscode.proto
% tar -cvf t2.tar dir2                         
a dir2
a dir2/terminal.proto
% buf --version                                
1.22.0
% buf build t2.tar 
% buf build t1.tar
dir1/._vscode.proto:1:1:invalid control character
dir1/._vscode.proto:1:1:syntax error: unexpected error
dir1/._vscode.proto:1:2:invalid control character
...

Verified on 1.22.0 and 1.26.2-dev (latest main).

[bufdev]

Can you see if buf build dir1/vscode.proto fails as well?

[oliversun9]

Can you see if buf build dir1/vscode.proto fails as well?

Both buf build dir1/vscode.proto and buf build dir2/terminal.proto succeed.

[oliversun9]

Some more information:

adding a print statement in storageArchive.Untar shows that during buf build t1.tar, the following files are looked at:

dir1/
dir1/._vscode.proto
dir1/vscode.proto

and during buf build t2.tar, it looked at these files:

dir2/
dir2/terminal.proto

This isn't surprising, given we have seen the error messages: dir1/._vscode.proto:1:1:invalid control character.

It turns out that the ._ files are used by the Mac to store extended attributes.

Some options:

1. Skip ._ files in a tar whenever we see one
2. Maintain the status quo
3. Skip xyz/._a.proto if xyz/a.proto is also present.

[timostamm]

The macos Archive Utility.app does something similar when creating ZIP archives:

$ unzip -l proto.zip 
Archive:  proto.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  10-17-2023 17:50   proto/
        4  10-17-2023 17:50   proto/x.proto
      327  10-17-2023 17:50   __MACOSX/proto/._x.proto
---------                     -------
      331                     3 files

[jchadwick-buf]

Hmm, that's interesting. I didn't catch that behavior with libarchive ZIP, unless I was missing an option. Maybe it should be a separate issue...

edit: Though, the same workaround works for both, so it's tempting to just make this behavior happen for all archives.

[oliversun9]

I remember making both a .zip and a .tar back then. The .zip archive worked but not the .tar.

[timostamm]

The macos Archive Utility.app creates these files. To reproduce, right click a directory with proto files in the Finder, select "Compress", and run buf build on the resulting zip file to see a long list of errors such as __MACOSX/proto/._x.proto:1:1:invalid control character.

Though, the same workaround works for both, so it's tempting to just make this behavior happen for all archives.

That's my thinking as well, and why I commented 🙂