[bug] Conan does not package all the files in the package folder

[ericriff]

I'm creating a rather large package (an SDK generated by YOCTO) and I've noticed that not all the files present in my package_folder end up being uploaded.

I tested the package building it locally and everything worked fine, but as soon as I pushed it to our server and tried to use that version, I started getting errors about missing files.

If I build the package locally I get 13403 files in my package folder ($ tree -a path-to-package-folder ). Regardless, the conan manifest says that the package has 11681 files. That's about 2k files missing!

I've tested this on Ubuntu16 + Conan 1.21 and also Ubuntu18 + conan 1.29.

I haven't run a side by side comparation of the missing files, but I know of at least one file that is being ignored by conan:
package/some/folders/etc/ssl/private.

Is this expected behavior? And by this I mean conan not uploading/packaging a 1:1 copy of everything that it is on the package folder.

Thanks!

[memsharded]

Hi @ericriff

Is it possible that many of those are actually symlinks and not real files?
Also, the manifest and the package itself omits certain files, like temporary .pyc, the __pycache__ folder, the .DS_store in OSX... Are you trying to package those temporary files? These files are excluded because they are not portable, they should be generated by the local python interpreter locally.

[ericriff]

From those 13K files about 600 are symlinks.
I figured that the packager is ignoring some files on purpose, that is why I asked.

Does the packager ignores empty folders? Because that one file I named before, which is causing me issues, is an empty folder.
If I pull the package, the deploy generator complains about a broken symlink here

├── certs -> ../../../etc/ssl/certs
├── ct_log_list.cnf
├── ct_log_list.cnf.dist
├── openssl.cnf -> ../../../etc/ssl/openssl.cnf
├── openssl.cnf.dist
└── private -> ../../../etc/ssl/private     <-- HERE

../../../etc/ssl/private does exist on my PC when I build the package, and it is an empty folder.

When I pull the package from out server, that path does not exist and the deploy generator hits an broken symlink and the whole execution stops.

EDIT: Of course ../../../etc/ssl/private does not leave the scope of my package folder.

[memsharded]

Yes, I think empty folders are not packaged, only files are packaged. Something similar to what git does with folders.

Indeed packaging in the tgz packages that contains symlinks, specially broken symlinks, empty folders, etc is very problematic. We are doing some deeper investigation into the topic in #7760, to learn what could be the best approach.

To clarify, the ../../../etc/ssl/private path resolves inside the package? how is this possible? Is the private file in a 3 nested child folder?

[ericriff]

Exactly, ../../../etc/ssl/private resolves inside the package folder.

I created a dummy file inside the empty folder and now it gets packaged, so my symlink is not broken anymore.
Therefore I can confirm that conan ignores the empty folders, which can cause issues if you rely on them.
My workaround worked for this specific case, but it doesn't scale at all. Users may find other issues in the future related to these missing folders.

[memsharded]

Closing this ticket as outdated.
Modern alternatives like copy() instead of self.copy() and the --deployer in Conan 2.0 are more explicit about what is copied, so this shouldn't be a big issue anymore, users have more control and can do the copies they want more easily.