CP-49213: Add new tar unpacking module

[BengangY]

Add a module Tar_ext to unpack a tar file.
During the unpacking process, verify the tar file, containing total file size,
file type, file path, file integrity, etc.

[edwintorok]

Can we add the script that generated the tar file instead of the binary tar files?
Binary files are best avoided, even if just for test code.

[psafont]

Can this module be used to replace existing tar-unpacking code?

[Vincent-lau]

Is xapi/ the right place for this module? I feel this is just a utility module rather than a xapi functionality?

[BengangY]

Can this module be used to replace existing tar-unpacking code?

Some code can be replaced. But most of the tar-unpacking code doesn't unpack the files to disk. The extracted files are processed directly and not saved on disk.

[gangj]

Can this module be used to replace existing tar-unpacking code?

Some code can be replaced. But most of the tar-unpacking code doesn't unpack the files to disk. The extracted files are processed directly and not saved on disk.

Can we make "saving to files" or "extract into some memory" options of this module?

[BengangY]

Can this module be used to replace existing tar-unpacking code?

Some code can be replaced. But most of the tar-unpacking code doesn't unpack the files to disk. The extracted files are processed directly and not saved on disk.

Can we make "saving to files" or "extract into some memory" options of this module?

I don't think we need this option. It can call directly Tar_unix.Archive.with_next_file to process tar data in memory by self-defined function.

[lindig]

Files changed 302 - something is wrong here. You can't expect us to review this.

[BengangY]

Files changed 302 - something is wrong here. You can't expect us to review this.

Yes, I was trying to resolve the build and test failure. Now it's fixed.

[gangj]

This logic should be put in gen_tar_ext_test_file.sh as otherwise it will be required every where the gen_tar_ext_test_file.sh is called.

[lindig]

I would like to suggest to think about the following approach: use Unix.realpath to resolve a path coming from a Tar file and see that it is prefixed by the current directory and reject it otherwise.

[gangj]

Just a question for this tar unpacking feature, what's the performance impact (CPU usage) on the host as the unpacking might take long time for a big file, I think we should avoid consuming too much CPU of the host.

[lindig]

This looks good to me. I now understand the protection against escaping the directory better and think it's ok (checking against paths ., .., and /).

[BengangY]

Just a question for this tar unpacking feature, what's the performance impact (CPU usage) of the host as the unpacking might take long time for a big file, I think we should avoid consuming too much CPU of the host.

just tested it, I uploaded a tar file of about 500M, and it used not more than 8% (1 processor of 16). This took about 1 second as I uploaded the file from local, so it finished soon.

[BengangY]

Is xapi/ the right place for this module? I feel this is just a utility module rather than a xapi functionality?

I don't find a proper place for it. But it's only used for xapi in the current stage, so I think it's OK to place here.