📖 xai lib utils zip component

[MFA-X-AI]

Description

During Spark Submit in cluster mode, users would need to bundle their python packages and xai_component library. This PR provides the ZipDirectory component which can do it in python.

It is smart enough to create a new zip if zip_fn doesn't exist (if not provided, will use xircuits_name as fn), and add it to the zip if fn already exists. It is also has the option to include / exclude the root dir.

For example if you'd like to zip your venv site packages you could zip it like:

venv.zip
└── site-packages
    └── lib1
    └── lib2

or you can zip it excluding site-packages:

venv.zip
    └── lib1
    └── lib2

I've also added a xircuits in the examples folder (since people would need to do it before spark submitting anyways) and for you to try.

Pull Request Type

• Xircuits Core (Jupyterlab Related changes)
• Xircuits Canvas (Custom RD Related changes)
• Xircuits Component Library
• Testing Automation
• Documentation
• Others (Please Specify)

Type of Change

• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Tests

You can either start from a new .xircuits or use UtilsZipDirectory.xircuits

1. Check Create - Add Zip File
   1. Chain 2 ZipDirectory components
   2. Either provide them the same zip_fn string or leave it blank. Run it.
   3. The first component should create the zip, while the 2nd should update it.

Executing: ZipDirectory
UtilsZipDirectory.zip created at /your/path/
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 680.03it/s]

Executing: ZipDirectory
UtilsZipDirectory.zip updated at /your/path/
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 111.10it/s]

2. Check Exclude - Include Root Dir

   1. Chain 2 ZipDirectory components
   2. Either provide them the same zip_fn string or leave it blank.
   3. Set include_dir as False. Run it.
   4. If you ran UtilsZipDirectory.xircuits, the output should be:

include_dir = True:

venv.zip
    └── examples
        └── xircuits example 1
        └── xircuits example ...
    
    └── xai_components
        └── xai_lib 1
        └── xai_lib ...

include_dir = False:

venv.zip
        └── xircuits example 1
        └── xircuits example ...
        └── xai_lib 1
        └── xai_lib ...

Tested on?

• Windows
• Linux Ubuntu
• Centos
• Mac
• Others (State here -> xxx )

Notes

The component uses tqdm + os.walk which I've read that it might consume unlimited RAM if there's a circular symlink path... but that's unlikely to happen right?

[AdrySky]

LGTM.

The component uses tqdm + os.walk which I've read that it might consume unlimited RAM if there's a circular symlink path... but that's unlikely to happen right?

Though, not sure about this. I think we'll deal with it when it does goes wrong.