Using softlinks for pointers to libraries

[simon-streamotion]

I'd like to have a directory setup something similar to the following:

root
  -- common_libs
  -- pipe_1
     -- pipe1_files
     -- softlink_to_../common_libs
  -- pipe_2 
     -- pipe2_files
     -- softlink_to_../common_libs

The idea is that there is just one common_libs directory containing all my common libraries, and each pipeline dir contains a soflink to that directory. That way, I can easily import the common libs into the stages of my pipeline.

The problem arises when I try to run the pipeline on kubeflow - the stages fail when trying to import anything from common_libs because it cannot find the file.

It appears the tar command used to create the archive for the pipeline stages includes the softlink to common_libs rather than the files themselves. This is the default behaviour of tar but I believe it can be changed by using the --dereference flag - which includes a copy of the file in the archive rather than the link.

Any chance of changing the tar command for building the stage archives to include the dereference flag ?

Many thanks -- Simon

[lresende]

@simon-streamotion Thanks for the interesting scenario, Elyra can certainly be updated to support it. As usual, you can provide a pull request, or we will add to our milestones to take a look at it.

[lresende]

Also, if you are interested in providing a patch, this is most likely the location that needs to be updated:
https://github.com/elyra-ai/elyra/blob/master/elyra/util/archive.py

[lresende]

BTW, I didn't realize this was just a discussion, let me transfer this to an issue.

[lresende]

It doesn't seem I can just convert this to an issue. @simon-streamotion would you mind creating a new issue, basically by copying the title and description to the new issue.

[simon-streamotion]

No worrries @lresende.
Created issue #1648

[ptitzler]

Resolved in version 3.0+ by #1689

[simon-streamotion]

@ptitzler & team - many thanks for fixing this :-)