-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Placeholders break soopervisor #116
Comments
note that this will also close #114 |
hey, @feribg. thoughts? wanna tackle this? |
Yeah sounds fair, how about append or override null keys in env.yaml re the last step ? |
Do you mean here we should actually add the values function Should we not have code to remove those from evaluating in the runtime in the docker container then ? I guess it will still look for {{git_hash}} to replace at runtime. Also I think we can use this generic mechanism and extend the scope here so that the env var substitution can be done here as well. IE referring env vars as placeholders in the pipeline config and then injecting them into this new .env file. This might require some convention for which im not sure |
For the env, I think this is where we can pass |
@edublancas Here are the preliminary PRs I still have one question re |
can you clarify what you mean by run and build? I think However, I believe this is not what you want in your use case, but you want the If so, I think the simplest way to solve it is by adding a new placeholder automatically, e.g., |
That is indeed correct I want Currently now is being evaluated when a task starts (this is why we have a problem with dependencies because we put In a sense we almost need |
In Ploomber, we have placeholders that users can put in their
pipeline.yaml
. For example, to index experiments by{{git_hash}}
or timestamp with{{now}}
. However, the value of such placeholders is computed when loading thepipeline.yaml
.This works fine in Ploomber since the pipelines execute in a single machine; however, when using soopervisor, this breaks. For example, the
{{git_hash}}
isn't available because we don't copy the git repository, only the existing files. And{{now}}
breaks because it's evaluated for each task. Hence, each one sees a different timestamp.We can solve this by resolving placeholders when executing
soopervisor export
and putting a "resolved"pipeline.yaml
in the Docker container. Ploomber already has a feature that allows defining placeholders in a separate file (env.yaml
), we can use this feature to facilitate the implementation.This is a high level description of what I think it's the best way to do this:
docker.build
is the function that builds the docker image, and it already receives the path to thepipeline.yaml
in theentry_point
argument. We can use this value and pass it to this function (note that this is defined in ploomber), which will return is whatenv.yaml
file theentry_point
will use, let's call this valuepath_to_env
.We can use this function to get the values of the placeholders (like {{now}} and {{git_hash}}).
Then, we apply some logic depending on the value of
path_to_env
, right before compressing the source code.if
path_to_env
isNone
, we create anenv.yaml
in the same directory as thepipeline.yaml
, and write the contents of the placeholders, ifpath_to_env
is not None, then we load its contents and append the values of the placeholders. However, we should not override keys (only add the ones that do not exist).The text was updated successfully, but these errors were encountered: