Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explode flattened buildpacks when pack build #1787

Merged
merged 39 commits into from
Jul 10, 2023
Merged

Explode flattened buildpacks when pack build #1787

merged 39 commits into from
Jul 10, 2023

Commits on Jun 2, 2023

  1. Avoid writing duplicate tar files for flattened buildpacks 

    When `pack build --buildpack <flattened buildpack>` is invoked,
`pack` doesn't know the buildpack is flattened, so it treats it like
`pack build --buildpack bp/1 --buildpack bp2`,
resulting in unnecessary tar files being created
and duplicate layers being added to the ephemeral builder,
which can exhaust disk space depending on the environment.

This optimization avoids writing the tar file more than once
by returning a no-op read closer
when the diffID has already been queued for downloading.

Signed-off-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    735bfc8 View commit details
    Browse the repository at this point in the history

  2. WIP 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    362052a View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2023

  1. Implementing logic split flattened modules into different tar files, and 

    calculate which buildpacks must be added into the ephemeral builder

Skipping pack build -b <flattened-buildpack> in windows container
because acceptance tests doesn't support it yet

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    03808b7 View commit details
    Browse the repository at this point in the history

  2. skipping --flatten test on previous versions of pack 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    f5cce3c View commit details
    Browse the repository at this point in the history

  3. buildpack directories not supported on windows 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    c456691 View commit details
    Browse the repository at this point in the history

  4. WIP - logging again, it seems we have a concurrency issue 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    f938d7e View commit details
    Browse the repository at this point in the history

  5. fixes 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    36fcb55 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. removing some structs that were duplicated 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    c8e0705 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. Fixing weird behavior in the acceptance tests 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    47f76c4 View commit details
    Browse the repository at this point in the history

  2. Applying review suggestions from Joe Kimmel 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    12126f3 View commit details
    Browse the repository at this point in the history

  3. Apply suggestions from code review 

    Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante @natalieparellano
    jjbustamante and natalieparellano authored Jun 20, 2023
    Configuration menu
    Copy the full SHA
    f23c0df View commit details
    Browse the repository at this point in the history

  4. fixing assertions in flattened buildpack tests 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    39b275e View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2023

  1. fixing unit tests assertions 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    3903d96 View commit details
    Browse the repository at this point in the history

  2. Apply suggestions from code review 

    Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante @natalieparellano
    jjbustamante and natalieparellano authored Jun 21, 2023
    Configuration menu
    Copy the full SHA
    1255cc0 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2023

  1. refactoring the implementation of ToNLayerTar method to use the recur… 

    …sion suggested by Natalie

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    d38479d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into fix/flatten-tar-extras-2

    @jjbustamante
    jjbustamante authored Jun 22, 2023
    Configuration menu
    Copy the full SHA
    1a47dd7 View commit details
    Browse the repository at this point in the history

  3. adding some feedback from review 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    ff30cd4 View commit details
    Browse the repository at this point in the history

  4. fixing path errors on windows 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    1adc59e View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2023

  1. trying to fix the path issue on windows on a different way, cleaning … 

    …all the paths when creating the build modules

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    dceef0d View commit details
    Browse the repository at this point in the history

  2. trying to fix the path issue on windows on a different way, cleaning … 

    …all the paths when creating the build modules

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    7b867ad View commit details
    Browse the repository at this point in the history

  3. attempt to fix windows errors 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    0325a3d View commit details
    Browse the repository at this point in the history

  4. fixing windows issue 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    c3c11cc View commit details
    Browse the repository at this point in the history

  5. adding test case for windows 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    144a88f View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. Removing skips on Windows 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
    @natalieparellano
    natalieparellano committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    39f29d2 View commit details
    Browse the repository at this point in the history

  2. for -> from 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
    @natalieparellano
    natalieparellano committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    b17287b View commit details
    Browse the repository at this point in the history

  3. fixing windows issue 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    31060e0 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2023

  1. Cleanups from PR review 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
    @natalieparellano
    natalieparellano committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    6f50a75 View commit details
    Browse the repository at this point in the history

  2. Fix more stuff 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
    @natalieparellano
    natalieparellano committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    e7c0a12 View commit details
    Browse the repository at this point in the history

  3. Update comment 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
    @natalieparellano
    natalieparellano committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    cead534 View commit details
    Browse the repository at this point in the history

  4. Variable rename and pend tests to see if acceptance is passing 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
    @natalieparellano
    natalieparellano committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    e14d0d9 View commit details
    Browse the repository at this point in the history

  5. Fix units by not writing parent headers twice 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
    @natalieparellano
    natalieparellano committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    03116fe View commit details
    Browse the repository at this point in the history

  6. Try to fix Windows 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
    @natalieparellano
    natalieparellano committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    84ee822 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. Other small cleanups 

    Signed-off-by: Natalie Arellano <narellano@vmware.com>
    @natalieparellano
    natalieparellano committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    d5cc7d9 View commit details
    Browse the repository at this point in the history

  2. merging suggestions from Natalie and Joe to simplify the code 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    e73699a View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into fix/flatten-tar-extras-2

    @jjbustamante
    jjbustamante authored Jun 28, 2023
    Configuration menu
    Copy the full SHA
    744aefc View commit details
    Browse the repository at this point in the history

  4. removing unused code 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    c5aa199 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2023

  1. adding test coverage 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    d5ab961 View commit details
    Browse the repository at this point in the history

  2. Attempt to fix CodeQL errors 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    f956222 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2023

  1. removing Focus 

    Signed-off-by: Juan Bustamante <jbustamante@vmware.com>
    @jjbustamante
    jjbustamante committed Jun 30, 2023
    Configuration menu
    Copy the full SHA
    d9a254b View commit details
    Browse the repository at this point in the history