diff --git a/proposals/20200506-artifacts-scope-part-1.md b/proposals/20200506-artifacts-scope-part-1.md index dee04956160..4f5a547d444 100644 --- a/proposals/20200506-artifacts-scope-part-1.md +++ b/proposals/20200506-artifacts-scope-part-1.md @@ -4,7 +4,7 @@ The **Falco Artifact Scope** proposal is divided in two parts: 1. the Part 1 - *this document*: the State of Art of Falco artifacts 2. the [Part 2](./20200506-artifacts-scope-part-2.md): the intended state moving forward -## Summary +## Summary As a project we would like to support the following artifacts. @@ -16,7 +16,7 @@ Inspired by many previous issues and many of the weekly community calls. ## Terms -**falco** +**falco** *The Falco binary* @@ -30,12 +30,12 @@ Inspired by many previous issues and many of the weekly community calls. **package** -*An installable artifact that is operating system specific. All packages MUST be hosted on bintray.* +*An installable artifact that is operating system specific. All packages MUST be hosted on [bintray](https://bintray.com/falcosecurity).* **image** *OCI compliant container image hosted on dockerhub with tags for every release and the current master branch.* - + # Packages @@ -52,11 +52,11 @@ List of currently official container images (for X86 64bits only): | Name | Directory | Description | |---|---|---| -| [falcosecurity/falco:latest](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:master](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/stable | Falco (DEB built from git tag or from the master) with all the building toolchain. | -| [falcosecurity/falco:latest-slim](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_-slim](https://hub.docker.com/repository/docker/falcosecurity/falco),[falcosecurity/falco:master-slim](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/slim | Falco (DEB build from git tag or from the master) without the building toolchain. | -| [falcosecurity/falco-driver-loader:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:master](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader) | docker/driver-loader | `falco-driver-loader` as entrypoint with the building toolchain. | -| [falcosecurity/falco-builder:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-builder) | docker/builder | The complete build tool chain for compiling Falco from source. See [the documentation](https://falco.org/docs/source/) for more details on building from source. Used to build Falco (CI). | -| [falcosecurity/falco-tester:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-tester) | docker/tester | Container image for running the Falco test suite. Used to run Falco integration tests (CI). | +| [falcosecurity/falco:latest](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:master](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/stable | Falco (DEB built from git tag or from the master) with all the building toolchain. | +| [falcosecurity/falco:latest-slim](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_-slim](https://hub.docker.com/repository/docker/falcosecurity/falco),[falcosecurity/falco:master-slim](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/slim | Falco (DEB build from git tag or from the master) without the building toolchain. | +| [falcosecurity/falco-driver-loader:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:master](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader) | docker/driver-loader | `falco-driver-loader` as entrypoint with the building toolchain. | +| [falcosecurity/falco-builder:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-builder) | docker/builder | The complete build tool chain for compiling Falco from source. See [the documentation](https://falco.org/docs/source/) for more details on building from source. Used to build Falco (CI). | +| [falcosecurity/falco-tester:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-tester) | docker/tester | Container image for running the Falco test suite. Used to run Falco integration tests (CI). | | _to not be published_ | docker/local | Built on-the-fly and used by falco-tester. | **Note**: `falco-builder`, `falco-tester` (and the `docker/local` image which it's built on the fly by the `falco-tester` one) are not integrated into the release process because they are development and CI tools that need to be manually pushed only when updated. @@ -76,7 +76,7 @@ This new [contrib](https://github.com/falcosecurity/contrib) repository will be ### repository -"_Incubating level_" projects such as [falco-exporter](https://github.com/falco-exporter) can be promoted from `contrib` to their own repository. +"_Incubating level_" projects such as [falco-exporter](https://github.com/falco-exporter) can be promoted from `contrib` to their own repository. This is done as needed, and can best be measured by the need to cut a release and use the GitHub release features. Again, this is at the discretion of the Falco open source community. @@ -92,7 +92,7 @@ The *Part 1* is mainly intended as a cleanup process. For each item not listed above, ask if it needs to be moved or deleted. After the cleanup process, all items will match the *Part 1* of this proposal. - + ### Action Items Here are SOME of the items that would need to be done, for example: diff --git a/proposals/20200818-artifacts-storage.md b/proposals/20200818-artifacts-storage.md new file mode 100644 index 00000000000..48f3cc8c9e0 --- /dev/null +++ b/proposals/20200818-artifacts-storage.md @@ -0,0 +1,83 @@ +# Falco Artifacts Storage + +This document reflects the way we store the Falco artifacts. + +## Terms & Definitions + +- [Falco artifacts](./20200506-artifacts-scope-part-1.md) +- Bintray: artifacts distribution platform + +## Packages + +The Falco packages are **automatically** built and sent to [bintray](https://bintray.com/falcosecurity) in the following cases: + +- a pull request gets merged into the master branch (**Falco development releases**) +- a new Falco release (git tag) happens on the master branch (**Falco stable releases**) + +The only prerequisite is that the specific Falco source code builds successfully and that the tests pass. + +As per [Falco Artifacts Scope (#1)](./20200506-artifacts-scope-part-1.md) proposal we provide three kind of Falco packages: + +- DEB +- RPM +- Tarball + +Thus, we have three repositories for the Falco stable releases: + +- https://bintray.com/falcosecurity/deb +- https://bintray.com/falcosecurity/rpm +- https://bintray.com/falcosecurity/bin + +And three repositories for the Falco development releases: + +- https://bintray.com/falcosecurity/deb-dev +- https://bintray.com/falcosecurity/rpm-dev +- https://bintray.com/falcosecurity/bin-dev + +## Drivers + +The process of publishing a set of prebuilt Falco drivers is implemented by the **Drivers Build Grid (DBG)** in the [test-infra](https://github.com/falcosecurity/test-infra/tree/master/driverkit) repository (`driverkit` directory). + +This process is driven by the configuration files (YAML) present in the `driverkit/config` directory in the [test-infra](https://github.com/falcosecurity/test-infra/tree/master/driverkit) repository. + +Each of these files represents a prebuilt driver (eventually two: kernel module and eBPF probe, when possible) that will be published on [bintray](https://bintray.com/falcosecurity) if it builds correctly. + +Every time the `driverkit/config` directory on the master branch has some changes from the previous commit the CI system, which you can find defined in the [.circleci/config.yml](https://github.com/falcosecurity/test-infra/blob/master/.circleci/config.yml) file, takes care of building and publishing all the drivers. + +The driver versions we ship prebuilt drivers for are: + +- the driver version associated with the last Falco stable version ([see here](https://github.com/falcosecurity/falco/blob/c4b7f17271d1a4ca533b2e672ecaaea5289ccdc5/cmake/modules/sysdig.cmake#L29)) +- the driver version associated with the penultimate Falco stable version + +The prebuilt drivers get published into [this](https://bintray.com/falcosecurity/driver) generic artifacts repository. + +You can also visualize the full list of prebuilt drivers by driver version visiting this [URL](https://dl.bintray.com/falcosecurity/driver). + +### Notice + +The generation of new prebuilt drivers takes usually place with a frequency of 1-2 weeks, on a **best-effort** basis. + +Thus, it can happen the list of available prebuilt drivers does not yet contain the driver version currently on Falco master. + +Nevertheless, this process is an open, auditable, and transparent one. + +So, by sending a pull-request towards [test-infra](https://github.com/falcosecurity/test-infra) repository containing the configuration YAML files you can help the Falco community stay on track. + +Some pull-requests you can look at to create your own are: + +- https://github.com/falcosecurity/test-infra/pull/165 +- https://github.com/falcosecurity/test-infra/pull/163 +- https://github.com/falcosecurity/test-infra/pull/162 + +While, the documentation of the YAML configuration files can be found [here](https://github.com/falcosecurity/driverkit/blob/master/README.md). + +## Container images + +As per Falco packages, also the Falco official container images are **automatically** published to the [dockerhub](https://hub.docker.com/r/falcosecurity/falco). + +These images are built and published in two cases: + +- a pull request gets merged into the master branch (**Falco development releases**) +- a new Falco release (git tag) happens (**Falco stable releases**) + +For a detailed explanation of the container images we build and ship look at the following [documentation](https://github.com/falcosecurity/falco/blob/master/docker/README.md). diff --git a/proposals/20200901-artifacts-cleanup.md b/proposals/20200901-artifacts-cleanup.md new file mode 100644 index 00000000000..4038da38dcc --- /dev/null +++ b/proposals/20200901-artifacts-cleanup.md @@ -0,0 +1,102 @@ +# Falco Artifacts Cleanup + +This document reflects when and how we clean up the Falco artifacts from their storage location. + +## Motivation + +The [bintray](https://bintray.com/falcosecurity) open-source plan offers 10GB free space for storing artifacts. + +They also kindly granted us an additional 5GB of free space. + +## Goal + +Keep the storage space usage under 15GB by cleaning up the [Falco artifacts](./20200506-artifacts-scope-part-1.md) from the [storage](./20200818-artifacts-storage). + +## Status + +To be implemented. + +## Packages + +### Tarballs from Falco master + +At the moment of writing this document, this kind of Falco package requires approx. 50MB (maximum detected size) of storage space. + +Since, historically, the [bin-dev](https://bintray.com/falcosecurity/bin-dev) repository is the less used one, this document proposes to keep only the last 10 **Falco development releases** it contains. + +This means that the [bin-dev](https://bintray.com/falcosecurity/bin-dev) repository will take at maximum 500MB of storage space. + +### DEB from Falco master + +At the moment of writing this document, this kind of Falco package requires approx. 5.1MB (maximum detected size) of storage space. + +Historically, every Falco release is composed by less than 50 merges (upper limit). + +So, to theoretically retain all the **Falco development releases** that led to a Falco stable release, this document proposes to keep the last 50 Falco DEB packages. + +This means that the [deb-dev](https://bintray.com/falcosecurity/deb-dev) repository will take at maximum 255MB of storage space. + +### RPM from Falco master + +At the moment of writing this document, this kind of Falco package requires approx. 4.3MB (maximum detected size) of storage space. + +For the same exact reasons explained above this document proposes to keep the last 50 Falco RPM packages. + +This means that the [rpm-dev](https://bintray.com/falcosecurity/rpm-dev) repository will take at maximum 215MB of storage space. + +### Stable releases + +This document proposes to retain all the stable releases. + +This means that all the Falco packages present in the Falco stable release repositories will be kept. + +The [bin](https://bintray.com/falcosecurity/bin) repository contains a Falco tarball package for every release. +This means it grows in space of ~50MB each month. + +The [deb](https://bintray.com/falcosecurity/deb) repository contains a Falco DEB package for every release. +This means it grows in space of ~5MB each month. + +The [rpm](https://bintray.com/falcosecurity/rpm) repository contains a Falco RPM package for every release. +This means it grows in space of ~4.3MB each month. + +### Considerations + +Assuming the size of the packages does not surpass the numbers listed in the above sections, the **Falco development releases** will always take less that 1GB of artifacts storage space. + +Assuming 12 stable releases at year, at the current size of packages, the **Falco stable releases** will take approx. 720MB of storage space every year. + +### Implementation + +The Falco CI will have a new CI job - called `cleanup/packages-dev` - responsible for removing the **Falco development releases** depending on the above plan. + +This job will be triggered after the `publish/packages-dev` completed successfully. + +## Drivers + +As explained in the [Artifacts Storage](./20200818-artifacts-storage) proposal, we build the drivers for the **last two driver versions** associated with **latest Falco stable releases**. +Then, we store those drivers into a [generic bintray repository](https://bintray.com/falcosecurity/driver) from which the installation process automatically downloads them, if suitable. + +This document proposes to implement a cleanup mechanism that deletes all the other driver versions available. + +At the moment of writing, considering only the last two driver versions (**ae104eb**, **85c8895**) associated with the latest Falco stable releases, we ship ~340 eBPF drivers, each accounting for ~3.1MB of storage space, and 1512 kernel modules (~3.1MB size each, too). + +Thus, we obtain an estimate of approx. 2.875GB for **each** driver version. + +This document proposes to only store the last two driver versions associates with the latest Falco stable releases. And deleting the other ones. + +This way, assuming the number of prebuilt drivers does not skyrocket, we can reasonably estimate the storage space used by prebuilt drivers to be around 6GB. + +Notice that, in case a Falco stable release will not depend on a new driver version, this means the last two driver versions will, in this case, cover more than the two Falco stable releases. + +### Archivation + +Since the process of building drivers is time and resource consuming, this document also proposes to move the driver versions in other storage facilities. + +The candidate is an AWS S3 bucket responsible for holding the deleted driver version files. + +### Implementation + +The [test-infra](https://github.com/falcosecurity/test-infra) CI, specifically its part dedicated to run the **Drivers Build Grid** that runs every time it detects changes into the `driverkit` directory of the [test-infra](https://github.com/falcosecurity/test-infra) repository, +will have a new job - called `drivers/cleanup` - responsible for removing all the Falco driver versions except the last two. + +This job will be triggered after the `drivers/publish` completed successfully on the master branch. \ No newline at end of file