Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposals: Artifacts Storage + Artifacts Cleanup #1375

Merged
merged 5 commits into from
Sep 7, 2020
Merged

proposals: Artifacts Storage + Artifacts Cleanup #1375

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
59fd137
new(proposals): initial document about SoA of artifacts storage
leodido Aug 18, 2020
684c149
update(proposals): artifacts storage proposal
leodido Sep 1, 2020
d19e1f9
new(proposals): artifacts cleanup (packages part)
leodido Sep 1, 2020
b675316
update(proposals): artifacts cleanup (prebuilt drivers part)
leodido Sep 1, 2020
cda28a2
update(proposals): clarify that prebuilding drivers is on a best-effo…
leodido Sep 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions proposals/20200506-artifacts-scope-part-1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The **Falco Artifact Scope** proposal is divided in two parts:
1. the Part 1 - *this document*: the State of Art of Falco artifacts
2. the [Part 2](./20200506-artifacts-scope-part-2.md): the intended state moving forward

## Summary
## Summary

As a project we would like to support the following artifacts.

Expand All @@ -16,7 +16,7 @@ Inspired by many previous issues and many of the weekly community calls.

## Terms

**falco**
**falco**

*The Falco binary*

Expand All @@ -30,12 +30,12 @@ Inspired by many previous issues and many of the weekly community calls.

**package**

*An installable artifact that is operating system specific. All packages MUST be hosted on bintray.*
*An installable artifact that is operating system specific. All packages MUST be hosted on [bintray](https://bintray.com/falcosecurity).*

**image**

*OCI compliant container image hosted on dockerhub with tags for every release and the current master branch.*


# Packages

Expand All @@ -52,11 +52,11 @@ List of currently official container images (for X86 64bits only):

| Name | Directory | Description |
|---|---|---|
| [falcosecurity/falco:latest](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:master](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/stable | Falco (DEB built from git tag or from the master) with all the building toolchain. |
| [falcosecurity/falco:latest-slim](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_-slim](https://hub.docker.com/repository/docker/falcosecurity/falco),[falcosecurity/falco:master-slim](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/slim | Falco (DEB build from git tag or from the master) without the building toolchain. |
| [falcosecurity/falco-driver-loader:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:master](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader) | docker/driver-loader | `falco-driver-loader` as entrypoint with the building toolchain. |
| [falcosecurity/falco-builder:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-builder) | docker/builder | The complete build tool chain for compiling Falco from source. See [the documentation](https://falco.org/docs/source/) for more details on building from source. Used to build Falco (CI). |
| [falcosecurity/falco-tester:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-tester) | docker/tester | Container image for running the Falco test suite. Used to run Falco integration tests (CI). |
| [falcosecurity/falco:latest](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:master](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/stable | Falco (DEB built from git tag or from the master) with all the building toolchain. |
| [falcosecurity/falco:latest-slim](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_-slim](https://hub.docker.com/repository/docker/falcosecurity/falco),[falcosecurity/falco:master-slim](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/slim | Falco (DEB build from git tag or from the master) without the building toolchain. |
| [falcosecurity/falco-driver-loader:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:master](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader) | docker/driver-loader | `falco-driver-loader` as entrypoint with the building toolchain. |
| [falcosecurity/falco-builder:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-builder) | docker/builder | The complete build tool chain for compiling Falco from source. See [the documentation](https://falco.org/docs/source/) for more details on building from source. Used to build Falco (CI). |
| [falcosecurity/falco-tester:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-tester) | docker/tester | Container image for running the Falco test suite. Used to run Falco integration tests (CI). |
| _to not be published_ | docker/local | Built on-the-fly and used by falco-tester. |

**Note**: `falco-builder`, `falco-tester` (and the `docker/local` image which it's built on the fly by the `falco-tester` one) are not integrated into the release process because they are development and CI tools that need to be manually pushed only when updated.
Expand All @@ -76,7 +76,7 @@ This new [contrib](https://github.com/falcosecurity/contrib) repository will be

### repository

"_Incubating level_" projects such as [falco-exporter](https://github.com/falco-exporter) can be promoted from `contrib` to their own repository.
"_Incubating level_" projects such as [falco-exporter](https://github.com/falco-exporter) can be promoted from `contrib` to their own repository.

This is done as needed, and can best be measured by the need to cut a release and use the GitHub release features. Again, this is at the discretion of the Falco open source community.

Expand All @@ -92,7 +92,7 @@ The *Part 1* is mainly intended as a cleanup process.
For each item not listed above, ask if it needs to be moved or deleted.
After the cleanup process, all items will match the *Part 1* of this proposal.


### Action Items

Here are SOME of the items that would need to be done, for example:
Expand Down
83 changes: 83 additions & 0 deletions proposals/20200818-artifacts-storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Falco Artifacts Storage

This document reflects the way we store the Falco artifacts.

## Terms & Definitions

- [Falco artifacts](./20200506-artifacts-scope-part-1.md)
- Bintray: artifacts distribution platform

## Packages

The Falco packages are **automatically** built and sent to [bintray](https://bintray.com/falcosecurity) in the following cases:

- a pull request gets merged into the master branch (**Falco development releases**)
leodido marked this conversation as resolved.
Show resolved Hide resolved
- a new Falco release (git tag) happens on the master branch (**Falco stable releases**)

The only prerequisite is that the specific Falco source code builds successfully and that the tests pass.

As per [Falco Artifacts Scope (#1)](./20200506-artifacts-scope-part-1.md) proposal we provide three kind of Falco packages:

- DEB
- RPM
- Tarball

Thus, we have three repositories for the Falco stable releases:

- https://bintray.com/falcosecurity/deb
- https://bintray.com/falcosecurity/rpm
- https://bintray.com/falcosecurity/bin

And three repositories for the Falco development releases:

- https://bintray.com/falcosecurity/deb-dev
- https://bintray.com/falcosecurity/rpm-dev
- https://bintray.com/falcosecurity/bin-dev

## Drivers

The process of publishing a set of prebuilt Falco drivers is implemented by the **Drivers Build Grid (DBG)** in the [test-infra](https://github.com/falcosecurity/test-infra/tree/master/driverkit) repository (`driverkit` directory).

This process is driven by the configuration files (YAML) present in the `driverkit/config` directory in the [test-infra](https://github.com/falcosecurity/test-infra/tree/master/driverkit) repository.

Each of these files represents a prebuilt driver (eventually two: kernel module and eBPF probe, when possible) that will be published on [bintray](https://bintray.com/falcosecurity) if it builds correctly.

Every time the `driverkit/config` directory on the master branch has some changes from the previous commit the CI system, which you can find defined in the [.circleci/config.yml](https://github.com/falcosecurity/test-infra/blob/master/.circleci/config.yml) file, takes care of building and publishing all the drivers.

The driver versions we ship prebuilt drivers for are:

- the driver version associated with the last stable Falco version ([see here](https://github.com/falcosecurity/falco/blob/c4b7f17271d1a4ca533b2e672ecaaea5289ccdc5/cmake/modules/sysdig.cmake#L29))
- the driver version associated with the current development Falco version - ie., the one on [master](https://github.com/falcosecurity/falco/blob/master/cmake/modules/sysdig.cmake#L30)

The prebuilt drivers get published into [this](https://bintray.com/falcosecurity/driver) generic artifacts repository.

You can also visualize the full list of prebuilt drivers by driver version visiting this [URL](https://dl.bintray.com/falcosecurity/driver).

### Notice

The generation of new prebuilt drivers takes usually place with a frequency of 1-2 weeks.
leodido marked this conversation as resolved.
Show resolved Hide resolved
leodido marked this conversation as resolved.
Show resolved Hide resolved

Thus, it can happen the list of available prebuilt drivers does not yet contain the driver version currently on Falco master.

Nevertheless, this process is an open, auditable, and transparent one.

So, by sending a pull-request towards [test-infra](https://github.com/falcosecurity/test-infra) repository containing the configuration YAML files you can help the Falco community stay on track.

Some pull-requests you can look at to create your own are:

- https://github.com/falcosecurity/test-infra/pull/165
- https://github.com/falcosecurity/test-infra/pull/163
- https://github.com/falcosecurity/test-infra/pull/162

While, the documentation of the YAML configuration files can be found [here](https://github.com/falcosecurity/driverkit/blob/master/README.md).

## Container images

As per Falco packages, also the Falco official container images are **automatically** published to the [dockerhub](https://hub.docker.com/r/falcosecurity/falco).

These images are built and published in two cases:

- a pull request gets merged into the master branch (**Falco development releases**)
- a new Falco release (git tag) happens (**Falco stable releases**)

For a detailed explanation of the container images we build and ship look at the following [documentation](https://github.com/falcosecurity/falco/blob/master/docker/README.md).