The Agent embeds a full-fledged CPython interpreter so it requires the development files to be available in the dev env. The Agent can embed Python2 and/or Python3, you will need development files for all versions you want to support.
If you're on OSX/macOS, installing Python 2.7 and/or 3.8 with Homebrew brings along all the development files needed:
Please note that not using Python versions explicitly supported, you may have problems running the built Agent's Python checks, especially if using a virtualenv. At this time, only Python 3.8 is confirmed to work as expected in the development environment.
brew install python@2
brew install python@3.8
On Linux, depending on the distribution, you might need to explicitly install the development files, for example on Ubuntu:
sudo apt-get install python2.7-dev
sudo apt-get install python3.8-dev
On Windows, install Python 2.7 and/or 3.8 via the official installer.
You will also need the Visual Studio for Visual Studio for Python installer
Download the gcc toolchain.
- From the graphical package manager, select and install the needed libraries, leave the default (select all) if you're unsure.
- Make sure to select
x86_64. - Add installation folder to the
%PATH%.
To protect and isolate your system-wide python installation, a python virtual environment is highly recommended (though optional). It will help keep a self-contained development environment and ensure a clean system Python.
Please note that due to the way
some virtual environments handle executable paths
(e.g. python -m venv), not all virtual environment options will be able to run the built
agent correctly. At this time, the only confirmed virtual enviroment that is known for
sure to work is virtualenv.
- Install the virtualenv module:
python3 -m pip install virtualenv - Create the virtual environment:
virtualenv $GOPATH/src/github.com/DataDog/datadog-agent/venv - Activate the virtualenv (OS-dependent). This must be done for every new terminal before you start.
If using virtual environments when running the built Agent, you may need to override the built
Agent's search path for Python check packages using the PYTHONPATH variable (your target path
must have the pre-requisite core integration packages installed
though).
PYTHONPATH="./venv/lib/python3.8/site-packages:$PYTHONPATH" ./agent run ...Invoke is a task runner written in Python that is extensively used in this project to orchestrate builds and test runs. Our invoke tasks are only compatible with Python 3, thus you will need to use Python 3 to run them.
Though you may install invoke in a variety of way we suggest you use
the provided requirements
file and pip:
pip install -r requirements.txtThis procedure ensures you not only get the correct version of invoke, but
also any additional python dependencies our development workflow may require,
at their expected versions.
It will also pull other handy development tools/deps (reno, or docker).
Tasks are usually parameterized and Invoke comes with some default values that
are used in the official build. Such values are listed in the invoke.yaml
file at the root of this repo and can be overridden by setting INVOKE_* env
variables (see Invoke docs for more details).
You must install Golang version 1.15. Make
sure that $GOPATH/bin is in your $PATH otherwise invoke cannot use any
additional tool it might need.
Please note that versions of Golang that aren't an exact match to the version specified in our build images (see e.g. here) may not be able to build the agent and/or the rtloader binary properly.
From the root of datadog-agent, run invoke install-tools to install go tooling, then invoke deps to install go dependencies. This uses go to install the necessary dependencies.
When working on the Agent codebase you can choose among two different ways to build the binary, informally named System and Embedded builds. For most contribution scenarios you should rely on the System build (the default) and use the Embedded one only for specific use cases. Let's explore the differences.
System builds use your operating system's standard system libraries to satisfy the Agent's external dependencies. Since, for example, macOS 10.11 may provide a different version of Python than macOS 10.12, system builds on each of these platforms may produce different Agent binaries. If this doesn't matter to you—perhaps you just want to contribute a quick bugfix—do a System build; it's easier and faster than an Embedded build. System build is the default for all build and test tasks, so you don't need to configure anything there. But to make sure you have system copies of all the Agent's dependencies, skip the Embedded build section below and read on to see how to install them via your usual package manager (apt, yum, brew, etc).
Embedded builds download specifically-versioned dependencies and compile them locally from sources. We run Embedded builds to create Datadog's official Agent releases (i.e. RPMs, debs, etc), and while you can run the same builds while developing locally, the process is as slow as it sounds. Hence, you should only use them when you care about reproducible builds. For example:
- you want to build an agent binary that can be used as-is to replace the binary of an existing agent installation
- some dependencies are not available on your system
- you're working or debugging at a very low level: let's say you're adding a function to the Python bindings, you want to make sure you're using the exact same versions of Python as the official Agent packages
Embedded builds rely on Omnibus to download
and build dependencies, so you need a recent ruby environment with bundler
installed. See how to build Agent packages with Omnibus for more
details.
The agent is able to collect systemd journal logs using a wrapper on the systemd utility library.
On Ubuntu/Debian:
sudo apt-get install libsystemd-dev
On Redhat/CentOS:
sudo yum install systemd-devel
If you want to build a Docker image containing the Agent, or if you wan to run system and integration tests you need to run a recent version of Docker in your dev environment.
We use Doxygen to generate the documentation for the rtloader part of the Agent.
To generate it (using the invoke rtloader.generate-doc command), you'll need to have Doxygen installed on your system and available in your $PATH. You can compile and install Doxygen from source with the instructions available here.
Alternatively, you can use already-compiled Doxygen binaries from here.
To get the dependency graphs, you may also need to install the dot executable from graphviz and add it to your $PATH.
It is optional but recommended to install pre-commit to run a number of checks done by the CI locally.
To install it, run:
python3 -m pip install pre-commit
pre-commit installThe shellcheck pre-commit hook requires having the shellcheck binary installed and in your $PATH.
To install it, run:
inv install-shellcheck --destination <path>(by default, the shellcheck binary is installed in /usr/local/bin).
If you want to skip pre-commit for a specific commit you can add --no-verify to the git commit command.
If you want to run one of the checks manually, you can run pre-commit run <check name>.
You can run it on all files with the --all-files flag.
pre-commit run flake8 --all-files # run flake8 on all filesSee pre-commit run --help for further options.
Microsoft Visual Studio Code is recommended as it's lightweight and versatile.
Building on Windows would require multiple 3rd-party softwares to be installed. To avoid the complexity, it is recommended to make the code change in VS Code then do the build in Docker image. For complete information, see Build the Agent packages