Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docu for deployers and a simple example #2946

Merged
merged 21 commits into from
Feb 6, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5f8e71e
Add deployer documentation and simple example
AbrilRBS Feb 2, 2023
8220988
Fix note formatting and typo
AbrilRBS Feb 2, 2023
43c14b1
Use root conanfile for copy
AbrilRBS Feb 2, 2023
d71b4bb
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
5f5c892
Merge branch 'conan-io:release/2.0.0-beta' into release/2.0.0-beta
AbrilRBS Feb 6, 2023
721372a
Address review comments
AbrilRBS Feb 6, 2023
c6f43d3
Extra changes
AbrilRBS Feb 6, 2023
72ed52a
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
0aaecd0
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
1b1363f
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
9d501ef
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
12fac7a
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
80ed7f1
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
26da31a
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
f3a31e1
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
21f66b4
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
faf1365
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
27448c7
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
f774c5e
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
81c0993
Remove fn parenthesis to full deployers
AbrilRBS Feb 6, 2023
125bba4
Update reference/extensions/deployers.rst
franramirez688 Feb 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Examples

examples/conanfile/conanfile
examples/extensions/commands/custom_commands
examples/extensions/deployers/custom_deployers
examples/tools/cmake/cmake
examples/tools/files/files
examples/tools/meson/meson
Expand Down
14 changes: 14 additions & 0 deletions examples/extensions/deployers/custom_deployers.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.. _examples_extensions_deployers:




Custom deployers
================


.. toctree::
:maxdepth: 2


sources/custom_deployer_sources
76 changes: 76 additions & 0 deletions examples/extensions/deployers/sources/custom_deployer_sources.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
.. examples_extensions_deployers_sources:

Copy sources from all your dependencies
=======================================



Please, first of all, clone the sources to recreate this project. You can find them in the
`examples2.0 repository <https://github.com/conan-io/examples2>`_ in GitHub:

.. code-block:: bash

$ git clone https://github.com/conan-io/examples2.git
$ cd examples2/examples/extensions/deployers/sources
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get:

cd: no such file or directory: examples2/examples/extensions/deployers/sources

is it expected? or typo in path?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is expected as the PR in examples2 has not yet been merged



In this example we are going to see how to create and use a custom deployer.
This deployer copies all the source files from your dependencies and puts them in a specific output folder

.. note::

To better understand this example, it is highly recommended to have previously read the :ref:`Deployers <reference_extensions_deployer_direct_deploy>` reference.


Locate the deployer
-------------------

In this case, the deployer is located in the same directory than our example conanfile,
but as show in :ref:`Deployers <reference_extensions_deployer_direct_deploy>` reference,
Conan will look for the specified deployer in a few extra places besides the current working directory,
including as an absolute path, and in ``[YOUR_CONAN_HOME]/extensions/deploy/``.


Run it
------

For our example, we have a simple recipe that only lists ``zlib`` as a requirement.
With the help of the ``tools.build:download_source=True`` conf, we can force the invocation of its ``source()`` method,
which will ensure that sources are available even if no build needs to be carried out.

Now, you should be able to use the new deployer in both ``conan install`` and ``conan graph`` commands for any given recipe:

.. code-block:: bash

$ conan graph info . -c tools.build:download_source=True --deploy=sources_deploy



Inspecting the command output we can see that it copied the sources of all our dependencies (direct and transitive)
to a ``dependency_sources`` folder. After this, extra preprocessing could be accomplished to more specific needs.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dependencies_sources might be better?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this dependency_sources folder located?


Code tour
---------

The **source_deploy.py** file has the following code:


**sources_deploy.py**

.. code-block:: python

from conan.tools.files import copy
import os
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, as it's for conan 2 and python 3+, it may already use Pathlib



def deploy(graph, output_folder):
for name, dep in graph.root.conanfile.dependencies.items():
copy(graph.root.conanfile, "*", dep.folders.source_folder, os.path.join(output_folder, "dependency_sources", str(dep)))


deploy()
++++++++

The ``deploy()`` method is called by Conan, and gets both a dependency graph (``conans.client.graph.graph.DepsGraph``)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better not document a private object yet (nothing in the conans namespace should be in the docs at all), because users might thing they can instantiate or use their methods. Until we document the API of the graph, better treat it as opaque as possible, just saying that it has a root.conanfile would be enough.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if consumer receives some sort of opaque object, they need to know that are the valid methods/attributes to be safely called.
if object itself cannot be documented, then it should define some sort of an interface.
e.g. how do I know that graph has graph.root, and graph.root has graph.root.conanfile, and graph.root.conanfile has graph.root.conanfile.dependencies that is some sort of dict.
how do I know that are these object and are they stable, if they are never explained?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment, it is enough to say that graph.root.conanfile is available, and that has the documented dependencies attribute, which is good for most cases. Let's just document that, but not the whole object private interface.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and documentation on the dependencies interface says explicitly: This information does not exist in some recipe methods, only in those methods that evaluate after the full dependency graph has been computed., At the moment, this information should only be used in generate() and validate() methods. Any other use, please submit a Github issue.. it doesn't seem to say it's available in deploy() method and allowed to be used there. that's a bit contradictory from the documentation to say in one place it's not allowed, and in another it's allowed 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are referencing Conan 1.X docs. The docs for 2.0 will be (and already are) very different to those of 1.X. Deployers is a 2.0 feature, and this PR is for 2.0-beta

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, yes, that would be a bug in the docs, seems it was a copy&paste from 1.X

and an output folder path as arguments. It iterates all the dependencies of our recipe,
and copies every source file to their respective folder under ``dependency_sources`` using :ref:`conan.tools.copy<conan_tools_files_copy>`.
62 changes: 61 additions & 1 deletion reference/extensions/deployers.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,65 @@
.. _reference_extensions_deployers:

Deployers
---------
=========

Deployers are a mechanism to facilitate copying files form one folder, usually the Conan cache, to user folders.
While Conan provides two built-in ones (``full_deploy`` and ``direct_deploy``), users can easily manage their own
with ``conan config install``.

Deployers run before generators, and they can change the target folders.
For example, if the ``--deploy=full_deploy`` deployer runs before ``CMakeDeps``,
the files generated by ``CMakeDeps`` will point to the local copy in the user folder done by the ``full_deploy`` deployer,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do they need to affect on generators behavior? that's super confusing part.
as a consumer, I'd expect deployers just to deploy (copy) things, and otherwise, leave other parts untouched.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is one of most requested behaviors for deployers and generators.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's kinda surprising behavior that deployers may modify files generated by generators. can it be made more explicit? e.g. --deploy=full_deploy --deploy-modify-generator=CMakeDeps?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is kinda unspecified, what happens in case of multiple deployers, if all of them want to change target directories, what happens?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, the last one would win

and not to the Conan cache.

Deployers can be multi-configuration. Running ``conan install . --deploy=full_deploy`` repeatedly for different profiles
can achieve a fully self-contained project, including all the artifacts, binaries, and build files.
This project would be completely independent of Conan and no longer require it at all to build.


Built-in deployers
------------------

.. _reference_extensions_deployer_full_deploy:

full_deploy()
^^^^^^^^^^^^^

Deploys to ``output_folder`` + ``host/dep/0.1/Release/x86_64`` subfolder


.. _reference_extensions_deployer_direct_deploy:

direct_deploy()
^^^^^^^^^^^^^^^

Deploys only your direct dependencies to ``output_folder``



Custom deployers
----------------

Custom deployers can be managed via ``conan config install``. When looking for a specific deployer,
Conan will look in these locations for the deployer in the following order:

#. Absolute paths
#. Relative to cwd
#. In the ``conancache/extensions/deploy`` folder
#. Built-in deployers

For each installed file, Conan will look for a ``deploy()`` method which to call.
The function signature of your custom deployers should be as follows:

**my_custom_deployer.py**

.. code-block:: python

def deploy(graph: conans.client.graph.graph.DepsGraph, output_folder: str):

(Note that the arguments are passed as named parameters, so both the ``graph`` and ``output_folder`` names are mandatory)

Your custom deployer can then be invoked as if it were a built-in deployer using the filename in which it's found,
in this case ``conan install . --deploy=my_custom_deployer``.

See the :ref:`custom deployers<examples_extensions_deployers>` section for examples on how to implement your own deployers.