[IMP] Add abstract dependencies installation #33

lasley · 2017-06-16T00:54:37Z

I think you guys might like this one. I haven't really tested much yet

Add dependencies installation for apt, npm, gem & pip from files in custom/dependencies folder
Support old requirements path

yajo

I love the feature, just some comments to make it fit the project perfectly 😉

BTW, this would need an update in the scaffolding too, but I can do it myself when merging.

yajo · 2017-06-16T08:51:58Z

build.d/80-dependencies

+DIR_CUSTOM = '/opt/odoo/custom/dependencies'
+
+REQ_TYPES = {
+    'apt': 'apt-get install -y',


Add --no-install-recommends

yajo · 2017-06-16T08:52:06Z

build.d/80-dependencies

+
+REQ_TYPES = {
+    'apt': 'apt-get install -y',
+    'pip': 'pip install',


Add --no-cache-dir.

yajo · 2017-06-16T08:52:31Z

build.d/80-dependencies

+    'apt': 'apt-get install -y',
+    'pip': 'pip install',
+    'npm': 'npm install -g',
+    'gem': 'gem install',


Add --no-rdoc --no-ri --no-update-sources

yajo · 2017-06-16T08:53:16Z

README.md

+            apt_dependencies.txt
+            gem_dependencies.txt
+            npm_dependencies.txt
+            pip_dependencies.txt


Since the folder is already called dependencies, do not be redundant and remove the _dependencies suffix 😊

yajo · 2017-06-16T08:53:47Z

README.md

@@ -200,6 +205,8 @@ Important notes:
 A normal [pip `requirements.txt`][] file, to install dependencies for your
 addons when building the subimage.

+###


I guess you wanted to document this feature, so please do 😉

This one is still empty.

Ooops - I never actually did this

I did it for you, don't worry 😆

yajo · 2017-06-16T08:59:42Z

build.d/80-dependencies

+
+DIR_CUSTOM = '/opt/odoo/custom/dependencies'
+
+REQ_TYPES = {


Call it INSTALL instead.

yajo · 2017-06-16T09:15:22Z

build.d/80-dependencies

+def do_install(command, requirements_file):
+    with open(requirements_file, 'r') as fh:
+        requirements = fh.read().strip()
+    command += ' %s' % requirements


Yikes, this loop is interesting, but I feel each install method has its things.

For instance,

apt needs an apt-get update before installing the build deps, and needs apt-get -y autoremove && rm -Rf /var/lib/apt/lists/* after installing run deps.

pip can better use its --requirements flag instead of this read-and-append hack. (maybe not required, see next point).

the rest of methods would benefit of removing lines from its requirements file that start by #, so we allow comments there (for pip this would mean the same as the point above).

rm /tmp/* would be great too at the end of everything.

But I wanted to be lazy! New code will be better 🚀

yajo · 2017-06-16T09:15:46Z

build.d/80-dependencies

+    with open(requirements_file, 'r') as fh:
+        requirements = fh.read().strip()
+    command += ' %s' % requirements
+    proc = subprocess.Popen(command.split(' '))


if you call split without arguments, in case there are multiple spaces it will split correctly, and strip before:

>>> " asdfasdf a sfdasdf ".split(" ") ['', '', '', 'asdfasdf', '', 'a', 'sfdasdf', '', '', '', ''] >>> " asdfasdf a sfdasdf ".split() ['asdfasdf', 'a', 'sfdasdf']

yajo · 2017-06-16T09:18:20Z

build.d/80-dependencies

+        if name == 'pip':
+            found_pip = True
+
+if not found_pip:


You can drop this line and stick with the if below.

yajo · 2017-06-16T09:18:30Z

build.d/80-dependencies

+
+if not found_pip:
+    req_file = '/opt/odoo/custom/src/requirements.txt'
+    if os.path.isfile(req_file):


Add a deprecation comment here.

yajo

You have to rebase please.

Design seems great, although there some little clunky pieces such as the apt stuff that should be treated differently depending on if we have apt, apt_build, none or both.

The Install class is OK, but I feel it a little hard to track. I mean that, after all, this code you are adding is not meant to be reused anywhere, so no need to add classes and so on; you can just add some helper methods to the odoobaselib, and then do a simple, hardcoded, sequential script that just makes its checks and installs everything, just like you would do with bash. All will be clearer, and the apt problem will be easier to fix IMHO.

However, as long as it works fine, you are free to use the class-based design if you still feel it's better 😉

yajo · 2017-06-19T08:32:42Z

README.md

@@ -200,6 +205,8 @@ Important notes:
 A normal [pip `requirements.txt`][] file, to install dependencies for your
 addons when building the subimage.

+###


This one is still empty.

yajo · 2017-06-19T08:34:14Z

lib/installer.py

+
+    def _install_apt(self):
+        self._run_command(
+            'apt-get update',


I'd like to avoid running this twice if we have both apt and apt_build. Maybe something like if not self.apt_updated: ...?

yajo · 2017-06-19T08:35:26Z

lib/installer.py

+            'apt-get -y autoremove',
+        )
+        self._run_command(
+            'rm -Rf /var/lib/apt/lists/*',


In the same line as above comment, if we have both apt and apt_build, this should go in the last step only, to avoid updating twice.

yajo · 2017-06-19T08:37:47Z

lib/installer.py

+    def _remove_apt(self):
+        for requirement in self.requirements:
+            self._run_command(
+                'apt-get purge -y %s' % requirement,


Add also an apt-get autoremove -y after the purge. It will remove all dangling dependencies too.

yajo · 2017-06-19T08:40:55Z

lib/installer.py

+    def _install_requirements(self, command, split=True, shell=False):
+        if split:
+            command = command.split()
+        for requirement in self.requirements:


I think all of the supported package managers allow passing all dependencies in a single command, can't we drop this loop?

We can, however I found it difficult in my debugging to find which packages were failing. If we try to install everything at once, installation issues have to be handled one at a time. If we separate and install, all the issues can be identified and resolved after the first build or so because we have a complete picture of the situation.

yajo · 2017-06-19T08:41:14Z

lib/installer.py

+
+    def _install_gem(self):
+        self._install_requirements(
+            'gem install --no-rdoc --no-ri --no-update-sources',


Add gem cleanup after this.

yajo · 2017-06-19T08:41:31Z

lib/installer.py

+
+    def _install_npm(self):
+        self._run_command(
+            'npm install -g',


Add npm cleanup after this.

yajo · 2017-06-19T08:42:01Z

lib/installer.py

+                shell=shell,
+            )
+
+    def _run_command(self, command, split=True, shell=False):


Didn't you add a helper for this already in odoobaselib?

lasley · 2017-06-19T14:49:33Z

I mean that, after all, this code you are adding is not meant to be reused anywhere, so no need to add classes and so on; you can just add some helper methods to the odoobaselib, and then do a simple, hardcoded, sequential script that just makes its checks and installs everything, just like you would do with bash.

I've already reused the class in #28 & it's used in the removal script too. IMO at the point three scripts are using the same context, a class was a good idea.

* Add dependencies installation for apt, npm, gem & pip from files in custom/dependencies folder * Support old requirements path

lasley · 2017-06-19T16:49:47Z

Rebase complete & PR comments attended to (except #33 (comment)). Note that I squashed all existing commits into 3ed2d8e in order to make the rebase easier. The commit responding to the most recent review is dc5c4c8

yajo suggested changes Jun 16, 2017

View reviewed changes

lasley force-pushed the feature/apt-dependencies branch 8 times, most recently from c4f3c06 to c9d889c Compare June 18, 2017 23:24

yajo suggested changes Jun 19, 2017

View reviewed changes

lasley force-pushed the feature/apt-dependencies branch from d5f3b19 to 56edd85 Compare June 19, 2017 15:34

[IMP] Add abstract dependencies installation

3ed2d8e

* Add dependencies installation for apt, npm, gem & pip from files in custom/dependencies folder * Support old requirements path

lasley force-pushed the feature/apt-dependencies branch 2 times, most recently from ade9b2f to c0c38bd Compare June 19, 2017 15:37

PR updates

dc5c4c8

lasley force-pushed the feature/apt-dependencies branch from c0c38bd to dc5c4c8 Compare June 19, 2017 15:40

yajo closed this in 30364c2 Jun 20, 2017

lasley deleted the feature/apt-dependencies branch June 20, 2017 16:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IMP] Add abstract dependencies installation #33

[IMP] Add abstract dependencies installation #33

lasley commented Jun 16, 2017

yajo left a comment

yajo Jun 16, 2017

yajo Jun 16, 2017

yajo Jun 16, 2017

yajo Jun 16, 2017

yajo Jun 16, 2017

yajo Jun 19, 2017

lasley Jun 20, 2017

yajo Jun 21, 2017

yajo Jun 16, 2017

yajo Jun 16, 2017

lasley Jun 16, 2017

yajo Jun 16, 2017

yajo Jun 16, 2017

yajo Jun 16, 2017

yajo left a comment

yajo Jun 19, 2017

yajo Jun 19, 2017

yajo Jun 19, 2017

yajo Jun 19, 2017

yajo Jun 19, 2017

lasley Jun 19, 2017 •

edited

Loading

yajo Jun 19, 2017

yajo Jun 19, 2017

yajo Jun 19, 2017

lasley commented Jun 19, 2017

lasley commented Jun 19, 2017


		DIR_CUSTOM = '/opt/odoo/custom/dependencies'

		REQ_TYPES = {

[IMP] Add abstract dependencies installation #33

[IMP] Add abstract dependencies installation #33

Conversation

lasley commented Jun 16, 2017

yajo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yajo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lasley Jun 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lasley commented Jun 19, 2017

lasley commented Jun 19, 2017

lasley Jun 19, 2017 •

edited

Loading