[Request] Pre-install platformio inside OctoPrint image

[jrmoserbaltimore]

What were you doing?

Attempted to install Marlin Flasher and flash new firmware.

What did you expect to happen?

Upload firmware after installing the plug-in.

What happened instead?

Discovered I needed to alter the system environment before Marlin Flasher would work.

Version of OctoPi

0.18.0

Feature request

See instructions for Marlin Flasher regarding PlatformIO installation. This is a simple system tweak adding a Python 3 virtualenv.

The installation would be as such:

sudo apt install python3 python3-pip python3-virtualenv
cd ~
virtualenv -p python3 pio
cd pio
. ./bin/activate
pip3 install platformio

However, python3-virtualenv appears to be broken, so a pip3 install virtualenv may be necessary instead of installing the python3-virtualenv package.

Then a udev profile is necessary:

sudo wget \
  https://raw.githubusercontent.com/platformio/platformio-core/master/scripts/99-platformio-udev.rules \
  -O /etc/udev/rules.d/99-platformio-udev.rules
sudo usermod -aG dialout,plugdev $USER

Where $USER is pi here, the user as which OctoPrint runs.

This allows plugins to access platformio and perform firmware upgrades on printers, using the path $HOME/pio/bin/pio as the platformio executable. No adjustments are necessary for OctoPrint, as this is a system modification allowing OctoPrint plugins to perform firmware updates. Marlin Flasher will actually build a PlatformIO project directly on the pi and install the firmware if successful.

[guysoft]

It might make sense to ship the 99-platformio-udev.rules.
However regarding the package, why not use /home/pi/oprint/bin/pip install ?

[cp2004]

According to the author's setup page:

Just don't install it in the same virtualenv as OctoPrint uses, they have some conflicting dependencies.

[guysoft]

That's beyond me, I can't support two seprate virtualenv. I barely find the time to maintain the one.
And it would seem like bloating to ship the two.

[jrmoserbaltimore]

I don't see what maintenance all that takes. It's just the one pip3 install platformio; put it in the build script for the image and you're good.

I guess it's also possible to spawn a shell that leaves the virtualenv and recreates the platformio virtualenv (to upgrade/clean it), if you wanted to do all that. Again, pretty much 3 steps (delete existing directory; create virtualenv; pip3 install platformio).

[cp2004]

You have 8 commands in the request, that's hardly 'just one line'. It does seem unnecessary to me, for one plugin that's not used by many people. Sure, it's used by some but out of 115k tracked installs only 1.5k have this plugin installed (last 30 days)

The extra maintenance comes in when it breaks, not the initial installation.

[guysoft]

Maintenance has costs.
That pip install also requires a virtualenv, aka a whole septate environment that would be shipped hundreds of thousands of times.
It also means that when it breaks someone needs to keep an eye on it. Add tests etc. I am not going to do that for a plugin that is run by a commercial company. I am a volunteer doing this in my spare time.
I would consider adding a script that would install the virtualenv in to /home/pi/scripts, however shipping it would be irresponsible on my end, it would bloat the system.

[jrmoserbaltimore]

Setting up the udev rule and adding a script to install/reinstall would make a lot of sense to me. It'd at least give an out-of-the-box way to install firmware updates (using one of the plugins available through the config menu, anyway).

Didn't realize size was a factor. OctoPi is about 2G, while an OctoPrint docker container on alpine is 540M, and a python virtualenv with platformio in it is 27M (a significant chunk of an Alpine-based image). Might be able to shrink the download by about 80% by rebasing on alpine.

[guysoft]

1. OctoPi compressed is under 700MB. It holds more stuff than the Alpine container. Kernel, modules and more.

Alpine is a built for containers, they only ship tarballs. No .img which is not compatible with CustomPiOS
3. This is a lot of work and I am a volunteer

[jrmoserbaltimore]

You know what? Never mind. These are all thinly-veiled excuses for I don't know what, but the amount of work is trivial—it's just people using OctoPi expect an out-of-the-box solution and they have to go wandering around google for half an hour or so to figure out what the best way to get it to do a firmware update might be, since a plug-in can't just ship the change (it's against OctoPrint policy). Once you actually figure out what to do, it's pretty much run this and everything is good to go 40 seconds later, so who cares anyway?

Your excuses are that it's irresponsible to ship something that compresses to 6M more, and that it's too much work, and everything else under the sun. Your tone is hostile. It's like dealing with Theo da Raadt, who pretty much shouts at everyone who disagrees with him and informs them he's much smarter than they are in the first thirty seconds of the conversation.

So forget it. I don't care, I figured it would be nice for anyone who didn't want to get under the hood and so got a turnkey solution like OctoPi, but it turns out the guy maintaining OctoPi is one of those people. If users want to update the firmware on their 3D printer, they can unplug it from OctoPi, carry it into whatever room has their computer, and plug it in there; or they can learn to not use a turnkey solution but maybe actually install it themselves. It's probably unsustainable to rely on software maintained by someone like you.

[cp2004]

I don't think such hostile criticism is valid.

I don't see constructive responses on either side. Comments like 'I don't see what maintenance it takes' and 'It's just one line' are unhelpful to someone maintaining this for nearly 8 years. That's a lot of commitment. There is much more to maintaining software than writing things once, and Guy has proven that he cares about this.

Equally, he never explicitly said exactly what could be done, but said 'It might make sense' (udev rules) or 'I would consider' (installation script) - never 'Absolutely not, such as stupid idea'. There are some quick comments that come across as strong disagreement yes, but nothing against the core idea.

I agree that it does not make sense to preinstall everything, the key argument is that if you do it for this plugin, why not preinstall all dependencies for all plugins? That's when you end up with bloat. Scripts to do this for the minority who want this? Makes sense.

Please, both of you, more constructive conversation. Less of the 'I don't see why not' approach. People are busy, and stress levels are generally high at the moment for everyone.

[guysoft]

@jrmoserbaltimore You are welcome to fork OctoPi and go under the hood. I am not stopping you. Its actually nice to hear that it seams so trivial for you. When I started out there was no documentation on how to do that at all, I had to write it myself.
You made a request, as the title suggests, and I am explaining to you what is blocking it. Which is maintaining an extra virtualenv is indeed not something that has any justification. I see you are saying I am making excuses, so let me be direct:

You are not special, you are not getting a virtualenv. It will be irresponsible because it means I would go against people expecting OctoPi not to be bloated, and an unfair advantage to one commercial company. it would be irresponsible from ME. Not YOU.

I gave a logical path which is to add a script to install what is needed. Its a script that can be invoked by a plugin.

Here is an example of how this should go, this issue is about Bed Level Visualizer plugin with its maintainer asking for numpy to be added: #676
We talked it out and figured out why there was a technical limitation. And the minimal change so it would all work right. Turns out the upgrade to python3 in the last release made the build use a wheel making the install faster.

You didn't even say what dependences are so critical and important that they demand a virtualenv. It might be that a simple change, one line, in this plugin would not require a virtualenv at all.

[foosel]

Just to chime in here: @jrmoserbaltimore I'm with @cp2004, you completely overreacted here. There's absolutely no need for such a hostile reaction and to call @guysoft names just because he didn't immediately implement your request.

Speaking of which - I agree with his assessment, adding platformio on the off chance that maybe someone might want to install this plugin is a bad cost/usefulness ratio (it's not just "one line"). A quick install script on the other hand would make sense. Heck, the plugin could even do the needful itself - as long as things happen with the user's consent, installing something from within a plugin is totally fine, and it wouldn't be the first plugin to do something like that either. And frankly, that would probably be the best solution for everyone involved here really.

So... chill. We are all affected by the current situation out there, we should try to get along even more than usual instead of attacking each other. See also OctoPrint On Air #37

[tprelog]

This almost seems like an xy problem --

The user is requesting platformio but what they really want is to upgrade their printer firmware using OctoPi

If users want to update the firmware on their 3D printer, they can unplug it from OctoPi, carry it into whatever room has their computer, and plug it in there; or they can learn to not use a turnkey solution but maybe actually install it themselves

For which a solution may already exist -- OctoPi plugin to upgrade your printers firmware

[jrmoserbaltimore]

Yes, I'm aware. I actually filed an upstream ticket and it was closed because OctoPrint does not allow plugins that install such software (i.e. you'd have to go manually fetch and install the plugin).

The OctoPrint firmware update plugin is also in in the list in Octoprint, and I in fact installed it. It didn't work.

Board-Specific Configuration
Plugin settings vary depending on the flashing tool and are documented on the page for each flash method. Follow the instructions on the appropriate page to install and configure any necessary tools:

Atmega (AVR) based boards
AT90USB based boards
LPC176x and other boards which are updated from the SD card
SAM based boards
STM32 based boards which do not update from the SD card

It didn't work because I did not install any of this. The plugin could install this stuff, in theory, but apparently that would get it excluded from the OctoPrint plugin feed.

It took me a few hours to figure out how to get Platformio installed on my desktop, no big deal. I had marlin compilation crash several times (it doesn't work under the usual Arduino toolchain) and had to install Python and then run some commands in Powershell and then it worked, and then I could use a plugin in VSCode to compile it. The Marlin Updater plugin lets you alter the Configuration.h file, zip it up, upload, and it'll do all that for you.

My concern is simply that a normal user who downloads an image, clicks the tool to spit the image onto an SD card, plugs it in, and points Cura at it (by searching OctoPrint in the Cura Marketplace within Cura) can click around and feel out how to get this working without any real technical knowledge. If you ever want to update your firmware, you can plug into your computer and click "Update Firmware" in Cura. If your printer is in another room connected to your Wifi, you can open a shell, ssh into OctoPi, manually install some software, alter some of the system configuration….

I went straight for the path of least resistance, which is how I ended up just using the plug-in that uses the build environment on the OctoPi. I realize the firmware updaters can be also supported by using the other tools, as well, and that complexity and potential support issue can be pushed off on guy instead of the user; I figured "tell it to do X and walk away" was asking approximately zero since the maintenance should be approximately zero. I actually considered just tossing the commands into the build script and sending a pull request, but I'm not set up for a build environment here.

As to hostility, I respond with open hostility to anything that looks like obvious bs or passive-aggression. "It would be irresponsible and a huge amount of maintenance I can't handle to have a separate virtualenv that downloads in 6MB compressed and consumes 30MB on the system bloat this out so that firmware install plugins can be used by people who basically point and click" sounds like bs to me. So does balking at enabling a free, MIT-licensed plugin written by some guy in Belgium because he is apparently a "commercial company"—who cares about the identity of people involved in open source software?

It's also disingenuous for someone to account things like cd and sudo as commands adding to the complexity of the process; although admittedly I implicitly narrowed the scope to setting up and configuring the specific software (which is one command) and ignored the housekeeping (make sure python is installed and that the virtualenv thing works, which is already true on OctoPi anyway, and switch into the virtualenv to run the pip command). I tend to filter out things that get chucked into the same bin as "first, turn the computer on."

There are precious few explanations for any of these responses; although perhaps I should apply Hanlon's Razor more often—especially when everything seemed peachy fine when it could be "in the same virtualenv," hopefully not interacting somehow weird, breaking OctoPrint in strange and unexpected ways, and maximizing the maintenance overhead. If I wanted to avoid maintenance problems down the road, I'd split things off into separate Docker containers, or Python virtualenvs which are a good deal more lightweight. It seems moving to a situation likely to reduce maintenance overhead was the trigger for the maintenance overhead being too much.

[cp2004]

The plugin could install this stuff, in theory, but apparently that would get it excluded from the OctoPrint plugin feed.

To clarify - it could install this with a press of a button, but not automatically without the user's consent.

As to hostility, I respond with open hostility anything that looks like obvious bs or passive-aggression.

Please, don't. Just because one person objects, does not warrant a hostile response. You must respect the people who you are asking to put in work for you.

[foosel]

I respond with open hostility to anything that looks like obvious bs or passive-aggression

And I'm kindly asking you to not do that in the broader OctoPrint community, because that kind of attitude only leads to unnecessary escalation over misunderstandings.

Things are easily misread, especially in written language, especially when cultural differences or language barriers are involved, especially when people need to be terse for the sake of efficiency. You are dealing with a huge multi cultural community and with project maintenance on a very small number of shoulders.

If your initial response to an argument is to go into full attack mode at perceived bs, you need to change that if you want to get any kind of positive outcome from discussions in this community.

[jrmoserbaltimore]

Please, don't. Just because one person objects, does not warrant a hostile response. You must respect the people who you are asking to put in work for you.

Yes well that ship sailed after "this is fine, if I can lump it together and all that it implies into the isolated environment with the tool I'm currently maintaining" into "oh crap, no, isolating it in its own environment so it doesn't cause perhaps unexpected maintenance problems would cause too much maintenance!" and ranting about something about "commercial companies" and the horrible bloat of a few megabytes of lightweight virtualenv containers.

I closed the ticket specifically to withdraw the request as a way to make the point that those kinds of responses have made me lose respect for the maintainer. If you think there's some misreading or language barrier between the semantics, you're not paying attention to the words being said. The assertions made were clear; they were also conflicting and ridiculous, at least to someone who has broken things down into separate isolated environments (mainly via Docker) as a way to reduce maintenance overhead, and who knows the scale of storage these days. The willingness to intermix another large body of changes that may alter the virtualenv into the main virtualenv, but unwillingness to separate large and complex bodies of code because of "maintenance costs," shows either a complete lack of understanding of how any of this works or the belief that the guy on the other end doesn't know anything about computers and hopefully will go away if given some vague excuse.

So I decided to "go away"—i.e. withdraw the request, since that seems to be what he wanted. Not like any of that matters, and it's pointless to argue anyway due to the way social dynamics work, but I'm still going to put strain on that kind of crap. I don't expect any positive outcomes from discussions in a community like that—I saw what happened to Michael Meeks and Kon Colivas, and I've dealt with Theo (who both times did what I told him to do 3 years later, resisting mostly by claiming everyone knows his name so I must be wrong, which was hilarious at the time and still is) and Drepper before. Your community gets binned with their communities, not for saying no—I don't frequently get what I want, usually for good reason or just plain lack of interest by the maintainers—but for saying no in a dishonest and patronizing fashion.

Now enough of this.