Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node-Red - add support for node-red-contrib-generic-ble #70

Merged
merged 3 commits into from
Oct 13, 2020

Conversation

Paraphraser
Copy link

Following on from issue #64, this adds support for Bluetooth in
Node-Red:

  1. Adds two volume mappings to the template service.yml:

    - /var/run/docker.sock:/var/run/docker.sock
    - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
    
  2. Adds device mappings to the template service.yml:

    devices:
    	- "/dev/ttyAMA0:/dev/ttyAMA0"
    	- "/dev/vcio:/dev/vcio"
    	- "/dev/gpiomem:/dev/gpiomem"
    
  3. Changes build.sh to:

    • unconditionally satisfy the eudev-dev dependency for
      node-red-contrib-generic-ble by inserting these lines in the
      generated Dockerfile:

       USER root
       RUN apk update
       RUN apk upgrade
       RUN apk add --no-cache eudev-dev
      
    • Add node-red-contrib-generic-ble to the list of optional nodes.

  4. Changes menu.sh to:

    • define a user_in_group function to facilitate checking whether
      the current user is a member of a particular group.
    • extends the initial checking performed by the menu to ensure that
      the current user is a member of both docker and bluetooth groups.

On the first run of the menu after this patch is applied, most users
will likely encounter a reboot prompt enforcing membership of the
bluetooth group.

Following on from issue SensorsIot#64, this adds support for Bluetooth in
Node-Red:

1. Adds two volume mappings to the template `service.yml`:

	```
	- /var/run/docker.sock:/var/run/docker.sock
	- /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
	```

2. Adds device mappings to the template `service.yml`:

	```
	devices:
		- "/dev/ttyAMA0:/dev/ttyAMA0"
		- "/dev/vcio:/dev/vcio"
		- "/dev/gpiomem:/dev/gpiomem"
	```

3. Changes `build.sh` to:

	* unconditionally satisfy the `eudev-dev` dependency for
`node-red-contrib-generic-ble` by inserting these lines in the
generated `Dockerfile`:

		```
		USER root
		RUN apk update
		RUN apk upgrade
		RUN apk add --no-cache eudev-dev
		```

	* Add `node-red-contrib-generic-ble` to the list of optional nodes.

4. Changes `menu.sh` to:

	* define a `user_in_group` function to facilitate checking whether
the current user is a member of a particular group.
	* extends the initial checking performed by the menu to ensure that
the current user is a member of both `docker` and `bluetooth` groups.

> On the first run of the menu after this patch is applied, most users
will likely encounter a reboot prompt enforcing membership of the
`bluetooth` group.
@Slyke Slyke mentioned this pull request Jun 17, 2020
Checks for existence of group.
Slyke pushed a commit that referenced this pull request Jun 20, 2020
@Slyke
Copy link
Collaborator

Slyke commented Jun 20, 2020

I just added all the logic in this PR to the new-menu branch on PR #77

If you have this repo as your origin, you can:

git checkout master
git pull
git checkout new-menu

To test it out.

@Paraphraser
Copy link
Author

I can't think why it should affect anything but I set up my test RPi like this:

$ git clone https://github.com/SensorsIot/IOTstack.git IOTstack
$ cd IOTstack
$ git remote add -f Paraphraser https://github.com/Paraphraser/IOTstack.git
$ git fetch Paraphraser

after that I can create a temporary branch to merge whichever of my pending pull requests I'm interested in:

$ git checkout -b merge-pull-requests
$ git merge -m "local merge branch" remotes/Paraphraser/20200610-templates-quotePorts

switching back and forth between that and master as appropriate.

With that as background, I applied your commands:

$ git checkout master
$ git pull
$ git checkout new-menu

The checkout failed so I changed to:

$ git checkout remotes/origin/new-menu
$ git status
HEAD detached at origin/new-menu
nothing to commit, working tree clean

Then I ran the menu.

$ ./menu.sh 
Checking for project update
fatal: Couldn't find remote ref remotes/origin/new-menu
An update is available for IOTstack
fatal: Couldn't find remote ref remotes/origin/new-menu
HEAD detached at origin/new-menu
nothing to commit, working tree clean
Traceback (most recent call last):
  File "./scripts/python_deps_check.py", line 3, in <module>
    import blessed
ModuleNotFoundError: No module named 'blessed'
Traceback (most recent call last):
  File "./scripts/python_deps_check.py", line 3, in <module>
    import blessed
ModuleNotFoundError: No module named 'blessed'
Python Version: 'Python 3.7.3'. Python is up to date.
PyYAML Version: 'Unknown'. ./menu.sh: line 55: [: Unknown: integer expression expected
./menu.sh: line 63: [: Unknown: integer expression expected
./menu.sh: line 72: [: Unknown: integer expression expected
PyYAML is outdated.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-dev is already the newest version (3.7.3-1).
python3-dev set to manually installed.
python3-pip is already the newest version (18.1-5+rpt1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pyyaml==5.3.1
  Downloading https://www.piwheels.org/simple/pyyaml/PyYAML-5.3.1-cp37-cp37m-linux_armv7l.whl (44kB)
    100% |################################| 51kB 82kB/s 
Collecting blessed
  Downloading https://files.pythonhosted.org/packages/97/68/db2c1fedc40efc428128a21de4be3111584fd82844053428b40e41514fcb/blessed-1.17.8-py2.py3-none-any.whl (76kB)
    100% |################################| 81kB 1.6MB/s 
Requirement already satisfied, skipping upgrade: six>=1.9.0 in /usr/lib/python3/dist-packages (from blessed) (1.12.0)
Collecting wcwidth>=0.1.4 (from blessed)
  Downloading https://files.pythonhosted.org/packages/ef/94/a17155b400812f0558093c6fb99f92ba823e24757d5884e3ec60f5e81319/wcwidth-0.2.4-py2.py3-none-any.whl
Installing collected packages: pyyaml, wcwidth, blessed
Successfully installed blessed-1.17.8 pyyaml-5.3.1 wcwidth-0.2.4
Docker version 19.03.11 >= 18.2.0. Docker is good to go.
Project dependencies up to date

The first time I let it do what it wanted by way of project update. After the installation of Python, I tried to build the stack. The display re-painted with "IOTstack Build Menu" but that was the only thing on the screen. I hit ESC and tried again.

The second time:

Checking for project update
fatal: Couldn't find remote ref remotes/origin/new-menu
An update is available for IOTstack
Python Version: 'Python 3.7.3'. Python is up to date.
PyYAML Version: 'pyyaml 5.3.1'. PyYAML is up to date.
Blessed Version: 'blessed 1.17.8'. Blessed is up to date.
Docker version 19.03.11 >= 18.2.0. Docker is good to go.
Project dependencies up to date

I tried to build the stack and got the same result. Rinse, repeat for a few more attempts.

I did not try the Delete 'out of date' prompt files option.

Any ideas?

Screen Shot 2020-06-21 at 18 15 04

@Slyke
Copy link
Collaborator

Slyke commented Jun 21, 2020

Hmmm. It looks like the git errors are from origin not existing. What does the git remote get-url origin command return?

You should be able to run the new menu directly with python3 ./scripts/menu_main.py does it produce the same result?

I had seen this blank screen a lot when developing, and it was always due to it somehow using Python 2.7 executable, or another version of Blessed running from Python 2. I'm going to do a fresh image of a RPi right now and see if I can reproduce this with the latest commit.

If you press the esc key when it's stuck like that, does it go back to the main menu? It should at least print the top of the menu:
image
If it's not, it means term.center() is causing an error for some reason.
I just added a error message if the except gets triggered during the rendering of the build screen to try to track down what's causing this error.

This is my output:

Project is up to date
Python Version: 'Python 3.7.3'. Python is up to date.
PyYAML Version: 'pyyaml 5.3.1'. PyYAML is up to date.
Blessed Version: 'blessed 1.17.8'. Blessed is up to date.
Docker version 19.03.8 >= 18.2.0. Docker is good to go.

@Paraphraser
Copy link
Author

pi@sec-dev:~/IOTstack $ git remote get-url origin
https://github.com/SensorsIot/IOTstack.git
pi@sec-dev:~/IOTstack $ 

Then python3 ./scripts/menu_main.py has same result - blank build screen.

@Slyke
Copy link
Collaborator

Slyke commented Jun 21, 2020

For the git issue, what happens if you run the command: git name-rev --name-only HEAD? This command should determine what your current branch is, and possibly the source of the fatal: Couldn't find remote ref remotes/origin/new-menu issue.

Running ./menu.sh --branch new-menu should tell the script to use the branch new-menu without it checking git.

My SD card juuust finished being imaged. Going to boot it up right now and install IOTstack on it.

@Paraphraser
Copy link
Author

IMG_0059
IMG_0058
Tried direct with Python a couple of times. Tried menu.sh. Tried clearing update warning block. Always variations on the same theme - seems stuck updating and blank build.

@Paraphraser
Copy link
Author

pi@sec-dev:~/IOTstack $ git name-rev --name-only HEAD
remotes/origin/new-menu
pi@sec-dev:~/IOTstack $

Trying ./menu.sh --branch new-menu still prompted for updates, seemed to do something, then in build produced

There was an error rendering the menu:
'latin-1' codec can't encode characters in position 35-116: ordinal not in range(256)
Press [Esc] to go back

Then I tried ./menu.sh --branch remotes/origin/new-menu and skipped letting it do updates. The screen kept repainting over and over.

Cleared the updates block. Tried again, this time permitting updates. Still constant repainting plus this left behind:

pi@sec-dev:~/IOTstack $ ./menu.sh --branch remotes/origin/new-menu
Checking for project update
fatal: Couldn't find remote ref remotes/origin/new-menu
An update is available for IOTstack
fatal: Couldn't find remote ref remotes/origin/new-menu
HEAD detached at origin/new-menu
nothing to commit, working tree clean
Python Version: 'Python 3.7.3'. Python is up to date.
PyYAML Version: 'pyyaml 5.3.1'. PyYAML is up to date.
Blessed Version: 'blessed 1.17.8'. Blessed is up to date.
Docker version 19.03.11 >= 18.2.0. Docker is good to go.
Project dependencies up to date

Wondered about "detached head" so tried git checkout master and then ./menu.sh --branch remotes/origin/new-menu and, bingo, choosing build showed the container list.

Exit. Retry from ./menu and it worked again. There must be some difference between my checkout command and what the script does internally. Whaddyathink?

@Slyke
Copy link
Collaborator

Slyke commented Jun 21, 2020

Ah, this error I think is the reason: 'latin-1' codec can't encode characters in position 35-116: ordinal not in range(256) Looks like your terminal can only render ASCII characters, or something. I was initially thinking that it was maybe the console was too small and OSX was sending something weird to blessed and was causing it to crash.

I will remove the fancy pants stuff, and lets see if that makes it work!

@Slyke
Copy link
Collaborator

Slyke commented Jun 21, 2020

Okay, I just pushed a new version of the build menu, where I've removed all non ASCII characters. Does that make it work? If not try starting ./menu.sh with a maximised console, just in case that is causing it.

@Paraphraser
Copy link
Author

pi@sec-dev:~/IOTstack $ git pull origin
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 17 (delta 10), reused 17 (delta 10), pack-reused 0
Unpacking objects: 100% (17/17), done.
From https://github.com/SensorsIot/IOTstack
   442888e..44c5789  new-menu   -> origin/new-menu
   d378790..ff99331  udpate-documentation-custom-settings -> origin/udpate-documentation-custom-settings
Already up to date.
pi@sec-dev:~/IOTstack $ ./menu.sh --branch remotes/origin/new-menu
~/IOTstack ~/IOTstack
checking for project update
From https://github.com/SensorsIot/IOTstack
 * branch            master     -> FETCH_HEAD
Project is up to date
checking docker version
Docker version >= 18.2.0. You are good to go.
Build cancelled
pi@sec-dev:~/IOTstack $ 

No funny stuff with updates. Build stack = instant proper menu.

I use a mixture of macOS Terminal + ssh, and an iPad app called Termius which sshes to the Pi. Similar behaviours on both. Whether that means anything ...

@Slyke
Copy link
Collaborator

Slyke commented Jun 21, 2020

Wait, does that mean it works :D?

@Slyke
Copy link
Collaborator

Slyke commented Jun 21, 2020

I would also try recloning it from SensorsIoT and running it that way. The other menus such as Docker Commands, Miscellaneous Commands and so on still have the special characters that would cause a crash if having non-ASCII characters is the issue. Either way, since it is a potential bug that causes a hard lockup, I will add in a check to see if it will error or not, and switch to basic ASCII if so.

Do let me know if you find any other bugs! I'm still testing it out myself. Once I'm happy I'll merge it into the experimental branch. Most of the MVPs are done, just need to add the latin/ASCII switch and then the main testing begins!

Experimental PR: #77

@Paraphraser
Copy link
Author

Long story - Discord discussion, trying things in VNC, wondering about date formats, experimenting, then trying sudo dpkg-reconfigure locales and adding

IMG_0060

Do you think that might be implicated? Feels counterintuitive because, if anything, this looks like it should be enabling non-ASCII rather than the other way around. Also can't really see how it would affect ssh sessions. It's just the only thing I could think of that might be implicated.

@Paraphraser
Copy link
Author

Wait, does that mean it works :D?

I'm thinking "no". I'm thinking that:

$ git checkout master
$ ./menu.sh --branch remotes/origin/new-menu

isn't actually doing anything on remotes/origin/new-menu. I was so surprised by seeing something working that I didn't realise it was "old menu".

I'll give a clean checkout a whirl.

@Paraphraser
Copy link
Author

Clean checkout is an improvement so I guess tangling things up with my fork did cause hassles. Learnt something there!

Basic commands and responses:

$ git clone https://github.com/SensorsIot/IOTstack.git ~/IOTstack
Cloning into '/home/pi/IOTstack'...
remote: Enumerating objects: 129, done.
remote: Counting objects: 100% (129/129), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 1727 (delta 61), reused 110 (delta 49), pack-reused 1598
Receiving objects: 100% (1727/1727), 814.10 KiB | 1003.00 KiB/s, done.
Resolving deltas: 100% (963/963), done.

$ cd IOTstack
$ git checkout master
Already on 'master'
Your branch is up to date with 'origin/master'.

$ git pull
Already up to date.

$ git checkout new-menu
Branch 'new-menu' set up to track remote branch 'new-menu' from 'origin'.
Switched to a new branch 'new-menu'

$ ./menu.sh 
Checking for project update
From https://github.com/SensorsIot/IOTstack
 * branch            new-menu   -> FETCH_HEAD
Project is up to date
Python Version: 'Python 3.7.3'. Python is up to date.
PyYAML Version: 'pyyaml 5.3.1'. PyYAML is up to date.
Blessed Version: 'blessed 1.17.8'. Blessed is up to date.
Docker version 19.03.11 >= 18.2.0. Docker is good to go.
Project dependencies up to date

touch: cannot touch './services/nodered/Dockerfile': No such file or directory
./.templates/nodered/build.sh: line 54: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 58: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 59: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 60: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 61: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 63: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 69: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 69: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 69: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 69: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 69: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 69: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 69: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 69: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 72: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 73: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 74: ./services/nodered/Dockerfile: No such file or directory
./.templates/nodered/build.sh: line 76: ./services/nodered/Dockerfile: No such file or directory
Finished NodeRed Build script
Issue running build:
[Errno 2] No such file or directory: './services/docker-compose.save.yml'

In the build menu I chose:

  • Node-Red
  • InfluxDB
  • Grafana
  • Eclipse Mosquitto
  • Postgres
  • PiHole
  • Gitea

Observations (I realise it's early days so this is just my two cents which, given that we're talking Australian Dollars is, to quote Terry Pratchett, less than a Hershebian Half Dong):

  1. I could not find Portainer in the list. Deliberate omission?
  2. I think the list should be kept in alphabetical order.
  3. I don't think end-around wrapping of the list is useful. Much better to bang into the top or bang into the bottom. You know where you are.

Between 2 & 3 I found that I had to make a separate list and check things off as I found them just to make sure I hadn't missed anything. I kept thinking "it shouldn't be this hard". 2¢, remember!!

Hitting RETURN took me to the list of nodes for Node-Red. I selected the ones I use but I seem to recall reading somewhere that that bit isn't working yet so maybe all the ./services/nodered/Dockerfile messages were to be expected.

On completion of the node selection, nothing seemed to happen. I eventually pressed a key (down-arrow, I think) and the main menu repainted. I selected Exit. I got a docker-compose.yaml for which a quick once-over returned "seems reasonable", plus that error about a .save.yml.

Trying to bring the stack up got:

pi@sec-dev:~/IOTstack $ docker-compose up -d
ERROR: Couldn't find env file: /home/pi/IOTstack/services/grafana/grafana.env

Contents of directory after all this:

$ ls -a
.  ..  .bash_aliases  docker-compose.yml  docs  duck  .git  .github  .gitignore  LICENSE  menu.sh  mkdocs.yml  .native  README.md  scripts  .templates  .tmp

so definitely no volumes or services.

Repeating a menu run and re-selecting the same containers and node-red nodes produced the same result.

@Slyke
Copy link
Collaborator

Slyke commented Jun 22, 2020

Not sure why you can't see portainer:
image
image

The only reason it wouldn't show up in the build menu is if the build.py file errored, or doesn't exist.

I fixed the NodeRed and Grafana issues! Please pull latest and give it a go when you get a chance. There's a good chance that there's a bunch of services that will error when building them. I've only tested a few of them to ensure the new menu can work.

@Paraphraser
Copy link
Author

Setup, with existing IOTstack folder (cloned from my fork with SensorsIot as upstream):

$ git remote remove upstream
$ git remote set-url origin https://github.com/SensorsIot/IOTstack.git
$ git pull origin master

In theory, that should be the same as a clone of SensorsIot

$ git checkout new-menu
$ ./menu.sh

Screen Shot 2020-06-22 at 16 33 16

Screen Shot 2020-06-22 at 16 33 40

Screen Shot 2020-06-22 at 16 33 58

Some things to note:

  1. Portainer appears in this list. Why not yesterday? Mystery. Perhaps it had something to do with yesterday being a fresh clone of SensorsIot vs today being git commands.
  2. Your post above shows things in alphabetical order. I am not seeing that.

I selected (in no particular order):

  • influxdb
  • portainer
  • postgres
  • grafana
  • mosquitto
  • gitea
  • pihole
  • nodered

For Node-Red, the defaults plus SQLite and BLE.

The result was a YAML file and no complaints. 🤪

Screen Shot 2020-06-22 at 16 35 25

First time I've seen that screen!!!

Then:

$ docker-compose pull
$ docker-compose up --build -d

That seemed to come up without trouble but, although the pull seemed to do things:

Pulling portainer ... done
Pulling nodered   ... done
Pulling influxdb  ... done
Pulling grafana   ... done
Pulling mosquitto ... done
Pulling postgres  ... done
Pulling pihole    ... done
Pulling gitea     ... done

it didn't build Node-Red so I assumed anything with a Dockerfile still needs special treatment.

The treatment I use is:

$ docker-compose down
$ docker rmi "iotstack_nodered" "nodered/node-red"

but that moaned like crazy about the image being in use with a stopped container. I got rid of the one it was complaining about but it just moved onto grizzling about another stopped container. In the end, I just:

$ docker container ls -aq
f4a237d7f01f
55ad937f541a
091ea88a9707
9907d3d0daab
d1ec529f87c1
4f576482613a
6397d385d1c8
$ docker container rm «each of the above»

Then I could finally get rid of the nodered base. I fired up:

$ docker-compose up --build -d

and it started building Node-Red with the usual piles of warnings for SQLite and friends.

Once that finished, a docker ps suggested everything was up and nothing was restarting. I could get to Node-Red, PiHole, Portainer and Grafana and, by inference to InfluxDB. I didn't try Mosquitto.

I was able to add a BLE node to Node-Red and it scanned successfully for nearby signals so I call that a successful test too.

How much of this is explained by:

  • Your changes in the last 24 hours; and/or
  • The git operations to take my fork out of the picture so that ~/IOTstack looked the same as a clone of SensorsIot; and/or
  • Not actually being a fresh clone so things like "services" and "volumes" were in place already.

is something I'll leave to you to decide. 👍👍👍

@Paraphraser
Copy link
Author

Paraphraser commented Jun 22, 2020

The UTF-8 thing is something that makes very little sense. The file:

~/IOTstack/docs/Mosquitto.md

has some U+2102 characters ("⇒") near the top.

  • ssh from the macOS terminal (correct rendering):

Screen Shot 2020-06-22 at 17 39 50

  • VNC then open the Raspbian terminal (incorrect rendering):

Screen Shot 2020-06-22 at 17 40 01

  • VNC, launch File Manager, find the file and hit the spacebar (correct rendering):

Screen Shot 2020-06-22 at 17 41 21

I use the first approach (ssh) for 99.9% of everything I do. That's all I use for muckin' around with IOTstack. This suggests everything in the ssh path understands a bit more than ASCII so I'm at a loss to explain what we were seeing before.

Oh, I should add that I did the same test on my "live" RPi and it was the same. I had not run sudo dpkg-reconfigure locales on that box so my earlier comment is probably a red herring.

@Slyke
Copy link
Collaborator

Slyke commented Jun 22, 2020

Awesome! Glad it's finally, (almost) working correctly haha. Yeah, not sure why portainer wasn't showing yesterday. I guess it doesn't really matter any more since it's showing, but if you wanted to, you could check out the specific commit that it wasn't showing with, and see if it now shows, that would at least tell us if it was some weird merge issue on your end, or if it was the actual stack.

It's also possible that some code that I changed did cause stuff to start working.

If the other menus are working from the main menu then I'll add back in the characters it was complaining about on the build menu, I'll let you know on Discord when I've done that.

For now, I will begin testing all the services and setting up the build scripts with them, as lots of stuff has changed on many of the services since I originally started this, and despite merging master and trying to keep it up to date, it looks like some issues have slipped in! My first order of business will be NodeRed.

@Paraphraser
Copy link
Author

Paraphraser commented Jun 22, 2020 via email

@Slyke
Copy link
Collaborator

Slyke commented Jun 22, 2020

I think that's just how the operating system is reporting the list of files back. It is easy to fix though so that it always reports the list back in alphabetical order. I'll change it to that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants