Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install On QNAP NAS with Graphana Port Change #288

Closed
aesculus opened this issue Jun 2, 2023 · 51 comments
Closed

Install On QNAP NAS with Graphana Port Change #288

aesculus opened this issue Jun 2, 2023 · 51 comments
Labels
troubleshooting Help setup or fix an installation

Comments

@aesculus
Copy link

aesculus commented Jun 2, 2023

I am making some progress on getting the dashboard running on a QNAP NAS. There are a lot of nuances to this but I am making progress. If/when I ever get it working I will document my actions in case someone else wants to do the same.

Right now the install went well but I am at the last bit to configure Graphana, but I cannot get to the admin screen. I forgot to change the port from 9000 in the graphana.env file until after the install.

If I list the containers in my Docker environment we see this:

8e0a76f9dd17   telegraf:1.26.1                  "/entrypoint.sh tele…"   About an hour ago   Up 37 minutes   8092/udp, 8125/udp, 8094/tcp       telegraf
56288caf65a0   grafana/grafana:9.1.2-ubuntu     "/run.sh"                About an hour ago   Up 1 second     3000/tcp, 0.0.0.0:9000->9000/tcp   grafana
6b90aa2e6acc   jasonacox/weather411:0.2.1       "python3 server.py"      About an hour ago   Up 37 minutes   0.0.0.0:8676->8676/tcp             weather411
ac6aeaa3a7fb   influxdb:1.8                     "/entrypoint.sh infl…"   About an hour ago   Up 37 minutes   0.0.0.0:8086->8086/tcp             influxdb
da46dd4a2a6b   jasonacox/pypowerwall:0.6.2t26   "python3 server.py"      About an hour ago   Up 37 minutes   0.0.0.0:8675->8675/tcp             pypowerwall

Any ideas how I can get graphana to change the port the UI is on?

@aesculus aesculus added the troubleshooting Help setup or fix an installation label Jun 2, 2023
@jasonacox
Copy link
Owner

Hi @aesculus - thanks for posting your troubleshooting with QNAP NAS.

Why do you need to change the port for Grafana? In any case, the envrionment variables set the configuration for the Grafana container and you would change the port with this, as seen in the grafana.env file.

GF_SERVER_HTTP_PORT=9000

@aesculus
Copy link
Author

aesculus commented Jun 3, 2023

Conflict with another program.

I changed the file and rebooted but for some reason it did not take.

@jasonacox
Copy link
Owner

jasonacox commented Jun 3, 2023

You need to rebuild the container if you change the environmentals for it to pick it up.

docker stop grafana
docker rm grafana
./compose-dash.sh up -d # or whatever you need to do to rebuild your containers

@BuongiornoTexas
Copy link
Contributor

The short up time for grafana suggests a different problem - grafana isn't starting at all.

Up 1 second

I suspect taking a look at your logs will show a permission or other grafana config problem.

@aesculus
Copy link
Author

aesculus commented Jun 3, 2023

Still looking into this. You are correct. All the other services have been up for 17 hours but the Graphana instance keeps cycling on/off every few seconds.

Any thoughts on what I should look for? Its running under an User ID with RW for the app folder and even has admin privileges until I get all this working OK.

The QNAP has a container environment that is "unique". If I install Graphana inside the container program but outside of the PW-DB app it works and loads fine.

The only error I see within the app environment in Graphana is this:

Error: ✗ Get "https://grafana.com/api/plugins/repo/grafana-piechart-panel": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

temp

@jasonacox
Copy link
Owner

Error: ✗ Get "https://grafana.com/api/plugins/repo/grafana-piechart-panel": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

That happens to be the first plugin that Grafana tries to download on startup. During startup, the Grafana container needs to access the public internet to download the panel plugins that the project uses. Do you need to tell ContainerStation to allow DNS lookups and access to the internet?

@aesculus
Copy link
Author

aesculus commented Jun 3, 2023

From the image posted above you can see the services (including Graphana) that seem to be sending and receiving data, so I think they have access.

Something is causing the Graphana service to abort and restart constantly, but only within the PW-DB app instance. Weird.

@jasonacox
Copy link
Owner

The environmental configs for grafana for the powerwall-dashboard project require it to download additional plugins. The error you posted from grafana indicates it is not able to access the internet. When that happens, grafana will stop running and the container will restart. That is why it keep restarting.

The "powerwall-dashboard" APP in your tool above likely defines how DNS will work for the app. If it is similar to docker-compose, it is doing local DNS to allow the containers to find each other with short names. I wonder if it also has a configuration you can change to allow it to use public DNS servers (e.g. 8.8.8.8) and if it also has security controls to define what is allowed to access the public internet (including NAT settings).

I'm not familiar with ContainerStation, unfortunately. But as we have seen, NAS setups usually require some additional tuning. Has anyone else been able to get the project to work in a ContainerStation?

@aesculus
Copy link
Author

aesculus commented Jun 3, 2023

Nobody else has done this for QNAP. I will take the information and see if I can find others who have run into a similar problem or post a question on the QNAP forum.

Stay tuned.

@BuongiornoTexas
Copy link
Contributor

It may also be related to the problem I get with docker rootless, which also shows this grafana cycling problem on an initial install - it is also plugin related for me, but the issue is not internet access, but rather the plugins can't get written to the host system.

Can you get more detail on the error from the docker logs? docker logs -f grafana

For me, the problem is the default install assumes that docker will run as root, which effectively allows it to ignore directory permissions when writing files from the container to the host system. Rootless docker doesn't allow this, and requires that I set up the correct subuid permissions on the host so that grafana can save the plugins. E.g. #284, detail on fix in #22 (comment)

@jasonacox
Copy link
Owner

Good point @BuongiornoTexas . I was tuning in to Client.Timeout exceeded while awaiting headers which seems more connectivity related but it could just be what manifest during a failure to write as well. It would be good to see if there are other log entries.

@aesculus
Copy link
Author

aesculus commented Jun 3, 2023

Egads. I had huge post that did not get set. Let me rebuild it.

@aesculus
Copy link
Author

aesculus commented Jun 3, 2023

The QNAP NAS has a private web interface to composer which is too limited to be any good. You have the create the YAML on the web page and it cannot refer to other files etc. It also displays the docker applications but you cannot control them if you created them in a terminal. Anyway ...

As an experiement I created a sample all (postgrsql) from the QNAP app to see how it behaved. There is a web based admin app that goes with it that can show both a service and a gui that depends on the service (sql server).

Here is the YAML:
`version: '3'

services:
postgres:
image: postgres:11.6
restart: on-failure
ports:
- 127.0.0.1:5432:5432
volumes:
- /share/Container/postgresdb:/var/lib/postgresql/data
environment:
- POSTGRES_USER=qnap_postgres
- POSTGRES_PASSWORD=qnap_postgres

phppgadmin:
image: edhongcy/phppgadmin:latest
restart: on-failure
ports:
- 7070:80
- 7443:443
depends_on:
- postgres
environment:
- PHP_PG_ADMIN_SERVER_DESC=PostgreSQL
- PHP_PG_ADMIN_SERVER_HOST=postgres
- PHP_PG_ADMIN_SERVER_PORT=5432
- PHP_PG_ADMIN_SERVER_SSL_MODE=allow
- PHP_PG_ADMIN_SERVER_DEFAULT_DB=template1
- PHP_PG_ADMIN_SERVER_PG_DUMP_PATH=/usr/bin/pg_dump
- PHP_PG_ADMIN_SERVER_PG_DUMPALL_PATH=/usr/bin/pg_dumpall
- PHP_PG_ADMIN_DEFAULT_LANG=auto
- PHP_PG_ADMIN_AUTO_COMPLETE=default on
- PHP_PG_ADMIN_EXTRA_LOGIN_SECURITY=false
- PHP_PG_ADMIN_OWNED_ONLY=false
- PHP_PG_ADMIN_SHOW_COMMENTS=true
- PHP_PG_ADMIN_SHOW_ADVANCED=false
- PHP_PG_ADMIN_SHOW_SYSTEM=false
- PHP_PG_ADMIN_MIN_PASSWORD_LENGTH=1
- PHP_PG_ADMIN_LEFT_WIDTH=200
- PHP_PG_ADMIN_THEME=default
- PHP_PG_ADMIN_SHOW_OIDS=false
- PHP_PG_ADMIN_MAX_ROWS=30
- PHP_PG_ADMIN_MAX_CHARS=50
- PHP_PG_ADMIN_USE_XHTML_STRICT=false`

This results in this in docker for the container list:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4b29b94a8929 edhongcy/phppgadmin:latest "/sbin/entrypoint ap…" About an hour ago Up About an hour 0.0.0.0:7070->80/tcp, 0.0.0.0:7443->443/tcp postgresql_phppgadmin_1 60f10be6c1a3 postgres:11.6 "docker-entrypoint.s…" About an hour ago Up About an hour 127.0.0.1:5432->5432/tcp postgresql_postgres_1

Note how the port mapping and server IP addresses show up. And I can confirm that I can indeed get to the admin app with nas_local_ip_address:7070
so this demonstrates that it does work.

So I imagine the QNAP version of composer may be acting differently or expect a different way of assigning ports.

Now this is not a good test for internet access since I am just going local, but I could always port forward the graphana service if needed.

Is this an apples to oranges thing or does this help at all?

@jasonacox
Copy link
Owner

I ran a Google search for qnap grafana container can't download plugin and ran across an issue here grafana/grafana#13920 that talks about a QNAP user needing to add this environmental variable:

GODEBUG=netdns=cgo

There is also a way to manually download the plugins and mount them as a volume but I'm not sure how this work for QNAP: grafana/grafana#13920 (comment)

@aesculus
Copy link
Author

aesculus commented Jun 4, 2023

Progress:

I decided to eliminate the grafana plugin since that seemed to be the issue. I just commented it out in the grafana.env file and then restarted. It took a few minutes for grafana to fully load but when it did the QNAP reported grafana had the 9000 port URL and selecting launched the Welcome to Grafana web page.

I could make all the changes on the grafana admin but could NOT get the sun and moon so I will need to do a bit more research on this.

From here I will try to load the plugin manually from the link above.

So not done yet, but progress none the less.

EDIT: Got Sun and Moon loaded but not sure how I did it.

So here is what the dashboard looks like. Next I guess is to see if I can get the plugin installed manually. Probably tomorrow.

temp

@aesculus
Copy link
Author

aesculus commented Jun 4, 2023

Got the other plugin installed manually. Still some features that don't work/issues:

  • When I attempt to add other plugins/data sources the Grafana admin program limits my search because it says it cannot connect to the Grafana server on the internet
  • You can see that the piechart panel does not show. I checked the piechart plugin and it was loaded
  • Not sure what the big white empty space is

temp

@jasonacox
Copy link
Owner

Nice job getting as far as you have!

The blank panel is the powerflow animation - it is an iFrame in an text panel (rendering html). I don't know why it isn't rendering but there is an environmental setting that is required to make it work:

GF_PANELS_DISABLE_SANITIZE_HTML=true

It should look like this:

image

@aesculus
Copy link
Author

aesculus commented Jun 5, 2023

It of course is set in the installation and I found it in the config file too. I don't think this is the problem and its probably related to our communications issue that prevented me from installing the plugins automatically too.

When I moved the mouse over the white space I got this:

image

I am really going to have to figure out how to resolve the communications problem for Grafana as that seems to be the root of all evil here.

EDIT: When I install Grafana directly using the QNAP container it does not suffer the coms issue. So now to figure out what the difference is.

@aesculus
Copy link
Author

aesculus commented Jun 5, 2023

More progress: I was able to install grafana as an app in the QNAP container and it had internet access. When you install apps in the QNAP GUI you cannot make references to other files (well I don't know how) so I had to embed items from the grafana.env and compose.env into my YAML. I was uncertain about some of the volume information so I commented some of it out and am now going to see if I can trace back some of this stuff. Note I had to change the name to grafana1 since I already had a grafana instance in the dashboard app created via putty.

If you could provide some insight into the volume variables maybe I can sort it out.

`version: '3'

services:
grafana:
image: grafana/grafana:9.1.2-ubuntu
container_name: grafana1
hostname: grafana
restart: always
user: "1005:1005"
volumes:
- /share/Container/grafana1:/var/lib/grafana1
# - type: bind
# source: ./grafana
# target: /var/lib/grafana
ports:
- 9002:9002
environment:
- GF_PATHS_PROVISIONING=/var/lib/grafana1/provisions
- GF_SERVER_HTTP_PORT=9002
- GF_ANALYTICS_CHECK_FOR_UPDATES=false
- GF_ANALYTICS_REPORTING_ENABLED=false
- GF_PANELS_DISABLE_SANITIZE_HTML=true
- GF_SECURITY_ALLOW_EMBEDDING=true`

@jasonacox
Copy link
Owner

When you write to a local file inside a Docker container, it is important to understand that the changes made to that file are stored within the container's writable layer. By default, Docker uses a copy-on-write mechanism, which means that any modifications made to the container's file system are saved in a separate layer known as the container layer.

When you restart a Docker container, the container layer is discarded, and the container starts fresh from the base image or the previously saved state. This means that any changes made to the container's file system, including the local files you wrote to, will not persist across container restarts.

In the docker compose.env file, we specify a mounted host directory. This is done to retain the changes made to a local file inside a Docker container across container restarts, you have a few options:

Mount a host directory as a volume: By mounting a host directory as a volume inside the container, you can store the file on the host machine's file system rather than the container's file system. This way, the file will persist even if the container is restarted. We achieve this by using the -v or --volume flag with the docker run command or by specifying the volume in the compose.env.

How did you set up the InfluxDB container? I worried that if you commented out the volume mounts that all of the data it is currently collecting is getting stored in the container layer that will be deleted when you restart that container.

I suspect the QNAP tool, which is built on the NAS, has a way for you to mount a NAS path to the container. I'm afraid I don't know how to do that.

# source: ./grafana
# target: /var/lib/grafana

The "source" is the file system that persists. The "target" is where that volume shows up inside the container. You don't want to change that one because the container expects the persistent data to go there.

@aesculus
Copy link
Author

aesculus commented Jun 6, 2023

Great info. This will help me decipher what is going on under the covers.

I think the QNAP YAML was just missing the headers (source and target). The : separating the two folders. I will investigate.

My first goal was to install the grafana so I could get the internet access, which it has. Now I will work on the folder structure and then add the other components to the YAML too.

There may still be hope for some sort of script as they provide a terminal interface after the initial app creation but that is TBD.

Still not understanding why doing this from a script prevented the grafana from having internet access but ...

@jasonacox
Copy link
Owner

Awesome, thanks for logging all your progress here. Hopefully it will help anyone else with a QNAP who follows your footsteps. :)

@aesculus
Copy link
Author

aesculus commented Jun 7, 2023

Still at it trying to determine why in QNAP using it's compose UI the Grafana can see the internet but installing by terminal I cannot.

I verified that these two statements seem the same with the following differences. In the first Bind is implied. It also creates the folder if not present. If you use the field based definitions in QNAP it will error if the folder is not present first.

`version: '3'

services:
grafana:
image: grafana/grafana:9.1.2-ubuntu
container_name: grafana2
hostname: grafana
restart: always
user: "1005:1005"
volumes:
#- /share/Container/grafana2:/var/lib/grafana2
- type: bind
source: /share/Container/grafana2
target: /var/lib/grafana2
ports:
- 9003:9003
environment:
- GF_PATHS_PROVISIONING=/var/lib/grafana2/provisions
- GF_SERVER_HTTP_PORT=9003
- GF_ANALYTICS_CHECK_FOR_UPDATES=false
- GF_ANALYTICS_REPORTING_ENABLED=false
- GF_PANELS_DISABLE_SANITIZE_HTML=true
- GF_SECURITY_ALLOW_EMBEDDING=true`

BTW I am not sure what the GF_PATHS_PROVISIONING as there is no folder like that in my system and it throws errors during the compose because it cannot find any provisioning data.

Anyway except for the obvious IDs, folders, ports and other instance differences, the only things that I could find between the QNAP version that worked and the terminal version that doesn't are these (using docker container inspect):

"Dns": [], "DnsOptions": [], "DnsSearch": [],

These are from the non internet working version (installed with setup.sh)

"Dns": null, "DnsOptions": null, "DnsSearch": null,

and these are from the working one (installed with QNAP GUI).

Not sure how an empty set [] is different than a null, but perhaps it's a nuance. Nor do I know where I can change the terminal one so it's also a null to test to see if that actually makes a difference.

EDIT: I used Putty and added a folder in the QNAP application folder (the one QNAP uses when you create an app via its GUI). Basically I used the yml file from the last test and just changed the folder names and port. Using

docker-compose -f docker-compose.yml up -d

I installed this app and then after a few minutes I opened grafana on that port and checked that it had internet access. So my next test is to do this from a shareable folder not under QNAP docker control and see what happens. BTW the inspect details were exactly the same as what would have been created if using the GUI. So now it appears that not doing it in the QNAP docker area or via the setup.sh is the issue.

Stay tuned.

EDIT 2:

Well I did the same simple Grafana test on a shared volume right next to the Powerwall-Dashboard folder with the same ID I used to install the PWDB and it has internet access.

The only thing different is that I cannot manage it from the QNAP container GUI because it's not in that containers application folder.

I guess my next step is to wipe all the containers and start fresh but not wipe the share I created to place the PWDB in.

@jasonacox
Copy link
Owner

jasonacox commented Jun 8, 2023

That's fantastic @aesculus ! Great progress.

"Dns": [], "DnsOptions": [], "DnsSearch": [],
"Dns": null, "DnsOptions": null, "DnsSearch": null,

For those interested in why this matters, I actually asked ChatGPT and it gave me the answer:

In Docker, the "Dns" field in the container inspect output refers to the DNS servers used by the container for name resolution. The difference between an empty set [] and null in this context is significant.

When the "Dns" field is set to null, it means that the container is using the default DNS configuration of the Docker daemon running on the host. This typically includes the DNS servers configured on the host machine.

On the other hand, when the "Dns" field is set to an empty set [], it means that the container has explicitly configured an empty list of DNS servers. This can cause issues with name resolution, as the container won't have any DNS servers to query for domain name resolution.

To resolve the issue of the container created using Docker Compose not being able to talk to the internet, you can explicitly specify the DNS configuration for the container within your Docker Compose file.

To see the full response from our friendly neighborhood AI: https://chat.openai.com/share/43ed295b-993b-4c4f-9782-a746b9140e8c

@aesculus
Copy link
Author

aesculus commented Jun 8, 2023

Wow. That is scary and also very informative.

So I may need to add these DNS null entries into my YAML file. I might try a clean install again without it and see what it does. Not sure why using setup.sh caused this originally.

BTW is there any reason that the folder name has to be Powerwall-Dashboard? I am thinking of storing the dashboard in QNAPs app container folder that only supports a-z and 0-9 characters. so like 'powerwalldashboard'. This way I would have more visibility within QNAP of the resource.

@aesculus
Copy link
Author

aesculus commented Jun 9, 2023

Will I reset everything and started from scratch using the QNAP container application folder for the PW DB. Everything seemed to work according to plan including the access to all Grafana content but I have still run into some issues on the dashboard I will need to work through.
image

@jasonacox
Copy link
Owner

BTW is there any reason that the folder name has to be Powerwall-Dashboard?

Yes, this project uses git to manage updates. If you look at upgrade.sh it basically does a git pull to update all the files. The folder name comes from the GitHub location and name of the project (Powerwall-Dashboard). You could manually change that (rename or move it).

The image you post looks like the one where Grafana is no longer able to access the internet again. I don't really understand why the docker compose on the QNAP is adding the [] empty array to the dns setting. The powerwall.yml doesn't list dns so it should be null. Is there a way to upgrade docker on the QNAP? I don't know if that would help but I can't explain why it works different on the QNAP than any other OS.

Worse case, you could go back to launching the grafana container separately in the directory where it works. I have seen that other NAS users have to manually launch containers to get around limitation of the shell their NAS provides.

@aesculus
Copy link
Author

aesculus commented Jun 9, 2023

Well it's not the DNS issue as it is set at null. So on to something else.

Can you do a docker container inspect on your grafana container and attach it here so I can do a compare?

@jasonacox
Copy link
Owner

Yes, of course!

pi@piserver:~ $ docker inspect grafana

[
    {
        "Id": "438abac8a5c1709d5dcc2a8bddd15771d1af7d2c2806da136a9218b04d36578a",
        "Created": "2023-05-05T04:49:18.300659456Z",
        "Path": "/run.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 25031,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-05-05T04:49:21.368398321Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:f348250d5001df4d0af64e73eb2eb75027ccf922c5d81e84396498183a540148",
        "ResolvConfPath": "/var/lib/docker/containers/438abac8a5c1709d5dcc2a8bddd15771d1af7d2c2806da136a9218b04d36578a/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/438abac8a5c1709d5dcc2a8bddd15771d1af7d2c2806da136a9218b04d36578a/hostname",
        "HostsPath": "/var/lib/docker/containers/438abac8a5c1709d5dcc2a8bddd15771d1af7d2c2806da136a9218b04d36578a/hosts",
        "LogPath": "/var/lib/docker/containers/438abac8a5c1709d5dcc2a8bddd15771d1af7d2c2806da136a9218b04d36578a/438abac8a5c1709d5dcc2a8bddd15771d1af7d2c2806da136a9218b04d36578a-json.log",
        "Name": "/grafana",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "powerwall-dashboard_default",
            "PortBindings": {
                "9000/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "9000"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/home/pi/Powerwall-Dashboard/grafana",
                    "Target": "/var/lib/grafana"
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/59195b0392de360b40184938b1ff38ebf414b8b9dba676c972334440249bbbf0-init/diff:/var/lib/docker/overlay2/701f6bc4f2a5cdb29021c529632bc17235f4dcde45ab38606ac375d4648c7819/diff:/var/lib/docker/overlay2/9a09fb5b28b35409bad145084d195faf313852cf6c15a18e530142d0bc7696c8/diff:/var/lib/docker/overlay2/b25123e6f5c86c1dc841599f74cf21506eb0751a1086676cf16d33d92c2c1658/diff:/var/lib/docker/overlay2/38ce7fc0cbc57b72e587e2a75c360584b99d91e531cafaba08c8371e31431e15/diff:/var/lib/docker/overlay2/f5adacfcce105255013f9e672f282b6eafa24fa730c1e3e7c97a8d32ac16d5f3/diff:/var/lib/docker/overlay2/9bed38f6d554816125f1d8e20338c9bc070f3f1f294761aa510729d94311b1a5/diff:/var/lib/docker/overlay2/837d5ce01403b713a8e590073286434b28bd6c17c13c906103b433676d2217ca/diff",
                "MergedDir": "/var/lib/docker/overlay2/59195b0392de360b40184938b1ff38ebf414b8b9dba676c972334440249bbbf0/merged",
                "UpperDir": "/var/lib/docker/overlay2/59195b0392de360b40184938b1ff38ebf414b8b9dba676c972334440249bbbf0/diff",
                "WorkDir": "/var/lib/docker/overlay2/59195b0392de360b40184938b1ff38ebf414b8b9dba676c972334440249bbbf0/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/pi/Powerwall-Dashboard/grafana",
                "Destination": "/var/lib/grafana",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "grafana",
            "Domainname": "",
            "User": "1000:1000",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3000/tcp": {},
                "9000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "GF_PATHS_PROVISIONING=/var/lib/grafana/provisions",
                "GF_SERVER_HTTP_PORT=9000",
                "GF_INSTALL_PLUGINS=grafana-piechart-panel, https://github.com/yesoreyeram/yesoreyeram-boomtable-panel/releases/download/v1.5.0-alpha.3/yesoreyeram-boomtable-panel-1.5.0-alpha.3.zip;yesoreyeram-boomtable-panel, grafana-influxdb-flux-datasource, fetzerch-sunandmoon-datasource, simpod-json-datasource",
                "GF_ANALYTICS_CHECK_FOR_UPDATES=false",
                "GF_ANALYTICS_REPORTING_ENABLED=false",
                "GF_PANELS_DISABLE_SANITIZE_HTML=true",
                "GF_SECURITY_ALLOW_EMBEDDING=true",
                "PATH=/usr/share/grafana/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GF_PATHS_CONFIG=/etc/grafana/grafana.ini",
                "GF_PATHS_DATA=/var/lib/grafana",
                "GF_PATHS_HOME=/usr/share/grafana",
                "GF_PATHS_LOGS=/var/log/grafana",
                "GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
            ],
            "Cmd": null,
            "Image": "grafana/grafana:9.1.2-ubuntu",
            "Volumes": null,
            "WorkingDir": "/usr/share/grafana",
            "Entrypoint": [
                "/run.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "a30d5674e0f3ad0667e9657f62118af0f95e83a933414b4f96f7906bf6abf1ce",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "powerwall-dashboard",
                "com.docker.compose.project.config_files": "powerwall.yml",
                "com.docker.compose.project.working_dir": "/home/pi/Powerwall-Dashboard",
                "com.docker.compose.service": "grafana",
                "com.docker.compose.version": "1.29.2"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "36c0b16509ca177c6b342b19666f4e4c53431aeb621d051eef7f013459f4d389",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "3000/tcp": null,
                "9000/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "9000"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "9000"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/36c0b16509ca",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "powerwall-dashboard_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "grafana",
                        "438abac8a5c1"
                    ],
                    "NetworkID": "344f7d555c341325120452837c3cf17e3a1081473188a18d5037a881736cff31",
                    "EndpointID": "aa568648d98350a9972c26d7ba1245b0e5435139fd0703653b3f5556cdee69e9",
                    "Gateway": "172.21.0.1",
                    "IPAddress": "172.21.0.6",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:15:00:06",
                    "DriverOpts": null
                }
            }
        }
    }
]

@aesculus
Copy link
Author

aesculus commented Jun 10, 2023

OK. Here is mine attached.

Quick compare I find some things you have that I don't related to the plugins.

"GF_INSTALL_PLUGINS=grafana-piechart-panel, https://github.com/yesoreyeram/yesoreyeram-boomtable-panel/releases/download/v1.5.0-alpha.3/yesoreyeram-boomtable-panel-1.5.0-alpha.3.zip;yesoreyeram-boomtable-panel, grafana-influxdb-flux-datasource, fetzerch-sunandmoon-datasource, simpod-json-datasource"

These being absent on my part explain a lot. But I used Grafana to install them (except pie chart) so????

new install inspect.txt

EDIT: My Bad. I had commented that line out in my grafana.env file during my first test installs. Let me remove this and try to reinstally grafana and see if I pick these up. If not I will clear the whole think and reinstall again.

@jasonacox
Copy link
Owner

Ok, and doing a quick diff, there are a few other things different but I don't know if they matter:

https://www.diffchecker.com/EniDWHH4/

@aesculus
Copy link
Author

Yeah. Saw those. Mostly directory stuff we expect. A few other tidbits but not as big as commenting out all the plugins. :-)

@aesculus
Copy link
Author

aesculus commented Jun 10, 2023

OK. We are back pretty close now. Just that one plugin on the upper right. Something to figure out tomorrow.

image

@jasonacox
Copy link
Owner

Fantastic! That upper panel is pypowerwall driven animation. It would be interesting to see if you can hit this:

http://x.x.x.x:8675/example.html

(of course replace x.x.x.x with the address of the hosting running grafana).

image

If nothing shows up, then the next issue is getting the pypowerwall container to be accessible to your local network (on port 8675).

@aesculus
Copy link
Author

aesculus commented Jun 10, 2023

Not accessible. So now to figure out how to get that port working.

Using verify.sh

`Checking pypowerwall

  • Config File pypowerwall.env: GOOD
  • Container (pypowerwall): GOOD
  • Service (port 8675): GOOD
  • Version: 0.6.2 Proxy t26
  • Powerwall State: CONNECTED - Firmware Version: 23.4.2-1
  • ERROR: Your pypowerwall settings are missing TZ.
    `
    Not sure why the TZ issue. I am in PDT so did not change that setting.

@aesculus
Copy link
Author

Success:
image

Made a couple of changes. One I changed all the users to 1005:100

But the big change was probably changing the pwpowerwall port from 127.0.0.1:8675:8675 to 8675:8675. Not sure why I went that route before but saw that somewhere else that suggested it. In this case I think it was wrong. Your instruction to see if the example.html worked got me thinking it must be an outside port access so I made it look like the Grafana one.

@aesculus
Copy link
Author

I'll do a writeup on the most optimal install for QNAP once I get my negative solar issue resolved in python. Then we can drop this thread.

Although I tried to do something within the QNAP Container Station domain, I pretty much ended up doing the same approach that was offered up at the end of the Synology thread.

#22 (comment)

@jasonacox
Copy link
Owner

Awesome!! It's looks great @aesculus - great job. Thanks for the step-by-step discussion and yes, your advices and write-up for other QNAP users would be helpful. Thank you! 🙏

@container-noob
Copy link

I'll do a writeup on the most optimal install for QNAP once I get my negative solar issue resolved in python. Then we can drop this thread.

Although I tried to do something within the QNAP Container Station domain, I pretty much ended up doing the same approach that was offered up at the end of the Synology thread.

#22 (comment)

Looking forward to the writeup as I have been trying to get this to work on my QNAP as well. Appreciate the time and effort you put in.

@aesculus
Copy link
Author

aesculus commented Jun 21, 2023

@container-noob

Lets try these instructions on you and if they work then we can have @jasonacox post them on a new thread and put this one into the archives.

Installing Powerwall-Dashboard on the QNAP NAS

The QNAP NAS supports the Docker container and Docker-Compose facility through the use of a terminal app such as Putty or the terminal app in a Mac. You will need to enable SSH before you can use this to install the Powerwall-Dashboard. The QNAP NAS also has a pseudo Docker-Compose app called Container Station that unfortunately is not suited for installing or maintaining the Powerwall-Dashboard but can be used to monitor it on the NAS.

These are the instructions on how to install the application on the NAS:

  1. Log into the NAS via an administrator ID
  2. Confirm that you have installed Container Station in the AppCenter
  3. First, we will need to create a folder to contain the dashboard in the QNAPs file system. Create a Shared File in the root of your volume named ‘Powerwall-Dashboard’. Pay attention to the capitalization.
  4. Next unzip the contents of the Powerwall-Dashboard-main file from Git into this directory
  5. Next, we will need a user to define and run the dashboard, you cannot use your root userid for this purpose. Create a new user, for example ‘dockuser’ with the default permissions. Give them R/W permission to the Powerwall-Dashboard folder.
  6. We will need to let this user access the system via SSH (Putty) so go to Control-Panel > Network & File Services > Telnet / SSH and if you have restrictions ensure that the userid has Access Permission.
  7. We will also need to make sure the ‘dockuser’ has execute permission too on all the .py and .sh files in the Powerwall-Dashboard folder and subfolders.
  8. Next open a terminal session like Putty and login with the ‘dockuser’ account.
  9. Change the directory to ‘Powerwall-Dashboard’.
  10. Type id and then enter to get the ‘dockuser’ ID and gid.
  11. Edit the ‘powerwall.yml’ file and change the user: entries for ‘dockuser’ userid and gid like user: "1005:100". Save the file
  12. Now you can follow the instructions in the read me just after Setup.
  13. Once everything is running in Docker and you have your Powerwall-Dashboard displaying properly, you can go back to the QNAP Admin interface and load ContainerStation from the AppCenter menu. You will see ‘powerwall-dashboard’ as an App in the Overview panel.
  14. While you cannot control anything from this limited Docker-Compose view, you can visualize and monitor the resources that the dashboard is consuming on your NAS. You can also view the logs for each container, memory, and other run statistics. None of the action buttons like Stop or Remove will work on the externally installed app. You will have to do that sort of thing via a terminal session and docker or docker-compose commands, but that is out of scope here.

@jasonacox
Copy link
Owner

jasonacox commented Jun 21, 2023

Thanks for doing this, @aesculus ! 🙏

@container-noob
Copy link

container-noob commented Jun 22, 2023 via email

@aesculus
Copy link
Author

aesculus commented Jun 23, 2023

For some reason the images did not come through.

Note I forgot a step. In the compose.yml you need to change the user to GRAFANAUSER="1005:100" or the same as you did for the powerwall.yml file. Same in the py.yml.

Anyway did you use the sample grafana.env file? It should be OK. You don't want to do anything I tried before my instruction posts. I was wandering around trying to do something in ContainerStation at first.

Also what does the ContainerStation log look like for the dashboard app Grafana container?

BTW I think the var/lib files are within the VM and I don't know how to see those. Nothing is in the var/lib disk files for me.

@container-noob
Copy link

container-noob commented Jun 23, 2023 via email

@aesculus
Copy link
Author

aesculus commented Jul 8, 2023

I discovered that with my manual install I could not use the Upgrade script because it depended on Git. So here is the set of instructions using Git:

Installing Powerwall-Dashboard on the QNAP NAS

The QNAP NAS supports the Docker container and Docker-Compose facility through the use of a terminal app such as Putty or the terminal app in a Mac. You will need to enable SSH before you can use this to install the Powerwall-Dashboard. The QNAP NAS also has a pseudo Docker-Compose app called Container Station that unfortunately is not suited for installing or maintaining the Powerwall-Dashboard but can be used to monitor it on the NAS.

These are the instructions on how to install the application on the NAS:

  1. Log into the NAS via an administrator ID
  2. Confirm that you have installed Container Station in the AppCenter
  3. Install the Text Editor from the Utilities
  4. You will need to install Git. I was successful using QGit from the QNAP club. https://www.myqnap.org/product/qgit/
  5. We will need a user to define and run the dashboard, you cannot use your root userid for this purpose. Create a new user, for example ‘dockuser’ with the default permissions. Give them R/W permission to the Powerwall-Dashboard folder.
  6. We will need to let this user access the system via SSH (Putty) so go to Control-Panel > Network & File Services > Telnet / SSH and if you have restrictions ensure that the userid has Access Permission.
  7. Next open a terminal session like Putty and login with the ‘dockuser’ account.
  8. Type id and then enter to get the ‘dockuser’ UID and GID (ie. 1005/100).
  9. Copy these numbers down as you will need them later
  10. Change the directory to the shared data directory (ex. cd /share/CACHEDEV1_DATA).
  11. Install the Powerwall-Dashboard via Git as described in the Setup with Option 1
  12. After the install is done and you have updated Grafana we need to make some changes to the configuration.
  13. Next, we will need to make the Powerwall-Dashboard folder a shared folder in the QNAP so you can get access to it from the File Manager
  14. On the QNAP web page go to Control Panel – Shared Folders
  15. Select Create – Shared Folders
  16. Name the folder Powerwall-Dashboard
  17. On the Path option choose ‘Enter path manually’.
  18. Scroll the pull down until you find the ‘Powerwall-Dashboard’ folder and select it. Select Next.
  19. Make sure your user (ex dockuser) has full R/W access.
  20. Select Next and then Finish
  21. Using the Text Editor edit the ‘powerwall.yml’ file and change the user: entries for the ‘dockuser’ userid and gid like (user: "1005:100"). Save the file
  22. Change the user in the same way in the compose.env file too.
  23. From the terminal session change the directory to ‘Powerwall-Dashboard’ (ex cd Powerwall-Dashboard)
  24. Stop the dashboard (ex ./compose-dash.sh down).
  25. Restart the dashboard (ex. ./compose-dash.sh up -d).
  26. Once everything is running in Docker and you have your Powerwall-Dashboard displaying properly, you can go back to the QNAP Admin interface and load ContainerStation from the AppCenter menu. You will see ‘powerwall-dashboard’ as an App in the Overview panel.
  27. While you cannot control anything from this limited Docker-Compose view, you can visualize and monitor the resources that the dashboard is consuming on your NAS. You can also view the logs for each container, memory, and other run statistics. None of the action buttons like Stop or Remove will work on the externally installed app. You will have to do that sort of thing via a terminal session and docker or docker-compose commands, but that is out of scope here.

@container-noob
Copy link

container-noob commented Jul 10, 2023 via email

@container-noob
Copy link

container-noob commented Jul 10, 2023 via email

@aesculus
Copy link
Author

Perfect. I was just about to ask you if you noticed those other nuanced places.

Now if you venture further in this to add

https://github.com/BuongiornoTexas/pwdusage

Be aware that I struggled here too. I would be curious to see if you get it running right away or not. It just would not expose the usage_engine port at all until a number of days later. I did some testing I reported at the end of this thread and that seemed to clear the log jam, or it just fixed itself.

BuongiornoTexas/pwdusage#5

@container-noob
Copy link

container-noob commented Jul 10, 2023 via email

@aesculus
Copy link
Author

It is. But its free when your usage is small. You have the register and then get a code to put in the dashboard.

@aesculus
Copy link
Author

Working fine so closing this thread. Thanks for the support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
troubleshooting Help setup or fix an installation
Projects
None yet
Development

No branches or pull requests

4 participants