Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compress data #1443

Merged
merged 1 commit into from
Dec 29, 2022
Merged

Compress data #1443

merged 1 commit into from
Dec 29, 2022

Conversation

tzachi-dar
Copy link
Contributor

Using nightscout on google cloud there is a limit on the BW that can be used for free.
The maximum free amount that can be used is 1G per month.
It seems that two rigs use that BW in a day, not in a month.

This checkin decreases the amount of data that is used significantly by doing two things:

  1. Compressing in a few places where curl was not used to compress.
  2. Compress the js usage of the code.
  3. Decrease the number of device status entries that are brought for pushover from 100 to 20. It seems to me that 20 entries should cover one hour of data, but I'm not 100% sure how things work.

Without this changes each pushover takes 250k and there is one of this calls every minute. After compressing, we get to ~10k of data. This is still too much, so I have changed this to be 20 entries only.
If that does not make sense, please let me know.

@sulkaharo
Copy link
Collaborator

Note the REST API supports if-modified-since headers, use of which will hugely reduce bandwidth consumption. This is being used in loading the glucose values but could be added to other data as well?

@tzachi-dar
Copy link
Contributor Author

Thanks for all the responders.

Here is some more info about the issue:
After moving to google cloud and running there for 3 weeks without the free tier, I have realized that data consumption is going to be more than 30G in a month. Since the free tier is limited to 1G of traffic, it seems that I would have to pay around 4$ a month for this BW.

That said, I have started looking for where the extra BW is coming from. I have used the nginx logs for that. It was very clear that the highest usage of data was coming from the device status command asking for a status of 100 and also non compressed. I have also changed all other places that are used in my use case to be compressed.
This has reduced the BW (as nginx) reports it to ~0.3G a month.

To be honest, I'm not using pushover, and I have removed this calls from my devices.
Still, Google is charging me for about 3G per month (which is about 10 times better that what I have started with but 10 times worth than what nginx is counting). I have used tcpdump to get a sense of what is going on, and it seems that google calculation are right. There is a very high overhead to opening an SSL connection.
See attached snipset for a typical https connection:
image

So, it seems that getting to a smaller number requires using less connections and not further compression.

All that said (if my PR will be expected as is), it seems that the expected price is going to be ~30 cents per month. So ROI is going to be very hard.
Looking at the pushover calls, it seems that data is filtered on the last 10 minutes, and that can be done on the server.
I'll change my PR to do that.
If there is a simple way to know if pushover is used, please let me know, and I'll also add that.
Going further probably means that we need to keep one connection alive and do multiple requests on it, but I have a filling this will add complexity to the project.
Another option might be to create a tunnel from the rig to the cloud and use html inside that rig. I'm not sure if it actually works or would save BW.

So, to sumarize:
If there is a way to know if pushover is used, please let me know.
I'll change the pushover call to ask for 10 minutes of data instead of 20 objects.
Any other ideas that will not make the project more complicated are welcomed. (probably calling less than one minute is the simplest but this might also have influence on total rig performance, so I don't like the idea)

@scottleibrand
Copy link
Contributor

scottleibrand commented Dec 22, 2022

Looks like oref0-pushover never gets to the point of doing the NS calls if you don't have pusover_token and pushover_user set in your preferences.json:

if [ -z $TOKEN ] || [ -z $USER ]; then
print_usage
exit 1
fi

So if you don't use Pushover, you can remove the pushover optimizations and then we can merge this with the --compressed flags etc.

@tzachi-dar
Copy link
Contributor Author

Thanks @scottleibrand I see your point about not calling if pusover_token is not set.
I have removed this part from the PR.

@scottleibrand scottleibrand merged commit 1a25d53 into openaps:dev Dec 29, 2022
@scottleibrand scottleibrand mentioned this pull request Jan 17, 2023
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants