SR3 AMQP Data Pump Assistance For Mirroring Directory

[MichaelDavisBaskaran]

I have currently set up an AMQP server that is able to get data from dd.weather.gc.ca using SR3. However when trying to mirror the directory of alerts from the website to a mounted directory it does not work. Is there any issue with mirroring currently or is there a different way to approach mirroring to a mounted directory versus locally? Any clarification pertaining to this would be greatly appreciated thanks!

[petersilva]

Mirroring should be just a subscriber that publishes to your AMQP broker after it downloads. I guess you can do a subscriber first... see if that works, then at posting after download to your config ... perhaps share the configuration you are using... this kind of thing should just work.

[MichaelDavisBaskaran]

Thank you for the quick response, appreciate it! Here, I will share with you the configuration I was able to come up with to achieve this after going through the documentation. I can confirm that the directory path used is writable and has received files through other configurations.

[petersilva]

subtopic *.WXO-DD.alerts.#

Your topic is not going to work try this one. but alerts are not issued very often.
If you want to see it working immediately, might be fun to use:

subtopic *.WXO-DD.bulletins.#

Then you will get files being copied immediately. You can have both subtopics active
and get bulletins immediately, and alerts will come in also.

[MichaelDavisBaskaran]

Awesome, thank you so much! I will test out both sets of subtopics within two different configuration files. However, when you get a chance may you please explain to me how you came up with these particular subtopics. I was under the assumption of just ensuring the web url path matches with my subtopic path. The reason I ask is because I eventually want to set up a mirrored configuration that pulls in other sets of data other than alerts as well.

[petersilva]

uh... this answer is too long... but I guess it has the virtue of being thorough... The topics... do exactly match the (real) directory tree:

https://dd.weather.gc.ca/20250127/WXO-DD/alerts/

Is the directory where the alerts are written. The first directory (* in the subtopic) is the date. The second sub-topic is WXO-DD is called a source in sarracenia parlance, it is a descriptive name meant to represent where a data set comes from. On internal servers there are dozens of different sources for data, but they get coalesced into one for the public. Other sources may show up in due course.

dd.weather's structure now matches exactly how all of our other deployments of sarracenia data pumps are done. dd was the original deployment and used to have a pre-standard tree. This change was announced here:

https://comm.collab.science.gc.ca/mailman3/hyperkitty/list/dd_info@comm.collab.science.gc.ca/thread/SQX45GNAB5MKY4Z5FELOWP4FIFKVB5LV/

prior to standardization:

• different datasets had different lifetimes, managed by scripts that would walk large sub-trees.
  • The load on the system placed by these cleaning scripts would get larger, the more data we added.
  • Users, when they fell behind, would end up asking questions because data they wanted had been removed before they had downloaded it.
  • when new data was added:
    • often new cleaning scripts were needed (we were into dozens of them.) and site failures can happen when large datasets aren't cleaned up.
    • Also since there was no pattern... there was no way for users to know that old data was being accidentially kept, so when the problem was fixed (and less data is kept) it could also cause complaints.

with the standard tree, we remove the tree from a given date after a set number of days has passed.

• orders of magnitude less load on the servers.
• clarity to users... people can tell if the entire date tree is missing, they usually guess that's why they can't get the data. Also... the data is kept longer than before because it's easier to manage.
• we now keep ALL the data for a given day. For example, for model data, we used to keep it for 3 to 12 hours... now we keep it for weeks.

So that's the official new tree. We did however understand that people were used to the old tree (in place since 2004.) so there is a forest of links created to make the new organization look familiar. For example... alerts/ is linked to todays' date/ WXO-DD/alerts this link gets replaced at midnight every day. so that makes it look like alerts/ is the same as it always was... but it doesn't contain data from previous days anymore.

I hope that clears things up. Does it make more sense? or did I just overwhelm you with a wall of text... sorry if I did that...

[MichaelDavisBaskaran]

That definitely makes much more sense now so I really do appreciate this insight. I will not lie it was quite overwhelming as I am pretty new to exploring with SR3. This raises the question, how would you go about setting subtopics when the webpage you are looking to draw data from is not linked to this official directory. Essentially what I am asking is if it possible to use sr3 to get data from websites other than dd.weather.gc.ca that are not linked on the official page https://hpfx.collab.science.gc.ca such as https://cwfis.cfs.nrcan.gc.ca/downloads/ which contains information pertaining to fire data.

[petersilva]

Ideally sites use sr3, and then the notifications are available so you can download them and save the load, on client and server of polling them to determine what new info is available. Ideally, you might contact them and ask them if they are interested in adding sr3 notifications to their server, and that would make it easier both you and us.

We already poll that server internally to create notifications for hotspot data that we download. None of that configuration is currently public. We have been pondering making more configurations and plugins public for a while. I'll try to scrape up what we use so you can try it out and put it here. it might take a few days.

[MichaelDavisBaskaran]

Okay, if you are able to put some suggestions into what you use that would be extremely helpful. On a different note, is it possible to send an email notification once a specific alert is received to notify a server. For example is it possible to send an email when an alert within the CWTO folder is received. (since we know alerts don't typically always come on a set schedule, I am looking for a method to be able to know when an alert is received and then trigger an email.)

[petersilva]

At the top of the thread you said you have an AMQP server. Let's call it mybroker ... so in the scripts that subscribe to alerts, you add lines like:

# all the subscriber stuff you already have is above, at the end:

# post to the local broker...
post_broker amqp://myuser@mybroker
post_exchange xs_myuser_alerts

Then you set up a subscriber on the local broker:

broker amqp://myuser@mybroker
exchange xs_myuser
topicPrefix v02.post
subtopic #

sendTo   smtp://your_local_relay.example.com
email_from my_email@example.com

callback send.email

accept .* send_to_me@example.com and_me@example.com and_me_too@example.com

so that's an outline... you can try it with bulletins so it's way easier to test (products show up constantly.)
but then change the configs for the data you really want.

[petersilva]

First problem: It's winter... there isn't much forest fire data available ;-) so this is hard to test.
but fwiw:

The following would be an example of a poll routine that would produce messages that the rest of the sarracenia components can consume so that it works just like hpfx or a datamart.

~/.config/sr3/poll/hotspots

# where to post the notifications 
post_broker  amqp://feeder@localhost  
post_exchange xs_feeder_hotspots

# options
sleep   2m
timeout 60s

callback poll.hotspots_html_page

permCopy  off
pollUrl   http://cwfis.cfs.nrcan.gc.ca/

# where to get the products
path    /downloads/hotspots
accept  .*/2.*[0-9].csv$

reject  .*2017.*
reject  .*a.csv.*

path    /downloads/hotspots/cffeps
accept  .*/in.*[0-9].csv$

acceptUnmatched False

You might need an entry in ~/.config/sr3/credentials.conf like so:

http://cwfis.cfs.nrcan.gc.ca

that would be if the poll spits out some missing or bad credential errors.

Note, this has a callback routine... callback poll.hotspots_html_page in it to reformat the html listing from the site..

You would need to put the plugin code in ~/.config/sr3/plugins/poll/hotspots_html_page.py
with the source below:

#!/usr/bin/env python3
"""
Description:
    This is an html page plugin used for NRCAN data.

Ported to sr3 by ANL 2022/12/19

Usage:
    callback poll.hotspots_html_page
"""

from sarracenia.flowcb.poll import Poll
import html.parser, logging, paramiko
import time

logger = logging.getLogger(__name__)


class Hotspots_html_page(Poll):

    def __init__(self, options):
        super().__init__(options, logger)
        self.last = None
        self.myfname = None
        self.fdate = None

    def handle_starttag(self, tag, attrs):
        for attr in attrs:
            c,n = attr
            if self.last == 'file' and c == 'href':
               self.myfname = n
               return
            if n != '' :
               self.last = n

    def handle_data(self, data):
        data = data.strip()
        data = data.strip('\t')
        if data == '': return
        if self.last == 'date':
            self.fdate = data
            return

        if self.last != 'size': return
        if self.myfname == None: return

        mysize = data.lower()
        mysize = mysize.replace('\t', '')
        mysize = mysize.replace(' ', '')
        mysize = mysize.replace('b', '')

        # date example  2019-06-12 06:06:04
        mydate = self.fdate
        try:
            t = time.strptime(mydate, '%d-%b-%Y %H:%M:%S')
        except:
            t = time.strptime(mydate, '%Y-%m-%d %H:%M:%S')
        mydate = time.strftime('%b %d %H:%M', t)

        self.entries[self.myfname] = '-rwxr-xr-x 1 101 10 ' + mysize + ' ' + mydate + ' ' + self.myfname

        logger.debug("(%s) = %s" % (self.myfname, self.entries[self.myfname]))

        self.last = None
        self.myfname = None

so with that you will be able to create messages on your local broker, and subscribe to them so that the download happens.

[MichaelDavisBaskaran]

Thanks for being so helpful Peter, I am going to take all this shared knowledge and try to get it up and working! I will reach out if I run into anything I am not able to work around.

[MichaelDavisBaskaran]

Wait so is it possible to directly integrate a send email script within the configuration file that is already subscribed to alerts using callback rather than connecting with a local amqp broker and triggering it from there to minimize work load?

[petersilva]

one component can either download or send... not both. You need two components, and they talk to each other via messages. At least that is how it currently works.... improvements welcome, but that's a biggie. ;-)

[MichaelDavisBaskaran]

Hi @petersilva I have noticed that when I run the following configuration file that my directory is mirroring as expected however the actual file itself does not seem to show. So, for example right now I have created an instance of Observer that is watching my mounted directory and it alerts me when a file is received such as T_WWCN12_C_CWNT_202501311750_3167450066.cap. But when I go into my directory it is not actually present. Any suggestions on how to resolve this?

[MichaelDavisBaskaran]

Also, I was wondering if there is any newsletter or anything of that sort to be subscribed and notified of any updates or changes?

[petersilva]

mirror off

with mirror on ... it will reproduce the entire directory tree from the source onto the destination. with it off, it places the file in the specified direcory

[MichaelDavisBaskaran]

Sorry, just got around to this, would this mean that I would need my current configuration file with mirror on running in parallel to a bunch of other configuration files with all the other possible file path locations such as /WeatherDashboard/MSCDatamartClone2/alerts/20250206/WXO-DD/alerts/cap/20250206/CWVR/16.

I tried to include both mirror off and on in the same configuration folder and nothing was created, which is why I am wondering if this is the only other way to go about it?

[petersilva]

Let us set aside uploading to S3 for now... You just need one subscriber to download ... so you would have a file called something like ~/.config/sr3/subscribe/alert_download.conf

broker amqps://dd.weather.gc.ca/
topicPrefix v02.post

subtopic *.WXO-DD.alerts.cap.#

mirror on

directory /WeatherDashboard/MSCDatamartClone2/alerts
accept .*

The sr3 package includes a bunch of example configurations you can see with the sr3 list ie command,
then you can add an example to play with. so I pick the datamart alert subscriber example:

sr3 add subscribe/ddc_cap-xml.conf
sr3 edit subscribe/ddc_cap-xml   # change the directory to match what you want.
sr3 declare
mkdir ~/MSCDatamartClone2/
sr3 start subscribe/ddc_cap-xml

In the sr3 edit command, I just changed the directory setting (for me it is ${HOME}/MSCDatamartClone2/alerts )
What path do you want to store alerts in?:

• /WeatherDashboard/MSCDatamartClone2/alerts/20250205/WXO-DD/alertsa/cap/20250206/CWVR/16 (mirror: on)
• /WeatherDashboard/MSCDatamartClone2/alerts (mirror: off)

with mirror on, it got an alert, and ended up here:

home/peter/MSCDatamartClone2/alerts/20250208/WXO-DD/alerts/cap/20250208/CWNT/05:
total 32
drwxrwxr-x 2 peter peter  4096 Feb  8 00:36 .
drwxrwxr-x 3 peter peter  4096 Feb  8 00:36 ..
-rw-rw-r-- 1 peter peter 18046 Feb  8 00:36 T_WWCN12_C_CWNT_202502080533_3712498709.cap

The mirror created the entire tree directories in place on the source tree... if you just want to put the
Common Alerting Protocol (CAP) files in the alerts directory, you set mirror off.

so anyways... This is to download to your local debian vm. Please confirm this works for you and
that you understand the file placement. Then we will move onto posting. Thanks.

[MichaelDavisBaskaran]

total 48
drwxrwxr-x. 2 yr10 sg_dvmlp 0 Feb 10 17:38 .
drwxrwxr-x. 2 yr10 sg_dvmlp 49152 Feb 10 13:23 ..
drwxrwxr-x. 2 yr10 sg_dvmlp 0 Feb 10 13:23 20250210

[petersilva]

is the sarra process running under the sg_dvmlp user?

[petersilva]

sorry... the sarra process you need to be running under the yr10 user... the other thing is the group, I guess.

[MichaelDavisBaskaran]

Oh I see, I will have to test that out but hopefully that should resolve this matter. My apologies just recently the permissions were under my user account but I will try from the listed account.

If it works I will be sure to update this thread. Once again thank you so much for all the insight and guidance in resolving this.

[MichaelDavisBaskaran]

Hi @petersilva, it works as expected! You've been really helpful thanks!

[petersilva]

so now you are downloading ok... Are you ok to publish to a local broker, and uplink to S3?

[petersilva]

never mind. I was confused with another thread. I gather the problem is solved.

[MichaelDavisBaskaran]

Yes, I am actually working on implementing the email functionality in another way.