Apprise to Support Persistent Storage

[caronc]

💡 The Idea
Apprise needs to support persistent storage.

• It requires a nice library like the Attachments, and Configuration that the Plugin library can readily/easily access.

• The persistent storage should be a switch that defaults to being off, but can be toggled to on from the AppriseAsset() object (storage_mode="memory" by default)

• The Persistent storage would be adapted by the NotifyBase object making it accessible in every plugins environment without the need of an extra import.

• The CLI tool has Persistent Storage enabled by default.

• The CLI tool provides an extra --storage-mode which can be set to auto (default), flush, and memory.

  • auto: This is the default option and pesistent storage is used when applicable (only the plugins that require it take advantage of local cache made available to them).
  • flush: Similar to auto except that any changes made are immediately flushed to disk. This mode creates a higher i/o but enforces the content on disk is the latest.
  • memory: Effectively turns off Persistent storage. No plugins are allowed to write to disk. This is exactly the way Apprise was prior ro the Persistent Storage feature.

• To remove all accumulated persistent storage generated through the CLI tool, you can run the following:

  apprise storage clean

• You can compliment this call by providing URL IDs and/or --tag (or -g) values to focus on only cleaning specific persistently cached data. For example:

  # Assuming we want to target the URL ID of abc123xy
  apprise storage clean abc123xy

  You can also clear cache based on tag references:

  # Assuming we want to target the URL(s) associated with the tag 'family'
  apprise storage clean --tag family

• The current version of Apprise would be written to disk in the persisent storage space (if enabled by the AppriseAsset() object). Subsequent runs of Apprise would then check this version and compare it to the current one running. If they differ, all persistent data would be reset/cleared and the version file updated.

  • Note: This will allow newer versions of Apprise to introduce new schemas and saving mechanisms and not be broken due to past/earlier releases.

Today the use cases for Persistent Storage are:

• Twitter IDs: Today a clean application entrance of Apprise to a Twitter endpoint requires the system to lookup the @user and get it's ID (An extra GET Request). From there it hangs onto it in memory for future notifications. With Persistent storage, this can be written to disk and save future hits even when the program is restarted from scratch. This would be a noticeable performance increase
• Mastodon: Only for DM's to ourselves (visibility=direct) requires an additional hit to the Mastodon API to retrieve our information. Caching this information would be 1 less URL hit later on.
• Opsgenie: Tracking the last message sent would allow subsequent messages to auto-close the previous;
  • Ref: Send close alert to Opsgenie #748
• Matrix: This plugin is in need of a re-write, but if it could write to disk, the amount of URL hits saved would be quite significant.
• MQTT: Retain previous message - Ref MQTT Retain Flag #799
• WhatsApp: Potential support (still not sure how to deal with QR-Code issue) can be added Ref WhatsApp Support #808
• Matrix: Retain session - Ref Matrix: Logging-in with an access_token #895

#640 would tie back to this enhancement as well.

Plugin Logic

class MyNotification(NotifyBase):
 # ...
 def send(self, body, *args, **kwargs):
       # Value is returned from memory if stored there
       # if not found, and persistent storage is enabled, then the value is read from disk and stored into memory
       # The value is then returned
       # `None` is returned if content does not exist in either location
       value = self.store.get('key')

       # Store key/value pair in memory (and Disk if persistent flag is set)
       self.store.set('key', 'value')

       #
       # Some extra options you can use with the set()
       #

       # Invalidate the data (forcing get() to return None after time is reached)
       self.store.set('key', 'value', expires=datetime(now) + timedelta(hours=1))

       # By default persistent will always be set to True, but perhaps you don't want the content to
       # persist and only exist for the life of the application instead.
       self.store.set('key', 'value', persistent=False)
       
       # Flushes elements to disk if they are configured to persist there; this 
       # never needs to be called directly as the PersistentStore object looks
       # after this automatically.  Those set to `auto` (default) mode can
       # leverage this to force an early write/sync to disk
       self.store.flush()

       # The following would both flush the key from memory and persistent
       # storage if present.
       self.store.clear('key')

       # Similar to flush, this works as well to provide a focused group clear
       self.store.clear('key1', 'key2', 'key3', ...)

       # Remove all entries in memory and all entries in the persistent store
       self.store.clear()

       # Allow Apprise to persist files outside of key/value pairs as well.  Such as maybe
       # an initial online registration that occurs and it needs to store a private/public
       # key.
       with self.store.open("key", "wb") as fp:
           fp.write()

       with self.store.open("key", "rb") as fp:
           content = fp.read()

       # To remove all persistent files you could use:
       self.store.clear()

       # For consistency with the rest of the above logic:, this would clear a file
       self.store.delete('key')

       # multiple keys can be cleared if required
       self.store.delete('key1', 'key2', 'key3')

Persistent Store Details
I was thinking of the following; but am open for discussion:

• Use an sqlite database, or perhaps just convert content to JSON and write straight to disk (+ zip) might be easier.

• sha1() of Apprise URL would id the directory/namespace it all content would write to. e.g: 549f6c03f0fb0095e02a0e68deb1b49e149f29d6

  • key/value pairs using set(), get(), and clear() would write to a file called db stored in the namespace;
    - e.g: %{namespace}/db would be something like: 549f6c03f0fb0095e02a0e68deb1b49e149f29d6/db
  • This would allow multiple variations of a URL to write to their own persistent store.
  • data written with self.persistent.open() would be written to the same %{namespace} but would reside in a sub directory called persist.
    - e.g: %{namespace}/persist/ would be something like: 549f6c03f0fb0095e02a0e68deb1b49e149f29d6/persist/

• Namespaces would all reside in the following path (additionally prefixed with a %{schema} such as discord or twitter, etc to make it easier to further troubleshoot or allow users to clear specific persisten directories..

  • ~/.apprise/persist/%{schema}/%{namespace}/ (Linux)
  • ~/.config/apprise/persist/%{schema}/%{namespace}/ (Linux)
  • %APPDATA%/Apprise/persist/%{schema}/%{namespace}/ (Windows)
  • %LOCALAPPDATA%/Apprise/persist/%{schema}/%{namespace}/ (Windows)

• the sha1() generated namespace can be overridden by the Notify() object itself through a defined namespace(self) function. Users can further look at the variables provided (ignoring extra kwargs that should not change the variance of the persist directory) and return a more specific/focused (and re-usable) location. Otherwise the parent NotifyBase() will return the general sha1(self.url()) one.

🔨 Breaking Feature
The functions/integration should be able to be implemented in such a way there would be no breaking change.

This may introduce plugins that depend on persistent storage to work and would have to be reported disabled via the Apprise.details() when the flag is off.

[caronc]

Issue resolved