Parse Sparkle appcasts to detect need for Cask updates

[phinze]

Many mac apps use the Sparkle update framework:

http://sparkle.andymatuschak.org/

It's got a standardized "publish an update" format:

https://github.com/andymatuschak/Sparkle/wiki/publishing-an-update

We could leverage this to detect when an app might need an update.

That would be pretty cool yeah? 😎

[jfsiii]

I came here to post this very issue :)

How do you envision this being implemented? I assume it'd be a Ruby Module but haven't done any research to see if any already exist.

I'll wait to hear your thoughts before starting in with a list of questions.

[jfsiii]

I was just talking with a co-worker and he pointed out that products which use Sparkle will have something like:

<key>SUFeedURL</key>
<string>http://gmask.clockwise.ee/check_update/</string>

in their Contents\Info.plist file.

This means we could programmaticly determine if an app was using Sparkle, then handle the update gracefully.

This also goes along with my "can we add logic/processing" question in #308 (comment)

[tjluoma]

FWIW I have been researching this on my own for the past several months, and there are LOT of apps which do NOT make it easy to find their Sparkle feeds. (No, I don't know why either.)

In fact I would say that the majority of the apps that I have looked at do not put their feed info in the Info.plist file.

I finally installed an HTTP "sniffer" to discover some of them, only to discover that some of them seem to fetch this information by HTTPS (grrrr).

Also, while there seems to be some amount of similarity between Sparkle feeds, there is enough difference between them that I have been parsing each one of them rather than trying to come up with "one parser to rule them all" so to speak. This can be as simple as some apps ONLY keep the latest version listed in their Sparkle feed, while others seem to keep ALL of them. Some put the latest information on top, others put it on the bottom. And so on.

Last but not least: while many of them seem to use "sparkle:version" and/or "sparkle:shortVersionString" to indicate the latest version, some of them seem to have come up with their own version naming scheme. (Again, I don't know why, maybe some of them pre-date this "standard"?)

Since I don't know ruby (hangs head in geek shame) I have just been parsing them using curl and awk, sed, tr, grep.

This is not (at all) intended to suggest that the developers not try to use Sparkle feeds, just to highlight the fact that it might not be as easy as it initially seems.

[rolandwalker]

I neglected to cross-reference this original issue, but we have been working on appcasts since #3105.

We aren't ready to close this — not nearly — because the appcast stanza doesn't yet do something useful. But there is progress.

[alebcay]

Indeed, parsing the appcasts is probably the most difficult part of this process. Just from looking at the Appcast XMLs of two Casks, I realize they don't have some field with a "short" version number that is easily accessible (e.g. 1.2.3), but rather, there were only <title> fields nested inside <item> fields with titles varying from "Version 1.2.3" to something like "ExampleApp v1.2.3 - build 1234". There does not seem to be a very good one size fits all method to parsing appcasts. Perhaps we should look to how the Sparkle framework itself reads appcast XMLs to determine how to handle them.

[tjluoma]

FWIW:

I have been working on this problem for awhile now for a personal
project, and have discovered that some use CFBundleShortVersionString
and some use CFBundleIdentifier for the version numbers. Some use extra
text in the version fields, as you discovered.

Many appcast feeds only show the most recent version, but some of them
seem to show (almost) every version.

Most put new entries on the top, but some put them at the bottom.

Some apps make it almost impossible to even find their appcast feed,
and getting any results from it depends on sending app information in a
very specific format. Determining some of that can only be done using
something like Charles Proxy and watching outgoing requests.

Most aggravating are those whose apps obviously have some process for
checking for new versions, but they have 'rolled their own' and make it
difficult to figure out how to use it.

I am not suggesting that Cask should not do this -- it absolutely
should. I am just saying that there will not be a "one size fits all"
way to do this for all apps.

For my project I ended up writing per-feed parsing scripts (mostly just
one-liners), and then make a note whether it is giving me a
CFBundleShortVersionString or CFBundleIdentifier.

On 20 Jun 2014, at 22:55, Caleb Xu wrote:

Indeed, parsing the appcasts is probably the most difficult part of
this process. Just from looking at the Appcast XMLs of two Casks, I
realize they don't have some field with a "short" version number that
is easily accessible (e.g. 1.2.3), but rather, there were only
<title> fields nested inside <item> fields with titles varying
from "Version 1.2.3" to something like "ExampleApp v1.2.3 - build
1234". There does not seem to be a very good one size fits all method
to parsing appcasts. Perhaps we should look to how the Sparkle
framework itself reads appcast XMLs to determine how to handle them.

---

Reply to this email directly or view it on GitHub:
#260 (comment)

[rolandwalker]

Thanks for the data points. In #4847 adds a :format key to the appcast stanza which we can hopefully use to define some of these variations in the future.

More importantly, a :sha256 key is being added. The first implementation plan is to simply checksum the appcast content, and detect that updates are available when the content changes.

[timsutton]

Also FWIW:

I also wrote a Sparkle feed parser for a Python-based project for sysadmins, AutoPkg, which collects community-built 'recipes' for obtaining and deploying software updates to managed Mac clients. I stumbled across this because we also have a trivial parser for pulling the URL out of a cask formula.

Here's our 'processor' that handles returning a URL for the latest download from an appcast feed.

https://github.com/autopkg/autopkg/blob/master/Code/autopkglib/SparkleUpdateInfoProvider.py

It's been a while since I looked at it - it hasn't needed any revision since I wrote it over a year ago - but essentially, Sparkle first looks for version and might use a shortVersionString for the user-facing version. These probably usually map to CFBundleVersion and CFBundleShortVersionString.

They also fall back to parsing the URL for a version but without any real validation of a version string:
https://github.com/sparkle-project/Sparkle/blob/def8cfa1ff0f52abd07d8ffe22df96e1e38e785f/Sparkle/SUAppcastItem.m#L70-L77

I think there have been a couple cases where a vendor's appcast feed server only returns something when passed a specific User-Agent header. Although these might have been for applications which are paid and for which the vendor doesn't provide a public download.

[rolandwalker]

Thanks @timsutton!

For pulling the URL out of a Cask, try brew cask _stanza url. Theoretically, commands with leading underscore are internal/undocumented, but in practice I'm sure that one won't change.

[timsutton]

Thanks, that could be useful. For our project, however, we don't have any requirement of Homebrew and cask, and instead just read and parse a formula's .rb file via GitHub raw urls.

[rolandwalker]

That's interesting, because we were just talking with @sdegutis about whether Cask files should be more static for the purpose of interop with other projects (on IRC and in #5458).

Meanwhile (in the other direction) @vitorgalvao has been changing many Casks since #4910 so that the url value is calculated dynamically from the version number (example: vagrant-manager.rb). While that means less typing is required to accomplish an update, it also means that your parser may stop working.

I guess I have gone offtopic for the current issue. If you have any problems with interop, do file an issue or catch us on IRC. Being scriptable and hackable is one of our goals, as described in https://github.com/caskroom/homebrew-cask/blob/master/doc/HACKING.md#what-is-the-design-philosophy .

[thesoftwarephilosopher]

(:+1: for static configuration files)

[kornelski]

So, regarding the feeds:

• Order of elements in the feed is not important. Sparkle checks all of them and picks the latest version.
• sparkle:version is used to find latest version. sparkle:shortVersionString is only for UI.
• There's a standard format for versions: https://github.com/sparkle-project/Sparkle/blob/master/Tests/SUVersionComparisonTest.m
• Feeds containing only the latest version should be expected, it's RSS after all. You will have to poll for changes (fortunately these feeds don't change too often).
• SUFeedURL in the Info.plist is the recommended option for declaring appcasts. There is a programmatic way to set it, but I hoped apps would use the API only to set special beta/nightly appcasts :/

[rolandwalker]

Many thanks, @pornel!

As mentioned elsewhere, we have identified appcasts for almost one third of all app Casks by extracting SUFeedURL from Info.plist. Appcast info is now stored in each Cask, but we have not yet started making use of it.

[timsutton]

@rolandwalker
Thanks for the heads-up about the change. At least in this case, it would be easy for me to just string-sub in #{version} myself.

Our project always had the goal of static recipe files, and instead provide different "Processors" that are able to parse feeds and deal with URLs but then figure out if the download has changed compared to the previous one, compare with versions in a repo, etc. Users who want to write a recipe to get the latest app by a vendor that doesn't use Sparkle but instead wrote their own custom auto-updater mechanism (of which there are many!), they end up reverse-engineering the feed system and implementing the necessary logic in their own processor and ship that with the recipe files.

The benefit is that while users keep their community repos up to date to keep track with any changes by the vendor, the recipes themselves don't need to change from minor version to minor version of a given app. Of course, the benefit of brew-cask work is that you distribute the work among a ton of contributors, and the casks are very easy for anyone to contribute to.

[jm3]

Hi, is this still being worked on? Anything I can help test or document?

[vitorgalvao]

@jm3 Many casks have the proper stanza, but the feature is completely non-working yet. We don’t even have the proper idea of how we should be about doing it.

[adidalal]

Closing based on discussion in #15908 (:formathas been removed for the time being)