Sparkle 2: Support custom appcast formats

[ezekg]

Summary

Follow up to these old issues, #1080, #905, #427, #155. It seems to have been requested for years. I figured a new thread would be nice, especially since Sparkle 2 is under development. I'd really like to see support for a JSON appcast. XML is old and constantly has new security issues, and it's really tough for new APIs to support it due to this. Adding JSON support would really modernize Sparkle, especially from the perspective of a software distribution service, which is nowadays typically some sort of JSON-based API backed by a storage service such as AWS S3 using a non-public storage bucket i.e. signed URLs per-asset (for example, this is how services such as GitHub Releases operate, for public and private repos).

Possible Fix

Allow an update server to respond with JSON instead of XML. Ideally, we could allow the server to respond with any payload, by offering a way to delegate transforming an arbitrary server response into an SUAppcast. This would allow Sparkle to support nearly any distribution server, including more modern setups like AWS S3 using signed URLs, or GitHub releases, since the developer could transform response values into a valid update. But if that's not feasible, we can define a required or default schema that a server must follow for JSON response formats.

With that said, I see 2 really good choices: (feel free to chime in with thoughts)

1. Add support for defining a delegate to parse the update server response and transform it into a valid SUAppcast. This delegate could make additional requests if needed, or simply parse the custom server response, possibly JSON possibly not, into an SUAppcast with valid SUAppcastItem items. E.g. add SUAppcast:initWithDelegate.
2. Add SUAppcast:initWithJSONData much like SUAppcast:initWithXMLData.

Both would be ideal, since 2 could be an implementation of 1, i.e. an out-of-the-box standardized JSON delegate. Having 2 would open up the door for third-party delegate packages, to support the various distribution services that all have different response formats and access requirements. This would put the onus on third-parties to build and maintain delegates, instead of the Sparkle team, who would only maintain the XML and possibly the official JSON delegates.

Version

Sparkle 2

---

Disclaimer: I'm a small business owner, and I currently run a software distribution API service. I'd really like to natively support Sparkle for my macOS customers. But I would like to avoid implementing a custom server endpoint that responds with XML solely to support Sparkle, so I'm opening this issue to start the conversation around Sparkle 2 supporting a wider variety of update server response formats, by allowing the community to build out appcast delegate packages to support services such as GitHub Releases, S3, R2, as well as my own service.

Please forgive me if I don't fully understand Sparkle’s internals. I'm new to the code base, and to Swift/Obj-C, but have been reviewing it over the last months to try and come up with a feasible way forward here.

I am willing to put a bounty on this of $1,000/mo, to fund development, given the Sparkle maintainers approve of the final direction that should be taken.

[jbrayton]

I like the idea of an app being able to supply a delegate that retrieves its own SUAppcast, as an alternative to Sparkle retrieving an appcast based on a feed URL. That delegate might retrieve update information from a JSON feed or from an API. If the Sparkle team would accept a pull request with this functionality, I can work on it. Thoughts?

[zorgiepoo]

I'm somewhat neutral about JSON support.. I see the simplistic/popular appeal but at the same time there are drawbacks

If we support signed appcasts which I may have future interest in, how do we properly support that with a JSON format which lacks comments? I think we should have an answer to this.

I guess we would have to give up CDATA / embedded release notes too because nobody will want to write those in JSON. But maybe that is okay.

I don't think we should expose a delegate-able customizable spec API, at least not to start out with. Most in support of JSON would want just a standard json format that is well documented on our website, automatable to write, and something the ecosystem as a whole wants to support.. Sparkle also supports other apps being able to update Sparkle-based bundles too (sparkle-cli is one such example, but there are others, as well as website providers like MacUpdate or brew-cask..). Maybe it is even essential to have a tool that can convert between formats, or maybe not..

I would initially say this is out of scope for 2.0.0 which I'm trying to get in a shippable state, but hey, if in the framework it's as simple as adding a single parse method to SUAppcast.m that maps all of our current features I may reconsider.. Other necessary work involved is adding unit tests (maybe just for the mapping of properties), generate_appcast support, website documentation updates. A contributor would have to push for this, current maintainers aren't going to invest in work here.

[ezekg]

Signed appcasts could be handled using standard HTTP headers, the same way APIs including my own offer cryptographic signatures of JSON response payloads. No need to do anything fancy on that front with non-standard JSON comments.

Another way could be to have an appcast top level object containing the appcast data, and then a signature property for the signature of the signed appcast object.

{
  “signature”: “...”,
  “appcast”: {
      ..
  }
}

But offering a delegate system could allow for a variety of different ways to solve this, depending on how the backing storage system operates. For example, my API would probably use HTTP headers. An S3 delegate might utilize an object’s metadata for signature storage. Again, the onus for all that would fall on the delegate implementation, not the Sparkle team so it provides flexibility without burdening the maintainers. Like a plugin economy. 🙂

[zorgiepoo]

Sorry I don't really want to support creating custom mappings of properties or expose appcast item creation to clients. This also fragments other support / Sparkle-based tooling. This is a bigger ask than JSON support which you can see hasn't been done yet. I am more open to supporting JSON as a secondary standard format, despite its shortcomings if they can be worked around..

I don't think signed appcasts can be complicated enough to require web server sending custom metadata back. Including the signature in the appcast is possible, possibly, if the signed contents strips out the signature field and Sparkle strips out contents on verification.. This is definitely less pleasant and possibly less secure than an XML comment to me. Ok, I can think of ways around this though, let's move that aside..

Thinking about it again, embedded release notes support is also important (eg look at the one the Sparkle Test App uses), even if it's not as pretty in JSON.. Apps do use this today and it's also somewhat related with signed appcasts.

[ezekg]

That is pretty disappointing to hear. I thought what I came up with was a good solution to support a wider variety of update servers, but I do respect your decision. I appreciate the work that you've done with Sparkle. 👍

Can you shine some light on exactly why you think it'd fragment the community? As far as I could tell, the XML appcast isn't used in too many places and it's not really a "necessary" part of Sparkle's main functionality — it's just a means to securely get upgrade data into Sparkle, right?

Also — including an embedded HTML string inside of JSON for release notes is a non-issue. Any good JSON serializer will automatically escape the Markdown/HTML/etc., including any with newlines, so I don't see the issue here. It’ll be unescaped when deserialized. JSON isn't meant to be pretty. JSON is meant to be consumed by a machine, not written by hand or read by a human.

[zorgiepoo]

The appcast format is a big part of Sparkle -- maybe not code-wise, but lots of other infrastructure & services (eg brew-task, macupdate, other implementations of Sparkle for other OS's, more) rely on it. Other software that updates user's software for them rely on it. Sparkle 2 even includes an API and a tool to update other Sparkle based bundles. People even subscribe to software update feeds with RSS readers. I debug other people's apps when investigating Sparkle issues. Asking for a way to customize the appcast format is going to fragment support for all these other use cases. Further allowing custom mappings causes fragmentation by allowing these different formats or backends to not implement specific features, despite the framework otherwise supporting it, which users may need but don't know it just yet and realize too late.

I don't want to also maintain API support for that customization or maintain compatibility with other mappings, and I don't want users to be tied to a "middle-man". I have recently even been making the details of creating appcast items / appcasts private and not exposed to minimize past misuse and future compatibility breaking changes.

generate_appcast could probably indeed handle serializing out inline HTML in JSON.

I bet a big part of people wanting JSON support over XML actually is that it's easier to read, write, modify, and validate by eye. So I disagree about the "pretty" statement. All the test appcasts I use are crafted or altered later or copied/pasted by hand and no doubt other developers do this too for their own software. People like to make claims in their favor like nobody should be modifying appcasts by hand, or comments are not used by people, or who uses (or writes by hand) embedded release notes -- but the fact is these things are untrue and they are very important to consider in making a good assessment of benefits/shortcomings here.

I realize we also have somewhat different goals.. I care more about the app developer experience of integrating Sparkle, not so much about backend providers adopting Sparkle... I don't really believe supporting a standard JSON or XML format is a significant issue for them.. and needing to provide an additional API to users to understand a custom format has costs from my point of view.

[ezekg]

Okay, thanks for your time. 👍 And just as a note — as an API provider, I do think XML is a significant issue.