JSON project format #3981

tresf · 2017-11-18T01:46:12Z

Proposal to add a new JSON-backed, ZIP-encapsulated project format.

The Problem

The .mmpz format is confusing because it's a standard compression with a byte offset that makes it unreadable by common archive tools like 7zip.
- This can be even more difficult for bug reporters/testers because the recommended technique for a bug report is to provide an uncompressed project file, but this can prove difficult if LMMS can't open it, yet is required for decompressing it.
The .xml project format is very resilient but XML can be tricky for power-users to hand-edit and is less popular format for newcomers (as opposed to .yml, .json, etc)
Although the .mmpz format is compressed, no current standard exists for storing and extracting project resources (samples, patches, soundfonts, plugins, etc).
- A new project format can help blueprint portable project files, also making it easier to debug loading issues, since it's easy for the power-user to hand-edit a .zip file and folders/files within.

Possible Solutions

Change the .mmpz save code to fix the byte offset. This would cause compatibility issues, but could address the 7zip problems.
-- OR --
Create a new format based on the popular .zip format, designed using standards that our power users already know and are familiar with.

Mockup

Benefits

JSON based, easier to read
Standard ZIP format
- Provides quick access for developers to open/debug projects
- Provides a scalable method for eventually bundling portable project files (Portable Project files #1257)

Before

<?xml version="1.0"?>
<!DOCTYPE lmms-project>
<lmms-project version="1.0" creator="LMMS" creatorversion="1.1.3" type="song">
  <head timesig_numerator="4" mastervol="100" timesig_denominator="4" bpm="140" masterpitch="0"/>
  <song>
    <trackcontainer width="600" x="5" y="5" maximized="0" height="300" visible="1" type="song" minimized="0">
      <track muted="0" type="0" name="TripleOscillator" solo="0">

After (mockup)

{
  lmms-project: {
    version: 1.0,
    creator: "LMMS",
    creatorversion: "1.1.3",
    type: "song",
    head: {
      timesig_numerator: 4,
      mastervol: 100,
      timesig_denominator: 4,
      bpm: 140,
      masterpitch: 0
    },
    song: {
      trackcontainer: {
        width: 600,
        x: 5,
        y: 5,
        maximized: 0,
        height: 300,
        visible: 1,
        type: "song",
        minimized: 0,
        track: [
          {
            muted: 0,
            type: 0,
            name: "TripleOscillator",
            solo: 0,

The text was updated successfully, but these errors were encountered:

JohannesLorenz · 2019-06-08T20:26:18Z

Did you have plans how to handle external resources? Is this specific point different between XML and json?

tresf · 2019-06-10T14:22:21Z

@JohannesLorenz it would just mimic the existing folder layout (samples, etc). I'd even go as far as to say presets should be stashed as separate files (instead of embedding them) so that they can be easily shared with others.

vlad0337187 · 2019-06-13T10:14:43Z

Good idea.
Also I'd suggest to use YAML or CSON for this =)

tresf · 2019-06-13T14:52:05Z

From what I've read YAML is less common for data and more common for configuration. YAML becomes valuable if we have a lot of self-references.

Quoting j2y:

YAML vs JSON
YAML is best suited for configuration where JSON is better as a serialization format or serving up data for your APIs.

YAML is by no means a holy grail or a replacement for JSON - you should use the data format that makes the most sense for what you are trying to accomplish.

But in some cases, YAML has a couple of big advantages over JSON, including the ability to self reference, support for complex datatypes, embedded block literals, comments, and more.

Write you configuration files in YAML format where you have the opportunity - it is designed to be readable and editable by humans.

JSON, in contrast, is only designed to be human readable - intentionally lacking features to support editing. Lets start with lack of comment support - this is intentionally left out of the JSON spec because its not what the format was designed for.

A big win for YAML is that it does support comments. This is very useful especially when you use it for configuration. For data interchange, many of YAMLs features lose their appeal.

YAML parsers are younger and have been known to be less secure.

JSON vs YAML
JSON wins as a serialization format. It is more explicit and more suitable for data interchange between your apis.

JSON ships with a far simpler spec than YAML. You can learn JSON a lot faster than you can learn YAML, because it is not nearly as robust in its feature set.

YAML is a superset of JSON, which means you can parse JSON with a YAML parser.

Try mixing JSON and YAML in the same document: [..., ..] for annotating arrays and { "foo" : "bar"} for objects.

vlad0337187 · 2019-06-15T00:33:31Z

@tresf , I thought it could be human-readable =)

0xf0xx0 · 2019-10-26T22:11:17Z

Related thing: .lmmsrc.json

mnini · 2020-08-18T20:45:39Z

Yaml is best suitable, if you want to edit something or whatever, I suggest comments in projects files with ranges of instruments...

It's easier to script by keyboard, without playing audio, than tunin stuff in engine, because mouse clics are too far away to work with at certain parts of interface and entering numbers by hand trough lmms takes a lot of time...

I suggest you to not use JSON, because JSON is more horrific to edit then xml, t. javascript programmer
When you delete list there's some comma issue and stuff like that, it's really horryfying, yaml is clear without any wrongness.

Veratil · 2020-08-18T21:42:22Z

If we go with YAML, yaml.org points to libfyaml being a spec 1.2 compliant (with 1.3 features it says). It's MIT so it's compatible with GPL. It uses CMake and gives instructions of how to include it.

Cons listed:

Windows - libfyaml is not supporting windows yet.
Unicode - libfyaml only supports UTF8 and has no support for wide character input.

musikBear · 2020-08-20T13:37:59Z

beside the technical considerations between formats of files, would this be a finite break with backward compatibility?

tresf · 2020-08-20T15:36:21Z

@musikBear the general consensus is to preserve backwards compatibility at all costs for the foreseeable future for any feature; mmpz would be supported still.

0xf0xx0 · 2020-08-20T20:12:46Z

so, if we move to yaml, will there be a way to convert existing project files/presets to yaml? imo keeping support for xml longer than a few versions would be a bit of a hassle, especially if something changed in how the file is structured. Maybe LMMS could automatically convert on project save?

vlad0337187 · 2020-08-20T20:25:35Z

I think it could auto-convert on project open.
So there will be no need to save .mmpz any more.

Veratil · 2020-08-20T20:46:06Z

Maybe LMMS could automatically convert on project save?

I think it could auto-convert on project open.
So there will be no need to save .mmpz any more.

Since we're also considering a file extension change (I assume from @tresf possible solutions and screenshot example), it probably wouldn't be an auto-convert on save, but only when using save as to explicitly specify the new format. Using the same extension is dangerous since we don't really have any header identifier in the file to say that it's this or that format. My opinion is to have a new file format which will be default on save as, but if using the older mmp format then it will still fall back to using that.

This means our save pipeline needs to be modified which will need to be tested in depth.

musikBear · 2020-08-21T13:04:42Z

@Veratil wrote:

My opinion is to have a new file format which will be default on save as, but if using the older mmp format then it will still fall back to using that.
💯

Absolutely 100% spot ON

0xf0xx0 · 2020-08-27T18:33:03Z

Using the same extension is dangerous since we don't really have any header identifier in the file to say that it's this or that format.

one way to avoid that is to attempt to read as yaml, and if it fails it falls back to xml and then converts on save. Would that not be a good way, and why not?

tresf · 2020-08-27T18:40:28Z

We aren't treating a single file extension as two separate formats. It's not worth talking about.

DomClark · 2020-08-29T19:20:29Z

I suggest comments in projects files with ranges of instruments

(I assume this means annotating all values, not just literal pitch ranges, given the objection in the comment to "entering numbers by hand through lmms".)

I'm against this.

It would bloat all project files for a feature only a few people will use.
It would tie the range information to individual project files, rather than versions of LMMS. What matters is what values can be loaded, not what values could have been saved.
It would only provide information about things currently in the project files. It wouldn't help when adding new data to a project by hand.
It wouldn't be easy for a project editing tool to use this information, since it would be in a custom format within a comment, rather than something common or standard.

I do like the idea of providing meta-information about the project file format though. I think JSON schema would be suitable here: a schema can be referenced just once at the root of a project file, we can keep a schema up to date as LMMS is developed, it can describe all possible aspects of a project file, and there are libraries for most common languages to use schemas and validate files against them.

When you delete list there's some comma issue and stuff like that, it's really horryfying

I agree that forbidding trailing commas is irritating (C++ has that issue with member initialiser lists which I don't like), but I think "really horrifying" is an exaggeration and this is a rather weak reason not to use JSON. It's not like YAML doesn't have nasty gotchas either: unquoted strings can end up being interpreted as different types in rather unintuitive ways; it's insecure (YAML libraries for multiple languages have or have had arbitrary code execution problems or even features) and we don't want to propagate malware; etc. Not to mention that it's considerably more complex than JSON.

I don't see the point of overcomplicating this; it's primarily meant to be a serialisation format after all, and JSON is far from illegible.

0xf0xx0 · 2020-08-29T21:57:45Z

about the trailing commas, doesnt the JSON library for C/C++ handle that? so you wouldn't need to worry about trailing commas anyways

musikBear · 2020-08-30T14:07:03Z

comments in projects files with ranges of instruments

I have thought about a kind of 'report-feature' Its just some loose thoughts..
The short explanation would be in the line of:
*Right-click a connection/ controller/ preset & (?)
*Select option 'To-journal'
LMMS would then write a simple text-explanation of the item set-up.
That could be done in project-notes, or it could be done in a new .Jour text-document

0xf0xx0 · 2020-08-31T14:56:09Z

@musikBear sooo... the help feature?

tresf · 2020-08-31T15:01:33Z

@gingkathfox @musikBear can we please stop speculative rhetoric? JSON (or YAML) is meant to improve the experience. It's NOT meant to make a project file a human editable text document. If a UI feature is needed (such as link-specific annotations) track that in a feature. It'll make it just fine into the project file regardless of the format.

I don't see the point of overcomplicating this

👍

I think "really horrifying" is an exaggeration and this is a rather weak reason not to use JSON

💯

mnini · 2020-09-13T08:06:19Z

Project files doesn't have to break backward compatibility. @Veratil I suggest that opening filetype can go opening by extension, changing new extension to .lp (lmms project) and make project file be folder of actual project, from which you can choose on and can be saved to another projects, and projects folders like these could be renderable whole at the time into multiple files. Each of those project folders could contain samples used included in zip, so everything to re-render on another machine would come with a project file, which is nowadays issue. Even if I download something from .lmms portal I have problem opening files.

@musikBear just notes file in zip directory with project is what I would suggest, and projects could be folders, remember some people have to transfer samples as individual files and this could help that.

@gingkathfox trialing commas are standard of JSON since it changed for saving one line change of github of JSON, however JSON from C++ library is not beautified and I would keep the white space, therefore I would use YAML, maybe some extended YAML, that would allow comments that are not project file, but "range" of value in the line, because then you could just write project file to make music, which would be iconic. Project files over 5MB renders for hours on normal machine, therefore I really think there's enough space to keep white space, it's literally nothing, even on slow DSL internet, and file size doesn't have to be hyper optimalized if it would be negative in one way.

DomClark · 2020-09-13T17:17:55Z

however JSON from C++ library is not beautified and I would keep the white space

Qt certainly can (https://doc.qt.io/qt-5/qjsondocument.html#JsonFormat-enum), as can RapidJSON (https://rapidjson.org/classrapidjson_1_1_pretty_writer.html). I'm sure the same goes for most other libraries.

mnini · 2020-09-19T18:40:33Z

Sorry my bad. But how you comment inside JSON?

Veratil · 2020-09-19T18:51:32Z

{
  "comment": "This is a comment"
}

😁

tresf · 2020-09-22T15:30:45Z

The moderation is done by humans. Let's keep this on-topic and civil please.

tresf · 2020-09-22T15:36:28Z

Yes, but if you want to specify range for every variable it looks utterly [****].

In regards to project formats being a home for in-depth documentation (such as descriptions and value ranges), the project file is currently not the place for it.

I agree that polluting the document for commenting purposes is a gross implementation, but that's not the purpose of a project file. Range is part of the instruments, knobs sliders and plugins and managed as such in code. Tracking this information in the project file is not currently a requirement in the software and likewise would need a strong argument for this to keep it in the design.

0xf0xx0 · 2020-09-22T17:55:04Z

i mean, i dont get all this about comments and range, a project would be literally this:

{
    "name": "This is totally a real project!",
    "instruments: [
        "tripleoscillator": {
            "triposcsetting1": "triposcsetting1value"
        }
    ]
}

obviously this is oversimplified, but thats beside the point. the point is you wont need comments. if youre poking at the project file manually, i think we can safely assume you know where to find everything. So heres what i think LMMS should do with project files:

LMMS should check for legal values. We dont want the master fader to be +500 and completely destroy whatever is playing the project. Also, the project file shouldn't have to worry about ranges. The project files purpose is to save data. It doesn't (and shouldn't) care about what are legal values, and what the range of say, the w/d knob on a effect is.
LMMS should not worry about commenting and prettifying. If the end user wants to edit it they can easily use VSCode and a plugin to make it human-readable. But LMMS should name each key something that explains itself, like, say, TripleOscillatorMasterVolume for the Vol knob or TripleOscillatorResoAttack for the reso filter.

will edit with more as i think of them

tresf · 2020-09-22T18:33:35Z

@gingkathfox agree with everything except prettifying. I always prettify by default and as long as there's no major file size penalty, will be my recommendation. Case and point, you just did it to prove your point. 😉 Minified adds a small amount of storage space for -- what I consider -- drastic degradation in readability.

messmerd · 2023-09-29T02:50:24Z

Here's something worth looking into:
https://github.com/bitwig/dawproject

It's an open exchange format for DAWs, and allows project files that follow the format to be opened in any DAW that supports it (currently Bitwig Studio and PreSonus Studio One). It's XML in a ZIP container.

tresf added enhancement ux labels Nov 18, 2017

tresf mentioned this issue Dec 9, 2017

Absolute home path written in .lmmsrc.xml #2933

Open

tresf mentioned this issue Jan 31, 2018

lmms -d is not possible on Windows #4145

Closed

tresf mentioned this issue Apr 9, 2018

Better default working directory #4288

Merged

tresf mentioned this issue Jul 30, 2018

Accessing Presets/Samples in .zip folder #4474

Open

tresf mentioned this issue Jun 8, 2019

Improved savefile layout for external resources #4922

Open

husamalhomsi removed the ux label Jul 31, 2019

Spekular mentioned this issue Aug 18, 2020

YAML for project files #5638

Closed

This comment has been minimized.

Sign in to view

Spekular mentioned this issue Dec 7, 2020

Reintroduce freeze function #762

Open

Spekular mentioned this issue May 1, 2022

project file folders for better organisation #6389

Closed

consolegrl mentioned this issue Sep 30, 2023

Implement Import/Export of the dawproject open format. #6909

Open

TechnoPorg mentioned this issue Mar 10, 2024

Asset System Rewrite #7138

Open

PhysSong mentioned this issue Jul 21, 2024

Sample Track Recording Stage One #5990

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JSON project format #3981

JSON project format #3981

tresf commented Nov 18, 2017 •

edited

Loading

JohannesLorenz commented Jun 8, 2019

tresf commented Jun 10, 2019

vlad0337187 commented Jun 13, 2019

tresf commented Jun 13, 2019

vlad0337187 commented Jun 15, 2019

0xf0xx0 commented Oct 26, 2019

mnini commented Aug 18, 2020

Veratil commented Aug 18, 2020

musikBear commented Aug 20, 2020

tresf commented Aug 20, 2020

0xf0xx0 commented Aug 20, 2020

vlad0337187 commented Aug 20, 2020

Veratil commented Aug 20, 2020

musikBear commented Aug 21, 2020

0xf0xx0 commented Aug 27, 2020

tresf commented Aug 27, 2020

DomClark commented Aug 29, 2020

0xf0xx0 commented Aug 29, 2020

musikBear commented Aug 30, 2020

0xf0xx0 commented Aug 31, 2020

tresf commented Aug 31, 2020

mnini commented Sep 13, 2020

DomClark commented Sep 13, 2020

mnini commented Sep 19, 2020

Veratil commented Sep 19, 2020

This comment has been minimized.

This comment has been minimized.

tresf commented Sep 22, 2020

tresf commented Sep 22, 2020 •

edited

Loading

0xf0xx0 commented Sep 22, 2020 •

edited

Loading

tresf commented Sep 22, 2020 •

edited

Loading

messmerd commented Sep 29, 2023

JSON project format #3981

JSON project format #3981

Comments

tresf commented Nov 18, 2017 • edited Loading

JohannesLorenz commented Jun 8, 2019

tresf commented Jun 10, 2019

vlad0337187 commented Jun 13, 2019

tresf commented Jun 13, 2019

vlad0337187 commented Jun 15, 2019

0xf0xx0 commented Oct 26, 2019

mnini commented Aug 18, 2020

Veratil commented Aug 18, 2020

musikBear commented Aug 20, 2020

tresf commented Aug 20, 2020

0xf0xx0 commented Aug 20, 2020

vlad0337187 commented Aug 20, 2020

Veratil commented Aug 20, 2020

musikBear commented Aug 21, 2020

0xf0xx0 commented Aug 27, 2020

tresf commented Aug 27, 2020

DomClark commented Aug 29, 2020

0xf0xx0 commented Aug 29, 2020

musikBear commented Aug 30, 2020

0xf0xx0 commented Aug 31, 2020

tresf commented Aug 31, 2020

mnini commented Sep 13, 2020

DomClark commented Sep 13, 2020

mnini commented Sep 19, 2020

Veratil commented Sep 19, 2020

This comment has been minimized.

This comment has been minimized.

tresf commented Sep 22, 2020

tresf commented Sep 22, 2020 • edited Loading

0xf0xx0 commented Sep 22, 2020 • edited Loading

tresf commented Sep 22, 2020 • edited Loading

messmerd commented Sep 29, 2023

tresf commented Nov 18, 2017 •

edited

Loading

tresf commented Sep 22, 2020 •

edited

Loading

0xf0xx0 commented Sep 22, 2020 •

edited

Loading

tresf commented Sep 22, 2020 •

edited

Loading