Unique file name extension for glTF, instead of .json

[antont]

Could glTF use a more specific file name extension instead of .json? For example .gltf?

It would be easier to support loading scenes / objects from glTF files when also other formats are supported. When a file is loaded it is most straighforward to see the type from the file name extension instead opening it and reading headers. For example to pass it on to the appropriate file loader.

I am now using the .gltf extension for this purpose in our framework where use Three.js for graphics and support also Three's JSON and OpenCTM loaders. With this unique file name extension it was trivial to add glTF support the same way:

else if (Tundra.suffixMatch(url, ".gltf"))
fn = this.loadGltf;

in https://github.com/realXtend/WebTundra/blob/dev/src/view/ThreeView.js#L894

I can well believe that there may be benefits to using the .json extension too, at least to tell that the file is indeed in json encoding. However it may be that the benefits of a unique / specific extension are greater. Also COLLADA files are called .dae and not .xml and that is AFAIK good.

In the similar issue in the three.js tracker it seems that mrdoob decided on using: ".3geo, .3mat and .3obj" instead of .js or .json - mrdoob/three.js#3298 (i didn't verify the current status of that now though, we've used .js and .json for three.js formats now)

[fabrobinet]

I understand the point, but currently we handle the glTF extension as a bundle.
There is a -b option in the converter where you specify that you want the output as a bundle. Means that all files related to the glTF asset (images, binary file...) will be copied inside a folder, such as myAsset.glTF

[emackey]

Should that be the default conversion? Is there any advantage to having the files stay separate on a production server?

Could the main file be something more descriptive than JSON, such as .glmesh or something?

[fabrobinet]

I don't mind changing this mode to be the default, it was just added, so I didn't want this change to be disturbing. For the JSON, it could be renamed as .scene ?

[emackey]

I like .scene. Looks like Ogre is using it as XML though. Maybe use .glscene or something?

[antont]

ah great news about the bundle .. I think, have to go through our usage though to see if it really works for us and solves this. possibly so, we've usually liked bundles. had missed the addition in the converter and was only looking at the spec about file names when creating this issue.

.scene might be good for the json. i don't think the collision with Ogre's dotscene is an absolute blocker as that format is afaik not used anywhere on the web atm and not too much otherwise either (we use Ogre in realXtend's native/c++ Tundra and support dotscene import but don't usually use it at runtime).

good catch anyhow that collision and .glscene might indeed be better.

[fabrobinet]

I don't think either the collision with ogre would be a problem. Purely subjective... but I don't really like glscene. A detail....but it is probably better to keep extension size <= 4 otherwise it might appear truncated sometime...

[RemiArnaud]

file extensions are not that important in regards to web servers. In fact,
it is recommended not to use file extensions in URL and instead using Accept
request-header on a URL
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
It is however very convenient to have a common extension for the 'bundle'.
I would recommend looking into something like a .kmz for example, when
someone want to download the bundle as a file, using a zip container.

What we should probably look into is to agree on a mime type to be able to
query a gltf scene using Accept request-header.
FYI - collada .dae (equivalent to gltf scenes) is model/vnd.collada+xml,
and was registered with IANA
so we coud have model/vnd.gltf+json for gltf scenes?

On Thu, Apr 3, 2014 at 10:21 AM, Fabrice Robinet
notifications@github.comwrote:

I don't think either the collision with ogre would be a problem. Purely
subjective... but I don't really like glscene. A detail....but it is
probably better to keep extension size <= 4 otherwise it might appear
truncated sometime...

Reply to this email directly or view it on GitHubhttps://github.com//issues/260#issuecomment-39479091
.

[antont]

sure that's a theory but how much is it the reality?

I've never seen a web page that would use a png file in any other way than the file having the .png suffix -- that's how web servers set the mime-type then but AFAIK for less common types that often doesn't happen in common web hosting setups and browsers fall back to looking at the extension. I may be completely wrong about this though.

for example Facebook refers to my cover image with a URL that has the jpg extension: https://fbcdn-sphotos-c-a.akamaihd.net/hphotos-ak-ash2/t31.0-8/s720x720/466996_10151606782755854_631840940_o.jpg

[emackey]

The MIME type only comes into play during transfer to client. I was thinking more about disk-based storage, both on the author's system and on the server's drive. I would think it would be better not to collide with other 3D packages like Ogre, since authors may have that installed. But in any case that collision is less problematic than colliding with all other .json files, especially if ThreeJS already stores its own format in a .json file.

Would be interesting to rename the scene .gltf and the combined file .gltz (following the cue from kml/kmz). But I can understand if other folks want to stick with .scene and .gltf.

[antont]

BTW Oge's dotscene is not a part of basic ogre (like .mesh and .skeleton) but an addon. And Ogre is a lib and not an application that would be associated to handle files when opened by the operating system or anything like that. The Ogitor scene editor might register as a .scene file handler though.

I like the gltf/gltz idea :)

[twinup]

@antont The URL could be anything, the fact that it ends in .jpg or .asp does not mean anything at all.
The reason your web browser receive the data and know what to do with it is because the response from the server contains the header 'Content-Type:image/jpeg'

Facebook is not a good example, as they seem to only return jpg anyways - by choice. But who knows , since it is a url, they could take the extension in the URL and use it to feed the option to an image converter, such as this API http://cloudinary.com/documentation/image_transformations#format_conversion.

Here's an example where the API provide 10 different possible formats to use for a query, using a parameter rather than an extension http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#//02r3000000wq000000. And then there is the modern http 1.1 that I already mentioned, header content negociation.

Basically, there very well could be a header parameter for the facebook URL that will ask for a png format, and the URL could still end in a .jpg. So, basically, never use the url as an indication of what the type is.
You have to use the mime type. (and then you can decide what filename extension you should be using)

[fabrobinet]

Remi please use same account for same thread :).

[RemiArnaud]

sorry, can't see what account I'm logged in, it is hidden on the top of the page.
gmail for example does not scroll the header on the page. github should do the same.

[antont]

@RemiArnaud / @screampoint - yes I know the theory and am all for mimetypes but, again, it is not what we face on the web today.

two more examples, this time with 3d content:

1. we support also CTM in WebTundra, the example for that (same as the xml3d.js CTM example) is at http://playsign.tklapp.com:8000/WebTundra/examples/xml3d/example-ctm.html and the ctm file used for the mesh is http://playsign.tklapp.com:8000/WebTundra/examples/xml3d/raf22031.ctm
2. we use Amazon S3 often to host the files for actual production use, one of the examples loads a scene from there using Three's json scene format, http://playsign.tklapp.com:8000/WebTundra/examples/xml3d/example-jsonscene.html . That uses DDS textures, one of them is: http://webtundra.s3-external-3.amazonaws.com/oulu9-jsonscene/Kauppurienkatu11.dds

Guess what the mimetype returned by those servers is for the .ctm and .dds files? binary/octet-stream . So how would we make things work today if we "never use url as an indication" as you suggest? Call Amazon and ask them to add mimetypes for all the exotic file types we require in each project, and ask for customers to wait until they fix things for us? Even if we could fix it for our services the next user of the library using some other web hosting service would then encounter it as a bug.

So yes by all means define mimetypes as well but it seems to me that more urgently we need descriptive file extensions. Perhaps some day then when these are old standards we can trust servers to return correct headers like they do for png and jpg today. In the meantime I don't think we are rushing to remove all code that looks at file extensions..

[RemiArnaud]

here's another interesting example:

this page http://www.wikihow.com/Image:Remove-Pages-from-a-PDF-File-Step-3-Version-2.jpg returns an html page.

So there are several issues mixed up. In your own web page, you can ask for whatever url you want, and then use the content the way you assume the server will send to you. Easiest way to do that is a very simple server that map URL to folders/files.

When I click on the second link or fourth link, the browser will send me the filename in the header has well. So the filename that the browser will save the file into has nothing to do with what is in the url.

What is going on here has nothing to do with mime type. The browser or the server knows nothing about the content or format.

I would not constrict the 'face of the web' of those examples where the web server is not doing anything else than transmitting files that it ignores everything about. This is how the web started, but nowadays servers are capable of a lot more than that.

BTW, I would not recommend using those URLs as simple references to a file system, if you intend to be able to change the content. There are several options that I let you think about.
Just imagine that you are working on a game, and want to provide updates. You have many players already in the game, and you don't want to mess them up. But you want the new players to get the new content. Say you want to change the texture with a new texture file. Just that one resource.
One other thing to keep in mind is that you want to enable web browser to use their cache, so players don't have to reload everything every time they get back to the game.

[antont]

ok fine, so we are not looking at URLs but instead the file name that the browser gets from the headers - that seems good. as long as the name has a useful extension :)

BTW I still like the gltz / gltf idea .. 'gl transfer zip bundle' and 'gl transfer format / file'. also other files inside the bundle have specific names, like .glsl and .dds, so why not .gltf for the json as that is the new thing that the gltf spec defines. the other things are specified in their own specs.

@emackey asked whether there are any production benefits with separate files instead of always using the bundle. I think @RemiArnaud provided one in his latest: allows to update a single texture in a game without requiring browsers that already have the files in cache to re-download everything. http's last-modified works for that, AFAIK ok.

[emackey]

@RemiArnaud good point about updating a single texture. This means some production servers will have the individual files hosted, so it's even more important to move away from .json.

With the JPG example, I've written web services placed on URLs (not ending in .jpg) that dynamically generate an image with the correct MIME type. But this is the 2% use case in my code, the remaining 98% of my JPGs are flat files on disk. These flat files derive their MIME types from the file extension.

Some use of GLTF will be dynamic, computed on the fly with the appropriate headers. But I suspect a lot of it will be static assets that have made their way through the converter, and are sitting on a drive someplace, waiting to be used. Avoiding collisions with file extensions, particularly with other 3D formats that could be on the same server, is important to support this case.

@antont Thanks, I do hope that gltf and gltz would be a recognizable pattern to newcomers. If they've been using COLLADA inside kmz, it should seem familiar.

[pjcozzi]

Just chiming in now:

• .gltf is OK with me and better than the proposed .scene especially since the top-level gltf property contains scene and scenes.
• The current bundle and proposed .gltz are variations on embedding all the resources into the JSON file (Format should support references and inline data #68), which we have a small client-side library for reading. I don't know how useful the current bundle is without also zipping it. Zipping will require the client to be able to unzip it, which requires a non-trivial third-party library AFAIK. Providing this many options and this much client-side pain isn't exactly the glTF manifesto. For a draft 1.0, I would only advocate embedding resources, then perhaps do .gltz if needed to improve file size for bundles later. We have to fight feature creep so we can finish a 1.0.
• The MIME type discussion is all you guys; I don't have anything to contribute.

[antont]

ah sorry for my ignorance - i didn't realise that current gltf bundles are simply inlining, thanks for info.

afaik bundling is considered useful even without zipping on the web, reduces the number of requests required. unzipping on the other hand can be done by the browser itself (and even automatically by both server and browser) as a part of how http works.

[pjcozzi]

afaik bundling is considered useful even without zipping on the web, reduces the number of requests required

My - perhaps incorrect - understanding is that the current bundle is just a folder; it still has separate files in it. @fabrobinet can confirm.

unzipping on the other hand can be done by the browser itself (and even automatically by both server and browser) as a part of how http works.

This is based on the HTTP header? This makes the option a lot more attractive. Can you point me to an example? I can't quite picture the exposed JavaScript API.

[antont]

I mean just the ability of web servers to zip the data they send, a single file. Yes it is based on HTTP headers. But no it is not exposed to Javascript - your web app just sees files like normally.

http://en.wikipedia.org/wiki/HTTP_compression has an example: "Accept-Encoding: gzip, deflate"

So if you put all the files in one and then use that compression you get everything in one zip .. and the native code in browser handles the unzipping. It can be done live by the server but IIRC also so that you just put precompressed mydata.gz files there.

Having that compression on is AFAIK common on the web. We use it e.g. in our mobile (iOS) game to compress leaderboard etc. json sends from web service to the obj-c written game .. just set that header in the request and make sure the server is configured to serve so.

Interestingly enough the Three.js entry which I linked to earlier mentions that they started using .js as the file name extension for their JSON because one of the devs hinted that it's usually configured to be gzipped by default. The Blender Three.js exporter at least still gives .js files (.. we've been renaming those to .json so far in our projects).

[tparisi]

This can't be an issue unique to glTF. The problem is that an application is potentially dealing with different "vocabularies" in JSON. Same idea as XML... it's one content type that can actually implement multiple languages. Has anybody seen any work done on a standard way to identify a JSON "doc type" ?

[antont]

perhaps some json headers can be nice but speaking of XML I suppose everyone likes the fact that Collada files are .dae and not .xml ..

[pjcozzi]

@antont if I am following you correctly, you are talking about serving a single file zipped, which is a no-brainer I agree. I was asking if the browser would handle a zip of several files for bundling but it sounds like no, which is why I am advocating for the embedded resources into the JSON file, which can, of course, still be zipped.

[antont]

@pjcozzi yes that's my understanding too, browsers can't handle zip bundles, the http-compression biz is for single files. this seem to confirm, people use a javascript unzip impl when have to unpack such an archive in the browser, zip.js: http://stackoverflow.com/questions/2095697/unzipping-files

so yes the way you've designed and implemented this in #68 already is exactly what's right for delivering a compressed 'bundle' (joined / inlined / embedded json(s) + glsls + bins and optionally even images) to browsers.

i'm not sure what how #68 works says about file name extensions but am still hopeful based on these talks that the gltf json will get some useful extension also in the case when nothing is embedded :)

[pjcozzi]

am still hopeful based on these talks that the gltf json will get some useful extension also in the case when nothing is embedded :)

Agreed. .gltf is my recommendation.

[fabrobinet]

@pjcozzi can you sum up the conclusions if any ? sorry I didn't really follow all that thread I was taking time off... Here are my questions and remarks.

• To answer your previous question, yes .glTF as it is now is just a bundle and doesn't help WRT multiple XHR but it still have value for applications (like the one I am working on...) to easily/cleanly handle assets when performance during authoring is not critical and then you can benefit of the flexibility of being able to access the content of the bundle easily...
• I am OK and happy to add a .glTZ option to produce this .glTF as a ZIP.
• OK to generate bundle as the default ?

No we have the issue of how to name the JSON:

• When inside the bundle, any extension is OK as long as the JSON file is formed as filename.EXT.
• Then what the case without the bundle ? I suggestion would be to use .glTF extension for it and consequently use another extension for the bundle to avoid confusion. What about .glTB (as gl transmission bundle) ? and we could just keep glTZ.

Note: I just made fixes in dev-6 branches for bundles, so better waiting for dev-6 to be merged before using them...

[pjcozzi]

Summary:

• Rename .json to .gltf
• We will embed resources as discussed in Format should support references and inline data #68. I see no value in standardizing the current bundle or zipping the current bundle. It's fine as a converter option since you need/want it, but is not needed for the spec. We should implement Format should support references and inline data #68 and make it the default.

[fabrobinet]

• Yes bundle are are just converter options.
• If you are talking about just embedding shaders by default, yes for Format should support references and inline data #68.

[pjcozzi]

If you are talking about just embedding shaders by default, yes for #68.

I am talking about embedding shaders, textures, and geometry/animation/skins by default, but if you only want to embed shaders by default, it is OK with me since this is a converter decision.

[fabrobinet]

We're good then. This is not really a format change, so let's target dev-6 too.

[pjcozzi]

👍

[fabrobinet]

that's dev-7 then...