File format specification

[peter88213]

Is there a specification for the .nwx project file format that says something about the order of content item entries?
With my yWriter converter, I first built the project tree based on the parent entries and then serialized it again. In the last version, I abandoned that and just used the flat XML structure to obtain the document structure.
As it looks, novelWriter writes the parts, chapters and scenes of the document in the right order and puts "outtakes" and "trash" at the end. Thus, the "Parent" information for each node of the project tree is basically redundant. Or not so? Can one rely on the project tree being properly serialized in the .nwx project file?

[vkbo]

The order of the content in the XML is the linear order of items in the project tree from top to bottom. Linear meaning it ignores the parent/child hierarchy (column number) and follows the row number. You can rely on this being the case when novelWriter writes the XML. However, the parent reference is needed in order to re-create the tree structure when the project is loaded, otherwise it would end up as a flat list. The order attribute is presently redundant though. It is mostly written to the XML for debug purposes.

The algorithm that loads the project tree is able to handle cases where the content entries in the file have been re-ordered or appear in the wrong order from a hierarchical point of view. It will detect, for instance, if a child item is loaded before its parent item, in which case it will postpone the loading of that item and try again later. So the loading is robust in terms of parent/child relationship, but a re-ordered XML will still mess up the order of sibling items at any given level. Postponing the loading of an item will also put it in the wrong location after the tree is built.

Since the intended order is saved to the XML, I could write code that reorders the final tree based on the order attribute in the XML, but since there is no scenario internally in the XML handling that could produce such an outcome as it stands. Therefore I never prioritised this. Of course, when the file is generated externally such a scenario is possible, so if needed I can look into this. In that case, the order of entries in the XML would in principle no longer matter at all.

The early versions of novelWriter ran on Python 3.4 and up, and prior to 3.6 the entries of a dictionary were ordered by their hash key. Since Python 3.6 the order is effectively the same as they were added, although this isn't guaranteed until Python 3.7. Due to the history of Python version support, the project content is stored separately (in a dict) from the project order (in a list) and there are the mentioned checks on loading that makes sure everything is ok before it is loaded into the project tree widget. From next release I'm dropping Python 3.6, so now it's actually possible to redesign this process, but I will not change the XML format in any case.

For reference, the function that serves the entries to the tree widget during load time can be found here:

novelWriter/novelwriter/core/project.py

Lines 1143 to 1183 in 7f43584

	def getProjectItems(self):
	"""This function ensures that the item tree loaded is sent to
	the GUI tree view in such a way that the tree can be built. That
	is, the parent item must be sent before its child. In principle,
	a proper XML file will already ensure that, but in the event the
	order has been altered, or a file is orphaned, this function is
	capable of handling it.
	"""
	sentItems = []
	iterItems = self.projTree.handles()
	n = 0
	nMax = min(len(iterItems), 10000)
	while n < nMax:
	tHandle = iterItems[n]
	tItem = self.projTree[tHandle]
	n += 1
	if tItem is None:
	# Technically a bug since treeOrder is built from the
	# same data as projTree
	continue
	elif tItem.itemParent is None:
	# Item is a root, or already been identified as an
	# orphaned item
	sentItems.append(tHandle)
	yield tItem
	elif tItem.itemParent in sentItems:
	# Item's parent has been sent, so all is fine
	sentItems.append(tHandle)
	yield tItem
	elif tItem.itemParent in iterItems:
	# Item's parent exists, but hasn't been sent yet, so add
	# it again to the end, but make sure this doesn't get
	# out hand, so we cap at 10000 items
	logger.warning("Item '%s' found before its parent", tHandle)
	iterItems.append(tHandle)
	nMax = min(len(iterItems), 10000)
	else:
	# Item is orphaned
	logger.error("Item '%s' has no parent in current tree", tHandle)
	tItem.setParent(None)
	yield tItem

[peter88213]

Thank you for clarifying this.

Due to the history of Python version support, the project content is stored separately (in a dict) from the project order (in a list)

This is also how I implemented the yWriter metamodel, although I am aware that since (C)Python 3.6 dictionaries are ordered by design. But when restructuring or merging projects, the lists seem a bit handier to me.

But essential for me is the convention that the order in the XML file matters. This allows for much more compact code for reading the file.
Then I can easily build the document structure from the heading hierarchy.
When creating a .nwx file, the correct sequence of items is automatically generated. Of course, I also create the "parent" attributes, but this again requires additional effort.

If there should ever be an official specification of your file format, it would definitely be beneficial to mention the aspect of order in the XML structure.

Cheers,
Peter

[vkbo]

Sure, I can add a file spec to the technical section of the docs.

[vkbo]

I've had the file format spec document for a while now, but I added it to the documentation today. It's available from the Project Format Changes page. It is available as a PDF. The raw document is in the repo.

In the code I've separated out the XML reading and writing into a separate module. The XML writer class is available here:

novelWriter/novelwriter/core/projectxml.py

Lines 475 to 485 in ecff40d

	class ProjectXMLWriter:
	def __init__(self, path):
	self._path = Path(path)
	self._error = None
	return
	##
	# Properties

It should make it easier to re-implement in various tool. I've modified the format to be a lot simpler in general, as is evident from the class.

[peter88213]

I'm looking at the new file format and am considering updating my yWriter file converter.
Your file format specification as PDF is good work. However, for version comparison, a DTD would be very helpful. Is there something like that for NovelWriter?

[vkbo]

On another note, is the converter tool functional now so that I can link to it from the documentation?

I am planning on adding a HowTo section one I have enough "hows" to write one.

[peter88213]

At the moment, the specification is as follows:

• Read and write file format version 1.3 created with novelWriter 1.5 and 1.6
• Read file format version 1.4 created with novelWriter 1.7

For file format v1.5 support, I opened an issue and started a branch. However, I cannot make no statement as to when the release will be ready. Actually, I wanted to interpose the topic today, but it looks like the weekend is already over and I have to turn away from the computer screen now.

[vkbo]

As long as there is a text that describes which versions are supported, there is no problem adding it to the docs. But as the weekend is indeed over, I will not do anything on this today either.

As a sidenote, there was never a release 1.7, hence there is probably no 1.4 format projects out there save for a few users who tested the 2.0 RC 1 pre-release. I would just ignore 1.4 altogether. Anyone using it should anyway update to the full 2.0 release which uses project format 1.5.

[peter88213]

That's okay with me.
See: peter88213/yw2nw#2

[peter88213]

Meanwhile, I implemented full v 1.5 support, and dropped v 1.3. With that, I'm ending the development of this tool unless serious bugs are reported. After all, I have no idea if there is anyone out there who uses this.