Wrong export byte with CE-only tokens

[PeterTillema]

When a program contains tokens which are only available on the CE (such as Wait, Asm84CEPrgm), the bytes at 08 09 0A should be 1A 0A 13, not 1A 0A 00. When there are CSE/CE tokens (such as RED) it should be 1A 0A 0F. If no special CSE/CE token is present, it should be 1A 0A 0A. At least, TI-Connect CE tells me that, I can't find any documentation for the CE file formats.

[adriweb]

I'll double check in TI-Connect CE as well but that seems correct indeed.

[PeterTillema]

Update: all the exported variables should have 1A 0A 0A, except in the following cases:

• if a string/appvar/(protected) program contains any CSE or CE tokens (like the colors), it should be 1A 0A 0F
• if a string/appvar/(protected) program contains CE tokens (like Wait), it should be 1A 0A 13
• Pictures and GDB's should also be 1A 0A 0F (probably because those are also for the CSE/CE only).

So far I didn't see any exception of the types I checked.

[adriweb]

So according to https://github.com/TI-Planet/z80_basic_tokens/blob/master/tokens.csv

• if a string/appvar/(protected) program contains any CSE or CE tokens (like the colors), it should be 1A 0A 0F
• Pictures and GDB's should also be 1A 0A 0F (probably because those are also for the CSE/CE only).

so, if any token is >= 0xEF41 (until 0xEF6C apparently)

• if a string/appvar/(protected) program contains CE tokens (like Wait), it should be 1A 0A 13

This might have some other finer version bytes (10, 11, 12, 13 ?), considering:

• 0xEF73 -> 0xEF7A are from OS 5.0
• 0xEF81 -> 0xEF98 are from OS 5.2
• 0xEF9E -> 0xEFA6 are from OS 5.3

(Has someone tested what's in those ranges holes, if anything?)

[debrouxl]

The 0A, 0F and 13 values look familiar. See enum CalcProductIDs in https://github.com/debrouxl/tilibs/blob/experimental2/libticalcs/trunk/src/ticalcs.h .
If you use 0A or 04 instead of the more usual 00 in e.g. a string variable, among other types of variables whose format shouldn't have changed since before the advent of the 84+ in 2005 (right?), does TI-Connect <= 4 still accept to send the given variables to a 83+(SE) ?

[adriweb]

well it probably doesn't matter if it can't send them to older calcs since they wouldn't actually be compatible anyway (but if it does transfer some vartypes correctly despite that correctly, an ERR:VERSION might appear I guess?)

[debrouxl]

Of course, 84+CSE/TI-eZ80 PPRGMs / Pics / GDBs, 83PCE EP TI-Python APPVs, 83PCE(EP) exact math var types, and other similar types couldn't be handled properly by the monochrome 84+ or 83+ families anyway. However, if there's an actual risk of doing so, I think that we want to avoid annoying users by using compatibility levels higher than they need to be (say, 0A instead of 00), for the types of variables which should work just fine on older models. That's just what I was trying to state here :)

Some people make TI-Basic programs + variables they depend on suitable for multiple models.

[adriweb]

Well I guess we could probably keep 00 instead of 0A until we find a more accurate rule.
But for the ranges I've noted above, there isn't any confusing cases.

[PeterTillema]

Which means that (complex) numbers, (complex) lists, matrices, window, table and zoom variables should still have 00, while the others at least 0A, and eventually a higher number (will explore later).

[PeterTillema]

I've tested several "new" tokens so far, but it seems every CE-only token is 13, CSE/CE is still 0F and normally it's 0A. Can't guarentee this applies to everything of course.

[adriweb]

Right, this then appears to be indeed a calculator product ID mapping directly, as @debrouxl mentioned.
So, let's only enforce the dedicated version byte on things we're sure are only usable for the calc models in question (for tokenized variables, we'll have to check token bytes individually and set the minimum version accordingly, for the newer vartypes it's easier)

[PeterTillema]

well it probably doesn't matter if it can't send them to older calcs since they wouldn't actually be compatible anyway (but if it does transfer some vartypes correctly despite that correctly, an ERR:VERSION might appear I guess?)

Indeed, if the version byte for variables (not the global version byte) is higher than allowed, you get an error with TI-Connect CE. I tested it by changing the default $0C for the CE to $0D and it indeed didn't work. Note that this doesn't apply to the global variable. Changing that from $13 to $14 works fine.

--

The global version (offset 08 09 0A in the exported file) has these options:

• 1F 0A 0A for all CE programs, no matter the data
• 1F 0A 0F for CE programs with CSE/CE tokens in it, like RED
• 1F 0A 13 for CE programs with CE-only tokens in it, like Wait

Each variable in the exported file has a version byte too, and you can find the possible values here: https://wikiti.brandonw.net/index.php?title=83Plus:OS:Variable_Versions

[PeterTillema]

Any updates on this?

[adriweb]

Not yet unfortunately. Would you have a pseudo-code algorithm of what you think would be best to have in CEmu?

[PeterTillema]

Well, it is actually very easy. Loop through the program, and get all tokens. Then check the version of each token, which is like checking if the bytes are between 2 bounds, and return the version of the token. Then the final version is the maximum of all those tokens, so for every token: version = max(token_version, version);.

[adriweb]

Alright, looks like it would be a feature of ti_vars_lib to be able to "compute" the minimum-compatible version, since that's an information specific to tokens. CEmu doesn't deal with that "low" of a level for file exporting purposes, but the lib does, and since CEmu has it readily available, it could use it. The problem may be that the lib is only used for the front end Qt stuff, not in the core where transfers happen, so it's not like the core could just call a function of the lib. There's a bit of thinking to do first as to how/when/where the front end could pass the info to the core... but that would mean any non-Qt version (web, sdl, cli...) of CEmu would not have this feature. But you know, we could also just "give up" and add this token version-processing in the core in some static function somewhere around the variable exporting code in order to get the best version field value (in addition to ti_vars_lib since that's always useful anyway)

[adriweb]

Just leaving that here for reference:
The third byte is actually what's called the "owner calculator ID", and it can have the following values (that I have seen so far):

Min compatible calc	Value
TI-83+	0x04
TI-84+	0x0A
TI-82 Advanced	0x0B
TI-84+ C SE	0x0F
TI-83PCE/84+CE	0x13
TI-84+ T	0x1B

I don't know why there's a new value for the 84+T, is there any new token we hadn't noticed?
Edit: well I guess it doesn't necessarily mean there's new stuff, it's just the calculator ID.

[PeterTillema]

The third byte is actually what's called the "owner calculator ID", and it can have the following values (that I have seen so far):

https://github.com/debrouxl/tilibs/blob/master/libticalcs/trunk/src/ticalcs.h#L166-L191 here you go, as ldebroux already pointed out.

[adriweb]

Yes I know, it's in my edit of the comment above :P
But I was just listing the values we've seen so far, although all of them may not really matter for what we want to implement here.

[adriweb]

The doc on the wiki was updated (by @LogicalJoe), and I'll try to take care of all that soon™ on the tivars_lib_cpp side.
Then CEmu will probably have to query the lib to know which version byte to use, at least for some types...

[calc84maniac]

Just adding a comment here before I push my fix, based on my observations of TI-OS behavior. There are very specific situations when the OS updates the version field of a variable:

For programs, the version is updated only when the program editor is closed. I was unable to identify a situation where protected programs or appvars had their version updated. This makes some sense because protected ASM programs can contain invalid tokens, and obviously the same goes for appvars.

For image files, the version seems to be set properly at creation (which makes sense, because they're created in Archive). The same goes for group files, where the version is set to the maximum version of each of the grouped files.

For most other variable types (including strings, equations, and all numeric/list/matrix types), the version is updated either when the variable is archived or when the variable is about to be sent from RAM. This is the case that needs to be handled explicitly in CEmu, by calculating an updated version field only if the variable is in RAM.

In all cases, it seems correct that the product ID is set based on the version of the sent variable, as discussed in earlier comments. When creating a .8cg group file from CEmu (I'm not sure if TI-Connect CE has an equivalent for this) it makes sense to me to set the product ID based on the maximum version of each of the grouped files.