3.0.0 Schema changes

[gilesknap]

This change allows us to more effectively use anchors and aliases to (multiple) inherit groups of parameters from other definitions or from shared fields.

Also includes a converter tool to convert support yaml and ioc yaml to work with the latest round of changes to ibek.

The names of the classes in the code have also been changed to reflect the name changes in the schema detailed below.

The schema changes are as follows:

• defs are now entity_models
• args are now parameters
• parameters are a dictionary of dictionaries instead of a list of dictionaries
• parameters no longer have a name field as the key in the dictionary is the name
• values is now post_defines and is also a dictionary of dictionaries
• pre_values is now pre_defines and is also a dictionary of dictionaries
• __utils__ is now _global
• _global functions have changed:
  • get_var -> get
  • set_var -> set
  • counter -> incrementor
    • behaviour has changed: the --increment and --stop flags are interpreted
      on every invocation of the incrementor function (so non linear counters
      are possible)
    • incrementors current value is accessible via _global.get(incrementorName)

Backward compatible changes:

• object refs are literally the object - so you can modify a referenced object's
  attributes (with appropriate care!!)
• pre_defines and post_defines now may have optional type: which defaults to str and may be
  • list (of Any), int, float, bool
• each entity_model may have a shared: field which is a list of Any
• there is also a global shared: at the root level
• shared: are scratch areas in which to place anchors that can be referred to by aliases - and are therefore useful for placing repeating patterns (especially of parameters) and then using aliases where these repeats are required.

[gilesknap]

NOTE: this is breaking the system test because it uses ioc-template-example and that in turn is using ibek-support which has not been updated to the new schema changes yet.

NEXT CALL: adding CI to ibek support that checks all of this so I can

• validate that CI runs against current ibek-support with ibek 2.x
• do the conversion on all ibek-support
• validate that the CI runs with the new ibek 3.x (the CI will include a requirements.txt for a compatible ibek version)
• fix the ioc-template to use the above version of ibek-support AND the 3.x version of ibek (not required for this system test but required so that ioc-template-example's own CI works)
• re-generate ioc-template-example
• re run the CI for ibek

Some notes on the remaining points to complete in this PR

[GDYendell]

OK, I now realise I don't like this name. Classes shouldn't be verbs. And DefinesTypes is very unclear.

It also leads me to a question. Are pre_defines and post_defines different to Definitions?

[gilesknap]

yes this was one of the points in slack we have definitions that contain pre_defines and post_defines.

Your verb point is also good.

So how about values then!! ??

[gilesknap]

i.e.

pre_values
post_values

[GDYendell]

Should they not be called pre/post args given that they are inserted into the Entity args?

Or perhaps, the section in the yaml could stay as defines, but the object is Pre/Post Arg. And the description could be Pre/Post -defined Args for Entity.

[gilesknap]

I'm not that keen on the trend we have started of using different class names in the code than the key names in yaml.
Maybe we could fix that and this issue by renaming
defs:
to
entityDefinitions:

which are what we called the class in the code.

[GDYendell]

EntityDefinition in yaml and code sounds good.

These are distinct from Parameters (no longer called Args and now appearing under params: in yaml) because they are calculated - never supplied by writer of an ioc.yaml - thus distinctly not parameters (or args!)

Ah right they are being renamed here. Then should this be updated? Or are there now Args and Params?

It looks like (what is currently called) Defines are converted directly into the EntityFactory args (just a dictionary of builtins) and then that is used to populate the fields of the dynamically generated Entity model, which I think makes them parameters, right?

[gilesknap]

Yes all use of 'arg' in the code should be replaced with param and your link is to an oversight.

You are right that 'defines' get treated like 'params' because they go in the Entity Model. But you are not supposed to be able to use them as params - now I'm worried that you can - just checking that now ...

[gilesknap]

Yes - you can reference definitions in the ioc.yaml so that was unintentional.

Now the question is is that a bad thing. i.e. although they are intended as calculated values or constants, you can still override them in the ioc.yaml if you wish.

I think that probably it is a bad thing because the order of interpretation of jinja strings is supposed to be clear and I'm feeling that this is muddying it. Also - I did not want to confuse the ioc instance writer with many extra parameters that they should not normally set.

So I need to fix that and then still come up with the correct name.

I used pre_defines and post_defines because they are SUPPOSED to be like defines in C ie. calculated values or constants.

[gilesknap]

To help think about this here is a reminder of an example use of defines.

ibek/tests/samples/support/fastVacuum.ibek.support.yaml

Lines 114 to 140 in d82c40f

	post_defines:
	gaugeNum:
	description: auto gauge count
	type: int
	value: |-
	{{ _global.incrementor(master, start=1) }}
	fan:
	description: fan number
	value: |-
	{{ "%02d" % (gaugeNum / 7 + 1) }}
	mask:
	description: mask for the channel
	value: |-
	{{ _global.incrementor("mask_{}".format(master), 2, 2**gaugeNum) }}
	lnk_no:
	description: link number
	value: |-
	{{ ((gaugeNum - 1) % 6) + 1 }}
	gaugePV:
	description: Gauge PV
	value: |-
	{{ master.device }}:GAUGE:{{id}}_0

These are all calcualted values that help to generate values for db templates - not something the individual IOC instance creator wants to worry about.

[gilesknap]

@GDYendell please can we spend 20 mins IRL discussing this today as I'd like to resolve ASAP.

[gilesknap]

@GDYendell after around 12 hours in a debugger it finally came out pretty in my opinion.

All jina rendering and reference to pre_defines and post_defines are all handled in the Entity model validator here:

ibek/src/ibek/ioc.py

Lines 97 to 122 in 2bd5cb3

	@model_validator(mode="after")
	def add_ibek_attributes(self):
	"""
	Whole Entity model validation
	Do jinja rendering of pre_defines/ parameters / post_defines
	in the correct order.
	Also adds pre_defines and post_defines to the model instance, making
	them available for the phase 2 (final) jinja rendering performed in
	ibek.runtime_cmds.generate().
	"""
	if self.__definition__.pre_defines:
	for name, define in self.__definition__.pre_defines.items():
	self._process_field(name, define.value, define.type)
	if self.__definition__.parameters:
	for name, parameter in self.__definition__.parameters.items():
	self._process_field(name, getattr(self, name), parameter.type)
	if self.__definition__.post_defines:
	for name, define in self.__definition__.post_defines.items():
	self._process_field(name, define.value, define.type)
	return self

(I think we eventually agreed that define may be used as a noun, right?)

[GDYendell]

I think this PR needs to be renamed "Refactor the world"

[gilesknap]

I think this PR needs to be renamed "Refactor the world"

Was just thinking what to call it.

But here is the cool bit.

• I wrote CI for ibek-support that checks a bunch of generic ioc builds and ioc instance renders for all ioc instances in bl47p bl45p and bl01t
• then I ran the auto converter on all of the support yaml and ioc yaml
• plus updated the CI ibek version to latest
• the CI picked up a single change which is that an enum was now correctly rendering in a subst file

Now this has only checked support yaml for those things we have actual IOC instances for but it is a very good cross-section.

[gilesknap]

@GDYendell

I have clarified the ID discussion as follows:

ibek/src/ibek/ioc.py

Lines 124 to 139 in db46dba

	def __str__(self):
	"""
	When a jinja template refers to an object by itself e.g.
	# this is the startup entry for {{ my_entity }}
	Jinja will attempt to render the object as a string and this
	method will be called.
	The behaviour is to print the ID of the object. Thus we look up
	which of our object's fields is the ID field and return the
	value of that field.
	"""
	id_name = self._model._get_id_arg()
	if id_name:
	return getattr(self, id_name)
	else:
	raise ValueError(f"Entity {self} has no id field")

and

ibek/src/ibek/entity_model.py

Lines 178 to 190 in 0043ef5

	def _get_id_arg(self) -> str | None:
	"""
	Lookup which of this object's fields represents its ID. The ID is the
	field of type IdParam whose value is the identifying name for
	this object.
	The jinja templates of other objects can refer to this object by the
	value of its ID field.
	"""
	for name, param in self.parameters.items():
	if isinstance(param, IdParam):
	return name
	return None