Proposed RFC Feature Engine, Project and Gem Versions #44
Comments
|
Relates to #o3de/o3de#9669
I sort of want a way to automatically update a file in git to store and lookup the last commit timestamp. Then in my client and server I can see major.minor.patch + timestamp. If I notice the time stamps on client and server are a week apart vs 1 month apart, I can start getting suspicious that I'm running two incompatible builds should there be any weird behavior. |
|
@AMZN-Gene, re: using this for server/client mismatch, @kberg-amzn has some thoughts on why network or client/server versioning should not be tied to engine, project or gem versioning, let me know if that changes your feedback. re: patch numbers changing often - that is certainly a possibility, but I would discourage bumping the patch number with every bug fix, minor feature or chore and only recommend patch changes for ones that are significant enough that a dependency needs to know about it - like security fixes. |
|
Sounds like a good approach, I especially like the decision to use semver and the Python versioning scheme. A few questions, more for understanding the proposal than real feedback:
I am wondering if it's even necessary/recommended to set |
|
Thanks for the feedback @fabioanderegg,
The name of the engine in
The
Yes, adding support for
In some cases you must use |
|
In your example, the development version is not greater that the release version. Will it be the case? And in most of the cases, when handling multiple release versions in the code, the code of the latest release will need to be activated for the development version too. You can see an example here : https://github.com/PopcornFX/O3DEPopcornFXPlugin/blob/9ba771490535ab5f4f61d65ac1ba7024cb535db7/Code/Source/Integration/Render/AtomIntegration/PopcornFXFeatureProcessor.cpp#L133 |
|
I like the proposal and especially the tracking of fine-grained features with I have one concern about using the engine name in Although the first case is simple to fix (i.e. a developer must always define I was thinking about the following suggestion, starting from the assumption that engine variants (e.g. source, sdk) with the same version should not provide different list of features if they are released by the same entity (e.g. O3DE, a game studio):
This allows users to customize their engine installation names (even if targeting the same engine version) without loosing the compatibility check for gems, whereas developers haven't to explicit all engine APIs if they don't want. |
|
@ValPKFX Thank you for your feedback
Thanks for pointing that out, I'll update the design to include @loherangrin thanks for pointing out the issue surrounding multiple engine names/versions.
The In your example, if a developer wanted to indicate their gem was compatible with any engine, they could leave the Thoughts? As you pointed out, if a gem developer wanted to make their gem compatible with multiple engine distributions from various providers (O3DE, game studios etc) they should use That said, we will always let the user use any combination of engine and gems they want, but steer them toward compatible versions if there are any available, and otherwise point out to them when a gem/engine combination isn't known to be compatible.
So if I understand correctly, the Changing what the I think it would be good to say the engine and gems provided in a source and SDK variant of the engine are the same and they can be treated the same for version compatibility checks, but we can only guarantee that for the O3DE distributions. Our documentation shows how to make an SDK and install it without making any changes to the provided Gems or engine, but I think the goal of making a custom SDK is to make changes for distributing and so in most cases users who make a custom SDK will have functionality that is different from what O3DE provides. That said, I'm sure you're right that they'll want to make it easy for their engine to be listed as compatible with gems that are known compatible with another engine. |
Thank @AMZN-alexpete for the explanation, I got the specification in the most strict way. So, when the compatibility status is unknown, is the user allowed to continue anyway showing some additional warning message (or nothing at all)?
It looks good to me, assuming that an alias cannot be expanded recursively (e.g. if
Sorry for not having added an example. My proposal was limited to add a new field in O3DE releases can keep the two different names in However, your previous addition of |
Hi @loherangrin! A user will always be allowed to continue and add a gem even if it is not known to be compatible, but there will be some kind of visual indication to the user that the gem is not known compatible. Thank you for providing an example of |
|
@ValPKFX @fabioanderegg during the sig-core weekly meeting today we discussed how we could simplify the engine version fields and thought it best to make the following changes:
@loherangrin we discussed the |
|
@AMZN-alexpete That being said, although the actual compatibility logic should be only tied to the |
We certainly could have a helper function to return the engine version based on the display version - though maintaining a map may not be necessary because it's unlikely users will have many engines installed on their machine at a time. A simple way for users to discover the engine version would be for us to add that information to the Project Manager's Engine screen. |
|
That's true, that would definitely be simpler. Depending on how we do it though, I would be concerned about it being error-prone with typos for a given engine screen. A mapping structure can help us with validation, but if the use case is too simple it likely doesn't justify the effort into making that structure. |
|
For the Feature Design Description section, step 4 mentions that o3de.py and Project Manager will need to be updated to handle the version specifications
CMake should also be mentioned in that entry. For the workflows sections, it says that the Pretty much I imagine before the the stabilization branch is created the engine.json has the engine major and minor versions for the next release. development branch engine.json"engine_version": "1.1.0"When the stabilization branch is created they both have an engine_version of "1.1.0". development branch engine.json"engine_version": "1.1.0"stabilization branch engine.json"engine_version": "1.1.0"But right after the stabilization branch is created a commit can be made to the development to update the minor version to 1.2.0 development branch engine.json"engine_version": "1.2.0"stabilization branch engine.json"engine_version": "1.1.0"Afterwards the workflow could specify that the the major and minor versions should NEVER be updated in the stabilization branch. Now I agree with making the engine display version just a user friendly version for marketing purposes I think we should first try just using the I think it adds complexity that is not needed and can potentially be a flaw in the version system if we allow it, so I think we should let some time pass to handle it. Furthermore users wanting to specify a dependency on a specific version would have to look in the engine.json determine the version , which has both fields anyway. So it is not like the user can have the excuse to say they only know the "marketing" version and not the actual version. Finally the version information isn't targeting for novice users. It requires at least the ability to locate the "engine_version" field in the engine.json file. |
|
@lumberyard-employee-dm Thanks for the feedback - I agree, a minor version update would be better than a patch change. I've made that change and updated the Feature Design Description to also mention CMake. |
|
This RFC looks good with me and I am OK with signing off with the plan to implement this semantic version scheme as well as the mechanism for specifying dependencies on engine api versions, engine version and gem versions. |
|
One consideration that was brought up was - do we need template.json support for |
|
How would non-official/custom engine versioning work? One would want to fork the engine for their own work without conflating custom engine semver with official engine semver. In the gem world one could simply give the gem a new name to fork it. But in the engine world there is simply 'the engine' with a version without scope for renaming. |
The "myengine==1.2.3"A common example of this would be game teams that have a custom internal version of the engine. As soon as that game team starts to change the engine version to reflect changes they made to the engine, they would now be responsible for determining what gems are compatible with their engine and updating those @flappybirdace Did I understand your question correctly? |
Yes. My bad, I didn't read the conversation correctly. |
@AMZN-alexpete It should be mentioned that the "engien_path" field is only read from the |
@lumberyard-employee-dm I was actually just thinking of a common use-case where it might be good to support relative paths for the "engine_path" field in Thoughts? |
|
The problem with that approach is that an individual user would have no way to override the engine using the "engine" key without modifying the project.json file itself which is against the entire purpose of having a Let's say as a common team user, they would like to use "custom-user-o3de" via the |
I see what you mean. One option would be for them to unset the "engine_path" var by setting it to "" in their |
lemonade-dm
added
the
kind/roadmap
Summary:
A versioning scheme is needed to determine code compatibility between engines, projects and gems. Currently, only the engine has a version scheme tied to 6 month releases which is not granular enough.
Version and dependency information will be added inside
engine.json,project.jsonandgem.jsonand relevant tools will be updated to use this information. Version information will be incremented to indicate API or other relevant changes.What is the relevance of this feature?
Versioning and dependency information will allow users and tools to make informed decisions regarding compatibility. Given the distributed nature of O3DE, an established version and dependency scheme is the only scalable way to allow de-centralized control over compatibility.
Feature design description:
The following changes will be made
Semantic version fields in the format
<major>.<minor>.<patch>(e.g. 1.21.9) will be added inengine.json,project.jsonandgem.json<major>is for API-breaking changes<minor>is for non-API-breaking changes that add new APIs or change them in a non-breaking way<patch>is for all other non-API-breaking changes, usually important fixesDependency fields containing lists of dependencies in the format
<name><version specifier>(e.g. o3de >= 1.19.1) will be added inproject.jsonandgem.jsonwhere version specifiers are compatible with PEP 440 so we can use existing Python versioning functionality. If a version specifier is omitted for a gem dependency, the project's engine version will be used to determine the latest compatible gem version.*IMPORTANT* Version specifiers indicate "known compatibility". This means, there isn't a way with the proposed versioning system to specify "known incompatibility". We use the version specifiers to recommend the most compatible gem, but we warn users when they attempt to use a version that is not listed as compatible.CMake will make #defines available with version information for compile-time compatibility control. e.g.
EDITOR_VERSION_MAJOR,EDITOR_VERSION_MINOR, etc.The o3de.py CLI and Project Manager and CMake will take into account version and dependency specifications to display version information, determine compatibility and when a project needs to be re-compiled. The UX changes for these tools are not part of this RFC.
Workflows:
developmentbranch, thegem_version,engine_api_versionsandengine_versionwill be updated as important changes are made.engine_version. For example, if they change the minor version of a gem and zero out the patch version, they should also increase the minor version of theengine_versionand zero out the patch version.engine_version, developers will no longer need to manually update that field.stabilizationbranch is created in preparation for a release it should reflect version information that themainbranch will have whenstabilizationis merged tomain.engine_versionminor version should be immediately incremented in development after creating thestabilizationbranch so there is less time that the two branches share the sameengine_versionvalue.engine_display_versionin engine.json should be set to the appropriateYY.MM.XXrelease version prior to merging tomain, preferably the last change submitted to thestabilizationbranch before merging tomainto avoid accidentally merging this value back to development.stabilizationtodevelopmentbranches, developers should be careful not to bring theengine_display_versionorengine_versionfromstabilizationintodevelopment.developmenttostabilizationbranches, developers should be careful to only include appropriate version changes. For example, if the major gem version was bumped indevelopmentbut you're only merging over a patch fix for that gem into stabilization - do not bump the major gem version, just the patch.stabilizationbranch should not change because only bug fixes should be merged to the branch which should never result in more than the patch version changing.stabilizationis merged tomainit should have the correctengine_display_version.Technical design description:
The versioning system will be backward compatible and optional. If there is no versioning or dependency information available the system will fallback to not enforcing compatibility, but will warn the user.
JSON file changes
engine.jsonis modified so we have version fields for release and development and a newengine_api_versionsfield is added so gems can depend on specific versions of APIs inside the core engine.{ "engine_display_version":"22.05.1", // rename O3DEVersion field, set to 0.0.0 in development "engine_version":"1.0.1", // use engine_version for all compatibility checks "engine_api_versions": { // versions of general APIs provided by the core engine (not gems) "editor":"3.1.4", "framework":"2.0.0", "launcher":"1.0.0", "tools":"4.0.0" }, ... }project.jsonwill now include a project and engine version and fields for engine and gem dependencies. Also, acompatible_enginesfield is added as a simple way for project maintainers to indicate known good versions of the engine (and gems) their project is compatible with.{ "project_version":"1.0.0", // not needed for dependencies, but useful for users and added for consistency "engine_finder_cmake":"cmake/EngineFinder.cmake", // path to cmake script used to find the engine "engine":"o3de", // engine_name this project was registered with "engine_version":"10.1.4", // engine version this project was registered with "compatible_engines": [ // if empty (default) or missing, the project is assumed compatible with every engine "o3de>=1.0.0", // project is compatible with any o3de engine greater than or equal to version 1.0.0 "o3de-install==1.2.3" // project is ALSO compatible with the o3de-install engine version 1.2.3 ], "engine_api_dependencies": [ // declaration of dependency on engine api versions, defaults to empty "framework~=2.0.0" ], "gem_dependencies": [ // rename "gem_names" to "gem dependencies" and support optional version specifiers "example~=2.3", // project depends on example gem version 2.3.x "other==1.2.3", // project ALSO depends on other gem version 1.2.3 "lmbrcentral", // if no version specifier, use latest version compatible with project's engine ... ], ... }project.jsonanduser/project.json<project>/user/project.jsoncan be used to overrideproject.jsonsettings locally. These properties can be set usingo3de edit-project-properties --user. See O3DE CLI --user option changes for details.engine_finder_cmakewill be contain the relative path to the appropriate.cmakefile that will be used to find the engine for the project. Currently this file is hardcoded to becmake/EngineFinder.cmakebut we need the flexibility to easily update and revert this logic for future engines and projects.engine_pathcan be used to specify the path to the engine. The path may be local or relative if inuser/project.jsonbut may only be relative in the sharedproject.json. This field is provided for users to explicitly set the path to the engine, which is especially useful if they have multiple copies of an engine with the same name and version.gem.jsonwill now include a version field and fields for engine and gem dependencies. It will also have acompatible_enginesfield as a simple way for gem maintainers to indicate known good versions of the engine their gem is compatible with.{ "gem_version":"0.0.0", // default gem version is 0.0.0 "compatible_engines": [ // if empty (default) or missing, the gem is assumed compatible with every engine "o3de>=0.0.0" , // gem is compatible with any version of an engine named o3de "o3de-sdk>=2.0.1, <=3.1.0" // gem is ALSO compatible with o3de-sdk engine versions from 2.0.1 up to 3.1.0 ], "engine_api_dependencies": [ // optional declaration of dependency on engine api versions, default is empty "framework~=2.0.0" ], "gem_dependencies": [ // rename "dependencies" to "gem_dependencies" and add version specifiers "AWSCore>=1.0.0" // NEW version specifiers added ], ... }CLI tool changes
The
o3de.pyCLI tool will have the following updates:--forceparam can be used to force the action.project.jsongem_dependenciesfield using a version specifier of== <gem version>if the gem has version information and the user selects the option to use a specific gem versiono3de.pyCLI changes--useroption will be added toregister,edit-project-propertiesandenable-gem/disable-gemto use theuser/project.jsonand manipulate it. If--userCLI operations fail the command does not fall back to just use theproject.json.upgradecommand will be added that will be used to perform project upgrades. Initially this command will compare the project's engine version and the current engine version and execute Python commands to upgrade project files, outputting the list of files changed, where backups were stored, and letting the user know that, if they're using source control, they should check these files in.Project Manager changes
The Project Manager tool will have the following updates:
CMake changes
The following changes will be made to the CMake build scripts:
CMake scripts will be updated to surface the version information from each
.jsonfile as#definesthat can be used by code in one of the following formats:ENGINE_VERSION_<MAJOR/MINOR/PATCH>ENGINE_<engine API>_API_VERSION_<MAJOR/MINOR/PATCH><PROJECT/GEM>_<project name/gem name>_VERSION_<MAJOR/MINOR/PATCH>examplewith version1.2.3would have the following defines made available:frameworkwith version2.3.0would have the following defines:CMake scripts will be updated to take into account gem versions as well as gem names when determining correct sub-directories
CMakeList.txtin projects will be updated to read theproject.jsonand then theuser/project.jsonfor the "engine_finder_cmake" entry and theninclude()that path.EngineFinder.cmakefile will be updated to check project <-> engine compatibility.Content updates
1.0.01.0.0What are the advantages of the feature?
What are the disadvantages of the feature?
How will this be implemented or integrated into the O3DE environment?
This does not require any new libraries. It will expand on both C++ source code and o3de python CLI code.
Are there any alternatives to this feature?
Two other similar approaches were considered but not selected due to their reliance on git and not all customers will use git as source control.
Storing the version information in .h (header) files does not satisfy the requirement of being able to determine version information when the source isn't available.
GitHub Tags
Utilize the git tag system to mark commits that increment version.
Pros:
Cons:
Version Per Commit
Each commit is considered a separate incremental version.
Pros:
Cons:
For example, a customer would need to depend on
commit >=<sha1 hash>buto3de>=23e38520e9fcould be after commit hasho3de>=a086bcbe0How will users learn this feature?
Are there any open questions?
1.0.0?gem_dependencieswhen enabling a gem in a project?engine_version?engine_api_versionslist - are there better ones than those proposed?Examples
Project using latest version of gems for an engine
When you intend to use the latest versions of gems that are compatible with whatever engine your project uses, you would leave the version specifier portion of the
gem_depencenciesblank and provide anengine_nameandengine_version. This will likely be common for teams that use the pre-built SDK.The following example project will attempt to use the latest version of
PopcornFXandKytheraAIgems that are compatible with theo3de-sdkengine with version1.2.3, and because theLmbrCentralgem is provided with the engine, no version specifier is necessary.project.json{ "project_name":"Example", "engine_name":"o3de-sdk", "engine_version":"1.2.3", "gem_dependencies": [ "LmbrCentral", "PopcornFX", "KytheraAI", ], }Project requiring specific gem versions
Game studios with engineers dedicated to engine integrations may prefer to download all the necessary gems for a project and check them into source control and specify the exact version to use to discourage non-engineers from using other versions that may not be approved.
In this example, the project specifies a custom internal engine and version, and the exact versions of the
PopcornFXandKytheraAIgems to use. TheLmbrCentralstill does not need a version specifier because it is always provided with the engine, but the team could provide a version specifier if desired for consistency.project.json{ "project_name":"Example", "engine_name":"o3de-internal", "engine_version":"1.0.1", "gem_dependencies": [ "LmbrCentral", "PopcornFX==1.2.3", "KytheraAI==2.3.4", ], }Gem compatible with engine release versions
When you intend to provide a gem that only needs to be compatible with released engine versions you can specify those exact versions in the
compatible_enginesfield. You will not be able to use a range because that would include development engine versions.The following example declares this gem is compatible with engines version
1.2.3and2.3.4Gems\Example\1.2.3\gem.json{ "gem_name":"Example", "compatible_engines": [ "o3de-sdk == 1.2.3", "o3de-sdk == 2.3.4", "o3de == 1.2.3", "o3de == 2.3.4" ], }Gem depending on specific APIs
When you intend to provide a gem that should be compatible with any future engine so long as a core API doesn't change you can use the
engine_api_dependenciesandgem_dependenciesfields .The following example shows a gem that should be compatible with all versions of the engine containing the
frameworkAPI major version2Gems\Example\1.2.3\gem.json{ "gem_name":"Example", "engine_api_dependencies": [ "framework ~= 2.0.0" ], }The text was updated successfully, but these errors were encountered: