-
-
Notifications
You must be signed in to change notification settings - Fork 403
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* Initial version with just a local cache and all still in version control * Additional cache for current versions * Move pattern matching into script * Remove built files and cache from git * Ignore patterns * Make sure taxonmies aren't loaded before they are built * Load taxonomies after build, but packaging still failing * Packaging taxonomy fixes for build * Trying to avoid circular references * Test data changes TBA * More check / test fixes * Make sure lang is always built before taxonomies * Create build_taxonomies_test target when main containers aren't available * Adding txt files back for analysis * Keep results files for analysis * Re-copied from latest main * Taxonomy rebuild copied from main * Example issue. After rebuild of taxonomies this change still wasn't picked up * Fixes so packaging can build * Lang fix * Fixes to ensure building in correct order * Fixing issues with building from scratch * Tidy up * Fix test results * Remove result.txt again * Include nutrient_levels in git in case taxonomies built before lang * Add build-cache submodule * More test fixes * Cache update * Updated to track head * Update cache reference * Use github for cache * Build traces from allergens rather than using symlink * Remove symnlinks for all and put logic in retrieve_tags_taxonomy * tidy fix * Remove module and use GitHub API to push * Perl tidy and removed excess logging * Reverting unecessary test expected result changes * Use taxonomy build cache in pull_request * Use taxonomy build cache in code_cov.yml * fix: typo * Addressing PR feedback * Typos and ensure cache is updated on build_lang * Perltidy fix * More PR feedback * Perltidy fix * Perl critic fix --------- Co-authored-by: Alex Garel <alex@garel.org>
- Loading branch information
Showing
91 changed files
with
1,826 additions
and
520,408 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
Cached copies of taxonomy build results are stored here. | ||
|
||
If no local cache is available then https://github.com/openfoodfacts/openfoodfacts-build-cache is checked for a copy. | ||
|
||
If the taxonomy needs to be built then this will be uploaded back to the repo if the GITHUB_TOKEN environment variable is set. | ||
|
||
The token is a personal access token, created here: https://github.com/settings/tokens. Only the public_repo scope is needed. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
Taxonomies have a significant impact on OFF processing and automated test results so need to be rebuilt before running any tests. However, this process takes some time, so the built taxonomy files are cached in a GitHub repository so that they only need to be rebuilt when there is a genuine change. | ||
|
||
# How it works | ||
A hash is calculated for all of the source files used to build a particular taxonomy and GitHub is then checked to see if a cache already exists for that hash. | ||
|
||
If no cached build is found then the taxonomy is rebuilt and cached locally. | ||
|
||
If the GITHUB_TOKEN environemnt variable is set then the cached build is also uploaded to the https://github.com/openfoodfacts/openfoodfacts-build-cache repository. Note that no token is required to download previous cached builds from the repo. | ||
|
||
# Obtaining a token | ||
|
||
The GITHUB_TOKEN is a personal access token, created here: https://github.com/settings/tokens. Only the public_repo scope is needed. | ||
|
||
# Considerations | ||
|
||
In maintianing this code be aware of the following complications... | ||
|
||
## Circular Dependencies | ||
|
||
There is a cicular dependency between taxonomies, languages and foods. The foods library is used to create the source for the nutrient_levels taxonomy, which uses transalations from languages. However, languages depends on the languages taxonomy... | ||
|
||
This is currently resolved by building the taxonomy on the fly if it is requested but not currently built. | ||
|
||
## Taxonomy Dependencies | ||
|
||
Some taxonomies perform lookups on others, e.g. additives_classes are referenced by additives, so the referenced taxonomy needs to be built first. The build order is determined in the Config_off.pm file. | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.