Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorproate linkml-runtime, linkml-model/linkml-files.py changes from linkml-runtime #201

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

sierra-moxon
Copy link
Member

@sierra-moxon sierra-moxon commented Jul 15, 2024

please see: linkml/linkml-runtime#320

back-porting changes in the runtime PR to linkml-model (current SoT for linkml-model/linkml_files.py methods).

this PR was a result of my dive into converting schemaview (linkml-runtime) tests to pytest when I ran into an error in linkml_files.py and test_linkml_files.py in linkml-runtime/linkml-model. It might be that we want to reconsider vendoring.

my biggest concern here with this PR, is that the tests for linkml-model and linkml-runtime that exercise linkml_files.py were different. If I put back in (as I did with this PR), the tests currently in linkml-model, there are slight changes in resulting expected URLs for model files - the new URLs do resolve, so I've adjusted the tests. We likely need @cmungall's review here before merge.

GITHUB_PATH_FOR(Source.META, Format.NATIVE_JSONLD, "v0.0.1")
current_loc = re.sub(r'linkml-model/[0-9a-f]*/', 'linkml-model/SHA/', GITHUB_PATH_FOR(Source.TYPES, Format.YAML))
assert "https://raw.githubusercontent.com/linkml/linkml-model/SHA/model/schema/types.yaml" == current_loc
assert 'https://raw.githubusercontent.com/linkml/linkml-model/missing_branch/linkml_model/owl/mappings.owl.ttl' == \
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: change in path here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the thing more worthy of fixing to me here is /SHA/model and missing_branch/linkml_model having different directory structure, but if the URLs resolve and they give us the thing we were after then they are the correct URLs in my book!

@sierra-moxon sierra-moxon marked this pull request as ready for review July 16, 2024 01:02
Copy link
Contributor

@sneakers-the-rat sneakers-the-rat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically i feel like i have already spent too much time for the three of us overengineering this thing, and am fine with whatever as long as the paths work ;) some minor suggestions on test semantics but take em or leave em

Comment on lines +152 to +176
def test_fixed_meta_url():
"""
One fixed canary value - the METAMODEL_URI as used in linkml main shouldn't change
"""
assert URL_FOR(Source.META, Format.YAML) == 'https://w3id.org/linkml/meta.yaml'
assert URL_FOR(Source.META, Format.JSONLD) == 'https://w3id.org/linkml/meta.context.jsonld'


root_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "linkml_model"))

SKIP_GITHUB_API = False # True means don't do the github API tests


class LinkMLFilesTestCase(unittest.TestCase):
""" Test that linkml_model/linkml_files.py work """
def test_basic_rules(self):
self.assertEqual("https://w3id.org/linkml/annotations.yaml",
URL_FOR(Source.ANNOTATIONS, Format.YAML))
self.assertEqual("https://w3id.org/linkml/meta.model.context.jsonld",
URL_FOR(Source.META, Format.NATIVE_JSONLD))
self.assertEqual(os.path.join(root_path, "model/schema/meta.yaml"),
LOCAL_PATH_FOR(Source.META, Format.YAML))
print(LOCAL_PATH_FOR(Source.META, Format.YAML))
self.assertEqual(os.path.join(root_path, "jsonld/types.model.context.jsonld"),
LOCAL_PATH_FOR(Source.TYPES, Format.NATIVE_JSONLD))
self.assertEqual("https://linkml.github.io/linkml-model/model/schema/meta.yaml",
GITHUB_IO_PATH_FOR(Source.META, Format.YAML))
self.assertEqual("https://linkml.github.io/linkml-model/jsonld/types.model.context.jsonld",
GITHUB_IO_PATH_FOR(Source.TYPES, Format.NATIVE_JSONLD))
self.assertEqual("https://raw.githubusercontent.com/linkml/linkml-model/main/jsonld/meta.model.context.jsonld",
GITHUB_PATH_FOR(Source.META, Format.NATIVE_JSONLD, ReleaseTag.LATEST))
self.assertEqual("https://raw.githubusercontent.com/linkml/linkml-model/testing_branch/owl/mappings.owl.ttl",
GITHUB_PATH_FOR(Source.MAPPINGS, Format.OWL, branch="testing_branch"))

@unittest.skipIf(SKIP_GITHUB_API, "Github API tests skipped")
def test_github_specific_rules(self):
"""
Test accesses that require github API to access

def test_basic_rules():
assert "https://w3id.org/linkml/annotations.yaml" == URL_FOR(Source.ANNOTATIONS, Format.YAML)
assert "https://w3id.org/linkml/meta.context.jsonld" == URL_FOR(Source.META, Format.NATIVE_JSONLD)
assert os.path.join(root_path, "model/schema/meta.yaml") == LOCAL_PATH_FOR(Source.META, Format.YAML)
print(LOCAL_PATH_FOR(Source.META, Format.YAML))
assert os.path.join(root_path, "jsonld/types.context.jsonld") == LOCAL_PATH_FOR(Source.TYPES, Format.NATIVE_JSONLD)
assert "https://linkml.github.io/linkml-model/latest/linkml_model/model/schema/meta.yaml" == GITHUB_IO_PATH_FOR(
Source.META, Format.YAML)
assert "https://linkml.github.io/linkml-model/latest/linkml_model/jsonld/types.context.jsonld" == \
GITHUB_IO_PATH_FOR(Source.TYPES, Format.NATIVE_JSONLD)
assert "https://raw.githubusercontent.com/linkml/linkml-model/main/linkml_model/jsonld/meta.context.jsonld" == \
GITHUB_PATH_FOR(Source.META, Format.NATIVE_JSONLD, ReleaseTag.LATEST)
assert "https://raw.githubusercontent.com/linkml/linkml-model/testing_branch/linkml_model/owl/mappings.owl.ttl" == \
GITHUB_PATH_FOR(Source.MAPPINGS, Format.OWL, branch="testing_branch")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_basic_rules is basically the same thing as test_fixed_meta_url - in general i only test fixed values when there is a specific reason for it to be fixed forever, so that's why i only tested those two URLs, but i think if we're going to test the rest of these then they should be in a test structure that reflects the meaning of the test :)

test each of these functions (URL_FOR, LOCAL_PATH_FOR, GITHUB_IO_PATH_FOR and GITHUB_PATH_FOR) separately, and some of what we want to test for might already be covered by other tests.

  • URL_FOR, LOCAL_PATH_FOR, and GITHUB_IO_PATH_FOR are trivial string formatting functions, so they can be covered by unit-style tests.
  • LOCAL_PATH_FOR has a test_local_paths test for existence for all the files in the expected formats already.
  • URL_FOR has test_url_for_format that tests the resolution of the generated URLs for the formats that have rewrite rules in the w3id repo.
  • GITHUB_PATH_FOR is substantially different than the others and should be tested separately for sure,

ideally these would have docstrings about why they should be fixed forever (if they were just fixed forever, why wouldn't we just hardcode the strings in the module itself and call it a day?), but idrc about that ;)

GITHUB_PATH_FOR(Source.MAPPINGS, Format.OWL, branch="testing_branch")


SKIP_GITHUB_API = False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally this would be a pytest option or dependent on some flag, if we want to keep skipping them - i marked them as skip because they kept failing bc the ratelimits on github API were very low, but if we want to rescue them then yes a flag that can be accessed would be good :)

GITHUB_PATH_FOR(Source.META, Format.NATIVE_JSONLD, "v0.0.1")
current_loc = re.sub(r'linkml-model/[0-9a-f]*/', 'linkml-model/SHA/', GITHUB_PATH_FOR(Source.TYPES, Format.YAML))
assert "https://raw.githubusercontent.com/linkml/linkml-model/SHA/model/schema/types.yaml" == current_loc
assert 'https://raw.githubusercontent.com/linkml/linkml-model/missing_branch/linkml_model/owl/mappings.owl.ttl' == \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the thing more worthy of fixing to me here is /SHA/model and missing_branch/linkml_model having different directory structure, but if the URLs resolve and they give us the thing we were after then they are the correct URLs in my book!

@@ -33,11 +33,13 @@ packages = [
[tool.poetry.dependencies]
python = "^3.9"
linkml-runtime = "^1.6.3"
rdflib = "^7.0.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oof if we are gaining rdflib as a runtime dependency just to use the Namespace class to declare a module constant i woudl call that highly undesirable and worthy of thinking about whether we need that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants