Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite ModelBindingsMapper to more efficiently handle parallel lists #170

Open
kevinstadler opened this issue Dec 11, 2024 · 0 comments
Open
Assignees
Labels
bug Something isn't working priority Needs urgent fix refactor Implementation changes that preserve external interfaces and the behavior of the software.

Comments

@kevinstadler
Copy link

kevinstadler commented Dec 11, 2024

Mapping this query result onto the following flat model takes the mapper ~20 minutes at 100% CPU, even though the 9840 query result lines actually all get mapped to a single result object (see below).

from typing import Annotated

from pydantic import AnyUrl, BaseModel, Field  # noqa: F401
from rdfproxy import ConfigDict, SPARQLBinding


class Person_PersonAppellationAssertion(BaseModel):
    model_config = ConfigDict(
        title="Name(s) in the sources",
        model_bool="id",
    )
    id: Annotated[
        AnyUrl | None, SPARQLBinding("person__person_appellation_assertion")
    ] = Field(default=None, exclude=True)
    person_appellation_is: Annotated[
        str | None,
        SPARQLBinding("person__person_appellation_assertion__person_appellation_is"),
    ] = None


class Person(BaseModel):
    model_config = ConfigDict(
        title="Person",
        model_bool="id",
        group_by="id",
    )
    id: Annotated[AnyUrl | None, SPARQLBinding("person")] = Field(
        default=None, exclude=True
    )
    person_id_assignment: Annotated[
        list[AnyUrl], SPARQLBinding("person__person_id_assignment")
    ]
    person_appellation_assertion: Annotated[
        list[Person_PersonAppellationAssertion],
        SPARQLBinding("person__person_appellation_assertion"),
    ]
    person_gender_assertion: Annotated[
        list[AnyUrl], SPARQLBinding("person__person_gender_assertion")
    ]
    person_ethnicity_assertion: Annotated[
        list[AnyUrl], SPARQLBinding("person__person_ethnicity_assertion")
    ]
    person_descriptive_name: Annotated[
        list[str], SPARQLBinding("person__person_descriptive_name")
    ]
    person_possession_assertion: Annotated[
        list[AnyUrl], SPARQLBinding("person__person_possession_assertion")
    ]
    person_status_assertion: Annotated[
        list[AnyUrl], SPARQLBinding("person__person_status_assertion")
    ]
    person_religion_assertion: Annotated[
        list[AnyUrl], SPARQLBinding("person__person_religion_assertion")
    ]
    person_occupation_assertion: Annotated[
        list[AnyUrl], SPARQLBinding("person__person_occupation_assertion")
    ]
    person_language_assertion: Annotated[
        list[AnyUrl], SPARQLBinding("person__person_language_assertion")
    ]
    person_kinship_assertion: Annotated[
        list[AnyUrl], SPARQLBinding("person__person_kinship_assertion")
    ]
    birth_circumstances_claim: Annotated[
        AnyUrl | None, SPARQLBinding("person__birth_circumstances_claim")
    ] = None
    person_death_assertion: Annotated[
        AnyUrl | None, SPARQLBinding("person__person_death_assertion")
    ] = None

The correct single result which is eventually returned (the underlying GraphDB query result has 9840 lines because it shows every possible permutation of the 41*8*6*5 contained list items):

{
  "items": [
    {
      "person_id_assignment": [
        "https://r11.eu/rdf/resource/5cc4b389-7235-4c53-8d12-3f64d109f374"
      ],
      "person_appellation_assertion": [
        {
          "person_appellation_is": "Ἄννα Κομνηνή"
        },
        {
          "person_appellation_is": "Komnenos"
        },
        {
          "person_appellation_is": "Κομνηνοῦ"
        },
        {
          "person_appellation_is": "Komnenos"
        },
        {
          "person_appellation_is": "Κομνηνοῦ"
        }
      ],
      "person_gender_assertion": [
        "https://r11.eu/rdf/resource/11511d7f-85de-45bb-95be-71147c7efdfa"
      ],
      "person_ethnicity_assertion": [],
      "person_descriptive_name": [
        "Anna Komnene, daughter of Alexios I and historian"
      ],
      "person_possession_assertion": [],
      "person_status_assertion": [
        "https://r11.eu/rdf/resource/313a689f-7f50-4046-9a72-0897f5f3475b",
        "https://r11.eu/rdf/resource/0ceb6cd4-aeae-46cf-9815-fc18fed4f5d9",
        "https://r11.eu/rdf/resource/12dbe1bf-76f5-43d5-aacb-d30d61d35fd4",
        "https://r11.eu/rdf/resource/c90895b0-9e7d-4bfe-ad61-9cde457a9234",
        "https://r11.eu/rdf/resource/6f1b93a1-e211-488a-80f7-49e93ad82364",
        "https://r11.eu/rdf/resource/f10b66a4-4463-4d48-ab45-8b15d30c59d0"
      ],
      "person_religion_assertion": [
        "https://r11.eu/rdf/resource/4c3310d4-c971-43d2-bf33-f7d448af5403"
      ],
      "person_occupation_assertion": [
        "https://r11.eu/rdf/resource/bce19957-5f5c-443d-8565-dd105e3c83d2",
        "https://r11.eu/rdf/resource/5d3d09e5-f8c4-4d7b-9f81-f8094d1e9cfd",
        "https://r11.eu/rdf/resource/333d0b6e-c429-40b4-82c7-60221394b87d",
        "https://r11.eu/rdf/resource/4b8cb787-005c-4727-9ede-98ee688127ff",
        "https://r11.eu/rdf/resource/4907b39b-54c5-42e2-8803-77877255d75a",
        "https://r11.eu/rdf/resource/e7a3c038-8cdd-4174-8d23-0ee1fb88a86d",
        "https://r11.eu/rdf/resource/5e85c552-a970-486b-ad0a-b2069cbde43d"
      ],
      "person_language_assertion": [],
      "person_kinship_assertion": [
        "https://r11.eu/rdf/resource/31baa2c0-9671-431a-af2f-2bcfc7c21918",
        "https://r11.eu/rdf/resource/c281d177-d43f-4234-bd26-6a111daa7c32",
        "https://r11.eu/rdf/resource/c7df7030-b0f2-4515-95ee-5176e9dcfe4b",
        "https://r11.eu/rdf/resource/3f77ce12-5bc6-49bd-9bf2-8362e8fa0aac",
        "https://r11.eu/rdf/resource/a0dce488-1700-452c-8238-50b238001d65",
        "https://r11.eu/rdf/resource/789de3c3-c593-4738-8c23-b8a44bb5fd21",
        "https://r11.eu/rdf/resource/d366bc1d-db29-4c70-91f3-4440f5ae6942",
        "https://r11.eu/rdf/resource/93bceed3-5813-461f-9d54-7f6aa8bf398f",
        "https://r11.eu/rdf/resource/0e19f63c-d447-4976-9fa4-f9a162586819",
        "https://r11.eu/rdf/resource/28aced50-bbc6-4f57-a702-c93e371afcd6",
        "https://r11.eu/rdf/resource/99c09a15-551c-4cf1-87ea-8e5f9821bddf",
        "https://r11.eu/rdf/resource/85e4377e-3f2a-4bd2-a29c-ffa5aaefb5f8",
        "https://r11.eu/rdf/resource/1f67d784-4908-4d1c-8b8f-bc5c37a3f426",
        "https://r11.eu/rdf/resource/76c145b7-e4d2-4044-bceb-c6bf8144577d",
        "https://r11.eu/rdf/resource/b033a046-98b2-448c-9ede-ce4867eaee53",
        "https://r11.eu/rdf/resource/cdb2d7e4-3796-445f-87f8-e586354d2e48",
        "https://r11.eu/rdf/resource/15f15a33-c66f-4b71-a352-92ac88ddc6fa",
        "https://r11.eu/rdf/resource/71de9652-80c2-4600-bddd-b1574bd005d1",
        "https://r11.eu/rdf/resource/0ec2f987-8e35-440d-a413-dbbba92decc4",
        "https://r11.eu/rdf/resource/e26ce0a4-2953-40ea-ac71-eeb73a66dba5",
        "https://r11.eu/rdf/resource/7d67a657-039d-4b56-923c-da816f3135b9",
        "https://r11.eu/rdf/resource/33193d5c-2c82-43bd-9f0b-fe841040d967",
        "https://r11.eu/rdf/resource/6b53e825-d94b-4899-b9fc-adf100909f44",
        "https://r11.eu/rdf/resource/c4f94d49-7464-4b5e-9f2d-eed5c882f82f",
        "https://r11.eu/rdf/resource/1e7aa33d-612f-4e2d-bd83-e9be8e80c8b5",
        "https://r11.eu/rdf/resource/fc5039a3-0050-4381-b737-41c531904022",
        "https://r11.eu/rdf/resource/fee3d0ac-d8ea-4ab9-815e-01f864009c34",
        "https://r11.eu/rdf/resource/d1e4510a-1faf-4c22-b9d1-5fd385163e20",
        "https://r11.eu/rdf/resource/8d98af5b-9947-462b-9d8d-3c81a674cf0f",
        "https://r11.eu/rdf/resource/6347fa5c-08bb-4e26-83ea-127704d32131",
        "https://r11.eu/rdf/resource/30694857-d6b7-48bb-b649-4319486b459a",
        "https://r11.eu/rdf/resource/52954b06-f672-40dc-ae24-48f64c4b6e86",
        "https://r11.eu/rdf/resource/18a1d4ea-3ea0-4fba-917f-900d53937327",
        "https://r11.eu/rdf/resource/256654d1-7117-4c60-8a93-48b7cded0e6e",
        "https://r11.eu/rdf/resource/f7c4e73d-d8c2-4cd5-b2a7-5f5d24d3038f",
        "https://r11.eu/rdf/resource/8c50834a-57c8-4cb5-9463-14559eaae6dd",
        "https://r11.eu/rdf/resource/58290a40-c49c-47ba-8eab-044bcc6233ee",
        "https://r11.eu/rdf/resource/dc70ecf8-8e87-4065-8eb7-d786fa1af5a8",
        "https://r11.eu/rdf/resource/ad36a4ba-f08f-40f3-9eae-77fc4b3c24a4",
        "https://r11.eu/rdf/resource/f74daa5a-d226-46cc-b289-61d02e3e7072",
        "https://r11.eu/rdf/resource/fca3c996-3ca9-4af8-ab9f-de6dc3cb7aed"
      ],
      "birth_circumstances_claim": "https://r11.eu/rdf/resource/65d74ec81e751",
      "person_death_assertion": "https://r11.eu/rdf/resource/022e98a6-b69d-4afc-9afd-47c920956f27"
    }
  ],
  "page": 1,
  "size": 10,
  "total": 1,
  "pages": 1
}
@kevinstadler kevinstadler added bug Something isn't working priority Needs urgent fix labels Dec 11, 2024
@lu-pl lu-pl self-assigned this Dec 17, 2024
@kevinstadler kevinstadler added this to the v0.2.0 release milestone Dec 18, 2024
@kevinstadler kevinstadler changed the title ModelBindingsMapper takes ages to process model with parallel lists Rewrite ModelBindingsMapper to more efficiently handle parallel lists Dec 18, 2024
@kevinstadler kevinstadler added the refactor Implementation changes that preserve external interfaces and the behavior of the software. label Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working priority Needs urgent fix refactor Implementation changes that preserve external interfaces and the behavior of the software.
Projects
None yet
Development

No branches or pull requests

2 participants