Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

platforms: cache results or reduce overheads #5242

Open
oliver-sanders opened this issue Nov 28, 2022 · 0 comments
Open

platforms: cache results or reduce overheads #5242

oliver-sanders opened this issue Nov 28, 2022 · 0 comments
Labels
efficiency For notable efficiency improvements
Milestone

Comments

@oliver-sanders
Copy link
Member

platform_from_name is fairly quick, ~1e-5 for me, however, it can get called a lot of times causing this to add up substantially.

Consider this example:

#!Jinja2

{% set N = 1000 %}

[scheduling]
    [[graph]]
        R1 = """
{% for i in range(N) %}
            x_{{ i }}
{% endfor %}
        """

[runtime]
{% for i in range(N) %}
    [[x_{{ i }}]]
{% endfor %}

There is only one platform, however, platform_from_name will be called at least 1000 times during validation.

Here's profiling results from a large parameterised workflow showing ~20s spent on platform_from_name during validation:

Screenshot from 2022-11-28 11-44-51

This time is almost all spent on deepcopy and regex stuff.

Pull requests welcome!

@oliver-sanders oliver-sanders added the efficiency For notable efficiency improvements label Nov 28, 2022
@oliver-sanders oliver-sanders added this to the some-day milestone Nov 28, 2022
@oliver-sanders oliver-sanders linked a pull request Nov 28, 2022 that will close this issue
8 tasks
@oliver-sanders oliver-sanders modified the milestones: some-day, 8.1.1, cylc-8.x Jan 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
efficiency For notable efficiency improvements
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant