[v1.10.0-rc1] openstudio hung when I clicked Find Component (under the Components & Measures dropdown)

[jmarrec]

Issue overview

openstudio hung when I clicked Find Component (under the Components & Measures dropdown)

Reported by @James-py in #819 (comment)

Current Behavior

I can reproduce on Windows 11 and Ubuntu 24.04.

If I leave it running for a (long!!!!! like 15minutes) while it does eventually load properly...

Expected Behavior

Steps to Reproduce

Possible Solution

Details

Environment

Some additional details about your environment for this issue (if relevant):

• Platform (Operating system, version):
• Version of OpenStudioApplication (if using an intermediate build, include SHA):

Context

[jmarrec]

I'm 99% certain this is due to

• Issue #166 - Sort measures alphabetically in GUI #795

Yep. It's looking for

tid = 127 # == Construction Assembly
filterType = "components"

And due to #795 it's now trying to parse ALL results. In this specific case, there are 390 pages of responses (3891 lastTotalResults). Hence why it takes for ever.

Each call to remoteBCL.searchComponentLibrary(searchString, tid, currentPage) takes about 2 seconds on my machine. That means 390 * 2 / 60.0 = 13 min just for fetching.

[jmarrec]

@macumber @antonszilasi Do we want to address this or should I revert the PR for the time being?

• Issue #166 - Sort measures alphabetically in GUI #795

I suppose we could at least disable the pre-fetching & sorting for the components and keep them for the measures. Or maybe do something like if lastTotalResults > some value or numResultPages() > 10 or something.

[jmarrec]

In the meantime I can just use the API v2 with json and force it to load everything in one go, and while it's still too long, it only takes 9.54 s

r = requests.get('https://bcl.nrel.gov/api/search/*.json?fq=bundle:component&fq=component_tags:Construction%20Assembly&show_rows=100000')
r.raise_for_status()
data = r.json()
assert len(data['result']) == 3891

[jmarrec]

from tqdm.auto import tqdm
import time
import requests
from urllib.parse import quote_plus
import math
from concurrent.futures import ThreadPoolExecutor, as_completed
from enum import StrEnum

class Bundle(StrEnum):
    measure = "measure"
    component = "component"

def fetchAndSortResponses(bundle: Bundle, parentTaxonomy: str):

    # Initial request to get total page count
    rows = 50
    page = 0

    # URL template using encoded taxonomy
    url_template = (
        'https://bcl.nrel.gov/api/search/*.json'
        f'?fq=bundle:{bundle}'
        f'&fq=component_tags:{quote_plus(parentTaxonomy)}'
        '&show_rows={rows}&page={page}'
    )
    
    r = requests.get(url_template.format(rows=rows, page=page))
    r.raise_for_status()
    data = r.json()

    num_pages = math.ceil(data['complete_results_count'] / rows)
    all_results = data['result']  # Save first page results

    # Function to fetch one page
    def fetch_page(page_number):
        try:
            response = requests.get(url_template.format(rows=rows, page=page_number))
            response.raise_for_status()
            return response.json()['result']
        except Exception as e:
            print(f"Error fetching page {page_number}: {e}")
            return []

    # Parallel fetching of remaining pages
    with ThreadPoolExecutor(max_workers=min(num_pages, 128)) as executor:
        futures = [executor.submit(fetch_page, p) for p in range(1, num_pages)]
        with tqdm(total=len(futures), desc="Fetching pages") as pbar:
            for future in as_completed(futures):
                all_results.extend(future.result())
                pbar.update(1)

    print(f"Retrieved {len(all_results)} components")
    all_results = [result[bundle.name] for result in all_results]
    all_results.sort(key=lambda x: x['name'])
    return all_results

results = fetchAndSortResponses(bundle=Bundle.component, parentTaxonomy='Construction Assembly')

4.41s.

[macumber]

Yeah, let me try to do something like this, I don't want to have to back this out since it is one of Anton's first contributions. I'll get to it this weekend.

[macumber]

This does look like a big change so we may have to revert this out. @jmarrec do you have any access to the repository for the BCL itself? I am wondering if there are any undocumented options for sort_by or order_by? If we could simply request results from the BCL in sorted order that would save us from having to get all the results at once. Even knowing what query library is used in the API could give some hints for undocumented options

Something like: https://bcl.nrel.gov/api/search/?fq=component_tags:Construction%20Assembly&sort=Name.asc&show_rows=2

If that doesn't exist then maybe we re-open the original issue and point to a new enhancement on the BCL to provide a sort method.

[macumber]

Added BuildingComponentLibrary/bcl-manifest#47 to add sort_by to the BCL's public API