"In offline mode, executing BriefcaseConnection.openFile is very slow and causes the program to lag." #6592

jh9636 · 2024-04-07T12:39:58Z

jh9636
Apr 7, 2024

"After packaging an app with Electron, when offline, the first attempt to open a local BIM file using BriefcaseConnection.openFile is extremely slow and leads to program unresponsiveness. This issue does not occur when there is an internet connection."

I have used two different iTwin test programs, and both have encountered this issue.
The demo url:
AppUI Standalone Test App
Desktop Viewer React

This is a screenshot of the code causing the program to become unresponsive：

This is the source code：
LocalFileSupport.ts

/*---------------------------------------------------------------------------------------------
 * Copyright (c) Bentley Systems, Incorporated. All rights reserved.
 * See LICENSE.md in the project root for license terms and full copyright notice.
 *--------------------------------------------------------------------------------------------*/
import {
  BriefcaseConnection,
  IModelConnection,
  SnapshotConnection,
} from "@itwin/core-frontend";
import { SampleAppIModelApp } from "../index";
import { Logger, OpenMode } from "@itwin/core-bentley";
import { ElectronApp } from "@itwin/core-electron/lib/cjs/ElectronFrontend";

// cSpell:ignore TESTAPP FILEPATH

export class LocalFileSupport {
  public static localFilesSupported = (): boolean => {
    if (ElectronApp.isValid) return true;

    if (
      !SampleAppIModelApp.testAppConfiguration?.snapshotPath &&
      !SampleAppIModelApp.testAppConfiguration?.fullSnapshotPath
    ) {
      alert(
        "IMJS_UITESTAPP_SNAPSHOT_FILEPATH must be set on the backend and point to a folder containing local snapshot files."
      );
      return false;
    }

    return true;
  };

  public static getLocalFileSpecification = (): string | undefined => {
    return SampleAppIModelApp.testAppConfiguration?.fullSnapshotPath;
  };

  public static openLocalFile = async (
    fileSpec: string,
    definesFullPath = false
  ): Promise<IModelConnection | undefined> => {
    // Close the current iModelConnection
    await SampleAppIModelApp.closeCurrentIModel();

    let iModelConnection: IModelConnection | undefined;
    let filePath = "";

    // Open the iModel
    if (ElectronApp.isValid) {
      filePath = fileSpec;
      Logger.logInfo(
        SampleAppIModelApp.loggerCategory(LocalFileSupport),
        `openLocalFile: Opening standalone. path=${filePath}`
      );
      
      if (document.querySelector(".mask")) {
        const eleMask = document.querySelector(".mask");
        console.log(eleMask, "eleMask");
        (eleMask as any)!.style.display = "block";
      }
      try {
        iModelConnection = await BriefcaseConnection.openStandalone(
          filePath,
          OpenMode.Readonly,
          { key: filePath }
        );
      } catch (err: any) {
        Logger.logError(
          SampleAppIModelApp.loggerCategory(LocalFileSupport),
          `openLocalFile: BriefcaseConnection.openStandalone failed.`
        );
        alert(err.message);
        iModelConnection = undefined;
      }
    } else {
      if (
        SampleAppIModelApp.testAppConfiguration?.fullSnapshotPath &&
        definesFullPath
      )
        filePath = fileSpec;
      else
        filePath = `${SampleAppIModelApp.testAppConfiguration?.snapshotPath}/${fileSpec}`;
      Logger.logInfo(
        SampleAppIModelApp.loggerCategory(LocalFileSupport),
        `openLocalFile: Opening snapshot. path=${filePath}`
      );
      try {
        iModelConnection = await SnapshotConnection.openFile(filePath);
      } catch (err: any) {
        alert(err.message);
        iModelConnection = undefined;
      }
    }

    return iModelConnection;
  };
}

"I want to avoid encountering this issue when offline. I would greatly appreciate any helpful solutions you may have. Looking forward to your response."

Answered by jiaruiz717

Apr 17, 2024

@jh9636 We had a group discussion about it, this issue arises from cloudContainer.checkForChanges(); To get rid of it, we have the following suggestion:

Navigate to itwin/core-backend in your node_modules and find /lib/cjs/assets/Settings folder, if you use pnpm then the path may look something like \node_modules\.pnpm\@itwin+core-backend@4.4.6_@itwin+core-bentley@4.4.6_@itwin+core-common@4.4.6_@itwin+core-geometry@4.4.6\node_modules\@itwin\core-backend\lib\cjs\assets\Settings\backend.setting.json5
Find and open backend.setting.json5, add "syncOnConnect": false to cloud/containers"

cloud/containers": [
    {
      "name": "gcs/container",
      "baseUri": "https://geocoord-workspace.itw…

View full answer

pmconne · 2024-04-15T09:31:41Z

pmconne
Apr 15, 2024
Maintainer

Check for attempted network requests. Check browser console for errors.

1 reply

jh9636 Apr 19, 2024
Author

I'm very sorry for responding so late. Thank you very much for your advice.

MichaelBelousov · 2024-04-15T09:32:54Z

MichaelBelousov
Apr 15, 2024

Can you generate a CPU profile of your program to share, so you can confirm which function is taking so long and how long?

https://nodejs.org/en/learn/getting-started/profiling

6 replies

MichaelBelousov Apr 15, 2024

Yes, which takes 30 seconds, which a CPU profile would confirm

nick4598 Apr 15, 2024
Collaborator

@pmconne That was also my theory when I was discussing this with Dan late Friday last week. It seems very likely.

Assuming that is true,
I wonder whether we are actually able to use GCS data offline or not because I would expect this user to at least have some of those files locally at the time of noticing this issue. @jiaruiz717 will start taking a look at this today.

kabentley Apr 15, 2024

We shouldn't load any GCS data synchronously during iModel opening. All we do is open the container and read the names of the dbs. We then start a prefetch on the "base" WorkspaceDb. That runs in another thread and we don't await it. Of course if there is no network connection it's possible we're waiting for a network timeout to attempt to open the container.

If this is Electron, it shouldn't even have any network activity after the first run,

nick4598 Apr 16, 2024
Collaborator

@jiaruiz717 was able to provide me with two log files, one with internet and one without internet

With Internet

Info | CloudSqlite | 14:16:54.894 | r1 [] GET https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?...

Info | CloudSqlite | 14:16:54.926 | r1 [] [32ms] (http=200) (rc=0) HTTP/1.1 200 OK

Info | CloudSqlite | 14:16:54.927 | container gcs leaves POLL state

Info | GeoCoord | loaded gcsDb "base:1.0.2", from "https://geocoord-workspace.itwinjs.org/gcs" size=7, local=0

Info | GeoCoord | loaded gcsDb "allEarth:1.0.2", from "https://geocoord-workspace.itwinjs.org/gcs" size=166, local=0

Info | CloudSqlite | 14:16:58.106 | r2 [prefetch 0 of base:1.0.2] GET https://geocoord-workspace.itwinjs.org/gcs/6FC56086CCEFE209214A71C771F3933889619C52037C61EF.bcv?...

Info | CloudSqlite | 14:16:58.106 | r3 [prefetch 1 of base:1.0.2] GET https://geocoord-workspace.itwinjs.org/gcs/C58F6B3EABBFA1A7C332BCC7948417B8400E0CFC7A2A243C.bcv?...

Info | CloudSqlite | 14:16:58.106 | r4 [prefetch 2 of base:1.0.2] GET https://geocoord-workspace.itwinjs.org/gcs/A4E66AFACC494100B38D8265FA15895D8EA3D80E9E3E0724.bcv?...

Info | CloudSqlite | 14:16:58.213 | r5 [prefetch 3 of base:1.0.2] GET https://geocoord-workspace.itwinjs.org/gcs/E672E16EB5BEB3E5264A736F1E6CC2D41CE1316F2F947788.bcv?...

Info | CloudSqlite | 14:16:58.213 | r6 [prefetch 4 of base:1.0.2] GET https://geocoord-workspace.itwinjs.org/gcs/F8262DB2C305FEB98319A9139EBD104F8224D8B4E7972505.bcv?...

Info | CloudSqlite | 14:16:58.213 | r7 [prefetch 5 of base:1.0.2] GET https://geocoord-workspace.itwinjs.org/gcs/3FE490E3E07C62615D1F8C058CC4B821C5748763052DDBC5.bcv?...

Info | CloudSqlite | 14:16:58.426 | r2 [prefetch 0 of base:1.0.2] [320ms] (http=200) (rc=0) HTTP/1.1 200 OK

Info | CloudSqlite | 14:16:58.434 | r8 [prefetch 6 of base:1.0.2] GET https://geocoord-workspace.itwinjs.org/gcs/6BB06F9C41157680A4852FAEA0D0BC6E39E38718F9AEFD5F.bcv?...

Info | CloudSqlite | 14:16:58.594 | r8 [prefetch 6 of base:1.0.2] [160ms] (http=200) (rc=0) HTTP/1.1 200 OK

Info | CloudSqlite | 14:16:58.770 | r4 [prefetch 2 of base:1.0.2] [664ms] (http=200) (rc=0) HTTP/1.1 200 OK

Info | CloudSqlite | 14:16:58.781 | r3 [prefetch 1 of base:1.0.2] [675ms] (http=200) (rc=0) HTTP/1.1 200 OK

Info | CloudSqlite | 14:16:58.845 | r5 [prefetch 3 of base:1.0.2] [632ms] (http=200) (rc=0) HTTP/1.1 200 OK

Info | CloudSqlite | 14:16:58.851 | r6 [prefetch 4 of base:1.0.2] [638ms] (http=200) (rc=0) HTTP/1.1 200 OK

Info | CloudSqlite | 14:16:58.954 | r7 [prefetch 5 of base:1.0.2] [741ms] (http=200) (rc=0) HTTP/1.1 200 OK

Info | CloudSqlite | 14:16:58.959 | destroying BCV connection 1...

Without Internet

Info | CloudSqlite | 14:14:15.334 | invoking xAuth(azure?customuri=1&sas=1, https://geocoord-workspace.itwinjs.org, gcs)

Info | CloudSqlite | 14:14:15.336 | r0 [] GET https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?...

Info | CloudSqlite | 14:14:15.350 | r0 [] [14ms] (http=0) (rc=1) Couldn't resolve host name (retry in 0ms)

Info | CloudSqlite | 14:14:15.350 | r1 [] GET (nRetry=1) https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?...

Info | CloudSqlite | 14:14:15.353 | r1 [] [3ms] (http=0) (rc=1) Couldn't resolve host name (retry in 500ms)

Info | CloudSqlite | 14:14:15.854 | r2 [] GET (nRetry=2) https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?...

Info | CloudSqlite | 14:14:15.857 | r2 [] [3ms] (http=0) (rc=1) Couldn't resolve host name (retry in 1000ms)

Info | CloudSqlite | 14:14:16.858 | r3 [] GET (nRetry=3) https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?...

Info | CloudSqlite | 14:14:16.860 | r3 [] [2ms] (http=0) (rc=1) Couldn't resolve host name (retry in 2000ms)

Info | CloudSqlite | 14:14:18.862 | r4 [] GET (nRetry=4) https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?...

Info | CloudSqlite | 14:14:18.865 | r4 [] [3ms] (http=0) (rc=1) Couldn't resolve host name (retry in 4000ms)

Info | CloudSqlite | 14:14:22.866 | r5 [] GET (nRetry=5) https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?...

Info | CloudSqlite | 14:14:22.868 | r5 [] [2ms] (http=0) (rc=1) Couldn't resolve host name (retry in 8000ms)

Info | CloudSqlite | 14:14:30.869 | r6 [] GET (nRetry=6) https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?...

Info | CloudSqlite | 14:14:30.871 | r6 [] [2ms] (http=0) (rc=1) Couldn't resolve host name

Error | GeoCoord | Cannot load GCS workspace (1): Error: attach error: Couldn't resolve host name,container=https://geocoord-workspace.itwinjs.org/gcs, storage=azure, public=true, cacheDir=
Error | GeoCoord | could not load gcs database "gcs/entire-earth"

Afaik, the rootDir of the cachedirs were the same between the two runs since it was the same test and he validated that the cachedir still had content in it and was not cleaned up between the runs. We both read through the code and it looks like manifest fetching always takes place when the cache/container objects are setup/ connected for the first time in that process.

Someone would have to run their app with internet connection then shut it down after the cache/ container object is setup as its currently written as far as we can both tell. I could be missing something since the comment above 'connect' in CloudSqlite.ts says:

If the CloudCache was previously populated, this method may be called and will succeed even when offline or without a valid accessToken.

@kabentley regarding the above comment, do you know how to opt into this offline mode? Or is this only possible when the process starts off with internet connection?

jh9636 Apr 19, 2024
Author

I'm very sorry for the late reply. Thank you very much for your suggestions and help.

kabentley · 2024-04-16T13:22:52Z

kabentley
Apr 16, 2024

I now see that I added a syncOnConnect option to opening workspaces that defaults to true. That's so we always check to see if there is a new version available. That's what's causing the attempt to load the manifest.

Can you try adding in backend.setting.json5:

  "workspace/databases": [
    {
      "name": "gcs/base",
      "dbName": "base",
      "containerName": "gcs/container",
      "version": "^1",
      "priority": 10000,
      "prefetch": true,
      **"syncOnConnect": false**
    },
    {
      "name": "gcs/entire-earth",
      "dbName": "allEarth",
      "containerName": "gcs/container",
      "version": "^1",
      "priority": 100,
      **"syncOnConnect": false**
    },

and see if it works?

We'll have to give some way to tell when we're offline so we can avoid things like this.

1 reply

jh9636 Apr 19, 2024
Author

I'm very sorry for responding so late. Thank you very much for your advice.

jiaruiz717 · 2024-04-16T17:23:44Z

jiaruiz717
Apr 16, 2024
Collaborator

Adding syncOnConnect:false to backend.setting.json5 does not help to get rid of cloudContainer.checkForChanges() @nick4598 and I both tried.

Here's the log with Internet

Logs with Internet

Info | CloudSqlite | 11:54:45.917 | invoking xAuth(azure?customuri=1&sas=1, https://geocoord-workspace.itwinjs.org, gcs)

Info | CloudSqlite | 11:54:45.921 | r0 [] GET https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?... \

Info | CloudSqlite | 11:54:46.211 | r0 [] [290ms] (http=200) (rc=0) HTTP/1.1 200 OK

Trace | CloudSqlite | 11:54:50.299 | poll

Info | CloudSqlite | 11:54:50.299 | container gcs enters POLL state

Info | CloudSqlite | 11:54:50.300 | r1 [] GET https://geocoord-workspace.itwinjs.org/gcs/manifest.bcv?...

Info | CloudSqlite | 11:54:50.363 | r1 [] [63ms] (http=200) (rc=0) HTTP/1.1 200 OK

Info | CloudSqlite | 11:54:50.364 | container gcs leaves POLL state

According to the logs, there are two http calls to fetch the manifest file, the first call happens at cloudContainer.connect(this.workspace.getCloudCache()), the second call happens at cloudContainer.checkForChanges()
I also commented out cloudContainer.checkForChanges(), then there's only one http call to fetch the manifest file.

Further dive into cloudContainer.connect(this.workspace.getCloudCache()) on the native side, the http call happens at: rc = bcvContainerOpen(pCont->zStorage, pCont->zAccount, zAuth, pCont->zContainer, &pWrapper->pBcv, &zErr); and rc = bcvfsFetchManifest(pDisp, pBcv, &pMan, &zErr); in blockcachevfs.c without checking if the manifest exists or not.

We came up with several options

IModelHost.startup could set a property for internet access IModelHost.hasInternet = xxxxx, pass that value to the connect function. Then the SQLite team can support an offline mode based on the value we pass to connect. Here's a potential package we can use to check network availability: is-online
If it's easier to check if network is available on the native side. I'd like to keep the connect function unchanged. SQLite team can do it entirely on the native side, checking network availability, checking if manifest file exists, etc.

Any thoughts on the above options?

0 replies

kabentley · 2024-04-16T18:33:00Z

kabentley
Apr 16, 2024

can you please provide a calltrace of how we get to fecthManifest?

1 reply

jh9636 Apr 19, 2024
Author

I'm very sorry for responding so late. Thank you very much for your advice.

jiaruiz717 · 2024-04-16T19:17:10Z

jiaruiz717
Apr 16, 2024
Collaborator

Sure. Here's a calltrace to fetchManifest while connecting a cloud container to local cache

open an iModel SnapshotDb.openFile(testBimFileName);
load gcs data GeoCoordConfig.loadForImodel(this.workspace.settings); in IModeldb.ts
get container const container = ws.getContainer(containerProps); in GeoCoordConfig.ts
if the container in step 3 does not exist, add container workspace.addContainer(this); and connect the container to cloudCache cloudContainer.connect(this.workspace.getCloudCache()); in Workspace.ts
the connect function calls the native code CloudResult CloudContainer::Connect(CloudCache& cache) {} return sqlite3_bcvfs_attach(cache.m_vfs, GetOpenParams().c_str(), m_baseUri.c_str(), m_containerId.c_str(), m_alias.c_str(), attachFlags, msg);in CloudSqlite.cpp
sqlite3_bcvfs_attach calls rc = bcvfsAttach(pFs, zStorage, zAccount, zCont, zAlias, flags, &zErr); in blockcachevfs.c
bcvfsAttach calls rc = bcvfsFetchManifest(pDisp, pBcv, &pMan, &zErr); to fetch the manifest file

0 replies

jiaruiz717 · 2024-04-17T17:56:16Z

jiaruiz717
Apr 17, 2024
Collaborator

@jh9636 We had a group discussion about it, this issue arises from cloudContainer.checkForChanges(); To get rid of it, we have the following suggestion:

Navigate to itwin/core-backend in your node_modules and find /lib/cjs/assets/Settings folder, if you use pnpm then the path may look something like \node_modules\.pnpm\@itwin+core-backend@4.4.6_@itwin+core-bentley@4.4.6_@itwin+core-common@4.4.6_@itwin+core-geometry@4.4.6\node_modules\@itwin\core-backend\lib\cjs\assets\Settings\backend.setting.json5
Find and open backend.setting.json5, add "syncOnConnect": false to cloud/containers"

cloud/containers": [
    {
      "name": "gcs/container",
      "baseUri": "https://geocoord-workspace.itwinjs.org",
      "storageType": "azure",
      "containerId": "gcs",
      "isPublic": true,
      "syncOnConnect": false
    }
  ],

cloudContainer.checkForChanges() triggers a http call which has multiple retries(0+500+1000+2000+4000+8000=15500ms). We avoid it by adding "syncOnConnect": false. Opening a local BIM file will not encounter issues as long as you have cached GCS files in the cloud cache folder. But here are several things to consider:

If a local BIM file has never been opened before, opening it without Internet will still encounter this issue because we don't have the necessary cached GCS files in the cloud cache folder
When you replicate workflows that were previously done with Internet access in an offline environment, you are likely to operate without issues. However, please be aware that since GCS files are downloaded on demand, you could theoretically encounter a scenario where you need some GCS data that you do not have, leading to errors. However, the GCS workspaces are relatively stable and the only time GCS workspaces are updated is to add new coordinate systems which your current iModels would likely never need.

2 replies

jh9636 Apr 19, 2024
Author

You are right; the issue has indeed been resolved. I'm very grateful for the help you and your team have provided.

JoanaSmit Aug 14, 2024
Collaborator

@jiaruiz717 Is there a way to disable syncOnConnect when calling BriefcaseConnection.openFile without editing node_modules?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"In offline mode, executing BriefcaseConnection.openFile is very slow and causes the program to lag." #6592

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 11 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

"In offline mode, executing BriefcaseConnection.openFile is very slow and causes the program to lag." #6592

jh9636 Apr 7, 2024

Replies: 7 comments · 11 replies

pmconne Apr 15, 2024 Maintainer

jh9636 Apr 19, 2024 Author

MichaelBelousov Apr 15, 2024

MichaelBelousov Apr 15, 2024

nick4598 Apr 15, 2024 Collaborator

kabentley Apr 15, 2024

nick4598 Apr 16, 2024 Collaborator

jh9636 Apr 19, 2024 Author

kabentley Apr 16, 2024

jh9636 Apr 19, 2024 Author

jiaruiz717 Apr 16, 2024 Collaborator

kabentley Apr 16, 2024

jh9636 Apr 19, 2024 Author

jiaruiz717 Apr 16, 2024 Collaborator

jiaruiz717 Apr 17, 2024 Collaborator

jh9636 Apr 19, 2024 Author

JoanaSmit Aug 14, 2024 Collaborator

jh9636
Apr 7, 2024

Replies: 7 comments 11 replies

pmconne
Apr 15, 2024
Maintainer

jh9636 Apr 19, 2024
Author

MichaelBelousov
Apr 15, 2024

nick4598 Apr 15, 2024
Collaborator

nick4598 Apr 16, 2024
Collaborator

jh9636 Apr 19, 2024
Author

kabentley
Apr 16, 2024

jh9636 Apr 19, 2024
Author

jiaruiz717
Apr 16, 2024
Collaborator

kabentley
Apr 16, 2024

jh9636 Apr 19, 2024
Author

jiaruiz717
Apr 16, 2024
Collaborator

jiaruiz717
Apr 17, 2024
Collaborator

jh9636 Apr 19, 2024
Author

JoanaSmit Aug 14, 2024
Collaborator