Performace optimization for project-saving

[nighca]

Possible approaches:

• Avoid unnecessary uploading (when no change of file content) for

  • code files, done in Loading optimization for files #508
  • config files

  for models (sprite/stage/...)

• Avoid parseProjectData in save (models/common/cloud.ts)

• Upload (local) file when added to project, instead of upload when saving project (Will revisit this after Automatically save project to cloud #541 is done)

• Save code content to DB instead of OSS?

[aofei]

To minimize the occurrence of file uploads to OSS, we will implement the following changes:

• For plain text files, we will no longer upload them to OSS. Instead, we will use Data URLs to store their contents, e.g., data:text/plain;charset=utf-8,Hello%2C%20world%21. Additionally, we need to impose some length restrictions on file contents. Thanks to the flexibility of Data URLs, we may consider extending this approach to other file types in the future.
• For files with no content, such as empty code files or resource files, we will use data:, as the file URL.
• For other files, we will adopt an "upload only upon addition" policy, as these files are unlikely to change once added.

To accelerate file uploads, we may consider the following strategies:

• Given that our objects' universal URLs contain the file hash, and assuming that identical hashes indicate identical content, we can store all obtained universal URLs locally. Before uploading any file, we will compute its hash locally and check the local storage for existing entries. If a match is found, we can reuse the existing file without uploading, reducing redundant uploads (though this kind of improvement may not be that significant).
• To prevent excessive local storage usage due to saving universal URLs, we can utilize an eviction algorithm, such as LRU.

[nighca]

Now we save universal URLs with file instances (in file meta). Assuming that we save small plain text files directly in field files of project / asset. Most of the "uploading-to-OSS" behaviors are already avoided.

Then the adoption of local storage for existing entries (let's call it "entry-map") will not be urgent.

These files will still be uploaded:

• Asset files generated by Buider (edited with sound-editor, image-editor, etc.)
• Asset files directly from user's local disk

For asset files generated in Builder, they are edited in Builder, so it's not likely for them to keep the same hash after editing and the entry-map will not help.

For asset files directly from user's local disk, it's small probability event for a user to add the same file to project more than once —— whose uploading will be skipped with entry-map.

So I think we do not need to implement entry-map this time. It may become important in some future day, we can do it then.

Or we may treat the "universal URL in file meta" as a simple version of "entry-map", which does not require cache elimination mechanism like LRU.