Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: theme push is very slow in versions with the new colorful upload UI #4210

Open
2 tasks done
BearKid opened this issue Jul 16, 2024 · 11 comments
Open
2 tasks done
Labels
Area: @shopify/theme @shopify/theme package issues moderate lowest severity for theme related bug Type: Bug Something isn't working

Comments

@BearKid
Copy link

BearKid commented Jul 16, 2024

Please confirm that you have:

  • Searched existing issues to see if your issue is a duplicate. (If you’ve found a duplicate issue, feel free to add additional information in a comment on it.)
  • Reproduced the issue in the latest CLI version.

In which of these areas are you experiencing a problem?

Theme

Expected behavior

I expect shopify theme push -d in @shopify/cli@3.64.0 should complete in no more than 5s, should be same as @shopify/cli@3.47.5.

Actual behavior

I found that v3.56.x ~ v3.64.0 (the latest version at this moment) is very slow when uploading the theme files using shopify theme push -d.

  • Taking about 15s ~ 2minutes for a simple theme in development store that initially created by shopify.
  • Taking serveral minutes to upload my formal theme(not a big project) in my prod store.

I tested cli 3.47.x ~ 3.53.x, shopify theme push -d is far more faster than the new versions, taking no more than 5s.

Intuitively, the versions with the colorful upload progress bar are slow, and the versions with the old style upload progress bar are faster.

  • new UI
    image
  • old UI
    img_v3_02cr_6df5956c-2eda-4594-a6d9-0bff6068e9fg

Below is a more detail table
image

As you can see by the table above, my new PC doesn't run the old version of the CLI properly and I have to use the new version, so I'm desperate to find a solution to the slow uploads.

Verbose output

...

2024-07-16T04:10:29.212Z: Request to https://xxx-dev.myshopify.com/admin/api/unstable/themes/143758819578/assets/bulk.json completed in 1003 ms
With response headers:
 - content-type: application/json; charset=utf-8
 - x-request-id: 6e88e3c6-713f-48fa-9b81-9b2941e0db9f-1721103028

2024-07-16T04:10:29.214Z: File Upload Results:
-snippets/preorder-now-settings-1.liquid: success
-snippets/preorder-now.liquid: success
-snippets/price.liquid: success
-snippets/signup_event_fire.liquid: success
-templates/page.bss-b2b-wholesaler-18046.liquid: success
-templates/search.bss.b2b.liquid: success
-templates/search.preorder-now-search.liquid: success
-templates/customers/account.liquid: success
-templates/customers/register.liquid: success
2024-07-16T04:10:29.221Z: Uploading the following files:                                                                                                                                                                                                                                                            
-assets/pandectes-settings.json                                                                                                                                                                                                                                                                                     
-templates/product.json
2024-07-16T04:10:30.167Z: Request to https://xxx-dev.myshopify.com/admin/api/unstable/themes/143758819578/assets/bulk.json completed in 943 ms
With response headers:
 - content-type: application/json; charset=utf-8
 - x-request-id: ebed7cfc-672b-470e-8d97-b2623aef3cd1-1721103029

2024-07-16T04:10:30.169Z: File Upload Results:                                                                                                                                                                                                                                                                      
-assets/pandectes-settings.json: success                                                                                                                                                                                                                                                                            
-templates/product.json: success
2024-07-16T04:10:30.173Z: Uploading the following files:                                                                                                                                                                                                                                                            
-config/settings_schema.json                                                                                                                                                                                                                                                                                        
2024-07-16T04:10:30.898Z: Request to https://xxx-dev.myshopify.com/admin/api/unstable/themes/143758819578/assets/bulk.json completed in 723 ms
With response headers:
 - content-type: application/json; charset=utf-8
 - x-request-id: 29bb82c3-9245-47ec-afe3-f1ed7550b4c1-1721103030

2024-07-16T04:10:30.902Z: File Upload Results:                                                                                                                                                                                                                                                                      
-config/settings_schema.json: success                                                                                                                                                                                                                                                                               
2024-07-16T04:10:30.906Z: Uploading the following files:
-assets/appointly-main.js
2024-07-16T04:10:31.902Z: Request to https://xxx-dev.myshopify.com/admin/api/unstable/themes/143758819578/assets/bulk.json completed in 993 ms
With response headers:
 - content-type: application/json; charset=utf-8
 - x-request-id: cbd5f66c-639c-4380-90ba-8951f36c7c67-1721103030

2024-07-16T04:10:31.906Z: File Upload Results:                                                                                                                                                                                                                                                                      
-assets/appointly-main.js: success                                                                                                                                                                                                                                                                                  
2024-07-16T04:10:31.910Z: Uploading the following files:
-assets/bss-b2b-hook.js
-assets/bss-b2b-js.js
2024-07-16T04:10:35.136Z: Request to https://xxx-dev.myshopify.com/admin/api/unstable/themes/143758819578/assets/bulk.json completed in 3218 ms
With response headers:
 - content-type: application/json; charset=utf-8
 - x-request-id: b970d93d-6245-45e8-a4dc-525717de3278-1721103031

2024-07-16T04:10:35.148Z: File Upload Results:                                                                                                                                                                                                                                                                      
-assets/bss-b2b-hook.js: success                                                                                                                                                                                                                                                                                    
-assets/bss-b2b-js.js: success
2024-07-16T04:10:35.150Z: Uploading the following files:
-assets/bss-b2b-state.js
2024-07-16T04:10:36.371Z: Request to https://xxx-dev.myshopify.com/admin/api/unstable/themes/143758819578/assets/bulk.json completed in 1215 ms
With response headers:
 - content-type: application/json; charset=utf-8
 - x-request-id: 8ecbbb1f-dde9-43b1-8cab-fc2b1971b6b3-1721103035

2024-07-16T04:10:36.372Z: File Upload Results:                                                                                                                                                                                                                                                                      
-assets/bss-b2b-state.js: success

...

 Analytics event sent: {
  "command": "theme push",
  "time_start": 1721109452872,
  "time_end": 1721109601617,
  "total_time": 148745,
  "success": true,
  "cli_version": "3.64.0",
  "ruby_version": "3.3.4",
  "node_version": "22.4.1",
  "is_employee": false,
  "uname": "windows amd64",
  "env_ci": false,
  "env_plugin_installed_any_custom": false,
  "env_plugin_installed_shopify": "[\"@shopify/cli\"]",
  "env_shell": "cmd.exe",
  "env_device_id": "59e44e9b47f02007da55a01933fdbb74b0d207e4",
  "env_cloud": "localhost",
  "env_package_manager": "pnpm",
  "env_is_global": true,
  "cmd_app_warning_api_key_deprecation_displayed": false,
  "cmd_all_timing_network_ms": 144150,
  "cmd_all_timing_prompts_ms": 0,
  "cmd_all_launcher": "unknown",
  "cmd_all_topic": "theme",
  "cmd_all_plugin": "@shopify/cli",
  "cmd_all_force": false,
  "cmd_all_verbose": true,
  "cmd_all_path_override": true,
  "cmd_all_path_override_hash": "e23c73e2fe642a39a0a3686096356ed8af4ac0a5",
  "cmd_all_timing_active_ms": 4594,
  "cmd_all_exit": "ok",
  "args": "-d --store {mystore_name}.myshopify.com --verbose",
  "env_plugin_installed_all": "[\"@shopify/cli\"]",
  "metadata": "{\"extraPublic\":{},\"extraSensitive\":{}}"
}

Reproduction steps

  1. follow doc to Install Ruby+Devkit 3.x using https://rubyinstaller.org/downloads/
  2. pnpm i -g @shopify/cli@3.64.0 or npm i -g @shopify/cli@3.64.0
  3. shopify theme push -d --store xxx.myshopify.com

Operating System

Windows 10

Shopify CLI version (check your project's package.json if you're not sure)

3.64.0

Shell

Powersell

Node version (run node -v if you're not sure)

v22.4.1

What language and version are you using in your application?

No response

@BearKid BearKid added the Type: Bug Something isn't working label Jul 16, 2024
@BearKid
Copy link
Author

BearKid commented Jul 16, 2024

  • project 1 is about 23MB
  • project 2 is about 5MB

both my old PC and new PC are under the same network environment and using the proxy software Clash with TUN mode enabled.

@lucyxiang lucyxiang added Area: @shopify/theme @shopify/theme package issues moderate lowest severity for theme related bug labels Jul 16, 2024
@lucyxiang
Copy link
Contributor

Hey @BearKid, are you deleting your theme between these theme push calls? With each subsequent theme push, we only push the diffs so the initial call is the slowest.

You can test the TypeScript implementation (new colorful UI) vs Ruby implementation (what older CLI versions use) with the --stable flag (theme push -d --stable). --stable will run the old Ruby implementation while without it will run the new TypeScript implementation.

@BearKid
Copy link
Author

BearKid commented Jul 18, 2024

@lucyxiang thx for your reply.
It seems that the key point of the issue is that the cli unexpectedly uploaded some files that hadn't been changed.
Is it possible that cli diff has a problem with the handling of line breaks?

Hey @BearKid, are you deleting your theme between these theme push calls? With each subsequent theme push, we only push the diffs so the initial call is the slowest.

  1. No, I repeatly tested theme push -d without delete anything.
  2. I re-tested it in Project 1(23MB) today and unexpectedly found it to be a bit better than the previous tests (30s vs. 2minutes). Prior to this test, there was some changes to the environment: I realized that theme dev was running with an error, and based on this issue, I reinstalled Ruby + Devkit, downgrading its version from 3.3.4 to 3.1.2.
  3. In my case, the cli did not "only push the diffs". with --verbose, I found that the cli neither push all files or only push the diffs, it just picked some files to push.
  4. During the execution of the push command, the bulk api is called many times, some calls upload only one file, some calls upload only 2 files, and some upload several files. Below is part of the log:
2024-07-18T00:44:31.953Z: Uploading the following files:                                                                                                                                                                                                                                                            
-sections/pf-d37933f8.liquid                                                                                                                                                                                                                                                                                        
-sections/pf-d7f4e5c1.liquid
2024-07-18T00:44:33.798Z: Request to https://xxx.myshopify.com/admin/api/unstable/themes/170038395148/assets/bulk.json completed in 1839 ms
With response headers:
 - content-type: application/json; charset=utf-8
 - x-request-id: 04995c0c-c604-4e82-b7c3-a58d5f32e2b7-1721263472

2024-07-18T00:44:33.826Z: Uploading the following files:                                                                                                                                                                                                                                                            
-sections/pf-f330db7b.liquid                                                                                                                                                                                                                                                                                        
2024-07-18T00:44:35.068Z: Request to https://xxx.myshopify.com/admin/api/unstable/themes/170038395148/assets/bulk.json completed in 1222 ms
With response headers:
 - content-type: application/json; charset=utf-8
 - x-request-id: 1d0f47f1-3f19-4506-b36d-d7fa604de9bb-1721263474

2024-07-18T00:44:45.087Z: Analytics event sent: {
  "command": "theme push",
  "time_start": 1721263452745,
  "time_end": 1721263484353,
  "total_time": 31608,
  "success": true,
  "cli_version": "3.64.0",
  "ruby_version": "3.1.2",
  "node_version": "22.4.1",
  "is_employee": false,
  "uname": "windows amd64",
  "env_ci": false,
  "env_plugin_installed_any_custom": false,
  "env_plugin_installed_shopify": "[\"@shopify/cli\"]",
  "env_shell": "cmd.exe",
  "env_device_id": "59e44e9b47f02007da55a01933fdbb74b0d207e4",
  "env_cloud": "localhost",
  "env_package_manager": "pnpm",
  "env_is_global": true,
  "cmd_app_warning_api_key_deprecation_displayed": false,
  "cmd_all_timing_network_ms": 30398,
  "cmd_all_timing_prompts_ms": 0,
  "cmd_all_launcher": "unknown",
  "cmd_all_topic": "theme",
  "cmd_all_plugin": "@shopify/cli",
  "cmd_all_force": false,
  "cmd_all_verbose": true,
  "cmd_all_path_override": true,
  "cmd_all_path_override_hash": "e23c73e2fe642a39a0a3686096356ed8af4ac0a5",
  "cmd_all_timing_active_ms": 1209,
  "cmd_all_exit": "ok",
  "args": "-d --store xxx.myshopify.com --verbose",
  "env_plugin_installed_all": "[\"@shopify/cli\"]",
  "metadata": "{\"extraPublic\":{},\"extraSensitive\":{}}"
}


You can test the TypeScript implementation (new colorful UI) vs Ruby implementation (what older CLI versions use) with the --stable flag (theme push -d --stable). --stable will run the old Ruby implementation while without it will run the new TypeScript implementation.

I tested three times with --stable. I did not change or delete any file. please help to check the results:

  • Test 1 - 63s ~ 70s
    about 100 api requests with 100 files in total.
    image
2024-07-18T01:35:20.245Z: Analytics event sent: {
  "command": "theme push",
  "time_start": 1721266449275,
  "time_end": 1721266519459,
  "total_time": 70184,
  "success": true,
  "cli_version": "3.64.0",
  "ruby_version": "3.1.2",
  "node_version": "22.4.1",
  "is_employee": false,
  "uname": "windows amd64",
  "env_ci": false,
  "env_plugin_installed_any_custom": false,
  "env_plugin_installed_shopify": "[\"@shopify/cli\"]",
  "env_shell": "cmd.exe",
  "env_device_id": "59e44e9b47f02007da55a01933fdbb74b0d207e4",
  "env_cloud": "localhost",
  "env_package_manager": "pnpm",
  "env_is_global": true,
  "cmd_app_warning_api_key_deprecation_displayed": false,
  "cmd_all_timing_network_ms": 2211,
  "cmd_all_timing_prompts_ms": 0,
  "cmd_all_launcher": "unknown",
  "cmd_all_topic": "theme",
  "cmd_all_plugin": "@shopify/cli",
  "cmd_all_force": false,
  "cmd_all_verbose": true,
  "cmd_all_path_override": true,
  "cmd_all_path_override_hash": "e23c73e2fe642a39a0a3686096356ed8af4ac0a5",
  "cmd_all_timing_active_ms": 67972,
  "cmd_all_exit": "ok",
  "args": "-d --stable --store xxx.myshopify.com --verbose",
  "env_plugin_installed_all": "[\"@shopify/cli\"]",
  "metadata": "{\"extraPublic\":{},\"extraSensitive\":{}}"
}
  • Test 2 - take 9s ~12s (faster than tests without --stable)
    16 api requests with 16 files in total.
    image
2024-07-18T01:38:33.814Z: Analytics event sent: {
  "command": "theme push",
  "time_start": 1721266700287,
  "time_end": 1721266713038,
  "total_time": 12751,
  "success": true,
  "cli_version": "3.64.0",
  "ruby_version": "3.1.2",
  "node_version": "22.4.1",
  "is_employee": false,
  "uname": "windows amd64",
  "env_ci": false,
  "env_plugin_installed_any_custom": false,
  "env_plugin_installed_shopify": "[\"@shopify/cli\"]",
  "env_shell": "cmd.exe",
  "env_device_id": "59e44e9b47f02007da55a01933fdbb74b0d207e4",
  "env_cloud": "localhost",
  "env_package_manager": "pnpm",
  "env_is_global": true,
  "cmd_app_warning_api_key_deprecation_displayed": false,
  "cmd_all_timing_network_ms": 1625,
  "cmd_all_timing_prompts_ms": 0,
  "cmd_all_launcher": "unknown",
  "cmd_all_topic": "theme",
  "cmd_all_plugin": "@shopify/cli",
  "cmd_all_force": false,
  "cmd_all_verbose": true,
  "cmd_all_path_override": true,
  "cmd_all_path_override_hash": "e23c73e2fe642a39a0a3686096356ed8af4ac0a5",
  "cmd_all_timing_active_ms": 11125,
  "cmd_all_exit": "ok",
  "args": "-d --stable --store xxx.myshopify.com --verbose",
  "env_plugin_installed_all": "[\"@shopify/cli\"]",
  "metadata": "{\"extraPublic\":{},\"extraSensitive\":{}}"
}
  • Test 3 - same as the Test 2, the same 16 files uploaded.

@BearKid
Copy link
Author

BearKid commented Jul 18, 2024

Continuing from the previous comment.
Further, I re-tested the same project 1 on my old PC with shopify/cli@3.47.5, taking about 9s ~ 14s.
Same, I didn't change or delete any files when doing tests.

  • I found that this version of the cli uses the PUT /assets/bluk.json api and the DELETE /assets.json API. (v3.64.0 push with --stable uses PUT /assets.json API)
  • Repeatly executing the theme push -d command, the cli always uploads the same 21 files.

On my new PC, the 16 uploaded files under shopify/cli@3.64.0 theme push -d **--stable** have a large overlap with these 21 files. This explains the similar upload times for both commands.

@lucyxiang
Copy link
Contributor

Thank you for the details @BearKid, I'll bring this back to the team. If you are comfortable sharing the exact theme projects you're working with, that'll help us investigate, you can email them to me at lucy.xiang@shopify.com.

@BearKid
Copy link
Author

BearKid commented Jul 23, 2024

Thank you for the details @BearKid, I'll bring this back to the team. If you are comfortable sharing the exact theme projects you're working with, that'll help us investigate, you can email them to me at lucy.xiang@shopify.com.

Thx, I re-test in a clean project (created by git clone dawn.git from official git repository), and I just sent a email to you.

No luck, the cli without --stable uploaded a lot of files:
image

2024-07-23T09:20:32.091Z: Analytics event sent: 
{
  "command": "theme push",
  "time_start": 1721726390869,
  "time_end": 1721726431007,
  "total_time": 40138,
  "success": true,
  "cli_version": "3.64.0",
  "ruby_version": "3.1.2",
  "node_version": "22.4.1",
  "is_employee": false,
  "uname": "windows amd64",
  "env_ci": false,
  "env_plugin_installed_any_custom": false,
  "env_plugin_installed_shopify": "[\"@shopify/cli\"]",
  "env_shell": "cmd.exe",
  "env_device_id": "965d14299bfe08715bf65c528f0f88481987ba6d",
  "env_cloud": "localhost",
  "env_package_manager": "unknown",
  "env_is_global": true,
  "cmd_app_warning_api_key_deprecation_displayed": false,
  "cmd_all_timing_network_ms": 37321,
  "cmd_all_timing_prompts_ms": 0,
  "cmd_all_launcher": "unknown",
  "cmd_all_topic": "theme",
  "cmd_all_plugin": "@shopify/cli",
  "cmd_all_force": false,
  "cmd_all_verbose": true,
  "cmd_all_path_override": true,
  "cmd_all_path_override_hash": "fc9241ef609882b9d6bfbf1494a2267be28512fe",
  "cmd_all_timing_active_ms": 2816,
  "cmd_all_exit": "ok",
  "args": "-d --verbose --store xxx-dev.myshopify.com",
  "env_plugin_installed_all": "[\"@shopify/cli\"]",
  "metadata": "{\"extraPublic\":{},\"extraSensitive\":{}}"
}

after that, I run twice with --stable, at the first time the cli uploaded a lot of files, and the second time the cli still uploaded some files.

@lukeh-shopify
Copy link
Contributor

👋🏻 Hi all, thanks so much for your reports and logs. We are tracking this in some upcoming work, so hope to be able to resolve these issues soon. Thanks for your patience with this one

Copy link
Contributor

This issue seems inactive. If it's still relevant, please add a comment saying so. Otherwise, take no action.
→ If there's no activity within a week, then a bot will automatically close this.
Thanks for helping to improve Shopify's dev tooling and experience.

P.S. You can learn more about why we stale issues here.

Copy link
Contributor

This issue seems inactive. If it's still relevant, please add a comment saying so. Otherwise, take no action.
→ If there's no activity within a week, then a bot will automatically close this.
Thanks for helping to improve Shopify's dev tooling and experience.

P.S. You can learn more about why we stale issues here.

@supa-freak
Copy link

We are having the same experience and it hits our productivity quite a bit. Together with the 3.69.x introduced inability to build when triggered by github on cloud environments (crash due to prompting which doesnt work on remote), I feel the shopify cli is getting worse by the version. We will stay with an old version since it is much more effective and stable.

@catlee
Copy link
Contributor

catlee commented Nov 4, 2024

Thank you all for the updates!

If this is still occurring, it would help to know which files are getting re-uploaded each time.

(crash due to prompting which doesnt work on remote)

Would you mind sharing the command you're running in github, and what prompt is failing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: @shopify/theme @shopify/theme package issues moderate lowest severity for theme related bug Type: Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants