Subsequent delta updates on Windows fail 2/2

[mherrmann]

Description

On Windows, a delta update that gets applied after another delta update always fails and falls back to the full update.

Steps to Reproduce

1. Install Brave 1.49.120 using its offline installer. Accept the UAC prompt to perform a system-wide installation.
2. Close Brave and make sure you don't have internet so Brave doesn't accidentally get updated.
3. Open a Windows command prompt with Administrator privileges (right-click->"Run as Administrator").
4. Download the delta installer to go from Brave 1.49.120 to 1.49.128.
5. Enter the following in the command prompt: set BraveSoftwareUpdateIsMachine=1.
6. Execute the delta installer in the same command prompt. For example by typing: C:\Users\micha\Downloads\brave_installer-delta-x64.exe. If you are testing Release, don't append anything. If you are testing Nightly, append --chrome-sxs. If you are testing Beta, append --chrome-beta. For example: brave_installer-delta-x64.exe --chrome-beta.
7. Repeat steps 4. - 6. for the delta installer to go from Brave 1.49.128 to 1.49.132
8. Start Brave.

The above delta installer URLs are for the Release channel. For Beta, use and modify https://updates-cdn.bravesoftware.com/delta/Brave-Browser-Beta/x64-be/win/113.1.52.94/113.1.52.92/brave_installer-delta-x64.exe.

Actual result:

The version on brave://settings/help is 1.49.128.

Expected result:

The version on brave://settings/help should be 1.49.132.

Reproduces how often:

Always.

Steps to Reproduce more generally

Repeat the steps above for another first, second and third public versions of Brave. You can construct the delta URLs from the version numbers. Note: Deltas are only available for subsequent public versions. For example, there is also a 1.49.131 non-public/pre-release version. But because it isn't public, there is no delta for it.

Related issues

The following issue also affects delta updates on Windows: #28989

[mherrmann]

Reopening this because my fix in #17882 only fixed one cause of the problem. But at least one other cause still exists.

[stephendonner]

Reopening this because my fix in #17882 only fixed one cause of the problem. But at least one other cause still exists.

That's a different issue, I think you meant to point to the full link of the commit.

[mherrmann]

Thank you @stephendonner, yes, I meant brave/brave-core#17882.

[MadhaviSeelam]

Added OS/macOS label as per the testplan , Widevine functionality needs to be tested on macOS.

Specifically:

We need to make sure that DRM-protected content can still be played at the highest quality. It should be possible to test this with Netflix or Amazon Prime, verifying that the video quality offered by the site is as high as before this PR.

more info: https://bravesoftware.slack.com/archives/C012NN4F6UU/p1683648954812479

[stephendonner]

Verification PASSED using

Brave	1.52.94 Chromium: 113.0.5672.92 (Official Build) beta (x86_64)
Revision	b6f521170062a1fa8a82c33fb223b06fec566da1-refs/branch-heads/5672_63@{#10}
OS	macOS Version 11.7.6 (Build 20G1231)

Widevine component details:

Confirmed via brave://components that Widevine Content Decryption Module - Version: 4.10.2652.1 was downloaded, in all cases

brave://components	WidevineCdm folder

Netflix - PASSED

Steps:

1. installed 1.52.94
2. launched Brave
3. loaded netflix.com
4. clicked on the Sign In link
5. entered my credentials
6. clicked on the Sign In button
7. confirmed I got prompted to Install and run Widevine
8. clicked Allow
9. clicked on my user avatar
10. played Our Planet (https://www.netflix.com/watch/80094026)

Confirmed the content was streamed in High-Definition (HD) quality (through visual inspection)

Widevine prompt	HD content

Amazon Prime - PASSED

Steps:

1. installed 1.52.94
2. launched Brave
3. loaded amazon.com
4. clicked on Prime Video
5. logged in
6. confirmed I was prompted to Install and run Widevine
7. clicked on Allow
8. clicked to play John Wick 3 - https://www.amazon.com/gp/video/detail/B07SZZNFRM/ref=atv_hm_hom_c_lZOsi7_2_4?jic=8%7CEgNhbGw%3D

Confirmed the content was streamed in High-Definition (HD) quality, "Best" quality level @ 1080p (6.48 GB per hour)

Widevine prompt	HD content

HBOMax - PASSED

Steps:

1. installed 1.52.94
2. launched Brave
3. loaded hbomax.com
4. clicked on the Sign In button
5. clicked Sign in to your account
6. clicked Sign in With a Provider button
7. finally signed in
8. clicked on my username/avatar
9. confirmed I was prompted to Install and run Widevine
10. clicked on Moonage Daydream

Confirmed the content was streamed in High-Definition (HD) quality, through visual inspection

Widevine prompt	HD content

[mherrmann]

I believe the milestone for this should be 1.53.x and have changed it from 1.52.x. There were two PRs that fix this issue: 17882 is in 1.52.x while 18043 is in 1.53.x. I believe this can only be fully tested with 1.53.x. What do you think @stephendonner @MadhaviSeelam?

[stephendonner]

Makes perfect sense to me, @mherrmann 👍

[kjozwiak]

I believe the milestone for this should be 1.53.x and have changed it from 1.52.x. There were two PRs that fix this issue: 17882 is in 1.52.x while 18043 is in 1.53.x. I believe this can only be fully tested with 1.53.x. What do you think @stephendonner @MadhaviSeelam?

@mherrmann we usually keep issues in the milestone were the PRs landed. Ideally, we would have two issues for the above scenario. One via 1.52.x & 1.53.x. The one in 1.52.x would be labelled as QA/No as the above can only be verified via 1.53.x. Then another issue within 1.53.x labelled as QA/Yes. The rule that we usually follow is one PR per issue so you don't run into cases where one PR spans ~2-3 milestones and there's only one issue.

For now, leaving this in 1.53.x works 👍 Regarding 1.52.x, is there anything that needs to be checked? Just double checking that we're not going to ship with partial fixes/code that might cause problems.

[LaurenWags]

Removing QA Pass-macOS as milestone for this issue was adjusted.

[kjozwiak]

@kjozwiak I've created #30485 now to capture this :)

awesome, thanks @mherrmann 👍 Assuming #30551 doesn't block 1.52.x?

[mherrmann]

Assuming #30551 doesn't block 1.52.x?

No, it doesn't. Not in theory and also not in practice - I just checked from v1.52.109 and v1.52.102 and successfully received updates to v1.53.66.

[MadhaviSeelam]

Verification PASSED using

Brave | 1.53.81 Chromium: 114.0.5735.90 (Official Build) beta (64-bit)
-- | --
Revision | 386bc09e8f4f2e025eddae123f36f6263096ae49-refs/branch-heads/5735@{#1052}
OS | Windows 11 Version 22H2 (Build 22621.1702)

Pre-requisites:

• Uninstall Brave from your system (all channels)
• Run the following in an Administrator command prompt: "C:\Program Files (x86)\BraveSoftware\Update\BraveUpdate.exe" /uninstall
• The following registry key should now not exist: HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\BraveSoftware\Update\Clients.
• Delete the following file if it exists: C:\ProgramData\BraveSoftware\Update\Log\BraveUpdate.log.
• Set the registry value HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\BraveSoftware\UpdateDev\IsEnabledLogToFile to DWORD 1.

Upgrade 1.53.66 --> 1.53.76
Steps:

1. Followed above Pre-requisite steps before installing Brave
2. Installed BraveBrowserStandaloneBetaSetup.exe public beta installer for 1.53.66
3. launched Brave
4. Open brave://settings/help
5. click Relaunch
6. confirmed Brave updated to latest public beta version 1.53.76

Confirmed Delta updates received via brave_installer-delta-x64.exe for 1.53.66-->1.53.76`

[06/05/23 13:23:36.687][BraveUpdate:goopdate][15932:27616][Installing][display name: Brave Beta][app id: {103BD053-949B-43A8-9120-2E424887DE11}][installer path: C:\Program Files (x86)\BraveSoftware\Update\Install\{2FF268EE-DBD8-41A5-A307-2384995FE167}\brave_installer-delta-x64.exe][manifest args: --chrome-beta][installer data: ][untrusted data: ][priority: 10]
[06/05/23 13:23:36.687][BraveUpdate:goopdate][15932:27616][Running installer][C:\Program Files (x86)\BraveSoftware\Update\Install\{2FF268EE-DBD8-41A5-A307-2384995FE167}\brave_installer-delta-x64.exe][--chrome-beta][{103BD053-949B-43A8-9120-2E424887DE11}]
[06/05/23 13:23:55.943][BraveUpdate:goopdate][15932:27616][InstallerResult][{103BD053-949B-43A8-9120-2E424887DE11}][0]
[06/05/23 13:23:55.943][BraveUpdate:goopdate][15932:27616][InstallApp returned][0x0][{103BD053-949B-43A8-9120-2E424887DE11}][type:1][code: 30][][]
[06/05/23 13:23:55.953][BraveUpdate:goopdate][15932:27616][Started process][30708]
[06/05/23 13:23:55.962][BraveUpdate:goopdate][15932:27616][Started process][28944]

1.53.66	1.53.66 Installed logs	Update	1.53.76	Delta logs

Upgrade 1.53.71 --> 1.53.74

Note: Confirmed Delta updates not received via brave_installer-delta-x64.exe for 1.53.71-->1.53.74`

1.53.66	1.53.66 Installed logs	Update	1.53.76	Delta logs

Re- verified as following for 1.53.76 -->1.53.82-->1.53.83

1. Install Brave 1.53.76 using its offline installer.
2. Accepted the UAC prompt to perform a system-wide installation.
3. Closed Brave and disconnected from internet so Brave doesn't accidentally get updated.
4. Open a Windows command prompt with Administrator privileges (right-click->"Run as Administrator").
5. Download the delta installer to go from Brave 1.53.76 to 1.53.82.
6. Enter the following in the command prompt: set BraveSoftwareUpdateIsMachine=1.
7. Execute the delta installer in the same command prompt. For example by typing: C:\Users\mseel\Downloads\brave_installer-delta-x64.exe --chrome-beta.
8. Repeat steps 5 - 7 for the delta installer to go from Brave 1.53.82 to 1.53.83
9. Start Brave.

Actual Results:

• Got Deltas for the first update 1.52.76-->1.52.82
• However also see unspecified installation failure error.
• Looks like 1.53.83 is failed to install as I didn't see UAC when I executed delta installer for 1.53.82->1.53.83
• tried updating via brave://settings/help. Got the Update error
• Refresh the page and got full update.

[06/07/23 14:54:47.793][BraveUpdate:goopdate][31768:13864][Installing][display name: Brave Beta][app id: {103BD053-949B-43A8-9120-2E424887DE11}][installer path: C:\Program Files (x86)\BraveSoftware\Update\Install\{6951E7F1-EF83-4EA7-87DF-DB8E739D54E7}\brave_installer-delta-x64.exe][manifest args: --chrome-beta][installer data: ][untrusted data: ][priority: 0]
[06/07/23 14:54:47.793][BraveUpdate:goopdate][31768:13864][Running installer][C:\Program Files (x86)\BraveSoftware\Update\Install\{6951E7F1-EF83-4EA7-87DF-DB8E739D54E7}\brave_installer-delta-x64.exe][--chrome-beta][{103BD053-949B-43A8-9120-2E424887DE11}]
[06/07/23 14:54:58.629][BraveUpdate:goopdate][31768:13864][InstallerResult][{103BD053-949B-43A8-9120-2E424887DE11}][1]
[06/07/23 14:54:58.629][BraveUpdate:goopdate][31768:13864][Installer failed][C:\Program Files (x86)\BraveSoftware\Update\Install\{6951E7F1-EF83-4EA7-87DF-DB8E739D54E7}\brave_installer-delta-x64.exe][--chrome-beta][8]
[06/07/23 14:54:58.629][BraveUpdate:goopdate][31768:13864][InstallApp returned][0x80040902][{103BD053-949B-43A8-9120-2E424887DE11}][type:4][code: 8][Installation failed due to unspecified error. Please download Brave again.][]

ex	ex	ex	ex	ex
image

[stephendonner]

Verification PASSED using

Brave	1.53.83 Chromium: 114.0.5735.110 (Official Build) beta (64-bit)
Revision	1c828682b85bbc70230a48f5e345489ec447373e-refs/branch-heads/5735_90@{#13}
OS	Windows 10 Version 22H2 (Build 19045.3031)

Prerequisites

• Uninstall Brave from your system (all channels)
• Run the following in an Administrator command prompt: "C:\Program Files (x86)\BraveSoftware\Update\BraveUpdate.exe" /uninstall
• The following registry key should now NOT exist: HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\BraveSoftware\Update\Clients
• Delete the following file if it exists: C:\ProgramData\BraveSoftware\Update\Log\BraveUpdate.log
• Set the registry value HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\BraveSoftware\UpdateDev\IsEnabledLogToFile to DWORD 1

Upgrade 1.53.82 -> 1.53.83 - PASSED

Steps:

(continued from Prerequisites, above)

1. installed https://github.com/brave/brave-browser/releases/download/v1.53.82/BraveBrowserStandaloneBetaSetup.exe
2. launched Brave
3. opened brave://settings/help
4. clicked Relaunch
5. confirmed Brave updated to latest public-beta version (at the time), 1.53.83

Confirmed delta updated received via brave_installer-delta-x64.exe going from 1.53.82 to 1.53.83

1.53.82	installed logs	Update	1.53.83	Delta logs

Upgrade 1.53.76 -> 1.53.x - FAILED*

NOTE: Fails due to #30748

Steps:

1. installed Brave 1.53.76 using its offline installer
2. accepted the UAC prompt to perform a system-wide installation
3. closed Brave and disconnected from Wi-Fi/internet, so Brave doesn't accidentally get updated
4. opened a Windows command prompt with Administrator privileges (right-clicked->"Run as Administrator").
5. downloaded the delta installer to go from Brave 1.53.76 to 1.53.82.
6. entered the following in the command prompt: set BraveSoftwareUpdateIsMachine=1
7. executed the delta installer in the same command prompt. Entered: C:\Users\steph\Downloads\brave_installer-delta-x64.exe --chrome-beta
8. confirmed UAC prompt to update
9. repeated steps 5 - 7 for the delta installer to go from Brave 1.53.82 to 1.53.83
10. started Brave

example	example	example	example	example

[06/08/23 12:31:48.068][BraveUpdate:goopdate][2500:3744][InstallerResult][{103BD053-949B-43A8-9120-2E424887DE11}][1]
[06/08/23 12:31:48.068][BraveUpdate:goopdate][2500:3744][Installer failed][C:\Program Files (x86)\BraveSoftware\Update\Install\{5F516FFD-95F7-482C-903D-710159AB3B5F}\brave_installer-delta-x64.exe][--chrome-beta][8]
[06/08/23 12:31:48.068][BraveUpdate:goopdate][2500:3744][InstallApp returned][0x80040902][{103BD053-949B-43A8-9120-2E424887DE11}][type:4][code: 8][Installation failed due to unspecified error. Please download Brave again.][]

Encountered:

• Subsequent delta updates on Windows fail #30748

[MadhaviSeelam]

@mherrmann can you review actual results section for Re- verified as following for 1.53.76 -->1.53.82-->1.53.83 case in #29469 (comment) please? Delta updates are failing (step 8) on second delta installer.

cc: @stephendonner

[stephendonner]

@mherrmann can you review actual results section for Re- verified as following for 1.53.76 -->1.53.82-->1.53.83 case in #29469 (comment) please? Delta updates are failing (step 8) on second delta installer.

cc: @stephendonner

Perhaps (probably) due to #30748 (comment)?

[mherrmann]

Yes, unfortunately there was the second regression mentioned in #30748 by @stephendonner; I'm sorry.

[stephendonner]

Yes, unfortunately there was the second regression mentioned in #30748 by @stephendonner; I'm sorry.

No worries, and thanks for confirming. Marking as QA Pass-Win64 for the initial, passing case, and noted #30748 as the followup issue 👍 cc @MadhaviSeelam

[LaurenWags]

@mherrmann we're still ok for 1.53.x without the fix to #30748, correct? it seems that users still get updated, they just get the full installer?

cc @kjozwiak @rebron

[mherrmann]

@LaurenWags Correct!

[LaurenWags]

Excellent, thanks for confirming @mherrmann 👍🏻

[LaurenWags]

Verified with

Brave | 1.53.91 Chromium: 114.0.5735.110 (Official Build) beta (x86_64)
-- | --
Revision | 1c828682b85bbc70230a48f5e345489ec447373e-refs/branch-heads/5735_90@{#13}
OS | macOS Version 13.4 (Build 22F66)

Performing Widevine checks as per brave/brave-core#18043 (comment).

Widevine component info

Netflix - PASSED

Compared Netflix quality on 1.53.91 to 1.52.122 by playing the same video on both.
Confirmed video quality appeared the same (through visual inspection).

Max (formerly HBOMax) - PASSED

Compared Max quality on 1.53.91 to 1.52.122 by playing the same video on both.
Confirmed video quality appeared the same (through visual inspection).

Amazon Prime - PASSED

Compared Max quality on 1.53.91 to 1.52.122 by playing the same video on both.
Confirmed video quality appeared the same (through visual inspection).
Confirmed the content was streamed in High-Definition (HD) quality, "Best" quality level (6.48 GB per hour) per their controls.

1.52.x	1.53.x

[LaurenWags]

Changing to release-notes/exclude as there additional fixes (#30748) coming in subsequent version. Will include in release notes when all fixes are included.

@rebron @kjozwiak @mherrmann ok with this?

[kjozwiak]

Changing to release-notes/exclude as there additional fixes (#30748) coming in subsequent version. Will include in release notes when all fixes are included.

@rebron @kjozwiak @mherrmann ok with this?

SGTM 👍 No objections re: excluding the above till we have everything fixed/addressed.