x86 Windows machines are not updating #2221

kjozwiak · 2018-11-23T04:43:52Z

Description

Updates are currently failing on x86 machines. I initially ran into the issue when we were releasing 0.56.14 but no one else had x86 machines that could double check if I was the only one receiving this issue on my VM's. @GeetaSarvadnya mentioned that she also ran into the issue once on her x64 machine so we thought it was an intermittent issue. However, getting reports from reddit that x86 users are not getting new updates:

https://www.reddit.com/r/brave_browser/comments/9ywwpb/brave_version_05614_now_available_on_release/ea8p9jv

When attempting to update my Win x86 machine, I'm getting the following error under brave://settings/help:

An error occurred while checking for updates: The installer encountered error 255. (error code 7: 0x80040902: 255 -- system level).

Win 10 x86 logs: OmahaUpdate.log
Win 7 x86 logs: OmahaUpdate.log

Steps to Reproduce

launch a Win x86 VM and install https://github.com/brave/brave-browser/releases/download/v0.56.14/BraveBrowserStandaloneSetup32.exe
once installed, attempt to update Brave via brave://settings/help

Actual result:

When attempting to update an x86 machine, you'll get the following error and Brave won't upgrade:

An error occurred while checking for updates: The installer encountered error 255. (error code 7: 0x80040902: 255 -- system level).

Expected result:

x86 machines should be updating to the latest version without any issues.

Reproduces how often:

100% reproducible using the above STR.

Brave version (brave://version info)

Brave	0.56.15 Chromium: 70.0.3538.110 (Official Build) (32-bit)
Revision	ca97ba107095b2a88cf04f9135463301e685cbb0-refs/branch-heads/3538@{#1094}
OS	Windows

Brave	0.56.14 Chromium: 70.0.3538.102 (Official Build) (32-bit)
Revision	4bbeebac88fdc09c97265e47c205868bbd190497-refs/branch-heads/3538@{#1077}
OS	Windows

Reproducible on current release:

Does it reproduce on brave-browser dev/beta builds?

I'm not 100% sure if the other channels are available as there's no standalone installers for the previous versions

Additional Information

Win 10 x86 - Reproduced
Win 8.1 x86 - Reproduced
Win 7 x86 - Reproduced

The text was updated successfully, but these errors were encountered:

simonhong · 2018-11-23T13:54:47Z

I read the above reddit post and I think it is different issue with this because user saw up to date not 0x80040902(this error means silent installer failed)

mihaiplesa · 2018-11-23T14:33:25Z

Reproduced on Win 7 x64 using 32-bit standalone installer for 0.56.14 then attempted update.

After the error message appears (255) and an update is re-attempted it tries to get updates from the x86-rel-full channel which doesn't exist. Also, there's two different appid entries in the response and that doesn't look ok.

[11/23/18 14:25:12.604][BraveUpdate:goopdate][4000:1896][Send][url=https://updates.bravesoftware.com/service/update2][request=<?xml version="1.0" encoding="UTF-8"?><request protocol="3.0" version="1.3.99.0" shell_version="1.3.99.0" ismachine="1" sessionid="{DF73D6DA-3263-4972-83CD-C9EC025AF4B0}" installsource="scheduler" testsource="auto" requestid="{E9A87339-ED58-419B-A5BD-84A12751C352}" dedup="cr"><hw physmemory="4" sse="1" sse2="1" sse3="1" ssse3="1" sse41="1" sse42="1" avx="1"/><os platform="win" version="6.1.7601.0" sp="Service Pack 1" arch="x64"/><app appid="{AFE6A462-C574-4B8A-AF43-4CC60DF4563B}" version="70.0.56.14" nextversion="" ap="x86-rel-full" lang="en" brand="GGLS" client="" installage="0" installdate="4340"><updatecheck/><ping active="0" rd="4344" ping_freshness="{D620F9EC-0B92-4B97-8A29-46D3C5808B30}"/></app><app appid="{B131C935-9BE6-41DA-9599-1F776BEB8019}" version="1.3.99.0" nextversion="" lang="" brand="GGLS" client="" installage="0"><updatecheck/><ping r="-1" rd="-1"/></app></request>][filename=]
[11/23/18 14:25:13.038][BraveUpdate:goopdate][4000:1896][Send response received][result 0x0][status code 200][<?xml version='1.0' encoding='UTF-8'?>
<response protocol="3.0" server="prod">
  <daystart elapsed_seconds="51912" elapsed_days="4344"/>
  <app status="ok" appid="{AFE6A462-C574-4B8A-AF43-4CC60DF4563B}">
    <updatecheck status="noupdate"/>
    <ping status="ok"/>
  </app>
  <app status="ok" appid="{B131C935-9BE6-41DA-9599-1F776BEB8019}">
    <updatecheck status="noupdate"/>
    <ping status="ok"/>
  </app>
</response>
]

The correct channel name is x86-rel which it has tried initially before the error so there seems to be some logic in the installer that allows that.

[11/23/18 14:20:51.661][BraveUpdate:goopdate][5032:5116][Send][url=https://updates.bravesoftware.com/service/update2][request=<?xml version="1.0" encoding="UTF-8"?><request protocol="3.0" version="1.3.99.0" shell_version="1.3.99.0" ismachine="1" sessionid="{4DAD4CA5-A4F7-4AB7-86A3-76293AC1FEF6}" installsource="update3web-ondemand" testsource="auto" requestid="{90B08142-D10E-4A26-94CD-04381EFD8B34}" dedup="cr"><hw physmemory="4" sse="1" sse2="1" sse3="1" ssse3="1" sse41="1" sse42="1" avx="1"/><os platform="win" version="6.1.7601.0" sp="Service Pack 1" arch="x64"/><app appid="{AFE6A462-C574-4B8A-AF43-4CC60DF4563B}" version="70.0.56.14" nextversion="" ap="x86-rel" lang="en" brand="GGLS" client="" installage="0"><updatecheck/><ping active="1" a="-1" r="-1" ad="-1" rd="-1"/></app></request>][filename=]
[11/23/18 14:20:52.582][BraveUpdate:goopdate][4468:2336][DllEntry exit][0x00000000]
[11/23/18 14:20:52.628][BraveUpdate:goopdate][3148:3580][Setup::Uninstall]
[11/23/18 14:20:52.632][BraveUpdate:goopdate][3148:3580][CanUninstallGoogleUpdate returned 0]
[11/23/18 14:20:52.632][BraveUpdate:goopdate][3148:3580][DllEntry exit][0x00000000]
[11/23/18 14:21:03.149][BraveUpdate:goopdate][5032:5116][Send response received][result 0x0][status code 200][<?xml version='1.0' encoding='UTF-8'?>
<response protocol="3.0" server="prod">
  <daystart elapsed_seconds="51663" elapsed_days="4344"/>
  <app status="ok" appid="{AFE6A462-C574-4B8A-AF43-4CC60DF4563B}">
    <updatecheck status="ok">
      <urls>
        <url codebase="https://updates-cdn.bravesoftware.com:443/build/Brave-Release/x86-rel/win/76965817614351/"/>
      </urls>
      <manifest version="70.0.56.15">
        <packages>
          <package required="true" hash="cTd39dBQ7fkU5d1XMZ+dBQFpanw=" name="brave_installer-ia32.exe" size="54878696"/>
        </packages>
      </manifest>
    </updatecheck>
    <ping status="ok"/>
  </app>
</response>

Full output at BraveUpdate.log

mihaiplesa · 2018-11-23T14:41:02Z

The unexpected appid above also appears in logs attached in #2129

simonhong · 2018-11-25T23:23:58Z

It's normal to update check with different two appid because {B131C935-9BE6-41DA-9599-1F776BEB8019} is the appid of omaha client. Omaha client can check update itself.

We should check about x86-rel-full appid. It's strange because usually appid is set by tagging process.

mihaiplesa · 2018-11-25T23:53:18Z

@mbacchi any ideas?

kjozwiak · 2018-11-26T04:12:32Z

I read the above reddit post and I think it is different issue with this because user saw up to date not 0x80040902(this error means silent installer failed)

Agreed, I think it was a different issue as the user managed to finally update their Brave. Once you receive the error I mentioned above, every other attempt will display Brave is up to date even though there's a new version available. It seemed like the same issue as the user was running x86 and was a few versions behind but kept receiving Brave is up to date.

simonhong · 2018-12-03T01:45:13Z

@kjozwiak I see. I also saw error one time and then up to date is visible.

simonhong · 2018-12-03T02:20:16Z

Hmm, root cause of this issue is -full is appended to our ap. When I manually remove it from regedit, everything works fine. Finding where and why this -full is appended.

I found the why installer appends -full to existing ap.
When install starts, mini_installer appends -full to existing ap and then removes it when install is completed. The purpose of this is indicator of incremental update failure.
When chrome installer fails incremental updates, installer doesn't remove -full to use full install at the next trying by changing ap.

We always uses full update so appending and removing during the installation doesn't have meaning for us now.

Need to investigate why appended -full isn't removed on x86 only.

mihaiplesa · 2018-12-03T11:03:52Z

This is where I see it used https://github.com/brave/brave-core/search?q=-full&type=Code

simonhong · 2018-12-03T22:48:47Z

Hmm, I built standalone installer on my local and did update test.
Unfortunately, this problem wasn't reproduced with that. I'll dig more about this.

kjozwiak · 2018-12-03T22:58:14Z

Hmm, I built standalone installer on my local and did update test.
Unfortunately, this problem wasn't reproduced with that. I'll dig more about this.

@simonhong let me know if you need any help debugging 👍I can pretty much reproduce this 100% of the time using the STR in #2221 (comment) using either Win 10 x86 or Win 7 x86. Both are VM's.

The strange part is that I can't reproduce this on my Win 8.1 x86 VM. @simonhong let me know if there's anything that I can check on the Win 10 x86 or Win 7 x86 VM's which are failing.

simonhong · 2018-12-03T23:35:16Z

Yep, I also can reproduce this with released standalone installer and update trying... (So it's weird why manually built standalone and update trying works fine).
Anyway, I'll take a look more.

I think the main problem is the update fail on first update trying.
Because of that fail, I assume -full isn't removed from ap.
It should be reverted to x86-rel when install completed. But, it wasn't in our case.(still x86-rel-full)
I think not reverted ap caused up to date problem.

So, I think up to date problem would be disappeared after fixing update fail on first trying.

simonhong · 2018-12-04T01:42:24Z

@mbacchi @mihaiplesa Finally I reproduced this on my local with manually built installer and test update server.
When I omit update action in omaha server, I got same error.
Can you check install/update actions were set properly for each x86 versions in official server?
(I can't access our official omaha server.)
Every versions should have install/update event in Actions sections.

like below.

mihaiplesa · 2018-12-04T13:19:37Z

Have just checked and the uploaded versions look fine (unless it was happening on an older version we removed).

Will make sure we do it right from now.

Just curious, what happens if we mismatch and put --chrome-dev- instead of --chrome-beta for a beta version?

simonhong · 2018-12-04T21:43:02Z

If we use wrong argument(ex using --chrome-dev to beta), it will cause the problem.
installer determines which channel it should install by using this arguments.
I think using wrong arguments could cause install error.

mihaiplesa · 2018-12-04T21:57:06Z

Thanks for clarifying, we'll see if this happens again.

mihaiplesa · 2020-03-27T08:18:57Z

Re-opening to create secondary channels suffixed with -full and serve updates to them so they come back to the regular channel.

bsclifton · 2020-04-14T19:06:48Z

Thanks @mihaiplesa and @mbacchi for looking into solutions. After discussion, I think this is a fairly isolated problem. The only way the -full installer could be there would have been manually installing the 7zip executable (which we have removed now from GitHub pages)

@mbacchi and I are working on a help article that we can review with @Brave-Matt for what to do when update isn't working that would resolve this scenario

mbacchi · 2020-04-27T20:27:42Z

Documentation added here: https://support.brave.com/hc/en-us/articles/360042816611-Why-isn-t-Brave-updating-automatically-on-Windows-

kjozwiak added bug priority/P1 A very extremely bad problem. We might push a hotfix for it. feature/updater labels Nov 23, 2018

kjozwiak added this to the 1.x Backlog milestone Nov 23, 2018

kjozwiak added needs-investigation A bug not 100% confirmed/fixed priority/P2 A bad problem. We might uplift this to the next planned release. and removed priority/P1 A very extremely bad problem. We might push a hotfix for it. labels Nov 23, 2018

bbondy assigned simonhong and mihaiplesa Nov 23, 2018

mihaiplesa assigned mbacchi Nov 25, 2018

btlechowski mentioned this issue Nov 30, 2018

Stub installer fails to download the build on Windows 7 #2129

Closed

simonhong added the QA/Yes label Dec 11, 2018

kjozwiak removed bug feature/updater needs-investigation A bug not 100% confirmed/fixed priority/P2 A bad problem. We might uplift this to the next planned release. labels Dec 11, 2018

kjozwiak modified the milestones: 1.x Backlog, Dupe / Invalid / Not actionable Dec 11, 2018

rebron removed this from the Dupe / Invalid / Not actionable milestone May 10, 2019

NejcZdovc added this to the Dupe / Invalid / Not actionable milestone Jun 3, 2019

mbacchi mentioned this issue Mar 21, 2020

brave-installer not available for release 1.5.112 #8743

Closed

mihaiplesa reopened this Mar 27, 2020

kjozwiak removed the closed/invalid label Mar 27, 2020

kjozwiak removed this from the Dupe / Invalid / Not actionable milestone Mar 27, 2020

kjozwiak added the feature/updater label Mar 27, 2020

bsclifton mentioned this issue Mar 29, 2020

Error when trying to upgrade Brave via brave://settings/help #8906

Closed

rebron assigned bsclifton and unassigned mihaiplesa, mbacchi and simonhong Apr 14, 2020

rebron added the support addition/edits to articles on support website label Apr 14, 2020

bsclifton assigned mbacchi Apr 14, 2020

mbacchi closed this as completed Apr 27, 2020

bbondy added this to the Closed / Invalid milestone Jun 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x86 Windows machines are not updating #2221

x86 Windows machines are not updating #2221

kjozwiak commented Nov 23, 2018 •

edited

Loading

simonhong commented Nov 23, 2018

mihaiplesa commented Nov 23, 2018 •

edited

Loading

mihaiplesa commented Nov 23, 2018

simonhong commented Nov 25, 2018

mihaiplesa commented Nov 25, 2018

kjozwiak commented Nov 26, 2018 •

edited

Loading

simonhong commented Dec 3, 2018

simonhong commented Dec 3, 2018 •

edited

Loading

mihaiplesa commented Dec 3, 2018

simonhong commented Dec 3, 2018

kjozwiak commented Dec 3, 2018 •

edited

Loading

simonhong commented Dec 3, 2018 •

edited

Loading

simonhong commented Dec 4, 2018 •

edited

Loading

mihaiplesa commented Dec 4, 2018

simonhong commented Dec 4, 2018

mihaiplesa commented Dec 4, 2018

mihaiplesa commented Mar 27, 2020 •

edited

Loading

bsclifton commented Apr 14, 2020 •

edited

Loading

mbacchi commented Apr 27, 2020

x86 Windows machines are not updating #2221

x86 Windows machines are not updating #2221

Comments

kjozwiak commented Nov 23, 2018 • edited Loading

Description

Steps to Reproduce

Actual result:

Expected result:

Reproduces how often:

Brave version (brave://version info)

Reproducible on current release:

Additional Information

simonhong commented Nov 23, 2018

mihaiplesa commented Nov 23, 2018 • edited Loading

mihaiplesa commented Nov 23, 2018

simonhong commented Nov 25, 2018

mihaiplesa commented Nov 25, 2018

kjozwiak commented Nov 26, 2018 • edited Loading

simonhong commented Dec 3, 2018

simonhong commented Dec 3, 2018 • edited Loading

mihaiplesa commented Dec 3, 2018

simonhong commented Dec 3, 2018

kjozwiak commented Dec 3, 2018 • edited Loading

simonhong commented Dec 3, 2018 • edited Loading

simonhong commented Dec 4, 2018 • edited Loading

mihaiplesa commented Dec 4, 2018

simonhong commented Dec 4, 2018

mihaiplesa commented Dec 4, 2018

mihaiplesa commented Mar 27, 2020 • edited Loading

bsclifton commented Apr 14, 2020 • edited Loading

mbacchi commented Apr 27, 2020

kjozwiak commented Nov 23, 2018 •

edited

Loading

mihaiplesa commented Nov 23, 2018 •

edited

Loading

kjozwiak commented Nov 26, 2018 •

edited

Loading

simonhong commented Dec 3, 2018 •

edited

Loading

kjozwiak commented Dec 3, 2018 •

edited

Loading

simonhong commented Dec 3, 2018 •

edited

Loading

simonhong commented Dec 4, 2018 •

edited

Loading

mihaiplesa commented Mar 27, 2020 •

edited

Loading

bsclifton commented Apr 14, 2020 •

edited

Loading