Use BiDi protocol for Chrome tests #17962

calixteman · 2024-04-17T19:10:53Z

No description provided.

calixteman · 2024-04-17T19:11:29Z

/botio integrationtest

moz-tools-bot · 2024-04-17T19:11:31Z

From: Bot.io (Linux m4)

Received

Command cmd_integrationtest from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/395e307c994cbca/output.txt

moz-tools-bot · 2024-04-17T19:11:31Z

From: Bot.io (Windows)

Received

Command cmd_integrationtest from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/aead5c6b2e4df46/output.txt

timvandermeij

I'll have a detailed look at this tomorrow, but in the meantime here are my first observations.

test/test.mjs

test/integration/scripting_spec.mjs

test/integration/freetext_editor_spec.mjs

moz-tools-bot · 2024-04-17T19:19:11Z

From: Bot.io (Linux m4)

Success

Full output at http://54.241.84.105:8877/395e307c994cbca/output.txt

Total script time: 7.67 mins

Integration Tests: Passed

moz-tools-bot · 2024-04-17T19:30:02Z

From: Bot.io (Windows)

Success

Full output at http://54.193.163.58:8877/aead5c6b2e4df46/output.txt

Total script time: 18.50 mins

Integration Tests: Passed

calixteman · 2024-04-17T20:03:21Z

/botio test

moz-tools-bot · 2024-04-17T20:03:23Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/9e74d8b32fd9a8a/output.txt

moz-tools-bot · 2024-04-17T20:03:24Z

From: Bot.io (Windows)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/92a94d4274292a5/output.txt

moz-tools-bot · 2024-04-17T20:08:46Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/92a94d4274292a5/output.txt

Total script time: 5.37 mins

Unit tests: FAILED
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.193.163.58:8877/92a94d4274292a5/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2024-04-17T20:16:56Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/9e74d8b32fd9a8a/output.txt

Total script time: 13.53 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.241.84.105:8877/9e74d8b32fd9a8a/reftest-analyzer.html#web=eq.log

calixteman · 2024-04-18T08:00:19Z

@OrKoN the ref tests failed on the linux bot because of an OOM crash. The bot has a 64 bits OS with 16Gb of RAM and afaik we never had this kind of issues with CDP in the last years.
And on the windows bot, it failed in the middle of the unit test with an exception (TargetCloseError: Protocol error (Runtime.callFunctionOn): Target closed).

calixteman · 2024-04-18T11:21:01Z

/botio test

moz-tools-bot · 2024-04-18T11:21:03Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/17e8528589e75e3/output.txt

moz-tools-bot · 2024-04-18T11:21:03Z

From: Bot.io (Windows)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/81255050207bcff/output.txt

moz-tools-bot · 2024-04-18T11:31:00Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/17e8528589e75e3/output.txt

Total script time: 9.94 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.241.84.105:8877/17e8528589e75e3/reftest-analyzer.html#web=eq.log

OrKoN · 2024-04-18T11:50:33Z

@calixteman do all tests use Puppeteer or only integration?

OrKoN · 2024-04-18T11:51:01Z

I think I am not able to view the logs but I will give it a try later

moz-tools-bot · 2024-04-18T12:00:59Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/81255050207bcff/output.txt

Total script time: 39.92 mins

Unit tests: FAILED
Integration Tests: FAILED
Regression tests: FAILED

  different ref/snapshot: 5

Image differences available at: http://54.193.163.58:8877/81255050207bcff/reftest-analyzer.html#web=eq.log

calixteman · 2024-04-18T12:03:58Z

@calixteman do all tests use Puppeteer or only integration?

The unit/ref/integration tests are using Puppeteer.

calixteman · 2024-04-18T12:05:14Z

I think I am not able to view the logs but I will give it a try later

You should be able to just look at http://54.193.163.58:8877/81255050207bcff/output.txt

timvandermeij · 2024-04-18T12:44:45Z

@OrKoN The logs of the runs that crashed due to OOM can be found at http://54.241.84.105:8877/9e74d8b32fd9a8a/output.txt for the first try and at http://54.241.84.105:8877/17e8528589e75e3/output.txt for the second try. Both crashed during the reference tests. For completeness I'll also include the relevant bit here (the TEST-PASS line is the last bit of the PDF.js code):

TEST-PASS | eq test issue13751 | in firefox

<--- Last few GCs --->

[4032:0x6eba470]   422579 ms: Scavenge (reduce) 4087.8 (4127.6) -> 4087.9 (4128.6) MB, 2.1 / 0.0 ms  (average mu = 0.145, current mu = 0.047) allocation failure; 
[4032:0x6eba470]   422585 ms: Scavenge (reduce) 4088.8 (4128.6) -> 4088.7 (4129.3) MB, 3.1 / 0.0 ms  (average mu = 0.145, current mu = 0.047) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb7a940 node::Abort() [node]
 2: 0xa8e823  [node]
 3: 0xd5c940 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd5cce7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf3a3e5  [node]
 6: 0xf3b2e8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xf4b7f3  [node]
 8: 0xf4c668 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf26fce v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xf28397 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
11: 0xf088e0 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
12: 0xf00354 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node]
13: 0xf02608 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node]
14: 0x1042959 v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [node]
15: 0x1044676 v8::internal::JsonParser<unsigned short>::ParseJsonValue() [node]
16: 0x104515f v8::internal::JsonParser<unsigned short>::ParseJson() [node]
17: 0xde2523 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [node]
18: 0x16fb7b9  [node]
[11:30:57] 'runBotTest' errored after 8.62 min
[11:30:57] Error: Running browser tests failed.
    at ChildProcess.<anonymous> (file:///home/ubuntu/pdfjs/botio-files-pdfjs/private/17e8528589e75e3/gulpfile.mjs:731:15)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess.emit (node:domain:552:15)
    at maybeClose (node:internal/child_process:1091:16)
    at ChildProcess._handle.onexit (node:internal/child_process:302:5)
    at Process.callbackTrampoline (node:internal/async_hooks:130:17)

calixteman · 2024-04-18T13:11:36Z

/botio-linux test

moz-tools-bot · 2024-04-18T13:11:38Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/4340ac09990c704/output.txt

moz-tools-bot · 2024-04-18T13:21:01Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/4340ac09990c704/output.txt

Total script time: 9.37 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.241.84.105:8877/4340ac09990c704/reftest-analyzer.html#web=eq.log

calixteman · 2024-07-07T15:21:46Z

/botio test

moz-tools-bot · 2024-07-07T15:21:48Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/35bc713481f461b/output.txt

moz-tools-bot · 2024-07-07T15:21:48Z

From: Bot.io (Windows)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/85e50f48a0d2c83/output.txt

moz-tools-bot · 2024-07-07T15:51:26Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/35bc713481f461b/output.txt

Total script time: 29.63 mins

Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

  different ref/snapshot: 17
  different first/second rendering: 2

Image differences available at: http://54.241.84.105:8877/35bc713481f461b/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2024-07-07T16:06:15Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/85e50f48a0d2c83/output.txt

Total script time: 44.45 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

  different ref/snapshot: 4

Image differences available at: http://54.193.163.58:8877/85e50f48a0d2c83/reftest-analyzer.html#web=eq.log

timvandermeij · 2024-07-07T16:17:59Z

/botio test

moz-tools-bot · 2024-07-07T16:18:01Z

From: Bot.io (Windows)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.193.163.58:8877/9971aad506fbf14/output.txt

moz-tools-bot · 2024-07-07T16:18:01Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.241.84.105:8877/3327169103eb2ee/output.txt

moz-tools-bot · 2024-07-07T16:47:51Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/3327169103eb2ee/output.txt

Total script time: 29.82 mins

Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

  different ref/snapshot: 11

Image differences available at: http://54.241.84.105:8877/3327169103eb2ee/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2024-07-07T17:02:09Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/9971aad506fbf14/output.txt

Total script time: 44.12 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

  different ref/snapshot: 5
  different first/second rendering: 1

Image differences available at: http://54.193.163.58:8877/9971aad506fbf14/reftest-analyzer.html#web=eq.log

timvandermeij · 2024-07-07T17:03:58Z

/botio test

moz-tools-bot · 2024-07-07T17:04:00Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.241.84.105:8877/f7449f640d6e3fc/output.txt

moz-tools-bot · 2024-07-07T17:04:00Z

From: Bot.io (Windows)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.193.163.58:8877/217b665cb8096c4/output.txt

moz-tools-bot · 2024-07-07T17:33:39Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/f7449f640d6e3fc/output.txt

Total script time: 29.65 mins

Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

  different ref/snapshot: 15

Image differences available at: http://54.241.84.105:8877/f7449f640d6e3fc/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2024-07-07T17:47:28Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/217b665cb8096c4/output.txt

Total script time: 43.46 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

  different ref/snapshot: 5

Image differences available at: http://54.193.163.58:8877/217b665cb8096c4/reftest-analyzer.html#web=eq.log

timvandermeij · 2024-07-07T17:51:03Z

/botio test

moz-tools-bot · 2024-07-07T17:51:05Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.241.84.105:8877/7f3f5e033f0a2be/output.txt

moz-tools-bot · 2024-07-07T17:51:05Z

From: Bot.io (Windows)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.193.163.58:8877/56b1f221162c049/output.txt

moz-tools-bot · 2024-07-07T18:20:51Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/7f3f5e033f0a2be/output.txt

Total script time: 29.75 mins

Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

  different ref/snapshot: 14
  different first/second rendering: 2

Image differences available at: http://54.241.84.105:8877/7f3f5e033f0a2be/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2024-07-07T18:34:54Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/56b1f221162c049/output.txt

Total script time: 43.80 mins

Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

  different ref/snapshot: 4

Image differences available at: http://54.193.163.58:8877/56b1f221162c049/reftest-analyzer.html#web=eq.log

timvandermeij · 2024-07-07T18:37:19Z

/botio test

moz-tools-bot · 2024-07-07T18:37:22Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.241.84.105:8877/2c70720bd4ca7bf/output.txt

moz-tools-bot · 2024-07-07T18:37:22Z

From: Bot.io (Windows)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.193.163.58:8877/d00b709df3fcf0e/output.txt

moz-tools-bot · 2024-07-07T19:07:06Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/2c70720bd4ca7bf/output.txt

Total script time: 29.72 mins

Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

  different ref/snapshot: 12

Image differences available at: http://54.241.84.105:8877/2c70720bd4ca7bf/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2024-07-07T19:23:36Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/d00b709df3fcf0e/output.txt

Total script time: 46.24 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

  different ref/snapshot: 4

Image differences available at: http://54.193.163.58:8877/d00b709df3fcf0e/reftest-analyzer.html#web=eq.log

timvandermeij · 2024-07-07T20:09:18Z

It's really nice that after all this time we can now land this. Thanks everyone for helping out with this effort, because this now works due to a combination of fixes here and upstream in the Puppeteer project!

calixteman requested a review from timvandermeij April 17, 2024 19:10

timvandermeij added the test label Apr 17, 2024

timvandermeij reviewed Apr 17, 2024

View reviewed changes

test/test.mjs Show resolved Hide resolved

test/integration/scripting_spec.mjs Outdated Show resolved Hide resolved

test/integration/freetext_editor_spec.mjs Outdated Show resolved Hide resolved

calixteman force-pushed the chrome_bidi branch from c6c48e6 to c583cb9 Compare April 17, 2024 20:03

calixteman mentioned this pull request Apr 18, 2024

Use WebDriver BiDi for Chrome #17961

Open

calixteman force-pushed the chrome_bidi branch from c583cb9 to 743d6a3 Compare April 18, 2024 13:11

calixteman force-pushed the chrome_bidi branch from 54ed774 to 5bfe759 Compare July 7, 2024 15:21

timvandermeij approved these changes Jul 7, 2024

View reviewed changes

timvandermeij merged commit 1bdd692 into mozilla:master Jul 7, 2024
9 checks passed

npanchal108 mentioned this pull request Aug 31, 2024

[Snyk] Upgrade pdfjs-dist from 4.0.269 to 4.5.136 npanchal108/ecommercebase#5

Open

Use BiDi protocol for Chrome tests #17962

Use BiDi protocol for Chrome tests #17962

Conversation

calixteman commented Apr 17, 2024

calixteman commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Windows)

Received

timvandermeij left a comment

Choose a reason for hiding this comment

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Linux m4)

Success

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Windows)

Success

calixteman commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Windows)

Received

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Windows)

Failed

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Linux m4)

Failed

calixteman commented Apr 18, 2024

calixteman commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Windows)

Received

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Linux m4)

Failed

OrKoN commented Apr 18, 2024

OrKoN commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Windows)

Failed

calixteman commented Apr 18, 2024

calixteman commented Apr 18, 2024

timvandermeij commented Apr 18, 2024 • edited Loading

calixteman commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Linux m4)

Failed

calixteman commented Jul 7, 2024

moz-tools-bot commented Jul 7, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Jul 7, 2024

From: Bot.io (Windows)

Received

moz-tools-bot commented Jul 7, 2024

From: Bot.io (Linux m4)

Failed

moz-tools-bot commented Jul 7, 2024

From: Bot.io (Windows)

Failed

timvandermeij commented Jul 7, 2024

moz-tools-bot commented Jul 7, 2024

From: Bot.io (Windows)

Received

moz-tools-bot commented Jul 7, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Jul 7, 2024

From: Bot.io (Linux m4)

timvandermeij commented Apr 18, 2024 •

edited

Loading