Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky tests #784

Merged
merged 3 commits into from
Sep 21, 2018
Merged

Fix flaky tests #784

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e640bcb
Add xcrun simctl list step
fathyb Aug 31, 2018
db1a2b4
Fix flaky maxQueueSize test
fathyb Aug 30, 2018
98375d3
Increase E2E test timeout to 3 minutes
fathyb Sep 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ jobs:
xcode: "9.4.1"
steps:
- checkout
- run: xcrun simctl list
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we still need this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I added xcrun simctl list we've miraculously never had the error again. If I remove it we get the missing simulator error sporadically. This could also help us debug it next time.

- run:
name: Install build dependencies
command: |
Expand Down
19 changes: 9 additions & 10 deletions AnalyticsTests/AnalyticsTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,19 +121,18 @@ class AnalyticsTests: QuickSpec {
let integration = analytics.test_integrationsManager()?.test_segmentIntegration()
expect(integration).notTo(beNil())

var sent = 0

analytics.flush()
integration?.test_dispatchBackground {
if let count = integration?.test_queue()?.count {
sent = count
}
else {
sent = -1
waitUntil(timeout: 60) {done in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the timeout to check for the messages in webhook? if so, you may want to increase it to 3 minutes as 1 minute is sometimes not enough to find them in webhook (3 mins is what library-e2e-tester currently does).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't this one, but I pushed a new commit to increase the webhook timeout to 3 minutes - thanks!

let queue = DispatchQueue(label: "test")

queue.async {
while(integration?.test_queue()?.count != max) {
sleep(1)
}

done()
}
}

expect(sent).toEventually(equal(max))
}

it("protocol conformance should not interfere with UIApplication interface") {
Expand Down
4 changes: 2 additions & 2 deletions AnalyticsTests/EndToEndTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,9 @@ class AnalyticsE2ETests: QuickSpec {

analytics.track("E2E Test", properties: ["id": uuid])

self.waitForExpectations(timeout: 20)
self.waitForExpectations(timeout: 3 * 60)

for _ in 1...5 {
for _ in 1...3 * 30 {
sleep(2)
if hasMatchingId(messageId: uuid) {
return
Expand Down