Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add integration tests #535

Merged
merged 39 commits into from
Aug 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
ea58ebf
Add accessibility labels
bamx23 Jul 21, 2024
13dfb9b
Support install path override
bamx23 Jul 21, 2024
3a76e69
Add integration test
bamx23 Jul 21, 2024
38ad1d1
Add integration test CI job
bamx23 Jul 21, 2024
ee4fd04
Wait for crash
bamx23 Jul 21, 2024
03918c0
Do not CI tvOS
bamx23 Jul 21, 2024
6fbeb29
Assert Apple-formatted report
bamx23 Jul 21, 2024
631134e
Remove visionOS part
bamx23 Jul 21, 2024
cbb3ee2
Extract integration tests to a separate workflow
bamx23 Aug 3, 2024
962760a
Add testplan and fix scheme
bamx23 Aug 3, 2024
24c7c72
Lower repetitions number
bamx23 Aug 3, 2024
1385df0
Wait for install button
bamx23 Aug 3, 2024
f3438ef
Add some swifty sugar
bamx23 Aug 3, 2024
fb05c2e
Use directory sink for tests
bamx23 Aug 3, 2024
6b53f95
Add other crash types
bamx23 Aug 3, 2024
76ea91c
Fix watchOS
bamx23 Aug 3, 2024
40ee6a2
Split helper and crash triggers
bamx23 Aug 10, 2024
4838045
Add crash search and use it in tests
bamx23 Aug 10, 2024
2ec6b24
Just use scroll instead of search
bamx23 Aug 10, 2024
2bcc173
Use a different second test
bamx23 Aug 10, 2024
c51982a
Fix format
bamx23 Aug 10, 2024
2ee6540
Run tests in parallel
bamx23 Aug 10, 2024
e93316f
Remove typealias
bamx23 Aug 10, 2024
14ce43b
Limit number of scrolls
bamx23 Aug 10, 2024
3ebd3e9
Fix throws
bamx23 Aug 10, 2024
99d1986
Swipe slowly
bamx23 Aug 10, 2024
9c3300b
Even slower?
bamx23 Aug 10, 2024
5aa9a19
Add generic crashed thread validation
bamx23 Aug 10, 2024
b6a3707
Add mach exception test
bamx23 Aug 10, 2024
ee36794
Fix C++ test by enabling cxa_throw swap
bamx23 Aug 10, 2024
155a7f5
Fix toggle logic
bamx23 Aug 10, 2024
1f63cdb
Use env + script instead of UI elements
bamx23 Aug 11, 2024
bd862f3
Test Mach only on iOS
bamx23 Aug 11, 2024
5252ae8
Add tvOS test target
bamx23 Aug 11, 2024
fffa488
Address comments
bamx23 Aug 11, 2024
1a49e4c
Add platform-specific action delay
bamx23 Aug 11, 2024
f6bfde6
Add macOS
bamx23 Aug 11, 2024
362baa8
Remove test UI ids
bamx23 Aug 11, 2024
146b749
Use manual run for most changes
bamx23 Aug 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: Run Integration Tests

on:
workflow_dispatch:
pull_request:
paths:
- 'Samples/Tests/**'
- 'Samples/Common/Sources/IntegrationTestsHelper/**'
- '.github/workflows/integration-tests.yml'
push:
branches:
- master
schedule:
- cron: '0 0 1 * *'

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
integration-tests:
runs-on: macos-latest
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
- platform: iOS
- platform: macOS
- platform: tvOS
- platform: watchOS
steps:
- name: Checkout Code
uses: actions/checkout@v4

- name: Use Latest Stable Xcode
uses: maxim-lobanov/setup-xcode@v1
with:
xcode-version: latest-stable

- name: Install Tuist
run: |
brew tap tuist/tuist
brew install tuist@4.18.0

- name: Generate Project
working-directory: ./Samples
run: tuist generate --verbose --no-open

- name: Run Build
uses: mxcl/xcodebuild@v3.0.0
with:
action: test
workspace: "Samples/KSCrashSamples.xcworkspace"
scheme: "Sample"
platform: ${{ matrix.platform }}
14 changes: 14 additions & 0 deletions Samples/Common/Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ let package = Package(
name: "CrashTriggers",
targets: ["CrashTriggers"]
),
.library(
name: "IntegrationTestsHelper",
targets: ["IntegrationTestsHelper"]
),
.library(
name: "SampleUI",
targets: ["SampleUI"]
Expand All @@ -40,11 +44,21 @@ let package = Package(
.target(
name: "CrashTriggers"
),
.target(
name: "IntegrationTestsHelper",
dependencies: [
.target(name: "CrashTriggers"),
.product(name: "Recording", package: "KSCrash"),
.product(name: "Reporting", package: "KSCrash"),
.product(name: "Logging", package: "swift-log"),
]
),
.target(
name: "SampleUI",
dependencies: [
.target(name: "LibraryBridge"),
.target(name: "CrashTriggers"),
.target(name: "IntegrationTestsHelper"),
.product(name: "Logging", package: "swift-log"),
]
)
Expand Down
86 changes: 86 additions & 0 deletions Samples/Common/Sources/CrashTriggers/KSCrashTriggersHelper.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
//
// KSCrashTriggersHelper.mm
//
// Created by Nikolay Volosatov on 2024-08-10.
//
// Copyright (c) 2012 Karl Stenerud. All rights reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall remain in place
// in this source code.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
//

#import "KSCrashTriggersHelper.h"
#import "KSCrashTriggersList.h"

@implementation KSCrashTriggersHelper

+ (NSArray<NSString *> *)groupIds
{
return @[
#define __PROCESS_GROUP(GROUP, NAME) @ #GROUP,
__ALL_GROUPS
#undef __PROCESS_GROUP
];
}

+ (NSString *)nameForGroup:(NSString *)groupId
{
#define __PROCESS_GROUP(GROUP, NAME) \
if ([groupId isEqualToString:@ #GROUP]) { \
return NAME; \
}
__ALL_GROUPS
#undef __PROCESS_GROUP
return @"Unknown";
}

+ (NSArray<KSCrashTriggerId> *)triggersForGroup:(NSString *)groupId
{
NSMutableArray<KSCrashTriggerId> *result = [NSMutableArray array];
#define __PROCESS_TRIGGER(GROUP, ID, NAME) \
if ([groupId isEqualToString:@ #GROUP]) { \
[result addObject:TRIGGER_ID(GROUP, ID)]; \
}
__ALL_TRIGGERS
#undef __PROCESS_TRIGGER
return result;
}

+ (NSString *)nameForTrigger:(KSCrashTriggerId)triggerId
{
#define __PROCESS_TRIGGER(GROUP, ID, NAME) \
if ([triggerId isEqualToString:TRIGGER_ID(GROUP, ID)]) { \
return NAME; \
}
__ALL_TRIGGERS
#undef __PROCESS_TRIGGER
return @"Unknown";
}

+ (void)runTrigger:(KSCrashTriggerId)triggerId
{
#define __PROCESS_TRIGGER(GROUP, ID, NAME) \
if ([triggerId isEqualToString:TRIGGER_ID(GROUP, ID)]) { \
[KSCrashTriggersList trigger_##GROUP##_##ID]; \
return; \
}
__ALL_TRIGGERS
#undef __PROCESS_TRIGGER
}

@end
82 changes: 82 additions & 0 deletions Samples/Common/Sources/CrashTriggers/KSCrashTriggersList.mm
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
//
// KSCrashTriggersList.mm
//
// Created by Nikolay Volosatov on 2024-06-23.
//
// Copyright (c) 2012 Karl Stenerud. All rights reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall remain in place
// in this source code.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
//

#import "KSCrashTriggersList.h"

#import <mach/mach.h>
#import <signal.h>
#import <stdexcept>

@implementation KSCrashTriggersList

+ (void)trigger_nsException_genericNSException
{
NSException *exc = [NSException exceptionWithName:NSGenericException reason:@"Test" userInfo:@{ @"a" : @"b" }];
[exc raise];
}

+ (void)trigger_nsException_nsArrayOutOfBounds
{
NSArray *array = @[ @1, @2, @3 ];
[array objectAtIndex:10]; // This will throw an NSRangeException
}

+ (void)trigger_cpp_runtimeException
{
throw std::runtime_error("C++ exception");
}

+ (void)trigger_mach_badAccess
{
volatile int *ptr = (int *)0x42;
*ptr = 42; // This will cause an EXC_BAD_ACCESS (SIGSEGV)
}

+ (void)trigger_mach_busError
{
char *ptr = (char *)malloc(sizeof(int));
int *intPtr = (int *)(ptr + 1); // Misaligned pointer
*intPtr = 42; // This will cause an EXC_BAD_ACCESS (SIGBUS)
free(ptr);
}

+ (void)trigger_mach_illegalInstruction
{
void (*funcPtr)() = (void (*)())0xDEADBEEF;
funcPtr(); // This will cause an EXC_BAD_INSTRUCTION
}

+ (void)trigger_signal_abort
{
abort(); // This will raise a SIGABRT signal
}

+ (void)trigger_other_stackOverflow
{
[self trigger_other_stackOverflow];
}

@end
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
//
// KSCrashTriggersHelper.h
//
// Created by Nikolay Volosatov on 2024-08-10.
//
// Copyright (c) 2012 Karl Stenerud. All rights reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall remain in place
// in this source code.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
//

#import "KSCrashTriggersList.h"

NS_ASSUME_NONNULL_BEGIN

typedef NSString *KSCrashTriggerId NS_TYPED_ENUM NS_SWIFT_NAME(CrashTriggerId);
#define TRIGGER_ID(GROUP, ID) KSCrashTriggerId_##GROUP##_##ID
#define __PROCESS_TRIGGER(GROUP, ID, NAME) \
static KSCrashTriggerId const TRIGGER_ID(GROUP, ID) NS_SWIFT_NAME(GROUP##_##ID) = @"trigger-" #GROUP "-" #ID;
__ALL_TRIGGERS
#undef __PROCESS_TRIGGER

NS_SWIFT_NAME(CrashTriggersHelper)
@interface KSCrashTriggersHelper : NSObject

+ (NSArray<NSString *> *)groupIds;
+ (NSString *)nameForGroup:(NSString *)groupId;

+ (NSArray<KSCrashTriggerId> *)triggersForGroup:(NSString *)groupId;
+ (NSString *)nameForTrigger:(KSCrashTriggerId)triggerId;

+ (void)runTrigger:(KSCrashTriggerId)triggerId;

@end

NS_ASSUME_NONNULL_END
57 changes: 57 additions & 0 deletions Samples/Common/Sources/CrashTriggers/include/KSCrashTriggersList.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
//
// KSCrashTriggersList.h
//
// Created by Nikolay Volosatov on 2024-06-23.
//
// Copyright (c) 2012 Karl Stenerud. All rights reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall remain in place
// in this source code.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
//

#import <Foundation/Foundation.h>

NS_ASSUME_NONNULL_BEGIN

#define __ALL_GROUPS \
__PROCESS_GROUP(nsException, @"NSException") \
__PROCESS_GROUP(cpp, @"C++") \
__PROCESS_GROUP(mach, @"Mach") \
__PROCESS_GROUP(signal, @"Signal") \
__PROCESS_GROUP(other, @"Other")

#define __ALL_TRIGGERS \
__PROCESS_TRIGGER(nsException, genericNSException, @"Generic NSException") \
__PROCESS_TRIGGER(nsException, nsArrayOutOfBounds, @"NSArray out-of-bounds") \
__PROCESS_TRIGGER(cpp, runtimeException, @"Runtime Exception") \
__PROCESS_TRIGGER(mach, badAccess, @"EXC_BAD_ACCESS (SIGSEGV)") \
__PROCESS_TRIGGER(mach, busError, @"EXC_BAD_ACCESS (SIGBUS)") \
__PROCESS_TRIGGER(mach, illegalInstruction, @"EXC_BAD_INSTRUCTION") \
__PROCESS_TRIGGER(signal, abort, @"Abort") \
__PROCESS_TRIGGER(other, stackOverflow, @"Stack overflow")

NS_SWIFT_NAME(CrashTriggersList)
@interface KSCrashTriggersList : NSObject

#define __PROCESS_TRIGGER(GROUP, ID, NAME) +(void)trigger_##GROUP##_##ID;
__ALL_TRIGGERS
#undef __PROCESS_TRIGGER

@end

NS_ASSUME_NONNULL_END
Loading