-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Swiftlint Analyzer uses large amounts of memory #2812
Comments
Yup. Pretty wild. Might be worth moving to an on-disk database instead of a 100% in-memory representation to reduce memory usage. |
Hey @jpsim so, I dug into this a bit and it turns out SourceKitten is leaking a lot of memory. Take a look in the xcode memory graph debugger, you'll see leaked objects, they are named I don't believe this can be fixed without breaking changes to SourceKitten. Should be fine, since SourceKitten is still I think the best approach is to prevent the raw I created a hacked together branch that proof of concepts this fix. Wanted to make sure I could run swiftlint analyzer without crashes and without large amounts of memory being used. After this change memory usage dropped from 5GB for ~50 files to ~250MB. I think there is still another another leak, because there are are lots of xpc strings, and memory usage was still slowly climbing. My guess is it's the keys for the dictionaries, but that's just a guess and memory usage was much more reasonable. Also probably the root cause for: jpsim/SourceKitten#559 |
Amazing work @cltnschlosser ❤️ I think the other leak is "on purpose" where we cache byte-wise lookups into String since these are operations that we need to do so frequently: https://github.com/jpsim/SourceKitten/blob/0.23.2/Source/SourceKittenFramework/String+SourceKitten.swift#L71-L78 It'd be nice to find ways to speed up those operations sufficiently that we wouldn't need to maintain an indefinite cache. |
I moved all code that interacts with `sourcekitd_object_t` into `SourceKitObject.swift`. This way it's easier to manage the lifecycle of `sourcekitd_object_t` pointers. Relates to: realm/SwiftLint#2812 I think this could be simplified/improved even more by exposing SourceKitObject as a typealias for `[String: SourceKitObjectConvertible]`. Then when we make the request, converting to `sourcekitd_object_t` pointers, making the request, then freeing the pointers. This means that for the most part you're only working with swift types and the `sourcekitd_object_t` lifecycle is greatly shorted and simplified.
I used SourceKitten master with swiftlint master to do an analyze run of 4000+ swift file codebase. Memory usage stayed below 3GB the whole time :) Also it didn't crash 👍 Much better than the 100GB+ (most of which was swapped to disk) from before. |
Nice! Looking forward to integrating that into SwiftLint. |
New Issue Checklist
Describe the bug
Running swiflint analyzer over 4000 swift files, with both the
unused import
andunused private declaration
rules enabled. Currently ~800 files in and memory usage is over 80GB.I'll dig into this further later, to see if I can figure out why this is happening. Figured I'd create the issue now as a place to track the excessive memory usage.
Environment
SwiftLint version (run
swiftlint version
to be sure)? 0.33.1Installation method used (Homebrew, CocoaPods, building from source, etc)? Cocoapods
Are you using nested configurations? No
Which Xcode version are you using (check
xcode-select -p
)? 10.2.1The text was updated successfully, but these errors were encountered: