account doesn't work on macOS 12+

[lukasmalkmus]

Your Environment

• mas version: 1.8.3
• macOS version (system_profiler SPSoftwareDataType -detailLevel mini): 12.0 (21A5522h)

mas Install Method

• brew install mas (homebrew-core)
• [] mas-cli/tap
• [] .pkg installer from releases
• [] Built from source
  • Fork/branch: ? (e.g. mas-cli/main)
  • Xcode version: 10.?

Describe the Bug

mas account says I'm not signed in, although I am:

Not signed in
Error: Not signed in

To Reproduce

Steps to reproduce the behavior:

1. Run mas account
2. Make sure App Store has a signed in Apple ID
3. Run mas account again

Expected Behavior

Getting no error and being able to use other mas commands that depend on being signed din.

Actual Behavior

Not signed in
Error: Not signed in

Screenshots, Terminal Output

If applicable, add screenshots to help explain your problem.

$ mas account
Not signed in
Error: Not signed in

Additional Context

It stopped working around 2 weeks ago I guess? Maybe it worked with the previous version?

[malob]

Same here (with same macOS Monterey version). I noticed to it stopped working after it updated to Beta 7, mas had been working on previous betas without issue.

[chris-araman]

I can reproduce this. I am also running macOS Monterey beta 7.

@lukasmalkmus, @malob, do either or both of you see this when iCloud Private Relay is enabled? Is it reproducible with iCloud Private Relay disabled?

[malob]

@chris-araman, unfortunately I’m now away from my macOS machine for a few days, so won’t be able to test this until Thursday.

[phatblat]

I'm able to reproduce this with Private Relay disabled in macOS 12 beta 7.

Guessing this is due to an API change in Monterey I ran our update_headers script, and it appears class-dump can't find the binary.

↪ script/update_headers
/Users/phatblat/bin/class-dump
class-dump: Input file (/System/Library/PrivateFrameworks/CommerceKit.framework/CommerceKit) does not exist.
class-dump: Input file (/System/Library/PrivateFrameworks/StoreFoundation.framework/StoreFoundation) does not exist.

Dumper agrees that these frameworks are missing the binary. I wonder if Monterey has somehow hid them from our prying eyes 👀.

[chris-araman]

@phatblat, those binaries don't exist in the file system in Big Sur either, due to the dyld cache. There is some sample code from Apple (search for dsc_extractor.bundle) that can extract binaries from the cache.

That said, class-dump doesn't seem to be able to handle recent x86-64/x86-64h binaries extracted this way, and definitely doesn't handle arm64/arm64e binaries. Maybe Dumper does?

This is what I was trying to address for Apple Silicon Macs when I last contributed to class-dump. 😃

[phatblat]

I ran across Extracting libraries from dyld_shared_cache and was impressed with his/her understanding of the process. Using https://github.com/zhuowei/dsc_extractor_badly I was able to get a CommerceKit x86_64 binary. Neither class-dump nor Dumper were able to work with it, but IDA Free is able to understand this file.

I'll try updating the headers manually and then see which new methods might return the active account.

[tiiiecherle]

I have the same problem on the latest macOS 12 beta. As I am running tests for my install scripts in the moment it would be very nice to see this fixed.
Thanks in advance.

[ankushnarula]

Same problem with Release Candidate - macOS 12.0 Beta (21A5552a)

[cherryblossom000]

As Monterey has been released, it would be nice if mas supports it (fully).

(I understand it takes a lot of work to play around with private frameworks and fix code for macOS upgrades. Keep up the good work — mas is a very useful tool!)

[0xmachos]

I can confirm the same issue on macOS Monterey (12.0.1) (21A559)

$ mas version 
1.8.3

$ mas account 
Not signed in
Error: Not signed in

[NSBrianWard]

Same issue on macOS 12.0.1, 16" MacBook Pro 2021.

[SecKatie]

Same issue on macOS 12.0.1, 13" MacBook Pro 2020 M1

[luisnabais]

Same on macOS 12.0.1, 15.4" MacBook Pro 2018.

1.8.3
❯ mas account
Not signed in
Error: Not signed in```

[ghost]

Same here.

[ghost]

Hey Team,

This replicates on macOS 12.0.1 (21A559), MacBook Air (M1, 2020). Let me know if you need any specific tests/checks.

[StrangeRanger]

Same here

[jimscard]

Repros on 12.1 Beta as well.

[Drallas]

@chris-araman @phatblat
Any news if this issue is being adressed?
Is there a workarround?
How can we help?

[chris-araman]

Hi, all!

Any news if this issue is being addressed?

You're in the right place. Efforts toward resolving this issue will be tracked here. @phatblat and I are both volunteer maintainers with obligations apart from mas maintenance. We are not paid (though I do welcome sponsorship). We do this because we love open source and want to give back to the community. We are also mas users.

Is there a workarround?

We're not aware of a workaround at this time, other than using the App Store app, of course.

How can we help?

As you're probably aware, Apple has made some changes to the private frameworks mas uses to manage App Store apps on your Mac. Complicating things, the class-dump tool that was used by the original mas author to generate Objective-C headers from those private frameworks does not support extracting dylibs from a cache, and does not appear to support the x86_64h, arm64, or arm64e architectures. The original mas author is no longer involved in the mas or class-dump projects, and class-dump appears to have been abandoned.

A few things have to happen in order to resolve this issue:

• We need tooling to generate Objective-C headers from current CommerceKit.framework and StoreFoundation.framework. @phatblat has already done some investigation toward this effort, but there's more to do.
• This will probably require tooling to extract the binary modules of those frameworks from the dylib cache that was introduced in macOS 11 Big Sur. There is some sample dsc_extractor code available from Apple that seems to work well enough for this, but it isn't (yet?) maintained as a standalone tool. This snippet may have already been built into some third-party tools, so it might not be necessary to publish it as a standalone tool.
• We may or may not need to find or build tooling to support extracting modern arm64/arm64e binaries or generating headers from them.
• We then need to figure out what has changed in these private frameworks in order to determine whether there might be a new path to mas functionality. This may involve making new calls into these frameworks, or dumping headers for some additional framework(s). This one is the trickiest to estimate effort involved. It could be a one-liner, or it could be very involved, or it could be infeasible.

tl;dr: This is going to take time and effort. We welcome any helpful contributions from subject matter experts!

That said, additional reports of mas errors on Monterey aren't necessary or helpful right now. Please do try to minimize the noise on this thread while we work to solve the problem. We appreciate it!

[zcutlip]

When you use dsc_extractor to dump a dylib from the shared cache, it doesn't really give you a proper dylib. Lots of things get rewritten during the linking process that generates the shared cache, and dsc_extractor doesn't undo most of that. So, you mostly get a mach-o dylib that, at best, superficially resembles the original, but is mostly broken internally. @phatblat, you probably found lots of broken xrefs in IDA.

I'm not sure what the specific issues are with class-dump but I wouldn't be surprised if that's part of it.

Afaik, IDA still doesn't support the new dyld shard cache format in Monterey (split across several slices), but I have tooling that allows me to re-join the slices. I can then load it up and disassemble individual dylibs cleanly, for the Apple Silicon version (not the intel version, yet).

I don't know much about class-dump but if there exists (or someone wants to write) an IDA python script that replicates what it does, I can give it a try and share its output. Or really any IDA script that might give you useful info.

[zcutlip]

Regarding dyld_shared_cache, here are a couple resources for informational purposes. Neither of these help move the ball down the field, but hopefully they offer a bit of insight into the problem space.

There's a project called DyldExtractor which unfortunately doesn't work on macOS shared caches; it's iOS only. But looking through it can give you some insight into how hard a problem it is to reconstruct the dylibs. It's a pure python project that does not use dsc_extractor. It parses the shared cache and attempts to reconstruct the dylibs as faithfully as possible.
https://github.com/arandomdev/DyldExtractor

And here's a project that essentially dlopen()s dsc_extractor from Xcode to extract individual dylibs from the shared cache. Since it relies on Xcode's dsc_extractor it does support the new split format, but that also means it spits out broken dylibs. This doesn't get you any farther that what @phatblat probably has done, but just linking here as a generally useful tool.
https://github.com/keith/dyld-shared-cache-extractor

cheers,
zach

[chris-araman]

...I have tooling that allows me to re-join the slices. I can then load it up and disassemble individual dylibs cleanly...

Any chance you'd want to open source that? 😁 What are you using for disassembly?

FWIW, even the NSA doesn't seem to be able to grock Monterey's dylib_shared_cache yet.

[zcutlip]

Unfortunately I can't share the tool since it's work-related

As for disassembly, I'm using IDA to load the reconstructed dyld_shared_cache, and then disassemble individual dylibs. So, happy to run any IDA python scripts against my IDB and share the results if that helps.

And yeah I filed that bug on Ghidra :-)

[chris-araman]

@zcutlip, if you're able to somehow generate headers for these frameworks on macOS 12.0.1 and either post a link to them here, or fork and drop them in a branch for review, it'd be greatly appreciated. Not sure if that's possible given your current tooling. I'm not yet familiar with IDA, let alone Python scripting for it, unfortunately.

[blacktop]

Okay, got method selector strings resolved. Still CommerceKit. Haven't gotten to StoreKit yet. ida-output.txt

Remaining todo:

• Resolve type strings to actual parameter and return types
• Properties
• ivars
• ObjcExplore has a TODO for class methods for protocols, but I haven't seen any so far in IDA. Are they uncommon?
• what else?

sorry for jumping in, but have you tried my tool ipsw: ipsw dyld macho DSC StoreKit --objc ? Add the -V flag to try and decode the types.

[zcutlip]

I'm familiar with your tool, but wasn't aware of it's dsc/obj-c capabilities. Giving it at try now. Thanks!

[blacktop]

I'm familiar with your tool, but wasn't aware of it's dsc/obj-c capabilities. Giving it at try now. Thanks!

If you have bat installed (brew install bat) you can pipe the output of my tool to: bat -l m --tabs 0 -p --theme Nord --wrap=never --pager "less -S" to make it look amazing.

[zcutlip]

Yeah that's really cool. This gets us really close. I'll see if I can clean up the output to be a proper header file.

Very nice work on ipsw

[zcutlip]

Okay, making progress:

I'm dumping JSON from IDA pro, and I've written some python that shells out to classdump's formatType utility, to convert encoded type strings into method signatures.

I've attached a gist with JSON for CommerceKit as well as C-header formatted output from post-processing the JSON. There's definitely stuff missing, like ivars and properties, and probably a few other things. Also I'm not sure what magic class dump does to organize it into individual header files. For now it's all one big .h file.

Someone maybe take a look at let me know if we're getting close
https://gist.github.com/zcutlip/c74ceec2e818c630e655ee4c8f408878

Cheers

[zcutlip]

Oh, I think I get the organization. It's supposed to be one .h per protocol or class, is that right?

[zcutlip]

I'm now writing out individual .h files per protocol & class, but I'm now realizing there's a lot more to it. I'm currently dumping about 90-ish header files from CommerceKit alone, because there are lots of class and protocol structures in the binary that don't need to be dumped

I studied what class-dump does, and it's quite sophisticated. It actually builds a dependency graph and then is able to do things like add forward declarations and imports as well as work out what protocols & classes it doesn't need to dump. I don't think I'll be able to get to that level of refinement.

My proposals are:

1. Dump out JSON and let others work on a tooling to ingest it (either by modifying class-dump or some other way), or
2. Dump header files as best I can, and let others refine those header files and discard what they don't need

Here's an example of one of the header files I dumped, CKAuthenticationSettings.h:

@interface CKAuthenticationSettings : NSObject
{
}

// Class methods
+ (_Bool)supportsSecureCoding

// Instance methods
- (id)copyWithZone:(struct _NSZone *)arg1
- (id)init
- (void)encodeWithCoder:(id)arg1
- (id)initWithCoder:(id)arg1
- (void).cxx_destruct
- (id)_password
- (_Bool)showHelp
- (_Bool)createSession
- (void)setShowHelp:(_Bool)arg1
- (id)suggestedUsername
- (void)setSuggestedUsername:(id)arg1
- (id)authenticateArguments
- (void)setAuthenticateArguments:(id)arg1
- (id)createAccountArguments
- (void)setCreateAccountArguments:(id)arg1
- (_Bool)forceAccount
- (void)setForceAccount:(_Bool)arg1
- (void)setCreateSession:(_Bool)arg1
- (void)set_password:(id)arg1
- (_Bool)_loginToiCloud
- (void)set_loginToiCloud:(_Bool)arg1


@end  /* CKAuthenticationSettings */

I still need to do ivars and a couple other things, but that shouldn't be too bad.

Cheers

[chris-araman]

Someone maybe take a look at let me know if we're getting close

I haven't found time to dive into this gist yet, but I'm hoping to do so this week. Hopefully there are enough breadcrumbs here to piece together a plan for the original issue reported here, among others.

It's supposed to be one .h per protocol or class, is that right?

Yep!

My proposals are:

1. Dump out JSON and let others work on a tooling to ingest it (either by modifying class-dump or some other way), or
2. Dump header files as best I can, and let others refine those header files and discard what they don't need

Option 1 would probably be best in the long run if you were able to publish your IDA scripts as open source. That way, we could try to integrate the functionality directly into class-dump. If you aren't able to do so, let's go for the quicker, stop-gap progress of option 2.

Great work on this, @zcutlip. I really appreciate the time and effort you're putting into this, and I'm sure others here do too.

[chris-araman]

ObjcExplore has a TODO for class methods for protocols, but I haven't seen any so far in IDA. Are they uncommon?

It looks like several of the interfaces used by mas have class methods used to expose shared singletons.

I took a look at your progress, and I agree we're getting close. Your to-do list looks accurate. I haven't noticed anything else we'd need from a class dumper.

(To be clear, though, Apple still may not have left enough surface area to fix the account, signin, or purchase commands.)

[zcutlip]

I'll absolutely share the IDA plugin. I'm in the process of doing the following:

• Cleaning up the plugin so it's easier to use, gives you a choice of frameworks/dylibs to dump, etc.
• adding ivars, properties, etc. to the JSON
• rolling up a zip file with all the JSON and all the individual header files I'm able to generate

At that point, based on feedback,I can revise the JSON output to include any additional details as required

[zcutlip]

Got ivars going. Here's a sample

/*
 * CKAccountStore.h
 * ./classdump.py ../build/Release/formatType ./CommerceKit/ ../../IDA-ObjCExplorer/CommerceKit.json
 */


@interface CKAccountStore
{
    CKStoreClient *_storeClient _storeClient;
}

// Class methods
+ (id)accountStoreForStoreClient:(id)arg1;
+ (id)sharedAccountStore;

// Instance methods
- (void).cxx_destruct;
- (id)accounts;
- (id)primaryStoreAccount;
- (void)addAccount:(id)arg1;
- (id)primaryAccount;
- (id)accountWithAppleID:(id)arg1;
- (_Bool)isDemoModeEnabled;
- (id)initWithStoreClient:(id)arg1;
- (id)demoAccount;
- (id)storeClient;
- (id)storeAccountForDSID:(id)arg1;
- (id)addAccountObserver:(id)arg1;
- (id)_initWithStoreClient:(id)arg1;
- (id)knownAccounts;
- (void)signOutWithCompletionHandler:(id /* CDUnknownBlockType */)arg1;
- (void)removeAccountObserver:(id)arg1;
- (id)storeAccountForAppleID:(id)arg1;
- (void)getEligibilityForService:(long long)arg1 completionBlock:(id /* CDUnknownBlockType */)arg2;
- (_Bool)primaryAccountIsPresentAndSignedIn;
- (void)signOut;
- (id)addPrimaryAccountObserverWithBlock:(id /* CDUnknownBlockType */)arg1;
- (void)removePrimaryAccountObserver:(id)arg1;
- (id)accountForDSID:(id)arg1;
- (void)signIn;
- (void)signInWithSuggestedAppleID:(id)arg1 allowChangeOfAppleID:(_Bool)arg2 completionHandler:(id /* CDUnknownBlockType */)arg3;
- (void)viewAccount;
- (id)eligibilityForService:(long long)arg1;
- (void)getPasswordSettingsWithCompletionBlock:(id /* CDUnknownBlockType */)arg1;
- (void)updatePasswordSettings:(id)arg1 completionBlock:(id /* CDUnknownBlockType */)arg2;
- (void)setTouchIDStateForAccount:(id)arg1 state:(long long)arg2 completionBlock:(id /* CDUnknownBlockType */)arg3;
- (void)getTouchIDStateForAccount:(id)arg1 completionBlock:(id /* CDUnknownBlockType */)arg2;


@end  /* CKAccountStore */

I haven't looked at properties yet, but ivars weren't too bad. Hopefully properties will be equally straightforward.

I'm attaching a zip file of all the headers I've spit out so far, along with the JSON I spit out from IDA.

Cheers

EDIT: Note, many/most of the headers in the zip file are superfluous, but as mentioned previously, I'm not sure of a way to programmatically filter out the ones that don't need to be generated. So for now they're all in there, and we can work on a better way next.

commercekit.zip

[chris-araman]

I don't see CKDownloadDirectory or CKDownloadQueueObserver in the .zip. Are they visible to your script? What about those types mas uses from StoreFoundation?

[zcutlip]

I haven't started StoreFoundation yet. I have a separate IDB for it. I wanted to get the overall process working first,

I'll check on CKDownloadDirectory & CKDownloadQueueObserver in my CommerceKit IDB and figure out why they aren't being dumped.

[zcutlip]

It appears CKDownloadDirectory is just a standard C function exported by CommerceCore.framework (see attached screenshot). I won't be able to get normal exported functions by walking the objective-C structures. I'll have to walk the exports to get those.

As far as CKDownloadQueueObserver, I don't find any reference to it in the IDB. The closest thing there is is DownloadQueueObserver, for which there is a header file in the zip file attached above. Not sure if that's related or not.

[chris-araman]

CKDownloadQueueObserver is passed as arg1 to CKDownloadQueue.addObserver and removeObserver, and it looks like they're held in downloadQueueObservers. Maybe your work on properties will expose how to find that type. Perhaps it isn't exported because it's a callback/delegate protocol meant to be implemented by the caller.

[zcutlip]

Forgot to actually attach screenshot previously:

[zcutlip]

I should add that I'm happy to share my IDB if it's helpful. It's quite large though, around 18GB, so I'm not sure the best way to do that.

[chris-araman]

@zcutlip, thanks for the offer. I think we could rebuild an IDB if needed. Tooling to generate new headers will be more valuable than a one-time snapshot in the long run.