Richer IDL generation with compilation

[kklas]

If we have a, for example, Program<'info, SomeProgram> constraint in the accounts, we know that the account with the SomeProgram::id() pubkey needs to be passed in, but we're not able to leverage this information from the clients because it's not encoded within the IDL (aside from the hack in anchor ts where we match by account name only for specific accounts).

The reason this is not possible currently is because the IDL generation works by parsing the program source code which gives it access to the TokenStream (basically AST) but parsing the actual program ID of the referenced program from this is, to say the least, not practical.

In order to go around this, in this POC I've taken a slightly different approach. Instead of just parsing the program source and outputting the generated IDL, I've l codegened an intermediate binary which is then compiled and run to to print out the IDL. The benefit of this approach is that now instead of only having access to the AST, we can also evaluate expressions within the program source.

So, referring to the the above example, with the binary we can do:

"pubkey": SomeProgram::id().to_string()

and when this is executed it will output

"pubkey": "<...>"

giving the possibility of encoding more information in the IDL.

Expression evaluation is not only useful for the Program constraint, but also for seeds, seeds::program, etc. Essentially, anything that does rely on an input to the ix call (ix arg, passed account, or account data) we can now evaluate and put in the IDL.

Here's a more detailed example that was generated by this POC for a test program, and this is the IDL it generates (I've generated only the instructions field here for the POC but this would output the full IDL).

Notes:

• I’ve encoded the seeds as base58 which I think is really convenient
• I’ve added the optional pubkey field to account for when we know which account needs to be passed in (like with Program)

TODO:

• handle expressions which reference ix args or account inputs
  • we need to be careful not to just copy over the TokenStream into the binary source because if it references any inputs it will cause a compiler error
  • this can be handled similar to how the current pda parser works - we can whitelist expression types which we want to support (like the MethodCall) and comb through the AST to find any references to ix args or accounts. In that case we can put any info about which args or inputs are being referenced to the IDL if possible
• handle other constraints like Sysvar, address = <>, and seeds::program = <>
• handle modules

Consideration:
The POC command I've implemented generates the binary in the program source directory which is probably not ideal because it pollutes users codebase. We probably want to generate it in the .anchor directory or similar.
In that case we'd also need to copy the program's Cargo.toml in there to have the same dependencies.

@armaniferrante let me know if this approach is valid so I can finish this up.

[kklas]

Thinking about #1972 got me an idea how we can avoid having to codegen the idl binary file - we modify the AnchorSerialize / Deserialize / derive(Accounts) / program macros to output the IDL generation code when the cargo "idl" feature flag is active. Then we just run the program with cargo run --features idl which will now output the IDL.
This will also enable the IDL generation for external (imported) types.

[acheroncrypto]

Superseded by #2011 and #2824.