Imported types should be added to IDL

[callensm]

Problem

At present, if you use an enum or struct from another crate that meets the criteria for a (de)serializable type as an instruction argument, that type gets listed as a defined entry in the instruction's argument list but doesn't get it's buffer-layout translations embedded into the IDL on program parsing.

While this is simply not implemented to be supported, it's also a natural limitation of Rust due to the lack of reflection/introspection. Creating a wrapper type doesn't work either because the inner field of the struct still won't be inspectable in order to get the list of fields of the foreign type.

Possible Solution

A new functional macro to act as a syntactical wrapper around defining the list of external types that need to be parsed and imported into the program's IDL, but expands to an empty TokenStream on build (since IDL parsing is over a static file read and doesn't require builds)

Example

use anchor_lang::prelude::*;

declare_id!("...");

import_idl_types![
    external_program::MyAnchorType,
    other_program::ArgumentType,
    // etc.
];

#[program]
mod sample {
    // ...
}

When the new local program's IDL is parsed, this would cause the equivalent of anchor idl parse on the program entry points (likely from the local cargo cache copies) for each of the enumerated external types and merge their IDL type object with the array of types in the local program's IDL.

---

This has become a cross-program composability issue is several cases and would be a huge improvement in the IDL parsing processes to unblock the use of imported types in programs.

(cc @suscd)

[kklas]

What does listing the types within the import_idl_types! macro do here? We already know what the types are from parsing the ix signature and we can deduce that they are external either from from the import statements or from the fact that they don't appear in any of the program's modules.

Thinking about external types two things come to mind:

1. Generate the IDL of the external program and then combine the two IDLs together manually to get the final IDL with all of the required types. We can build anchor tooling to help with that.

2. Use the approach based on what I outlined here Richer IDL generation with compilation #1927. We could modify the AnchorSerialize / Deserialize macros to generate a "toIDL" method only when "idl" cargo feature flag is active and then just run the code to output the IDL.

[callensm]

@kklas the purpose of using something like import_idl_types! would be to verbosely specify the external crate (and import path) of the type. This would ultimately lead to the parsing of the external program to get it IDL but we don't want to merge the entirely of them together, that would cause enormous IDLs for complex program which means higher storage cost on chain...we only want to merge in the definitions of the types we need.

We already know what the types are from parsing the ix signature and we can deduce that they are external either from from the import statements or from the fact that they don't appear in any of the program's modules.

Correct...but what about the case of developers importing the types via a renamed use or a glob? Sure this would be handleable when there is only one glob import in the given file or module but what if there are two or more glob imports? Now you can't do a simple deduction of where the type came from.

There is also std::any::type_name<T>() that returns the fully qualifying crate path for a given type T as a string that is nice for things like this, however, T in this case is required to be a const or const-like definition (meaning you can't use something like an ident of a type, syn::ItemStruct, syn::ItemEnum, etc. during file parsing) which makes this approach infeasible.

There are lots of little edge cases like this that make crate deduction difficult. but if there was an expandable-to-nothing macro to provide a syntactical identifier around verbose type paths, we could then do the IDL parsing of the external programs to get the types we need to merge into the local IDL.

[kklas]

Fair enough, thanks for clarifying. To me it feels like we're re-implementing the compiler a bit with the IDL parsing.

[acheroncrypto]

Added in #2011 and #2824.