-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BYOC] Exclude external params from Graph Runtime #7564
Conversation
This patch exculdes external params from created Graph Runtime module to prevent duplication as the external params will be also serialized from Metadata module. Change-Id: I47eda7e53b85f0b840445b60d33c2939b1cf99a0
It seems to me that the parameter name prefix doesn't indicate if it is in the external module or metadata module? Could you show some use cases? cc @zhiics |
I understood that the agreement is |
In summary, a safer solution is to have a way to check whether the constant is alaready stored in the extenral module when building a module, but it's not easy and that's part of the reasons that we haven't fixed this issue until now. |
Agree, However it still looks to me as the agreement to put |
@comaniac Any suggestions then? My first attempt was to explicitly split params from internal (p prefixed) and external modules in different sets. However the subsequent modifications were quite invasive. Basically my understanding of the issue is graph runtime module should not need external constatns and metadata module should not contain internal constants. |
I personally don't think it's a good idea to make decisions depending on variable names. The reason was mentioned in my previous comment. The variable names are supposed to be referred by users for debugging or development only.
I don't have a good idea at this moment. It might be a way to extend the solution from @mbaret that assigns "null" pointers to the external module handled parameters to be a more general solution, but we need to be careful. Maybe we should start with an RFC for discussion. |
I noticed that there is a check implemented for demanding codegen_name to be placed first (utils.h) in case of external codegen uses its own implementation of ConstUpdater. Looks to me now that I am on a safe side with current patch |
Yeha that's why I asked the question about if btw, if the purpose of this PR is going to solve this issue (https://discuss.tvm.apache.org/t/model-graph-size-doubling-after-partitioning-for-arm-compute-library/9131), I think it would be much easier to add an ACL constant updater like what Ethos-N does. |
I doubt that what was implemented in Ethos-N is applicable to ACL (ACL relies on JSON codegen and Ethos-N uses C codegen). Could you elaborate your proposal? |
I think this is not related to what codegen you are using? Based on the functionality of this PR, what you need is getting rid of some constatnts in the metadata module. Then it should be fine to put this logic to the ACL specific constant updater. @mbaret could you comment whether this proposal would work? |
The constants removed by this PR are actually from graph runtime module. The metadata constants are kept intact. The usage of custom ConstUpdater (in its current implementation) seems unrealistic to me due to the updater is called before the metadata and graph runtime modules are created. Metadata module gets all the local consts and the consts fetched via ConstUpdater as one map. Then this map of all constants is copied to graph runtime module |
Thanks @d-smirnov for this fix! I was also trying to find ways to fix the duplicate weights bug but couldn't find any as clean as yours. To address @comaniac's concern about using the names, could we use some sort of pass like free_vars to prune params and remove any params that aren't used in the module that is being compiled? Also, this problem also affects relay vm. Can we do the same thing for VM? |
I think this is just a work around. One way I can think of is that we can make graph runtime similar to some other runtimes. Therefore, it is able to query the needed parameters. I had a local draft commit to do this some time ago but I could not find it now. |
Apologies @comaniac that I missed this earlier. We did explore using the same fix as for Ethos-N (giving a 'null' ConstantUpdater), but this doesn't work for ACL as ACL does load the constants from MetadataModule. Ethos-N just directly serializes the constants into the binary module so it doesn't care if they're missing in the MetadataModule. This problem seems quite fundamental to me and shouldn't just affect ACL, but any BYOC target that tries to make use of MetadataModule.
Given this is a user-facing issue (https://discuss.tvm.apache.org/t/model-graph-size-doubling-after-partitioning-for-arm-compute-library/9131) and we're very interested in getting a fix in soon, could you elaborate on your alternative proposal @zhiics? |
The approach I was thinking is that we can probably use the The way we filter the redundant parameters in the this PR can be used by users after the build (i.e. we can document it somewhere) but should not be part of the |
Could we improve ACL's customized runtime so that it could know if the constant should load from MetadataModule or itself? To me, Ethos-N is just an extreme case that completely gets rid of constants in MetadataModule, but constant updater should be capable of supporting all three cases: all in MetadataModule, some in MetadataModule, and none in MetadataModule. |
Closed as superseded with #7977 |
This patch exculdes external params from created Graph Runtime module
to prevent duplication as the external params will be also serialized
from Metadata module.