P4 module system

[ppenzin]

Hi,

After compiling a few mid-size examples I've ran into a couple issues with using preprocessor within single compilation. I am highly suspicious that this can come up much more often in really large codes.

First is that that it is impossible to know if all files #include everything necessary or just piggy-back on includes from above, like this:

// define control A
#include a.p4
// define control B that depends on control A
#include b.p4
...

If b.p4 is missing #include a.p4 this would still compile, because preporcessed source would have everything in correct order. This leads to errors after switching order of #include directives. The solution is to guard the entire source with #ifndef in a way done in C or C++ headers, but it requires a bit more discipline than in case of C family, since it has to be done for every source file (versus only header files).

Second is name conflicts in preprocessed source. Because all files that are #include'd into a single .p4i file every exposed name needs to be unique. It is also fixable by prepending names with something contextual (like ingress_A instead of A), but that can affect core reuse, since names like this can go out of date when the code is incorporated into a different application or even a different part of the same application.

Looks like there was some discussion about this in #30. I am wondering if this is something that would be addressed in future revisions of the standard. One way to address this would be some form of non-transitive 'import' keyword (meaning that it would pull in only one level of imported files), which would provide some form of name hiding and would ensure that every source includes its own dependencies.

[mihaibudiu]

You are right, P4 does not really have a module system. Today the only solution is to use the preprocessor, which makes a very crude module system. We have considered adding namespaces, which could help somewhat. There is some activity in the P4 design working group on designing features to make P4 more modular, but we have not yet worked on this particular aspect that you describe.

Support for modularity is very important. I think we should look carefully at the existing module systems and borrow the best features that are known to work. We would appreciate suggestions and proposals if you have any. I am thinking something inspired by Python or Haskell would perhaps work.

Are you aware of an existing design which uses non-transitive imports?

[ppenzin]

I was thinking of Python and Haskell as well. But in spirit of P4's simplicity explicit module definitions might not be necessary. Maybe have "import" make all definitions from imported file (besides ones imported into it) available in the current file. There can be limitations on where such "import" statement can be placed in the source file (probably top level only, may be before all declarations).

Let me know if I can do anything to help -- with prototyping or write up.

[mihaibudiu]

If you or your organization is a member of p4.org you should consider joining our design discussions for the modularity constructs in P4. If you are not perhaps you should consider joining - there is no cost, you just have to sign a CLA form. We have some online meetings roughly every two Mondays.

Any other contributions are encouraged, including a design proposal or even code prototyping. If you have a proposal for an 'include'-like construct design the right solution is to submit a pull request against the spec with your changes. (For us to accept the pull request you may need to be a member of p4.org.)

[mihaibudiu]

Personally I think we should give a high priority to a nice 'import' mechanism; it should be relatively simple and it should not take too many resources to design and implement, but the payoff may be substantial.
So thank you for bringing this up.

[jnfoster]

+1. Having namespaces and import would be a win and is easier technically than some of the issues we've been dealing with in the modularity group so far. We should fork off a parallel thread to explore this.

…

-N

On Fri, May 25, 2018 at 11:38 AM, Mihai Budiu ***@***.***> wrote: Personally I think we should give a high priority to a nice 'import' mechanism; it should be relatively simple and it should not take too many resources to design and implement, but the payoff may be substantial. So thank you for bringing this up. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#628 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwi0svqhBAkXC96UNch28LcBLreNM4Uks5t2CVygaJpZM4UJucu> .

[ppenzin]

I think I will be able to join, I'll try to follow the instructions on https://p4.org/join/

[ppenzin]

I have submitted the CLA form, but not able to join mailing lists for some reason. Let me know if you want me to be part of any of the meetings. In the meantime, I will try to put a PR together.

@jnfoster, what is the best way to start a thread? Another issue?

[mihaibudiu]

We haven't had any meetings lately.
Have you followed the directions here for joining the mailing list?
http://lists.p4.org/mailman/listinfo/p4-design_lists.p4.org

[ppenzin]

I got through, thank you! I am not on p4-dev yet though, not sure if that is important.

[jnfoster]

Petr, can you send me an email? I'll send you the info for the P4 Modularity meetings.

[jnfoster]

Marking this as stalled.