[WIP][DNM] lib: posix: Elaborate for wider yet more selective POSIX coverage

[pfalcon]

Basic requirements:

1. It should be possible to control whether include/posix/ is
   added to include search path or not.
2. It should be possible to use individual parts of POSIX API,
   without bringing in everything else. For example, if user is
   interested in just sleep(), should be possible to enable just
   that.
3. Keep existing CONFIG_POSIX_API as "master switch", which
   enables "a lot of POSIX APIs".

Fixes: #12965

Signed-off-by: Paul Sokolovsky paul.sokolovsky@linaro.org

[aescolar]

1. It should be possible to control whether include/posix/ is
   added to include search path or not.

include/posix was moved into that posix/ folder (by Anas) purposefully to avoid include conflicts with the natiev_posix board. The idea is that people should include as "posix/something" if they want to include the Zephyr POSIX API replacement

[aescolar]

This is not nice, as you just remove that type and therefore that API when compiling for the posix arch

[pfalcon]

This is not nice, as you just remove that type and therefore that API when compiling for the posix arch

No, sys/types.h absolutely gotta come from the underlying POSIX OS, and I make Zephyr's POSIX subsys defines not clash with that.

(You may ask where exactly host sys/types.h is included, and that's "good question". Current situation is that it's included "somewhere", and that already causes clashes. And the right solution for now is excluding tests where the conflict happens from being run, instead of wasting time on solving such riddles.)

[aescolar]

this should not be here, or you will not be able to select host files instead of Zephyr posix files and viceversa

[pfalcon]

this should not be here, or you will not be able to select host files instead of Zephyr posix files and viceversa

???

Assuming you put comment on the exact line, it simply is there. Saying that it should not be there means digging in history, and addressing that comment to the author of that line. (But from my PoV, it of course should be there. But of course, it should be controlled, like my patch does.)

[aescolar]

I know it is there.. ;). Werent you trying to improve it? :p
Done like that you have the problem with inclusion order dependencies (which there is today) in the posix API files, where some of its headers and the host headers have the same name.
Knowing this, may help you understand some of the errors you see when trying to modify this code.
My mistake, I had read that line too fast

[pfalcon]

I know it is there.. ;). Werent you trying to improve it? :p

So, let's start from the start. When someone wants to use a POSIX function, they go to a reference and look it up. E.g. for sleep() http://pubs.opengroup.org/onlinepubs/9699919799/functions/sleep.html , it says #include <unistd.h>. So, using #include <unistd.h> for sleep() should just work, ditto for all other POSIX functions. That's the baseline requires for POSIX subsystem (and POSIX subsystem, Zephyr's included).

Being able to use #include <posix/unistd.h> is convenience hack, until the above works perfectly. What my patch does is to allow to control of automatic include path for include/posix/ in fine-grained way (as required to to make it work for hundreds of POSIX symbols, one by one).

[aescolar]

If you include this always, you should have all its individual c files gated with something else

[pfalcon]

If you include this always, you should have all its individual c files gated with something else

Probably yes, and eventually yes. But what happens when all those gates are enabled? Random build failures from native_posix. And that's the problem and means to address it which we discuss.

[aescolar]

My comment was related to a) code side & b) if you include them without any Kconfig option, then you need to be careful that today posix_cheats only does the renaming if POSIX_API is set

[pfalcon]

then you need to be careful that today posix_cheats only does the renaming if POSIX_API is set

One way to be careful is again to never include posix_cheats when anything POSIX subsys related is used. As posix_cheats is mandatory for ARCH_POSIX, that leads to obvious conclusions.

[aescolar]

I think Ulf would tell you to not use select in this way if you can avoid it.
Would a "default y if POSIX_API" in this option below be good enough?

[pfalcon]

The purpose of that config option is to select a bunch of other options.

[aescolar]

Yes, I understand, but the keyword "select" has a meaning beyond the english meaning of select.
At least Ulf recommends to use select only to set promtless symbols when possible, as otherwise it can be very confusing.

[ulfalizer]

select might be alright as currently used here. Could turn messy if dependencies are added. The Kconfig best practices page has this:

Rare exceptions might include cases where you’re sure that the dependencies of the selecting
and selected symbol will never drift out of sync, e.g. when dealing with two simple symbols defined
close to one another within the same if.

I'd add this to the help text too: "Particular POSIX APIs can still be enabled separately when this option is disabled."

[pfalcon]

@aescolar

include/posix was moved into that posix/ folder (by Anas) purposefully to avoid include conflicts with the natiev_posix board. The idea is that people should include as "posix/something" if they want to include the Zephyr POSIX API replacement

Yes, it it was done Anas. But no, not for being used like you wrote. You can see that yourself: 274ad46 .

@aescolar: I much appreciate prompt response to this matter. But let me be straight, the whole discussion shows that: I don't understand how ARCH_POSIX works; you don't understand how Zephyr POSIX subsys works. We definitely should improve each of out areas, actually, I'm set to do just that for posix subsys. But first and formost, we should break the deadlock both POSIXes put on each other. So, please continue with #13068, but "arch_exclude: posix" at the right places (and we have it in a bunch of places already) is the thing for the time being.

[zephyrbot]

All checks are passing now.

Review history of this comment for details about previous failed status.
Note that some checks might have not completed yet.

[pfalcon]

@aescolar, @ulfalizer: Thanks for the suggestions, this is partly PoC and WIP PR, I'll be moving it into suggested and discussed directions over time.

[andyross]

Let's not blacklist x86_64. Those tests all build and run successfully now, so this is going to reduce coverage. And the diagnosis in the patch here is actually wrong.

The proper fix, if you can't find a way to arrange the includes to not hit the missing limits.h issue, is to add and maintain our own copies of those headers for minimal libc builds.

[andyross]

Yeah, that's not what happening here. What's actually happening is that the toolchain is trying to include a limits.h header from the "platform", and not finding one. Our SDK compilers provide this header somehow (probably coming out of the sysroot they were built for), but a bare toolchain does not.

Technically this is a zephyr bug, I think. The limits.h header is part of POSIX (as well as ISO C) and if we want to provide POSIX we need to have our own (or piggy back on the one from newlib where practical, I guess).

[pfalcon]

Let's not blacklist x86_64.

@andyross, I didn't want to say that, but our further conversation will 100% follow the same pattern as we had a few days earlier with @aescolar here and in #13054, because his first phrase was also "Let's not blacklist native_posix" (or the same in meaning) ;-).

Here's the matter: I'm working on improving POSIX subsys, and some archs and boards work along with me, and some greet me with random build crashes on each step. I won't be telling maintainers of those archs and boards "you should fix that", I am telling "can we please backlog these unneeded complications and revisit it later, when Zephyr's POSIX subsys is actually improved". If you say no, fine. The only risk here is that Zephyr POSIX subsys won't be improved (much/enough). Someone will certainly solve it all later ;-).