Questions about Fiwix

[ghaerr]

Hi @mikaku,

This isn't an issue, but rather some quick questions about your project. First off, wow, what a nice project!! Very impressive how you've built such a straightforwardly-coded UNIX-like kernel, I find the sources easy to read and understand, which isn't always the case for complex kernels :)

I'm the maintainer over at ELKS, which is a Linux kernel, C library and applications for 8086 and compatible CPUs in real mode, running a segmented architecture. I was thinking it might be fun to jump in and contribute to Fiwix. However, my development environment is macOS. Do you think that might be much of an issue, substituting say x86_64-linux-musl-gcc for CC? I thought to ask before diving in to see what you think might be other gotcha's in the kernel build. We had the same issue at ELKS 3 years ago, but now the entire kernel, C lib, applications and images can all be built on macOS.

It's pretty cool how you've got a framebuffer console and /dev/fb running. I was thinking it might be fun to port over Microwindows or Nano-X over, which, now that Fiwix has UNIX sockets, should run easily directly on top of 16/24/32bpp framebuffer, and could use serial mouse rather than a dedicated kernel mouse driver. Any interest in that?

At ELKS, we have a nicely-working TCP/IP stack and application set, although in our case it runs in userland due to size constraints, which I wouldn't recommend here. But it has a nice state machine, which could possibly be somewhat easily inserted under the Fiwix socket code. Looking closer, I see it probably has the wrong license though.

Finally, is there a build script for the FiwixOS binaries, or is that all magic for the time being?

Thank you for your Fiwix project, I'm having fun reading the kernel code. Nicely done!

[mikaku]

First off, wow, what a nice project!! Very impressive how you've built such a straightforwardly-coded UNIX-like kernel, I find the sources easy to read and understand, which isn't always the case for complex kernels :)

Thank you very much!, I really appreciate your words.

I'm the maintainer over at ELKS, which is a Linux kernel, C library and applications for 8086 and compatible CPUs in real mode, running a segmented architecture. I was thinking it might be fun to jump in and contribute to Fiwix.

Yes, I know the ELKS project. It helped me a lot when I implemented the UNIX socket interface in Fiwix. As you can see, the code is heavily influenced by ELKS and by very early Linux kernel versions as well.

I was thinking it might be fun to jump in and contribute to Fiwix.

You are very welcome.

Do you think that might be much of an issue, substituting say x86_64-linux-musl-gcc for CC

First of all, and as stated here, it's not expected that you use the compiler of your system to compile a totally different kernel. I recommend you to build a generic cross-compiler (I use GCC 4.7.4) and use that compiler to build the Fiwix kernel.

That said, the Fiwix code is pretty strict to ANSI C89, so as long as the macOS C compiler complies with ANSI C89, it shouldn't have much problems. Perhaps the inline assembly parts are not supported in the macOS CC?. In these cases we can include #ifdef preprocessor directives to help to circumvent small problems. But I don't like the idea to fill the code with excessive #ifdef, as this could reduce readability.

In fact, we already have included some #ifdef preprocessor directives when we added support to build the Fiwix kernel using the tcc in #63. This was necessary as part of the live-bootstrap project by the bootstrapping community. I recommend you to take a look at this project. They have their own IRC channel at Libera.Chat.

It's pretty cool how you've got a framebuffer console and /dev/fb running. I was thinking it might be fun to port over Microwindows or Nano-X over, which, now that Fiwix has UNIX sockets, should run easily directly on top of 16/24/32bpp framebuffer, and could use serial mouse rather than a dedicated kernel mouse driver. Any interest in that?

I'm really interested. One of my goals is to have X11 running on FiwixOS. As you might know, I'm targeting to run FiwixOS on old PCs (386, 486 and Pentium) with networking capabilities. After completing the UNIX socket implementation, I wanted to start including a simple NIC driver (RTL8139?), and a simple TCP/IP stack. Unfortunately, I have to be sincere here, despite I'm an syadmin, I'm not an expert in low level networking at kernel level. That's why I thought this would be a good exercise to increase my knowledge on TCP, IP, ARP, ICMP, etc.

At ELKS, we have a nicely-working TCP/IP stack and application set, although in our case it runs in userland due to size constraints, which I wouldn't recommend here. But it has a nice state machine, which could possibly be somewhat easily inserted under the Fiwix socket code.

I think Fiwix could benefit from your current TCP/IP stack implementation on ELKS. We'll have to include a list of ioctl() options to keep Linux 2.0 compatibility and be able to use GNU network tools, like Linux does. Please, feel free to investigate this further and let me know if I can be of help for you on this process.

Looking closer, I see it probably has the wrong license though.

I'm not an expert in licensing. I'm open to change the Fiwix kernel license if this can attract people to this project, as long as this doesn't affect other projects like Boostrappable that depends of Fiwix. I believe @rick-masters might bring more information on this.

Finally, is there a build script for the FiwixOS binaries, or is that all magic for the time being?

You mean the FiwixOS user land packages?. If so, it's all in the Installation CDROM. Mount that CDROM and you will find a script called install/pkgs/src/makeall.sh to build all the packages, and another one called install/pkgs/src/toolchain/make-toolchain.sh to build the GNU Toolchain (Binutils, GCC and Newlib C library).

Thank you for your Fiwix project, I'm having fun reading the kernel code. Nicely done!

Is a pleasure for me to see people enjoying Fiwix as much as I do.

[ghaerr]

Do you think that might be much of an issue, substituting say x86_64-linux-musl-gcc for CC
it's not expected that you use the compiler of your system to compile a totally different kernel.
I recommend you to build a generic cross-compiler (I use GCC 4.7.4) and use that compiler to build the Fiwix kernel.

Sorry - x86_64-linux-musl-gcc is a cross compiler, that is exactly what I was suggesting - using (in this case), the statically-compiled gcc running on macOS to compile the Fiwix kernel. x86_64-linux-musl-gcc is available as a download so it doesn't have to be built on macOS to be used (like we have to with ia16-gcc-elf on ELKS).

Perhaps the inline assembly parts are not supported in the macos CC?.

All the inline ASM should compile fine. I was more worried about the other aspects of the build mechanism, including other non-gcc/binutil tools, of which I haven't looked at closely yet.

But I don't like the idea to fill the code with excessive #ifdef, as this could reduce readability.

I completely agree! It is quite refreshing reading Fiwix sources with very few #ifdefs. I have been working for years trying to remove as many from ELKS as possible, but sometimes contributed platforms set that process backwards a bit.

With regards to Fiwix #ifdefs, I notice quite a few #ifdef __DEBUG__ around printk. How about the following idea, which would remove most all of them, thus returning the source readability to very high:

Instead of:

#ifdef __DEBUG__
    printk("\t(inode = %d)\n", i ? i->inode : -1);
#endif /*__DEBUG__ */

use

/* in a header file */
#ifdef __DEBUG__
void dprintk(const char *, ...);        /* printk when debugging on */
#else
#define dprintk(...)
#endif
...
    dprintk("\t(inode = %d)\n", i ? i->inode : -1);

Then modify printk to be vprintk(const char *fmt, va_list p) and have printk and dprintk call it. This also could have the advantage of a runtime or command line boot switch to turn dprintk output on or off without recompilation.

[mikaku]

How about the following idea, which would remove most all of them, thus returning the source readability to very high:

This is a great idea, I've never had satisfied with the excessive amount of #ifdef __DEBUG__ in the code.

The only thing that worries me with this new implementation, is if I'll be able to continue activating debug for a particular code. I mean, now I just add the line #define __DEBUG__ on particular system call and I can see the debug messages of that system call only.

Also, I don't understand the syntax of the line #define dprintk(...). Does that mean that it won't me compiled at all?

[ghaerr]

I don't understand the syntax of the line #define dprintk(...). Does that mean that it won't me compiled at all?

Yes. According to the rules of variadic macros, this particular method effectively replaces the entire line of code with nothing, resulting in no code produced when debug is off.

The only thing that worries me with this new implementation, is if I'll be able to continue activating debug for a particular code.

I see. I had thought, seeing the master Makefile that -D__DEBUG__ was only turned on for the entire system at once, rather than per system portion. I solved this in ELKS by creating a series of debug_xxx macros, where each can be individually turned on/off for finer precision debugging without #ifdefs in the main source. Here's debug.h using something like the following which might work for Fiwix:

#define DEBUG_FILE      0               /* sys open and file i/o*/
#define DEBUG_NET       1               /* networking*/
...
#if DEBUG_FILE
#define debug_file      dprintk
#else
#define debug_file(...)
#endif
#if DEBUG_NET
#define debug_net       dprintk
#else
#define debug_net(...)
#endif

Then, instead of using dprintk directly in the kernel, debug_file would be used in the file I/O code, debug_net for networking, etc. It's a little messy in debug.h, but keeps the rest of the kernel without #ifdefs. I had another thought where a bit mask of subsystems to display debugging information would be passed into dprintk as part of the debug_xxx macro allowing for runtime changing of debug settings without kernel recompilation, but for 16-bit ELKS, it was deemed a bit too expensive in terms of code size.

All-in-all, I like your idea of keeping the kernel sources easy to read, with very few explicit #ifdefs.

I'm going to continue to play with Fiwix and will look further into cross-compiling on macOS, thank you!

[ghaerr]

now I just add the line #define DEBUG on particular system call and I can see the debug messages of that system call only.

Yes, that could be done without the more complex debug.h described above by just passing -D__DEBUG__ to a particular file being compiled and using a single #ifndef __DEBUG__ / dprintk(...) macro in a kernel header file. dprintk could be renamed debug to keep things very simple.

[mikaku]

Yes. According to the rules of variadic macros, this particular method effectively replaces the entire line of code with nothing, resulting in no code produced when debug is off.

This is interesting, I've never heard of it before.

The problem is that the page says "Variadic macros became a standard part of the C language with C99.", which presumably GCC will compile regardless of -std=c89 because they include Extensions, but other compilers like tccmight not accept it.

[mikaku]

If you are using variadic macros, does that mean that GCC and macOS CC are the only compilers you use on ELKS and your code is not C89 strict?

[ghaerr]

but other compilers like tcc might not accept it.

Good point, I had forgotten Fiwix can be compiled with a compiler other than gcc. I'll have to experiment with tcc to see what it accepts. It's kind of nice that Fiwix keeps things simple enough that the complex gcc inline asm isn't used, thus increasing portability.

Speaking of which, I just added CROSS_COMPILE = x86_64-linux-musl- to Makefile, and the kernel compiles without error! It is dying on the final link stage, where my cross-compiler is complaining about an invalid format libgcc.a. I'll try another cross compiler and report results.

[mikaku]

Speaking of which, I just added CROSS_COMPILE = x86_64-linux-musl- to Makefile, and the kernel compiles without error!

You don't need to touch the Makefile to use your cross compiler.

I use the following command:

export CROSS_COMPILE=~/Fiwix/toolchain/4.7.4-elf/bin/i686-elf- ; make clean ; make

[ghaerr]

If you are using variadic macros, does that mean that GCC and macos CC are the only compilers you use on ELKS and your code is not C89 strict?

ELKS can't use macOS cc/clang as it only produces 64-bit code. The portability problem is much worse, as we now have to use a specialized port of GCC known as ia16-elf-gcc, which has all sorts of 8086-style segment extensions (__far pointers, etc.) This was discussed heavily a couple of years back when switching from BCC, a very old compiler, and for ELKS it was deemed ok as the project needed a stronger modern compiler that was able to generate tight code, for speed and space reasons.

That said, its nice to not have a project depend on a specific compiler. Given that Fiwix doesn't use GCC extensions and keeps x86 ASM in (very few) .S files, it seems that all that is needed is a compiler that produces ELF .o output, along with a few (incompatible) linker options.

For ELKS, we have to build the GCC ia16-elf-gcc cross compiler which is quite time consuming, although its only done once and isn't part of the standard "make all" build.

[ghaerr]

It does look like TCC supports variadic macros as a C99 extension. I'm not sure whether using variadic macros breaks the -std=c89 option or not on GCC.

[mikaku]

It is dying on the final link stage, where my cross-compiler is complaining about an invalid format libgcc.a. I'll try another cross compiler and report results.

If you are having problems linking the kernel with libgcc, you might try to #undef the options CONFIG_OFFSET64 and CONFIG_PRINTK64 in include/fiwix/config.h, and then remove the part -L$(LIBGCC) -lgcc in the Makefile.

You'll not have 64bit offsets (EXT2 filesystems bigger than 4GB) but you'll be able to play with Fiwix quickly.

[ghaerr]

If you are having problems linking the kernel with libgcc, you might try

Very cool - that worked! (Using x86_64-linux-musl-gcc). Now I need to figure out how to add the newly-built fiwix into the flat ATA disk raw image I'm using for QEMU.

I'm also downloading i686-elf-gcc and will try that without removing -lgcc to see whether that works. Thanks for the help, I hadn't realized that only the libgcc.a routines for 64-bit arithmetic were being used by gcc output.

[ghaerr]

I'm assuming the ATA raw disk image is EXT2 format, right? We can't write/mount EXT2 on macOS.

We got around this problem on ELKS by writing the mfs (Minix Filesystem) tool that allows creating MINIX filesystems from scratch on any OS. (I didn't say it was easy to build operating systems on macOS, right?! :)

[mikaku]

Now I need to figure out how to add the newly-built fiwix into the flat ATA disk raw image I'm using for QEMU.

You don't need to update the flat ATA disk raw image every time you changed the Fiwix kernel. Just use something like this:

qemu-system-i386 \
       -kernel fiwix -append "ro root=/dev/hda2 bga=1024x768x32" \
       -drive file=FiwixOS-3.3-i386.raw,format=raw,if=ide,cache=writeback,index=0 \
       -net none \
       -enable-kvm \
       -machine pc \
       -cpu 486 \
       -serial pty

[mikaku]

I'm assuming the ATA raw disk image is EXT2 format, right? We can't write/mount EXT2 on macOS.

Yes, it's EXT2, but beware because it uses revision level 0. That is the original EXT2 without any kind of features.

I didn't say it was easy to build operating systems on macOS, right?!

Heh 😃

[ghaerr]

-kernel fiwix -append "ro root=/dev/hda2 bga=1024x768x32" \

Fiwix is booting, built on macOS! Very nice. How does the QEMU -kernel option work, does QEMU know about EXT2 filesystems and thus able to redirect file reads to the host OS image, or is it more tricky than that?

Using the i686-elf-gcc toolchain isn't working, i686-elf-ld is failing not understanding the -nostartfiles and other options. So I'll just stay with the working x86_64-linux-musl-gcc toolchain for now, with 64-bit operations turned off.

I understand the Fiwix applications are all being built on Fiwix itself? And then is the output EXT2 image created through QEMU running makeall.sh then kept as the ATA raw disk image?

I suppose in order to give the Microwindows port a test run I'll still need to figure a way to get its source files (or cross-compiled binaries) onto a compatible Fiwix filesystem... I suppose I could try using my mfs to write a MINIX v1 filesystem and boot from that?

Thanks for your help. Getting Fiwix running using macOS is proving lots easier than I had thought.

[ghaerr]

How does the QEMU -kernel option work,

I think I get it now, since Fiwix doesn't depend on any non-standard boot mechanism (and doesn't include its own boot code), the QEMU -kernel and -append options just use a GRUB-style or standard boot to load the kernel, and pass the -append options to the boot loader. Nice.

I haven't found the script that adds a boot loader to the ATA raw disk image, I'll look at that stuff in a bit.

[mikaku]

How does the QEMU -kernel option work, does QEMU know about EXT2 filesystems and thus able to redirect file reads to the host OS image, or is it more tricky than that?

Fiwix is a Multiboot Specification v1 compliant (GRUB loader v1 also called GRUB legacy). When you use the -kernel parameter, QEMU boots the provided kernel and passes it a compliant Multiboot Spec. Fiwix then knows the memory map and other machine features.

I understand the Fiwix applications are all being built on Fiwix itself?

Yes, FiwixOS is what it's called a self-hosted operating system. All the packages has been built under FiwixOS (including the kernel, of course).

And then is the output EXT2 image created through QEMU running makeall.sh then kept as the ATA raw disk image?

The ATA raw disk image is created on my Host OS (Fedora Linux) using the simple command truncate -s 1GB FiwixOS-3.3-i386.raw. Then I use the Installation CDROM to install the FiwixOS there.

[mikaku]

I suppose in order to give the Microwindows port a test run I'll still need to figure a way to get its source files (or cross-compiled binaries) onto a compatible Fiwix filesystem... I suppose I could try using my mfs to write a MINIX v1 filesystem and boot from that?

First, I don't think porting your Microwindows will be trivial. I'd be really surprised if it is.

Also, Fiwix supports Minix v1 and v2 filesystems. Just need to activate the option CONFIG_MINIX in config.h.

[mikaku]

I haven't found the script that adds a boot loader to the ATA raw disk image, I'll look at that stuff in a bit.

The ATA raw disk image was created using the Installation CDROM and installing FiwixOS there.
I think that you should try install FiwixOS on your own ATA raw disk to have a better idea of how interact all the pieces.

[ghaerr]

FiwixOS is what it's called a self-hosted operating system. All the packages has been built under FiwixOS (including the kernel, of course).

That's pretty amazing. Especially building gcc and binutils on Fiwix itself.

using the simple command truncate -s 1GB FiwixOS-3.3-i386.raw.

I see. So that command actually only creates a 1GB hole (read as effectively zeros) and then the Installation CDROM actually writes the filesystem onto it in one of Fiwix's kernel-supported formats.

Then I use the Installation CDROM to install the FiwixOS there.
The ATA raw disk image was created using the Installation CDROM and installing FiwixOS there.

I see, truly self-hosted, including building the images themselves. Wow.

I think that you should try install FiwixOS on your own ATA raw disk to have a better idea of how interact all the pieces.

Ok, I'll try that.

I don't think porting your Microwindows will be trivial. I'd be really surprised if it is.

Microwindows can run without a keyboard or mouse by configuring it run by only requiring /dev/fb0 in 32/24/16 bpp, with almost no dependencies. I definitely want to try that. Do you support the Linux framebuffer ioctls to get the framebuffer size and address for mmap? If so, then possibly no modifications at all will be required for /dev/fb0 support.

Microwindows has a couple of drivers for the keyboard (scancode mode for full keyboard control w/shift keys etc) and a basic ASCII mode for systems without scan code keyboards. It's got its own self-contained MS or Logitech mouse driver that can run on the serial port, which I think QEMU can possibly redirect into from the host mouse. So I think we can get it running quickly.

Fiwix supports Minix v1 and v2 filesystems.

ELKS only supports Minix v1. The only real benefit here would be to allow creating Fiwix images directly from macOS without running Fiwix itself on QEMU to create the images. I'll look at your suggested approach and learn how you've done it. The other problem with Minix v1 could be the 14-character max filename, which sometimes causes problems these days.

[mikaku]

Do you support the Linux framebuffer ioctls?

Not yet, basically because I've not found a software that can use the framebuffer and do not requires a basic TCP/IP stack.

[ghaerr]

Do you support the Linux framebuffer ioctls?

How can an application get the size (width, height, stride) and address of the VESAFB framebuffer? The Microwindows driver then just mmaps that address into the process and the rest is handled internally.

All we really need is the hardware address of the framebuffer, the rest could be provided manually.

[mikaku]

How can an application get the size (width, height, stride) and address of the VESAFB framebuffer?

No application is using the framebuffer. I only use it to have a better console experience.

The only graphic applications that run on FiwixOS (like DOOM) are using the SVGAlib.

[mikaku]

All we really need is the hardware address of the framebuffer, the rest could be provided manually.

You can use /dev/fb0 to write anything into the framebuffer.

[ghaerr]

I've not found a software that can use the framebuffer and do not requires a basic TCP/IP stack.

Microwindows implements two APIs, a Win32-compatible and an X11-like (Nano-X). The former doesn't require any sockets as the application main loop is compiled into the "win32" app. Nano-X is normally compiled separately using a named FIFO to communicate between multiple applications and Nano-X but we also support linking the application into the server.

Thus, both APIs can work by drawing directly into a mmap-ed framebuffer.

The only graphic applications that run on FiwixOS (like DOOM) are using the SVGAlib.

I see. I'll just have to dig up how SVGAlib gets the framebuffer address, unless it's not using the framebuffer.

You can use /dev/fb0 to write anything to the framebuffer.

Yes - but the application normally opens /dev/fb0, then does a single ioctl to get the framebuffer address. Otherwise, one would need to use the open/read/write/lseek in order to draw, which would be very slow.

[mikaku]

The other problem with Minix v1 could be the 14-character max filename, which sometimes causes problems these days.

Minix v2 supports up to 30-character filenames, but that's not much either.

[ghaerr]

Here's a sample of how Microwindows uses /dev/fb0 to gain access:

    /* open framebuffer file for mmap*/
    if((env = getenv("FRAMEBUFFER")) == NULL)
        env = "/dev/fb0";
    fb = open(env, O_RDWR);
    if (fb >= 0)
    {
        /* mmap framebuffer into this address space*/
        int extra = getpagesize() - 1;
        psd->size = (psd->size + extra) & ~extra;   /* extend to page boundary*/
        psd->addr = mmap(NULL, psd->size, PROT_READ|PROT_WRITE, MAP_SHARED, fb, 0);
        if (psd->addr == NULL || psd->addr == (unsigned char *)-1)
        {
            EPRINTF("Error mmaping shared framebuffer %s: %m\n", env);
            close(fb);
            return NULL;
        }
    }

After this, psd->addr can be directly used as a pointer to the framebuffer, after which 32-bit RGBA can be read/written to.

[EDIT: Oops! I was incorrect: an ioctl is only used (previous to this listing) to get the framebuffer size. The mmap call then takes the framebuffer fd and maps it into the process address space. Will that work on Fiwix?]

[mikaku]

I see. I'll just have to dig up how SVGAlib gets the framebuffer address, unless it's not using the framebuffer.

SVGAlib doesn't use the framebuffer, and if you want to try the software using this library, you cannot boot the kernel using the framebuffer console, but the VGA text console. That is, without the parameter bga=.

Yes - but the application normally opens /dev/fb0, then does a single ioctl to get the framebuffer address. Otherwise, one would need to use the open/read/write/lseek in order to draw, which would be very slow.

I'm sorry, there are no ioctls implemented for the framebuffer.

[mikaku]

Yes - I believe what happens is that each mmap-able file device has a special fops entry point that can be called to do that for the specific device. So the kernel mmap gets the address by calling the function table entry that ends up calling a fb.c function to get the hardware address to map.

This is exactly what I was trying to say, but my English is not as good as I wanted.

We wouldn't need to go to that extreme initially though, as that requires another entry point for most all drivers. Instead, perhaps an ioctl to fb.c that ends up mapping the hardware address into the process space and returns the virtual address. That would allow getting framebuffer versions of graphics libraries working without waiting for a full TCP/IP stack.

I completely agree.

I think that only UNIX sockets are actually required, if all applications are local. Yes, this does require a pretty full higher-level socket implementation, but no TCP/IP underneath.

That would save a lot of time of implementation.

Having ported Microwindows to many OSes, I would suggest doing this in steps:

Looks like a good plan.

[ghaerr]

Looks like a good plan.

I'll play around with getting the framebuffer mapped in the fb driver. Given that macOS doesn't have the ability to read or write EXT2 virtual filesystems at all, how do you recommend getting source (or binary) files onto and off the root or other mounted Fiwix filesystem easily?

Also, is there a sample Makefile that might be used for cross-compiling user mode executables outside of a running Fiwix? Or is the idea to always use Fiwix for compiling all applications?

[mikaku]

how do you recommend getting source (or binary) files onto and off the root or other mounted Fiwix filesystem easily?

Since Fiwix also supports the ISO9660 filesystem, you might want to consider using that media to import software to your FiwixOS system, by using macOS tools to create CDROM images. And then you might use the Minix filesystem to export them out. Minix v2 would be specially useful as it supports 30-character filenames and up to 1GB of filesystem size.

Another alternative is to use the Paragon Software driver.

Also, is there a sample Makefile that might be used for cross-compiling user mode executables outside of a running Fiwix? Or is the idea to always use Fiwix for compiling all applications?

I used cross-compiling long time ago, in the very early stages of the FiwixOS, when I was creating a self-hosting operating system. Once the system was self-hosting capable, I no longer used that path.

For best results build your software under FiwixOS 3.3, or under an old Linux distro with kernels 2.0 or 2.2 since Fiwix is system call mostly compatible with those kernels. I regularly test the compatibility using the old RedHat Linux 4.2 and 5.2 distributions (both with Linux 2.0.x kernels), and also RedHat Linux 6.2 with kernel 2.2.x. I recommend you to statically link your binaries, to avoid dragging a lot of Linux libraries to FiwixOS.

[rick-masters]

Given that macOS doesn't have the ability to read or write EXT2 virtual filesystems at all, how do you recommend getting source (or binary) files onto and off the root or other mounted Fiwix filesystem easily?

@ghaerr
Not sure if this will help you but there is this project for creating and populating ext2,3,4 file systems from a program:
https://github.com/gkostka/lwext4

You'll want to use my fork which supports ext2 rev 0 and a couple other fixes/features:
https://github.com/rick-masters/lwext4/commits/live-bootstrap

Downloadable here:
https://github.com/rick-masters/lwext4/releases/download/v1.0.0-lb1/lwext4-1.0.0-lb1.tar.gz

We use this in live-bootstrap to build an ext2 for booting Fiwix:
https://github.com/fosslinux/live-bootstrap/tree/master/steps/lwext4-1.0.0-lb1

This is the program that creates the file system:
https://github.com/fosslinux/live-bootstrap/blob/master/steps/lwext4-1.0.0-lb1/files/make_fiwix_initrd.c

[ghaerr]

Thanks @rick-masters, with a little hack to the CMakeLists.txt file, I was able to get your version of lwext4 compiled on macOS! Also got make_fiwix_initrd to build.

I'd like to use the lwext4-mkfs (or possibly lwext4-generic?) tools to read/write from EXT2 V0, as then I could write a script to move things back and forth. I'm a bit concerned about specifying the EXT2 V0 version though: would lwext4-mkfs -i ext_image -e 1 work (where -e 1 specifies the value of F_SET_EXT2_V0 from your branch), or do I possibly need to modify the lwext4-mkfs.c source?

Is lwext4-generic just a test tool or is there a built program that can be used to read/write EXT2 files directly from the command line? I'm trying to determine whether there's anything in the lwext4 source that can be used as command line programs to list, read or write EXT2.

[ghaerr]

Another alternative is to use the Paragon Software driver.

This could potentially be very easy to use, but its a paid program; it says to support EXT2 but I am bit worried about exactly the difference(s) between the V0 and later version of EXT2 filesystem. Is the Fiwix EXT2 V0 incompatible with later versions of EXT2? Unfortunately, I am not much familiar with EXT2, although I am very familiar with MINIX v1.

The idea of doing development on a Fiwix MINIX v1 filesystem is quite possible, except I am not sure how to convert or create the Fiwix 3.3 ATA Raw disk image into a MINIX filesystem, from which I could use my ELKS mfs utility to get files back and forth. If Fiwix requires an image > 65MB, then I suppose I would have to enhance the mfs utility to support V2, of which I think that is almost done, but not tested. Another thought, I suppose, would be to automatically mount a MINIX v1 filesystem with my source code when running Fiwix under QEMU.

In some sense it might be good to figure a way to create a full image of Fiwix 3.3 binaries from a template of such on a host operating system, as then it would be possible to generate a specific distribution based on the users needs or set of source code being worked on. On the other hand, this may not be in the goals of Fiwix.

Sorry for all the trouble; it seems that tools supporting filesystem formats of self-hosted systems are in fact another subtle dependency not always realized when not running Linux.

[ghaerr]

Good news! After fooling around a bit with ELKS mfs and recompiling Fiwix to support MINIX, I was able to easily create EXT2 image from scratch and transfer an entire source distribution of Microwindows onto it, using:

$ cd microwindows
# create 1GB EXT2 filesystem w/30 char names and transfer entire src/ dir to it
$ mfs fwsrc.img genfs -2 -n30 -i32736 -s1000000 src
$ qemu-system-i386 \
    -kernel fiwix -append "ro root=/dev/hda2 bga=1024x768x32" \
    -drive file=FiwixOS-3.3-i386.raw,format=raw,cache=writeback,index=0 \
    -drive file=/Users/greg/net/microwindows/src/fwsrc.img,format=raw,cache=writeback,index=1 \
    -net none \
    -boot d \
    -machine pc \
    -cpu 486
[Then on Fiwix after login]
...
[(root) ~] # mount -t minix /dev/hdb /mnt

This should allow for quick creation of arbitrary source trees to be mounted and compiled within Fiwix. The kernel can be compiled outside and use the QEMU -kernel option for testing. Getting the files off EXT2 will be a bit harder for the moment, as mfs doesn't have a recursive directory copy out, but that can be alleviated for the time being by creating a tar file, copying that out, and running diff -urN on the contents for commit.

[ghaerr]

Also got all Microwindows Win32 API libraries and most applications compiled and linked, without too many issues. Will report small list of unimplemented Fiwix kernel features later.

Rewriting screen driver now for Fiwix, and will attempt mapping framebuffer address into process space hopefully tomorrow.

I'm impressed with the very full development environment all ported and running on Fiwix. Things are a bit slow compiling on QEMU though, but no strange errors at all. Turns out it is quicker to just mount an MINIX v2 filesystem and compile there than to copy source to Fiwix root EXT2 filesystem.

[mikaku]

it says to support EXT2 but I am bit worried about exactly the difference(s) between the V0 and later version of EXT2 filesystem.

The revision level 0 of the EXT2 filesystem is the original EXT2, it don't has any extra feature. For instance, the very next version of EXT2 introduced the sparse superblock after seeing that so many copies of it resulted on a lot of disk space lost on big filesystems. Over time, they have included more and more new features, but Fiwix only supports V0.

I can interact easily with the Fiwix filesystem from my Fedora because any Linux system supports all EXT2 versions. The problem here is that you are using a macOS. But Minix v2 can be of help for you since you made your own driver.

If Fiwix requires an image > 65MB, then I suppose I would have to enhance the mfs utility to support V2,

Yes, with only 64MB of size in the root filesystem you won't be able to do any type of development. GNU Toolchain, Bash, system tools, etc. all them weight 500MB at least. You'll need 1GB which means, you need Minix v2.

[rick-masters]

Thanks @rick-masters, with a little hack to the CMakeLists.txt file, I was able to get your version of lwext4 compiled on macOS! Also got make_fiwix_initrd to build.

I'd like to use the lwext4-mkfs (or possibly lwext4-generic?) tools to read/write from EXT2 V0, as then I could write a script to move things back and forth. I'm a bit concerned about specifying the EXT2 V0 version though: would lwext4-mkfs -i ext_image -e 1 work (where -e 1 specifies the value of F_SET_EXT2_V0 from your branch), or do I possibly need to modify the lwext4-mkfs.c source?

Is lwext4-generic just a test tool or is there a built program that can be used to read/write EXT2 files directly from the command line? I'm trying to determine whether there's anything in the lwext4 source that can be used as command line programs to list, read or write EXT2.

Sounds like you may have found another method, but here are my answers:

Sorry, I don't know about the tools you mentioned. I think they may be test tools. If I remember correctly, I couldn't find any simple command line tools in the project that could be used out-of-the-box. I found that surprising since it seemed like it wouldn't have been too hard for the project to include those.

I just ended up writing my own program. The make_fiwix_initrd.c program is probably your best bet as an example for how to use the library.

Unfortunately, the author of lwext4 does not appear to be maintaining the project. My Issues and PRs haven't received any attention.

[ghaerr]

Thanks @mikaku and @rick-masters for all your help. I'm trying to call do_mmap inside fb_ioctl and have possibly run into a problem (bear with me, I'm a bit new at fully understanding kernel page mapping and Fiwix). I'm doing:

unsigned int addr = do_mmap(NULL, (unsigned int)video.address, video.memsize, PROT_READ|PROT_WRITE,
        MAP_PRIVATE|MAP_ANONYMOUS, 0, P_MMAP, 0, NULL);

It seems that the physical (or perhaps already mapped kernel virtual address?) for the framebuffer is 0xfd00000. I'm looking to map that address (or its predecessor with the real VESA hardware address?) into the process address space, at any available address. The do_mmap code following is causing an -EINVAL return, presumably because one can't map addresses higher than PAGE_OFFSET into a user process address space:

    if(start > PAGE_OFFSET || start + length > PAGE_OFFSET) {
        return -EINVAL;
    }

Any thoughts on how to proceed?

Thank you!

[ghaerr]

I also don't yet understand this code in fb_init:

    /*
     * Frame buffer memory must be marked as reserved because its memory
     * range (e.g: 0xFD000000-0xFDFFFFFF) might conflict with the physical
     * memory below 1GB (e.g: 0x3D000000-0x3DFFFFFF + PAGE_OFFSET).
     */
    from = (unsigned int)video.address - PAGE_OFFSET;
    bios_map_reserve(from, from + video.memsize);

Are the kernel virtual addresses already mapped into the user process space, and/or does Fiwix keep seperate page tables for kernel and user mode? I suppose if 0xFD000000 is already mapped into the process space, perhaps it only needs to be given read/write permission for the user page table entry for the same address. I must admit I don't yet understand Fiwix enough to know how the system actually runs regarding virtual/physical address mappings etc.

[ghaerr]

I am guessing here that do_mmap is the wrong function to use - it builds a vma entry and waits for a page fault to map, right?

Instead, something like map_kaddr but for user address space (would that be map_page?) that would basically copy the existing kernel address page table entries for the 16MB at 0xFD000000 to some available user virtual address and mark them as present? I'm not quite able to determine exactly what code should be written for that.

It seems strange that 0xFD000000 would always be the VESA FB address; does multiboot setup protected mode page table entries for a kernel mode framebuffer address before passing control to Fiwix? Because the kernel/multiboot.c code just copies the address, which always seems to be 0xFD000000 to video.address, and doesn't create a Fiwix page table entry for it that I can see.

Sorry for all the questions! I hope to learn a bit more and will then get a graphical windowing environment running on Fiwix.

[mikaku]

Yes, Fiwix memory subsystem is not as good as we all would want, and there is no documentation either. Please, keep in mind that this is a kernel made by only one person (mostly).

I 'd like to rewrite the memory subsystem and fix some pending problems with RAMdisk drives and other aspects, but sometimes it's difficult to have a good window of spare time to do it with calm.

The framebuffer is mapped in a way that is not expected to be in the user address space. As I stated above, the only way for a user program to write to the /dev/fb0 is using the basic semantics (open, write and close). I know that this slows down the access to the framebuffer. Your Microwindows and X11 will want to access the framebuffer memory directly.

I think the function do_mmap() won't be of much help here. As you already noticed, you need something more like map_page but for a range of addresses instead of a single page.

I cannot answer all your questions quickly because there are some parts of the kernel that I forgot how I implemented them and it forces me to re-read and test de code to refresh my memory, which wants time.

[ghaerr]

Please, keep in mind that this is a kernel made by only one person (mostly).

Don't worry about that, I am already very impressed with what you have produced. Excellent job!!! I am really enjoying Fiwix, and find the coding very straightforward and well written.

Fiwix memory subsystem is not as good as we all would want

I have spent a couple more hours looking at code, and actually the memory and paging subsystem seems quite understandable, although of course complex. It is just taking me a bit of time myself to come to speed with 32-bit protected mode paging and memory management.

The framebuffer is mapped in a way that is not expected to be in the user address space.

Yes, it seems so, but I have a couple ideas below on how this might get done.

you need something more like map_page but for a range of addresses instead of a single page.

I am thinking of the following: it seems that normally, the kernel virtual addresses (KVA) start at 0xC0000000 (PAGE_OFFSET), and user addresses at 0x10000 up to 0x4000000 (MMAP_START) which includes the heap, and mmaps mapped from there up to the bottom of stack, where the stack starts growing downwards at PAGE_OFFSET.

Thus, yes it does seem there is not a good space to map the framebuffer address, but I am thinking it could be placed as a normal mmap (P_MAP) section by calling get_unmapped_vma_region to find a space above MMAP_START, then call vma_add_region to reserve it (like do_mmap does). However, instead of then waiting for a page fault, the kernel page table entries for video.address (0xFD000000) could be copied and new user page table entries created with PAGE_PRESENT flags in the vma region just reserved.

In this way, a vma region would get allocated for the 16MB framebuffer in the normal mmap user address space, but get immediately marked present using the values from the existing kernel framebuffer page table entries.

Thus, something like this:

   unsigned int start = get_unmapped_vma_region(video.memsize);
   if(!(vma = (struct vma *)kmalloc(sizeof(struct vma)))) {
            return -ENOMEM;
    }
    memset_b(vma, 0, sizeof(struct vma));
    vma->start = start;
    vma->end = start + video.memsize;
    vma->prot = PROT_READ|PROT_WRITE;
    vma->flags = MAP_PRIVATE|MAP_ANONYMOUS;
    vma->offset = 0;
    vma->s_type = P_MMAP;
    vma->inode = NULL;
    vma->o_mode = 0;
    vma->object = NULL;
    for (unsigned int addr = vma->start; addr < vma->end; addr += 4096) {
        unsigned int kaddr = get_kernel_pte_addr(video.addr + addr - vma->start);
        map_page(current, addr, kaddr, PROT_READ|PROT_WRITE);
    }

The new function get_kernel_pte_addr would return the physical address mapping from the kernel page table entry for the passed kernel virtual address (starting from 0xFD000000).

I cannot answer all your questions quickly

That's ok, thank you very much for your help! I am enjoying learning by using your project :)

[ghaerr]

Great news! Microwindows running on Fiwix!!

Microwindows.on.Fiwix.mov

I got the framebuffer mapping working using the idea above. At the moment no mouse and no keyboard, but hey, kind of neat :)

The VMA region mapped for the framebuffer needs unmapping before the process exits, and it seems that the kernel fb_close routine is not automatically called on the open framebuffer file descriptor. I've worked around that problem but will report back more issues.

One can also see the fbcon cursor on the left side of things. This can all be worked around with some more ioctls. Continuing to port Microwindows drivers over to Fiwix.

[ghaerr]

[ghaerr]

Status Update

I finished with a full port of Microwindows to Fiwix, and now both Win32 and Nano-X APIs are working, along with the mouse and keyboard. It's only tested on QEMU, and with a few exceptions, most demo programs are working well. I must say, it's been a pleasant experience working with both the Fiwix kernel and very complete development environment!

Things Added

• Added three ioctls to Fiwix framebuffer driver: map the framebuffer address to the current process space, and return X and Y resolution when a VBE 32-bit RGBA framebuffer is specified at boot.
• Framebuffer is unmapped from address space on close of file descriptor.
• config file to build Microwindows on Fiwix from scratch with no dependencies.

Things not finished

• Framebuffer address map only works for a single open call (!).
• It appears that Fiwix process exit does not call sys_close on each open file descriptor, which causes a hash table error on release_page. This is a problem when an application exits abnormally.
• The kernel doesn't remove the fbcon cursor when calling ioctl(ttyfd, KDSETMODE, KD_GRAPHICS). Calling ioctl(ttyfd, KDSETMODE, KD_TEXT) may want to restore the fbcon text console buffer.

Potential Kernel Issues Found

• Last line of console is sometimes not scrolled properly; can be duplicated using vi and scrolling up/down but also seen from shell on rarer occasions.
• No named pipes, so Nano-X applications are linked directly with the NX server, instead of running client-server.
• No struct dirent or DT_DIR using <dirent.h>. Not a big deal.
• Running bin/demo-font causes Page Fault then system panic in release_page. Unknown reason, could be NULL pointer in application? (Screenshot available).

Wishlist

• Add FreeType 2.10.4 to FiwixOS distribution; this allows for much better font support.
• Add Microwindows to FiwixOS distribution (will provide commit for this)
• Need C library header to put new framebuffer ioctls - IO_FB_ADDR, IO_FB_XRES, IO_FB_YRES.
• Figure out libgcc.a problem on cross compiler for macOS so 64-bit functions can be used.
• Building Microwindows on Fiwix is quite slow... is this possibly because of MINIX v2 filesystem use?

For testing, QEMU was setup to pass through the host mouse as a MS mouse, which is then decoded again by the Microwindows mouse driver on /dev/ttyS0. This was done using the following QEMU invocation:

qemu-system-i386 \
    -kernel fiwix -append "ro root=/dev/hda2 bga=1024x768x32" \
    -drive file=FiwixOS-3.3-i386.raw,format=raw,cache=writeback,index=0 \
    -drive file=/Users/greg/net/fwsrc3.img,format=raw,cache=writeback,index=1 \
    -chardev msmouse,id=chardev1 -device isa-serial,chardev=chardev1,id=serial1 \
    -net none \
    -boot d \
    -machine pc \
    -cpu 486

With a bit more work, the X11->Nano-X libraries could be built, which would allow for some X11 programs to run. But this isn't very feasible until we get Nano-X client/server running and X11 headers and other X11 client GUI source on Fiwix.

All-in-all, pretty cool! More screenshots and/or screencasts available if wanted.

[mikaku]

Great news! Microwindows running on Fiwix!!

This is ... well, I don't have words to express how I feel right now. Amazing!, really amazing!. Good Job Gregory!

Fiwix is pronounced as 'few weeks' because that should be the average time you need to learn how it works (and start tinkering). Definitely, you and @rick-masters made this name a good choice! :-)

Added three ioctls to Fiwix framebuffer driver: map the framebuffer address to the current process space, and return X and Y resolution when a VBE 32-bit RGBA framebuffer is specified at boot.

Are those ioctls fully compatible with Linux?

It appears that Fiwix process exit does not call sys_close on each open file descriptor, which causes a hash table error on release_page. This is a problem when an application exits abnormally.

Yes it does:

Fiwix/kernel/syscalls/exit.c

Lines 73 to 77 in 5bb7de7

	for(n = 0; n < OPEN_MAX; n++) {
	if(current->fd[n]) {
	sys_close(n);
	}
	}

The kernel doesn't remove the fbcon cursor when calling ioctl(ttyfd, KDSETMODE, KD_GRAPHICS). Calling ioctl(ttyfd, KDSETMODE, KD_TEXT) may want to restore the fbcon text console buffer.

Hmm, you're right. I think this ioctl only works in the VGA console (vgacon) but in fbcon, it doesn't takes the cursor into account.

Last line of console is sometimes not scrolled properly; can be duplicated using vi and scrolling up/down but also seen from shell on rarer occasions.

Yes, I know this issue. There are still some VT100 ANSI sequences not supported.

No named pipes, so Nano-X applications are linked directly with the NX server, instead of running client-server.

FIFOs (named pipes) are fully supported.

No struct dirent or DT_DIR using <dirent.h>. Not a big deal.

The header dirent.h exist, of course, otherwise it wouldn't be possible to make a simple directory listing. And regarding the DT_DIR type, check the file include/fiwix/dirent.h.

Running bin/demo-font causes Page Fault then system panic in release_page. Unknown reason, could be NULL pointer in application? (Screenshot available).

I don't know. Where is the screen shot?

Add FreeType 2.10.4 to FiwixOS distribution; this allows for much better font support.

Sure, as long as we can have a graphic environment, the number of packages will increase a lot. It will be just a matter to start porting packages.

Add Microwindows to FiwixOS distribution (will provide commit for this)

Just add a new entry in the script makeall.sh.

Figure out libgcc.a problem on cross compiler for macOS so 64-bit functions can be used.

Perhaps your cross compiler was built without libgcc. I recommend you to go to https://wiki.osdev.org/GCC_Cross-Compiler and rebuild a new one.

Building Microwindows on Fiwix is quite slow... is this possibly because of MINIX v2 filesystem use?

QEMU under Linux includes the option -enable-kvm to speed up emulation. I don't know if there is something similar when QEMU is running under macOS. In any case, block I/O performance in Fiwix is not good, I plan to rewrite it using I/O queues. I'm reading a lot of documentation on the subject. My ToDo list is enlarging it a lot lately. :-)

With a bit more work, the X11->Nano-X libraries could be built, which would allow for some X11 programs to run. But this isn't very feasible until we get Nano-X client/server running and X11 headers and other X11 client GUI source on Fiwix.

I'ld love to see xeyes running on Fiwix. :-)

All-in-all, pretty cool! More screenshots and/or screencasts available if wanted.

I guess that the lack of pty makes impossible to have a terminal under Microwindows/Nano-X. That's another pending feature for the near future.

In all, you did an incredible progress in just a few days. Congratulations!!!
Seeing all these graphic screenshots is like a dream come true.
Thank you very much!

[ghaerr]

Fiwix is pronounced as 'few weeks'

I was going to ask what Fiwix stood for! :)

Are those ioctls fully compatible with Linux?

No. For the first round, I punted as it was enough work figuring out how to map an existing KA into a user VA. Yes, the ioctls should be made compatible.

That's a bit more of a problem for me as, while adding (header) files to the kernel would be easy since they're in the Fiwix repo, I haven't yet figured out how to add files to the C library and/or non-repo /usr/include distribution, and don't want to go through a full distribution "make" to create a runtime image during a fast development cycle. This will be more of a problem given my preferred non-Linux (macOS) development environment.

IMO, what is needed to solve this problem would be the ability to (somehow) add (header?) files to a host-based repo, from which there could be a way to very quickly generate an entire FiwixOS disk image. This is exacerbated by the need to compile everything on a running Fiwix QEMU installation. I have some ideas on this regarding a dual-mechanism to create Fiwix images (both self-hosted and host-hosted) but I'm not sure that's within the goals of Fiwix. More on this later if your are interested.

It appears that Fiwix process exit does not call sys_close on each open file descriptor,
Yes it does:

Hmm, that does seem to. There's still a problem, when SIGINT is generated from ^C to exit Microwindows, it seems fb_close does not get called for some reason. I just checked the kernel/signal.c sources and it seems it should. I'll look further into this.

Yes, I know this issue. There are still some VT100 ANSI sequences not supported.

If you know which ones are required, I can implement them. It would be nice to have vi display properly, since it must be used for development on Fiwix.

FIFOs (named pipes) are fully supported.

Great! I'll change the compilation for Nano-X to client/server and see what happens :) That will be a big improvement.

The header dirent.h exist, of course, otherwise it wouldn't be possible to make a simple directory listing. And regarding the DT_DIR type, check the file include/fiwix/dirent.h.

You're saying both methods of reading directories should be supported... I'll check into that.

Running bin/demo-font causes Page Fault then system panic

I further debugged this, and its failing in the PCF font decoder... not sure exactly why yet, will report more soon. I suspect something to do with malloc incompatibility between systems, but bug is likely in PCF decoder.

Just add a new entry in the script makeall.sh.

I'm looking for that, where is it? I found make_media.sh and install.sh in FiwixOS repo? I haven't yet tried building a system using FiwixOS scripts, not sure that'll work on macOS anyways.

I'ld love to see xeyes running on Fiwix. :-)

Nano-X has xeyes running under X11 compatibility layer... I will compile this up after reviewing X11 issues later.

I guess that the lack of pty makes impossible to have a terminal under Microwindows/Nano-X. That's another pending feature for the near future.

Yes - the terminal demos had to be temporarily removed from the Microwindows build. Not a big deal at the moment, as Nano-X doesn't have a good terminal emulator anyways.

In all, you did an incredible progress in just a few days. Congratulations!!!

Thanks. In years past, I had worked hard on Microwindows such that it could be ported with very few dependencies. After getting a live usermode framebuffer, things progressed quickly.

Microwindows isn't really a desktop GUI environment. It's place in the universe is a full windowing system on small devices or bare hardware. At this point, it is very useful to get Fiwix up to speed with some basics for graphics. I have been thinking of rewriting Microwindows to produce a compositor-based solution. But even then, the problem of what API to bring over is a topic for long discussion, as there are so many, they're all large and very complicated, and mostly incompatible! That might be one of the reasons I like small systems.

[ghaerr]

It appears that Fiwix process exit does not call sys_close on each open file descriptor,
Yes it does
There's still a problem, when SIGINT is generated from ^C to exit Microwindows, it seems fb_close does not get called for some reason.

Chased this down: yes, fb_close is being called through sys_exit as you indicated. The problem I was seeing had to do with releasing the mapped page table entries for the framebuffer, which I was doing inside the fb_close routine. If that wasn't done before the process exited, the system would crash due to my buggy implementation.

sys_exit unmaps the process' PTEs in release_binary before the file descriptors are closed, which triggered my bug, which is that I misunderstood exactly how Fiwix page tables should be constructed and freed. An upcoming PR will include this proper implementation.

After more reading of Fiwix's memory management, page handling and mmap implementation, I now realize a far better solution will be to use mmap on the framebuffer file descriptor to map the physical framebuffer into user memory, which will match Linux. This gets rid of the incompatible ioctls I initially coded to get things running. It also separates the framebuffer map/unmap from the close handling, which also match the way Linux systems work. If a process doesn't munmap the framebuffer before exit, release_binary will do it automatically via the free_vma_pages and free_vma_regions routines.

I'm in the process of rewriting the Fiwix framebuffer and Microwindows drivers for this now.

I haven't yet figured out how to add files to the C library and/or non-repo /usr/include distribution

I think I can "punt" on this and include the linux-compatible frameuffer header files in the Fiwix source along with the rewritten framebuffer map/unmap handling, and use a MIcrowindows driver that for the time being assumes 1024x768. That will allow a Fiwix distribution to automatically place the updated kernel header files in the proper locations in the FiwixOS distribution. After that, I can update the Microwindows and Fiwix drivers to become fully Linux-compatible so that other framebuffer-based graphics applications or libraries should be portable to Fiwix.

[mikaku]

I'm in the process of rewriting the Fiwix framebuffer and Microwindows drivers for this now.

Yes, Fiwix memory management is not very well designed. Moreover, people from the #boostrappable project have problems because the RAMdisk drives are taking memory from the user space, instead of using memory above PAGE_OFFSET. I mean, Fiwix can have two virtual memory splits 3/1 and 2/2 (user/kernel). All the memory above is not used. If you need a big RAMdisk (e.g. 1500MB), your best path is go to 2/2 and you'll have around 400MB for user space regardless if your system has 4GB of physical memory.

I plan to rewrite how RAMdisk drives memory are used and take advantage of the unused memory. I hope our code will not collide. :-)

After that, I can update the Microwindows and Fiwix drivers to become fully Linux-compatible so that other framebuffer-based graphics applications or libraries should be portable to Fiwix.

That would be fantastic because X11 porting could take advantage of this.

Regarding to add the header files in the C library, this is something I do very often. As you might know FiwixOS uses the Newlib C v4.3 with my changes in sys/fiwix/. You might want to take a look into. You'll find it in the FiwixOS 3.3 Installation CDROM.

After the inclusion the UNIX domain socket and other new system calls, I have added new headers and functions in the C library, so the sources in the current Installation CDROM are already outdated.

Let me know if you want to be able to push PRs into the C library, and I'll create the repository.

[ghaerr]

if you want to be able to push PRs into the C library, and I'll create the repository.

Yes, that would be useful for creating the Linux ioctl compatibility in Fiwix, as otherwise I don't have easy access to any of the C library outside of the running QEMU Fiwix image.

Fiwix memory management is not very well designed.
people from the #boostrappable project have problems because the RAMdisk drives are taking memory from the user space, instead of using memory above PAGE_OFFSET.

It seems pretty straight-forwardly designed to me, but I'm still only a week into it :)

Can you explain in a bit more detail what the problem is here? It seems to me that MultiBoot first loads the optional initrd into physical RAM just after the kernel text/data (just before kpage_dir). Then mem_init remaps that initrd RAM above PAGE_OFFSET before loading the final kernel GDT, right?

So how is initrd taking up user space virtual addresses? Aren't all the initrd addresses in KVA?

[mikaku]

Can you explain in a bit more detail what the problem is here? It seems to me that MultiBoot first loads the optional initrd into physical RAM just after the kernel text/data (just before kpage_dir). Then mem_init remaps that initrd RAM above PAGE_OFFSET before loading the final kernel GDT, right?

So how is initrd taking up user space virtual addresses? Aren't all the initrd addresses in KVA?

Initially the Fiwix kernel only had the virtual memory split 3/1 (3GB user / 1GB kernel) and so the maximum available physical memory was 1GB, regardless if your PC had more memory. In this case the RAMdisk drives or the initrd were limited to a maximum of around 950MB, but if you tried to use all the memory, you cannot even login to the system because there would not be enough memory for the user applications.

In #34, @rick-masters suggested a patch to increase the limit of the size in the initrd images to be more than 1GB, which is a requirement for their Bootstrappable project. But somehow we agreed that the patch was a bit tricky.

So, in my to attempt to fix this, I presented a new path, the support to have the virtual memory split 2/2 (2GB user / 2GB kernel) with the new kernel option CONFIG_VM_SPLIT22 (disabled by default). This way you can have easily an initrd of 1.2GB in size and you still have around 750MB for the user space.

Recently though, people in the Bootstrappable project complained that with very large initrd files (let's say 1.5GB) they only have around 450MB for user, even when the system has 4GB of physical memory. They ask if there would be a change to use the unused 2GB of physical memory for the initrd files and the rest for the user space.

I've been testing several (old) Linux kernel versions 2.0, 2.2 and 2.4 and none of these versions is capable to have a RAMdisk drive bigger than 950MB, even when the system has 2GB of physical memory.

I continue thinking how can I implement this in a decent way (no tricky) but so far I'm failing on this.

It's interesting to see that modern Linux kernels have other different memory splits like 2.5GB/1.5GB and 1GB/3GB, to solve this kind of problems.

[mikaku]

If you know which ones are required, I can implement them. It would be nice to have vi display properly, since it must be used for development on Fiwix.

Yes, I know is very annoying. It forces me to keep refreshing the screen with CTRL+L all the time.
Sincerely, I'm not sure if this is a lack of support or just a bug in the console.

I'm looking for that, where is it? I found make_media.sh and install.sh in FiwixOS repo? I haven't yet tried building a system using FiwixOS scripts, not sure that'll work on macOS anyways.

The makeall.sh script is used to build all the user space packages. You'll find it in the FiwixOS repository inside the iso/install/pkgs/src/ directory. I recommend you to create a new ATA disk image (4GB) to build the packages and the GNU Toolchain. I have the FiwixOS 3.3 disk image in /dev/hda, and an ATA disk image of 4GB in /dev/hdd that contains all the src/ directory in the FiwixOS repository. Every time I port a new file, I just mount the filesystem /dev/hdd1 into /mnt/disk, then I modify the script makeall.sh with the changes needed by the new package and I test the build.