Add Windows support

[jezdez]

This is related to the things we discussed at PyCon

[benoitc]

wip. planned for R18.

[benoitc]

Some update on that:

• SOCKET support is quite done, a branch will be posted online next week

TODO:

handle service start/stop/restart to use gunicorn as a full windows service. It is replacing somehow the use of signals under unix.

[benoitc]

the support will finally happen in R20. i will create the new PR to track the development of this feature next week.

Things to fix are:

• adapt Worker notifications to windows
• port the signaling to windows: reload and reexec
• make sure sockets can be inherited among processes

[renierdbruyn]

What is the progress on adding windows support for gunicorn?. Or rather How far are you from R20? I am having the same problem as #587

[benoitc]

Slow progress. I want to release R19 with the threaded work now. I will push a version on monday. the windows support will arrive right after.

[helmus]

Is this windows supported version available somewhere ?

[renierdbruyn]

Windows support will be in R20, so I guess it will be a while till guincorn will have windows support... I created a vm with Ubuntu, and this works for me.....

[robertlagrant]

Just curious as to any progress in this. As a first pass it doesn't even need to be fast; a big advantage will be just being able to develop with Gunicorn on Windows and push to a Linux server (e.g. Heroku) without changing config.

[benoitc]

@robertlagrant it's on my todo. I need to find a windows license first. R20 will be released in october,

[helmus]

How about the VM images at modern.ie ? Those are free and legal
On Sep 22, 2014 3:30 PM, "Benoit Chesneau" notifications@github.com wrote:

@robertlagrant https://github.com/robertlagrant it's on my todo until
the end of the year. I need to find a windows license first.

—
Reply to this email directly or view it on GitHub
#524 (comment).

[tilgovi]

I used modern.ie images to test a web app in IE8 recently. Easy to get going and worked great.

[cyberdelia]

@benoitc If needed, I believe the PSF can provide you with a windows license.

[berkerpeksag]

@benoitc is there a WIP branch for this? I'm not a Windows user, but I can take a look at this.

[benoitc]

@berkerpeksag no not yet :/ we should really start one asap.

[benoitc]

like @cyberdelia said we could ask a license of windows at the PSF that the gunicorn project could use for such development. Thoughts?

[berkerpeksag]

It's a good idea. https://www.python.org/psf/grants/ has all information about grant proposals.

[zet4]

Any updates on this?

[jamespedid]

I am also looking for updates to this.

[benoitc]

@berkerpeksag let's have a talk asap on IRC, just ping me sometimes before since i'm away from any instant thing these days :)

[johann-petrak]

Is there a rough idea how it could be done on Windows to support proper multiprocessing?
Do we know what the major obstacles are so far for this getting implemented?

[platinops]

i may have some cycle yo spend on it during march. I will need to find a way to execute windows somewhere. you can already run it in the terminal anyway.

Could the Win10 dev VM's be a solution for you to run Windows and check gunicorn compatibility?

[arihant2math]

Perhaps we can use pywin32 for windows instead of fnctl on windows, it seems they have file locking capabilities .

[bjones1]

I've written #2636 which fixes about half of the problems porting gunicorn to Windows. The other half requires finding a Windows-compatible alternative to using os.pipe. Could we simply use a multiprocessing listener and client instead of the existing os.pipe?

• It doesn't require using any other part of multiprocessing, so it avoid that baggage/overhead.
• It produces a Connection object, which allows sending raw bytes for efficiency.
• It has a select-like function.
• It's (of course) multi-platform.

[benoitc]

I would admit that windows is a really low priority for me. Gunicorn is done first to be deployed on POSIX/UNIX platforms. That’s a desirable goal anyway :) But in any case we need a more native way to handle it close to windows libraries/system rather than using anything from multiprocessing. So we can understand how it works and optimise when needed without depending on a third party. This should work until Python stops to expose low level libraries. What we need is a way to share a socket between threads on windows and basic signaling. I think it would be good to list common patterns in windows and pick one for our implementation. Let’s start by doing this and we will see how it can be handled our way? Le sam. 28 août 2021 à 09:34, Bryan A. Jones ***@***.***> a écrit :

I've written #2636 <#2636> which fixes about half of the problems porting gunicorn to Windows. The other half requires finding a Windows-compatible alternative to using os.pipe. Could we simply use a multiprocessing listener and client <https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.connection> instead of the existing os.pipe? - It doesn't require using any other part of multiprocessing, so it avoid that baggage/overhead. - It produces a Connection <https://docs.python.org/3/library/multiprocessing.html#multiprocessing.connection.Connection> object, which allows sending raw bytes for efficiency. - It has a select-like function <https://docs.python.org/3/library/multiprocessing.html#multiprocessing.connection.wait> . - It's (of course) multi-platform. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#524 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADRIUEFRFIS3VU6ZQDWBDT7CGPNANCNFSM4AE7ECYA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Sent from my Mobile

[bjones1]

@benoitc, thanks for your feedback. Do we need only inter-thread synchronization, or is this actually inter-process synchronization? I assume inter-process, since you mention sockets and inter-thread is well supported already by the Python library.

[benoitc]

You’re right I meant inter-process synchronisation (or system threads) .I am pretty sure there is a signaling API that could do it. Le sam. 28 août 2021 à 10:18, Bryan A. Jones ***@***.***> a écrit :

@benoitc <https://github.com/benoitc>, thanks for your feedback. Do we need only inter-thread synchronization, or is this actually inter-process synchronization? I assume inter-process, since you mention sockets and inter-thread is well supported already by the Python library. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#524 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADRIWL3L6OPDDFAYA2RMDT7CLW5ANCNFSM4AE7ECYA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Sent from my Mobile

[bjones1]

I would admit that windows is a really low priority for me. Gunicorn is
done first to be deployed on POSIX/UNIX platforms. That’s a desirable goal
anyway :)

Thanks for taking the time to look at this. I really appreciate it. I know your time is very limited.

But in any case we need a more native way to handle it close to windows
libraries/system rather than using anything from multiprocessing. So we
can understand how it works and optimise when needed without depending on a
third party. This should work until Python stops to expose low level
libraries.

I'm a bit confused here. multiprocessing is from the Python standard library, so it's not a third-party library.

You’re right I meant inter-process synchronisation (or system threads) .I
am pretty sure there is a signaling API that could do it.

That's what I thought, but I wanted to check...

[benoitc]

Le sam. 28 août 2021 à 10:31, Bryan A. Jones ***@***.***> a écrit :

I would admit that windows is a really low priority for me. Gunicorn is done first to be deployed on POSIX/UNIX platforms. That’s a desirable goal anyway :) Thanks for taking the time to look at this. I really appreciate it. I know your time is very limited. But in any case we need a more native way to handle it close to windows libraries/system rather than using anything from multiprocessing. So we can understand how it works and optimise when needed without depending on a third party. This should work until Python stops to expose low level libraries. I'm a bit confused here. multiprocessing <https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.connection> is from the Python standard library, so it's not a third-party library.

multiprocessing is an abstraction over systems calls. I would really prefer we handle directly native calls so we can change any time or use only the part we need. If really needed we can also create a binding for the one not exposed.

You’re right I meant inter-process synchronisation (or system threads) .I am pretty sure there is a signaling API that could do it. That's what I thought, but I wanted to check... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#524 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADRIRN6UJAJ45RO43O74DT7CNHJANCNFSM4AE7ECYA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Sent from my Mobile

[bjones1]

multiprocessing is an abstraction over systems calls. I would really prefer
we handle directly native calls so we can change any time or use only the
part we need. If really needed we can also create a binding for the one not
exposed.

OK, sounds good. I'll look for signaling / socket IPC mechanisms and let you know what I find.

[tilgovi]

I would love to use multiprocessing, if it's appropriate. These standard library abstractions are a great thing about the language, especially because the low level implementation details for CPython are often well specified in the documentation.

If multiprocessing, or any other standard module, implements the right windows platform features to do what we need, we should consider using it. If it doesn't, we won't.

[tilgovi]

In the past, then I've read multiprocessing docs, I haven't been convinced! But I'd be open to being educated if it does seem appropriate.

I'm also open to measuring the impact if it turns out slower but vastly simpler.

[bjones1]

@tilgovi, thanks for your feedback. Unless @benoitc disagrees, I'll pursue an initial implementation using multiprocessing. With a working, multi-platform solution, we can then stay with the standard library or customize this working code based on the feedback I receive. I'd appreciate any other thoughts / ideas on the architecture you have -- would you mind posting them on #2637? Thanks again.

[benoitc]

well, i hate to repeat myself. I would like to have a clean implementation not based on multiprocessing. It's not about optimisation but also in the process learning and make gunicorn a better citizen in windows.

Multiprocessing and gunicorn doesn't share the same philosophy in handling multiprocessing. It's OK however to use low level python bindings, some coming with the multiprocessing. But definitly not higher level. We have to be cautious. And definitly removing anything using shared memory.

[johann-petrak]

I do not see how something not based on a standard python module but some native library should be the "cleaner" solution?
Clearly, multiprocessing is a part of Python that will get maintained and receive bug fixes in the future while depending on some native library may easily break things in the future.
Sure it makes sense to restrict the implementation to a relevant part of the abstractions present in multiprocessing or deal with situations where something needed is missing from the module, but why reinvent the wheel?

[bjones1]

I believe good code argues for itself. Would the gunicorn team be willing to look at an initial implementation based only on the inter-process communication components of multiprocessing (not the overall multiprocessing approach, which I also see as a poor fit for this project)? If this works, we can either manually edit the code (@benoitc's preference, I think), or rely on the ongoing maintenance and improvements provided by direct use of the Python standard library.

If not, I can look at a socket-based solution, or any of the alternatives listed in #2637.

[benoitc]

there are many reason to not use multiprocessing. One of them is to be able to work with other possible workers and provide a facility that work for all workers. It is also to ensure our maintenance window. Something really hard with multiprocessing. And I also like to not have depends on optinion of others about what is interprocess-communication.

My preference is the following:

• Have al the code handling multiprocessing embed in Gunicorn. This allows easy checking, and isolation.
• if C-code is needed then provide them directly in Gunicorn.
• When the a stdlib implementation is safe (thread safe, not sharing memory between processes) then use it
• Fork internally if needed for compatibility across version.

@bjones1 if you believe the code can be used for example to spawn workiers that include gevent and check their liveness we can go with it. Better to speak over the code anyway.

[bjones1]

@benoitc, my only goal is to provide inter-process communication code. I don't want to change or even touch the way gunicorn spawns/manages processes. As you say, I'll write code and listen to your feedback. A quick note: I'm @bjones1; sadly, @bjones claimed that userid before I joined github :(.

[tilgovi]

This is great. I see a lot of alignment. I'll give feedback on the issue you opened. Nice to see movement on this.

I agree with everyone so far. It is good to be in control, avoid high level abstractions that don't do exactly what we need, but also not reinvent the wheel. So we will evaluate deeper, together, as we look at real code possibilities.

[bjones1]

I've started work on this in #2643. Comments welcome.

[Andrioden]

Any news here? : )