New osrandom_engine in C #3229

tiran · 2016-11-10T22:59:50Z

Inspired by Python/random.c and the old implementation. Let's ask travis if it works on all platforms.

Signed-off-by: Christian Heimes christian@python.org

Test /dev/urandom path
Test getrandom() path
Test getentropy() path
Test CryptGenRandom path

mention-bot · 2016-11-10T22:59:52Z

@tiran, thanks for your PR! By analyzing the history of the files in this pull request, we identified @reaperhulk, @glyph and @alex to be potential reviewers.

glyph · 2016-11-10T23:10:13Z

On the face of it, this is a terrible idea; ~400 lines of C to replace ~40 lines of Python; 2 open file descriptors for your whole process lifetime instead of 1.

However, knowing the history of this I am sure there's a very good reason for this change. Can you link it to an issue describing said reasons, please? I think it's super important to thoroughly document what is going on here, since it is incredibly confusing; without very clear explication I am concerned that a future maintainer will haplessly reintroduce the bug that this is fixing.

(Secretly, of course, I want to read that description and have a brilliant idea that will salvage the Python implementation and not have to drag around half a thousand lines of duplicative tricky wrapper C code; alas, if I haven't by now, I very much doubt I'll be able to.)

tiran · 2016-11-10T23:27:22Z

Ha, mentionbot betrayed me to @glyph :)

@reaperhulk ask me to look into the matter. My patch is still in a very early phase. Basically I'm just abusing the build and CI testing to test my quick hack on other platforms. I'll take care of documentation later.

reaperhulk · 2016-11-11T02:27:21Z

@glyph: The answer is that we run into endless problems with calling into Python from C on the margins. The most recent of which were apps like Kodi, which embed Python as a scripting option. A Kodi plugin adds cryptography, which registers a callback into Python...but there's only one instance of OpenSSL in the process so now things that aren't even aware of Python are attempting to call back into Python. You could argue that disabling the osrandom engine is the right solution in such a scenario but "right" and "things people will do" are not always overlapping sets.

We also continue to have subinterpreter problems which are probably soluble but I am so very tired of trying to fix it when moving it entirely to C will instantly resolve this entire class of issue.

Combine that with our interpreter shutdown issues in newer Pythons and the fact that we need to entirely abandon our Python-based locking callbacks (see: #3226 and linked issues) and there's a strong incentive to move this entirely to C because frankly users will have less problems.

The cost is that we are once again hoisting the complexity CPython formerly provided for us into our own domain and will be responsible for managing that. I don't believe we can continue to say "this isn't a big deal" though. Every time cryptography fails in a weird, undebuggable way when dangerous, unsupported libraries like PyCrypto do not we lose.

Sorry for the semi-rambling response, but I thought you deserved a reasonable dump of my thoughts on why this has become a desirable scenario. The tl;dr is "being reliably functional in all the environments Python is used in trumps the ideological purity of using os.urandom"

reaperhulk · 2016-11-11T02:33:47Z

@tiran Thanks for the awesome work here so far. I look forward to reviewing it.

As an aside: CCRandomGenerateBytes might be an avenue we could pursue in a future PR on the mac side (available in 10.10+). SecRandomCopyBytes is also available since 10.7 but we'd have to investigate the implementation of each of those to see if they're truly suitable.

tiran · 2016-11-12T18:19:28Z

@reaperhulk The PR is ready for review. I added a CCRandomGenerateBytes implementation.

tiran · 2016-11-12T18:49:55Z

All Net/Free/Open BSD seem to use the same syscall name SYS_getentropy. I'm using #ifdef SYS_getentropy to select getentropy() instead of /dev/urandom.
https://bz-attachments.freebsd.org/attachment.cgi?id=149067

reaperhulk

First round of review, haven't looked closely at the actual functionality yet

reaperhulk · 2016-11-12T20:33:14Z

src/_cffi_src/openssl/src/osrandom_engine.c

+/****************************************************************************
+ * BSD getentropy
+ */
+#if CRYPTOGRAPHY_OSRANDOM_ENGINE == CRYPTOGRAPHY_OSRANDOM_ENGINE_GETRANDOM


This should be CRYPTOGRAPHY_OSRANDOM_ENGINE_GETENTROPY

reaperhulk · 2016-11-12T20:33:30Z

src/_cffi_src/openssl/src/osrandom_engine.c

+static int osrandom_rand_status(void) {
+    return 1;
+}
+#endif /* CRYPTOGRAPHY_OSRANDOM_ENGINE_GETRANDOM */


This comment should be CRYPTOGRAPHY_OSRANDOM_ENGINE_CC_RANDOM

Thx, I messed up some find and replace operations. The error was consistent over all files so it just worked.

reaperhulk · 2016-11-12T20:34:17Z

src/_cffi_src/openssl/src/osrandom_engine.c

+static int osrandom_rand_status(void) {
+    return 1;
+}
+#endif /* CRYPTOGRAPHY_OSRANDOM_ENGINE_GETRANDOM */


CRYPTOGRAPHY_OSRANDOM_ENGINE_GETENTROPY

reaperhulk · 2016-11-12T20:34:58Z

src/_cffi_src/openssl/src/osrandom_engine.c

+        }
+    }
+}
+#endif


Should this have a CRYPTOGRAPHY_OSRANDOM_NEEDS_DEV_URANDOM comment?

reaperhulk · 2016-11-12T20:35:10Z

src/_cffi_src/openssl/src/osrandom_engine.c

+ * Linux getrandom engine with fallback to dev_urandom
+ */
+
+#if CRYPTOGRAPHY_OSRANDOM_ENGINE == CRYPTOGRAPHY_OSRANDOM_ENGINE_GETENTROPY


CRYPTOGRAPHY_OSRANDOM_ENGINE_GETRANDOM

reaperhulk · 2016-11-12T20:35:17Z

src/_cffi_src/openssl/src/osrandom_engine.c

+    }
+    return 1;
+}
+#endif /* CRYPTOGRAPHY_OSRANDOM_ENGINE_GETENTROPY */


CRYPTOGRAPHY_OSRANDOM_ENGINE_GETRANDOM

reaperhulk · 2016-11-12T20:51:15Z

src/_cffi_src/openssl/src/osrandom_engine.h

+   /* for defined(BSD) */
+  #include <sys/param.h>
+
+  #ifdef BSD


Does FreeBSD define this? I think they typically use __FreeBSD__. What about macOS (which apparently has SYS_getentropy as of 10.12)?

Yes, it's defined by Free, Open, Net and Dragon BSD in sys/param.h

http://www.boost.org/doc/libs/1_60_0/libs/predef/doc/html/predef/reference/boost_os_operating_system_macros.html
http://nadeausoftware.com/articles/2012/01/c_c_tip_how_use_compiler_predefined_macros_detect_operating_system#BSD

reaperhulk · 2016-11-12T20:52:39Z

src/_cffi_src/openssl/src/osrandom_engine.h

+  #define CRYPTOGRAPHY_OSRANDOM_ENGINE CRYPTOGRAPHY_OSRANDOM_ENGINE_CC_RANDOM
+#elif defined(BSD) && defined(SYS_getentropy)
+  /* OpenBSD 5.6+ */
+  #define CRYPTOGRAPHY_OSRANDOM_ENGINE CRYPTOGRAPHY_OSRANDOM_ENGINE_GETRANDOM


This should be CRYPTOGRAPHY_OSRANDOM_ENGINE_GETENTROPY with the naming fixes requested above

reaperhulk · 2016-11-12T20:52:54Z

src/_cffi_src/openssl/src/osrandom_engine.h

+  #define CRYPTOGRAPHY_OSRANDOM_ENGINE CRYPTOGRAPHY_OSRANDOM_ENGINE_GETRANDOM
+#elif defined(__linux__) && defined(SYS_getrandom)
+  /* Linux 3.4.17+ */
+  #define CRYPTOGRAPHY_OSRANDOM_ENGINE CRYPTOGRAPHY_OSRANDOM_ENGINE_GETENTROPY


And this should be CRYPTOGRAPHY_OSRANDOM_ENGINE_GETRANDOM

reaperhulk · 2016-11-12T20:54:43Z

src/_cffi_src/openssl/src/osrandom_engine.h

+  #define CRYPTOGRAPHY_OSRANDOM_NEEDS_DEV_URANDOM 1
+#endif
+
+/* Open SSL 1.1.0+ has no ERR_R_RAND_LIB */


What do you think about always passing 0 and not worrying about a macro here?

reaperhulk · 2016-11-12T23:03:42Z

Coverage is failing because _osrandom_rand_bytes and _osrandom_rand_status need to be removed from src/cryptography/hazmat/bindings/openssl/binding.py

reaperhulk · 2016-11-13T14:51:23Z

tests/hazmat/backends/test_openssl.py

+        assert name
+        if sys.platform.startswith('linux'):
+            assert name in ['getrandom', '/dev/urandom']
+        elif sys.platform == 'win32':


Our CI can't cover this branch (we don't get coverage from our windows builders, although this is being executed) so we'll need to figure out a way to rewrite this. Maybe make the asserts a separate function that takes a string _engine_name_assert(sys.platform) and then

def _engine_name_assert(platform): if platform.startswith('linux'): assert name in ['getrandom', '/dev/urandom'] elif platform == 'win32': assert name == 'CryptGenRandom'

and then a separate test for _engine_name_assert that explicitly passes win32?

alex · 2016-11-13T22:33:14Z

Failing docs tests are relevant; looks like fallback needs to be added to the wordlist :-)

tiran · 2016-11-14T10:44:09Z

I added fallback to spelling wordlist. On my machine the doc tests are failing with a different error. My spell checker knows about fallback but does not recognize personalization:

hazmat/primitives/cryptographic-hashes.rst:129: (personalization)

reaperhulk · 2016-11-14T12:59:50Z

https://codecov.io/gh/pyca/cryptography/compare/3a15b03e92c9fdeadff04ddd2ce505028b279b86...075de30232f1fd6f1e0d541844e01eab93a006b3#74657374732F68617A6D61742F6261636B656E64732F746573745F6F70656E73736C2E7079-280...281 is our coverage problem that's causing the status check failures.

We do have FreeBSD 10 (but not 11) in our jenkins cluster, but no OpenBSD and we don't get coverage from those builders right now. 😢

In an ideal world what would CI look like here?

vstinner · 2016-11-15T10:48:08Z

src/_cffi_src/openssl/src/osrandom_engine.c

+        if (urandom_cache.fd >= 0) {
+            do {
+                n = close(fd);
+            } while (n < 0 && errno == EINTR);


It would be kind to call PyErr_CheckSignals() to fail fast on CTRL+C: PEP 475.

The engine does not not use any Python API at all. I don't consider SIGINT a problem here. The engine is a low level part of OpenSSL's CPRNG. The CPRNG API should take care of of interrupts.

Is closign an fd in a loop safe? I seem to recall it being dangerous :-/

reaperhulk

Looks like this needs a rebase as well now

reaperhulk · 2016-11-29T14:01:21Z

docs/hazmat/backends/openssl.rst

@@ -40,6 +40,10 @@ greater.
        Activates the OS random engine. This will effectively disable OpenSSL's
        default CSPRNG.

+    .. method:: osrandom_engine_implementation()


.. versionadded:: 1.7 please

changes made

alex · 2016-12-06T02:15:05Z

LICENSE

-The code used in the OpenSSL locking callback is derived from the same in
-Python itself, and is licensed under the terms of the PSF License Agreement.
+The code used in the OpenSSL locking callback and OS random engine is derived
+from the same in Python itself, and is licensed under the terms of the PSF


Ugh, nit, not technically a bug in your PR (but if I go fix it separately it'll just cause a conflict). Can you s/Python/CPython/

alex · 2016-12-06T02:15:43Z

docs/hazmat/backends/openssl.rst

+| Windows                                  | ``CryptGenRandom()``         |
+------------------------------------------+------------------------------+
+| Linux >= 3.4.17 with working             | ``getrandom(GRND_NONBLOCK)`` |
+| ``SYS_getrandom`` syscall                |                              |


You can drop the leading SYS here.

Unless you really insist, I'd rather keep SYS_getrandom because that is what the random engine actually uses. It's going to make it easier to distinguish between platforms with the syscall and platforms with a glibc wrapper later.

alex · 2016-12-06T02:18:22Z

src/_cffi_src/openssl/src/osrandom_engine.h

+    #ifndef GRND_NONBLOCK
+      #define GRND_NONBLOCK 0x0001
+    #endif /* GRND_NONBLOCK */
+  #endif /* _linux__ */


Missing leading _ here

alex · 2016-12-06T02:18:32Z

src/_cffi_src/openssl/src/osrandom_engine.h

+  #ifdef __linux__
+    /* for SYS_getrandom */
+    #include <sys/syscall.h>
+    #ifndef GRND_NONBLOCK


In what circumstances it this not defined?

The constant is defined in user land headers. Sometimes the running Kernel has the syscall but the userland hasn't caught up yet.

alex · 2016-12-06T02:21:28Z

tests/hazmat/backends/test_openssl.py

@@ -282,6 +271,23 @@ def test_activate_builtin_random_already_active(self):
        e = backend._lib.ENGINE_get_default_RAND()
        assert e == backend._ffi.NULL

+    def test_osrandom_engine_implementation(self):
+        name = backend.osrandom_engine_implementation()
+        assert name in ['/dev/urandom', 'CCRandomGenerateBytes',


CCRandomGenerateBytes is not used anywhere.

alex · 2016-12-06T02:23:30Z

src/_cffi_src/openssl/src/osrandom_engine.c

+    }
+}
+
+static const char* osurandom_get_implementation(void) {


* next to the function name, rather than the type please.

alex · 2016-12-06T02:25:05Z

src/_cffi_src/openssl/src/osrandom_engine.c

+        if (urandom_cache.fd >= 0) {
+            do {
+                n = close(fd);
+            } while (n < 0 && errno == EINTR);


Is closign an fd in a loop safe? I seem to recall it being dangerous :-/

alex · 2016-12-06T02:27:12Z

src/_cffi_src/openssl/src/osrandom_engine.c

+            /* Can fail with:
+             * - ENOSYS: Kernel does not support the syscall.
+             * - ENOPERM: seccomp prevents syscall.
+             * - EAGAIN: Kernel CRPNG has not been seeded yet.


In the even of EAGAIN I'm pretty sure we want to still use getrandom, not fall back.

Let's discuss this matter on IRC later. I don't want to repeat the /dev/urandom discussion from python-dev on github. :)

Ok, you know how to ping me.

Please give link to that conversation.

It would be nice to place explanation in documentation.

alex · 2016-12-06T02:27:53Z

src/_cffi_src/openssl/src/osrandom_engine.c

+        long n;
+        while (size > 0) {
+            do {
+                n = syscall(SYS_getrandom, buffer, size, GRND_NONBLOCK);


What does this do on large (>256) size?

http://man7.org/linux/man-pages/man2/getrandom.2.html

If the
/dev/urandom pool has been initialized, reads of up to 256 bytes will
always return as many bytes as requested and will not be interrupted
by signals. No such guarantees apply for larger buffer sizes. For
example, if the call is interrupted by a signal handler, it may
return a partially filled buffer, or fail with the error EINTR.

alex · 2016-12-06T02:28:21Z

src/_cffi_src/openssl/src/osrandom_engine.c

+}
+
+static int osrandom_rand_status(void) {
+    if (urandom_cache.fd < 0) {


Simplifies to return urandom_cache.fd > 0

>= 0 or != -1. Theoretically the fd can by 0,

Inspired by Python/random.c and the old implementation. Signed-off-by: Christian Heimes <christian@python.org>

* Fix naming bug caused by search 'n replace mistake * Make it easier to override osrandom auto-detection * Add engine ctrl and backend API to get implementation from ENGINE Signed-off-by: Christian Heimes <christian@python.org>

Signed-off-by: Christian Heimes <christian@python.org>

* read() returns size_t Signed-off-by: Christian Heimes <christian@python.org>

This change allows us to test all the engines in our CI: * getentropy (tested by macOS sierra) * getrandom (tested on several linux builders) * /dev/urandom (tested on FreeBSD, OS X 10.11 and below, & older linux) * CryptGenRandom (tested on windows builders) I also fixed bugs preventing compilation in the getentropy code

Signed-off-by: Christian Heimes <christian@python.org>

alex · 2016-12-07T21:43:55Z

The EAGAIN issue is the last remaining issue for me.

Add error reporting strings for various error cases. This gives us much nicer and understandable error messages. SYS_getrandom() EAGAIN is now an error. Cryptography refuses to initialize its osrandom engine when the Kernel's CPRNG hasn't been seeded yet. Signed-off-by: Christian Heimes <christian@python.org>

1.7.1 - 2016-12-13 ~~~~~~~~~~~~~~~~~~ * Fixed a regression in ``int_from_bytes`` where it failed to accept ``bytearray``. 1.7 - 2016-12-12 ~~~~~~~~~~~~~~~~ * Support for OpenSSL 1.0.0 has been removed. Users on older version of OpenSSL will need to upgrade. * Added support for Diffie-Hellman key exchange using :meth:`~cryptography.hazmat.primitives.asymmetric.dh.DHPrivateKeyWithSerialization.exchange` * The OS random engine for OpenSSL has been rewritten to improve compatibility with embedded Python and other edge cases. More information about this change can be found in the `pull request <https://github.com/pyca/cryptography/pull/3229>`_.

tiran force-pushed the new_osrandom_engine branch from b96bbe5 to 26836ff Compare November 10, 2016 23:19

tiran force-pushed the new_osrandom_engine branch 4 times, most recently from 77e9f2b to 1d38d6f Compare November 12, 2016 16:37

tiran changed the title ~~New osrandom_engine in C (WIP)~~ New osrandom_engine in C Nov 12, 2016

tiran force-pushed the new_osrandom_engine branch from 1d38d6f to aa43143 Compare November 12, 2016 18:44

tiran force-pushed the new_osrandom_engine branch from aa43143 to 6ed7f91 Compare November 12, 2016 20:06

reaperhulk reviewed Nov 12, 2016

View reviewed changes

tiran force-pushed the new_osrandom_engine branch 2 times, most recently from da3f779 to 807f437 Compare November 13, 2016 12:50

reaperhulk reviewed Nov 13, 2016

View reviewed changes

tiran force-pushed the new_osrandom_engine branch 2 times, most recently from 952ebe9 to 2f00fae Compare November 14, 2016 10:43

tiran force-pushed the new_osrandom_engine branch from 2f00fae to 075de30 Compare November 14, 2016 10:57

tiran force-pushed the new_osrandom_engine branch 3 times, most recently from 669295e to d4ae881 Compare November 15, 2016 07:48

vstinner reviewed Nov 15, 2016

View reviewed changes

tiran force-pushed the new_osrandom_engine branch from 19e1511 to e7ddb5f Compare November 22, 2016 10:12

This was referenced Nov 24, 2016

@ffi.def_extern() was not called in the current subinterpreter #3282

Closed

Regression: Cannot use cyrptography in embedded wsgi #2299

Closed

reaperhulk previously requested changes Nov 29, 2016

View reviewed changes

tiran force-pushed the new_osrandom_engine branch from e7ddb5f to c67e987 Compare November 29, 2016 14:31

alex reviewed Dec 6, 2016

View reviewed changes

tiran and others added 11 commits December 6, 2016 10:30

New osrandom_engine in C

e5ebfeb

Inspired by Python/random.c and the old implementation. Signed-off-by: Christian Heimes <christian@python.org>

osrandom_engine

1cf7222

* Fix naming bug caused by search 'n replace mistake * Make it easier to override osrandom auto-detection * Add engine ctrl and backend API to get implementation from ENGINE Signed-off-by: Christian Heimes <christian@python.org>

Better test coverage, documentation, LICENSE

247ea25

Signed-off-by: Christian Heimes <christian@python.org>

Coverage is hard.

fcec330

Signed-off-by: Christian Heimes <christian@python.org>

* enable win32 check

cb6f9e2

* read() returns size_t Signed-off-by: Christian Heimes <christian@python.org>

Add macOS to spelling list. Remove dead code from header file.

ad2d09e

Signed-off-by: Christian Heimes <christian@python.org>

getentropy() returns int and is restricted to 256 bytes on macOS, too.

158ba3b

Signed-off-by: Christian Heimes <christian@python.org>

add versionadded

086ebbf

Re-add import of os module

f6bc160

Fixes related to Alex's recent review.

8575fb2

Signed-off-by: Christian Heimes <christian@python.org>

tiran force-pushed the new_osrandom_engine branch from c05ea01 to 8575fb2 Compare December 6, 2016 09:52

tiran force-pushed the new_osrandom_engine branch from 8d8df86 to 2abbab4 Compare December 8, 2016 18:50

reaperhulk approved these changes Dec 9, 2016

View reviewed changes

reaperhulk merged commit 2e71776 into pyca:master Dec 9, 2016

tiran deleted the new_osrandom_engine branch December 9, 2016 16:51

reaperhulk mentioned this pull request Dec 11, 2016

Update FAQ Documentation Re: Subinterpreters #3308

Closed

noam09 mentioned this pull request Dec 13, 2016

Can't do anything, crash. Kolifanes/plugin.video.youtube#60

Closed

reaperhulk mentioned this pull request Mar 19, 2017

osrandom engine already registered #2287

Closed

github-actions bot locked as resolved and limited conversation to collaborators Aug 4, 2020

New osrandom_engine in C #3229

New osrandom_engine in C #3229

Conversation

tiran commented Nov 10, 2016 • edited by reaperhulk Loading

mention-bot commented Nov 10, 2016

glyph commented Nov 10, 2016

tiran commented Nov 10, 2016

reaperhulk commented Nov 11, 2016

reaperhulk commented Nov 11, 2016

tiran commented Nov 12, 2016

tiran commented Nov 12, 2016

reaperhulk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reaperhulk commented Nov 12, 2016

Choose a reason for hiding this comment

alex commented Nov 13, 2016

tiran commented Nov 14, 2016

reaperhulk commented Nov 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reaperhulk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex commented Dec 7, 2016

tiran commented Nov 10, 2016 •

edited by reaperhulk

Loading