Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New source creation uses Sequoia #6799

Closed
Tracked by #6399
legoktm opened this issue May 17, 2023 · 2 comments · Fixed by #6892
Closed
Tracked by #6399

New source creation uses Sequoia #6799

legoktm opened this issue May 17, 2023 · 2 comments · Fixed by #6892
Assignees
Labels
Rust Issues that touch Rust code

Comments

@legoktm
Copy link
Member

legoktm commented May 17, 2023

As part of our work to migrate to Sequoia, the first step is to have new source creation generate a PGP key pair using Sequoia, and then use those keys for encryption/decryption.

This work will include:

  • Writing Sequoia bindings for Python
  • Converting key storage to be in the database
  • Switching over new source creation to use those functions and store said keys in the database.

Most of this work already exists in the oxidize branch

@legoktm legoktm changed the title New source creation uses Sequoia (done in oxidize branch) New source creation uses Sequoia May 17, 2023
@legoktm legoktm added the Rust Issues that touch Rust code label May 17, 2023
@zenmonkeykstop zenmonkeykstop moved this to Cycle Backlog in SecureDrop dev cycle May 19, 2023
@zenmonkeykstop zenmonkeykstop moved this from Cycle Backlog to In Progress in SecureDrop dev cycle Jun 20, 2023
@zenmonkeykstop zenmonkeykstop added this to the SecureDrop 2.7.0 milestone Jul 12, 2023
@legoktm
Copy link
Member Author

legoktm commented Jul 18, 2023

What is coming in this PR:

  • Storing journalist public key out of the GPG keyring - including a hopefully not too hacky postinst migration
  • DB schema change, adding 3 columns to the source table
  • Changing all the EncryptionManager.encrypt_* functions to use redwood, and sometimes the decrypt function
  • Actually switching the new source creation to Sequoia

Honestly I've spent most of the time getting tests to pass, including:

  • Getting rid of all EncryptionManager._gpg direct calls and fixing all the other EncryptionManager calls really.
  • Encrypting the journalist test key with a passphrase because Sequoia wants that

Except some tests are failing with redwood.RedwoodError: OpenPgp(Missing session key: No session key decrypted) in my new utils.decrypt_as_journalist() function. I'm trying to get everything else passing before I dive into this one.

@legoktm
Copy link
Member Author

legoktm commented Jul 19, 2023

The missing session key issue seems to be with encrypting a message for multiple recipients. Here's a pure-Rust test case that fails:

    #[test]
    fn test_multiple_recipients() {
        // Generate 2 keys
        let (public_key1, secret_key1, _fingerprint1) = generate_source_key_pair(
            "correcthorsebatterystaple",
            "foo1@example.org",
        )
        .unwrap();
        let (public_key2, secret_key2, _fingerprint2) = generate_source_key_pair(
            "correcthorsebatterystaple",
            "foo2@example.org",
        )
        .unwrap();

        let tmp = NamedTempFile::new().unwrap();
        // Encrypt a message
        encrypt_message(
            vec![public_key1, public_key2],
            "Rust is great 🦀".to_string(),
            tmp.path().to_path_buf(),
        )
        .unwrap();
        let ciphertext = std::fs::read_to_string(tmp.path()).unwrap();
        // Verify ciphertext looks like an encrypted message
        assert!(ciphertext.starts_with("-----BEGIN PGP MESSAGE-----\n"));
        // Decrypt as key 1
        let plaintext = decrypt(
            ciphertext.clone().into_bytes(),
            secret_key1,
            "correcthorsebatterystaple".to_string(),
        )
        .unwrap();
        // Verify message is what we put in originally
        assert_eq!("Rust is great 🦀", &plaintext);
        // Decrypt as key 2
        let plaintext = decrypt(
            ciphertext.into_bytes(),
            secret_key2,
            "correcthorsebatterystaple".to_string(),
        )
        .unwrap(); // <-- fails here
        // Verify message is what we put in originally
        assert_eq!("Rust is great 🦀", &plaintext);
    }

legoktm added a commit that referenced this issue Jul 21, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like.

Fixes #6799.
legoktm added a commit that referenced this issue Jul 24, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like.

Fixes #6799.
legoktm added a commit that referenced this issue Jul 25, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
@legoktm legoktm moved this from In Progress to Ready For Review in SecureDrop dev cycle Jul 27, 2023
legoktm added a commit that referenced this issue Aug 1, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 2, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 2, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 3, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 3, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 9, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 9, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
@cfm cfm moved this from Ready For Review to Under Review in SecureDrop dev cycle Aug 14, 2023
legoktm added a commit that referenced this issue Aug 23, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 23, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 23, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 24, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 24, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Aug 25, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Sep 6, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Sep 6, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
legoktm added a commit that referenced this issue Sep 8, 2023
In summary, this change implements:
* Newly created sources have a Sequoia-generated key pair stored in the
database, not in the GPG keyring.
* All message and file encryption functions are handled by Sequoia. The
journalist public key is read off disk instead of the GPG keyring.
* Decryption of journalist messages for legacy GPG sources is still done
through pretty_bad_protocol since the secret key is still in the GPG
keyring.

== EncryptionManager ==

The journalist public key is now read out of the SECUREDROP_DATA_ROOT
folder instead of from the keyring, so we no longer need to know the
specific fingerprint.

Key generation is invoked directly in source_user.create_source(), so
generate_source_key_pair() is removed, though it lives on in tests that
continue to verify GPG-based source behavior.

Since we can easily get a source's public key out of the GPG keyring,
all of the encryption functions now use Sequoia. Functionally this is
inefficient since we need to pull the key out of GPG and then pass it to
Sequoia, but the next phase of the transition will move the public keys
into the database, which will make this more efficient.

== Missing keypairs ==

It's generally wrong to mutate state during GET requests, so move the
step where we'd generate a keypair on /lookup requests to right after a
successful login. This necessitated refactoring the flow of the login
function to exit early in all failure cases so it's more obvious the new
code only runs for a successful request.

And this nicely leaves us a spot to fit in the migration of the secret
key out of GPG (coming soon).

== Tests ==

We're no longer able to mock the key length of generated keys since it's
hardcoded in Rust, so the tests might overall be slower. We could
probably pregenerate a set of keys and cycle through them over the
entire test run, but that's left for a follow-up.

The journalist public key is now copied into each test's unique
data_root. It is no longer present in the GPG keyring. A number of tests
also imported the journalist secret key, decrypted a submission, and
then deleted the key from the keyring. A new
utils.decrypt_as_journalist() helper function uses Sequoia to do all of
that in a much simpler way.

Some EncryptionManager tests were duplicated to use the
create_legacy_gpg_key() helper, which generates a GPG key pair and
deletes the Sequoia-generated keys to mimic what a pre-Sequoia source
creation would be like. Tests that are explicitly testing GPG-based
functionality should have `gpg` in their name, so it's easy to run them
directly with `-k gpg`.

Fixes #6799.
@cfm cfm closed this as completed in #6892 Oct 2, 2023
@github-project-automation github-project-automation bot moved this from Under Review to Done in SecureDrop dev cycle Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Rust Issues that touch Rust code
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants