Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

All: Add new encryption methods based on native crypto libraries #10696

Merged
merged 78 commits into from
Oct 26, 2024
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
71f0204
Add basic framework for native encryption
wh201906 Jul 7, 2024
17aa0fd
Remove class
wh201906 Jul 8, 2024
a2be9f1
Rename to Crypto
wh201906 Jul 8, 2024
17badd5
Rename to Krypto
wh201906 Jul 11, 2024
80a764b
Merge branch 'dev' into wh201906/native_encryption
wh201906 Jul 19, 2024
4c8c2f3
Rename
wh201906 Jul 19, 2024
8387c96
Remove type any
wh201906 Jul 19, 2024
0276041
Add common interface for the Buffer in Node.js and react-native-buffer
wh201906 Jul 19, 2024
832e589
Clean up
wh201906 Jul 20, 2024
373ecac
Fix the type of digest
wh201906 Jul 21, 2024
d3bd657
Clean up
wh201906 Jul 21, 2024
e4eb58c
Rename
wh201906 Jul 23, 2024
f2ed473
Use enum for Digest
wh201906 Jul 23, 2024
0182cce
Merge branch 'dev' into wh201906/native_encryption
wh201906 Jul 26, 2024
48b54ec
Add a linter ignore rule for crypto terms
wh201906 Jul 28, 2024
70db354
Add encryption core and string wrapper
wh201906 Jul 28, 2024
52f8244
Clean up
wh201906 Aug 3, 2024
7b7595f
Integrate into EncryptionService
wh201906 Aug 3, 2024
26faf26
Add tests
wh201906 Aug 3, 2024
63f6ce9
Enable native encryption
wh201906 Aug 3, 2024
295ef6f
Merge branch 'dev' into wh201906/native_encryption
wh201906 Aug 3, 2024
9d39eb7
Tuning parameters
wh201906 Aug 4, 2024
bfc3ecb
Add performance test for crypto
wh201906 Aug 4, 2024
73b4c5a
Resolve some review suggestions
wh201906 Aug 6, 2024
bf0f126
Use featureFlag for enabling the new encryption method
wh201906 Aug 10, 2024
61dd12d
Set encryption chunk size to method-specific
wh201906 Aug 10, 2024
1536a04
Merge branch 'dev' to resolve conflicts
wh201906 Aug 10, 2024
8b20d12
Merge branch 'dev' into wh201906/native_encryption
laurent22 Aug 10, 2024
04dfc8b
Empty commit for triggering CI
wh201906 Aug 11, 2024
b912dca
Move featureFlag.useBetaEncryptionMethod to sync section
wh201906 Aug 11, 2024
2adf438
Remove translations in featureFlag item
wh201906 Aug 12, 2024
95281e7
Fix tests
wh201906 Aug 13, 2024
d780b09
Remove internal methods from interface definition
wh201906 Aug 20, 2024
cbf2563
Remove null value in parameter "salt"
wh201906 Aug 20, 2024
d7a3b9c
Remove null value in parameter "iv"
wh201906 Aug 20, 2024
885a4b6
Remove null value in parameter "associatedData"
wh201906 Aug 20, 2024
b96bd0d
Move default parameters in crypto.ts to encrypt()/decrypt() implement…
wh201906 Aug 20, 2024
b99f9fc
Modify EncryptionOptions
wh201906 Aug 20, 2024
7026dc2
Move associatedData from encrypt()/decrypt() to EncryptionParameters
wh201906 Aug 20, 2024
bca6e4d
Clean up
wh201906 Aug 20, 2024
d9d8eae
Rename a method parameter
wh201906 Aug 20, 2024
bb69f50
Move featureFlag.useBetaEncryptionMethod to sync.advanced
wh201906 Aug 20, 2024
bc9605e
Better chunkSize
wh201906 Aug 20, 2024
91f9c9b
Add type for encryption handler
wh201906 Aug 20, 2024
b329c0c
Add test cases for invalid ciphertext
wh201906 Aug 20, 2024
d0140da
Migrate to web crypto from Node crypto
personalizedrefrigerator Aug 21, 2024
41e7918
Enable integrated tests on web
personalizedrefrigerator Aug 21, 2024
6087f77
Revert an unintended change
wh201906 Aug 21, 2024
72366e2
Merge branch 'dev' into wh201906/native_encryption
wh201906 Aug 22, 2024
dfefff7
Fix randomBytes for a large output size
personalizedrefrigerator Aug 22, 2024
2fc951d
Merge branch 'personalizedrefrigerator:downstream-pr/use-web-crypto' …
wh201906 Aug 22, 2024
1f9525d
Remove digest name mapping
wh201906 Aug 22, 2024
10c185f
Less calculation and branches in loop
wh201906 Aug 22, 2024
d56dfc0
Remove parameter validation in hot path
wh201906 Aug 22, 2024
ab22f04
Reduce overhead
wh201906 Aug 22, 2024
701c313
Clean up .eslintrc.js
wh201906 Aug 22, 2024
b637071
Enhance nonce generation
wh201906 Aug 23, 2024
58aec1d
Merge branch 'dev' into wh201906/native_encryption
wh201906 Aug 23, 2024
cb9b077
Add digest name mapping for QuickCrypto.createHash()
wh201906 Aug 24, 2024
1a2c3d2
Reduce chunk size to prevent potential freezing
wh201906 Aug 24, 2024
2855ec7
Increase nonce size before hashing
wh201906 Aug 24, 2024
7dc4eec
Replace console with logger in crypto integration tests
wh201906 Aug 28, 2024
9a7b310
Move duplicate code to cryptoShared.ts
wh201906 Aug 28, 2024
2b42bbf
Fix timestamp bytes in nonce
wh201906 Aug 28, 2024
8cfd6dd
Merge branch 'dev' into wh201906/native_encryption
wh201906 Aug 28, 2024
6de1015
Merge branch 'dev' into wh201906/native_encryption
laurent22 Aug 31, 2024
bf0d512
Clean up
wh201906 Sep 6, 2024
8def1be
Merge branch 'dev' to resolve conflicts
wh201906 Sep 11, 2024
115b502
Fix yarn.lock
wh201906 Sep 11, 2024
8fcc8ee
Use it.each to make test cases independent of each other
wh201906 Sep 12, 2024
edd2a90
Add test cases for new encryption methods in Synchronizer.e2ee.test.ts
wh201906 Sep 12, 2024
a786478
Merge branch 'dev' to resolve conflicts
wh201906 Sep 16, 2024
d8f243a
Merge branch 'dev' to resolve conflicts
wh201906 Sep 18, 2024
e78cc10
Merge branch 'dev' to resolve conflicts
wh201906 Sep 20, 2024
5934ccb
Merge branch 'dev' to resolve conflicts
wh201906 Sep 21, 2024
9e22379
Merge branch 'dev' to resolve conflicts
wh201906 Sep 22, 2024
002f7de
Merge branch 'dev' to resolve conflicts
wh201906 Oct 15, 2024
98acc2a
Merge branch 'dev' to resolve conflicts
wh201906 Oct 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
Expand Up @@ -688,6 +688,7 @@ packages/app-mobile/root.js
packages/app-mobile/services/AlarmServiceDriver.android.js
packages/app-mobile/services/AlarmServiceDriver.ios.js
packages/app-mobile/services/e2ee/RSA.react-native.js
packages/app-mobile/services/e2ee/crypto.js
packages/app-mobile/services/plugins/PlatformImplementation.js
packages/app-mobile/services/profiles/index.js
packages/app-mobile/services/voiceTyping/vosk.android.js
Expand Down Expand Up @@ -1028,6 +1029,7 @@ packages/lib/services/debug/populateDatabase.js
packages/lib/services/e2ee/EncryptionService.test.js
packages/lib/services/e2ee/EncryptionService.js
packages/lib/services/e2ee/RSA.node.js
packages/lib/services/e2ee/crypto.js
packages/lib/services/e2ee/ppk.test.js
packages/lib/services/e2ee/ppk.js
packages/lib/services/e2ee/ppkTestUtils.js
Expand Down
2 changes: 2 additions & 0 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ module.exports = {
'tinymce': 'readonly',

'JSX': 'readonly',

'BufferEncoding': 'readonly',
},
'parserOptions': {
'ecmaVersion': 2018,
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -667,6 +667,7 @@ packages/app-mobile/root.js
packages/app-mobile/services/AlarmServiceDriver.android.js
packages/app-mobile/services/AlarmServiceDriver.ios.js
packages/app-mobile/services/e2ee/RSA.react-native.js
packages/app-mobile/services/e2ee/crypto.js
packages/app-mobile/services/plugins/PlatformImplementation.js
packages/app-mobile/services/profiles/index.js
packages/app-mobile/services/voiceTyping/vosk.android.js
Expand Down Expand Up @@ -1007,6 +1008,7 @@ packages/lib/services/debug/populateDatabase.js
packages/lib/services/e2ee/EncryptionService.test.js
packages/lib/services/e2ee/EncryptionService.js
packages/lib/services/e2ee/RSA.node.js
packages/lib/services/e2ee/crypto.js
packages/lib/services/e2ee/ppk.test.js
packages/lib/services/e2ee/ppk.js
packages/lib/services/e2ee/ppkTestUtils.js
Expand Down
1 change: 1 addition & 0 deletions packages/app-mobile/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
"react-native-paper": "5.12.3",
"react-native-popup-menu": "0.16.1",
"react-native-quick-actions": "0.3.13",
"react-native-quick-crypto": "0.7.1",
"react-native-rsa-native": "2.0.5",
"react-native-safe-area-context": "4.10.1",
"react-native-securerandom": "1.0.1",
Expand Down
49 changes: 49 additions & 0 deletions packages/app-mobile/services/e2ee/crypto.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import { Crypto, CryptoBuffer } from '@joplin/lib/services/e2ee/types';
import crypto from 'react-native-quick-crypto';
import { HashAlgorithm } from 'react-native-quick-crypto/lib/typescript/keys';

const cryptoLib: Crypto = {

getCiphers: (): string[] => {
return crypto.getCiphers();
},

getHashes: (): string[] => {
return crypto.getHashes();
},

randomBytes: async (size: number): Promise<CryptoBuffer> => {
return new Promise((resolve, reject) => {
crypto.randomBytes(size, (error, result) => {
if (error) {
reject(error);
} else {
resolve(result);
}
});
});
},

pbkdf2Raw: async (password: string, salt: CryptoBuffer, iterations: number, keylen: number, digest: string): Promise<CryptoBuffer> => {
const digestMap: { [key: string]: HashAlgorithm } = {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again please read the coding style guide, we explicitly ask not to set types this way. See https://joplinapp.org/help/dev/coding_style#avoid-inline-types

I really shouldn't have to keep reminding you to read it. It's understandable to miss a few details here and there but for this PR that's what, the third or fourth time that I remind you to read it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I only checked the method declaration. I will change it later.

'sha1': 'SHA-1',
'sha224': 'SHA-224',
'sha256': 'SHA-256',
'sha384': 'SHA-384',
'sha512': 'SHA-512',
'ripemd160': 'RIPEMD-160',
};
const digestAlgorithm: string = digestMap[digest.toLowerCase()] || digest;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

return new Promise((resolve, reject) => {
crypto.pbkdf2(password, salt, iterations, keylen, digestAlgorithm as HashAlgorithm, (error, result) => {
if (error) {
reject(error);
} else {
resolve(result);
}
});
});
},
};

export default cryptoLib;
laurent22 marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 2 additions & 0 deletions packages/app-mobile/utils/shim-init-react.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,12 @@ import { getLocales } from 'react-native-localize';
import { setLocale, defaultLocale, closestSupportedLocale } from '@joplin/lib/locale';
import type SettingType from '@joplin/lib/models/Setting';
import injectedJs from './injectedJs';
import cryptoLib from '../services/e2ee/crypto';

export default function shimInit() {
shim.Geolocation = GeolocationReact;
shim.sjclModule = require('@joplin/lib/vendor/sjcl-rn.js');
shim.cryptoLib = cryptoLib;

shim.fsDriver = () => {
if (!shim.fsDriver_) {
Expand Down
36 changes: 36 additions & 0 deletions packages/lib/services/e2ee/crypto.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
import { Crypto } from './types';
import { promisify } from 'util';
import crypto = require('crypto');

const cryptoLib: Crypto = {

getCiphers: (): string[] => {
return crypto.getCiphers();
},

getHashes: (): string[] => {
return crypto.getHashes();
},

randomBytes: async (size: number): Promise<Buffer> => {
const randomBytesAsync = promisify(crypto.randomBytes);
return randomBytesAsync(size);
},

pbkdf2Raw: async (password: string, salt: Buffer, iterations: number, keylen: number, digest: string): Promise<Buffer> => {
const digestMap: { [key: string]: string } = {
'sha-1': 'sha1',
'sha-224': 'sha224',
'sha-256': 'sha256',
'sha-384': 'sha384',
'sha-512': 'sha512',
'ripemd-160': 'ripemd160',
};
const digestAlgorithm: string = digestMap[digest.toLowerCase()] || digest;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really get this part - are the digestMap the list of supported digests? In that case shouldn't that be exposed to the library user? In other words, it seems there should be an enum that tells the list of supported digests.

And digest should probably have this type.

Also please don't use toLowerCase() for this - the input digest should be strictly correct and checked at compile time using a type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The digestMap of pbkdf2Raw() is for mapping the name according to the implementations. For Node.js, the hash algorithms are named like sha1, sha256 while for react-native-quick-crypto they are named like SHA-1, SHA-256. I planned to use the string type for the name, just like how Node.js does. But now I will change it to a enum because react-native-quick-crypto only supports a limited set of them.


const pbkdf2Async = promisify(crypto.pbkdf2);
return pbkdf2Async(password, salt, iterations, keylen, digestAlgorithm);
},
};

export default cryptoLib;
11 changes: 11 additions & 0 deletions packages/lib/services/e2ee/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,14 @@ export interface RSA {
publicKey(rsaKeyPair: RSAKeyPair): string;
privateKey(rsaKeyPair: RSAKeyPair): string;
}

export interface Crypto {
getCiphers(): string[];
getHashes(): string[];
randomBytes(size: number): Promise<CryptoBuffer>;
pbkdf2Raw(password: string, salt: CryptoBuffer, iterations: number, keylen: number, digest: string): Promise<CryptoBuffer>;
}

export interface CryptoBuffer extends Uint8Array {
toString(encoding?: BufferEncoding, start?: number, end?: number): string;
}
2 changes: 2 additions & 0 deletions packages/lib/shim-init-node.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import { ResourceEntity } from './services/database/types';
import { TextItem } from 'pdfjs-dist/types/src/display/api';
import replaceUnsupportedCharacters from './utils/replaceUnsupportedCharacters';
import { FetchBlobOptions } from './types';
import cryptoLib from './services/e2ee/crypto';

import FileApiDriverLocal from './file-api-driver-local';
import * as mimeUtils from './mime-utils';
Expand Down Expand Up @@ -135,6 +136,7 @@ function shimInit(options: ShimInitOptions = null) {
shim.Geolocation = GeolocationNode;
shim.FormData = require('form-data');
shim.sjclModule = require('./vendor/sjcl.js');
shim.cryptoLib = cryptoLib;
shim.electronBridge_ = options.electronBridge;

shim.fsDriver = () => {
Expand Down
3 changes: 3 additions & 0 deletions packages/lib/shim.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import * as React from 'react';
import { NoteEntity, ResourceEntity } from './services/database/types';
import type FsDriverBase from './fs-driver-base';
import type FileApiDriverLocal from './file-api-driver-local';
import { Crypto } from './services/e2ee/types';

export interface CreateResourceFromPathOptions {
resizeLargeImages?: 'always' | 'never' | 'ask';
Expand Down Expand Up @@ -286,6 +287,8 @@ const shim = {
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- Old code before rule was applied
sjclModule: null as any,

cryptoLib: null as Crypto,

// eslint-disable-next-line @typescript-eslint/no-explicit-any -- Old code before rule was applied
randomBytes: async (_count: number): Promise<any> => {
throw new Error('Not implemented: randomBytes');
Expand Down
3 changes: 2 additions & 1 deletion packages/tools/cspell/dictionary4.txt
Original file line number Diff line number Diff line change
Expand Up @@ -110,4 +110,5 @@ ENOTFOUND
Scaleway
Inkscape
Ionicon
Stormlikes
Stormlikes
ripemd
51 changes: 50 additions & 1 deletion yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -4367,6 +4367,16 @@ __metadata:
languageName: node
linkType: hard

"@craftzdog/react-native-buffer@npm:^6.0.5":
version: 6.0.5
resolution: "@craftzdog/react-native-buffer@npm:6.0.5"
dependencies:
ieee754: ^1.2.1
react-native-quick-base64: ^2.0.5
checksum: 921b8bc7f84778e355e81e475792399276d611a346a7e51b6266a45cf4aa82194beb3a8106af796ed143d958c8476070c59e3720c0eec0a3c31e368fbb08b350
languageName: node
linkType: hard

"@cronvel/get-pixels@npm:^3.4.0":
version: 3.4.0
resolution: "@cronvel/get-pixels@npm:3.4.0"
Expand Down Expand Up @@ -6902,6 +6912,7 @@ __metadata:
react-native-paper: 5.12.3
react-native-popup-menu: 0.16.1
react-native-quick-actions: 0.3.13
react-native-quick-crypto: 0.7.1
react-native-rsa-native: 2.0.5
react-native-safe-area-context: 4.10.1
react-native-securerandom: 1.0.1
Expand Down Expand Up @@ -36117,6 +36128,31 @@ __metadata:
languageName: node
linkType: hard

"react-native-quick-base64@npm:^2.0.5":
version: 2.1.2
resolution: "react-native-quick-base64@npm:2.1.2"
dependencies:
base64-js: ^1.5.1
peerDependencies:
react: "*"
react-native: "*"
checksum: 46f3b26f48b26978686b0c043336220d681e6a02af5abcf3eb4ab7b9216251d1eb2fac5c559e984d963e93f54bd9f323651daac09762196815558abbd551729b
languageName: node
linkType: hard

"react-native-quick-crypto@npm:0.7.1":
version: 0.7.1
resolution: "react-native-quick-crypto@npm:0.7.1"
dependencies:
"@craftzdog/react-native-buffer": ^6.0.5
events: ^3.3.0
readable-stream: ^4.5.2
string_decoder: ^1.3.0
util: ^0.12.5
checksum: 9a7a1fb1456410db30f062078f570bc566cac36fbc165e7d8ee8677bec09fcc96923de3cf7a0464804142af242a822bf66ea22460951399b9247e4a03fcfe059
languageName: node
linkType: hard

"react-native-rsa-native@npm:2.0.5":
version: 2.0.5
resolution: "react-native-rsa-native@npm:2.0.5"
Expand Down Expand Up @@ -36833,6 +36869,19 @@ __metadata:
languageName: node
linkType: hard

"readable-stream@npm:^4.5.2":
version: 4.5.2
resolution: "readable-stream@npm:4.5.2"
dependencies:
abort-controller: ^3.0.0
buffer: ^6.0.3
events: ^3.3.0
process: ^0.11.10
string_decoder: ^1.3.0
checksum: c4030ccff010b83e4f33289c535f7830190773e274b3fcb6e2541475070bdfd69c98001c3b0cb78763fc00c8b62f514d96c2b10a8bd35d5ce45203a25fa1d33a
languageName: node
linkType: hard

"readable-stream@npm:~2.0.0":
version: 2.0.6
resolution: "readable-stream@npm:2.0.6"
Expand Down Expand Up @@ -43293,7 +43342,7 @@ __metadata:
languageName: node
linkType: hard

"util@npm:^0.12.4":
"util@npm:^0.12.4, util@npm:^0.12.5":
version: 0.12.5
resolution: "util@npm:0.12.5"
dependencies:
Expand Down
Loading