Fix fog conformance OOM failure by reducing frontends allocation

[samdealy]

Motivation

Previously, we allocated space for 10,000 frontends in AkeEnclaveState. This caused an OOM failure when the fog conformance tests ran. After tinkering around with the numbers, I found that an allocation of 1000 frontends works fine, and then I cut that number in half because we should never need more than 300 frontends given our FVR needs.

[samdealy]

Current dependencies on/for this PR:

• master
  • PR Feature/fog view router #2634
    • PR Make query batch logic compatible with sharding strategy #2845
    • PR Replace ciphertext delta with explicit length field #2803
      • PR Make FVR integration test more complex #2804
        • PR Add unary apis to FogViewRouter #2839
          • PR Separate unary router config #2840
    • PR Stop using the first query response #2783
    • PR Collate remaining QueryResponse fields #2781
    • PR Calculate last known block txo count and add tests #2766
    • PR Set block-related data more accurately #2726
    • PR Create MVQ internal type #2721
    • PR Implement new highest processed block count logic #2722
    • PR Add block range to MVQ response #2720
    • PR Update block tracker to use sharding strategy #2711
    • PR Create fog view router test utils #2698
    • PR Implement debug for Fog View Router client error #2694
    • PR Implement nonce based auth for Fog View Store #2653
      • PR Close gRPC channel when client is dropped #2692
      • PR Fix reencrypt #2672
    • PR Create query_backend enclave methods #2627
    • PR Modify MVQ to use NonceMessage #2617
    • PR Fix fog conformance OOM failure by reducing frontends allocation #2660 👈
    • PR Create Fog View Router integration test #2375
      • PR Make router integration test for gaps in block ranges #2730
      • PR Use slices in fog view enclave impl methods #2712
      • PR Collate cumulative tx out count #2704
        • PR Implement missed blocks for FVR #2705
      • PR Use slices in fog view enclave impl methods #2703

This comment was auto-generated by Graphite.

[eranrund]

Hmm, doesn't this indicate we are very close to not being able to add stuff to the enclave?
Where did this OOM memory show?

[samdealy]

Hmm, doesn't this indicate we are very close to not being able to add stuff to the enclave? Where did this OOM memory show?

Not sure how close we are, but the OOM memory error manifested in the fog conformance tests.

[eranrund]

Hmm, doesn't this indicate we are very close to not being able to add stuff to the enclave? Where did this OOM memory show?

Not sure how close we are, but the OOM memory error manifested in the fog conformance tests.

Do you have a link to a log that shows it? I am curious if this is enclave-specific or if the container just ran out of memory

[samdealy]

Hmm, doesn't this indicate we are very close to not being able to add stuff to the enclave? Where did this OOM memory show?

Not sure how close we are, but the OOM memory error manifested in the fog conformance tests.

Do you have a link to a log that shows it? I am curious if this is enclave-specific or if the container just ran out of memory

Here's a link to the failure.

[nick-mobilecoin]

I'm probably loosing some allocations, but at a minimum it looks like the current values use up:

• backend_auth_pending: 13Mby 10_000 * (24 + 1232 + 64 + 64)
• clients: ~22Mby 10_000 * (32 + 2080 + 64 + 64)
• frontends: ~1Mby 500 * (32 + 2080 + 64 + 64)
• backends: ~22Mby 10_000 * (24 + 2080 + 64 + 64)

If clients are devices/users, I'm guessing that is the only one that needs to be so high.
It seems like the others should be able to be significantly smaller.

    #[test]
    fn sizes() {
        let size = mem::size_of::<AuthPending<X25519, Aes256Gcm, Sha512>>(); // 1232
        assert_eq!(1232, size);
        let size = mem::size_of::<ResponderId>(); //24
        assert_eq!(24, size);
        let size = mem::size_of::<Ready<Aes256Gcm>>(); //2080
        assert_eq!(2080, size);
        let size = mem::size_of::<NonceSession>(); //32
        assert_eq!(32, size);
        let size = mem::size_of::<ClientSession>(); //24
        assert_eq!(24, size);
    }

const MAX_FRONTEND_SESSIONS: usize = 500;
const MAX_BACKEND_SESSIONS: usize = 10_000;
const MAX_CLIENT_SESSIONS: usize = 10_000;
const MAX_BACKEND_AUTH_PENDING_REQUESTS: usize = 10_000;

/// LRU Cache.
pub struct LruCache<K, V> {
    /// Entries currently in cache.
    pub entries: Vec<Option<(Arc<K>, V)>>,

    /// A map of keys -> their index in the `entries` vector, used to speed up
    /// lookups.
    pub key_to_entry_index: HashMap<Arc<K>, usize>,

    /// Indexes of used entries inside the `entries` array. Sorted from newest
    /// to oldest.
    pub used_indexes: VecDeque<usize>,

    /// Indexes of free entries in the `entries` array.
    pub free_indexes: VecDeque<usize>,
}

[samdealy]

I'm probably loosing some allocations, but at a minimum it looks like the current values use up:

• backend_auth_pending: 13Mby 10_000 * (24 + 1232 + 64 + 64)
• clients: ~22Mby 10_000 * (32 + 2080 + 64 + 64)
• frontends: ~1Mby 500 * (32 + 2080 + 64 + 64)
• backends: ~22Mby 10_000 * (24 + 2080 + 64 + 64)

If clients are devices/users, I'm guessing that is the only one that needs to be so high. It seems like the others should be able to be significantly smaller.

    #[test]
    fn sizes() {
        let size = mem::size_of::<AuthPending<X25519, Aes256Gcm, Sha512>>(); // 1232
        assert_eq!(1232, size);
        let size = mem::size_of::<ResponderId>(); //24
        assert_eq!(24, size);
        let size = mem::size_of::<Ready<Aes256Gcm>>(); //2080
        assert_eq!(2080, size);
        let size = mem::size_of::<NonceSession>(); //32
        assert_eq!(32, size);
        let size = mem::size_of::<ClientSession>(); //24
        assert_eq!(24, size);
    }

const MAX_FRONTEND_SESSIONS: usize = 500;
const MAX_BACKEND_SESSIONS: usize = 10_000;
const MAX_CLIENT_SESSIONS: usize = 10_000;
const MAX_BACKEND_AUTH_PENDING_REQUESTS: usize = 10_000;

/// LRU Cache.
pub struct LruCache<K, V> {
    /// Entries currently in cache.
    pub entries: Vec<Option<(Arc<K>, V)>>,

    /// A map of keys -> their index in the `entries` vector, used to speed up
    /// lookups.
    pub key_to_entry_index: HashMap<Arc<K>, usize>,

    /// Indexes of used entries inside the `entries` array. Sorted from newest
    /// to oldest.
    pub used_indexes: VecDeque<usize>,

    /// Indexes of free entries in the `entries` array.
    pub free_indexes: VecDeque<usize>,
}

I think this a good point, but that it should be implemented in a follow up PR. We'll need to fix this enclave memory allocation issue in a more holistic way that is beyond the scope of this PR.

[jcape]

What's driving the tokio/mio update?

[jcape]

Approved pending Cargo.lock fix