Session.start event overwritten in DynamoDB Events table with Enhanced Logging sessions

[stevenGravy]

Description

If enhanced logging is enabled with dynamodb storage the session.start records are overwritten by session.command or session.network events.

What happened:
A amazon 2 linux vm has enhanced recording on and is set to the required kernel settings. After opening a session and exiting the session listing shows as this. Going to the dynamodb records there are no session.start records, only session.network and session.command for that session id.

What you expected to happen:

All session.start, session.command and session.network are stored and the node, username are available

How to reproduce it (as minimally and precisely as possible):

Deploy a teleport instance that uses dynamodb storage as below. Configure AWS settings so it can access a bucket and have a .aws setting.

  storage:
     type: dynamodb
     region: us-east-1
     table_name: configtable
     audit_events_uri: 'dynamodb://events'
     audit_sessions_uri: 's3://examplebucket/tele1c?region=us-east-1'

Configure a Vm as a teleport node for the instance that has the correct kernel settings (as for amazon 2 linux https://gravitational.com/teleport/docs/features/enhanced_session_recording/#amazon-linux). put in the enhanced logging options as below.

  enhanced_recording:
    # Enable or disable enhanced auditing for this node. Default value: false.
    enabled: true

    # Optional: command_buffer_size is optional with a default value of 8 pages. 
    command_buffer_size: 8

    # Optional: disk_buffer_size is optional with default value of 128 pages.
    disk_buffer_size: 128

    # Optional: network_buffer_size is optional with default value of 8 pages.
    network_buffer_size: 8

    # Optional: Controls where cgroupv2 hierarchy is mounted. Default value: 
    # /cgroup2.
    cgroup_path: /cgroup2

Open a session, run commands and exit. Confirm the session does not now the node or user in the session listing.

Environment

• Teleport version (use teleport version): 4.2.10 ent

• Tsh version (use tsh version): 4.2.10

• OS (e.g. from /etc/os-release): Ubuntu bionic

• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS EC2

[russjones]

Starting from a fresh install of DynamoDB, this is what I saw when I created one session with enhanced session recording enabled, typed ls and then exit.

[russjones]

This is opposed to logging to disk where I see the following events.

"user.update"
"session.start"
"resize"
"session.command"
"session.command"
"session.command"
"session.command"
"session.command"
"session.command"
"session.command"
"session.leave"
"session.end"
"session.data"

[fspmarshall]

Looks like this issue arises caused because the current event system uses a combination of EventIndex and SessionID as the primary key in dynamo. Since enhanced events are indipendently indexed, they end up having the same primary key as their standard counterparts.

It appears that the session streaming work being done by @klizhentas (slated for 4.4) will resolve this problem indirectly since it includes a rework of how events are stored.