Long sessions playback is sometimes interrupted

[klizhentas]

What happened:

Playback of the long sessions is sometimes incomplete.

What you expected to happen:

Playback recorded session to the end

How to reproduce it (as minimally and precisely as possible):

This issue is for teleport 2.6.0-alpha.0

Setup the teleport with dynamodb backend and s3 upload:

# Single-node Teleport cluster called "one" (runs all 3 roles: proxy, auth and node)
teleport:
  nodename: one
  advertise_ip: 172.10.1.1
  log:
    output: /var/lib/teleport/teleport.log
    severity: INFO

  data_dir: /var/lib/teleport
  storage:
      type: dynamodb
      table_name: test_grv8
      region: us-west-1
      audit_table_name: test_grv8_events
      audit_sessions_uri: s3://testgrv8records

Open the terminal session in the UI.

# run top for a few seconds
top
# produce significant output
ls -lR /
# print something and exit
echo howdy
exit

Then try to playback the session. For the first playback the session will be played back to about 70%
and stop. Second playback will work.

I've checked on backend and it's the same set of files, there is no eventual consistency as in 2.6.0 the session is fully downloaded on replay and is played from the same tarball.

That's why I'm assuming there is UI somehow involved.

Environment:

• Teleport version (use teleport version):

2.6.0-alpha.0

• Tsh version (use tsh version):
• OS (e.g. from /etc/os-release):

Browser environment

• Browser Version (for UI-related issues):
• Install tools:
• Others:

Relevant Debug Logs If Applicable

• tsh --debug
• teleport --debug

[kontsevoy]

@klizhentas how does tsh play behave?

[alex-kovoy]

I looked at the chunks and noticed that the 2nd chunk is empty on the first attempt.

When the player fetches the same chunk on consequent attempt, then it has data in it. It confirms that this is not the issue with the UI/JS. We need to look at the web handlers or further up.

5242880 is the maximum allowed chunk size.

[UPDATE]: note that we query for all chunks at the same time.

[klizhentas]

cool, thanks for your research, this gives me something to work on

[alex-kovoy]

The issue is back. It can repro it in the current master 3.0.0-alpha.3

[russjones]

Let's double check to see if this is still occurring after we've merged in #3549.