Make TSC the default system clock source

[cwhatley]

What would you like to be added:

Make the default system clock source "tsc" instead of "xen".

Why is this needed:

Docker containers that do frequent system time lookups can be badly affected by the overhead around the xen system clock source.

AWS's own tuning recommendations suggest this switch.

Is there any reason not to switch to tsc as the default for the EKS images?

[JasonSwindle]

I have a soon pending PR for this. I need to test it first.

JasonSwindle@9d57d76

[JasonSwindle]

This issue can be closed.

[micahhausler]

Closed in #272

[cwhatley]

This doesn't appear to be working anymore. See comments in #321

[redbelow]

seeing the same behavior in v1.15.11-eks-065dce

[cartermckinnon]

Has anyone continued to experience this issue on currently-supported AMI's?

[bryantbiggs]

I don't know if its related, but I usually do not have to open NTP/123 on my NACLs/security groups but with the EKS AMI I am seeing a lot of REJECTs in flow logs

Per this https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

The Amazon Time Sync Service is available through NTP at the 169.254.169.123 IPv4 address or the fd00:ec2::123 IPv6 address for any instance running in a VPC. The IPv6 address is only accessible on Instances built on the Nitro System. Your instance does not require access to the internet, and you do not have to configure your security group rules or your network ACL rules to allow access. The latest versions of Amazon Linux 2 and Amazon Linux AMIs synchronize with the Amazon Time Sync Service by default.

Is there any way to use this service at 169.254.169.123 instead of having to open up NTP/123 to a public route?

[cartermckinnon]

Looking into this. I think we should remove our chrony config, because the base AL2 AMI is already doing what's necessary for TSS.

As for #272 -- that PR was a no-op. It's setting the current clocksource during the AMI build process, and the default is reset upon reboot. I think we're fine to set clocksource=tsc on the kernel command line, but I need to verify this doesn't cause issues if/when that clocksource isn't available. I'm also not sure if tsc is considered stable on newer Intels versus kvm-clock (the recommended clocksource on Nitro-based instance types).

[cartermckinnon]

@ybstaked are you using Xen or Nitro-based instance types? FWICT, tsc is what's being used on Nitro-based instance types with our AMI, and the recommendation is to use kvm-clock; whether this could be responsible for the drift is an open question.

[ybstaked]

We are known to have issues on specifically at leastc6i.4xlarge and m6i2xlarge. These are:

• The validator keeps submitting outdated message to beacon node. "Sync Committee Message is too old to broadcast, discarding it" prysmaticlabs/prysm#11415
• https://www.coincashew.com/coins/overview-eth/guide-or-how-to-setup-a-validator-on-eth2-mainnet/part-i-installation/synchronizing-time-with-chrony
• https://www.reddit.com/r/ethstaker/comments/ilr1ma/keeping_systemtime_accurate/

[cartermckinnon]

Interesting; those are Nitro instance types. If you want to give kvm-clock a try, it'd be a good data point.

I'll go ahead and update #1328 to use kvm-clock on Nitro, it's the recommendation from EC2 anyway 👍