How to sync clocks of provisioned instances

[frejonb]

I'm running into an issue with prometheus where data shows as not available because the EC2 instance clock is behind by 7 minutes to the current time. This can be solver by using NTP but how can I add this during provisioning of the worker nodes? There doesn't seem to be an option in the bootstrap script.

Environment:

• AWS Region: eu-west-1
• Instance Type(s): t2.large
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.2
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version):
• AMI Version: 1.13
• Kernel (e.g. uname -a):
• Release information (run cat /etc/eks/release on a node):

[whereisaaron]

@frejonb eek! chronyd is supposed to be running on nodes to sync time against the tsc or else xen clock sources. Is that not happening on your nodes?

https://github.com/awslabs/amazon-eks-ami/blob/master/install-worker.sh#L67-L87

################################################################################
### Time #######################################################################
################################################################################

# Make sure Amazon Time Sync Service starts on boot.
sudo chkconfig chronyd on

# Make sure that chronyd syncs RTC clock to the kernel.
cat <<EOF | sudo tee -a /etc/chrony.conf
# This directive enables kernel synchronisation (every 11 minutes) of the
# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.
rtcsync
EOF

# If current clocksource is xen, switch to tsc
if grep --quiet xen /sys/devices/system/clocksource/clocksource0/current_clocksource &&
  grep --quiet tsc /sys/devices/system/clocksource/clocksource0/available_clocksource; then
    echo "tsc" | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource
else
    echo "tsc as a clock source is not applicable, skipping."
fi

[cwhatley]

Just came here b/c I was surprised to see that some g3.8xlarge instances I have in a cluster for 1.14 20191213 version of the AMIs had xen not tsc as the source. I don't think that code in the install-worker.sh script is working.

This should probably be reopened:
#249

Perhaps the problem is that the instances that are used to build the images are nitro which has kvm-clock as the default. If I understand the build process, this install-worker.sh script is run at image build time, so I think maybe a boot time check of the clock source is better than a build-time check.

[cartermckinnon]

I've reopened #249 and am closing this as duplicate.