-
Notifications
You must be signed in to change notification settings - Fork 158
RFC: Use RunTask API for attached runs
This is a proposal for moving attached runs (emp run without the -d flag) to use the ECS RunTask API, instead of using the Docker Daemon API directly. The benefits of this are that:
- It means containers started via an attached run can use ECS Task Roles.
- It normalizes how attached/detached containers are ran.
- Paves the way for the possibility of adding an
emp attachcommand.
Currently, "attached runs" are NOT implemented in the CloudFormation Scheduler, but rather in the Docker Scheduler.
When you run a command like emp run bash, Empire uses the Docker scheduler, connects to the specified Docker daemon, and uses the CreateContainer, StartContainer, and AttachToContainer API's.
I'm proposing that we instead just use the RunTask API (which detached runs already use), and use the Docker Daemon AttachToContainer API to attach to the container on the given host, after ECS has started it.
When a container is started via ECS, Empire will need to connect to the Docker Daemon on the host that ECS started the container on. Empire will open a new connection for each attach run.
Since ECS doesn't expose any information about the Docker container ID. Empire will use ListContainers to find containers where the com.amazonaws.ecs.task-arn label matches the started task, then attach to this container. I believe that the RunTask API will return before the actual Docker container is running, so Empire will need to wait for the task to enter the RUNNING state before connecting.
In order for us to use AttachToContainer, the container needs to have been started with the Tty and OpenStdin flags. Unfortunately, the ECS API's, and the ECS agent currently do not provide a method to pass these flags down to CreateContainer. In order for this to work, we would need to fork the agent, and add support for specifying these flags, possibly by setting some tags in the container definition:
docker.config.Tty=true
docker.config.OpenStdin=true
At the moment, attached runs are ran by the Docker daemon specified at start up (--docker.socket/DOCKER_HOST). This is also the daemon that Empire uses to pull images, when extracting a Procfile.
A new flag would be added to allow you to specify some container instance constraints for where attached runs would be ran, so that you can limit attached runs to a specific host or set of hosts. By default, the zero value would let the ECS scheduler handle what instance to schedule the container on.
After this change, the DOCKER_HOST flag would only be used to tell Empire what docker daemon to use.
Empire should assume that that Docker Daemon is exposed over TLS when connecting to the container instance.
Race condition between starting the task and attaching to the container
Since RunTask will return before the Docker container is started, Empire will need to poll the tasks status, and connect when it enters the 'RUNNING' state. It's possible that between this state change, and the time that Empire attaches to the container, the container could have stopped.
This is also true of the existing implementation.
No container instance matching constraints
Since the previous implementation would just blindly start a container on the Docker host, resource constraints of the container instance were not taken into consideration. After this change, it's possible that starting an attached run could fail if there is no container instance matching the constraints of the task.
docker pull output won't appear
The docker pull will be done by the ECS agent, so no way to grab this.