rpk container smoke test sporadically fails

[ivotron]

Initializing new client
--
  | client/metadata fetching metadata for all topics from broker 127.0.0.1:42583
  |  
  | Connected to broker at 127.0.0.1:42583 (unregistered)
  |  
  | client/brokers registered new broker #2 at 127.0.0.1:42583
  | client/brokers registered new broker #0 at 127.0.0.1:41913
  | Successfully initialized new client
  | Cluster didn't stabilize: Expected 3 nodes, got 2.
  | Retrying (1 retries left)
  | Error: Expected 3 nodes, got 2.
  | + cleanup
  | + echo 'Cleaning up'
  | + ./rpk container purge -v
  | Cleaning up
  | Stopping node 0
  | Stopping node 1
  | Stopping node 2
  | Removed node '1'
  | Removed container 'rp-node-1'
  | Removed node '0'
  | Removed container 'rp-node-0'
  | Removed node '2'
  | Removed container 'rp-node-2'
  | Deleted cluster data.

ref: https://buildkite.com/vectorized/redpanda/builds/1156#624c9c4e-690f-4ce6-a38f-5e579badf261

[ivotron]

a related error: due to the rpk container smoke test running right after ducktape tests are executed, the compose cluster is left up and running (see this for an example), which in some cases causes the containers started by rpk to fail. A fix for this has been pushed to vtools (link), which stops the compose cluster before the smoke tests begin. It might also be the fix for the OP issue.

[ivotron]

oh, and another commit enabled logging on failures (running docker logs for each rp-node container), which will show what is going on at the docker engine-level, in case this issue persists

[RafalKorepta]

https://buildkite.com/vectorized/redpanda/builds/3736#c848158a-fdc4-40fa-aac2-0c31ef261f92/4193-4675

[twmb]

This is possibly a duplicate of #2418. There's been no new post on this specific issue since Oct; closing for now, please reopen if this is ran into again.