Fail fast if Rsync fails due to "No space left on device", no need to do 10+ retries.

[djwhatle]

Is your feature request related to a problem? Please describe.

string"8 [1] recv_files(zbtl) recv_files(zbtl) 2021/04/14 19:06:58 [1] recv_files(zbvl) recv_files(zbvl) 2021/04/14 19:06:58 [1] recv_files(zceknl) recv_files(zceknl) 2021/04/14 19:06:58 [1] recv_files(zdafatai) recv_files(zdafatai) 2021/04/14 19:06:58 [1] recv_files(zedtnhu) recv_files(zedtnhu) 2021/04/14 19:06:58 [1] recv_files(zeebbgucg) recv_files(zeebbgucg) 2021/04/14 19:06:58 [1] recv_files(zeuibroxz) recv_files(zeuibroxz) 2021/04/14 19:06:58 [1] recv_files(ziazqgq) recv_files(ziazqgq) 2021/04/14 19:06:58 [1] recv_files(zilhouzey) recv_files(zilhouzey) 2021/04/14 19:06:58 [1] recv_files(zjbgp) recv_files(zjbgp) 2021/04/14 19:06:58 [1] recv_files(zjgz) recv_files(zjgz) 2021/04/14 19:06:58 [1] recv_files(zjorvbmdy) recv_files(zjorvbmdy) 2021/04/14 19:06:58 [1] recv_files(zmcaya) recv_files(zmcaya) rsync: write failed on "zqdabwv" (in pvc-02): No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] 92.79M 1% 109.11MB/s 0:00:00 (xfr#11, to-chk=0/524)2021/04/14 19:06:58 [1] <f+++++++++ zzlxhuprx 2021/04/14 19:06:58 [1] recv_files(zobaylvmk) recv_files(zobaylvmk) 2021/04/14 19:06:58 [1] recv_files(zqdabwv) recv_files(zqdabwv) 2021/04/14 19:06:58 [1] recv mapped zqdabwv of size 14159872 recv mapped zqdabwv of size 14159872 2021/04/14 19:06:58 [1] rsync: write failed on "zqdabwv" (in pvc-02): No space left on device (28) 2021/04/14 19:06:58 [1] rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] 2021/04/14 19:06:58 [1] [receiver] _exit_cleanup(code=11, file=receiver.c, line=374): about to call exit(11) [receiver] _exit_cleanup(code=11, file=receiver.c, line=374): about to call exit(11) 2021/04/14 19:06:58 [1] rsync error: error in file IO (code 11) at io.c(1642) [generator=3.1.3] 2021/04/14 19:06:58 [1] [generator] _exit_cleanup(code=11, file=io.c, line=1642): about to call exit(11) [generator] _exit_cleanup(code=11, file=io.c, line=1642): about to call exit(11) rsync error: error in file IO (code 11) at io.c(1642) [generator=3.1.3]

[djwhatle]

Had a brief conversation with @pranavgaikwad on this. It sounds like there would be some complexity involved to pull this off since DVMP status lags behind pod logs and DVM orchestrates the restart before it has final logs available from DVMP.

So DVM would need to delay the retry in order to be able to analyze this information.