Bug: failed to create k8s image tar file

[MaxMcAdam]

Describe the bug.

NMP-based cluster upgrade failed precheck with the following error

cluster_upgrade_worker.go:716] Cluster upgrade worker: Making tarball /var/horizon/nmp/<org_name>/<nmp_name>/amd64_anax_k8s.tar
cluster_upgrade_worker.go:718] Cluster upgrade worker: Failed to create image tar at /var/horizon/nmp/<org_name>/<nmp_name>/amd64_anax_k8s.tar, error: open amd64_anax_k8s/sha256: no such file or directory
cluster_upgrade_worker.go:469] Cluster upgrade worker: Failed to compare agent image version for nmp: <org_name>/<nmp_name>, error: open amd64_anax_k8s/sha256: no such file or directory
cluster_upgrade_worker.go:259] Cluster upgrade worker: Set status to precheck failed in db and status file for nmp <org_name>/<nmp_name>

Describe the steps to reproduce the behavior.

No response

Expected behavior.

No response

Screenshots.

No response

Operating Environment

k8s

Additional Information

No response

[dlarson04]

Adding root cause here...

The edgeNodeFiles relies on some docker functions such as docker pull and docker save... Here is what I see for the same image that gets pushed into CSS

With docker 20

# tar -tvf amd64_anax_k8s.tar
drwxr-xr-x 0/0               0 2024-06-12 12:25 08c26a9bd7e21ccdd71f50096823630380a3e5e9d5019048eedece0e35710d0d/
-rw-r--r-- 0/0               3 2024-06-12 12:25 08c26a9bd7e21ccdd71f50096823630380a3e5e9d5019048eedece0e35710d0d/VERSION
-rw-r--r-- 0/0             482 2024-06-12 12:25 08c26a9bd7e21ccdd71f50096823630380a3e5e9d5019048eedece0e35710d0d/json
-rw-r--r-- 0/0        90322944 2024-06-12 12:25 08c26a9bd7e21ccdd71f50096823630380a3e5e9d5019048eedece0e35710d0d/layer.tar

but with Docker 26

# tar -tvf amd64_anax_k8s.tar
drwxr-xr-x 0/0               0 2024-06-12 12:25 blobs/
drwxr-xr-x 0/0               0 2024-06-19 14:52 blobs/sha256/
-rw-r--r-- 0/0            4608 2024-06-12 12:25 blobs/sha256/09053b937453a5a5783b9ac634dac58e2b105d57d57fb2fd4bfd188fa1faf97d
-rw-r--r-- 0/0        90322944 2024-06-12 12:25 blobs/sha256/3429faf624656835ceb7d1ad6218d30dee13f826600fc774ecdd4c8379509c09
-rw-r--r-- 0/0             482 2024-06-12 12:25 blobs/sha256/37bf02a9431c5eac3ad161ed9e0426371d79c80315e75f6bb61f9ca61b705d27

The error on the NMP status that was shown was

"status": "precheck failed",
        "errorMessage": "Failed to compare agent image version for nmp: ISV_qatoolbox-pablo_andres_fuero/ga_nmp, error: open amd64_anax_k8s/sha256: no such file or directory"
      }