Issue with Deployment Tinkerer hanging after executing a remote command

[ranikamadurawe]

Description:
On occasion when communicating with multiple remote agents the deployment tinkerer API successfully receives and executes the command but does not update the success of the command in the Test Phase. So the Test Phase waits till the Timeout to continue

Suggested Labels:
Bug

Steps to reproduce:
Running a sample Test Grid job with multiple remote agents. The issue usually occurs with 3 or 4 remote agents

Logfiles

[09:43:06,708] Report generation skipped..

[09:43:06,708] Process to upload deployment-outputs to S3 has started for test-plan rnm-k8s_deployment_CentOS-7.5_MySQL-5.7_run22

[09:43:07,264] Found 1 agents for the test-plan.

[09:43:07,961] Tinkerer commands are sending out to nodes, Wait till the execution is complete. (TIME-OUT: 10 Minutes)

[09:43:07,971] Sending sync commands to http://34.68.130.7:80/tinkerer/api/ agent K8S:US:rnm-k8s_deployment_CentOS-7.5_MySQL-5.7_run22:wso2apim-with-analytics-apim-6f89d77db-7x7rs:10.8.0.155

[09:53:07,964] Tinkerer commands execution time-out. Gracefully moving to next steps..