Skip to content
This repository has been archived by the owner on May 28, 2021. It is now read-only.
This repository has been archived by the owner on May 28, 2021. It is now read-only.

mysql-agent can't recreate MySQL Cluster after full k8s cluster restart #83

Closed
gites opened this issue Apr 18, 2018 · 6 comments
Closed

mysql-agent can't recreate MySQL Cluster after full k8s cluster restart #83

gites opened this issue Apr 18, 2018 · 6 comments
Labels
bug

Comments

@gites
Copy link

gites commented Apr 18, 2018

BUG REPORT

Versions

MySQL Operator Version: 0.1.0

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration: minikube

  • OS (e.g. from /etc/os-release): Buildroot 2017.11

  • Kernel (e.g. uname -a): Linux minikube 4.9.64 #1 SMP Fri Mar 30 21:27:22 UTC 2018 x86_64 GNU/Linux

  • Others:

What happened?

Mysql-agent can't recreate MySQL Cluster after full k8s restart (the VM runing one node k8s was restarted)

What you expected to happen?

MySQL Cluster should have been recreated by mysql-agent

How to reproduce it (as minimally and precisely as possible)?

  1. Create one node k8s cluster
  2. Deploy MySQL Cluster to it
  3. Restart k8s VM
  4. Check cluster state

Anything else we need to know?

kubectl get pod -n ops                                 
NAME                              READY     STATUS    RESTARTS   AGE
mysql-cluster1-0                  1/2       Running   2          22m
mysql-cluster1-1                  1/2       Running   2          22m
mysql-cluster1-2                  1/2       Running   2          21m
mysql-operator-854d4684dc-t462l   1/1       Running   3          23m

kubectl -n ops log mysql-cluster1-0 mysql-agent
Starting mysql-agent version 0.1.0                  
ERROR: logging before flag.Parse: I0418 13:23:29.251728       1 flags.go:52] FLAG: --address="0.0.0.0"   
ERROR: logging before flag.Parse: I0418 13:23:29.253051       1 flags.go:52] FLAG: --alsologtostderr="false"                                                                                                       
ERROR: logging before flag.Parse: I0418 13:23:29.253064       1 flags.go:52] FLAG: --healthcheck-port="10512"                                                                                                      
ERROR: logging before flag.Parse: I0418 13:23:29.253068       1 flags.go:52] FLAG: --hostname="mysql-cluster1-0"                                                                                                   
ERROR: logging before flag.Parse: I0418 13:23:29.253072       1 flags.go:52] FLAG: --log-backtrace-at=":0"                                                                                                         
ERROR: logging before flag.Parse: I0418 13:23:29.253077       1 flags.go:52] FLAG: --log-dir=""          
ERROR: logging before flag.Parse: I0418 13:23:29.253080       1 flags.go:52] FLAG: --log-flush-frequency="5s"                                                                                                      
ERROR: logging before flag.Parse: I0418 13:23:29.253084       1 flags.go:52] FLAG: --logtostderr="true"  
ERROR: logging before flag.Parse: I0418 13:23:29.253087       1 flags.go:52] FLAG: --min-resync-period="12h0m0s"                                                                                                   
ERROR: logging before flag.Parse: I0418 13:23:29.253090       1 flags.go:52] FLAG: --mysql-cluster-name="mysql-cluster1"                                                                                           
ERROR: logging before flag.Parse: I0418 13:23:29.253093       1 flags.go:52] FLAG: --namespace="ops"     
ERROR: logging before flag.Parse: I0418 13:23:29.253203       1 flags.go:52] FLAG: --stderrthreshold="2" 
ERROR: logging before flag.Parse: I0418 13:23:29.253207       1 flags.go:52] FLAG: --v="4"               
ERROR: logging before flag.Parse: I0418 13:23:29.253209       1 flags.go:52] FLAG: --vmodule=""          
ERROR: logging before flag.Parse: I0418 13:23:29.480862       1 cluster_manager.go:105] Database not yet running. Waiting...                                                                                       
ERROR: logging before flag.Parse: I0418 13:24:39.530507       1 innodb_cluster.go:121] Clearing the MySQL binary logs                                                                                              
ERROR: logging before flag.Parse: I0418 13:24:39.641214       1 cluster_manager.go:280] Creating the cluster on the primary instance                                                                               
ERROR: logging before flag.Parse: E0418 13:24:39.718935       1 cluster_manager.go:121] Error bootstrapping cluster: creating cluster: mysqlsh --uri root:fwUSRT7OBXzV7N3I@mysql-cluster1-0:3306 --py -e dba.create_cluster('MySQLCluster'): err=exit status 1         
stdout:                                             
mysqlx: [Warning] Using a password on the command line interface can be insecure.                        

stderr:                                             
Traceback (most recent call last):                  
  File "<string>", line 1, in <module>              
SystemError: ArgumentError: Dba.create_cluster: A Cluster with the name 'MySQLCluster' already exists.   

: exit status 1                                     
failed to create new cluster                        
github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).bootstrap             
        /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:283  
github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Sync                  
        /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:119  
github.com/oracle/mysql-operator/cmd/mysql-agent/app.Run                                                 
        /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/app/mysql_agent.go:108                  
main.main                                           
        /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/main.go:40                              
runtime.main                                        
        /usr/local/go/src/runtime/proc.go:195       
runtime.goexit                                      
        /usr/local/go/src/runtime/asm_amd64.s:2337  

kubectl -n ops log mysql-cluster1-1 mysql-agent
Starting mysql-agent version 0.1.0
ERROR: logging before flag.Parse: I0418 13:23:28.683839       1 flags.go:52] FLAG: --address="0.0.0.0"
ERROR: logging before flag.Parse: I0418 13:23:28.683863       1 flags.go:52] FLAG: --alsologtostderr="false"
ERROR: logging before flag.Parse: I0418 13:23:28.683867       1 flags.go:52] FLAG: --healthcheck-port="10512"
ERROR: logging before flag.Parse: I0418 13:23:28.683871       1 flags.go:52] FLAG: --hostname="mysql-cluster1-1"
ERROR: logging before flag.Parse: I0418 13:23:28.683873       1 flags.go:52] FLAG: --log-backtrace-at=":0"
ERROR: logging before flag.Parse: I0418 13:23:28.683877       1 flags.go:52] FLAG: --log-dir=""
ERROR: logging before flag.Parse: I0418 13:23:28.683880       1 flags.go:52] FLAG: --log-flush-frequency="5s"
ERROR: logging before flag.Parse: I0418 13:23:28.683884       1 flags.go:52] FLAG: --logtostderr="true"
ERROR: logging before flag.Parse: I0418 13:23:28.683887       1 flags.go:52] FLAG: --min-resync-period="12h0m0s"
ERROR: logging before flag.Parse: I0418 13:23:28.683890       1 flags.go:52] FLAG: --mysql-cluster-name="mysql-cluster1"
ERROR: logging before flag.Parse: I0418 13:23:28.683892       1 flags.go:52] FLAG: --namespace="ops"
ERROR: logging before flag.Parse: I0418 13:23:28.683895       1 flags.go:52] FLAG: --stderrthreshold="2"
ERROR: logging before flag.Parse: I0418 13:23:28.683897       1 flags.go:52] FLAG: --v="4"
ERROR: logging before flag.Parse: I0418 13:23:28.683900       1 flags.go:52] FLAG: --vmodule=""
ERROR: logging before flag.Parse: I0418 13:24:29.479744       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0418 13:24:50.750149       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0418 13:25:00.885368       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0418 13:25:11.117466       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0418 13:25:21.515948       1 cluster_manager.go:127] Cluster not yet present. Waiting...

kubectl  -n ops  logs mysql-operator-854d4684dc-t462l 
Starting mysql-operator version '0.1.0'
ERROR: logging before flag.Parse: I0418 13:25:06.941234       1 options.go:90] No '/etc/mysql-operator/mysql-operator-config.yaml' was present.
ERROR: logging before flag.Parse: I0418 13:25:06.943858       1 flags.go:52] FLAG: --alsologtostderr="false"
ERROR: logging before flag.Parse: I0418 13:25:06.943882       1 flags.go:52] FLAG: --kubeconfig=""
ERROR: logging before flag.Parse: I0418 13:25:06.943887       1 flags.go:52] FLAG: --log-backtrace-at=":0"
ERROR: logging before flag.Parse: I0418 13:25:06.943893       1 flags.go:52] FLAG: --log-dir=""
ERROR: logging before flag.Parse: I0418 13:25:06.943897       1 flags.go:52] FLAG: --log-flush-frequency="5s"
ERROR: logging before flag.Parse: I0418 13:25:06.943901       1 flags.go:52] FLAG: --logtostderr="true"
ERROR: logging before flag.Parse: I0418 13:25:06.943904       1 flags.go:52] FLAG: --master=""
ERROR: logging before flag.Parse: I0418 13:25:06.943907       1 flags.go:52] FLAG: --min-resync-period="12h0m0s"
ERROR: logging before flag.Parse: I0418 13:25:06.943912       1 flags.go:52] FLAG: --mysql-agent-image="wcr.io/oracle/mysql-agent"
ERROR: logging before flag.Parse: I0418 13:25:06.943915       1 flags.go:52] FLAG: --mysql-server-image="mysql/mysql-server"
ERROR: logging before flag.Parse: I0418 13:25:06.944003       1 flags.go:52] FLAG: --namespace="ops"
ERROR: logging before flag.Parse: I0418 13:25:06.944008       1 flags.go:52] FLAG: --stderrthreshold="2"
ERROR: logging before flag.Parse: I0418 13:25:06.944011       1 flags.go:52] FLAG: --v="4"
ERROR: logging before flag.Parse: I0418 13:25:06.944014       1 flags.go:52] FLAG: --vmodule=""
ERROR: logging before flag.Parse: W0418 13:25:06.944020       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ERROR: logging before flag.Parse: I0418 13:25:07.007388       1 controller.go:172] Creating event broadcaster
ERROR: logging before flag.Parse: I0418 13:25:07.007552       1 operator_controller.go:91] Creating event broadcaster
ERROR: logging before flag.Parse: I0418 13:25:07.007673       1 operator_controller.go:99] Creating event broadcaster
ERROR: logging before flag.Parse: I0418 13:25:07.007802       1 controller.go:77] Creating event broadcaster
ERROR: logging before flag.Parse: I0418 13:25:07.008288       1 reflector.go:202] Starting reflector *v1.MySQLBackupSchedule (19h15m21.324440719s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.008382       1 reflector.go:240] Listing and watching *v1.MySQLBackupSchedule from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.008766       1 operator_controller.go:169] Starting OperatorController
ERROR: logging before flag.Parse: I0418 13:25:07.008792       1 operator_controller.go:172] Waiting for caches to sync
ERROR: logging before flag.Parse: I0418 13:25:07.008799       1 cache.go:30] Waiting for caches to sync for operator-restore-controller controller
ERROR: logging before flag.Parse: I0418 13:25:07.009099       1 controller.go:263] Starting MySQLCluster controller
ERROR: logging before flag.Parse: I0418 13:25:07.009111       1 controller.go:266] Waiting for MySQLCluster controller informer caches to sync
ERROR: logging before flag.Parse: I0418 13:25:07.009116       1 cache.go:30] Waiting for caches to sync for mysql cluster controller
ERROR: logging before flag.Parse: I0418 13:25:07.009400       1 operator_controller.go:159] Starting OperatorController
ERROR: logging before flag.Parse: I0418 13:25:07.009434       1 operator_controller.go:162] Waiting for caches to sync
ERROR: logging before flag.Parse: I0418 13:25:07.009440       1 cache.go:30] Waiting for caches to sync for operator-backup-controller controller
ERROR: logging before flag.Parse: I0418 13:25:07.009791       1 reflector.go:202] Starting reflector *v1.MySQLCluster (19h15m21.324440719s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.009803       1 reflector.go:240] Listing and watching *v1.MySQLCluster from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.009916       1 controller.go:140] Starting backup schedule controller
ERROR: logging before flag.Parse: I0418 13:25:07.010008       1 controller.go:143] Waiting for backup schedule controller caches to sync
ERROR: logging before flag.Parse: I0418 13:25:07.010153       1 reflector.go:202] Starting reflector *v1.Pod (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.010236       1 reflector.go:240] Listing and watching *v1.Pod from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.017895       1 reflector.go:202] Starting reflector *v1.ConfigMap (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.017933       1 reflector.go:240] Listing and watching *v1.ConfigMap from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018157       1 reflector.go:202] Starting reflector *v1.Service (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018197       1 reflector.go:240] Listing and watching *v1.Service from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018403       1 reflector.go:202] Starting reflector *v1beta1.StatefulSet (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018197       1 reflector.go:240] Listing and watching *v1.Service from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018403       1 reflector.go:202] Starting reflector *v1beta1.StatefulSet (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018422       1 reflector.go:240] Listing and watching *v1beta1.StatefulSet from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018572       1 reflector.go:202] Starting reflector *v1.MySQLBackup (19h15m21.324440719s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.019838       1 reflector.go:240] Listing and watching *v1.MySQLBackup from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.019767       1 reflector.go:202] Starting reflector *v1.MySQLRestore (19h15m21.324440719s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.020075       1 reflector.go:240] Listing and watching *v1.MySQLRestore from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.156798       1 shared_informer.go:122] caches populated
ERROR: logging before flag.Parse: I0418 13:25:07.156937       1 controller.go:147] Backup schedule controller caches are synced
ERROR: logging before flag.Parse: I0418 13:25:07.203952       1 controller.go:636] Processing object: mysql-cluster1
ERROR: logging before flag.Parse: I0418 13:25:07.210427       1 shared_informer.go:122] caches populated
ERROR: logging before flag.Parse: I0418 13:25:07.210466       1 cache.go:37] Caches are synced for operator-restore-controller controller
ERROR: logging before flag.Parse: I0418 13:25:07.210475       1 operator_controller.go:180] Caches are synced
ERROR: logging before flag.Parse: I0418 13:25:07.210546       1 shared_informer.go:122] caches populated
ERROR: logging before flag.Parse: I0418 13:25:07.210551       1 cache.go:37] Caches are synced for mysql cluster controller
ERROR: logging before flag.Parse: I0418 13:25:07.210556       1 controller.go:276] Starting MySQLCluster controller workers
ERROR: logging before flag.Parse: I0418 13:25:07.210563       1 controller.go:282] Started MySQLCluster controller workers
ERROR: logging before flag.Parse: I0418 13:25:07.210718       1 shared_informer.go:122] caches populated
ERROR: logging before flag.Parse: I0418 13:25:07.210818       1 cache.go:37] Caches are synced for operator-backup-controller controller
ERROR: logging before flag.Parse: I0418 13:25:07.210824       1 operator_controller.go:169] Caches are synced
ERROR: logging before flag.Parse: I0418 13:25:07.220728       1 controller.go:505] ops/mysql-cluster1: ss.Spec.Replicas=3, ss.Status.ReadyReplicas=0, ss.Status.Replicas=3
ERROR: logging before flag.Parse: I0418 13:25:07.244966       1 controller.go:315] Successfully synced 'ops/mysql-cluster1'
ERROR: logging before flag.Parse: I0418 13:25:07.246377       1 event.go:218] Event(v1.ObjectReference{Kind:"MySQLCluster", Namespace:"ops", Name:"mysql-cluster1", UID:"87e256bb-430a-11e8-9df3-080027ab5287", APIVersion:"mysql.oracle.com/v1", ResourceVersion:"775", FieldPath:""}): type: 'Normal' reason: 'Synced' MySQLCluster synced successfully
ERROR: logging before flag.Parse: I0418 13:25:07.253329       1 controller.go:505] ops/mysql-cluster1: ss.Spec.Replicas=3, ss.Status.ReadyReplicas=0, ss.Status.Replicas=3
ERROR: logging before flag.Parse: I0418 13:25:07.288727       1 controller.go:315] Successfully synced 'ops/mysql-cluster1'
ERROR: logging before flag.Parse: I0418 13:25:07.289259       1 event.go:218] Event(v1.ObjectReference{Kind:"MySQLCluster", Namespace:"ops", Name:"mysql-cluster1", UID:"87e256bb-430a-11e8-9df3-080027ab5287", APIVersion:"mysql.oracle.com/v1", ResourceVersion:"775", FieldPath:""}): type: 'Normal' reason: 'Synced' MySQLCluster synced successfully
ERROR: logging before flag.Parse: I0418 13:25:07.291028       1 controller.go:505] ops/mysql-cluster1: ss.Spec.Replicas=3, ss.Status.ReadyReplicas=0, ss.Status.Replicas=3
ERROR: logging before flag.Parse: I0418 13:25:07.291079       1 controller.go:315] Successfully synced 'ops/mysql-cluster1'
ERROR: logging before flag.Parse: I0418 13:25:07.291096       1 event.go:218] Event(v1.ObjectReference{Kind:"MySQLCluster", Namespace:"ops", Name:"mysql-cluster1", UID:"87e256bb-430a-11e8-9df3-080027ab5287", APIVersion:"mysql.oracle.com", ResourceVersion:"1467", FieldPath:""}): type: 'Normal' reason: 'Synced' MySQLCluster synced successfully
ERROR: logging before flag.Parse: I0418 13:30:26.174949       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1.Service total 0 items received
ERROR: logging before flag.Parse: I0418 13:30:36.206328       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1beta1.StatefulSet total 0 items received
ERROR: logging before flag.Parse: I0418 13:30:53.172602       1 reflector.go:428] github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72: Watch close - *v1.MySQLCluster total 1 items received
ERROR: logging before flag.Parse: I0418 13:31:37.206167       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1.Pod total 1 items received
ERROR: logging before flag.Parse: I0418 13:32:14.058020       1 reflector.go:428] github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72: Watch close - *v1.MySQLBackup total 0 items received
ERROR: logging before flag.Parse: I0418 13:32:18.056845       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1.ConfigMap total 0 items received
ERROR: logging before flag.Parse: I0418 13:33:26.057179       1 reflector.go:428] github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72: Watch close - *v1.MySQLRestore total 0 items received
ERROR: logging before flag.Parse: I0418 13:33:33.079811       1 reflector.go:428] github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72: Watch close - *v1.MySQLBackupSchedule total 0 items received
ERROR: logging before flag.Parse: I0418 13:36:40.210043       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1beta1.StatefulSet total 0 items received
@gites gites changed the title Mysql-agent can't recreat MySQL Cluster after full k8s clusterr estart Mysql-agent can't recreat MySQL Cluster after full k8s cluster restart Apr 18, 2018
@owainlewis owainlewis added the bug label Apr 18, 2018
@prydie prydie mentioned this issue Apr 20, 2018
Merged
@paolomainardi
Copy link

We are facing the same issue, @gites did you find something useful ?

@prydie
Copy link

prydie commented Apr 26, 2018

Hi @paolomainardi,

I’m ironing out the last few issues with the upgrade of the operator to MySQL 8 which will fix the issue (see #87 for details). In the meantime you may be able to rectify the situation on your existing cluster by kubectl exec’ing into the mysql-agentcontainer of the/a primary and using mysqlsh.

@paolomainardi
Copy link

ok thanks a lot @prydie

@gites
Copy link
Author

gites commented May 8, 2018

@prydie I did a test build from commit a0d35f4 and cluster is still failing after restrat, but now due to bad SSL settings.
The first pod is able to initialize cluster, but on the second one I see errors like this:

I0508 08:44:58.103437       1 cluster_manager.go:258] Attempting to rejoin instance to cluster           
I0508 08:44:58.103525       1 mysqlsh.go:160] Running command: mysqlsh [--no-wizard --uri root:root@mysql-cluster1-0.mysql-cluster1:3306 --py -e dba.get_cluster('MySQLCluster').rejoin_instance('root:root@mysql-cluster1-1.mysql-cluster1:3306', {'memberSslMode': 'DISABLED', 'ipWhitelist': '172.16.0.0/12'})]
I0508 08:44:58.246988       1 mysqlsh.go:162]     stdout:                                                
    stderr: mysqlsh: [Warning] Using a password on the command line interface can be insecure.           
Traceback (most recent call last):                  
  File "<string>", line 1, in <module>              
SystemError: RuntimeError: Cluster.rejoin_instance: The cluster has SSL (encryption) enabled. To add the instance 'mysql://root@mysql-cluster1-1.mysql-cluster1:3306' to the cluster either disable SSL on the cluster, remove the memberSslMode option or use it with any of 'AUTO' or 'REQUIRED'.


    err: exit status 1                              
E0508 08:44:58.247107       1 cluster_manager.go:263] Failed to rejoin cluster: SystemError: RuntimeError: Cluster.rejoin_instance: The cluster has SSL (encryption) enabled. To add the instance 'mysql://root@mysql-cluster1-1.mysql-cluster1:3306' to the cluster either disable SSL on the cluster, remove the memberSslMode option or use it with any of 'AUTO' or 'REQUIRED'.

Attaching logs from pods below:

agent0.log
agent1.log
agent2.log
mysql0.log
mysql1.log
mysql2.log
operator.log

@prydie
Copy link

prydie commented May 8, 2018

Hey @gites, yeah, we decided to merge the MySQL 8 change with that still outstanding as complete disaster recovery was already broken. I'm taking a look at it today. I'll update here once I've got something working 🙂.

@garthy
Copy link
Member

garthy commented May 8, 2018

FYI: https://bugs.mysql.com/bug.php?id=90793

@prydie prydie mentioned this issue May 8, 2018
Merged
@prydie prydie changed the title Mysql-agent can't recreat MySQL Cluster after full k8s cluster restart Mysql-agent can't recreate MySQL Cluster after full k8s cluster restart May 14, 2018
@prydie prydie changed the title Mysql-agent can't recreate MySQL Cluster after full k8s cluster restart mysql-agent can't recreate MySQL Cluster after full k8s cluster restart May 14, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@paolomainardi @prydie @owainlewis @garthy @gites