diff --git a/.gitignore b/.gitignore index 69999d7be..3680fd43e 100644 --- a/.gitignore +++ b/.gitignore @@ -21,3 +21,4 @@ main .vendor/go19 /bin /build +.vscode/ diff --git a/docs/agents.md b/docs/agents.md index 80036fedb..1d216bbf5 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -1,18 +1,56 @@ # Agents -You may optionally install [orchestrator-agent](https://github.com/github/orchestrator-agent) on your MySQL hosts. -`orchestrator-agent`* is a service which registers with your `orchestrator` server and accepts requests by `orchestrator` via web API. +You may optionally install [orchestrator-agent](https://github.com/opernark/orchestrator-agent) on your MySQL hosts. +*`orchestrator-agent`* is a service which registers with your `orchestrator` server and accepts requests by `orchestrator` via web API. -Supported requests relate to general, OS and LVM operations, such as: -- Stopping/starting MySQL service on host -- Getting MySQL OS info such as data directory, port, disk space usage -- Performing various LVM operations such as finding LVM snapshots, mounting/unmounting a snapshot -- Transferring data between hosts (e.g. via `netcat`) +**orchestrator-agent** is capable of seeding new replicas using different seed methods, providing operating system, file system and LVM information to *orchestrator*, managing MySQL service as well as invoke certain commands and scripts. -`orchestrator-agent` is an ongoing effort in solving host-specific operations. It was originally developed to overcome cloning and restoring issues, and later expanded to other areas. - -The information and API exposed by `orchestrator-agent` to `orchestrator` allow `orchestrator` to coordinate and operate seeding of new or corrupted machines by getting data from freshly available snapshots. Moreover, it allows `orchestrator` -to automatically suggest the source of data for a given MySQL machine, by looking up such hosts that actually have a -recent snapshot available, preferably in the same datacenter. +Generic functionality offered by **orchestrator-agent**: +- Detection of the MySQL service, starting and stopping (start/stop/status commands provided via configuration) +- Detection of MySQL port, data directory, databases +- Calculation of disk usage on data directory mount point, OS and RAM size +- Tailing the error log file +- Discovery (the mere existence of the *orchestrator-agent* service on a host may suggest the existence or need of existence of a MySQL service) + +Specialized functionality offered by **orchestrator-agent**: +- Seeding new slaves using different seed methods (LVM, Xtrabackup, Mydumper, Mysqldump, Clone plugin) +- Detection of LVM snapshots on MySQL host (snapshots that are MySQL specific) +- Creation of new snapshots +- Mounting/umounting of LVM snapshots +- Detection of DC-local and DC-agnostic snapshots available for a given cluster For security measures, an agent requires a token to operate all but the simplest requests. This token is randomly generated by the agent and negotiated with `orchestrator`. `orchestrator` does not expose the agent's token (right now some work needs to be done on obscuring the token on error messages). + +When using **orchestrator-agent** you will get access to new Agents UI, where you will be able to seed data to new hosts as well as monitor live seeding progress. + +In order to enable **orchestrator-agent** add following to orchestrator configuration file + +```json +{ + "ServeAgentsHttp": true, +} +``` + +# Seeding +## Supported seed methods +Orchestrator-agent supports following seed methods: +* Mysqldump +* Mydumper +* Xtrabackup +* LVM +* Clone plugin + +## Seed process +Seed process consists of 5 stages, running sequentaly one after another: +* Prepare. On this stage different preparations are performed. Depenging on seed method they can include creating specific directories, stopping MySQL, starting socat for data transfer +* Backup. On this stage data is backuped on source host and transfered to target host +* Restore. On this stage data is restored on source host +* ConnectSlave. On this stage backup metadata is read and target host is connected as slave to source host +* Cleanup. On this stage different cleanup activities are executed + +There are also additional parameters, which can help you with tunning **orchestrator-agent** and seed process: +* `UnseenAgentForgetHours` - Number of hours after which an unseen agent is forgotten +* `AgentPollMinutes` - Minutes between agent polling +* `MaxRetriesForSeedStage` - Number of maximum retries for each of the seed stages, after which seed will be marked as failed +* `SeedProcessIntervalSeconds` - Interval in seconds between processing active seeds +* `SeedBackupStaleFailMinutes` - Number of minutes after which a stale (no progress) seed in Backup stage is considered failed \ No newline at end of file diff --git a/go/agent/agent.go b/go/agent/agent.go index e82b7ff6f..982f90752 100644 --- a/go/agent/agent.go +++ b/go/agent/agent.go @@ -16,7 +16,63 @@ package agent -import "github.com/github/orchestrator/go/inst" +import ( + "bytes" + "encoding/json" + "fmt" + "net/http" + "time" + + "github.com/github/orchestrator/go/config" + "github.com/github/orchestrator/go/inst" + "github.com/openark/golib/log" + "github.com/openark/golib/sqlutils" +) + +//Agent describes an agent +type Agent struct { + Info *Info + Data *Data + LastSeen time.Time + Status AgentStatus + ClusterAlias string + MySQLVersion string +} + +// Info stores basic agent information +type Info struct { + Hostname string + Port int + Token string + MySQLPort int +} + +// Data stores additional agent information +type Data struct { + LocalSnapshotsHosts []string + SnaphostHosts []string + LogicalVolumes []*LogicalVolume + MountPoint *Mount + BackupDir string + BackupDirDiskFree int64 + BackupDirDiskUsed int64 + MySQLRunning bool + MySQLDatadir string + MySQLDatadirDiskUsed int64 + MySQLDatadirDiskFree int64 + MySQLDatabases map[string]*MySQLDatabase + AvailiableSeedMethods map[SeedMethod]*SeedMethodOpts + AgentCommands []string + NumCPU int + MemTotal int64 + OsName string +} + +// MySQLDatabase describes a MySQL database +type MySQLDatabase struct { + Engines []Engine + Size int64 +} // LogicalVolume describes an LVM volume type LogicalVolume struct { @@ -25,58 +81,403 @@ type LogicalVolume struct { Path string IsSnapshot bool SnapshotPercent float64 + CreatedAt time.Time } // Mount describes a file system mount point type Mount struct { - Path string - Device string - LVPath string - FileSystem string - IsMounted bool - DiskUsage int64 - MySQLDataPath string - MySQLDiskUsage int64 + Path string + Device string + LVPath string + FileSystem string + IsMounted bool + DiskUsage int64 } -// Agent presents the data of an agent -type Agent struct { - Hostname string - Port int - Token string - LastSubmitted string - AvailableLocalSnapshots []string - AvailableSnapshots []string - LogicalVolumes []LogicalVolume - MountPoint Mount - MySQLRunning bool - MySQLDiskUsage int64 - MySQLPort int64 - MySQLDatadirDiskFree int64 - MySQLErrorLogTail []string -} - -// SeedOperation makes for the high level data & state of a seed operation -type SeedOperation struct { - SeedId int64 - TargetHostname string - SourceHostname string - StartTimestamp string - EndTimestamp string - IsComplete bool - IsSuccessful bool -} - -// SeedOperationState represents a single state (step) in a seed operation -type SeedOperationState struct { - SeedStateId int64 - SeedId int64 - StateTimestamp string - Action string - ErrorMessage string -} - -// Build an instance key for a given agent -func (this *Agent) GetInstance() *inst.InstanceKey { - return &inst.InstanceKey{Hostname: this.Hostname, Port: int(this.MySQLPort)} +// SeedMethodOpts stores configuration options for SeedMethod +type SeedMethodOpts struct { + BackupSide SeedSide + SupportedEngines []Engine + BackupToDatadir bool +} + +type AgentStatus int + +const ( + Active AgentStatus = iota + Inactive +) + +func (a AgentStatus) String() string { + return [...]string{"Active", "Inactive"}[a] +} + +func (a AgentStatus) MarshalJSON() ([]byte, error) { + buffer := bytes.NewBufferString(`"`) + buffer.WriteString(a.String()) + buffer.WriteString(`"`) + return buffer.Bytes(), nil +} + +var ToAgentStatus = map[string]AgentStatus{ + "Active": Active, + "Inactive": Inactive, +} + +var AgentStatuses = []AgentStatus{Active, Inactive} + +// AuditAgentOperation creates and writes a new audit entry by given agent +func auditAgentOperation(auditType string, agent *Agent, message string) error { + instanceKey := &inst.InstanceKey{} + if agent != nil { + instanceKey = &inst.InstanceKey{Hostname: agent.Info.Hostname, Port: agent.Info.MySQLPort} + } + return inst.AuditOperation(auditType, instanceKey, message) +} + +// RegisterAgent registers a new agent +func RegisterAgent(agentInfo *Info) (string, error) { + agent := &Agent{Info: agentInfo, Data: &Data{}} + err := agent.getAgentData() + if err != nil { + return "", log.Errore(fmt.Errorf("Unable to get agent data: %+v", err)) + } + agent.Status = Active + agent.LastSeen = time.Now() + err = agent.registerAgent() + if err != nil { + return "", log.Errore(fmt.Errorf("Unable to save agent to database: %+v", err)) + } + + // Try to discover topology instances when an agent submits + go agent.discoverAgentInstance() + + return agentInfo.Hostname, err +} + +// ReadAgent returns an information about an agent and it's data from database +func ReadAgent(hostname string) (*Agent, error) { + whereCondition := ` + WHERE + ha.hostname = ? + ` + res, err := readAgents(whereCondition, sqlutils.Args(hostname), "") + if err != nil { + return nil, err + } + if len(res) == 0 { + return nil, fmt.Errorf("Agent %s not found", hostname) + } + return res[0], nil +} + +// ReadAgentInfo returns an information about an agent without data from database +func ReadAgentInfo(hostname string) (*Agent, error) { + whereCondition := ` + WHERE + hostname = ? + ` + res, err := readAgentsInfo(whereCondition, sqlutils.Args(hostname), "") + if err != nil { + return nil, err + } + if len(res) == 0 { + return nil, fmt.Errorf("Agent %s not found", hostname) + } + return res[0], nil +} + +// ReadAgents returns a list of all known agents with their data from database +func ReadAgents() ([]*Agent, error) { + return readAgents(``, sqlutils.Args(), "") +} + +// ReadAgentsInfo returns a list of all known agents without data from database +func ReadAgentsInfo() ([]*Agent, error) { + return readAgentsInfo(``, sqlutils.Args(), "") +} + +// ReadOutdatedAgents returns agents that need to be updated +func ReadOutdatedAgents() ([]*Agent, error) { + whereCondition := ` + WHERE + IFNULL(last_checked < now() - interval ? minute, 1) + ` + return readAgentsInfo(whereCondition, sqlutils.Args(config.Config.AgentPollMinutes), "") +} + +// ReadAgentsHosts returns a list of agent hostnames +func ReadAgentsHosts(hostname string) ([]string, error) { + if len(hostname) == 0 { + return readAgentsHosts(``, sqlutils.Args()) + } + return readAgentsHosts(`WHERE hostname RLIKE ?`, sqlutils.Args(fmt.Sprintf("^"+hostname))) +} + +// ReadAgentsPaged returns a list of all known agents with their data from database with pagination +func ReadAgentsPaged(page int) ([]*Agent, error) { + limit := fmt.Sprintf("limit %d offset %d", config.AuditPageSize, page*config.AuditPageSize) + return readAgents(``, sqlutils.Args(), limit) +} + +// ReadAgentsForClusterPaged returns an agents for cluster with pagination +func ReadAgentsForClusterPaged(clusterAlias string, page int) ([]*Agent, error) { + limit := fmt.Sprintf("limit %d offset %d", config.AuditPageSize, page*config.AuditPageSize) + whereCondition := ` + WHERE + di.suggested_cluster_alias = ? + ` + return readAgents(whereCondition, sqlutils.Args(clusterAlias), limit) +} + +// ReadAgentsInStatusPaged returns an agents in provided status with pagination +func ReadAgentsInStatusPaged(status AgentStatus, page int) ([]*Agent, error) { + limit := fmt.Sprintf("limit %d offset %d", config.AuditPageSize, page*config.AuditPageSize) + whereCondition := ` + WHERE + ha.status = ? + ` + return readAgents(whereCondition, sqlutils.Args(status.String()), limit) +} + +// executeAgentCommand requests an agent to execute a command via HTTP api +func (agent *Agent) executeAgentCommand(command string, onResponse *func([]byte)) error { + httpFunc := func(uri string) (resp *http.Response, err error) { + return httpGet(uri) + } + auditAgentOperation("agent-command", agent, command) + return executeCommandWithMethodFunc(agent.Info.Hostname, agent.Info.Port, agent.Info.Token, command, httpFunc, onResponse) +} + +// executeAgentPostCommand requests an agent to execute a command via HTTP POST +func (agent *Agent) executeAgentPostCommand(command string, content string, onResponse *func([]byte)) error { + httpFunc := func(uri string) (resp *http.Response, err error) { + return httpPost(uri, "text/plain", content) + } + auditAgentOperation("agent-command", agent, command) + return executeCommandWithMethodFunc(agent.Info.Hostname, agent.Info.Port, agent.Info.Token, command, httpFunc, onResponse) +} + +// If a mysql port is available, try to discover against it +func (agent *Agent) discoverAgentInstance() error { + instanceKey := &inst.InstanceKey{Hostname: agent.Info.Hostname, Port: agent.Info.MySQLPort} + instance, err := inst.ReadTopologyInstance(instanceKey) + if err != nil { + log.Errorf("Failed to read topology for %v. err=%+v", instanceKey, err) + return err + } + if instance == nil { + log.Errorf("Failed to read topology for %v", instanceKey) + return err + } + log.Infof("Discovered Agent Instance: %v", instance.Key) + return nil +} + +// UpdateAgent reads information from agent API and updates orchestrator database +func (agent *Agent) UpdateAgent() error { + log.Debugf("Updating information for agent %+v", agent.Info.Hostname) + if err := agent.updateAgentLastChecked(); err != nil { + return fmt.Errorf("Unable to update last_checked field for agent %s: %+v", agent.Info.Hostname, err) + } + err := agent.getAgentData() + if err != nil { + agent.Status = Inactive + if statusUpdateErr := agent.updateAgentStatus(); statusUpdateErr != nil { + return fmt.Errorf("Unable to update status for agent %s: %+v", agent.Info.Hostname, statusUpdateErr) + } + } + return err +} + +// GetAgentData gets information about MySQL\LVM from agent +func (agent *Agent) getAgentData() error { + onResponse := func(body []byte) { + err := json.Unmarshal(body, agent.Data) + if err != nil { + log.Errore(err) + } + } + if err := agent.executeAgentCommand("get-agent-data", &onResponse); err != nil { + return err + } + agent.Status = Active + agent.LastSeen = time.Now() + return agent.updateAgentData() +} + +func (agent *Agent) relaylogContentsTail(startCoordinates *inst.BinlogCoordinates, onResponse *func([]byte)) error { + return agent.executeAgentCommand(fmt.Sprintf("mysql-relaylog-contents-tail/%s/%d", startCoordinates.LogFile, startCoordinates.LogPos), onResponse) +} + +func (agent *Agent) applyRelaylogContents(content string) error { + return agent.executeAgentPostCommand("apply-relaylog-contents", content, nil) +} + +// prepare starts prepare stage for seed on agent +func (agent *Agent) prepare(seedID int64, seedMethod SeedMethod, seedSide SeedSide) error { + return agent.executeAgentCommand(fmt.Sprintf("prepare/%d/%s/%s", seedID, seedMethod.String(), seedSide.String()), nil) +} + +// backup starts backup stage for seed on agent +func (agent *Agent) backup(seedID int64, seedMethod SeedMethod, seedHost string, mysqlPort int) error { + return agent.executeAgentCommand(fmt.Sprintf("backup/%d/%s/%s/%d", seedID, seedMethod.String(), seedHost, mysqlPort), nil) +} + +// restore starts restore stage for seed on agent +func (agent *Agent) restore(seedID int64, seedMethod SeedMethod) error { + return agent.executeAgentCommand(fmt.Sprintf("restore/%d/%s", seedID, seedMethod.String()), nil) +} + +// cleanup starts cleanup stage for seed on agent +func (agent *Agent) cleanup(seedID int64, seedMethod SeedMethod, seedSide SeedSide) error { + return agent.executeAgentCommand(fmt.Sprintf("cleanup/%d/%s/%s", seedID, seedMethod.String(), seedSide.String()), nil) +} + +// cleanup starts cleanup stage for seed on agent +func (agent *Agent) postSeedCmd(seedID int64) error { + return agent.executeAgentCommand(fmt.Sprintf("post-seed-cmd/%d", seedID), nil) +} + +// AbortSeed stops seed on agent +func (agent *Agent) AbortSeed(seedID int64, seedStage SeedStage) error { + return agent.executeAgentCommand(fmt.Sprintf("abort-seed-stage/%d/%s", seedID, seedStage), nil) +} + +// ErrorLogTail returns tail of MySQL error log from agent +func (agent *Agent) ErrorLogTail() (output string, err error) { + onResponse := func(body []byte) { + output = string(body) + log.Debugf("output: %v", output) + } + if err := agent.executeAgentCommand("mysql-error-log-tail", &onResponse); err != nil { + return "", err + } + return output, err +} + +// CustomCommand executes specified command on agent +func (agent *Agent) CustomCommand(cmd string) (output string, err error) { + onResponse := func(body []byte) { + output = string(body) + log.Debugf("output: %v", output) + } + if err := agent.executeAgentCommand(fmt.Sprintf("custom-commands/%s", cmd), &onResponse); err != nil { + return "", err + } + return output, err +} + +// Unmount unmounts the designated snapshot mount point +func (agent *Agent) Unmount() error { + onResponse := func(body []byte) { + err := json.Unmarshal(body, agent.Data) + if err != nil { + log.Errore(err) + } + } + if err := agent.executeAgentCommand("umount", &onResponse); err != nil { + return err + } + return agent.updateAgentData() +} + +// MountLV requests an agent to mount the given volume on the designated mount point +func (agent *Agent) MountLV(lv string) error { + onResponse := func(body []byte) { + err := json.Unmarshal(body, agent.Data) + if err != nil { + log.Errore(err) + } + } + if err := agent.executeAgentCommand(fmt.Sprintf("mountlv?lv=%s", lv), &onResponse); err != nil { + return err + } + return agent.updateAgentData() +} + +// RemoveLV requests an agent to remove a snapshot +func (agent *Agent) RemoveLV(lv string) error { + onResponse := func(body []byte) { + err := json.Unmarshal(body, agent.Data) + if err != nil { + log.Errore(err) + } + } + if err := agent.executeAgentCommand(fmt.Sprintf("removelv?lv=%s", lv), &onResponse); err != nil { + return err + } + return agent.updateAgentData() +} + +// CreateSnapshot requests an agent to create a new snapshot -- a DIY implementation +func (agent *Agent) CreateSnapshot() error { + onResponse := func(body []byte) { + err := json.Unmarshal(body, agent.Data) + if err != nil { + log.Errore(err) + } + } + if err := agent.executeAgentCommand("create-snapshot", &onResponse); err != nil { + return err + } + return agent.updateAgentData() +} + +// MySQLStart requests an agent to start the MySQL service +func (agent *Agent) MySQLStart() error { + onResponse := func(body []byte) { + err := json.Unmarshal(body, agent.Data) + if err != nil { + log.Errore(err) + } + } + if err := agent.executeAgentCommand("mysql-start", &onResponse); err != nil { + return err + } + return agent.updateAgentData() +} + +// MySQLStop requests an agent to stop MySQL service +func (agent *Agent) MySQLStop() error { + onResponse := func(body []byte) { + err := json.Unmarshal(body, agent.Data) + if err != nil { + log.Errore(err) + } + } + if err := agent.executeAgentCommand("mysql-stop", &onResponse); err != nil { + return err + } + return agent.updateAgentData() +} + +// getMetdata returns SeedMetadata for seed +func (agent *Agent) getMetadata(seedID int64, seedMethod SeedMethod) (*SeedMetadata, error) { + seedMetadata := &SeedMetadata{} + onResponse := func(body []byte) { + err := json.Unmarshal(body, seedMetadata) + if err != nil { + log.Errore(err) + } + } + if err := agent.executeAgentCommand(fmt.Sprintf("get-metadata/%d/%s", seedID, seedMethod.String()), &onResponse); err != nil { + return nil, err + } + return seedMetadata, nil +} + +// SeedStageState gets current state for seed stage for seedID +func (agent *Agent) getSeedStageState(seedID int64, seedStage SeedStage) (*SeedStageState, error) { + seedStageState := &SeedStageState{} + onResponse := func(body []byte) { + err := json.Unmarshal(body, seedStageState) + if err != nil { + log.Errore(err) + } + } + if err := agent.executeAgentCommand(fmt.Sprintf("seed-stage-state/%d/%s", seedID, seedStage.String()), &onResponse); err != nil { + return nil, err + } + return seedStageState, nil } diff --git a/go/agent/agent_dao.go b/go/agent/agent_dao.go index 510bc3e85..956a2a38e 100644 --- a/go/agent/agent_dao.go +++ b/go/agent/agent_dao.go @@ -17,243 +17,146 @@ package agent import ( - "crypto/tls" "encoding/json" - "errors" "fmt" - "io/ioutil" - "net" - "net/http" - "strings" - "sync" - "time" "github.com/github/orchestrator/go/config" "github.com/github/orchestrator/go/db" - "github.com/github/orchestrator/go/inst" "github.com/openark/golib/log" "github.com/openark/golib/sqlutils" ) -type httpMethodFunc func(uri string) (resp *http.Response, err error) - -var SeededAgents chan *Agent = make(chan *Agent) - -var httpClient *http.Client -var httpClientMutex = &sync.Mutex{} - -// InitHttpClient gets called once, and initializes httpClient according to config.Config -func InitHttpClient() { - httpClientMutex.Lock() - defer httpClientMutex.Unlock() - - if httpClient != nil { - return - } - - httpTimeout := time.Duration(time.Duration(config.AgentHttpTimeoutSeconds) * time.Second) - dialTimeout := func(network, addr string) (net.Conn, error) { - return net.DialTimeout(network, addr, httpTimeout) - } - httpTransport := &http.Transport{ - TLSClientConfig: &tls.Config{InsecureSkipVerify: config.Config.AgentSSLSkipVerify}, - Dial: dialTimeout, - ResponseHeaderTimeout: httpTimeout, - } - httpClient = &http.Client{Transport: httpTransport} -} - -// httpGet is a convenience method for getting http response from URL, optionaly skipping SSL cert verification -func httpGet(url string) (resp *http.Response, err error) { - return httpClient.Get(url) -} - -// httpPost is a convenience method for posting text data -func httpPost(url string, bodyType string, content string) (resp *http.Response, err error) { - return httpClient.Post(url, bodyType, strings.NewReader(content)) -} - -// AuditAgentOperation creates and writes a new audit entry by given agent -func auditAgentOperation(auditType string, agent *Agent, message string) error { - instanceKey := &inst.InstanceKey{} - if agent != nil { - instanceKey = &inst.InstanceKey{Hostname: agent.Hostname, Port: int(agent.MySQLPort)} - } - return inst.AuditOperation(auditType, instanceKey, message) -} - -// readResponse returns the body of an HTTP response -func readResponse(res *http.Response, err error) ([]byte, error) { - if err != nil { - return nil, err - } - defer res.Body.Close() - - body, err := ioutil.ReadAll(res.Body) - if err != nil { - return nil, err - } - - if res.Status == "500" { - return body, errors.New("Response Status 500") - } - - return body, nil -} - -// SubmitAgent submits a new agent for listing -func SubmitAgent(hostname string, port int, token string) (string, error) { - _, err := db.ExecOrchestrator(` - replace - into host_agent ( - hostname, port, token, last_submitted, count_mysql_snapshots - ) VALUES ( - ?, ?, ?, NOW(), 0 - ) - `, - hostname, - port, - token, - ) - if err != nil { - return "", log.Errore(err) - } - - // Try to discover topology instances when an agent submits - go DiscoverAgentInstance(hostname, port) - - return hostname, err -} - -// If a mysql port is available, try to discover against it -func DiscoverAgentInstance(hostname string, port int) error { - agent, err := GetAgent(hostname) - if err != nil { - log.Errorf("Couldn't get agent for %s: %v", hostname, err) - return err - } - - instanceKey := agent.GetInstance() - instance, err := inst.ReadTopologyInstance(instanceKey) - if err != nil { - log.Errorf("Failed to read topology for %v. err=%+v", instanceKey, err) - return err - } - if instance == nil { - log.Errorf("Failed to read topology for %v", instanceKey) - return err - } - log.Infof("Discovered Agent Instance: %v", instance.Key) - return nil -} - -// ForgetLongUnseenAgents will remove entries of all agents that have long since been last seen. -func ForgetLongUnseenAgents() error { - _, err := db.ExecOrchestrator(` - delete - from host_agent - where - last_submitted < NOW() - interval ? hour`, - config.Config.UnseenAgentForgetHours, - ) - return err -} - -// ReadOutdatedAgentsHosts returns agents that need to be updated -func ReadOutdatedAgentsHosts() ([]string, error) { +// readAgentsHosts reads agents hostnames from backend table applying filters +func readAgentsHosts(whereCondition string, args []interface{}) ([]string, error) { res := []string{} - query := ` + query := fmt.Sprintf(` select hostname from host_agent - where - IFNULL(last_checked < now() - interval ? minute, 1) - ` - err := db.QueryOrchestrator(query, sqlutils.Args(config.Config.AgentPollMinutes), func(m sqlutils.RowMap) error { - hostname := m.GetString("hostname") - res = append(res, hostname) + %s + `, whereCondition) + + err := db.QueryOrchestrator(query, args, func(m sqlutils.RowMap) error { + res = append(res, m.GetString("hostname")) return nil }) - if err != nil { - log.Errore(err) + return res, fmt.Errorf("Unable to read agents hostnames: %+v", err) } - return res, err + return res, nil } -// ReadAgents returns a list of all known agents -func ReadAgents() ([]Agent, error) { - res := []Agent{} - query := ` +// readAgentsInfo reads agents information from backend table +func readAgentsInfo(whereCondition string, args []interface{}, limit string) ([]*Agent, error) { + res := []*Agent{} + query := fmt.Sprintf(` select hostname, port, token, - last_submitted, - mysql_port + last_seen, + mysql_port, + status from host_agent + %s order by - hostname - ` - err := db.QueryOrchestratorRowsMap(query, func(m sqlutils.RowMap) error { - agent := Agent{} - agent.Hostname = m.GetString("hostname") - agent.Port = m.GetInt("port") - agent.MySQLPort = m.GetInt64("mysql_port") - agent.Token = "" - agent.LastSubmitted = m.GetString("last_submitted") - + hostname desc + %s + `, whereCondition, limit) + err := db.QueryOrchestrator(query, args, func(m sqlutils.RowMap) error { + agent := &Agent{Info: &Info{}, Data: &Data{}} + agent.Info.Hostname = m.GetString("hostname") + agent.Info.Port = m.GetInt("port") + agent.Info.MySQLPort = m.GetInt("mysql_port") + agent.Info.Token = m.GetString("token") + agent.LastSeen = m.GetTime("last_seen") + agent.Status = ToAgentStatus[m.GetString("status")] res = append(res, agent) return nil }) - if err != nil { - log.Errore(err) + return res, fmt.Errorf("Unable to read agents info: %+v", err) } - return res, err - + return res, nil } -// readAgentBasicInfo returns the basic data for an agent directly from backend table (no agent access) -func readAgentBasicInfo(hostname string) (Agent, string, error) { - agent := Agent{} - token := "" - query := ` - select - hostname, - port, - token, - last_submitted, - mysql_port - from - host_agent - where - hostname = ? - ` - err := db.QueryOrchestrator(query, sqlutils.Args(hostname), func(m sqlutils.RowMap) error { - agent.Hostname = m.GetString("hostname") - agent.Port = m.GetInt("port") - agent.LastSubmitted = m.GetString("last_submitted") - agent.MySQLPort = m.GetInt64("mysql_port") - token = m.GetString("token") - +// readAgentsInfo reads agent information with agent data from backend table +func readAgents(whereCondition string, args []interface{}, limit string) ([]*Agent, error) { + res := []*Agent{} + query := fmt.Sprintf(` + SELECT + ha.hostname, + ha.port, + ha.token, + ha.last_seen, + ha.mysql_port, + ha.status, + ha.data, + di.major_version, + ifnull(ca.alias, di.cluster_name) as cluster_alias + FROM + host_agent ha + LEFT JOIN database_instance di ON di.hostname = ha.hostname AND di.port = ha.mysql_port + LEFT JOIN cluster_alias ca on ca.cluster_name = di.cluster_name + %s + ORDER BY + cluster_alias ASC, ha.hostname ASC + %s + `, whereCondition, limit) + err := db.QueryOrchestrator(query, args, func(m sqlutils.RowMap) error { + agent := &Agent{Info: &Info{}, Data: &Data{}} + agent.Info.Hostname = m.GetString("hostname") + agent.Info.Port = m.GetInt("port") + agent.Info.MySQLPort = m.GetInt("mysql_port") + agent.Info.Token = m.GetString("token") + agent.LastSeen = m.GetTime("last_seen") + agent.Status = ToAgentStatus[m.GetString("status")] + agent.ClusterAlias = m.GetString("cluster_alias") + agent.MySQLVersion = m.GetString("major_version") + err := json.Unmarshal([]byte(m.GetString("data")), agent.Data) + if err != nil { + return log.Errore(err) + } + res = append(res, agent) return nil }) + if err != nil { - return agent, "", err + log.Errore(err) } + return res, err +} - if token == "" { - return agent, "", log.Errorf("Cannot get agent/token: %s", hostname) +// registerAgent inserts info about agent to database +func (agent *Agent) registerAgent() error { + agentData, err := json.Marshal(agent.Data) + if err != nil { + return log.Errore(err) } - return agent, token, nil + _, err = db.ExecOrchestrator(` + replace + into host_agent ( + hostname, port, token, last_seen, mysql_port, status, data + ) VALUES ( + ?, ?, ?, ?, ?, ?, ? + ) + `, + agent.Info.Hostname, + agent.Info.Port, + agent.Info.Token, + agent.LastSeen, + agent.Info.MySQLPort, + agent.Status.String(), + agentData, + ) + return err } -// UpdateAgentLastChecked updates the last_check timestamp in the orchestrator backed database +// updateAgentLastChecked updates the last_check timestamp in the orchestrator backed database // for a given agent -func UpdateAgentLastChecked(hostname string) error { +func (agent *Agent) updateAgentLastChecked() error { _, err := db.ExecOrchestrator(` update host_agent @@ -261,684 +164,58 @@ func UpdateAgentLastChecked(hostname string) error { last_checked = NOW() where hostname = ?`, - hostname, + agent.Info.Hostname, ) - if err != nil { - return log.Errore(err) - } - - return nil + return err } -// UpdateAgentInfo updates some agent state in backend table -func UpdateAgentInfo(hostname string, agent Agent) error { +// updateAgentStatus updates a status of an agent +func (agent *Agent) updateAgentStatus() error { _, err := db.ExecOrchestrator(` update host_agent set - last_seen = NOW(), - mysql_port = ?, - count_mysql_snapshots = ? + status = ? where hostname = ?`, - agent.MySQLPort, - len(agent.LogicalVolumes), - hostname, - ) - if err != nil { - return log.Errore(err) - } - - return nil -} - -// baseAgentUri returns the base URI for accessing an agent -func baseAgentUri(agentHostname string, agentPort int) string { - protocol := "http" - if config.Config.AgentsUseSSL { - protocol = "https" - } - uri := fmt.Sprintf("%s://%s:%d/api", protocol, agentHostname, agentPort) - log.Debugf("orchestrator-agent uri: %s", uri) - return uri -} - -// GetAgent gets a single agent status from the agent service. This involves multiple HTTP requests. -func GetAgent(hostname string) (Agent, error) { - agent, token, err := readAgentBasicInfo(hostname) - if err != nil { - return agent, log.Errore(err) - } - - // All seems to be in order. Now make some inquiries from orchestrator-agent service: - { - uri := baseAgentUri(agent.Hostname, agent.Port) - log.Debugf("orchestrator-agent uri: %s", uri) - - { - availableLocalSnapshotsUri := fmt.Sprintf("%s/available-snapshots-local?token=%s", uri, token) - body, err := readResponse(httpGet(availableLocalSnapshotsUri)) - if err == nil { - err = json.Unmarshal(body, &agent.AvailableLocalSnapshots) - } - if err != nil { - log.Errore(err) - } - } - { - availableSnapshotsUri := fmt.Sprintf("%s/available-snapshots?token=%s", uri, token) - body, err := readResponse(httpGet(availableSnapshotsUri)) - if err == nil { - err = json.Unmarshal(body, &agent.AvailableSnapshots) - } - if err != nil { - log.Errore(err) - } - } - { - lvSnapshotsUri := fmt.Sprintf("%s/lvs-snapshots?token=%s", uri, token) - body, err := readResponse(httpGet(lvSnapshotsUri)) - if err == nil { - err = json.Unmarshal(body, &agent.LogicalVolumes) - } - if err != nil { - log.Errore(err) - } - } - { - mountUri := fmt.Sprintf("%s/mount?token=%s", uri, token) - body, err := readResponse(httpGet(mountUri)) - if err == nil { - err = json.Unmarshal(body, &agent.MountPoint) - } - if err != nil { - log.Errore(err) - } - } - { - mySQLRunningUri := fmt.Sprintf("%s/mysql-status?token=%s", uri, token) - body, err := readResponse(httpGet(mySQLRunningUri)) - if err == nil { - err = json.Unmarshal(body, &agent.MySQLRunning) - } - // Actually an error is OK here since "status" returns with non-zero exit code when MySQL not running - } - { - mySQLRunningUri := fmt.Sprintf("%s/mysql-port?token=%s", uri, token) - body, err := readResponse(httpGet(mySQLRunningUri)) - if err == nil { - err = json.Unmarshal(body, &agent.MySQLPort) - } - if err != nil { - log.Errore(err) - } - } - { - mySQLDiskUsageUri := fmt.Sprintf("%s/mysql-du?token=%s", uri, token) - body, err := readResponse(httpGet(mySQLDiskUsageUri)) - if err == nil { - err = json.Unmarshal(body, &agent.MySQLDiskUsage) - } - if err != nil { - log.Errore(err) - } - } - { - mySQLDatadirDiskFreeUri := fmt.Sprintf("%s/mysql-datadir-available-space?token=%s", uri, token) - body, err := readResponse(httpGet(mySQLDatadirDiskFreeUri)) - if err == nil { - err = json.Unmarshal(body, &agent.MySQLDatadirDiskFree) - } - if err != nil { - log.Errore(err) - } - } - { - errorLogTailUri := fmt.Sprintf("%s/mysql-error-log-tail?token=%s", uri, token) - body, err := readResponse(httpGet(errorLogTailUri)) - if err == nil { - err = json.Unmarshal(body, &agent.MySQLErrorLogTail) - } - if err != nil { - log.Errore(err) - } - } - } - return agent, err -} - -// executeAgentCommandWithMethodFunc requests an agent to execute a command via HTTP api, either GET or POST, -// with specific http method implementation by the caller -func executeAgentCommandWithMethodFunc(hostname string, command string, methodFunc httpMethodFunc, onResponse *func([]byte)) (Agent, error) { - agent, token, err := readAgentBasicInfo(hostname) - if err != nil { - return agent, err - } - - // All seems to be in order. Now make some inquiries from orchestrator-agent service: - uri := baseAgentUri(agent.Hostname, agent.Port) - - var fullCommand string - if strings.Contains(command, "?") { - fullCommand = fmt.Sprintf("%s&token=%s", command, token) - } else { - fullCommand = fmt.Sprintf("%s?token=%s", command, token) - } - log.Debugf("orchestrator-agent command: %s", fullCommand) - agentCommandUri := fmt.Sprintf("%s/%s", uri, fullCommand) - - body, err := readResponse(methodFunc(agentCommandUri)) - if err != nil { - return agent, log.Errore(err) - } - if onResponse != nil { - (*onResponse)(body) - } - auditAgentOperation("agent-command", &agent, command) - - return agent, err -} - -// executeAgentCommand requests an agent to execute a command via HTTP api -func executeAgentCommand(hostname string, command string, onResponse *func([]byte)) (Agent, error) { - httpFunc := func(uri string) (resp *http.Response, err error) { - return httpGet(uri) - } - return executeAgentCommandWithMethodFunc(hostname, command, httpFunc, onResponse) -} - -// executeAgentPostCommand requests an agent to execute a command via HTTP POST -func executeAgentPostCommand(hostname string, command string, content string, onResponse *func([]byte)) (Agent, error) { - httpFunc := func(uri string) (resp *http.Response, err error) { - return httpPost(uri, "text/plain", content) - } - return executeAgentCommandWithMethodFunc(hostname, command, httpFunc, onResponse) -} - -// Unmount unmounts the designated snapshot mount point -func Unmount(hostname string) (Agent, error) { - return executeAgentCommand(hostname, "umount", nil) -} - -// MountLV requests an agent to mount the given volume on the designated mount point -func MountLV(hostname string, lv string) (Agent, error) { - return executeAgentCommand(hostname, fmt.Sprintf("mountlv?lv=%s", lv), nil) -} - -// RemoveLV requests an agent to remove a snapshot -func RemoveLV(hostname string, lv string) (Agent, error) { - return executeAgentCommand(hostname, fmt.Sprintf("removelv?lv=%s", lv), nil) -} - -// CreateSnapshot requests an agent to create a new snapshot -- a DIY implementation -func CreateSnapshot(hostname string) (Agent, error) { - return executeAgentCommand(hostname, "create-snapshot", nil) -} - -// deleteMySQLDatadir requests an agent to purge the MySQL data directory (step before seed) -func deleteMySQLDatadir(hostname string) (Agent, error) { - return executeAgentCommand(hostname, "delete-mysql-datadir", nil) -} - -// MySQLStop requests an agent to stop MySQL service -func MySQLStop(hostname string) (Agent, error) { - return executeAgentCommand(hostname, "mysql-stop", nil) -} - -// MySQLStart requests an agent to start the MySQL service -func MySQLStart(hostname string) (Agent, error) { - return executeAgentCommand(hostname, "mysql-start", nil) -} - -// ReceiveMySQLSeedData requests an agent to start listening for incoming seed data -func ReceiveMySQLSeedData(hostname string, seedId int64) (Agent, error) { - return executeAgentCommand(hostname, fmt.Sprintf("receive-mysql-seed-data/%d", seedId), nil) -} - -// ReceiveMySQLSeedData requests an agent to start sending seed data -func SendMySQLSeedData(hostname string, targetHostname string, seedId int64) (Agent, error) { - return executeAgentCommand(hostname, fmt.Sprintf("send-mysql-seed-data/%s/%d", targetHostname, seedId), nil) -} - -// ReceiveMySQLSeedData requests an agent to abort seed send/receive (depending on the agent's role) -func AbortSeedCommand(hostname string, seedId int64) (Agent, error) { - return executeAgentCommand(hostname, fmt.Sprintf("abort-seed/%d", seedId), nil) -} - -func CustomCommand(hostname string, cmd string) (output string, err error) { - onResponse := func(body []byte) { - output = string(body) - log.Debugf("output: %v", output) - } - - _, err = executeAgentCommand(hostname, fmt.Sprintf("custom-commands/%s", cmd), &onResponse) - return output, err -} - -// seedCommandCompleted checks an agent to see if it thinks a seed was completed. -func seedCommandCompleted(hostname string, seedId int64) (Agent, bool, error) { - result := false - onResponse := func(body []byte) { - json.Unmarshal(body, &result) - } - agent, err := executeAgentCommand(hostname, fmt.Sprintf("seed-command-completed/%d", seedId), &onResponse) - return agent, result, err -} - -// seedCommandCompleted checks an agent to see if it thinks a seed was successful. -func seedCommandSucceeded(hostname string, seedId int64) (Agent, bool, error) { - result := false - onResponse := func(body []byte) { - json.Unmarshal(body, &result) - } - agent, err := executeAgentCommand(hostname, fmt.Sprintf("seed-command-succeeded/%d", seedId), &onResponse) - return agent, result, err -} - -// AbortSeed will contact agents associated with a seed and request abort. -func AbortSeed(seedId int64) error { - seedOperations, err := AgentSeedDetails(seedId) - if err != nil { - return log.Errore(err) - } - - for _, seedOperation := range seedOperations { - AbortSeedCommand(seedOperation.TargetHostname, seedId) - AbortSeedCommand(seedOperation.SourceHostname, seedId) - } - updateSeedComplete(seedId, errors.New("Aborted")) - return nil -} - -// PostCopy will request an agent to invoke post-copy commands -func PostCopy(hostname, sourceHostname string) (Agent, error) { - return executeAgentCommand(hostname, fmt.Sprintf("post-copy/?sourceHost=%s", sourceHostname), nil) -} - -// SubmitSeedEntry submits a new seed operation entry, returning its unique ID -func SubmitSeedEntry(targetHostname string, sourceHostname string) (int64, error) { - res, err := db.ExecOrchestrator(` - insert - into agent_seed ( - target_hostname, source_hostname, start_timestamp - ) VALUES ( - ?, ?, NOW() - ) - `, - targetHostname, - sourceHostname, + agent.Status.String(), + agent.Info.Hostname, ) - if err != nil { - return 0, log.Errore(err) - } - id, err := res.LastInsertId() - - return id, err + return err } -// updateSeedComplete updates the seed entry, signing for completion -func updateSeedComplete(seedId int64, seedError error) error { - _, err := db.ExecOrchestrator(` - update - agent_seed - set end_timestamp = NOW(), - is_complete = 1, - is_successful = ? - where - agent_seed_id = ? - `, - (seedError == nil), - seedId, - ) +// updateAgentData updates the data in the orchestrator backend database +// for a given agent +func (agent *Agent) updateAgentData() error { + agentData, err := json.Marshal(agent.Data) if err != nil { return log.Errore(err) } - - return nil -} - -// submitSeedStateEntry submits a seed state: a single step in the overall seed process -func submitSeedStateEntry(seedId int64, action string, errorMessage string) (int64, error) { - res, err := db.ExecOrchestrator(` - insert - into agent_seed_state ( - agent_seed_id, state_timestamp, state_action, error_message - ) VALUES ( - ?, NOW(), ?, ? - ) - `, - seedId, - action, - errorMessage, - ) - if err != nil { - return 0, log.Errore(err) - } - id, err := res.LastInsertId() - - return id, err -} - -// updateSeedStateEntry updates seed step state -func updateSeedStateEntry(seedStateId int64, reason error) error { - _, err := db.ExecOrchestrator(` - update - agent_seed_state - set error_message = ? - where - agent_seed_state_id = ? - `, - reason.Error(), - seedStateId, + _, err = db.ExecOrchestrator(` + update + host_agent + set + data = ?, + status = ?, + last_seen = ? + where + hostname = ?`, + agentData, + agent.Status.String(), + agent.LastSeen, + agent.Info.Hostname, ) - if err != nil { - return log.Errore(err) - } - - return reason + return err } -// FailStaleSeeds marks as failed seeds where no progress have been seen recently -func FailStaleSeeds() error { +// ForgetLongUnseenAgents will remove entries of all agents that have long since been last seen. +func ForgetLongUnseenAgents() error { _, err := db.ExecOrchestrator(` - update - agent_seed - set - is_complete=1, - is_successful=0 - where - is_complete=0 - and ( - select - max(state_timestamp) as last_state_timestamp - from - agent_seed_state - where - agent_seed.agent_seed_id = agent_seed_state.agent_seed_id - ) < now() - interval ? minute`, - config.Config.StaleSeedFailMinutes, + delete + from host_agent + where + last_seen < NOW() - interval ? hour`, + config.Config.UnseenAgentForgetHours, ) return err } - -// executeSeed is *the* function for taking a seed. It is a complex operation of testing, preparing, re-testing -// agents on both sides, initiating data transfer, following up, awaiting completion, diagnosing errors, claning up. -func executeSeed(seedId int64, targetHostname string, sourceHostname string) error { - - var err error - var seedStateId int64 - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("getting target agent info for %s", targetHostname), "") - targetAgent, err := GetAgent(targetHostname) - SeededAgents <- &targetAgent - if err != nil { - return updateSeedStateEntry(seedStateId, err) - } - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("getting source agent info for %s", sourceHostname), "") - sourceAgent, err := GetAgent(sourceHostname) - if err != nil { - return updateSeedStateEntry(seedStateId, err) - } - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Checking MySQL status on target %s", targetHostname), "") - if targetAgent.MySQLRunning { - return updateSeedStateEntry(seedStateId, errors.New("MySQL is running on target host. Cowardly refusing to proceeed. Please stop the MySQL service")) - } - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Looking up available snapshots on source %s", sourceHostname), "") - if len(sourceAgent.LogicalVolumes) == 0 { - return updateSeedStateEntry(seedStateId, errors.New("No logical volumes found on source host")) - } - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Checking mount point on source %s", sourceHostname), "") - if sourceAgent.MountPoint.IsMounted { - return updateSeedStateEntry(seedStateId, errors.New("Volume already mounted on source host; please unmount")) - } - - seedFromLogicalVolume := sourceAgent.LogicalVolumes[0] - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("%s Mounting logical volume: %s", sourceHostname, seedFromLogicalVolume.Path), "") - _, err = MountLV(sourceHostname, seedFromLogicalVolume.Path) - if err != nil { - return updateSeedStateEntry(seedStateId, err) - } - sourceAgent, err = GetAgent(sourceHostname) - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("MySQL data volume on source host %s is %d bytes", sourceHostname, sourceAgent.MountPoint.MySQLDiskUsage), "") - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Erasing MySQL data on %s", targetHostname), "") - _, err = deleteMySQLDatadir(targetHostname) - if err != nil { - return updateSeedStateEntry(seedStateId, err) - } - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Aquiring target host datadir free space on %s", targetHostname), "") - targetAgent, err = GetAgent(targetHostname) - if err != nil { - return updateSeedStateEntry(seedStateId, err) - } - - if sourceAgent.MountPoint.MySQLDiskUsage > targetAgent.MySQLDatadirDiskFree { - Unmount(sourceHostname) - return updateSeedStateEntry(seedStateId, fmt.Errorf("Not enough disk space on target host %s. Required: %d, available: %d. Bailing out.", targetHostname, sourceAgent.MountPoint.MySQLDiskUsage, targetAgent.MySQLDatadirDiskFree)) - } - - // ... - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("%s will now receive data in background", targetHostname), "") - ReceiveMySQLSeedData(targetHostname, seedId) - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Waiting %d seconds for %s to start listening for incoming data", config.Config.SeedWaitSecondsBeforeSend, targetHostname), "") - time.Sleep(time.Duration(config.Config.SeedWaitSecondsBeforeSend) * time.Second) - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("%s will now send data to %s in background", sourceHostname, targetHostname), "") - SendMySQLSeedData(sourceHostname, targetHostname, seedId) - - copyComplete := false - numStaleIterations := 0 - var bytesCopied int64 = 0 - - for !copyComplete { - targetAgentPoll, err := GetAgent(targetHostname) - if err != nil { - return log.Errore(err) - } - - if targetAgentPoll.MySQLDiskUsage == bytesCopied { - numStaleIterations++ - } - bytesCopied = targetAgentPoll.MySQLDiskUsage - - copyFailed := false - if _, commandCompleted, _ := seedCommandCompleted(targetHostname, seedId); commandCompleted { - copyComplete = true - if _, commandSucceeded, _ := seedCommandSucceeded(targetHostname, seedId); !commandSucceeded { - // failed. - copyFailed = true - } - } - if numStaleIterations > 10 { - copyFailed = true - } - if copyFailed { - AbortSeedCommand(sourceHostname, seedId) - AbortSeedCommand(targetHostname, seedId) - Unmount(sourceHostname) - return updateSeedStateEntry(seedStateId, errors.New("10 iterations have passed without progress. Bailing out.")) - } - - var copyPct int64 = 0 - if sourceAgent.MountPoint.MySQLDiskUsage > 0 { - copyPct = 100 * bytesCopied / sourceAgent.MountPoint.MySQLDiskUsage - } - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Copied %d/%d bytes (%d%%)", bytesCopied, sourceAgent.MountPoint.MySQLDiskUsage, copyPct), "") - - if !copyComplete { - time.Sleep(30 * time.Second) - } - } - - // Cleanup: - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Executing post-copy command on %s", targetHostname), "") - _, err = PostCopy(targetHostname, sourceHostname) - if err != nil { - return updateSeedStateEntry(seedStateId, err) - } - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("%s Unmounting logical volume: %s", sourceHostname, seedFromLogicalVolume.Path), "") - _, err = Unmount(sourceHostname) - if err != nil { - return updateSeedStateEntry(seedStateId, err) - } - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Starting MySQL on target: %s", targetHostname), "") - _, err = MySQLStart(targetHostname) - if err != nil { - return updateSeedStateEntry(seedStateId, err) - } - - seedStateId, _ = submitSeedStateEntry(seedId, fmt.Sprintf("Submitting MySQL instance for discovery: %s", targetHostname), "") - SeededAgents <- &targetAgent - - seedStateId, _ = submitSeedStateEntry(seedId, "Done", "") - - return nil -} - -// Seed is the entry point for making a seed -func Seed(targetHostname string, sourceHostname string) (int64, error) { - if targetHostname == sourceHostname { - return 0, log.Errorf("Cannot seed %s onto itself", targetHostname) - } - seedId, err := SubmitSeedEntry(targetHostname, sourceHostname) - if err != nil { - return 0, log.Errore(err) - } - - go func() { - err := executeSeed(seedId, targetHostname, sourceHostname) - updateSeedComplete(seedId, err) - }() - - return seedId, nil -} - -// readSeeds reads seed from the backend table -func readSeeds(whereCondition string, args []interface{}, limit string) ([]SeedOperation, error) { - res := []SeedOperation{} - query := fmt.Sprintf(` - select - agent_seed_id, - target_hostname, - source_hostname, - start_timestamp, - end_timestamp, - is_complete, - is_successful - from - agent_seed - %s - order by - agent_seed_id desc - %s - `, whereCondition, limit) - err := db.QueryOrchestrator(query, args, func(m sqlutils.RowMap) error { - seedOperation := SeedOperation{} - seedOperation.SeedId = m.GetInt64("agent_seed_id") - seedOperation.TargetHostname = m.GetString("target_hostname") - seedOperation.SourceHostname = m.GetString("source_hostname") - seedOperation.StartTimestamp = m.GetString("start_timestamp") - seedOperation.EndTimestamp = m.GetString("end_timestamp") - seedOperation.IsComplete = m.GetBool("is_complete") - seedOperation.IsSuccessful = m.GetBool("is_successful") - - res = append(res, seedOperation) - return nil - }) - - if err != nil { - log.Errore(err) - } - return res, err -} - -// ReadActiveSeedsForHost reads active seeds where host participates either as source or target -func ReadActiveSeedsForHost(hostname string) ([]SeedOperation, error) { - whereCondition := ` - where - is_complete = 0 - and ( - target_hostname = ? - or source_hostname = ? - ) - ` - return readSeeds(whereCondition, sqlutils.Args(hostname, hostname), "") -} - -// ReadRecentCompletedSeedsForHost reads active seeds where host participates either as source or target -func ReadRecentCompletedSeedsForHost(hostname string) ([]SeedOperation, error) { - whereCondition := ` - where - is_complete = 1 - and ( - target_hostname = ? - or source_hostname = ? - ) - ` - return readSeeds(whereCondition, sqlutils.Args(hostname, hostname), "limit 10") -} - -// AgentSeedDetails reads details from backend table -func AgentSeedDetails(seedId int64) ([]SeedOperation, error) { - whereCondition := ` - where - agent_seed_id = ? - ` - return readSeeds(whereCondition, sqlutils.Args(seedId), "") -} - -// ReadRecentSeeds reads seeds from backend table. -func ReadRecentSeeds() ([]SeedOperation, error) { - return readSeeds(``, sqlutils.Args(), "limit 100") -} - -// SeedOperationState reads states for a given seed operation -func ReadSeedStates(seedId int64) ([]SeedOperationState, error) { - res := []SeedOperationState{} - query := ` - select - agent_seed_state_id, - agent_seed_id, - state_timestamp, - state_action, - error_message - from - agent_seed_state - where - agent_seed_id = ? - order by - agent_seed_state_id desc - ` - err := db.QueryOrchestrator(query, sqlutils.Args(seedId), func(m sqlutils.RowMap) error { - seedState := SeedOperationState{} - seedState.SeedStateId = m.GetInt64("agent_seed_state_id") - seedState.SeedId = m.GetInt64("agent_seed_id") - seedState.StateTimestamp = m.GetString("state_timestamp") - seedState.Action = m.GetString("state_action") - seedState.ErrorMessage = m.GetString("error_message") - - res = append(res, seedState) - return nil - }) - - if err != nil { - log.Errore(err) - } - return res, err -} - -func RelaylogContentsTail(hostname string, startCoordinates *inst.BinlogCoordinates, onResponse *func([]byte)) (Agent, error) { - return executeAgentCommand(hostname, fmt.Sprintf("mysql-relaylog-contents-tail/%s/%d", startCoordinates.LogFile, startCoordinates.LogPos), onResponse) -} - -func ApplyRelaylogContents(hostname string, content string) (Agent, error) { - return executeAgentPostCommand(hostname, "apply-relaylog-contents", content, nil) -} diff --git a/go/agent/agent_http.go b/go/agent/agent_http.go new file mode 100644 index 000000000..818746bbe --- /dev/null +++ b/go/agent/agent_http.go @@ -0,0 +1,142 @@ +package agent + +import ( + "crypto/tls" + "encoding/json" + "fmt" + "io/ioutil" + "net" + "net/http" + "strings" + "sync" + "time" + + "github.com/github/orchestrator/go/config" + "github.com/openark/golib/log" +) + +type httpMethodFunc func(uri string) (resp *http.Response, err error) + +var httpClient *http.Client +var httpClientMutex = &sync.Mutex{} + +// APIResponseCode is an OK/ERROR response code +type APIResponseCode int + +const ( + ERROR APIResponseCode = iota + OK +) + +func (a *APIResponseCode) UnmarshalJSON(b []byte) error { + var value string + err := json.Unmarshal(b, &value) + if err != nil { + return err + } + *a = toAPIResponseCode[value] + return nil +} + +var toAPIResponseCode = map[string]APIResponseCode{ + "ERROR": ERROR, + "OK": OK, +} + +// APIResponse is a response returned as JSON to various requests. +type APIResponse struct { + Code APIResponseCode + Message string + Details string +} + +// InitHttpClient gets called once, and initializes httpClient according to config.Config +func InitHttpClient() { + httpClientMutex.Lock() + defer httpClientMutex.Unlock() + + if httpClient != nil { + return + } + + httpTimeout := time.Duration(time.Duration(config.AgentHttpTimeoutSeconds) * time.Second) + dialTimeout := func(network, addr string) (net.Conn, error) { + return net.DialTimeout(network, addr, httpTimeout) + } + httpTransport := &http.Transport{ + TLSClientConfig: &tls.Config{InsecureSkipVerify: config.Config.AgentSSLSkipVerify}, + Dial: dialTimeout, + ResponseHeaderTimeout: httpTimeout, + } + httpClient = &http.Client{Transport: httpTransport} +} + +// httpGet is a convenience method for getting http response from URL, optionaly skipping SSL cert verification +func httpGet(url string) (resp *http.Response, err error) { + return httpClient.Get(url) +} + +// httpPost is a convenience method for posting text data +func httpPost(url string, bodyType string, content string) (resp *http.Response, err error) { + return httpClient.Post(url, bodyType, strings.NewReader(content)) +} + +// executeAgentCommandWithMethodFunc requests an agent to execute a command via HTTP api, either GET or POST, +// with specific http method implementation by the caller +func executeCommandWithMethodFunc(hostname string, port int, token string, command string, methodFunc httpMethodFunc, onResponse *func([]byte)) error { + uri := baseAgentURI(hostname, port) + + var fullCommand string + if strings.Contains(command, "?") { + fullCommand = fmt.Sprintf("%s&token=%s", command, token) + } else { + fullCommand = fmt.Sprintf("%s?token=%s", command, token) + } + log.Debugf("orchestrator-agent command: %s", fullCommand) + agentCommandURI := fmt.Sprintf("%s/%s", uri, fullCommand) + + body, err := readResponse(methodFunc(agentCommandURI)) + if err != nil { + return err + } + if onResponse != nil { + (*onResponse)(body) + } + + return err +} + +// readResponse returns the body of an HTTP response +func readResponse(res *http.Response, err error) ([]byte, error) { + if err != nil { + return nil, err + } + defer res.Body.Close() + + body, err := ioutil.ReadAll(res.Body) + if err != nil { + return nil, err + } + + if res.StatusCode == http.StatusInternalServerError { + apiResponse := &APIResponse{} + err := json.Unmarshal(body, apiResponse) + if err != nil { + return nil, fmt.Errorf("response %s. Unable to unmarshall error message from agent", res.Status) + } + return nil, fmt.Errorf("message: %s, details: %s", apiResponse.Message, apiResponse.Details) + } + + return body, nil +} + +// baseAgentUri returns the base URI for accessing an agent +func baseAgentURI(agentHostname string, agentPort int) string { + protocol := "http" + if config.Config.AgentsUseSSL { + protocol = "https" + } + uri := fmt.Sprintf("%s://%s:%d/api", protocol, agentHostname, agentPort) + log.Debugf("orchestrator-agent uri: %s", uri) + return uri +} diff --git a/go/agent/agent_test.go b/go/agent/agent_test.go new file mode 100644 index 000000000..350178490 --- /dev/null +++ b/go/agent/agent_test.go @@ -0,0 +1,1763 @@ +package agent + +import ( + "database/sql" + "flag" + "fmt" + "io/ioutil" + "math/rand" + "net" + "net/http" + "net/http/httptest" + "os/exec" + "strconv" + "testing" + "time" + + logger "log" + + "github.com/go-martini/martini" + "github.com/martini-contrib/render" + "github.com/openark/golib/log" + + "github.com/github/orchestrator/go/config" + "github.com/github/orchestrator/go/db" + "github.com/github/orchestrator/go/inst" + mysqldrv "github.com/go-sql-driver/mysql" + "github.com/openark/golib/sqlutils" + "github.com/ory/dockertest" + . "gopkg.in/check.v1" +) + +// before running add following to your /etc/hosts file depending on number of agents you plan to use +// 127.0.0.2 agent1 +// 127.0.0.3 agent2 +// ... +// 127.0.0.n agentn + +// also if you are using Mac OS X, you will need to create the aliases with +// sudo ifconfig lo0 alias 127.0.0.2 up +// sudo ifconfig lo0 alias 127.0.0.3 up +// for each agent + +var agentsCount = 4 + +func init() { + log.SetLevel(log.DEBUG) +} + +var testname = flag.String("testname", "", "test names to run") + +func Test(t *testing.T) { TestingT(t) } + +type AgentTestSuite struct { + testAgents map[string]*testAgent + pool *dockertest.Pool + containers []*dockertest.Resource +} + +var _ = Suite(&AgentTestSuite{}) + +type testAgent struct { + agent *Agent + agentSeedStageStatus *SeedStageState + agentSeedMetadata *SeedMetadata + agentServer *httptest.Server + agentMux *martini.ClassicMartini + agentMySQLUseGTID bool + agentMySQLContainerIP string + agentMySQLContainerID string + agentContainerResource *dockertest.Resource +} + +func (s *AgentTestSuite) SetUpTest(c *C) { + if len(*testname) > 0 { + if c.TestName() != fmt.Sprintf("AgentTestSuite.%s", *testname) { + c.Skip("skipping test due to not matched testname") + log.Info("skipping test due to not matched testname") + } + } + log.Infof("Setting up test data for test %s", c.TestName()) + if _, err := db.ExecOrchestrator("TRUNCATE TABLE host_agent;"); err != nil { + log.Fatalf("Unable to truncate host_agent table: %s", err) + } + if _, err := db.ExecOrchestrator("TRUNCATE TABLE agent_seed;"); err != nil { + log.Fatalf("Unable to truncate agent_seed table: %s", err) + } + if _, err := db.ExecOrchestrator("TRUNCATE TABLE agent_seed_state;"); err != nil { + log.Fatalf("Unable to truncate agent_seed_state table: %s", err) + } + + for _, testAgent := range s.testAgents { + agentdb, _, err := sqlutils.GetDB(fmt.Sprintf("root:secret@(localhost:%s)/mysql", testAgent.agentContainerResource.GetPort("3306/tcp"))) + if err != nil { + log.Fatalf("Unable get agent container: %s", err) + } + if _, err = agentdb.Exec("DROP database if exists test_repl"); err != nil { + log.Fatalf("Unable to drop agent test DB: %s", err) + } + if _, err = agentdb.Exec("STOP SLAVE;"); err != nil { + log.Fatalf("Unable to execute stop slave: %s", err) + } + if _, err = agentdb.Exec("RESET SLAVE ALL;"); err != nil { + log.Fatalf("Unable to execute RESET SLAVE: %s", err) + } + if _, err = agentdb.Exec("RESET MASTER;"); err != nil { + log.Fatalf("Unable to execute RESET MASTER: %s", err) + } + if _, err = agentdb.Exec("Create database test_repl"); err != nil { + log.Fatalf("Unable to create agent test DB: %s", err) + } + if _, err = agentdb.Exec("Create table test_repl.test(id int);"); err != nil { + log.Fatalf("Unable create agent test table: %s", err) + } + if _, err = agentdb.Exec("insert into test_repl.test(id) VALUES (1), (2), (3), (4);"); err != nil { + log.Fatalf("Unable to insert data to agent test table: %s", err) + } + + if err = sqlutils.QueryRowsMap(agentdb, "show master status", func(m sqlutils.RowMap) error { + var err error + testAgent.agentSeedMetadata.LogFile = m.GetString("File") + testAgent.agentSeedMetadata.LogPos = m.GetInt64("Position") + testAgent.agentSeedMetadata.GtidExecuted = m.GetString("Executed_Gtid_Set") + return err + }); err != nil { + log.Fatalf("Unable to get agent binlog position: %s", err) + } + + mysqlDatabases := map[string]*MySQLDatabase{ + "sakila": { + Engines: []Engine{InnoDB}, + Size: 0, + }, + } + availiableSeedMethods := map[SeedMethod]*SeedMethodOpts{ + Mydumper: { + BackupSide: Target, + SupportedEngines: []Engine{ROCKSDB, MRG_MYISAM, CSV, BLACKHOLE, InnoDB, MEMORY, ARCHIVE, MyISAM, FEDERATED, TokuDB}, + BackupToDatadir: false, + }, + } + testAgent.agent = &Agent{ + Info: &Info{ + Hostname: testAgent.agent.Info.Hostname, + Port: testAgent.agent.Info.Port, + MySQLPort: testAgent.agent.Info.MySQLPort, + Token: "token", + }, + Data: &Data{ + LocalSnapshotsHosts: testAgent.agent.Data.LocalSnapshotsHosts, + SnaphostHosts: testAgent.agent.Data.SnaphostHosts, + LogicalVolumes: []*LogicalVolume{}, + MountPoint: &Mount{ + Path: "/tmp", + Device: "", + LVPath: "", + FileSystem: "", + IsMounted: false, + DiskUsage: 0, + }, + BackupDir: "/tmp/bkp", + BackupDirDiskUsed: 0, + BackupDirDiskFree: 10000, + MySQLRunning: true, + MySQLDatadir: "/var/lib/mysql", + MySQLDatadirDiskUsed: 10, + MySQLDatadirDiskFree: 10000, + MySQLDatabases: mysqlDatabases, + AvailiableSeedMethods: availiableSeedMethods, + }, + } + } +} + +func (s *AgentTestSuite) SetUpSuite(c *C) { + var testAgents = make(map[string]*testAgent) + s.testAgents = testAgents + s.containers = []*dockertest.Resource{} + pool, err := dockertest.NewPool("") + if err != nil { + log.Fatalf("Could not connect to docker: %s", err) + } + s.pool = pool + + // pulls an image, creates a container based on it and runs it + log.Info("Creating docker container for Orchestrator DB") + resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"}) + if err != nil { + log.Fatalf("Could not start resource: %s", err) + } + + s.containers = append(s.containers, resource) + + // configure Orchestrator + config.Config.MySQLOrchestratorHost = "127.0.0.1" + port, _ := strconv.Atoi(resource.GetPort("3306/tcp")) + config.Config.MySQLOrchestratorPort = uint(port) + config.Config.MySQLOrchestratorUser = "root" + config.Config.MySQLOrchestratorPassword = "secret" + config.Config.MySQLOrchestratorDatabase = "orchestrator" + config.Config.AgentsServerPort = ":3001" + config.Config.ServeAgentsHttp = true + config.Config.AuditToSyslog = false + config.Config.EnableSyslog = false + config.Config.MySQLConnectTimeoutSeconds = 600 + config.Config.MySQLConnectionLifetimeSeconds = 600 + config.Config.MySQLOrchestratorReadTimeoutSeconds = 600 + config.Config.MySQLTopologyReadTimeoutSeconds = 600 + config.Config.MySQLTopologySSLSkipVerify = true + config.Config.MySQLTopologyUseMixedTLS = true + config.Config.HostnameResolveMethod = "none" + config.Config.MySQLHostnameResolveMethod = "none" + falseFlag := false + trueFlag := true + config.RuntimeCLIFlags.Noop = &falseFlag + config.RuntimeCLIFlags.SkipUnresolve = &trueFlag + config.Config.SkipMaxScaleCheck = true + config.Config.MySQLTopologyUser = "orc_topology" + config.Config.MySQLTopologyPassword = "orc_topologypassword@" + config.Config.ReplicationCredentialsQuery = "select username, password from orchestrator_meta.replication;" + config.MarkConfigurationLoaded() + + // exponential backoff-retry, because the application in the container might not be ready to accept connections yet + if err := pool.Retry(func() error { + var testdb *sql.DB + var err error + testdb, _, err = sqlutils.GetDB(fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp"))) + mysqldrv.SetLogger(logger.New(ioutil.Discard, "discard", 1)) + if err != nil { + return err + } + return testdb.Ping() + }); err != nil { + log.Fatalf("Could not connect to docker: %s", err) + } + + log.Info("Creating Orchestrator DB") + _, err = db.OpenOrchestrator() + if err != nil { + log.Fatalf("Unable to create orchestrator DB: %s", err) + } + + // init few functions nessesary for test process + inst.InitializeInstanceDao() + InitHttpClient() + go func() { + for seededAgent := range SeededAgents { + instanceKey := &inst.InstanceKey{Hostname: seededAgent.Info.Hostname, Port: int(seededAgent.Info.MySQLPort)} + log.Infof("%+v", instanceKey) + } + }() + + // create mocks for agents + log.Info("Creating Orchestrator agents mocks") + for i := 1; i <= agentsCount; i++ { + mysqlDatabases := map[string]*MySQLDatabase{ + "sakila": { + Engines: []Engine{InnoDB}, + Size: 0, + }, + } + availiableSeedMethods := map[SeedMethod]*SeedMethodOpts{ + Mydumper: { + BackupSide: Target, + SupportedEngines: []Engine{ROCKSDB, MRG_MYISAM, CSV, BLACKHOLE, InnoDB, MEMORY, ARCHIVE, MyISAM, FEDERATED, TokuDB}, + }, + } + agent := &Agent{ + Info: &Info{ + Hostname: fmt.Sprintf("agent%d", i), + Port: 3002 + i, + Token: "token", + }, + Data: &Data{ + LocalSnapshotsHosts: []string{fmt.Sprintf("127.0.0.%d", i)}, + SnaphostHosts: []string{fmt.Sprintf("127.0.0.%d", i), "localhost"}, + LogicalVolumes: []*LogicalVolume{}, + MountPoint: &Mount{ + Path: "/tmp", + Device: "", + LVPath: "", + FileSystem: "", + IsMounted: false, + DiskUsage: 0, + }, + BackupDir: "/tmp/bkp", + BackupDirDiskFree: 10000, + MySQLRunning: true, + MySQLDatadir: "/var/lib/mysql", + MySQLDatadirDiskUsed: 10, + MySQLDatadirDiskFree: 10000, + MySQLDatabases: mysqlDatabases, + AvailiableSeedMethods: availiableSeedMethods, + }, + } + if err := s.createTestAgent(agent); err != nil { + log.Fatalf("unable to create test agent: %s", err) + } + } +} + +func (s *AgentTestSuite) createTestAgent(agent *Agent) error { + agentAddress := fmt.Sprintf("%s:%d", agent.Info.Hostname, agent.Info.Port) + m := martini.Classic() + m.Use(render.Renderer()) + testAgent := &testAgent{ + agent: agent, + agentSeedStageStatus: &SeedStageState{}, + agentSeedMetadata: &SeedMetadata{}, + agentMySQLUseGTID: true, + } + m.Get("/api/get-agent-data", func(r render.Render, res http.ResponseWriter, req *http.Request) { + r.JSON(200, testAgent.agent.Data) + }) + m.Get("/api/prepare/:seedID/:seedMethod/:seedSide", func(r render.Render, res http.ResponseWriter, req *http.Request) { + r.Text(202, "Started") + }) + m.Get("/api/backup/:seedID/:seedMethod/:seedHost/:mysqlPort", func(r render.Render, res http.ResponseWriter, req *http.Request) { + r.Text(202, "Started") + }) + m.Get("/api/restore/:seedID/:seedMethod", func(r render.Render, res http.ResponseWriter, req *http.Request) { + r.Text(202, "Started") + }) + m.Get("/api/cleanup/:seedID/:seedMethod/:seedSide", func(r render.Render, res http.ResponseWriter, req *http.Request) { + r.Text(202, "Started") + }) + m.Get("/api/post-seed-cmd/:seedID", func(r render.Render, res http.ResponseWriter, req *http.Request) { + r.Text(202, "Done") + }) + m.Get("/api/get-metadata/:seedID/:seedMethod", func(r render.Render, res http.ResponseWriter, req *http.Request) { + r.JSON(200, testAgent.agentSeedMetadata) + }) + m.Get("/api/seed-stage-state/:seedID/:seedStage", func(r render.Render, res http.ResponseWriter, req *http.Request) { + r.JSON(200, testAgent.agentSeedStageStatus) + }) + m.Get("/api/abort-seed-stage/:seedID/:seedStage", func(r render.Render, res http.ResponseWriter, req *http.Request) { + r.Text(200, "killed") + }) + testServer := httptest.NewUnstartedServer(m) + listener, _ := net.Listen("tcp", agentAddress) + testServer.Listener = listener + testServer.Start() + testAgent.agentServer = testServer + testAgent.agentMux = m + s.testAgents[agent.Info.Hostname] = testAgent + return s.createTestAgentsMySQLServers(testAgent) +} + +func (s *AgentTestSuite) createTestAgentsMySQLServers(agent *testAgent) error { + var testdb *sql.DB + rand.Seed(time.Now().UnixNano()) + serverID := rand.Intn(100000000) + + dockerCmd := []string{"mysqld", fmt.Sprintf("--server-id=%d", serverID), "--log-bin=/var/lib/mysql/mysql-bin"} + log.Infof("Creating docker container for agent %s", agent.agent.Info.Hostname) + + if agent.agentMySQLUseGTID { + dockerCmd = append(dockerCmd, "--enforce-gtid-consistency=ON", "--gtid-mode=ON") + } + resource, err := s.pool.RunWithOptions(&dockertest.RunOptions{Repository: "mysql", Tag: "5.7", Env: []string{"MYSQL_ROOT_PASSWORD=secret"}, Cmd: dockerCmd, CapAdd: []string{"NET_ADMIN", "NET_RAW"}}) + if err != nil { + return fmt.Errorf("Could not connect to docker: %s", err) + } + + agent.agentMySQLContainerIP = resource.Container.NetworkSettings.Networks["bridge"].IPAddress + agent.agentMySQLContainerID = resource.Container.ID + agent.agentContainerResource = resource + + s.containers = append(s.containers, resource) + + cmd := exec.Command("docker", "exec", "-i", agent.agentMySQLContainerID, "apt-get", "update") + if err := cmd.Run(); err != nil { + return err + } + cmd = exec.Command("docker", "exec", "-i", agent.agentMySQLContainerID, "apt-get", "install", "-y", "iptables") + if err := cmd.Run(); err != nil { + return err + } + + if err := s.pool.Retry(func() error { + var err error + testdb, _, err = sqlutils.GetDB(fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp"))) + mysqldrv.SetLogger(logger.New(ioutil.Discard, "discard", 1)) + if err != nil { + return err + } + return testdb.Ping() + }); err != nil { + return fmt.Errorf("Could not connect to docker: %s", err) + } + + if _, err = testdb.Exec("Create database orchestrator_meta;"); err != nil { + return err + } + if _, err = testdb.Exec("Create table orchestrator_meta.replication (`username` varchar(128) CHARACTER SET ascii NOT NULL DEFAULT '',`password` varchar(128) CHARACTER SET ascii NOT NULL DEFAULT '',PRIMARY KEY (`username`,`password`)) ENGINE=InnoDB DEFAULT CHARSET=utf8;"); err != nil { + return err + } + if _, err = testdb.Exec("CREATE USER `slave`@`%` IDENTIFIED BY 'slavepassword@';"); err != nil { + return err + } + if _, err = testdb.Exec("GRANT REPLICATION SLAVE ON *.* TO `slave`@`%`"); err != nil { + return err + } + if _, err = testdb.Exec("CREATE USER `orc_topology`@`%` IDENTIFIED BY 'orc_topologypassword@';"); err != nil { + return err + } + if _, err = testdb.Exec("GRANT ALL PRIVILEGES ON *.* TO `orc_topology`@`%`"); err != nil { + return err + } + if _, err = testdb.Exec("Insert into orchestrator_meta.replication(username, password) VALUES ('slave', 'slavepassword@');"); err != nil { + return err + } + + if _, err = testdb.Exec("Create database test_repl"); err != nil { + return err + } + if _, err = testdb.Exec("Create table test_repl.test(id int);"); err != nil { + return err + } + if _, err = testdb.Exec("insert into test_repl.test(id) VALUES (1), (2), (3), (4);"); err != nil { + return err + } + + if err = sqlutils.QueryRowsMap(testdb, "show master status", func(m sqlutils.RowMap) error { + var err error + agent.agentSeedMetadata.LogFile = m.GetString("File") + agent.agentSeedMetadata.LogPos = m.GetInt64("Position") + agent.agentSeedMetadata.GtidExecuted = m.GetString("Executed_Gtid_Set") + return err + }); err != nil { + return err + } + + mysqlPort, err := strconv.Atoi(resource.GetPort("3306/tcp")) + if err != nil { + return err + } + agent.agent.Info.MySQLPort = mysqlPort + + return nil +} + +// this is needed in order to redirect from host machine ip 172.0.0.x to container ip for replication to work +func (s *AgentTestSuite) createIPTablesRulesForReplication(targetAgent *testAgent, sourceAgent *testAgent) error { + cmd := exec.Command("docker", "exec", "-i", targetAgent.agentMySQLContainerID, "iptables", "-t", "nat", "-I", "OUTPUT", "-p", "tcp", "-o", "eth0", "--dport", fmt.Sprintf("%d", sourceAgent.agent.Info.MySQLPort), "-j", "DNAT", "--to-destination", fmt.Sprintf("%s:3306", sourceAgent.agentMySQLContainerIP)) + out, err := cmd.CombinedOutput() + if err != nil { + return fmt.Errorf("%s", string(out)) + } + cmd = exec.Command("docker", "exec", "-i", targetAgent.agentMySQLContainerID, "/bin/sh", "-c", fmt.Sprintf("echo %s %s >> /etc/hosts", sourceAgent.agentMySQLContainerIP, sourceAgent.agent.Info.Hostname)) + out, err = cmd.CombinedOutput() + if err != nil { + return fmt.Errorf("%s", string(out)) + } + return nil +} + +func (s *AgentTestSuite) TearDownSuite(c *C) { + log.Info("Teardown test data") + for _, container := range s.containers { + if err := s.pool.Purge(container); err != nil { + log.Fatalf("Could not purge resource: %s", err) + } + } + for _, testAgent := range s.testAgents { + testAgent.agentServer.Close() + } +} + +func (s *AgentTestSuite) registerAgents(c *C) { + for _, testAgent := range s.testAgents { + hostname, err := RegisterAgent(testAgent.agent.Info) + c.Assert(err, IsNil) + c.Assert(hostname, Equals, testAgent.agent.Info.Hostname) + } +} + +func (s *AgentTestSuite) getSeedAgents(c *C, targetTestAgent *testAgent, sourceTestAgent *testAgent) (targetAgent *Agent, sourceAgent *Agent) { + s.registerAgents(c) + targetAgent, err := ReadAgent(targetTestAgent.agent.Info.Hostname) + c.Assert(err, IsNil) + + sourceAgent, err = ReadAgent(sourceTestAgent.agent.Info.Hostname) + c.Assert(err, IsNil) + + return targetAgent, sourceAgent +} + +func (s *AgentTestSuite) readSeed(c *C, seedID int64, targetHostname string, sourceHostname string, seedMethod SeedMethod, backupSide SeedSide, status SeedStatus, stage SeedStage, retries int) *Seed { + seed, err := ReadSeed(seedID) + c.Assert(err, IsNil) + c.Assert(seed.TargetHostname, Equals, targetHostname) + c.Assert(seed.SourceHostname, Equals, sourceHostname) + c.Assert(seed.SeedMethod, Equals, seedMethod) + c.Assert(seed.BackupSide, Equals, backupSide) + c.Assert(seed.Status, Equals, status) + c.Assert(seed.Stage, Equals, stage) + c.Assert(seed.Retries, Equals, retries) + return seed +} + +func (s *AgentTestSuite) readSeedStageStates(c *C, seed *Seed, stateRecords int, targetTestAgent *testAgent, sourceTestAgent *testAgent) { + seedStates, err := seed.ReadSeedStageStates() + c.Assert(err, IsNil) + for _, seedState := range seedStates[:stateRecords] { + for _, agent := range []*testAgent{targetTestAgent, sourceTestAgent} { + if seedState.Hostname == agent.agent.Info.Hostname { + c.Assert(seedState.SeedID, Equals, agent.agentSeedStageStatus.SeedID) + c.Assert(seedState.Stage, Equals, agent.agentSeedStageStatus.Stage) + c.Assert(seedState.Status, Equals, agent.agentSeedStageStatus.Status) + } + } + } +} + +func (s *AgentTestSuite) TestAgentRegistration(c *C) { + s.registerAgents(c) +} + +func (s *AgentTestSuite) TestReadAgents(c *C) { + s.registerAgents(c) + + registeredAgents, err := ReadAgents() + c.Assert(err, IsNil) + c.Assert(registeredAgents, HasLen, 4) + + for _, testAgent := range s.testAgents { + for _, registeredAgent := range registeredAgents { + if registeredAgent.Info.Port == testAgent.agent.Info.Port && registeredAgent.Info.Hostname == testAgent.agent.Info.Hostname { + c.Assert(registeredAgent.Info, DeepEquals, testAgent.agent.Info) + c.Assert(registeredAgent.Data, DeepEquals, testAgent.agent.Data) + } + } + } +} + +func (s *AgentTestSuite) TestReadAgentsInfo(c *C) { + s.registerAgents(c) + + registeredAgents, err := ReadAgentsInfo() + c.Assert(err, IsNil) + c.Assert(registeredAgents, HasLen, 4) + + for _, testAgent := range s.testAgents { + for _, registeredAgent := range registeredAgents { + if registeredAgent.Info.Port == testAgent.agent.Info.Port && registeredAgent.Info.Hostname == testAgent.agent.Info.Hostname { + c.Assert(registeredAgent.Info, DeepEquals, testAgent.agent.Info) + c.Assert(registeredAgent.Data, DeepEquals, &Data{}) + } + } + } +} + +func (s *AgentTestSuite) TestReadAgent(c *C) { + testAgent := s.testAgents["agent1"] + + s.registerAgents(c) + + registeredAgent, err := ReadAgent(testAgent.agent.Info.Hostname) + c.Assert(err, IsNil) + c.Assert(registeredAgent.Info, DeepEquals, testAgent.agent.Info) + c.Assert(registeredAgent.Data, DeepEquals, testAgent.agent.Data) +} + +func (s *AgentTestSuite) TestReadAgentInfo(c *C) { + testAgent := s.testAgents["agent1"] + + s.registerAgents(c) + + registeredAgent, err := ReadAgentInfo(testAgent.agent.Info.Hostname) + c.Assert(err, IsNil) + c.Assert(registeredAgent.Info, DeepEquals, testAgent.agent.Info) + c.Assert(registeredAgent.Data, DeepEquals, &Data{}) +} + +func (s *AgentTestSuite) TestReadOutdatedAgents(c *C) { + config.Config.AgentPollMinutes = 2 + + s.registerAgents(c) + + outdatedAgents := []*testAgent{s.testAgents["agent1"]} + upToDateAgents := []*testAgent{s.testAgents["agent2"], s.testAgents["agent3"], s.testAgents["agent4"]} + + for _, outdatedAgent := range outdatedAgents { + db.ExecOrchestrator(fmt.Sprintf("UPDATE host_agent SET last_checked = NOW() - interval 60 minute WHERE hostname='%s'", outdatedAgent.agent.Info.Hostname)) + } + + for _, upToDateAgent := range upToDateAgents { + db.ExecOrchestrator(fmt.Sprintf("UPDATE host_agent SET last_checked = NOW() WHERE hostname='%s'", upToDateAgent.agent.Info.Hostname)) + } + + outdatedOrchestratorAgents, err := ReadOutdatedAgents() + c.Assert(err, IsNil) + c.Assert(outdatedOrchestratorAgents, HasLen, 1) + c.Assert(outdatedOrchestratorAgents[0].Info, DeepEquals, outdatedAgents[0].agent.Info) + c.Assert(outdatedOrchestratorAgents[0].Data, DeepEquals, &Data{}) +} + +func (s *AgentTestSuite) TestReadAgentsHosts(c *C) { + s.registerAgents(c) + + agents, err := ReadAgentsHosts("agent1") + + c.Assert(err, IsNil) + c.Assert(agents, HasLen, 1) + c.Assert(agents[0], Equals, "agent1") +} + +func (s *AgentTestSuite) TestReadAgentsPaged(c *C) { + s.registerAgents(c) + + agents, err := ReadAgentsPaged(0) + + c.Assert(err, IsNil) + c.Assert(agents, HasLen, 4) +} + +func (s *AgentTestSuite) TestReadAgentsInStatusPaged(c *C) { + s.registerAgents(c) + + agents, err := ReadAgentsInStatusPaged(Active, 0) + + c.Assert(err, IsNil) + c.Assert(agents, HasLen, 4) +} + +func (s *AgentTestSuite) TestUpdateAgent(c *C) { + testAgent := s.testAgents["agent1"] + + s.registerAgents(c) + + registeredAgent, err := ReadAgentInfo(testAgent.agent.Info.Hostname) + c.Assert(err, IsNil) + + testAgent.agent.Data.LocalSnapshotsHosts = []string{"127.0.0.10", "127.0.0.12"} + registeredAgent.Status = Inactive + + err = registeredAgent.updateAgentStatus() + c.Assert(err, IsNil) + + err = registeredAgent.UpdateAgent() + c.Assert(err, IsNil) + + c.Assert(registeredAgent.Info, DeepEquals, testAgent.agent.Info) + c.Assert(registeredAgent.Data, DeepEquals, testAgent.agent.Data) + c.Assert(registeredAgent.Status, Equals, Active) +} + +func (s *AgentTestSuite) TestUpdateAgentFailed(c *C) { + testAgent := s.testAgents["agent1"] + + s.registerAgents(c) + testAgent.agentServer.Close() + + registeredAgent, err := ReadAgentInfo(testAgent.agent.Info.Hostname) + c.Assert(err, IsNil) + + err = registeredAgent.UpdateAgent() + c.Assert(err, NotNil) + + registeredAgent, err = ReadAgentInfo(testAgent.agent.Info.Hostname) + c.Assert(err, IsNil) + c.Assert(registeredAgent.Status, Equals, Inactive) +} + +func (s *AgentTestSuite) TestForgetLongUnseenAgents(c *C) { + config.Config.UnseenAgentForgetHours = 1 + testAgent := s.testAgents["agent1"] + + s.registerAgents(c) + db.ExecOrchestrator(fmt.Sprintf("UPDATE host_agent SET last_seen = last_seen - interval 2 hour WHERE hostname='%s'", testAgent.agent.Info.Hostname)) + + err := ForgetLongUnseenAgents() + c.Assert(err, IsNil) + + _, err = ReadAgentInfo(testAgent.agent.Info.Hostname) + c.Assert(err, NotNil) +} + +func (s *AgentTestSuite) TestReadSeeds(c *C) { + targetTestAgent1 := s.testAgents["agent1"] + sourceTestAgent1 := s.testAgents["agent2"] + targetTestAgent2 := s.testAgents["agent3"] + sourceTestAgent2 := s.testAgents["agent4"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent1, sourceAgent1 := s.getSeedAgents(c, targetTestAgent1, sourceTestAgent1) + targetAgent2, sourceAgent2 := s.getSeedAgents(c, targetTestAgent2, sourceTestAgent2) + + _, err := NewSeed("Mydumper", targetAgent1, sourceAgent1) + c.Assert(err, IsNil) + + _, err = NewSeed("Mydumper", targetAgent2, sourceAgent2) + c.Assert(err, IsNil) + + seeds, err := ReadSeeds() + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 2) +} + +func (s *AgentTestSuite) TestReadSeedsPaged(c *C) { + targetTestAgent1 := s.testAgents["agent1"] + sourceTestAgent1 := s.testAgents["agent2"] + targetTestAgent2 := s.testAgents["agent3"] + sourceTestAgent2 := s.testAgents["agent4"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent1, sourceAgent1 := s.getSeedAgents(c, targetTestAgent1, sourceTestAgent1) + targetAgent2, sourceAgent2 := s.getSeedAgents(c, targetTestAgent2, sourceTestAgent2) + + _, err := NewSeed("Mydumper", targetAgent1, sourceAgent1) + c.Assert(err, IsNil) + + _, err = NewSeed("Mydumper", targetAgent2, sourceAgent2) + c.Assert(err, IsNil) + + seeds, err := ReadSeedsPaged(0) + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 2) +} + +func (s *AgentTestSuite) TestReadSeedsForTargetAgentPaged(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + + seeds, err := ReadSeedsForTargetAgentPaged(targetAgent, 0) + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 1) + c.Assert(seeds[0].TargetHostname, Equals, targetAgent.Info.Hostname) +} + +func (s *AgentTestSuite) TestReadSeedsForSourceAgentPaged(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + + seeds, err := ReadSeedsForSourceAgentPaged(sourceAgent, 0) + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 1) + c.Assert(seeds[0].SourceHostname, Equals, sourceAgent.Info.Hostname) +} + +func (s *AgentTestSuite) TestReadSeedsInStatusPaged(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + + seeds, err := ReadSeedsInStatusPaged(Scheduled, 0) + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 1) + seeds, err = ReadSeedsInStatusPaged(Running, 0) + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 0) +} + +func (s *AgentTestSuite) TestReadSeedsForAgent(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + + seeds, err := ReadSeedsForAgent(sourceAgent, "") + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 1) + c.Assert(seeds[0].SourceHostname, Equals, sourceAgent.Info.Hostname) + seeds, err = ReadSeedsForAgent(targetAgent, "") + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 1) + c.Assert(seeds[0].TargetHostname, Equals, targetAgent.Info.Hostname) +} + +func (s *AgentTestSuite) TestNewSeed(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) +} + +func (s *AgentTestSuite) TestNewSeedWrongSeedMethod(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("test", targetAgent, sourceAgent) + c.Assert(err, NotNil) +} + +func (s *AgentTestSuite) TestNewSeedSeedItself(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent1"] + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, NotNil) +} + +func (s *AgentTestSuite) TestNewSeedUnsupportedSeedMethod(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("Mysqldump", targetAgent, sourceAgent) + c.Assert(err, NotNil) +} + +func (s *AgentTestSuite) TestNewSeedUnsupportedSeedMethodForDB(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + sourceTestAgent.agent.Data.AvailiableSeedMethods[Xtrabackup] = &SeedMethodOpts{ + BackupSide: Target, + SupportedEngines: []Engine{MRG_MYISAM, CSV, BLACKHOLE, InnoDB, MEMORY, ARCHIVE, MyISAM, FEDERATED, TokuDB}, + } + sourceTestAgent.agent.Data.MySQLDatabases["test"] = &MySQLDatabase{ + Engines: []Engine{ROCKSDB}, + Size: 0, + } + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("Xtrabackup", targetAgent, sourceAgent) + c.Assert(err, NotNil) +} + +func (s *AgentTestSuite) TestNewSeedAgentHadActiveSeed(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + + _, err = NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, NotNil) +} + +func (s *AgentTestSuite) TestNewSeedNotEnoughSpaceInMySQLDatadir(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + targetTestAgent.agent.Data.MySQLDatadirDiskFree = 10 + sourceTestAgent.agent.Data.MySQLDatadirDiskUsed = 1000 + + // register agents to Orchestrator + s.registerAgents(c) + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + _, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, NotNil) +} + +func (s *AgentTestSuite) TestReadSeed(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) +} + +func (s *AgentTestSuite) TestReadActiveSeeds(c *C) { + targetTestAgent1 := s.testAgents["agent1"] + sourceTestAgent1 := s.testAgents["agent2"] + targetTestAgent2 := s.testAgents["agent3"] + sourceTestAgent2 := s.testAgents["agent4"] + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent1, sourceAgent1 := s.getSeedAgents(c, targetTestAgent1, sourceTestAgent1) + targetAgent2, sourceAgent2 := s.getSeedAgents(c, targetTestAgent2, sourceTestAgent2) + + seedID, err := NewSeed("Mydumper", targetAgent1, sourceAgent1) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + seedID, err = NewSeed("Mydumper", targetAgent2, sourceAgent2) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(2)) + + completedSeed := s.readSeed(c, seedID, targetAgent2.Info.Hostname, sourceAgent2.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + completedSeed.Status = Completed + completedSeed.updateSeedData() + + seeds, err := ReadActiveSeeds() + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 1) + + c.Assert(seeds[0].TargetHostname, Equals, targetTestAgent1.agent.Info.Hostname) + c.Assert(seeds[0].SourceHostname, Equals, sourceTestAgent1.agent.Info.Hostname) + c.Assert(seeds[0].SeedMethod, Equals, Mydumper) + c.Assert(seeds[0].BackupSide, Equals, Target) + c.Assert(seeds[0].Status, Equals, Scheduled) + c.Assert(seeds[0].Stage, Equals, Prepare) + c.Assert(seeds[0].Retries, Equals, 0) +} + +func (s *AgentTestSuite) TestReadActiveSeedsForAgent(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + for _, agent := range []*Agent{targetAgent, sourceAgent} { + seeds, err := ReadActiveSeedsForAgent(agent) + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 1) + c.Assert(seeds[0].TargetHostname, Equals, targetTestAgent.agent.Info.Hostname) + c.Assert(seeds[0].SourceHostname, Equals, sourceTestAgent.agent.Info.Hostname) + c.Assert(seeds[0].SeedMethod, Equals, Mydumper) + c.Assert(seeds[0].BackupSide, Equals, Target) + c.Assert(seeds[0].Status, Equals, Scheduled) + c.Assert(seeds[0].Stage, Equals, Prepare) + c.Assert(seeds[0].Retries, Equals, 0) + } + + activeSeed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + activeSeed.Status = Completed + activeSeed.updateSeedData() + + for _, agent := range []*Agent{targetAgent, sourceAgent} { + seeds, err := ReadActiveSeedsForAgent(agent) + c.Assert(err, IsNil) + c.Assert(seeds, HasLen, 0) + } + +} + +func (s *AgentTestSuite) TestReedSeedAgents(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + seedTargetAgent, seedSourceAgent, err := seed.readSeadAgents(false) + c.Assert(err, IsNil) + c.Assert(targetAgent.Info, DeepEquals, seedTargetAgent.Info) + c.Assert(sourceAgent.Info, DeepEquals, seedSourceAgent.Info) +} + +func (s *AgentTestSuite) TestProcessSeeds(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // create iptables rules for replication + err := s.createIPTablesRulesForReplication(targetTestAgent, sourceTestAgent) + c.Assert(err, IsNil) + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // ProcessSeeds. Orchestrator will ask agents to start Prepare stage. So it's state would change from Scheduled to Running + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Prepare, 0) + + // simulate that prepare stage is running on both agents + for _, agent := range []*testAgent{targetTestAgent, sourceTestAgent} { + agent.agentSeedStageStatus.SeedID = seedID + agent.agentSeedStageStatus.Stage = Prepare + agent.agentSeedStageStatus.Hostname = agent.agent.Info.Hostname + agent.agentSeedStageStatus.Timestamp = time.Now() + agent.agentSeedStageStatus.Status = Running + agent.agentSeedStageStatus.Details = "processing prepare stage" + } + + // check that SeedStageStates in Orchestator DB are the same, that were read from agents + s.readSeedStageStates(c, seed, 2, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. As seed is in status Running, Orchestrator will ask agent about seed stage status. As it's not Completed or Errored, it will just record SeedStageState in DB. + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Prepare, 0) + s.readSeedStageStates(c, seed, 2, targetTestAgent, sourceTestAgent) + + // now simulate that target agent completed Prepare stage + targetTestAgent.agentSeedStageStatus.Status = Completed + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Details = "completed prepare stage" + + // ProcessSeeds. We have stage: Prepare and status: Running as only target agent completed Prepare stage, but in SeedStageStates we will have one Completed record and one Running + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Prepare, 0) + s.readSeedStageStates(c, seed, 2, targetTestAgent, sourceTestAgent) + + // now simulate that source agent completed Prepare stage + sourceTestAgent.agentSeedStageStatus.Status = Completed + sourceTestAgent.agentSeedStageStatus.Timestamp = time.Now() + sourceTestAgent.agentSeedStageStatus.Details = "completed prepare stage" + + // ProcessSeeds. Now both agents completed Prepare stage, so Orchestrator will move Seed to Backup stage with status Scheduled. And we will have SeedStageState records with stage Prepare and status Completed + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Backup, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. Orchestrator will ask agent to start Backup stage. So it's state would change from Scheduled to Running + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Backup, 0) + + // simulate that Backup stage is running on target agent(as seedSide for Mydumper method is Target) + targetTestAgent.agentSeedStageStatus.Stage = Backup + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Running + targetTestAgent.agentSeedStageStatus.Details = "running backup stage" + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. TargetAgent is still running Backup Stage, so seed state won't change and we will have one SeedStageState record from target agent + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Backup, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // now simulate that target agent completed Backup stage + targetTestAgent.agentSeedStageStatus.Status = Completed + targetTestAgent.agentSeedStageStatus.Details = "completed backup stage" + + // ProcessSeeds. Target agent had completed Backup stage, so Orchestrator will move Seed to Restore stage with status Scheduled. And we will have one SeedStageState record with stage Backup and status Completed + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Restore, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. Orchestrator will ask agent to start Restore stage. So it's state would change from Scheduled to Running + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Restore, 0) + + // simulate that Backup stage is running on target agent(as restore is always processed by targetAgent) + targetTestAgent.agentSeedStageStatus.Stage = Restore + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Running + targetTestAgent.agentSeedStageStatus.Details = "running restore stage" + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. TargetAgent is still running Restore Stage, so seed state won't change and we will have one SeedStageState record from target agent + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Restore, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // now simulate that target agent completed Restore stage + targetTestAgent.agentSeedStageStatus.Status = Completed + targetTestAgent.agentSeedStageStatus.Details = "completed restore stage" + + //register agents one more time to update mysqlport + s.registerAgents(c) + + // update targetAgent with source agent replication pos\gtid + targetTestAgent.agentSeedMetadata = sourceTestAgent.agentSeedMetadata + + // ProcessSeeds. Target agent had completed Restore stage, so Orchestrator will move Seed to ConnectSlave stage with status Scheduled. And we will have one SeedStageState record with stage Restore and status Completed + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, ConnectSlave, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. Orchestrator will ask agents to start ConnectSlave stage. So it's state would change from Scheduled to Running + ProcessSeeds() + targetTestAgent.agentSeedStageStatus.Stage = ConnectSlave + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Running + targetTestAgent.agentSeedStageStatus.Details = "running connectSlave stage" + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, ConnectSlave, 0) + + // targetAgent completed ConnectSlave stage + targetTestAgent.agentSeedStageStatus.Status = Completed + targetTestAgent.agentSeedStageStatus.Details = "completed connectSlave stage" + + // Read seed one more time. It's state should be cleanup and status Scheduled and we should have one SeedStageState record with stage ConnectSlave and status Completed + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Cleanup, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. Orchestrator will ask agents to start Cleanup stage. So it's state would change from Scheduled to Running + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Cleanup, 0) + + // simulate that cleanup stage is running on both agents + for _, agent := range []*testAgent{targetTestAgent, sourceTestAgent} { + agent.agentSeedStageStatus.Stage = Cleanup + agent.agentSeedStageStatus.Timestamp = time.Now() + agent.agentSeedStageStatus.Status = Running + agent.agentSeedStageStatus.Details = "processing cleanup stage" + } + + // check that SeedStageStates in Orchestator DB are the same, that were read from agents + s.readSeedStageStates(c, seed, 2, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. Both agents are still running Cleanup Stage, so seed state won't change and we will have 2 SeedStageState record from each agent + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Cleanup, 0) + s.readSeedStageStates(c, seed, 2, targetTestAgent, sourceTestAgent) + + // now simulate that source agent completed Cleanup stage + sourceTestAgent.agentSeedStageStatus.Status = Completed + sourceTestAgent.agentSeedStageStatus.Timestamp = time.Now() + sourceTestAgent.agentSeedStageStatus.Details = "completed cleanup stage" + + // ProcessSeeds. We have stage: Cleanup and status: Running as only source agent completed Cleanup stage, but in SeedStageStates we will have one Completed record and one Running + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Cleanup, 0) + s.readSeedStageStates(c, seed, 2, targetTestAgent, sourceTestAgent) + + // now simulate that target agent completed Prepare stage + targetTestAgent.agentSeedStageStatus.Status = Completed + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Details = "completed cleanup stage" + + // ProcessSeeds. This is the last stage, so we will have seed in completed Status. + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Completed, Cleanup, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) +} + +func (s *AgentTestSuite) TestProcessSeedsErrorOnPrepare(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + config.Config.MaxRetriesForSeedStage = 2 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + ProcessSeeds() + + // One agent is OK, another had error + for _, agent := range []*testAgent{targetTestAgent, sourceTestAgent} { + agent.agentSeedStageStatus.SeedID = seedID + agent.agentSeedStageStatus.Stage = Prepare + agent.agentSeedStageStatus.Hostname = agent.agent.Info.Hostname + agent.agentSeedStageStatus.Timestamp = time.Now() + } + targetTestAgent.agentSeedStageStatus.Status = Running + targetTestAgent.agentSeedStageStatus.Details = "processing prepare stage" + sourceTestAgent.agentSeedStageStatus.Status = Error + sourceTestAgent.agentSeedStageStatus.Details = "error processing prepare stage" + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Error, Prepare, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. After error our seed will be moved back to Prepare stage Scheduled status and retries increased + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 1) + + // so our SeedStages should also be in Scheduled status + sourceTestAgent.agentSeedStageStatus.Status = Scheduled + targetTestAgent.agentSeedStageStatus.Status = Scheduled + s.readSeedStageStates(c, seed, 2, targetTestAgent, sourceTestAgent) +} + +func (s *AgentTestSuite) TestProcessSeedsErrorOnBackup(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + config.Config.MaxRetriesForSeedStage = 2 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // change seed status to Scheduled and stage to Backup + seed.Status = Scheduled + seed.Stage = Backup + seed.updateSeedData() + ProcessSeeds() + + // As Mydumper is executed on target, set in status to Error + targetTestAgent.agentSeedStageStatus.SeedID = seedID + targetTestAgent.agentSeedStageStatus.Stage = Backup + targetTestAgent.agentSeedStageStatus.Hostname = targetTestAgent.agent.Info.Hostname + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Error + targetTestAgent.agentSeedStageStatus.Details = "error processing backup stage" + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Error, Backup, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. After error our seed will be moved back to Prepare stage Scheduled status and retries increased (due to backup stage is always moved to scheduled) + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 1) + + // so our SeedStage should also be in Scheduled status + targetTestAgent.agentSeedStageStatus.Status = Scheduled + targetTestAgent.agentSeedStageStatus.Stage = Prepare + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) +} + +func (s *AgentTestSuite) TestProcessSeedsErrorOnRestore(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + config.Config.MaxRetriesForSeedStage = 2 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // change seed status to Scheduled and stage to Restore + seed.Status = Scheduled + seed.Stage = Restore + seed.updateSeedData() + ProcessSeeds() + + // As Restore is always executed on target, set in status to Error + targetTestAgent.agentSeedStageStatus.SeedID = seedID + targetTestAgent.agentSeedStageStatus.Stage = Restore + targetTestAgent.agentSeedStageStatus.Hostname = targetTestAgent.agent.Info.Hostname + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Error + targetTestAgent.agentSeedStageStatus.Details = "error processing restore stage" + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Error, Restore, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. After error our seed will be moved back to Restore stage Scheduled status and retries increased (due to backup stage is always moved to scheduled) + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Restore, 1) + + // so our SeedStage should also be in Scheduled status + targetTestAgent.agentSeedStageStatus.Status = Scheduled + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) +} + +func (s *AgentTestSuite) TestProcessSeedsErrorOnCleanup(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + config.Config.MaxRetriesForSeedStage = 2 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // change seed status to Running and stage to Cleanup + seed.Status = Running + seed.Stage = Cleanup + seed.updateSeedData() + + // One agent is OK, another had error + for _, agent := range []*testAgent{targetTestAgent, sourceTestAgent} { + agent.agentSeedStageStatus.SeedID = seedID + agent.agentSeedStageStatus.Stage = Cleanup + agent.agentSeedStageStatus.Hostname = agent.agent.Info.Hostname + agent.agentSeedStageStatus.Timestamp = time.Now() + } + targetTestAgent.agentSeedStageStatus.Status = Running + targetTestAgent.agentSeedStageStatus.Details = "processing cleanup stage" + sourceTestAgent.agentSeedStageStatus.Status = Error + sourceTestAgent.agentSeedStageStatus.Details = "error processing cleanup stage" + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Error, Cleanup, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. After error our seed will be moved back to Cleanup stage Scheduled status and retries increased + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Cleanup, 1) + + // so our SeedStages should also be in Scheduled status + sourceTestAgent.agentSeedStageStatus.Status = Scheduled + targetTestAgent.agentSeedStageStatus.Status = Scheduled + s.readSeedStageStates(c, seed, 2, targetTestAgent, sourceTestAgent) +} + +func (s *AgentTestSuite) TestProcessSeedsErrorOnConnectSlave(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + config.Config.MaxRetriesForSeedStage = 2 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // change seed status to Running and stage to ConnectSlave + seed.Status = Running + seed.Stage = ConnectSlave + seed.updateSeedData() + + targetTestAgent.agentSeedStageStatus.SeedID = seedID + targetTestAgent.agentSeedStageStatus.Stage = ConnectSlave + targetTestAgent.agentSeedStageStatus.Hostname = targetTestAgent.agent.Info.Hostname + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Error + targetTestAgent.agentSeedStageStatus.Details = "error processing ConnectSlave stage" + + // As ConnectSlave requires agent metadata, set it + targetTestAgent.agentSeedMetadata = &SeedMetadata{ + LogFile: "", + LogPos: 10, + GtidExecuted: "", + } + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Error, ConnectSlave, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. After error our seed will be moved back to ConnectSlave stage + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, ConnectSlave, 1) + + // so our SeedStage should also be in Scheduled status + targetTestAgent.agentSeedStageStatus.Status = Scheduled + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) +} + +func (s *AgentTestSuite) TestProcessSeedsFailed(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // MaxRetriesForStage set to 0, so it will be failed after first error + config.Config.MaxRetriesForSeedStage = 0 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // change seed status to Scheduled and stage to Restore + seed.Status = Scheduled + seed.Stage = Restore + seed.updateSeedData() + ProcessSeeds() + + // As Restore is always executed on target, set in status to Error + targetTestAgent.agentSeedStageStatus.SeedID = seedID + targetTestAgent.agentSeedStageStatus.Stage = Restore + targetTestAgent.agentSeedStageStatus.Hostname = targetTestAgent.agent.Info.Hostname + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Error + targetTestAgent.agentSeedStageStatus.Details = "error processing restore stage" + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Error, Restore, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. After error our seed will be Failed + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Failed, Restore, 0) + + // so our SeedStage should also be in Scheduled status + targetTestAgent.agentSeedStageStatus.Status = Failed + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + activeSeeds, err := ReadActiveSeeds() + c.Assert(err, IsNil) + c.Assert(activeSeeds, HasLen, 0) +} + +func (s *AgentTestSuite) TestAbortSeedScheduled(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + err = seed.AbortSeed() + c.Assert(err, IsNil) + + activeSeeds, err := ReadActiveSeeds() + c.Assert(err, IsNil) + c.Assert(activeSeeds, HasLen, 0) +} + +func (s *AgentTestSuite) TestAbortSeedCompleted(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // MaxRetriesForStage set to 0, so it will be failed after first error + config.Config.MaxRetriesForSeedStage = 0 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // change seed status to Completed and stage to Cleanup + seed.Status = Completed + seed.Stage = Cleanup + seed.updateSeedData() + + err = seed.AbortSeed() + c.Assert(err, NotNil) +} + +func (s *AgentTestSuite) TestAbortSeedRestore(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // MaxRetriesForStage set to 0, so it will be failed after first error + config.Config.MaxRetriesForSeedStage = 0 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // change seed status to Scheduled and stage to Restore + seed.Status = Scheduled + seed.Stage = Restore + seed.updateSeedData() + ProcessSeeds() + + // As Restore is always executed on target, set in status to Error + targetTestAgent.agentSeedStageStatus.SeedID = seedID + targetTestAgent.agentSeedStageStatus.Stage = Restore + targetTestAgent.agentSeedStageStatus.Hostname = targetTestAgent.agent.Info.Hostname + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Running + targetTestAgent.agentSeedStageStatus.Details = "processing restore stage" + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Restore, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + err = seed.AbortSeed() + c.Assert(err, IsNil) + + activeSeeds, err := ReadActiveSeeds() + c.Assert(err, IsNil) + c.Assert(activeSeeds, HasLen, 0) +} + +func (s *AgentTestSuite) TestProcessAborting(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + // MaxRetriesForStage set to 0, so it will be failed after first error + config.Config.MaxRetriesForSeedStage = 0 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // change seed status to Aborting and stage to Restore + seed.Status = Aborting + seed.Stage = Restore + seed.updateSeedData() + ProcessSeeds() + + activeSeeds, err := ReadActiveSeeds() + c.Assert(err, IsNil) + c.Assert(activeSeeds, HasLen, 0) +} + +func (s *AgentTestSuite) TestAbortSeedCleanup(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + config.Config.MaxRetriesForSeedStage = 2 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + + // change seed status to Running and stage to Cleanup + seed.Status = Running + seed.Stage = Cleanup + seed.updateSeedData() + + // One agent is OK, another had error + for _, agent := range []*testAgent{targetTestAgent, sourceTestAgent} { + agent.agentSeedStageStatus.SeedID = seedID + agent.agentSeedStageStatus.Stage = Cleanup + agent.agentSeedStageStatus.Hostname = agent.agent.Info.Hostname + agent.agentSeedStageStatus.Timestamp = time.Now() + agent.agentSeedStageStatus.Status = Running + agent.agentSeedStageStatus.Details = "processing cleanup stage" + } + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Cleanup, 0) + s.readSeedStageStates(c, seed, 2, targetTestAgent, sourceTestAgent) + + err = seed.AbortSeed() + c.Assert(err, IsNil) + + activeSeeds, err := ReadActiveSeeds() + c.Assert(err, IsNil) + c.Assert(activeSeeds, HasLen, 0) +} + +func (s *AgentTestSuite) TestProcessSeedsStaleBackup(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + config.Config.MaxRetriesForSeedStage = 2 + config.Config.SeedBackupStaleFailMinutes = 1 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + ProcessSeeds() + + // change seed status to Scheduled and stage to Backup + seed.Status = Scheduled + seed.Stage = Backup + seed.updateSeedData() + ProcessSeeds() + + // As Mydumper is executed on target, set that backup stage is running on targetAgent + targetTestAgent.agentSeedStageStatus.SeedID = seedID + targetTestAgent.agentSeedStageStatus.Stage = Backup + targetTestAgent.agentSeedStageStatus.Hostname = targetTestAgent.agent.Info.Hostname + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Running + targetTestAgent.agentSeedStageStatus.Details = "processing backup stage" + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Backup, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // simulate that seed is in process and BackupDirDiskUsed is increasing on targetAgents + targetTestAgent.agent.Data.BackupDirDiskUsed = 10 + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Backup, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // simulate that seed is in process and BackupDirDiskUsed is not increasing on targetAgents. Increase timestamp to simulate stale + targetTestAgent.agent.Data.BackupDirDiskUsed = 10 + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now().Add(5 * time.Minute) + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Error, Backup, 0) + targetTestAgent.agentSeedStageStatus.Status = Error + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. After error our seed will be moved back to Prepare stage Scheduled status and retries increased (due to backup stage is always moved to scheduled) + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 1) + + // so our SeedStage should also be in Scheduled status + targetTestAgent.agentSeedStageStatus.Status = Scheduled + targetTestAgent.agentSeedStageStatus.Stage = Prepare + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) +} + +func (s *AgentTestSuite) TestProcessSeedsStaleRestore(c *C) { + targetTestAgent := s.testAgents["agent1"] + sourceTestAgent := s.testAgents["agent2"] + + config.Config.MaxRetriesForSeedStage = 2 + config.Config.SeedBackupStaleFailMinutes = 1 + + // register agents to Orchestrator + s.registerAgents(c) + + targetAgent, sourceAgent := s.getSeedAgents(c, targetTestAgent, sourceTestAgent) + + seedID, err := NewSeed("Mydumper", targetAgent, sourceAgent) + c.Assert(err, IsNil) + c.Assert(seedID, Equals, int64(1)) + + // Orchestrator registered seed. It's will have first stage - Prepare and status Scheduled + seed := s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Prepare, 0) + ProcessSeeds() + + // change seed status to Scheduled and stage to Restore + seed.Status = Scheduled + seed.Stage = Restore + seed.updateSeedData() + ProcessSeeds() + + // Restore stage is running on targetAgent + targetTestAgent.agentSeedStageStatus.SeedID = seedID + targetTestAgent.agentSeedStageStatus.Stage = Restore + targetTestAgent.agentSeedStageStatus.Hostname = targetTestAgent.agent.Info.Hostname + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now() + targetTestAgent.agentSeedStageStatus.Status = Running + targetTestAgent.agentSeedStageStatus.Details = "processing restore stage" + + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Restore, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // simulate that seed is in process and MySQLDatadirDiskUsed is increasing on targetAgents + targetTestAgent.agent.Data.MySQLDatadirDiskUsed = 10 + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Running, Restore, 0) + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // simulate that seed is in process and MySQLDatadirDiskUsed is not increasing on targetAgents. Increase timestamp to simulate stale + targetTestAgent.agent.Data.MySQLDatadirDiskUsed = 10 + targetTestAgent.agentSeedStageStatus.Timestamp = time.Now().Add(5 * time.Minute) + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Error, Restore, 0) + targetTestAgent.agentSeedStageStatus.Status = Error + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) + + // ProcessSeeds. After error our seed will be moved back to Restore stage Scheduled status and retries increased (due to backup stage is always moved to scheduled) + ProcessSeeds() + seed = s.readSeed(c, seedID, targetAgent.Info.Hostname, sourceAgent.Info.Hostname, Mydumper, Target, Scheduled, Restore, 1) + + // so our SeedStage should also be in Scheduled status + targetTestAgent.agentSeedStageStatus.Status = Scheduled + targetTestAgent.agentSeedStageStatus.Stage = Restore + s.readSeedStageStates(c, seed, 1, targetTestAgent, sourceTestAgent) +} diff --git a/go/agent/instance_topology_agent.go b/go/agent/instance_topology_agent.go index d23167651..4cb1c4e7d 100644 --- a/go/agent/instance_topology_agent.go +++ b/go/agent/instance_topology_agent.go @@ -41,6 +41,16 @@ func SyncReplicaRelayLogs(instance, otherInstance *inst.Instance) (*inst.Instanc return instance, log.Errorf("SyncReplicaRelayLogs: replication on %+v must not run", otherInstance.Key) } + agent, err := ReadAgent(instance.Key.Hostname) + if err != nil { + return instance, log.Errore(err) + } + + otherAgent, err := ReadAgent(otherInstance.Key.Hostname) + if err != nil { + return otherInstance, log.Errore(err) + } + log.Debugf("SyncReplicaRelayLogs: correlating coordinates of %+v on %+v", instance.Key, otherInstance.Key) _, _, nextCoordinates, found, err = inst.CorrelateRelaylogCoordinates(instance, nil, otherInstance) if err != nil { @@ -52,12 +62,13 @@ func SyncReplicaRelayLogs(instance, otherInstance *inst.Instance) (*inst.Instanc log.Debugf("SyncReplicaRelayLogs: correlated next-coordinates are %+v", *nextCoordinates) InitHttpClient() - if _, err := RelaylogContentsTail(otherInstance.Key.Hostname, nextCoordinates, &onResponse); err != nil { + + if err := otherAgent.relaylogContentsTail(nextCoordinates, &onResponse); err != nil { goto Cleanup } log.Debugf("SyncReplicaRelayLogs: got content (%d bytes)", len(content)) - if _, err := ApplyRelaylogContents(instance.Key.Hostname, content); err != nil { + if err := agent.applyRelaylogContents(content); err != nil { goto Cleanup } log.Debugf("SyncReplicaRelayLogs: applied content (%d bytes)", len(content)) diff --git a/go/agent/seed.go b/go/agent/seed.go new file mode 100644 index 000000000..5553be1ce --- /dev/null +++ b/go/agent/seed.go @@ -0,0 +1,1137 @@ +package agent + +import ( + "bytes" + "encoding/json" + "fmt" + "sync" + "time" + + "github.com/github/orchestrator/go/config" + "github.com/github/orchestrator/go/inst" + "github.com/openark/golib/log" + "github.com/openark/golib/sqlutils" +) + +var SeededAgents chan *Agent = make(chan *Agent) + +type seedProgress struct { + bytesCopied int64 + databasesSize int64 + updatedAt time.Time +} + +var seedOperationProgress = make(map[int64]*seedProgress) + +type SeedSide int + +const ( + Target SeedSide = iota + Source +) + +var toSeedSide = map[string]SeedSide{ + "Target": Target, + "Source": Source, +} + +func (s SeedSide) String() string { + return [...]string{"Target", "Source"}[s] +} + +func (s SeedSide) MarshalJSON() ([]byte, error) { + buffer := bytes.NewBufferString(`"`) + buffer.WriteString(s.String()) + buffer.WriteString(`"`) + return buffer.Bytes(), nil +} + +func (s *SeedSide) UnmarshalJSON(b []byte) error { + var value string + err := json.Unmarshal(b, &value) + if err != nil { + return err + } + *s = toSeedSide[value] + return nil +} + +type Engine int + +const ( + ROCKSDB Engine = iota + MRG_MYISAM + CSV + BLACKHOLE + InnoDB + MEMORY + ARCHIVE + MyISAM + FEDERATED + TokuDB +) + +func (e Engine) String() string { + return [...]string{"ROCKSDB", "MRG_MYISAM", "CSV", "BLACKHOLE", "InnoDB", "MEMORY", "ARCHIVE", "MyISAM", "FEDERATED", "TokuDB"}[e] +} + +// MarshalJSON marshals the enum as a quoted json string +func (e Engine) MarshalJSON() ([]byte, error) { + buffer := bytes.NewBufferString(`"`) + buffer.WriteString(e.String()) + buffer.WriteString(`"`) + return buffer.Bytes(), nil +} + +var toEngine = map[string]Engine{ + "ROCKSDB": ROCKSDB, + "MRG_MYISAM": MRG_MYISAM, + "CSV": CSV, + "BLACKHOLE": BLACKHOLE, + "InnoDB": InnoDB, + "MEMORY": MEMORY, + "ARCHIVE": ARCHIVE, + "MyISAM": MyISAM, + "FEDERATED": FEDERATED, + "TokuDB": TokuDB, +} + +func (e *Engine) UnmarshalJSON(b []byte) error { + var value string + err := json.Unmarshal(b, &value) + if err != nil { + return err + } + *e = toEngine[value] + return nil +} + +type SeedMethod int + +const ( + ClonePlugin SeedMethod = iota + LVM + Mydumper + Mysqldump + Xtrabackup +) + +var toSeedMethod = map[string]SeedMethod{ + "ClonePlugin": ClonePlugin, + "LVM": LVM, + "Mydumper": Mydumper, + "Mysqldump": Mysqldump, + "Xtrabackup": Xtrabackup, +} + +func (m SeedMethod) String() string { + return [...]string{"ClonePlugin", "LVM", "Mydumper", "Mysqldump", "Xtrabackup"}[m] +} + +func (m *SeedMethod) UnmarshalText(b []byte) error { + *m = toSeedMethod[string(b)] + return nil +} + +func (m SeedMethod) MarshalText() ([]byte, error) { + return []byte(m.String()), nil +} + +type SeedStatus int + +const ( + Scheduled SeedStatus = iota + Running + Completed + Error + Failed + Aborting +) + +var SeedStatuses = []SeedStatus{Scheduled, Running, Completed, Error, Failed, Aborting} + +func (s SeedStatus) String() string { + return [...]string{"Scheduled", "Running", "Completed", "Error", "Failed", "Aborting"}[s] +} + +func (s SeedStatus) MarshalJSON() ([]byte, error) { + buffer := bytes.NewBufferString(`"`) + buffer.WriteString(s.String()) + buffer.WriteString(`"`) + return buffer.Bytes(), nil +} + +func (s *SeedStatus) UnmarshalJSON(b []byte) error { + var value string + err := json.Unmarshal(b, &value) + if err != nil { + return err + } + *s = ToSeedStatus[value] + return nil +} + +var ToSeedStatus = map[string]SeedStatus{ + "Scheduled": Scheduled, + "Running": Running, + "Completed": Completed, + "Error": Error, + "Failed": Failed, + "Aborting": Aborting, +} + +type SeedStage int + +const ( + Prepare SeedStage = iota + Backup + Restore + ConnectSlave + Cleanup +) + +func (s SeedStage) String() string { + return [...]string{"Prepare", "Backup", "Restore", "ConnectSlave", "Cleanup"}[s] +} + +func (s SeedStage) MarshalJSON() ([]byte, error) { + buffer := bytes.NewBufferString(`"`) + buffer.WriteString(s.String()) + buffer.WriteString(`"`) + return buffer.Bytes(), nil +} + +func (s *SeedStage) UnmarshalJSON(b []byte) error { + var value string + err := json.Unmarshal(b, &value) + if err != nil { + return err + } + *s = toSeedStage[value] + return nil +} + +var toSeedStage = map[string]SeedStage{ + "Prepare": Prepare, + "Backup": Backup, + "Restore": Restore, + "ConnectSlave": ConnectSlave, + "Cleanup": Cleanup, +} + +// Seed makes for the high level data & state of a seed operation +type Seed struct { + SeedID int64 + TargetHostname string + SourceHostname string + SeedMethod SeedMethod + BackupSide SeedSide + Status SeedStatus + Stage SeedStage + Retries int + StartTimestamp time.Time + EndTimestamp time.Time + UpdatedAt time.Time +} + +// SeedStageState represents a single stage in a seed operation +type SeedStageState struct { + SeedStageID int64 + SeedID int64 + Stage SeedStage + Hostname string + Timestamp time.Time + Status SeedStatus + Details string +} + +// SeedStageState represents a metadata information for seed with binlog coordinates or GtidExecuted +type SeedMetadata struct { + LogFile string + LogPos int64 + GtidExecuted string +} + +func byteCount(b int64) string { + const unit = 1000 + if b < unit { + return fmt.Sprintf("%d B", b) + } + div, exp := int64(unit), 0 + for n := b / unit; n >= unit; n /= unit { + div *= unit + exp++ + } + return fmt.Sprintf("%.1f %cB", + float64(b)/float64(div), "kMGTPE"[exp]) +} + +func auditSeedOperation(agent *Agent, message string) error { + instanceKey := &inst.InstanceKey{} + if agent != nil { + instanceKey = &inst.InstanceKey{Hostname: agent.Info.Hostname, Port: agent.Info.MySQLPort} + } + return inst.AuditOperation("seed", instanceKey, message) +} + +// NewSeed is the entry point for making a seed. It's registers seed in db, so it can be later processed asynchronously +func NewSeed(seedMethodName string, targetAgent *Agent, sourceAgent *Agent) (int64, error) { + var seedMethod SeedMethod + var ok bool + if seedMethod, ok = toSeedMethod[seedMethodName]; !ok { + return 0, log.Errorf("SeedMethod %s not found", seedMethodName) + } + if err := targetAgent.getAgentData(); err != nil { + return 0, log.Errorf("Unable to get information from agent on target host %s", targetAgent.Info.Hostname) + } + if err := sourceAgent.getAgentData(); err != nil { + return 0, log.Errorf("Unable to get information from agent on source host %s", sourceAgent.Info.Hostname) + } + if targetAgent.Info.Hostname == sourceAgent.Info.Hostname { + return 0, log.Errorf("Cannot seed %s onto itself", targetAgent.Info.Hostname) + } + if _, ok = targetAgent.Data.AvailiableSeedMethods[seedMethod]; !ok { + return 0, log.Errorf("Seed method %s not supported on target host %s", seedMethod.String(), targetAgent.Info.Hostname) + } + if _, ok = sourceAgent.Data.AvailiableSeedMethods[seedMethod]; !ok { + return 0, log.Errorf("Seed method %s not supported on source host %s", seedMethod.String(), sourceAgent.Info.Hostname) + } + for db, opts := range sourceAgent.Data.MySQLDatabases { + if !enginesSupported(sourceAgent.Data.AvailiableSeedMethods[seedMethod].SupportedEngines, opts.Engines) { + return 0, log.Errorf("Database %s had a table with engine unsupported by seed method %s", db, seedMethod.String()) + } + } + targetInstance, err := inst.ReadTopologyInstance(&inst.InstanceKey{Hostname: targetAgent.Info.Hostname, Port: targetAgent.Info.MySQLPort}) + if err != nil { + return 0, log.Errorf("Unable to read target agent instance %s", targetAgent.Info.Hostname) + } + sourceInstance, err := inst.ReadTopologyInstance(&inst.InstanceKey{Hostname: sourceAgent.Info.Hostname, Port: sourceAgent.Info.MySQLPort}) + if err != nil { + return 0, log.Errorf("Unable to read source agent instance %s", targetAgent.Info.Hostname) + } + if targetInstance.IsSmallerMajorVersion(sourceInstance) { + return 0, log.Errorf("MySQL version on target agent is smaller that on source. Source agent MySQL version: %s, target agent MySQL version: %s", sourceInstance.Version, targetInstance.Version) + } + if !sourceInstance.LogBinEnabled { + return 0, log.Errorf("Binlog is not enabled on MySQL source agent host %s", sourceAgent.Info.Hostname) + } + if sourceInstance.IsReplica() { + if !sourceInstance.LogSlaveUpdatesEnabled { + return 0, log.Errorf("log-slave-updates is not enabled on MySQL source agent host %s, but the host is a replica", sourceAgent.Info.Hostname) + } + } + if targetInstance.IsReplica() { + return 0, log.Errorf("Cannot seed on target agent host %s, because it is replica. Please reset slave before starting seed", targetAgent.Info.Hostname) + } + if len(targetInstance.SlaveHosts) > 0 { + return 0, log.Errorf("Cannot seed on target agent host %s, because it has slave attached. Please disconnect all slaves before seed", targetAgent.Info.Hostname) + } + activeSourceSeeds, _ := ReadActiveSeedsForAgent(sourceAgent) + activeTargetSeeds, _ := ReadActiveSeedsForAgent(targetAgent) + if len(activeSourceSeeds) > 0 { + return 0, log.Errorf("Source agent has active seeds") + } + if len(activeTargetSeeds) > 0 { + return 0, log.Errorf("Target agent has active seeds") + } + if sourceAgent.Data.MySQLDatadirDiskUsed > targetAgent.Data.MySQLDatadirDiskFree { + return 0, log.Errorf("Not enough disk space on target host %s. Required: %d, available: %d", targetAgent.Info.Hostname, sourceAgent.Data.MySQLDatadirDiskUsed, targetAgent.Data.MySQLDatadirDiskFree) + } + // do we need to check this? logical backup usually less than actual data size, so let's use 0.6 multiplier + if !sourceAgent.Data.AvailiableSeedMethods[seedMethod].BackupToDatadir { + if int64(float64(sourceAgent.Data.MySQLDatadirDiskUsed)*0.6) > targetAgent.Data.BackupDirDiskFree { + return 0, log.Errorf("Not enough disk space on target host backup directory %s. Database size: %d, available: %d", targetAgent.Info.Hostname, sourceAgent.Data.MySQLDatadirDiskUsed, targetAgent.Data.BackupDirDiskFree) + } + } + // special checks for LVM seed method + if seedMethod == LVM { + if sourceAgent.Data.MountPoint.IsMounted { + return 0, log.Errorf("Volume already mounted on source host %s. Please unmount", sourceAgent.Info.Hostname) + } + if len(sourceAgent.Data.LogicalVolumes) == 0 { + return 0, log.Errorf("No logical volumes found on source host %s", sourceAgent.Info.Hostname) + } + } + seed := &Seed{ + TargetHostname: targetAgent.Info.Hostname, + SourceHostname: sourceAgent.Info.Hostname, + SeedMethod: seedMethod, + BackupSide: sourceAgent.Data.AvailiableSeedMethods[seedMethod].BackupSide, + Status: Scheduled, + Stage: Prepare, + Retries: 0, + StartTimestamp: time.Now(), + UpdatedAt: time.Now(), + } + if err := seed.registerSeedEntry(); err != nil { + return 0, log.Errore(err) + } + SeededAgents <- targetAgent + + auditSeedOperation(nil, fmt.Sprintf("Registered new seed. SeedID: %d, source host: %s, target host: %s, seed method: %s", seed.SeedID, seed.SourceHostname, seed.TargetHostname, seed.SeedMethod.String())) + return seed.SeedID, nil +} + +func enginesSupported(first, second []Engine) bool { + hashmap := make(map[Engine]int) + for _, value := range first { + hashmap[value]++ + } + + for _, value := range second { + if count, found := hashmap[value]; !found { + return false + } else if count < 1 { + return false + } else { + hashmap[value] = count - 1 + } + } + return true +} + +// ReadSeed returns an information about seed from database +func ReadSeed(seedID int64) (*Seed, error) { + whereCondition := ` + where + agent_seed_id = ? + ` + res, err := readSeeds(whereCondition, sqlutils.Args(seedID), "") + if err != nil { + return nil, err + } + if len(res) == 0 { + return nil, fmt.Errorf("Seed %d not found", seedID) + } + return res[0], nil +} + +// ReadActiveSeeds reads active seeds (where status != Completed or Failed) +func ReadActiveSeeds() ([]*Seed, error) { + whereCondition := ` + where + status not in (?, ?) + ` + return readSeeds(whereCondition, sqlutils.Args(Completed.String(), Failed.String()), "") +} + +// ReadSeeds reads seeds from backend table. +func ReadSeeds() ([]*Seed, error) { + return readSeeds(``, sqlutils.Args(), "") +} + +// ReadSeedsPaged returns a list of seeds with pagination +func ReadSeedsPaged(page int) ([]*Seed, error) { + limit := fmt.Sprintf("limit %d offset %d", config.AuditPageSize, page*config.AuditPageSize) + return readSeeds(``, sqlutils.Args(), limit) +} + +// ReadSeedsForTargetAgentPaged returns a list of seeds for specified targetagent with pagination +func ReadSeedsForTargetAgentPaged(targetAgent *Agent, page int) ([]*Seed, error) { + whereCondition := ` + where + target_hostname = ? + ` + limit := fmt.Sprintf("limit %d offset %d", config.AuditPageSize, page*config.AuditPageSize) + return readSeeds(whereCondition, sqlutils.Args(targetAgent.Info.Hostname), limit) +} + +// ReadSeedsForSourceAgentPaged returns a list of seeds for specified sourceagent with pagination +func ReadSeedsForSourceAgentPaged(sourceAgent *Agent, page int) ([]*Seed, error) { + whereCondition := ` + where + source_hostname = ? + ` + limit := fmt.Sprintf("limit %d offset %d", config.AuditPageSize, page*config.AuditPageSize) + return readSeeds(whereCondition, sqlutils.Args(sourceAgent.Info.Hostname), limit) +} + +// ReadSeedsInStatusPaged returns seeds in specified status with pagination +func ReadSeedsInStatusPaged(status SeedStatus, page int) ([]*Seed, error) { + limit := fmt.Sprintf("limit %d offset %d", config.AuditPageSize, page*config.AuditPageSize) + whereCondition := ` + where + status = ? + ` + return readSeeds(whereCondition, sqlutils.Args(status.String()), limit) +} + +// ReadSeedsInStatus returns seeds in specified status +func ReadSeedsInStatus(status SeedStatus) ([]*Seed, error) { + whereCondition := ` + where + status = ? + ` + return readSeeds(whereCondition, sqlutils.Args(status.String()), ``) +} + +// ReadSeedsForAgent reads seeds where agent participates either as source or target in given status +func ReadSeedsForAgent(agent *Agent, limit string) ([]*Seed, error) { + whereCondition := ` + where + target_hostname = ? + or source_hostname = ? + + ` + return readSeeds(whereCondition, sqlutils.Args(agent.Info.Hostname, agent.Info.Hostname), limit) +} + +// ReadActiveSeedsForAgent reads active seeds where agent participates either as source or target +func ReadActiveSeedsForAgent(agent *Agent) ([]*Seed, error) { + whereCondition := ` + where + status not in (?, ?) + and ( + target_hostname = ? + or source_hostname = ? + ) + ` + return readSeeds(whereCondition, sqlutils.Args(Completed.String(), Failed.String(), agent.Info.Hostname, agent.Info.Hostname), "") +} + +// ProcessSeeds is the main function in seed process +func ProcessSeeds() []*Seed { + activeSeeds, err := ReadActiveSeeds() + if err != nil { + log.Errore(fmt.Errorf("Unable to read active seeds: %+v", err)) + return nil + } + if len(activeSeeds) > 0 { + inst.AuditOperation("process-seeds", nil, fmt.Sprintf("Will process %d active seeds", len(activeSeeds))) + var wg sync.WaitGroup + for _, seed := range activeSeeds { + wg.Add(1) + switch seed.Status { + case Scheduled: + go seed.processScheduled(&wg) + case Running: + go seed.processRunning(&wg) + case Error: + go seed.processErrored(&wg) + case Aborting: + go seed.processAborting(&wg) + } + } + wg.Wait() + inst.AuditOperation("process-seeds", nil, "All active seeds processed") + } + return activeSeeds +} + +// readSeadAgents returns target and source agents associated with seed from db. If updateAgentData is specified, it will use agent API in order to update agent data +func (s *Seed) readSeadAgents(updateAgentsData bool) (targetAgent *Agent, sourceAgent *Agent, err error) { + targetAgent, err = ReadAgentInfo(s.TargetHostname) + if err != nil { + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Unable to read target agent info: %+v", err)) + return nil, nil, log.Errore(err) + } + if updateAgentsData { + if err = targetAgent.getAgentData(); err != nil { + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Unable to update agent data: %+v", err)) + return nil, nil, log.Errore(err) + } + } + sourceAgent, err = ReadAgentInfo(s.SourceHostname) + if err != nil { + s.Status = Error + s.updateSeed(sourceAgent, fmt.Sprintf("Unable to read source agent info: %+v", err)) + return nil, nil, log.Errore(err) + } + if updateAgentsData { + if err = sourceAgent.getAgentData(); err != nil { + s.Status = Error + s.updateSeed(sourceAgent, fmt.Sprintf("Unable to update agent data: %+v", err)) + return nil, nil, log.Errore(err) + } + } + return targetAgent, sourceAgent, nil +} + +func (s *Seed) processAborting(wg *sync.WaitGroup) { + defer wg.Done() + s.AbortSeed() +} + +func (s *Seed) processScheduled(wg *sync.WaitGroup) { + defer wg.Done() + switch s.Stage { + case Prepare: + targetAgent, sourceAgent, err := s.readSeadAgents(false) + if err != nil { + return + } + targetInstanceKey := &inst.InstanceKey{Hostname: targetAgent.Info.Hostname, Port: targetAgent.Info.MySQLPort} + downtimeDuration := time.Duration(config.SeedDowntimeHours) * time.Hour + downtime := inst.NewDowntime(targetInstanceKey, "orchestrator-agent", "seed-in-process", downtimeDuration) + if err = inst.BeginDowntime(downtime); err != nil { + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error starting downtime on targetAgent: %+v", err)) + return + } + agents := make(map[*Agent]SeedSide) + agents[targetAgent] = Target + agents[sourceAgent] = Source + for agent, seedSide := range agents { + if err := agent.prepare(s.SeedID, s.SeedMethod, seedSide); err != nil { + s.Status = Error + s.updateSeed(agent, fmt.Sprintf("Error calling prepare API on agent: %+v", err)) + return + } + s.Status = Running + s.updateSeed(agent, "Stage started") + } + case Backup: + targetAgent, sourceAgent, err := s.readSeadAgents(false) + if err != nil { + return + } + agent := targetAgent + seedHost := sourceAgent.Info.Hostname + if s.BackupSide == Source { + agent = sourceAgent + seedHost = targetAgent.Info.Hostname + } + if err := agent.backup(s.SeedID, s.SeedMethod, seedHost, sourceAgent.Info.MySQLPort); err != nil { + s.Status = Error + s.updateSeed(agent, fmt.Sprintf("Error calling backup API on agent: %+v", err)) + return + } + s.Status = Running + s.updateSeed(agent, "Stage started") + if _, ok := seedOperationProgress[s.SeedID]; ok { + delete(seedOperationProgress, s.SeedID) + } + case Restore: + targetAgent, _, err := s.readSeadAgents(false) + if err != nil { + return + } + if err := targetAgent.restore(s.SeedID, s.SeedMethod); err != nil { + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error calling restore API on agent: %+v", err)) + return + } + s.Status = Running + s.updateSeed(targetAgent, "Stage started") + if _, ok := seedOperationProgress[s.SeedID]; ok { + delete(seedOperationProgress, s.SeedID) + } + case Cleanup: + targetAgent, sourceAgent, err := s.readSeadAgents(false) + if err != nil { + return + } + agents := make(map[*Agent]SeedSide) + agents[targetAgent] = Target + agents[sourceAgent] = Source + for agent, seedSide := range agents { + if err := agent.cleanup(s.SeedID, s.SeedMethod, seedSide); err != nil { + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error calling prepare API on agent: %+v", err)) + return + } + s.Status = Running + s.updateSeed(agent, "Stage started") + } + case ConnectSlave: + targetAgent, _, err := s.readSeadAgents(false) + if err != nil { + return + } + s.Status = Running + s.updateSeed(targetAgent, "Stage started") + } +} + +func (s *Seed) processRunning(wg *sync.WaitGroup) { + defer wg.Done() + switch s.Stage { + case Prepare, Cleanup: + targetAgent, sourceAgent, err := s.readSeadAgents(false) + if err != nil { + return + } + agents := make(map[*Agent]bool) + agents[targetAgent] = false + agents[sourceAgent] = false + for agent := range agents { + stageAlreadyCompleted, err := s.isSeedStageCompletedForAgent(agent) + if err != nil { + s.Status = Error + s.updateSeed(agent, fmt.Sprintf("Error getting information about completed stages from Orchestrator: %+v", err)) + return + } + if !stageAlreadyCompleted { + agentSeedStageState, err := agent.getSeedStageState(s.SeedID, s.Stage) + if err != nil { + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error getting seed stage state information from agent: %+v", err)) + return + } + submitSeedStageState(agentSeedStageState) + if agentSeedStageState.Status == Completed { + agents[agent] = true + auditSeedOperation(agent, fmt.Sprintf("SeedID: %d, Stage: %s, Status: %s, Retries: %d", s.SeedID, s.Stage.String(), Completed.String(), s.Retries)) + } + if agentSeedStageState.Status == Error { + s.Status = Error + s.updateSeed(agent, "") + return + } + } else { + agents[agent] = true + } + } + if agents[targetAgent] && agents[sourceAgent] { + if s.Stage == Prepare { + s.Stage = s.Stage + 1 + s.Status = Scheduled + s.updateSeed(nil, "") + } else { + for _, agent := range []*Agent{targetAgent, sourceAgent} { + if err := agent.postSeedCmd(s.SeedID); err != nil { + s.Status = Error + s.updateSeedState(agent.Info.Hostname, fmt.Sprintf("Failed to execute post seed command on agent: %+v", err)) + } else { + s.Status = Completed + s.updateSeedState(agent.Info.Hostname, "Executed post-seed command") + } + } + targetInstanceKey := &inst.InstanceKey{Hostname: targetAgent.Info.Hostname, Port: targetAgent.Info.MySQLPort} + if _, err = inst.EndDowntime(targetInstanceKey); err != nil { + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error ending downtime on targetAgent: %+v", err)) + return + } + s.Status = Completed + s.EndTimestamp = time.Now() + s.updateSeed(nil, "") + for _, agent := range []*Agent{targetAgent, sourceAgent} { + s.updateSeedState(agent.Info.Hostname, fmt.Sprintf("Seed completed")) + } + } + } + case Backup: + targetAgent, sourceAgent, err := s.readSeadAgents(true) + if err != nil { + return + } + agent := targetAgent + if s.BackupSide == Source { + agent = sourceAgent + } + agentSeedStageState, err := agent.getSeedStageState(s.SeedID, s.Stage) + if err != nil { + s.Status = Error + s.updateSeed(agent, fmt.Sprintf("Error getting seed stage state information from agent: %+v", err)) + return + } + // Create struct to store seed progress in case if they not exists + if _, ok := seedOperationProgress[s.SeedID]; !ok { + var databasesSize int64 + for _, dbProps := range sourceAgent.Data.MySQLDatabases { + databasesSize += dbProps.Size + } + seedOperationProgress[s.SeedID] = &seedProgress{ + bytesCopied: 0, + databasesSize: databasesSize, + updatedAt: time.Now(), + } + } + if agentSeedStageState.Status == Running { + // check that seed is not stale + bytesCopied := targetAgent.Data.BackupDirDiskUsed + if agent.Data.AvailiableSeedMethods[s.SeedMethod].BackupToDatadir == true { + bytesCopied = targetAgent.Data.MySQLDatadirDiskUsed + } + // if bytesCopied increased - just update info for this SeedID + if bytesCopied > seedOperationProgress[s.SeedID].bytesCopied { + seedOperationProgress[s.SeedID].bytesCopied = bytesCopied + seedOperationProgress[s.SeedID].updatedAt = agentSeedStageState.Timestamp + } else { + // else check diff between agentSeedStageState.Timestamp and seedOperationProgress[s.SeedID].updatedAt. If it is more than config.Config.StaleSeedFailMinutes - abort seed and mark it as failed + if agentSeedStageState.Timestamp.Sub(seedOperationProgress[s.SeedID].updatedAt).Minutes() >= float64(config.Config.SeedBackupStaleFailMinutes) { + if err := agent.AbortSeed(s.SeedID, s.Stage); err != nil { + s.Retries++ + s.updateSeed(agent, fmt.Sprintf("Error aborting stale seed on agent: %+v", err)) + return + } + s.Status = Error + s.updateSeed(agent, fmt.Sprintf("No data copied in SeedBackupStaleFailMinutes interval (%d minutes). Aborting seed", config.Config.SeedBackupStaleFailMinutes)) + return + } + } + agentSeedStageState.Details = fmt.Sprintf("Copied: %s. MySQL databases size: %s", byteCount(bytesCopied), byteCount(seedOperationProgress[s.SeedID].databasesSize)) + agentSeedStageState.Timestamp = time.Now() + submitSeedStageState(agentSeedStageState) + } + if agentSeedStageState.Status == Completed { + submitSeedStageState(agentSeedStageState) + auditSeedOperation(agent, fmt.Sprintf("SeedID: %d, Stage: %s, Status: %s, Retries: %d", s.SeedID, s.Stage.String(), Completed.String(), s.Retries)) + s.Stage = s.Stage + 1 + s.Status = Scheduled + s.Retries = 0 + s.updateSeed(agent, "") + delete(seedOperationProgress, s.SeedID) + } + if agentSeedStageState.Status == Error { + submitSeedStageState(agentSeedStageState) + s.Status = Error + s.updateSeed(agent, "") + delete(seedOperationProgress, s.SeedID) + } + case Restore: + targetAgent, sourceAgent, err := s.readSeadAgents(true) + if err != nil { + return + } + agentSeedStageState, err := targetAgent.getSeedStageState(s.SeedID, s.Stage) + if err != nil { + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error getting seed stage state information from agent: %+v", err)) + return + } + if agentSeedStageState.Status == Running { + if targetAgent.Data.AvailiableSeedMethods[s.SeedMethod].BackupToDatadir == false { + // Create struct to store seed progress in case if they not exists + if _, ok := seedOperationProgress[s.SeedID]; !ok { + var databasesSizeSource int64 + for _, dbPropsSource := range sourceAgent.Data.MySQLDatabases { + databasesSizeSource += dbPropsSource.Size + } + seedOperationProgress[s.SeedID] = &seedProgress{ + bytesCopied: 0, + databasesSize: databasesSizeSource, + updatedAt: time.Now(), + } + } + // check that seed is not stale + var databasesSize int64 + for _, dbProps := range targetAgent.Data.MySQLDatabases { + databasesSize += dbProps.Size + } + // if bytesCopied increased - just update info for this SeedID + if databasesSize > seedOperationProgress[s.SeedID].bytesCopied { + seedOperationProgress[s.SeedID].bytesCopied = databasesSize + seedOperationProgress[s.SeedID].updatedAt = agentSeedStageState.Timestamp + } else { + // else check diff between agentSeedStageState.Timestamp and seedBackupProgress[s.SeedID].updatedAt. If it is more than config.Config.StaleSeedFailMinutes - abort seed and mark it as failed + if agentSeedStageState.Timestamp.Sub(seedOperationProgress[s.SeedID].updatedAt).Minutes() >= float64(config.Config.SeedBackupStaleFailMinutes) { + if err := targetAgent.AbortSeed(s.SeedID, s.Stage); err != nil { + s.Retries++ + s.updateSeed(targetAgent, fmt.Sprintf("Error aborting stale seed on agent: %+v", err)) + return + } + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("No data restored in SeedBackupStaleFailMinutes interval (%d minutes). Aborting seed", config.Config.SeedBackupStaleFailMinutes)) + return + } + } + agentSeedStageState.Details = fmt.Sprintf("Restored: %s. MySQL databases size: %s", byteCount(databasesSize), byteCount(seedOperationProgress[s.SeedID].databasesSize)) + agentSeedStageState.Timestamp = time.Now() + } + submitSeedStageState(agentSeedStageState) + } + if agentSeedStageState.Status == Completed { + submitSeedStageState(agentSeedStageState) + auditSeedOperation(targetAgent, fmt.Sprintf("SeedID: %d, Stage: %s, Status: %s, Retries: %d", s.SeedID, s.Stage.String(), Completed.String(), s.Retries)) + s.Stage = s.Stage + 1 + s.Status = Scheduled + s.Retries = 0 + if targetAgent.Data.AvailiableSeedMethods[s.SeedMethod].BackupToDatadir == false { + delete(seedOperationProgress, s.SeedID) + } + s.updateSeed(targetAgent, "") + } + if agentSeedStageState.Status == Error { + if targetAgent.Data.AvailiableSeedMethods[s.SeedMethod].BackupToDatadir == false { + delete(seedOperationProgress, s.SeedID) + } + submitSeedStageState(agentSeedStageState) + s.Status = Error + s.updateSeed(targetAgent, "") + } + case ConnectSlave: + targetAgent, sourceAgent, err := s.readSeadAgents(false) + if err != nil { + return + } + var gtidHint inst.OperationGTIDHint = inst.GTIDHintNeutral + seedMetadata, err := targetAgent.getMetadata(s.SeedID, s.SeedMethod) + if err != nil { + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error calling getMetadata API on agent: %+v", err)) + return + } + s.updateSeedState(targetAgent.Info.Hostname, fmt.Sprintf("Got seed metadata from agent. LogFile: %s, LogPos: %d, GtidExecuted: %s", seedMetadata.LogFile, seedMetadata.LogPos, seedMetadata.GtidExecuted)) + log.Debugf("Got seed metadata from agent. LogFile: %s, LogPos: %d, GtidExecuted: %s", seedMetadata.LogFile, seedMetadata.LogPos, seedMetadata.GtidExecuted) + slaveInstanceKey := &inst.InstanceKey{Hostname: targetAgent.Info.Hostname, Port: targetAgent.Info.MySQLPort} + masterInstanceKey := &inst.InstanceKey{Hostname: sourceAgent.Info.Hostname, Port: sourceAgent.Info.MySQLPort} + slave, err := inst.ReadTopologyInstance(slaveInstanceKey) + if err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error reading topology instance: %+v", err)) + return + } + master, err := inst.ReadTopologyInstance(masterInstanceKey) + if err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(sourceAgent, fmt.Sprintf("Error reading topology instance: %+v", err)) + return + } + if s.Retries > 0 && slave.IsReplicaOf(master) && slave.Slave_SQL_Running && slave.Slave_IO_Running { + //everything is already good, seems like we do not mark seed stage as completed so just do it + auditSeedOperation(targetAgent, fmt.Sprintf("SeedID: %d, Stage: %s, Status: %s, Retries: %d", s.SeedID, s.Stage.String(), Completed.String(), s.Retries)) + s.Stage = s.Stage + 1 + s.Status = Scheduled + s.Retries = 0 + s.updateSeed(targetAgent, "") + } + if slave.ReplicationThreadsExist() && !slave.ReplicationThreadsStopped() { + slave, err = inst.StopSlave(slaveInstanceKey) + if err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error stopping slave threads: %+v", err)) + return + } + } + slave, err = inst.ResetSlave(slaveInstanceKey) + if err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error executing RESET SLAVE: %+v", err)) + return + } + s.updateSeedState(targetAgent.Info.Hostname, "Executed RESET SLAVE") + slave, err = inst.ResetMaster(slaveInstanceKey) + if err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error executing RESET MASTER: %+v", err)) + return + } + s.updateSeedState(targetAgent.Info.Hostname, "Executed RESET MASTER") + if len(seedMetadata.GtidExecuted) > 0 { + if err = inst.SetGTIDPurged(slave, seedMetadata.GtidExecuted); err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error setting GTID_PURGED: %+v", err)) + return + } + } + if master.UsingGTID() { + gtidHint = inst.GTIDHintForce + } + binlogCoordinates := &inst.BinlogCoordinates{ + LogFile: seedMetadata.LogFile, + LogPos: seedMetadata.LogPos, + Type: inst.BinaryLog, + } + slave, err = inst.ChangeMasterTo(slaveInstanceKey, masterInstanceKey, binlogCoordinates, false, gtidHint) + if err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error executing CHANGE MASTER TO: %+v", err)) + return + } + s.updateSeedState(targetAgent.Info.Hostname, "Executed CHANGE MASTER TO binlog position") + replicationUser, replicationPassword, err := inst.ReadReplicationCredentials(slaveInstanceKey) + if err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error getting replication credentials: %+v", err)) + return + } + slave, err = inst.ChangeMasterCredentials(slaveInstanceKey, replicationUser, replicationPassword) + if err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error executing CHANGE MASTER TO with replication credentials: %+v", err)) + return + } + s.updateSeedState(targetAgent.Info.Hostname, "Executed CHANGE MASTER TO master credentials") + if slave.AllowTLS { + if _, err := inst.EnableMasterSSL(slaveInstanceKey); err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error executing CHANGE MASTER TO enable SSL: %+v", err)) + return + } + } + slave, err = inst.StartSlave(slaveInstanceKey) + if err != nil { + log.Errore(err) + s.Status = Error + s.updateSeed(targetAgent, fmt.Sprintf("Error executing START SLAVE: %+v", err)) + return + } + s.updateSeedState(targetAgent.Info.Hostname, "Executed START SLAVE") + s.Status = Completed + auditSeedOperation(targetAgent, fmt.Sprintf("SeedID: %d, Stage: %s, Status: %s, Retries: %d", s.SeedID, s.Stage.String(), Completed.String(), s.Retries)) + s.updateSeedState(targetAgent.Info.Hostname, "Completed connect slave stage") + s.Stage = s.Stage + 1 + s.Status = Scheduled + s.Retries = 0 + s.updateSeed(targetAgent, "") + } +} + +func (s *Seed) processErrored(wg *sync.WaitGroup) { + defer wg.Done() + targetAgent, sourceAgent, err := s.readSeadAgents(false) + if err != nil { + return + } + if s.Retries >= config.Config.MaxRetriesForSeedStage { + s.Status = Failed + s.EndTimestamp = time.Now() + if s.Stage == Prepare || s.Stage == Cleanup { + for _, agent := range []*Agent{targetAgent, sourceAgent} { + s.updateSeed(agent, fmt.Sprintf("Seed failed. Seed stage retries: %d, MaxRetriesForSeedStage: %d", s.Retries, config.Config.MaxRetriesForSeedStage)) + } + } + if s.Stage == Restore || s.Stage == ConnectSlave { + s.updateSeed(targetAgent, fmt.Sprintf("Seed failed. Seed stage retries: %d, MaxRetriesForSeedStage: %d", s.Retries, config.Config.MaxRetriesForSeedStage)) + } + if s.Stage == Backup { + s.Stage = Prepare + if s.BackupSide == Target { + s.updateSeed(targetAgent, fmt.Sprintf("Seed failed. Seed stage retries: %d, MaxRetriesForSeedStage: %d", s.Retries, config.Config.MaxRetriesForSeedStage)) + } else { + s.updateSeed(sourceAgent, fmt.Sprintf("Seed failed. Seed stage retries: %d, MaxRetriesForSeedStage: %d", s.Retries, config.Config.MaxRetriesForSeedStage)) + } + } + return + } + // update set SeedStatus Scheduled and update seed according to stage + s.Status = Scheduled + s.Retries++ + switch s.Stage { + case Prepare, Cleanup: + // prepare and cleanup stages are executed on both agents, so we need to check them and if it's running - stop it. + for _, agent := range []*Agent{targetAgent, sourceAgent} { + agentSeedStageState, err := agent.getSeedStageState(s.SeedID, s.Stage) + if err != nil { + s.Retries++ + s.updateSeed(agent, fmt.Sprintf("Error getting seed stage state information from agent: %+v", err)) + return + } + if agentSeedStageState.Status == Running { + if err := agent.AbortSeed(s.SeedID, s.Stage); err != nil { + s.Retries++ + s.updateSeed(agent, fmt.Sprintf("Error aborting seed on agent: %+v", err)) + return + } + s.updateSeedState(agent.Info.Hostname, fmt.Sprintf("Aborted %s seed stage due to error on another agent", s.Stage.String())) + } + } + for _, agent := range []*Agent{targetAgent, sourceAgent} { + s.updateSeed(agent, fmt.Sprintf("Seed will be restarted from %s stage", s.Stage.String())) + } + case Restore, ConnectSlave: + s.updateSeed(targetAgent, fmt.Sprintf("Seed will be restarted from %s stage", s.Stage.String())) + // if stage is backup we need to restart from prepare stage (because we can have operations like stating netcat\socat on target in prepare stage) + case Backup: + s.Stage = Prepare + if s.BackupSide == Target { + s.updateSeed(targetAgent, fmt.Sprintf("Seed will be restarted from %s stage", s.Stage.String())) + } else { + s.updateSeed(sourceAgent, fmt.Sprintf("Seed will be restarted from %s stage", s.Stage.String())) + } + } +} + +func (s *Seed) updateSeed(agent *Agent, seedStateDetails string) { + if err := s.updateSeedData(); err == nil { + auditSeedOperation(agent, fmt.Sprintf("SeedID: %d, Stage: %s, Status: %s, Retries: %d", s.SeedID, s.Stage.String(), s.Status.String(), s.Retries)) + } + if seedStateDetails != "" { + s.updateSeedState(agent.Info.Hostname, seedStateDetails) + } +} + +func (s *Seed) updateSeedState(hostname string, details string) error { + seedStageState := &SeedStageState{ + SeedID: s.SeedID, + Stage: s.Stage, + Hostname: hostname, + Timestamp: time.Now(), + Status: s.Status, + Details: details, + } + return submitSeedStageState(seedStageState) +} + +// AbortSeed aborts seed operation +func (s *Seed) AbortSeed() error { + previousStatus := s.Status + if previousStatus != Aborting { + s.Status = Aborting + s.updateSeedData() + } + targetAgent, sourceAgent, err := s.readSeadAgents(false) + if err != nil { + return err + } + if previousStatus == Scheduled || previousStatus == Error { + for _, agent := range []*Agent{targetAgent, sourceAgent} { + s.updateSeedState(agent.Info.Hostname, fmt.Sprintf("Aborted %s seed stage", s.Stage.String())) + } + s.Status = Failed + s.EndTimestamp = time.Now() + return s.updateSeedData() + } else if previousStatus == Failed || previousStatus == Completed { + return fmt.Errorf("Unable to abort seed in %s status", previousStatus.String()) + } + switch s.Stage { + case Prepare: + for _, agent := range []*Agent{targetAgent, sourceAgent} { + agentSeedStageState, err := agent.getSeedStageState(s.SeedID, s.Stage) + if err != nil { + return err + } + if agentSeedStageState.Status == Running || (agent.Info.Hostname == targetAgent.Info.Hostname && s.BackupSide == Source) { + if err := agent.AbortSeed(s.SeedID, s.Stage); err != nil { + return err + } + s.updateSeedState(agent.Info.Hostname, fmt.Sprintf("Aborted %s seed stage", s.Stage.String())) + } + } + case Cleanup: + for _, agent := range []*Agent{targetAgent, sourceAgent} { + agentSeedStageState, err := agent.getSeedStageState(s.SeedID, s.Stage) + if err != nil { + return err + } + if agentSeedStageState.Status == Running { + if err := agent.AbortSeed(s.SeedID, s.Stage); err != nil { + return err + } + s.updateSeedState(agent.Info.Hostname, fmt.Sprintf("Aborted %s seed stage", s.Stage.String())) + } + } + case Backup, Restore: + agent := targetAgent + if s.BackupSide == Source && s.Stage == Backup { + agent = sourceAgent + } + agentSeedStageState, err := agent.getSeedStageState(s.SeedID, s.Stage) + if err != nil { + return err + } + if agentSeedStageState.Status == Running { + if err := agent.AbortSeed(s.SeedID, s.Stage); err != nil { + return err + } + s.updateSeedState(agent.Info.Hostname, fmt.Sprintf("Aborted %s seed stage", s.Stage.String())) + } + } + s.Status = Failed + s.EndTimestamp = time.Now() + return s.updateSeedData() +} diff --git a/go/agent/seed_dao.go b/go/agent/seed_dao.go new file mode 100644 index 000000000..329be4dc4 --- /dev/null +++ b/go/agent/seed_dao.go @@ -0,0 +1,212 @@ +package agent + +import ( + "fmt" + "time" + + "github.com/github/orchestrator/go/db" + "github.com/openark/golib/log" + "github.com/openark/golib/sqlutils" +) + +// registerSeedEntry registers a new seed operation entry, returning seed with it's unique id +func (seed *Seed) registerSeedEntry() error { + res, err := db.ExecOrchestrator(` + insert + into agent_seed ( + seed_method, target_hostname, source_hostname, start_timestamp, updated_at, status, backup_side, retries, stage + ) VALUES ( + ?, ?, ?, ?, ?, ?, ?, 0, ? + ) + `, + seed.SeedMethod.String(), + seed.TargetHostname, + seed.SourceHostname, + seed.StartTimestamp, + seed.UpdatedAt, + seed.Status.String(), + seed.BackupSide.String(), + seed.Stage.String(), + ) + if err != nil { + return err + } + id, err := res.LastInsertId() + if err != nil { + return err + } + seed.SeedID = id + return nil +} + +// updateSeedData updates an information about seed +func (seed *Seed) updateSeedData() error { + _, err := db.ExecOrchestrator(` + update + agent_seed + set + stage = ?, + status = ?, + retries = ?, + updated_at = NOW(), + end_timestamp = ? + where + agent_seed_id = ? + `, + seed.Stage.String(), + seed.Status.String(), + seed.Retries, + seed.EndTimestamp, + seed.SeedID, + ) + if err != nil { + return log.Errore(fmt.Errorf("Unable to update seed: %+v", err)) + } + return nil +} + +// isSeedStageCompletedForAgent checks if current seed stage is already completed for current agent +func (seed *Seed) isSeedStageCompletedForAgent(agent *Agent) (bool, error) { + var cnt int + query := ` + SELECT + count(*) as cnt + FROM + agent_seed_state + WHERE + agent_seed_id = ? + AND stage = ? + AND agent_hostname = ? + AND status = ? + AND agent_seed_state_id > ( + SELECT + MAX(agent_seed_state_id) + FROM + agent_seed_state + WHERE + agent_seed_id = ? + AND stage = ? + AND agent_hostname = ? + AND status = ? + ) + ` + if err := db.QueryOrchestrator(query, sqlutils.Args(seed.SeedID, seed.Stage.String(), agent.Info.Hostname, Completed.String(), seed.SeedID, seed.Stage.String(), agent.Info.Hostname, Scheduled.String()), func(m sqlutils.RowMap) error { + cnt = m.GetInt("cnt") + return nil + }); err != nil { + return false, log.Errore(fmt.Errorf("Unable to get information about completed stages for seed: %+v", err)) + } + if cnt > 0 { + return true, nil + } + return false, nil +} + +// submitSeedStageState submits a seed stage state: a single step in the overall seed process +func submitSeedStageState(seedStageState *SeedStageState) error { + _, err := db.ExecOrchestrator(` + insert + into agent_seed_state ( + agent_seed_id, stage, agent_hostname, state_timestamp, status, details + ) VALUES ( + ?, ?, ?, ?, ?, ? + ) + `, + seedStageState.SeedID, + seedStageState.Stage.String(), + seedStageState.Hostname, + time.Now(), + seedStageState.Status.String(), + seedStageState.Details, + ) + if err != nil { + return log.Errore(fmt.Errorf("Unable to submit seed stage to database: %+v", err)) + } + return nil +} + +// readSeeds reads seed from the backend table +func readSeeds(whereCondition string, args []interface{}, limit string) ([]*Seed, error) { + res := []*Seed{} + query := fmt.Sprintf(` + select + agent_seed_id, + target_hostname, + source_hostname, + seed_method, + backup_side, + stage, + status, + retries, + start_timestamp, + end_timestamp, + updated_at + from + agent_seed + %s + order by + agent_seed_id desc + %s + `, whereCondition, limit) + err := db.QueryOrchestrator(query, args, func(m sqlutils.RowMap) error { + seed := Seed{} + seed.SeedID = m.GetInt64("agent_seed_id") + seed.TargetHostname = m.GetString("target_hostname") + seed.SourceHostname = m.GetString("source_hostname") + seed.SeedMethod = toSeedMethod[m.GetString("seed_method")] + seed.BackupSide = toSeedSide[m.GetString("backup_side")] + seed.Stage = toSeedStage[m.GetString("stage")] + seed.Status = ToSeedStatus[m.GetString("status")] + seed.Retries = m.GetInt("retries") + seed.StartTimestamp = m.GetTime("start_timestamp") + seed.EndTimestamp = m.GetTime("end_timestamp") + seed.UpdatedAt = m.GetTime("updated_at") + + res = append(res, &seed) + return nil + }) + + if err != nil { + log.Errore(err) + } + return res, err +} + +// ReadSeedStageStates reads states for a given seed operation +func (seed *Seed) ReadSeedStageStates() ([]SeedStageState, error) { + res := []SeedStageState{} + query := ` + select + agent_seed_state_id, + agent_seed_id, + stage, + agent_hostname, + state_timestamp, + status, + details + from + agent_seed_state + where + agent_seed_id = ? + order by + agent_seed_state_id desc + ` + err := db.QueryOrchestrator(query, sqlutils.Args(seed.SeedID), func(m sqlutils.RowMap) error { + seedState := SeedStageState{} + seedState.SeedStageID = m.GetInt64("agent_seed_state_id") + seedState.SeedID = m.GetInt64("agent_seed_id") + seedState.Stage = toSeedStage[m.GetString("stage")] + seedState.Hostname = m.GetString("agent_hostname") + seedState.Timestamp = m.GetTime("state_timestamp") + seedState.Status = ToSeedStatus[m.GetString("status")] + seedState.Details = m.GetString("details") + + res = append(res, seedState) + return nil + }) + + if err != nil { + log.Errore(err) + } + return res, err +} diff --git a/go/app/cli.go b/go/app/cli.go index 6d8860a8e..aaf449f95 100644 --- a/go/app/cli.go +++ b/go/app/cli.go @@ -1706,7 +1706,11 @@ func Cli(command string, strict bool, instance string, destination string, owner } case registerCliCommand("custom-command", "Agent", "Execute a custom command on the agent as defined in the agent conf"): { - output, err := agent.CustomCommand(hostnameFlag, pattern) + agent, err := agent.ReadAgentInfo(hostnameFlag) + if err != nil { + log.Fatale(err) + } + output, err := agent.CustomCommand(pattern) if err != nil { log.Fatale(err) } diff --git a/go/app/http.go b/go/app/http.go index 92b2f31e3..c429a5578 100644 --- a/go/app/http.go +++ b/go/app/http.go @@ -181,6 +181,7 @@ func agentsHttp() { agent.InitHttpClient() go logic.ContinuousAgentsPoll() + go logic.ContinuousSeedProcess() http.AgentsAPI.URLPrefix = config.Config.URLPrefix http.AgentsAPI.RegisterRequests(m) diff --git a/go/config/config.go b/go/config/config.go index af3139a4a..d1b1717d8 100644 --- a/go/config/config.go +++ b/go/config/config.go @@ -58,6 +58,7 @@ const ( PseudoGTIDExpireMinutes = 60 CheckAutoPseudoGTIDGrantsIntervalSeconds = 60 SelectTrueQuery = "select 1" + SeedDowntimeHours = 96 ) var deprecatedConfigurationVariables = []string{ @@ -197,6 +198,11 @@ type Configuration struct { AgentSSLCertFile string // Name of Agent SSL certification file, applies only when AgentsUseSSL = true AgentSSLCAFile string // Name of the Agent Certificate Authority file, applies only when AgentsUseSSL = true AgentSSLValidOUs []string // Valid organizational units when using mutual TLS to communicate with the agents + UnseenAgentForgetHours uint // Number of hours after which an unseen agent is forgotten + AgentPollMinutes uint // Minutes between agent polling + MaxRetriesForSeedStage int // Number of maximum retries for each of the seed stages, after which seed will be marked as failed + SeedProcessIntervalSeconds uint // Interval in seconds between processing active seeds + SeedBackupStaleFailMinutes uint // Number of minutes after which a stale (no progress) seed in Backup stage is considered failed UseSSL bool // Use SSL on the server web port UseMutualTLS bool // When "true" Use mutual TLS for the server's web and API connections SSLSkipVerify bool // When using SSL, should we ignore SSL certification error @@ -206,11 +212,6 @@ type Configuration struct { SSLValidOUs []string // Valid organizational units when using mutual TLS StatusEndpoint string // Override the status endpoint. Defaults to '/api/status' StatusOUVerify bool // If true, try to verify OUs when Mutual TLS is on. Defaults to false - AgentPollMinutes uint // Minutes between agent polling - UnseenAgentForgetHours uint // Number of hours after which an unseen agent is forgotten - StaleSeedFailMinutes uint // Number of minutes after which a stale (no progress) seed is considered failed. - SeedAcceptableBytesDiff int64 // Difference in bytes between seed source & target data size that is still considered as successful copy - SeedWaitSecondsBeforeSend int64 // Number of seconds for waiting before start send data command on agent AutoPseudoGTID bool // Should orchestrator automatically inject Pseudo-GTID entries to the masters PseudoGTIDPattern string // Pattern to look for in binary logs that makes for a unique entry (pseudo GTID). When empty, Pseudo-GTID based refactoring is disabled. PseudoGTIDPatternIsFixedSubstring bool // If true, then PseudoGTIDPattern is not treated as regular expression but as fixed substring, and can boost search time @@ -376,11 +377,11 @@ func newConfiguration() *Configuration { SSLPrivateKeyFile: "", SSLCertFile: "", SSLCAFile: "", - AgentPollMinutes: 60, + AgentPollMinutes: 30, + SeedBackupStaleFailMinutes: 60, UnseenAgentForgetHours: 6, - StaleSeedFailMinutes: 60, - SeedAcceptableBytesDiff: 8192, - SeedWaitSecondsBeforeSend: 2, + MaxRetriesForSeedStage: 3, + SeedProcessIntervalSeconds: 60, AutoPseudoGTID: false, PseudoGTIDPattern: "", PseudoGTIDPatternIsFixedSubstring: false, diff --git a/go/db/db.go b/go/db/db.go index bac6758f8..3f13955c7 100644 --- a/go/db/db.go +++ b/go/db/db.go @@ -194,6 +194,9 @@ func translateStatement(statement string) (string, error) { if IsSQLite() { statement = sqlutils.ToSqlite3Dialect(statement) } + if config.Config.IsMySQL() && strings.Contains(statement, "/* mysql-skip */") { + statement = "" + } return statement, nil } @@ -264,19 +267,21 @@ func deployStatements(db *sql.DB, queries []string) error { if err != nil { return log.Fatalf("Cannot initiate orchestrator: %+v; query=%+v", err, query) } - if _, err := tx.Exec(query); err != nil { - if strings.Contains(err.Error(), "syntax error") { - return log.Fatalf("Cannot initiate orchestrator: %+v; query=%+v", err, query) - } - if !sqlutils.IsAlterTable(query) && !sqlutils.IsCreateIndex(query) && !sqlutils.IsDropIndex(query) { - return log.Fatalf("Cannot initiate orchestrator: %+v; query=%+v", err, query) - } - if !strings.Contains(err.Error(), "duplicate column name") && - !strings.Contains(err.Error(), "Duplicate column name") && - !strings.Contains(err.Error(), "check that column/key exists") && - !strings.Contains(err.Error(), "already exists") && - !strings.Contains(err.Error(), "Duplicate key name") { - log.Errorf("Error initiating orchestrator: %+v; query=%+v", err, query) + if len(query) > 0 { + if _, err := tx.Exec(query); err != nil { + if strings.Contains(err.Error(), "syntax error") { + return log.Fatalf("Cannot initiate orchestrator: %+v; query=%+v", err, query) + } + if !sqlutils.IsAlterTable(query) && !sqlutils.IsCreateIndex(query) && !sqlutils.IsDropIndex(query) { + return log.Fatalf("Cannot initiate orchestrator: %+v; query=%+v", err, query) + } + if !strings.Contains(err.Error(), "duplicate column name") && + !strings.Contains(err.Error(), "Duplicate column name") && + !strings.Contains(err.Error(), "check that column/key exists") && + !strings.Contains(err.Error(), "already exists") && + !strings.Contains(err.Error(), "Duplicate key name") { + log.Errorf("Error initiating orchestrator: %+v; query=%+v", err, query) + } } } } diff --git a/go/db/generate_patches.go b/go/db/generate_patches.go index ca66f2c8b..0c3aed0d7 100644 --- a/go/db/generate_patches.go +++ b/go/db/generate_patches.go @@ -546,4 +546,174 @@ var generateSQLPatches = []string{ database_instance ADD COLUMN region varchar(32) CHARACTER SET ascii NOT NULL AFTER data_center `, + ` + DROP INDEX last_submitted_idx_host_agent ON host_agent + `, + ` + DROP INDEX token_idx_host_agent ON host_agent + `, + ` + ALTER TABLE + host_agent /* sqlite3-skip */ + DROP COLUMN count_mysql_snapshots + `, + ` + ALTER TABLE + host_agent /* sqlite3-skip */ + DROP COLUMN last_submitted + `, + ` + ALTER TABLE + host_agent /* sqlite3-skip */ + ADD COLUMN status varchar(16) CHARACTER SET ascii NOT NULL AFTER mysql_port + `, + ` + ALTER TABLE + host_agent /* sqlite3-skip */ + ADD COLUMN data TEXT CHARACTER SET ascii NOT NULL AFTER status + `, + ` + DROP INDEX is_complete_idx_agent_seed ON agent_seed + `, + ` + DROP INDEX is_successful_idx_agent_seed ON agent_seed + `, + ` + DROP INDEX start_timestamp_idx_agent_seed ON agent_seed + `, + ` + ALTER TABLE + agent_seed /* sqlite3-skip */ + DROP COLUMN is_successful + `, + ` + ALTER TABLE + agent_seed /* sqlite3-skip */ + DROP COLUMN is_complete + `, + ` + ALTER TABLE + agent_seed /* sqlite3-skip */ + ADD COLUMN seed_method varchar(32) CHARACTER SET ascii NOT NULL AFTER source_hostname + `, + ` + ALTER TABLE + agent_seed /* sqlite3-skip */ + ADD COLUMN backup_side varchar(8) NOT NULL AFTER seed_method + `, + ` + ALTER TABLE + agent_seed /* sqlite3-skip */ + ADD COLUMN stage varchar(16) NOT NULL AFTER backup_side + `, + ` + ALTER TABLE + agent_seed /* sqlite3-skip */ + ADD COLUMN status varchar(16) NOT NULL AFTER stage + `, + ` + ALTER TABLE + agent_seed /* sqlite3-skip */ + ADD COLUMN retries int(10) unsigned NOT NULL AFTER status + `, + ` + ALTER TABLE + agent_seed /* sqlite3-skip */ + ADD COLUMN updated_at timestamp NOT NULL DEFAULT '1971-01-01 00:00:00' AFTER end_timestamp + `, + ` + DROP INDEX agent_seed_idx_agent_seed_state ON agent_seed_state + `, + ` + ALTER TABLE + agent_seed_state /* sqlite3-skip */ + DROP COLUMN state_action + `, + ` + ALTER TABLE + agent_seed_state /* sqlite3-skip */ + DROP COLUMN error_message + `, + ` + ALTER TABLE + agent_seed_state /* sqlite3-skip */ + ADD COLUMN agent_hostname varchar(128) NOT NULL DEFAULT '' AFTER agent_seed_id + `, + ` + ALTER TABLE + agent_seed_state /* sqlite3-skip */ + ADD COLUMN stage varchar(16) NOT NULL AFTER agent_hostname + `, + ` + ALTER TABLE + agent_seed_state /* sqlite3-skip */ + ADD COLUMN status varchar(16) NOT NULL AFTER state_timestamp + `, + ` + ALTER TABLE + agent_seed_state /* sqlite3-skip */ + ADD COLUMN details text NOT NULL AFTER status + `, + ` + DROP TABLE host_agent; /* mysql-skip */ + `, + ` + DROP TABLE agent_seed; /* mysql-skip */ + `, + ` + DROP TABLE agent_seed_state; /* mysql-skip */ + `, + ` + CREATE TABLE IF NOT EXISTS host_agent /* mysql-skip */ ( + hostname varchar(128) NOT NULL, + port smallint(5) unsigned NOT NULL, + token varchar(128) NOT NULL, + last_checked timestamp NULL DEFAULT NULL, + last_seen timestamp NULL DEFAULT NULL, + mysql_port smallint(5) unsigned DEFAULT NULL, + status varchar(16) NOT NULL, + data text NOT NULL, + PRIMARY KEY (hostname) + ) ENGINE=InnoDB DEFAULT CHARSET=ascii + `, + ` + CREATE TABLE IF NOT EXISTS agent_seed /* mysql-skip */ ( + agent_seed_id int(10) unsigned NOT NULL AUTO_INCREMENT, + target_hostname varchar(128) NOT NULL, + source_hostname varchar(128) NOT NULL, + seed_method varchar(32) NOT NULL, + backup_side varchar(8) NOT NULL, + stage varchar(16) NOT NULL, + status varchar(16) NOT NULL, + retries int(10) unsigned NOT NULL, + start_timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + end_timestamp timestamp NOT NULL DEFAULT '1971-01-01 00:00:00', + updated_at timestamp NOT NULL DEFAULT '1971-01-01 00:00:00', + PRIMARY KEY (agent_seed_id) + ) ENGINE=InnoDB DEFAULT CHARSET=ascii + `, + ` + CREATE TABLE IF NOT EXISTS agent_seed_state /* mysql-skip */ ( + agent_seed_state_id int(10) unsigned NOT NULL AUTO_INCREMENT, + agent_seed_id int(10) unsigned NOT NULL, + agent_hostname varchar(128) NOT NULL DEFAULT '', + stage varchar(16) NOT NULL, + state_timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + status varchar(16) NOT NULL, + details text NOT NULL, + PRIMARY KEY (agent_seed_state_id) + ) ENGINE=InnoDB DEFAULT CHARSET=ascii + `, + ` + CREATE INDEX status_idx_host_agent ON host_agent (status) + `, + ` + CREATE INDEX status_idx_agent_seed ON agent_seed (status) + `, + ` + CREATE INDEX agent_seed_id_idx_agent_seed_state ON agent_seed_state (agent_seed_id) + `, + ` + CREATE INDEX agent_seed_id_idx_agent_hostname ON agent_seed_state (agent_hostname) + `, } diff --git a/go/http/agents_api.go b/go/http/agents_api.go index 806ac4b5f..c1e7b0ca6 100644 --- a/go/http/agents_api.go +++ b/go/http/agents_api.go @@ -17,9 +17,9 @@ package http import ( + "encoding/json" "fmt" "net/http" - "strconv" "strings" "github.com/go-martini/martini" @@ -35,15 +35,17 @@ type HttpAgentsAPI struct { var AgentsAPI HttpAgentsAPI = HttpAgentsAPI{} -// SubmitAgent registeres an agent. It is initiated by an agent to register itself. -func (this *HttpAgentsAPI) SubmitAgent(params martini.Params, r render.Render) { - port, err := strconv.Atoi(params["port"]) +// RegisterAgent registeres an agent. It is initiated by an agent to register itself. +func (this *HttpAgentsAPI) RegisterAgent(params martini.Params, r render.Render, req *http.Request) { + var agentInfo agent.Info + decoder := json.NewDecoder(req.Body) + err := decoder.Decode(&agentInfo) if err != nil { r.JSON(200, &APIResponse{Code: ERROR, Message: err.Error()}) return } - output, err := agent.SubmitAgent(params["host"], port, params["token"]) + output, err := agent.RegisterAgent(&agentInfo) if err != nil { r.JSON(200, &APIResponse{Code: ERROR, Message: err.Error()}) return @@ -76,33 +78,12 @@ func (this *HttpAgentsAPI) GetHostAttributeByAttributeName(params martini.Params r.JSON(200, output) } -// AgentsHosts provides list of agent host names -func (this *HttpAgentsAPI) AgentsHosts(params martini.Params, r render.Render, req *http.Request) string { - agents, err := agent.ReadAgents() - hostnames := []string{} - for _, agent := range agents { - hostnames = append(hostnames, agent.Hostname) - } - - if err != nil { - r.JSON(200, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) - return "" - } - - if req.URL.Query().Get("format") == "txt" { - return strings.Join(hostnames, "\n") - } else { - r.JSON(200, hostnames) - } - return "" -} - // AgentsInstances provides list of assumed MySQL instances (host:port) func (this *HttpAgentsAPI) AgentsInstances(params martini.Params, r render.Render, req *http.Request) string { - agents, err := agent.ReadAgents() + agents, err := agent.ReadAgentsInfo() hostnames := []string{} for _, agent := range agents { - hostnames = append(hostnames, fmt.Sprintf("%s:%d", agent.Hostname, agent.MySQLPort)) + hostnames = append(hostnames, fmt.Sprintf("%s:%d", agent.Info.Hostname, agent.Info.MySQLPort)) } if err != nil { @@ -124,10 +105,9 @@ func (this *HttpAgentsAPI) AgentPing(params martini.Params, r render.Render, req // RegisterRequests makes for the de-facto list of known API calls func (this *HttpAgentsAPI) RegisterRequests(m *martini.ClassicMartini) { - m.Get(this.URLPrefix+"/api/submit-agent/:host/:port/:token", this.SubmitAgent) + m.Post(this.URLPrefix+"/api/register-agent", this.RegisterAgent) m.Get(this.URLPrefix+"/api/host-attribute/:host/:attrVame/:attrValue", this.SetHostAttribute) m.Get(this.URLPrefix+"/api/host-attribute/attr/:attr/", this.GetHostAttributeByAttributeName) - m.Get(this.URLPrefix+"/api/agents-hosts", this.AgentsHosts) m.Get(this.URLPrefix+"/api/agents-instances", this.AgentsInstances) m.Get(this.URLPrefix+"/api/agent-ping", this.AgentPing) } diff --git a/go/http/api.go b/go/http/api.go index 391337a9c..dc6f80085 100644 --- a/go/http/api.go +++ b/go/http/api.go @@ -21,6 +21,7 @@ import ( "fmt" "net" "net/http" + "regexp" "strconv" "strings" "time" @@ -1816,6 +1817,18 @@ func (this *HttpAPI) Clusters(params martini.Params, r render.Render, req *http. r.JSON(http.StatusOK, clusterNames) } +// ClustersAliases provides list of known clusters aliases +func (this *HttpAPI) ClustersAliases(params martini.Params, r render.Render, req *http.Request) { + clusterAliases, err := inst.ReadClustersAliases(req.URL.Query().Get("clusteralias")) + + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, clusterAliases) +} + // ClustersInfo provides list of known clusters, along with some added metadata per cluster func (this *HttpAPI) ClustersInfo(params martini.Params, r render.Render, req *http.Request) { clustersInfo, err := inst.ReadClustersInfo("") @@ -2428,18 +2441,80 @@ func (this *HttpAPI) WriteBufferMetricsAggregated(params martini.Params, r rende r.JSON(http.StatusOK, aggregated) } -// Agents provides complete list of registered agents (See https://github.com/github/orchestrator-agent) -func (this *HttpAPI) Agents(params martini.Params, r render.Render, req *http.Request, user auth.User) { - if !isAuthorizedForAction(req, user) { - Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) +// AgentsStatuses provides list of all possible agent statuses +func (this *HttpAPI) AgentsStatuses(params martini.Params, r render.Render, req *http.Request, user auth.User) { + if !config.Config.ServeAgentsHttp { + Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) return } + filter := req.URL.Query().Get("status") + fmt.Println(filter) + var agentStatuses []string + for _, agentStatus := range agent.AgentStatuses { + if len(filter) > 0 { + if matched, _ := regexp.MatchString(fmt.Sprintf("(?i)^%s.*", filter), agentStatus.String()); matched { + agentStatuses = append(agentStatuses, agentStatus.String()) + } + } else { + agentStatuses = append(agentStatuses, agentStatus.String()) + } + } + r.JSON(http.StatusOK, agentStatuses) +} + +// SeedsStatuses provides list of all possible seeds statuses +func (this *HttpAPI) SeedsStatuses(params martini.Params, r render.Render, req *http.Request, user auth.User) { if !config.Config.ServeAgentsHttp { Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) return } + filter := req.URL.Query().Get("status") + fmt.Println(filter) + var seedStatuses []string + for _, seedStatus := range agent.SeedStatuses { + if len(filter) > 0 { + if matched, _ := regexp.MatchString(fmt.Sprintf("(?i)^%s.*", filter), seedStatus.String()); matched { + seedStatuses = append(seedStatuses, seedStatus.String()) + } + } else { + seedStatuses = append(seedStatuses, seedStatus.String()) + } + } + r.JSON(http.StatusOK, seedStatuses) +} - agents, err := agent.ReadAgents() +// Agents provides complete list of registered agents (See https://github.com/github/orchestrator-agent) +func (this *HttpAPI) Agents(params martini.Params, r render.Render, req *http.Request, user auth.User) { + if !config.Config.ServeAgentsHttp { + Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) + return + } + var agents []*agent.Agent + + page, err := strconv.Atoi(req.URL.Query().Get("page")) + if err != nil || page < 0 { + agents, err = agent.ReadAgents() + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + r.JSON(http.StatusOK, agents) + return + } + if hostname := req.URL.Query().Get("hostname"); hostname != "" { + var foundAgent *agent.Agent + foundAgent, err = agent.ReadAgent(hostname) + if err == nil { + agents = append(agents, foundAgent) + } + } else if clusterName := req.URL.Query().Get("clusteralias"); clusterName != "" { + agents, err = agent.ReadAgentsForClusterPaged(clusterName, page) + } else if status := req.URL.Query().Get("status"); status != "" { + agentStatus := agent.ToAgentStatus[status] + agents, err = agent.ReadAgentsInStatusPaged(agentStatus, page) + } else { + agents, err = agent.ReadAgentsPaged(page) + } if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) @@ -2449,8 +2524,57 @@ func (this *HttpAPI) Agents(params martini.Params, r render.Render, req *http.Re r.JSON(http.StatusOK, agents) } +// AgentsHosts provides list of agents hostnames +func (this *HttpAPI) AgentsHosts(params martini.Params, r render.Render, req *http.Request, user auth.User) { + if !config.Config.ServeAgentsHttp { + Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) + return + } + agentsHosts, err := agent.ReadAgentsHosts(req.URL.Query().Get("hostname")) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, agentsHosts) +} + // Agent returns complete information of a given agent func (this *HttpAPI) Agent(params martini.Params, r render.Render, req *http.Request, user auth.User) { + if !config.Config.ServeAgentsHttp { + Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) + return + } + + agent, err := agent.ReadAgent(params["host"]) + + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, agent) +} + +// AgentData returns data for give agent +func (this *HttpAPI) AgentData(params martini.Params, r render.Render, req *http.Request, user auth.User) { + if !config.Config.ServeAgentsHttp { + Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) + return + } + + agent, err := agent.ReadAgent(params["host"]) + + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, agent.Data) +} + +// UpdateAgent updates information about a given agent in database +func (this *HttpAPI) UpdateAgent(params martini.Params, r render.Render, req *http.Request, user auth.User) { if !isAuthorizedForAction(req, user) { Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) return @@ -2460,18 +2584,22 @@ func (this *HttpAPI) Agent(params martini.Params, r render.Render, req *http.Req return } - agent, err := agent.GetAgent(params["host"]) - + agent, err := agent.ReadAgentInfo(params["host"]) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return } + if err = agent.UpdateAgent(); err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + r.JSON(http.StatusOK, agent) } -// AgentUnmount instructs an agent to unmount the designated mount point -func (this *HttpAPI) AgentUnmount(params martini.Params, r render.Render, req *http.Request, user auth.User) { +// AgentTailMySQLErrorLog returns tail of MySQL error log from agent +func (this *HttpAPI) AgentTailMySQLErrorLog(params martini.Params, r render.Render, req *http.Request, user auth.User) { if !isAuthorizedForAction(req, user) { Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) return @@ -2481,8 +2609,13 @@ func (this *HttpAPI) AgentUnmount(params martini.Params, r render.Render, req *h return } - output, err := agent.Unmount(params["host"]) + agent, err := agent.ReadAgentInfo(params["host"]) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + output, err := agent.ErrorLogTail() if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return @@ -2491,6 +2624,31 @@ func (this *HttpAPI) AgentUnmount(params martini.Params, r render.Render, req *h r.JSON(http.StatusOK, output) } +// AgentUnmount instructs an agent to unmount the designated mount point +func (this *HttpAPI) AgentUnmount(params martini.Params, r render.Render, req *http.Request, user auth.User) { + if !isAuthorizedForAction(req, user) { + Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) + return + } + if !config.Config.ServeAgentsHttp { + Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) + return + } + + agent, err := agent.ReadAgentInfo(params["host"]) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + if err := agent.Unmount(); err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, agent) +} + // AgentMountLV instructs an agent to mount a given volume on the designated mount point func (this *HttpAPI) AgentMountLV(params martini.Params, r render.Render, req *http.Request, user auth.User) { if !isAuthorizedForAction(req, user) { @@ -2502,14 +2660,18 @@ func (this *HttpAPI) AgentMountLV(params martini.Params, r render.Render, req *h return } - output, err := agent.MountLV(params["host"], req.URL.Query().Get("lv")) - + agent, err := agent.ReadAgentInfo(params["host"]) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return } - r.JSON(http.StatusOK, output) + if err := agent.MountLV(req.URL.Query().Get("lv")); err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, agent) } // AgentCreateSnapshot instructs an agent to create a new snapshot. Agent's DIY implementation. @@ -2523,14 +2685,18 @@ func (this *HttpAPI) AgentCreateSnapshot(params martini.Params, r render.Render, return } - output, err := agent.CreateSnapshot(params["host"]) - + agent, err := agent.ReadAgentInfo(params["host"]) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return } - r.JSON(http.StatusOK, output) + if err := agent.CreateSnapshot(); err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, agent) } // AgentRemoveLV instructs an agent to remove a logical volume @@ -2544,14 +2710,18 @@ func (this *HttpAPI) AgentRemoveLV(params martini.Params, r render.Render, req * return } - output, err := agent.RemoveLV(params["host"], req.URL.Query().Get("lv")) - + agent, err := agent.ReadAgentInfo(params["host"]) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return } - r.JSON(http.StatusOK, output) + if err := agent.RemoveLV(req.URL.Query().Get("lv")); err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, agent) } // AgentMySQLStop stops MySQL service on agent @@ -2565,14 +2735,18 @@ func (this *HttpAPI) AgentMySQLStop(params martini.Params, r render.Render, req return } - output, err := agent.MySQLStop(params["host"]) - + agent, err := agent.ReadAgentInfo(params["host"]) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return } - r.JSON(http.StatusOK, output) + if err := agent.MySQLStop(); err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, agent) } // AgentMySQLStart starts MySQL service on agent @@ -2586,14 +2760,18 @@ func (this *HttpAPI) AgentMySQLStart(params martini.Params, r render.Render, req return } - output, err := agent.MySQLStart(params["host"]) - + agent, err := agent.ReadAgentInfo(params["host"]) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return } - r.JSON(http.StatusOK, output) + if err := agent.MySQLStart(); err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, agent) } func (this *HttpAPI) AgentCustomCommand(params martini.Params, r render.Render, req *http.Request, user auth.User) { @@ -2606,7 +2784,13 @@ func (this *HttpAPI) AgentCustomCommand(params martini.Params, r render.Render, return } - output, err := agent.CustomCommand(params["host"], params["command"]) + agent, err := agent.ReadAgentInfo(params["host"]) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + output, err := agent.CustomCommand(params["command"]) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) @@ -2627,8 +2811,17 @@ func (this *HttpAPI) AgentSeed(params martini.Params, r render.Render, req *http Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) return } - - output, err := agent.Seed(params["targetHost"], params["sourceHost"]) + targetAgent, err := agent.ReadAgentInfo(params["targetHost"]) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + sourceAgent, err := agent.ReadAgentInfo(params["sourceHost"]) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + output, err := agent.NewSeed(params["seedMethod"], targetAgent, sourceAgent) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) @@ -2640,16 +2833,18 @@ func (this *HttpAPI) AgentSeed(params martini.Params, r render.Render, req *http // AgentActiveSeeds lists active seeds and their state func (this *HttpAPI) AgentActiveSeeds(params martini.Params, r render.Render, req *http.Request, user auth.User) { - if !isAuthorizedForAction(req, user) { - Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) - return - } if !config.Config.ServeAgentsHttp { Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) return } - output, err := agent.ReadActiveSeedsForHost(params["host"]) + hostAgent, err := agent.ReadAgentInfo(params["host"]) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + output, err := agent.ReadActiveSeedsForAgent(hostAgent) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) @@ -2661,17 +2856,35 @@ func (this *HttpAPI) AgentActiveSeeds(params martini.Params, r render.Render, re // AgentRecentSeeds lists recent seeds of a given agent func (this *HttpAPI) AgentRecentSeeds(params martini.Params, r render.Render, req *http.Request, user auth.User) { - if !isAuthorizedForAction(req, user) { - Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) + if !config.Config.ServeAgentsHttp { + Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) return } + + hostAgent, err := agent.ReadAgentInfo(params["host"]) + + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + output, err := agent.ReadSeedsForAgent(hostAgent, "limit 10") + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + + r.JSON(http.StatusOK, output) +} + +// AgentFailedSeeds lists all failed seeds +func (this *HttpAPI) AgentFailedSeeds(params martini.Params, r render.Render, req *http.Request, user auth.User) { if !config.Config.ServeAgentsHttp { Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) return } - output, err := agent.ReadRecentCompletedSeedsForHost(params["host"]) - + output, err := agent.ReadSeedsInStatus(agent.Failed) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return @@ -2682,40 +2895,45 @@ func (this *HttpAPI) AgentRecentSeeds(params martini.Params, r render.Render, re // AgentSeedDetails provides details of a given seed func (this *HttpAPI) AgentSeedDetails(params martini.Params, r render.Render, req *http.Request, user auth.User) { - if !isAuthorizedForAction(req, user) { - Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) - return - } if !config.Config.ServeAgentsHttp { Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) return } - seedId, err := strconv.ParseInt(params["seedId"], 10, 0) - output, err := agent.AgentSeedDetails(seedId) + seedID, err := strconv.ParseInt(params["seedId"], 10, 0) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + seed, err := agent.ReadSeed(seedID) if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return } - r.JSON(http.StatusOK, output) + r.JSON(http.StatusOK, seed) } // AgentSeedStates returns the breakdown of states (steps) of a given seed func (this *HttpAPI) AgentSeedStates(params martini.Params, r render.Render, req *http.Request, user auth.User) { - if !isAuthorizedForAction(req, user) { - Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) - return - } if !config.Config.ServeAgentsHttp { Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) return } - seedId, err := strconv.ParseInt(params["seedId"], 10, 0) - output, err := agent.ReadSeedStates(seedId) + seedID, err := strconv.ParseInt(params["seedId"], 10, 0) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + seed, err := agent.ReadSeed(seedID) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + output, err := seed.ReadSeedStageStates() if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return @@ -2724,28 +2942,55 @@ func (this *HttpAPI) AgentSeedStates(params martini.Params, r render.Render, req r.JSON(http.StatusOK, output) } -// Seeds retruns all recent seeds +// Seeds returns all seeds func (this *HttpAPI) Seeds(params martini.Params, r render.Render, req *http.Request, user auth.User) { - if !isAuthorizedForAction(req, user) { - Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) - return - } if !config.Config.ServeAgentsHttp { Respond(r, &APIResponse{Code: ERROR, Message: "Agents not served"}) return } - output, err := agent.ReadRecentSeeds() + var seeds []*agent.Seed + + page, err := strconv.Atoi(req.URL.Query().Get("page")) + if err != nil || page < 0 { + seeds, err = agent.ReadSeeds() + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + r.JSON(http.StatusOK, seeds) + return + } + if targetHostname := req.URL.Query().Get("targethostname"); targetHostname != "" { + targetAgent, err := agent.ReadAgentInfo(targetHostname) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + seeds, err = agent.ReadSeedsForTargetAgentPaged(targetAgent, page) + } else if sourceHostname := req.URL.Query().Get("sourcehostname"); sourceHostname != "" { + sourceAgent, err := agent.ReadAgentInfo(sourceHostname) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + seeds, err = agent.ReadSeedsForSourceAgentPaged(sourceAgent, page) + } else if status := req.URL.Query().Get("status"); status != "" { + seedStatus := agent.ToSeedStatus[status] + seeds, err = agent.ReadSeedsInStatusPaged(seedStatus, page) + } else { + seeds, err = agent.ReadSeedsPaged(page) + } if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return } - r.JSON(http.StatusOK, output) + r.JSON(http.StatusOK, seeds) } -// AbortSeed instructs agents to abort an active seed +// AbortSeed instructs agents to abort a running seed func (this *HttpAPI) AbortSeed(params martini.Params, r render.Render, req *http.Request, user auth.User) { if !isAuthorizedForAction(req, user) { Respond(r, &APIResponse{Code: ERROR, Message: "Unauthorized"}) @@ -2756,14 +3001,21 @@ func (this *HttpAPI) AbortSeed(params martini.Params, r render.Render, req *http return } - seedId, err := strconv.ParseInt(params["seedId"], 10, 0) - err = agent.AbortSeed(seedId) - + seedID, err := strconv.ParseInt(params["seedId"], 10, 0) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + seed, err := agent.ReadSeed(seedID) + if err != nil { + Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) + return + } + err = seed.AbortSeed() if err != nil { Respond(r, &APIResponse{Code: ERROR, Message: fmt.Sprintf("%+v", err)}) return } - r.JSON(http.StatusOK, err == nil) } @@ -3673,6 +3925,7 @@ func (this *HttpAPI) RegisterRequests(m *martini.ClassicMartini) { this.registerAPIRequest(m, "cluster-osc-slaves/:clusterHint", this.ClusterOSCReplicas) this.registerAPIRequest(m, "set-cluster-alias/:clusterName", this.SetClusterAliasManualOverride) this.registerAPIRequest(m, "clusters", this.Clusters) + this.registerAPIRequest(m, "clusters-aliases", this.ClustersAliases) this.registerAPIRequest(m, "clusters-info", this.ClustersInfo) this.registerAPIRequest(m, "masters", this.Masters) @@ -3820,21 +4073,28 @@ func (this *HttpAPI) RegisterRequests(m *martini.ClassicMartini) { // Agents this.registerAPIRequest(m, "agents", this.Agents) + this.registerAPIRequest(m, "agents-statuses", this.AgentsStatuses) + this.registerAPIRequest(m, "agents-hosts", this.AgentsHosts) this.registerAPIRequest(m, "agent/:host", this.Agent) + this.registerAPIRequest(m, "agent-data/:host", this.AgentData) + this.registerAPIRequest(m, "agent-update/:host", this.UpdateAgent) + this.registerAPIRequest(m, "agent-mysql-error-log/:host", this.AgentTailMySQLErrorLog) this.registerAPIRequest(m, "agent-umount/:host", this.AgentUnmount) this.registerAPIRequest(m, "agent-mount/:host", this.AgentMountLV) this.registerAPIRequest(m, "agent-create-snapshot/:host", this.AgentCreateSnapshot) this.registerAPIRequest(m, "agent-removelv/:host", this.AgentRemoveLV) this.registerAPIRequest(m, "agent-mysql-stop/:host", this.AgentMySQLStop) this.registerAPIRequest(m, "agent-mysql-start/:host", this.AgentMySQLStart) - this.registerAPIRequest(m, "agent-seed/:targetHost/:sourceHost", this.AgentSeed) + this.registerAPIRequest(m, "agent-seed/:seedMethod/:targetHost/:sourceHost", this.AgentSeed) this.registerAPIRequest(m, "agent-active-seeds/:host", this.AgentActiveSeeds) this.registerAPIRequest(m, "agent-recent-seeds/:host", this.AgentRecentSeeds) + this.registerAPIRequest(m, "agents-failed-seeds", this.AgentFailedSeeds) this.registerAPIRequest(m, "agent-seed-details/:seedId", this.AgentSeedDetails) this.registerAPIRequest(m, "agent-seed-states/:seedId", this.AgentSeedStates) this.registerAPIRequest(m, "agent-abort-seed/:seedId", this.AbortSeed) this.registerAPIRequest(m, "agent-custom-command/:host/:command", this.AgentCustomCommand) this.registerAPIRequest(m, "seeds", this.Seeds) + this.registerAPIRequest(m, "seeds-statuses", this.SeedsStatuses) // Configurable status check endpoint m.Get(config.Config.StatusEndpoint, this.StatusCheck) diff --git a/go/inst/cluster_alias_dao.go b/go/inst/cluster_alias_dao.go index 7eede7c75..33819a626 100644 --- a/go/inst/cluster_alias_dao.go +++ b/go/inst/cluster_alias_dao.go @@ -125,17 +125,20 @@ func UpdateClusterAliases() error { left join database_instance_downtime using (hostname, port) where suggested_cluster_alias!='' - /* exclude newly demoted, downtimed masters */ + /* exclude newly demoted, downtimed masters and hosts which are seeding */ and ifnull( database_instance_downtime.downtime_active = 1 and database_instance_downtime.end_timestamp > now() - and database_instance_downtime.reason = ? + and ( + database_instance_downtime.reason = ? + or database_instance_downtime.reason = ? + ) , 0) = 0 order by ifnull(last_checked <= last_seen, 0) asc, read_only desc, num_slave_hosts asc - `, DowntimeLostInRecoveryMessage) + `, DowntimeLostInRecoveryMessage, DowntimeSeedInProcessMessage) return log.Errore(err) } if err := ExecDBWriteFunc(writeFunc); err != nil { diff --git a/go/inst/instance_dao.go b/go/inst/instance_dao.go index c2c5979b6..e91347124 100644 --- a/go/inst/instance_dao.go +++ b/go/inst/instance_dao.go @@ -19,6 +19,7 @@ package inst import ( "bytes" "database/sql" + "encoding/json" "errors" "fmt" "regexp" @@ -67,6 +68,10 @@ func (this InstancesByCountSlaveHosts) Less(i, j int) bool { return len(this[i].SlaveHosts) < len(this[j].SlaveHosts) } +type agentSnapshots struct { + LogicalVolumes []struct{} `json:"LogicalVolumes"` +} + // instanceKeyInformativeClusterName is a non-authoritative cache; used for auditing or general purpose. var instanceKeyInformativeClusterName *cache.Cache var forgetInstanceKeys *cache.Cache @@ -90,10 +95,10 @@ func init() { writeBufferLatency.AddMany([]string{"wait", "write"}) writeBufferLatency.Start("wait") - go initializeInstanceDao() + go InitializeInstanceDao() } -func initializeInstanceDao() { +func InitializeInstanceDao() { config.WaitForConfigurationToBeLoaded() instanceWriteBuffer = make(chan instanceUpdateObject, config.Config.InstanceWriteBufferSize) instanceKeyInformativeClusterName = cache.New(time.Duration(config.Config.InstancePollSeconds/2)*time.Second, time.Second) @@ -604,7 +609,9 @@ func ReadTopologyInstanceBufferable(instanceKey *InstanceKey, bufferWrites bool, replicaKey, err := NewResolveInstanceKey(host, port) if err == nil && replicaKey.IsValid() { - instance.AddReplicaKey(replicaKey) + if !RegexpMatchPatterns(replicaKey.StringCode(), config.Config.DiscoveryIgnoreReplicaHostnameFilters) { + instance.AddReplicaKey(replicaKey) + } foundByShowSlaveHosts = true } return err @@ -632,7 +639,9 @@ func ReadTopologyInstanceBufferable(instanceKey *InstanceKey, bufferWrites bool, logReadTopologyInstanceError(instanceKey, "ResolveHostname: processlist", resolveErr) } replicaKey := InstanceKey{Hostname: cname, Port: instance.Key.Port} - instance.AddReplicaKey(&replicaKey) + if !RegexpMatchPatterns(replicaKey.StringCode(), config.Config.DiscoveryIgnoreReplicaHostnameFilters) { + instance.AddReplicaKey(&replicaKey) + } return err }) @@ -1979,7 +1988,9 @@ func ResolveUnknownMasterHostnameResolves() error { } // ReadCountMySQLSnapshots is a utility method to return registered number of snapshots for a given list of hosts +// as json funcs are only on MySQL 5.7+, so we will unmarshall result into a struct func ReadCountMySQLSnapshots(hostnames []string) (map[string]int, error) { + agentLogicalVolumes := &agentSnapshots{} res := make(map[string]int) if !config.Config.ServeAgentsHttp { return res, nil @@ -1987,7 +1998,7 @@ func ReadCountMySQLSnapshots(hostnames []string) (map[string]int, error) { query := fmt.Sprintf(` select hostname, - count_mysql_snapshots + data from host_agent where @@ -1997,7 +2008,11 @@ func ReadCountMySQLSnapshots(hostnames []string) (map[string]int, error) { `, sqlutils.InClauseStringValues(hostnames)) err := db.QueryOrchestratorRowsMap(query, func(m sqlutils.RowMap) error { - res[m.GetString("hostname")] = m.GetInt("count_mysql_snapshots") + err := json.Unmarshal([]byte(m.GetString("data")), agentLogicalVolumes) + if err != nil { + return log.Errore(err) + } + res[m.GetString("hostname")] = len(agentLogicalVolumes.LogicalVolumes) return nil }) @@ -2067,6 +2082,58 @@ func ReadClusters() (clusterNames []string, err error) { return clusterNames, nil } +// ReadClustersAliases reads names of all known clusters aliases with possible filtering +func ReadClustersAliases(clusterAlias string) (clusterAliases []string, err error) { + var clusters []ClusterInfo + if len(clusterAlias) == 0 { + clusters, err = ReadClustersInfoFiltrable(``, sqlutils.Args()) + } else { + clusters, err = ReadClustersInfoFiltrable(`WHERE alias RLIKE ?`, sqlutils.Args(fmt.Sprintf("^"+clusterAlias))) + } + if err != nil { + return clusterAliases, err + } + for _, clusterInfo := range clusters { + clusterAliases = append(clusterAliases, clusterInfo.ClusterAlias) + } + return clusterAliases, nil +} + +// ReadClustersInfoFiltrable reads names of all known clusters and some aggregated info using specified where condition +func ReadClustersInfoFiltrable(whereCondition string, args []interface{}) ([]ClusterInfo, error) { + clusters := []ClusterInfo{} + + query := fmt.Sprintf(` + select + cluster_name, + count(*) as count_instances, + ifnull(min(alias), cluster_name) as alias, + ifnull(min(domain_name), '') as domain_name + from + database_instance + left join cluster_alias using (cluster_name) + left join cluster_domain_name using (cluster_name) + %s + group by + cluster_name`, whereCondition) + + err := db.QueryOrchestrator(query, args, func(m sqlutils.RowMap) error { + clusterInfo := ClusterInfo{ + ClusterName: m.GetString("cluster_name"), + CountInstances: m.GetUint("count_instances"), + ClusterAlias: m.GetString("alias"), + ClusterDomain: m.GetString("domain_name"), + } + clusterInfo.ApplyClusterAlias() + clusterInfo.ReadRecoveryInfo() + + clusters = append(clusters, clusterInfo) + return nil + }) + + return clusters, err +} + // ReadClusterInfo reads some info about a given cluster func ReadClusterInfo(clusterName string) (*ClusterInfo, error) { clusters, err := ReadClustersInfo(clusterName) diff --git a/go/inst/instance_topology.go b/go/inst/instance_topology.go index 2d75240de..9f2165856 100644 --- a/go/inst/instance_topology.go +++ b/go/inst/instance_topology.go @@ -1338,7 +1338,7 @@ func ErrantGTIDResetMaster(instanceKey *InstanceKey) (instance *Instance, err er // We've just made the destructive operation. Again, allow for retries: for i := 0; i < countRetries; i++ { - err = setGTIDPurged(instance, gtidSubtract) + err = SetGTIDPurged(instance, gtidSubtract) if err == nil { break } diff --git a/go/inst/instance_topology_dao.go b/go/inst/instance_topology_dao.go index b964fa6e5..54925636c 100644 --- a/go/inst/instance_topology_dao.go +++ b/go/inst/instance_topology_dao.go @@ -855,7 +855,7 @@ func ResetMaster(instanceKey *InstanceKey) (*Instance, error) { } // skipQueryClassic skips a query in normal binlog file:pos replication -func setGTIDPurged(instance *Instance, gtidPurged string) error { +func SetGTIDPurged(instance *Instance, gtidPurged string) error { if *config.RuntimeCLIFlags.Noop { return fmt.Errorf("noop: aborting set-gtid-purged operation on %+v; signalling error but nothing went wrong.", instance.Key) } diff --git a/go/inst/instance_utils.go b/go/inst/instance_utils.go index 755a65958..348083d65 100644 --- a/go/inst/instance_utils.go +++ b/go/inst/instance_utils.go @@ -24,6 +24,7 @@ import ( var ( DowntimeLostInRecoveryMessage = "lost-in-recovery" + DowntimeSeedInProcessMessage = "seed-in-process" ) // majorVersionsSortedByCount sorts (major) versions: diff --git a/go/logic/orchestrator.go b/go/logic/orchestrator.go index 107e52b5c..b505098f6 100644 --- a/go/logic/orchestrator.go +++ b/go/logic/orchestrator.go @@ -34,7 +34,7 @@ import ( "github.com/github/orchestrator/go/kv" ometrics "github.com/github/orchestrator/go/metrics" "github.com/github/orchestrator/go/process" - "github.com/github/orchestrator/go/raft" + orcraft "github.com/github/orchestrator/go/raft" "github.com/github/orchestrator/go/util" "github.com/openark/golib/log" "github.com/patrickmn/go-cache" @@ -72,6 +72,8 @@ var recentDiscoveryOperationKeys *cache.Cache var pseudoGTIDPublishCache = cache.New(time.Minute, time.Second) var kvFoundCache = cache.New(10*time.Minute, time.Minute) +var continuousSeedProcessRunning uint32 + func init() { snapshotDiscoveryKeys = make(chan inst.InstanceKey, 10) @@ -617,20 +619,20 @@ func ContinuousDiscovery() { } } -func pollAgent(hostname string) error { - polledAgent, err := agent.GetAgent(hostname) - agent.UpdateAgentLastChecked(hostname) - - if err != nil { - return log.Errore(err) - } - - err = agent.UpdateAgentInfo(hostname, polledAgent) - if err != nil { - return log.Errore(err) +// ContinuousSeedProcess starts an asynchronuous infinite process of active seeds processing +func ContinuousSeedProcess() { + tick := time.Tick(time.Duration(config.Config.SeedProcessIntervalSeconds) * time.Second) + for range tick { + if IsLeader() { + if atomic.CompareAndSwapUint32(&continuousSeedProcessRunning, 0, 1) { + agent.ProcessSeeds() + atomic.StoreUint32(&continuousSeedProcessRunning, 0) + } else { + inst.AuditOperation("process-seeds", nil, "Unable to start seed processing as it is already running") + } + } } - return nil } // ContinuousAgentsPoll starts an asynchronuous infinite process where agents are @@ -641,19 +643,22 @@ func ContinuousAgentsPoll() { go discoverSeededAgents() - tick := time.Tick(config.HealthPollSeconds * time.Second) + tick := time.Tick(time.Duration(config.Config.AgentPollMinutes) * time.Minute) caretakingTick := time.Tick(time.Hour) for range tick { - agentsHosts, _ := agent.ReadOutdatedAgentsHosts() - log.Debugf("outdated agents hosts: %+v", agentsHosts) - for _, hostname := range agentsHosts { - go pollAgent(hostname) + outdatedAgents, _ := agent.ReadOutdatedAgents() + + for _, outdatedAgent := range outdatedAgents { + go func(agent *agent.Agent) { + if err := agent.UpdateAgent(); err != nil { + log.Errore(fmt.Errorf("Unable to update agent data for agent %s: %+v", agent.Info.Hostname, err)) + } + }(outdatedAgent) } // See if we should also forget agents (lower frequency) select { case <-caretakingTick: agent.ForgetLongUnseenAgents() - agent.FailStaleSeeds() default: } } @@ -661,7 +666,7 @@ func ContinuousAgentsPoll() { func discoverSeededAgents() { for seededAgent := range agent.SeededAgents { - instanceKey := &inst.InstanceKey{Hostname: seededAgent.Hostname, Port: int(seededAgent.MySQLPort)} + instanceKey := &inst.InstanceKey{Hostname: seededAgent.Info.Hostname, Port: int(seededAgent.Info.MySQLPort)} go inst.ReadTopologyInstance(instanceKey) } } diff --git a/resources/public/css/jquery-ui.css b/resources/public/css/jquery-ui.css new file mode 100644 index 000000000..53b1271cc --- /dev/null +++ b/resources/public/css/jquery-ui.css @@ -0,0 +1,544 @@ +/*! jQuery UI - v1.11.4 - 2020-03-24 +* http://jqueryui.com +* Includes: core.css, autocomplete.css, menu.css, theme.css +* To view and modify this theme, visit http://jqueryui.com/themeroller/?ffDefault=Arial%2CHelvetica%2Csans-serif&fsDefault=1em&fwDefault=normal&cornerRadius=3px&bgColorHeader=e9e9e9&bgTextureHeader=flat&borderColorHeader=dddddd&fcHeader=333333&iconColorHeader=444444&bgColorContent=ffffff&bgTextureContent=flat&borderColorContent=dddddd&fcContent=333333&iconColorContent=444444&bgColorDefault=f6f6f6&bgTextureDefault=flat&borderColorDefault=c5c5c5&fcDefault=454545&iconColorDefault=777777&bgColorHover=ededed&bgTextureHover=flat&borderColorHover=cccccc&fcHover=2b2b2b&iconColorHover=555555&bgColorActive=007fff&bgTextureActive=flat&borderColorActive=003eff&fcActive=ffffff&iconColorActive=ffffff&bgColorHighlight=fffa90&bgTextureHighlight=flat&borderColorHighlight=dad55e&fcHighlight=777620&iconColorHighlight=777620&bgColorError=fddfdf&bgTextureError=flat&borderColorError=f1a899&fcError=5f3f3f&iconColorError=cc0000&bgColorOverlay=aaaaaa&bgTextureOverlay=flat&bgImgOpacityOverlay=0&opacityOverlay=30&bgColorShadow=666666&bgTextureShadow=flat&bgImgOpacityShadow=0&opacityShadow=30&thicknessShadow=5px&offsetTopShadow=0px&offsetLeftShadow=0px&cornerRadiusShadow=8px +* Copyright jQuery Foundation and other contributors; Licensed MIT */ + +/* Layout helpers +----------------------------------*/ +.ui-helper-hidden { + display: none; +} +.ui-helper-hidden-accessible { + border: 0; + clip: rect(0 0 0 0); + height: 1px; + margin: -1px; + overflow: hidden; + padding: 0; + position: absolute; + width: 1px; +} +.ui-helper-reset { + margin: 0; + padding: 0; + border: 0; + outline: 0; + line-height: 1.3; + text-decoration: none; + font-size: 100%; + list-style: none; +} +.ui-helper-clearfix:before, +.ui-helper-clearfix:after { + content: ""; + display: table; + border-collapse: collapse; +} +.ui-helper-clearfix:after { + clear: both; +} +.ui-helper-clearfix { + min-height: 0; /* support: IE7 */ +} +.ui-helper-zfix { + width: 100%; + height: 100%; + top: 0; + left: 0; + position: absolute; + opacity: 0; + filter:Alpha(Opacity=0); /* support: IE8 */ +} + +.ui-front { + z-index: 100; +} + + +/* Interaction Cues +----------------------------------*/ +.ui-state-disabled { + cursor: default !important; +} + + +/* Icons +----------------------------------*/ + +/* states and images */ +.ui-icon { + display: block; + text-indent: -99999px; + overflow: hidden; + background-repeat: no-repeat; +} + + +/* Misc visuals +----------------------------------*/ + +/* Overlays */ +.ui-widget-overlay { + position: fixed; + top: 0; + left: 0; + width: 100%; + height: 100%; +} +.ui-autocomplete { + position: absolute; + top: 0; + left: 0; + cursor: default; +} +.ui-menu { + list-style: none; + padding: 0; + margin: 0; + display: block; + outline: none; +} +.ui-menu .ui-menu { + position: absolute; +} +.ui-menu .ui-menu-item { + position: relative; + margin: 0; + padding: 3px 1em 3px .4em; + cursor: pointer; + min-height: 0; /* support: IE7 */ + /* support: IE10, see #8844 */ + list-style-image: url("data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"); +} +.ui-menu .ui-menu-divider { + margin: 5px 0; + height: 0; + font-size: 0; + line-height: 0; + border-width: 1px 0 0 0; +} +.ui-menu .ui-state-focus, +.ui-menu .ui-state-active { + margin: -1px; +} + +/* icon support */ +.ui-menu-icons { + position: relative; +} +.ui-menu-icons .ui-menu-item { + padding-left: 2em; +} + +/* left-aligned */ +.ui-menu .ui-icon { + position: absolute; + top: 0; + bottom: 0; + left: .2em; + margin: auto 0; +} + +/* right-aligned */ +.ui-menu .ui-menu-icon { + left: auto; + right: 0; +} + +/* Component containers +----------------------------------*/ +.ui-widget { + font-family: Arial,Helvetica,sans-serif; + font-size: 1em; +} +.ui-widget .ui-widget { + font-size: 1em; +} +.ui-widget input, +.ui-widget select, +.ui-widget textarea, +.ui-widget button { + font-family: Arial,Helvetica,sans-serif; + font-size: 1em; +} +.ui-widget-content { + border: 1px solid #dddddd; + background: #ffffff; + color: #333333; +} +.ui-widget-content a { + color: #333333; +} +.ui-widget-header { + border: 1px solid #dddddd; + background: #e9e9e9; + color: #333333; + font-weight: bold; +} +.ui-widget-header a { + color: #333333; +} + +/* Interaction states +----------------------------------*/ +.ui-state-default, +.ui-widget-content .ui-state-default, +.ui-widget-header .ui-state-default { + border: 1px solid #c5c5c5; + background: #f6f6f6; + font-weight: normal; + color: #454545; +} +.ui-state-default a, +.ui-state-default a:link, +.ui-state-default a:visited { + color: #454545; + text-decoration: none; +} +.ui-state-hover, +.ui-widget-content .ui-state-hover, +.ui-widget-header .ui-state-hover, +.ui-state-focus, +.ui-widget-content .ui-state-focus, +.ui-widget-header .ui-state-focus { + border: 1px solid #cccccc; + background: #ededed; + font-weight: normal; + color: #2b2b2b; +} +.ui-state-hover a, +.ui-state-hover a:hover, +.ui-state-hover a:link, +.ui-state-hover a:visited, +.ui-state-focus a, +.ui-state-focus a:hover, +.ui-state-focus a:link, +.ui-state-focus a:visited { + color: #2b2b2b; + text-decoration: none; +} +.ui-state-active, +.ui-widget-content .ui-state-active, +.ui-widget-header .ui-state-active { + border: 1px solid #003eff; + background: #007fff; + font-weight: normal; + color: #ffffff; +} +.ui-state-active a, +.ui-state-active a:link, +.ui-state-active a:visited { + color: #ffffff; + text-decoration: none; +} + +/* Interaction Cues +----------------------------------*/ +.ui-state-highlight, +.ui-widget-content .ui-state-highlight, +.ui-widget-header .ui-state-highlight { + border: 1px solid #dad55e; + background: #fffa90; + color: #777620; +} +.ui-state-highlight a, +.ui-widget-content .ui-state-highlight a, +.ui-widget-header .ui-state-highlight a { + color: #777620; +} +.ui-state-error, +.ui-widget-content .ui-state-error, +.ui-widget-header .ui-state-error { + border: 1px solid #f1a899; + background: #fddfdf; + color: #5f3f3f; +} +.ui-state-error a, +.ui-widget-content .ui-state-error a, +.ui-widget-header .ui-state-error a { + color: #5f3f3f; +} +.ui-state-error-text, +.ui-widget-content .ui-state-error-text, +.ui-widget-header .ui-state-error-text { + color: #5f3f3f; +} +.ui-priority-primary, +.ui-widget-content .ui-priority-primary, +.ui-widget-header .ui-priority-primary { + font-weight: bold; +} +.ui-priority-secondary, +.ui-widget-content .ui-priority-secondary, +.ui-widget-header .ui-priority-secondary { + opacity: .7; + filter:Alpha(Opacity=70); /* support: IE8 */ + font-weight: normal; +} +.ui-state-disabled, +.ui-widget-content .ui-state-disabled, +.ui-widget-header .ui-state-disabled { + opacity: .35; + filter:Alpha(Opacity=35); /* support: IE8 */ + background-image: none; +} +.ui-state-disabled .ui-icon { + filter:Alpha(Opacity=35); /* support: IE8 - See #6059 */ +} + +/* Icons +----------------------------------*/ + +/* states and images */ +.ui-icon { + width: 16px; + height: 16px; +} +.ui-icon, +.ui-widget-content .ui-icon { + background-image: url("images/ui-icons_444444_256x240.png"); +} +.ui-widget-header .ui-icon { + background-image: url("images/ui-icons_444444_256x240.png"); +} +.ui-state-default .ui-icon { + background-image: url("images/ui-icons_777777_256x240.png"); +} +.ui-state-hover .ui-icon, +.ui-state-focus .ui-icon { + background-image: url("images/ui-icons_555555_256x240.png"); +} +.ui-state-active .ui-icon { + background-image: url("images/ui-icons_ffffff_256x240.png"); +} +.ui-state-highlight .ui-icon { + background-image: url("images/ui-icons_777620_256x240.png"); +} +.ui-state-error .ui-icon, +.ui-state-error-text .ui-icon { + background-image: url("images/ui-icons_cc0000_256x240.png"); +} + +/* positioning */ +.ui-icon-blank { background-position: 16px 16px; } +.ui-icon-carat-1-n { background-position: 0 0; } +.ui-icon-carat-1-ne { background-position: -16px 0; } +.ui-icon-carat-1-e { background-position: -32px 0; } +.ui-icon-carat-1-se { background-position: -48px 0; } +.ui-icon-carat-1-s { background-position: -64px 0; } +.ui-icon-carat-1-sw { background-position: -80px 0; } +.ui-icon-carat-1-w { background-position: -96px 0; } +.ui-icon-carat-1-nw { background-position: -112px 0; } +.ui-icon-carat-2-n-s { background-position: -128px 0; } +.ui-icon-carat-2-e-w { background-position: -144px 0; } +.ui-icon-triangle-1-n { background-position: 0 -16px; } +.ui-icon-triangle-1-ne { background-position: -16px -16px; } +.ui-icon-triangle-1-e { background-position: -32px -16px; } +.ui-icon-triangle-1-se { background-position: -48px -16px; } +.ui-icon-triangle-1-s { background-position: -64px -16px; } +.ui-icon-triangle-1-sw { background-position: -80px -16px; } +.ui-icon-triangle-1-w { background-position: -96px -16px; } +.ui-icon-triangle-1-nw { background-position: -112px -16px; } +.ui-icon-triangle-2-n-s { background-position: -128px -16px; } +.ui-icon-triangle-2-e-w { background-position: -144px -16px; } +.ui-icon-arrow-1-n { background-position: 0 -32px; } +.ui-icon-arrow-1-ne { background-position: -16px -32px; } +.ui-icon-arrow-1-e { background-position: -32px -32px; } +.ui-icon-arrow-1-se { background-position: -48px -32px; } +.ui-icon-arrow-1-s { background-position: -64px -32px; } +.ui-icon-arrow-1-sw { background-position: -80px -32px; } +.ui-icon-arrow-1-w { background-position: -96px -32px; } +.ui-icon-arrow-1-nw { background-position: -112px -32px; } +.ui-icon-arrow-2-n-s { background-position: -128px -32px; } +.ui-icon-arrow-2-ne-sw { background-position: -144px -32px; } +.ui-icon-arrow-2-e-w { background-position: -160px -32px; } +.ui-icon-arrow-2-se-nw { background-position: -176px -32px; } +.ui-icon-arrowstop-1-n { background-position: -192px -32px; } +.ui-icon-arrowstop-1-e { background-position: -208px -32px; } +.ui-icon-arrowstop-1-s { background-position: -224px -32px; } +.ui-icon-arrowstop-1-w { background-position: -240px -32px; } +.ui-icon-arrowthick-1-n { background-position: 0 -48px; } +.ui-icon-arrowthick-1-ne { background-position: -16px -48px; } +.ui-icon-arrowthick-1-e { background-position: -32px -48px; } +.ui-icon-arrowthick-1-se { background-position: -48px -48px; } +.ui-icon-arrowthick-1-s { background-position: -64px -48px; } +.ui-icon-arrowthick-1-sw { background-position: -80px -48px; } +.ui-icon-arrowthick-1-w { background-position: -96px -48px; } +.ui-icon-arrowthick-1-nw { background-position: -112px -48px; } +.ui-icon-arrowthick-2-n-s { background-position: -128px -48px; } +.ui-icon-arrowthick-2-ne-sw { background-position: -144px -48px; } +.ui-icon-arrowthick-2-e-w { background-position: -160px -48px; } +.ui-icon-arrowthick-2-se-nw { background-position: -176px -48px; } +.ui-icon-arrowthickstop-1-n { background-position: -192px -48px; } +.ui-icon-arrowthickstop-1-e { background-position: -208px -48px; } +.ui-icon-arrowthickstop-1-s { background-position: -224px -48px; } +.ui-icon-arrowthickstop-1-w { background-position: -240px -48px; } +.ui-icon-arrowreturnthick-1-w { background-position: 0 -64px; } +.ui-icon-arrowreturnthick-1-n { background-position: -16px -64px; } +.ui-icon-arrowreturnthick-1-e { background-position: -32px -64px; } +.ui-icon-arrowreturnthick-1-s { background-position: -48px -64px; } +.ui-icon-arrowreturn-1-w { background-position: -64px -64px; } +.ui-icon-arrowreturn-1-n { background-position: -80px -64px; } +.ui-icon-arrowreturn-1-e { background-position: -96px -64px; } +.ui-icon-arrowreturn-1-s { background-position: -112px -64px; } +.ui-icon-arrowrefresh-1-w { background-position: -128px -64px; } +.ui-icon-arrowrefresh-1-n { background-position: -144px -64px; } +.ui-icon-arrowrefresh-1-e { background-position: -160px -64px; } +.ui-icon-arrowrefresh-1-s { background-position: -176px -64px; } +.ui-icon-arrow-4 { background-position: 0 -80px; } +.ui-icon-arrow-4-diag { background-position: -16px -80px; } +.ui-icon-extlink { background-position: -32px -80px; } +.ui-icon-newwin { background-position: -48px -80px; } +.ui-icon-refresh { background-position: -64px -80px; } +.ui-icon-shuffle { background-position: -80px -80px; } +.ui-icon-transfer-e-w { background-position: -96px -80px; } +.ui-icon-transferthick-e-w { background-position: -112px -80px; } +.ui-icon-folder-collapsed { background-position: 0 -96px; } +.ui-icon-folder-open { background-position: -16px -96px; } +.ui-icon-document { background-position: -32px -96px; } +.ui-icon-document-b { background-position: -48px -96px; } +.ui-icon-note { background-position: -64px -96px; } +.ui-icon-mail-closed { background-position: -80px -96px; } +.ui-icon-mail-open { background-position: -96px -96px; } +.ui-icon-suitcase { background-position: -112px -96px; } +.ui-icon-comment { background-position: -128px -96px; } +.ui-icon-person { background-position: -144px -96px; } +.ui-icon-print { background-position: -160px -96px; } +.ui-icon-trash { background-position: -176px -96px; } +.ui-icon-locked { background-position: -192px -96px; } +.ui-icon-unlocked { background-position: -208px -96px; } +.ui-icon-bookmark { background-position: -224px -96px; } +.ui-icon-tag { background-position: -240px -96px; } +.ui-icon-home { background-position: 0 -112px; } +.ui-icon-flag { background-position: -16px -112px; } +.ui-icon-calendar { background-position: -32px -112px; } +.ui-icon-cart { background-position: -48px -112px; } +.ui-icon-pencil { background-position: -64px -112px; } +.ui-icon-clock { background-position: -80px -112px; } +.ui-icon-disk { background-position: -96px -112px; } +.ui-icon-calculator { background-position: -112px -112px; } +.ui-icon-zoomin { background-position: -128px -112px; } +.ui-icon-zoomout { background-position: -144px -112px; } +.ui-icon-search { background-position: -160px -112px; } +.ui-icon-wrench { background-position: -176px -112px; } +.ui-icon-gear { background-position: -192px -112px; } +.ui-icon-heart { background-position: -208px -112px; } +.ui-icon-star { background-position: -224px -112px; } +.ui-icon-link { background-position: -240px -112px; } +.ui-icon-cancel { background-position: 0 -128px; } +.ui-icon-plus { background-position: -16px -128px; } +.ui-icon-plusthick { background-position: -32px -128px; } +.ui-icon-minus { background-position: -48px -128px; } +.ui-icon-minusthick { background-position: -64px -128px; } +.ui-icon-close { background-position: -80px -128px; } +.ui-icon-closethick { background-position: -96px -128px; } +.ui-icon-key { background-position: -112px -128px; } +.ui-icon-lightbulb { background-position: -128px -128px; } +.ui-icon-scissors { background-position: -144px -128px; } +.ui-icon-clipboard { background-position: -160px -128px; } +.ui-icon-copy { background-position: -176px -128px; } +.ui-icon-contact { background-position: -192px -128px; } +.ui-icon-image { background-position: -208px -128px; } +.ui-icon-video { background-position: -224px -128px; } +.ui-icon-script { background-position: -240px -128px; } +.ui-icon-alert { background-position: 0 -144px; } +.ui-icon-info { background-position: -16px -144px; } +.ui-icon-notice { background-position: -32px -144px; } +.ui-icon-help { background-position: -48px -144px; } +.ui-icon-check { background-position: -64px -144px; } +.ui-icon-bullet { background-position: -80px -144px; } +.ui-icon-radio-on { background-position: -96px -144px; } +.ui-icon-radio-off { background-position: -112px -144px; } +.ui-icon-pin-w { background-position: -128px -144px; } +.ui-icon-pin-s { background-position: -144px -144px; } +.ui-icon-play { background-position: 0 -160px; } +.ui-icon-pause { background-position: -16px -160px; } +.ui-icon-seek-next { background-position: -32px -160px; } +.ui-icon-seek-prev { background-position: -48px -160px; } +.ui-icon-seek-end { background-position: -64px -160px; } +.ui-icon-seek-start { background-position: -80px -160px; } +/* ui-icon-seek-first is deprecated, use ui-icon-seek-start instead */ +.ui-icon-seek-first { background-position: -80px -160px; } +.ui-icon-stop { background-position: -96px -160px; } +.ui-icon-eject { background-position: -112px -160px; } +.ui-icon-volume-off { background-position: -128px -160px; } +.ui-icon-volume-on { background-position: -144px -160px; } +.ui-icon-power { background-position: 0 -176px; } +.ui-icon-signal-diag { background-position: -16px -176px; } +.ui-icon-signal { background-position: -32px -176px; } +.ui-icon-battery-0 { background-position: -48px -176px; } +.ui-icon-battery-1 { background-position: -64px -176px; } +.ui-icon-battery-2 { background-position: -80px -176px; } +.ui-icon-battery-3 { background-position: -96px -176px; } +.ui-icon-circle-plus { background-position: 0 -192px; } +.ui-icon-circle-minus { background-position: -16px -192px; } +.ui-icon-circle-close { background-position: -32px -192px; } +.ui-icon-circle-triangle-e { background-position: -48px -192px; } +.ui-icon-circle-triangle-s { background-position: -64px -192px; } +.ui-icon-circle-triangle-w { background-position: -80px -192px; } +.ui-icon-circle-triangle-n { background-position: -96px -192px; } +.ui-icon-circle-arrow-e { background-position: -112px -192px; } +.ui-icon-circle-arrow-s { background-position: -128px -192px; } +.ui-icon-circle-arrow-w { background-position: -144px -192px; } +.ui-icon-circle-arrow-n { background-position: -160px -192px; } +.ui-icon-circle-zoomin { background-position: -176px -192px; } +.ui-icon-circle-zoomout { background-position: -192px -192px; } +.ui-icon-circle-check { background-position: -208px -192px; } +.ui-icon-circlesmall-plus { background-position: 0 -208px; } +.ui-icon-circlesmall-minus { background-position: -16px -208px; } +.ui-icon-circlesmall-close { background-position: -32px -208px; } +.ui-icon-squaresmall-plus { background-position: -48px -208px; } +.ui-icon-squaresmall-minus { background-position: -64px -208px; } +.ui-icon-squaresmall-close { background-position: -80px -208px; } +.ui-icon-grip-dotted-vertical { background-position: 0 -224px; } +.ui-icon-grip-dotted-horizontal { background-position: -16px -224px; } +.ui-icon-grip-solid-vertical { background-position: -32px -224px; } +.ui-icon-grip-solid-horizontal { background-position: -48px -224px; } +.ui-icon-gripsmall-diagonal-se { background-position: -64px -224px; } +.ui-icon-grip-diagonal-se { background-position: -80px -224px; } + + +/* Misc visuals +----------------------------------*/ + +/* Corner radius */ +.ui-corner-all, +.ui-corner-top, +.ui-corner-left, +.ui-corner-tl { + border-top-left-radius: 3px; +} +.ui-corner-all, +.ui-corner-top, +.ui-corner-right, +.ui-corner-tr { + border-top-right-radius: 3px; +} +.ui-corner-all, +.ui-corner-bottom, +.ui-corner-left, +.ui-corner-bl { + border-bottom-left-radius: 3px; +} +.ui-corner-all, +.ui-corner-bottom, +.ui-corner-right, +.ui-corner-br { + border-bottom-right-radius: 3px; +} + +/* Overlays */ +.ui-widget-overlay { + background: #aaaaaa; + opacity: .3; + filter: Alpha(Opacity=30); /* support: IE8 */ +} +.ui-widget-shadow { + margin: 0px 0 0 0px; + padding: 5px; + background: #666666; + opacity: .3; + filter: Alpha(Opacity=30); /* support: IE8 */ + border-radius: 8px; +} diff --git a/resources/public/css/orchestrator.css b/resources/public/css/orchestrator.css index 1a9987da6..d0fb74267 100644 --- a/resources/public/css/orchestrator.css +++ b/resources/public/css/orchestrator.css @@ -994,3 +994,184 @@ body { .instance.instance-search p { margin: 8px 0px; } + +.table_actions { + float: right; +} + +#seeds_new_seed #agent_filter_button .input-group .form-control:last-child.agent-filter #agent_start_seed_button .input-group .form-control:not(:first-child):not(:last-child).agent-filter { + border-top-left-radius: 4px; + border-top-right-radius: 4px; + border-bottom-left-radius: 4px; + border-bottom-right-radius: 4px; +} + +.agent-filter-group { + margin-left: 60%; + margin-bottom: 15px; +} + +.agent-filter-empty-space { + width:5px; + padding-left:0px; + padding-right:0px; + border:none; + background-color: #fff; +} + +.new-seed-dropdown { + min-width: 300px; +} + +table.agents-table > tbody > tr > td { + vertical-align: middle; +} + +table.agents-table > thead > tr > th { + vertical-align: middle; +} + +.error-log-large-width { + margin: 0 auto; + width: 90%; +} + +table.agent-table > tbody > tr > td:first-child { + width: 60% !important; + vertical-align: middle; +} + +.ui-menu-item { + margin: 0 0 5px 0; +} + +.ui-autocomplete { + cursor: default; + max-height: 300px; + overflow-y: auto; + overflow-x: hidden; + position: absolute; + top: 100%; + left: 0; + z-index: 1000; + float: left; + display: none; + min-width: 160px; + _width: 160px; + padding: 4px 20px 0px 4px; + margin: 2px 0 0 0; + list-style: none; + background-color: #ffffff; + border-color: #ccc; + border-color: rgba(0, 0, 0, 0.2); + border-style: solid; + border-width: 1px; + -webkit-border-radius: 5px; + -moz-border-radius: 5px; + border-radius: 5px; + -webkit-box-shadow: 0 5px 10px rgba(0, 0, 0, 0.2); + -moz-box-shadow: 0 5px 10px rgba(0, 0, 0, 0.2); + box-shadow: 0 5px 10px rgba(0, 0, 0, 0.2); + -webkit-background-clip: padding-box; + -moz-background-clip: padding; + background-clip: padding-box; + *border-right-width: 2px; + *border-bottom-width: 2px; +} + +.ui-autocomplete .ui-menu-item > a.ui-corner-all { + display: block; + padding: 3px 15px; + clear: both; + font-weight: normal; + line-height: 18px; + color: #555555; + white-space: nowrap; +} +.ui-autocomplete .ui-menu-item > a.ui-corner-all.ui-state-hover, .ui-autocomplete .ui-menu-item > a.ui-corner-all.ui-state-active { + color: #ffffff; + text-decoration: none; + background-color: #0088cc; + border-radius: 0px; + -webkit-border-radius: 0px; + -moz-border-radius: 0px; + background-image: none; +} + +.ui-helper-hidden-accessible { + display: none; +} + +#agent_new-seed-from { + margin-bottom: 10px; +} + +#agent_seed_hostname_input { + width:100%; +} + +#new-seed-selector { + width:35%; +} + +#agent_seed_side, #agent_seed_hostname, #agent_seed_seed_method { + font-size:13px; +} + +#agent_seed_method_container { + height: 34px; + margin-bottom: 10px; +} + +#agent_seed_method_label { + width: 35%; + height: 34px; + float: left; + display: flex; + align-items: center; + font-weight: 400; + font-size:13px +} + +#agent_seed_seed_method { + width: 64%; + float: right; +} + +#agent_start_seed_button_container { + width:100%; + text-align: right; +} + +#seeds_filter, #agent_execute_command_button, #agent_seed_hostname { + border-top-left-radius: 4px; + border-top-right-radius: 4px; + border-bottom-left-radius: 4px; + border-bottom-right-radius: 4px; +} + +#agents_filter { + width: 50%; + left: 50%; + border-top-left-radius: 4px; + border-top-right-radius: 4px; + border-bottom-left-radius: 4px; + border-bottom-right-radius: 4px; +} + +#agents_filter_type { + width:70%; + left:100% +} + +#seeds_new_seed_form_group { + margin-left: 10%; + margin-right: 10%; + margin-top: 5px; + margin-bottom:5px +} + +#seeds_start_seed { + margin-top:5px; + margin-left:61% +} \ No newline at end of file diff --git a/resources/public/js/agent.js b/resources/public/js/agent.js index c30a19409..32fcf8e69 100644 --- a/resources/public/js/agent.js +++ b/resources/public/js/agent.js @@ -2,75 +2,148 @@ $(document).ready(function() { showLoader(); var hasActiveSeeds = false; + var currentAgentSeedMethods = []; + var targetAgentCluster = ""; + var sourceAgentCluster = ""; $.get(appUrl("/api/agent/" + currentAgentHost()), function(agent) { showLoader(); - agent.AvailableLocalSnapshots || (agent.AvailableLocalSnapshots = []) - agent.AvailableSnapshots || (agent.AvailableSnapshots = []) + agent.AvailableLocalSnapshots || (agent.AvailableLocalSnapshots = []); + agent.AvailableSnapshots || (agent.AvailableSnapshots = []); displayAgent(agent); - }, "json"); + }, "json").fail(function(operationResult) { + hideLoader(); + if (operationResult.responseJSON.Code == "ERROR") { + addAlert(operationResult.responseJSON.Message) + return + } + }); $.get(appUrl("/api/agent-active-seeds/" + currentAgentHost()), function(activeSeeds) { showLoader(); activeSeeds.forEach(function(activeSeed) { - appendSeedDetails(activeSeed, "[data-agent=active_seeds]"); + appendSeedDetails(activeSeed, "#agent_active_seeds"); }); if (activeSeeds.length == 0) { - $("div.active_seeds").parent().hide(); - $("div.seed_states").parent().hide(); + $("#agent_active_seeds_container").parent().hide(); + $("#agent_active_seeds_states_container").parent().hide(); } if (activeSeeds.length > 0) { hasActiveSeeds = true; activateRefreshTimer(); - $.get(appUrl("/api/agent-seed-states/" + activeSeeds[0].SeedId), function(seedStates) { + $.get(appUrl("/api/agent-seed-states/" + activeSeeds[0].SeedID), function(seedStates) { showLoader(); seedStates.forEach(function(seedState) { - appendSeedState(seedState); + appendSeedState(seedState, "#seed_states"); }); }, "json"); } }, "json"); + $.get(appUrl("/api/agent-recent-seeds/" + currentAgentHost()), function(recentSeeds) { showLoader(); recentSeeds.forEach(function(recentSeed) { - appendSeedDetails(recentSeed, "[data-agent=recent_seeds]"); + appendSeedDetails(recentSeed, "#agent_recent_seeds"); }); if (recentSeeds.length == 0) { - $("div.recent_seeds").parent().hide(); + $("#agent_recent_seeds_container").parent().hide(); } }, "json"); function displayAgent(agent) { - if (!agent.Hostname) { - $("[data-agent=hostname]").html('Not found'); - return; + currentAgentSeedMethods = Object.keys(agent.Data.AvailiableSeedMethods); + if (!(Object.keys(agent.Data.AvailiableSeedMethods).includes("LVM"))) { + $("#agent_snapshots_info_container").hide(); } - $("[data-agent=hostname]").html(agent.Hostname) - $("[data-agent=hostname_search]").html( - '' + agent.Hostname + '' + '
' + if (Object.keys(agent.Data.MySQLDatabases).length == 0) { + $("#agent_databases_container").hide(); + } + $("#agent_hostname").html(agent.Info.Hostname); + $("#agent_hostname").html(agent.Info.Hostname + '
' ); - $("[data-agent=port]").html(agent.Port) - $("[data-agent=last_submitted]").html(agent.LastSubmitted) + $("#agent_port").html(agent.Info.Port); + $("#agent_cluster_name").html('' + agent.ClusterAlias + '')) + sourceAgentCluster = agent.ClusterAlias + $("#agent_status").html(agent.Status) + $("#agent_last_seen").html(moment.utc(agent.LastSeen).format("YYYY-MM-DD HH:mm:ss")) + $("#agent_os").html(agent.Data.OsName); + $("#agent_cpu").html(agent.Data.NumCPU); + $("#agent_ram").html(toHumanFormat(agent.Data.MemTotal)); - var mySQLStatus = "" + agent.MySQLRunning + '
' + - (agent.MySQLRunning ? '' : - '') + + $("#agent_avaliable_seed_methods").html(Object.keys(agent.Data.AvailiableSeedMethods).join()) + if (agent.Data.AgentCommands !== null) { + $("#agent_commands").html('
'); + agent.Data.AgentCommands.forEach(function(command) { + $('