PMM-4172 PT MySQL summary. (#510)

* [PMM-4172] Struct and functions for pt-mysql-summary created

* [PMM-4172] Fixed type name error.

* [PMM-4172] Struct and functions for pt-mysql-summary created

* PMM-6746 psmdb clusters (#498)

* PMM-6746 Implementation of PSMDB Cluster API.

* PMM-6746 Return better error message.

* PMM-6746 Return states and update API.

* PMM-6746 Update API dependencies.

* PMM-6746 remove unused variable.

* PMM-6746 Fix merge conflicts.

* PMM-6746 Rename xtradb to mongodb.

* PMM-6746 Rename pxc to psmdb.

* PMM-6746 Return replicaset in response.

* PMM-6746 Fix List endpoint.

* PMM-6746 Update dependencies.

Co-authored-by: Alexey Palazhchenko <[email protected]>

* PMM-6401 Drop prometheus (#486)

* PMM-6389 Add VictoriaMetrics
Adds two new services - victoriametrics and vmalert
 victoriametrics replaces promtheus as metrics storage,
 scrapes targets.
 vmalert evaluates alerting and recording rules,
 sends notifications to alertmanager.
 Added tests and test data.
 Added wrappers for prometheus service functions.

* added return fast for enabled func, removed typo

* added params model for victoriametrics,
fixed issue with path to prometheus base file,
added external url for grafana explore

* added context to errors,
 removed templating flag,
added description to configuration flag,
removed unneeded struct,
fixed params init,
move baseConfigPath to Constant

* fixed linter error

* changed alertmanagerURL parse behaviour,
fixed formatting issues,
added VM filter to envvars

* PMM-6629 Hack protobuf dependency for saas (#461)

* updated docker image version for tests

* moved victoriametrics cache settings to db settings with envvar override,
changed alertmanager url parsing

* PMM-6588 Return information if DBaaS functionality is enabled in API. (#458)

* PMM-6588 Return information if DBaaS functionality is enabled in API.

* PMM-6588 Fix golint.

* PMM-6588 Refactoring.

* PMM-6588 Fix Gopkg.lock.

Co-authored-by: Alexey Palazhchenko <[email protected]>

* fix linter errors

* Update services/prometheus/prometheus.go

Co-authored-by: Nurlan Moldomurov <[email protected]>

* updated parseBool description, fixed linter and build errors

* SAAS-63 Better Starlark sandbox (#462)

* PMM-6629 Hack protobuf dependency for saas

* SAAS-63 Better Starlark sandbox

Squashed percona/pmm-managed#451

* SAAS-63 Temporary hack

* SAAS-63 Correct unit of CPU time

* SAAS-63 Minor changes

* SAAS-63 Revert temporary hack for Travis CI

* SAAS-63 Use the same field names

* SAAS-63 Set limits in main

* SAAS-63 Log stderr

* SAAS-63 Remove version from tests

* SAAS-63 Fix data race

* SAAS-63 Remove unused constant

Co-authored-by: Alexey Palazhchenko <[email protected]>
Co-authored-by: Alexey Palazhchenko <[email protected]>

* PMM-4597 Add default values in JSON APIs (#449)

* removed parseBool func, changed templates for supervisor, fixed linter error

* SAAS-321 Merge PERCONA_TEST_XXX_HOST environment variables (#466)

* SAAS-321 - created a shared func to validate and get SAAS host, panic when old SAAS env variables are passed

* SAAS-321 - using old SAAS env variables if they are provided

* Update services/checks/checks.go

Co-authored-by: Artem Gavrilov <[email protected]>

* Update services/telemetry/telemetry.go

Co-authored-by: Artem Gavrilov <[email protected]>

* Update services/platform/platform.go

Co-authored-by: Artem Gavrilov <[email protected]>

* SAAS-321 - updated services

* SAAS-321 - updated GetSAASHost function

* SAAS-321 - review changes, updated related services tests

* SAAS-321 - review changes

* Update main.go

Co-authored-by: Alexey Palazhchenko <[email protected]>

* SAAS-321 - review changes, using net.SplitHostPort for validation

* SAAS-321 - lock file

* SAAS-321 Refactor to DRY

* SAAS-321 - adding WILL BE REMOVED SOON warning at the Docker logs level, adding tests and FIXME

* SAAS-321 - FIXME review changes

Co-authored-by: Artem Gavrilov <[email protected]>
Co-authored-by: Alexey Palazhchenko <[email protected]>

* PMM-6566 Start DBaaS on PMM server (#469)

* PMM-6566 Add dbaas to supervisord.

* PMM-6567 Fix test.

* PMM-6566 Renamed, pass value, add default value.

* PMM-6566 Add missing empty line at end of the file.

* PMM-6566 Skip test for dbaas-controller if it is not enabled.

* PMM-6566 Fix.

* PMM-6566 Move it to settings.

* PMM-6566 Fix.

* PMM-6566 Remove duplicated dbaas settings property.

* PMM-6567 Required changes.

* PMM-6566 Changes.

* PMM-6566 Fix.

* PMM-6566 Remove empty line.

* PMM-6566 Fix.

* PMM-6566 Required changes.

* PMM-6566 Changes.

* PMM-6566 Required changes.

* PMM-6567 pg_stat_monitor (#457)

* PMM-6460 Add new agent type.

* PMM-6460 Fix new agent type.

* PMM-6460 Deps.

* PMM-6460 Deps.

* PMM-6567 PMM branch change.

* PMM-6567 PMM update.

* PMM-6567 Add skip for prometheus scrape config.

* PMM-6567 Deps.

* PMM-6567 Fix tests for new agent type.

* PMM-6567 Changes.

* PMM-6567 Add missing methods.

* PMM-6567 Deps.

* PMM-6567 Lint.

* PMM-6567 Deps.

* PMM-6567 Deps.

* PMM-6567 Add new metrics to request for QAN-API.

* PMM-6567 Changes.

* PMM-6567 Remove new agent type from database for now.

* PMM-6567 Fix.

* PMM-6567 Required changes.

* PMM-6567 Deps.

* Revert "PMM-6567 Deps."

This reverts commit 52b2b3224701cad2ecefb698564bba85d2d9ca28.

* PMM-6567 Deps.

* PMM-6567 Deps.

* Changed namings

* fixed conflict

* fixed gopkg branch, formating

* reverted docker-compose image, fixed prometheus test, is it issue with promtool?

* Renamed Zero test to FormerZero

* changed naming and added lines breakers,
changed vmalert template
added DisableVMCache option
renamed VM_CACHE_ENABLE to ENABLE_VM_CACHE
changed tests

* small typo fix

* added mocked vmdb and vmalert services to server_test

* PMM-6401 do not run Prometheus if PERCONA_TEST_ENABLE_VM=1 env flag is set

* PMM-6401 properly use feature image with VictoriaMetrics that can read Prometheus data

* updated tests
added force supervisor update
mocked http client for victoriametrics
fixed testdata

* PMM-6401 update feature build

* PMM-6401 drop prometheus

* Test with Go 1.15

* SAAS-315 Pass additional check context (#480)

* SAAS-315 pass additional check context

* SAAS=315 pass funcs

* SAAS-315 Update testdata check

Co-authored-by: Alexey Palazhchenko <[email protected]>
Co-authored-by: Alexey Palazhchenko <[email protected]>

* Allow failures with Go 1.15 for now

* PMM-6401 remove unused function scrapeConfigForPrometheus

* PMM-6401 run victoriametrics on port 9090 as prometheus

* fix conflict

* fixes test, adds prometheus flags for migration simplify

* typo fix

* PMM-6401 update feature build

* PMM-6401 store files related to victoriametrics in logs

* PMM-6401 typo fixes

* PMM-6401 remove unused prometheus flags

* PMM-6401 return error if base prom config contains remote_write section

* PMM-6401 collect prometheus.log file from the previous prometheus installation

* PMM-6401 return back stopsignal=SIGTERM for prometheus

* PMM-6401 update feature build

* PMM-6401 start `promethes` in order to create log file

* fixes logs test, renames prometheus files

* sets prometheus autostart to false

* bumped fb docker image

* Update docker-compose.yml

* PMM-6401 Update old path

* PMM-6401 Add FIXME

* PMM-6401 Add another TODO

Co-authored-by: f41gh7 <[email protected]>
Co-authored-by: Alexey Palazhchenko <[email protected]>
Co-authored-by: Nurlan Moldomurov <[email protected]>
Co-authored-by: Palash Nigam <[email protected]>
Co-authored-by: Alexey Palazhchenko <[email protected]>
Co-authored-by: Vasyl Yurkovych <[email protected]>
Co-authored-by: Artem Gavrilov <[email protected]>
Co-authored-by: JiriCtvrtka <[email protected]>

* PMM-6050 Add external group (#500)

* PMM-6050 Add external group

* PMM-6050 Add external group to service server.

* PMM-6050 Update dep and test.

* PMM-6050 Fix order in test. Add Group to repsonce.

* PMM-6050 Fix test.

* PMM-6050 Improve external group.

* PMM-6050 Moved external_group from service to model.

* PMM-6050 Update dep

* PMM-4172 Fix build.

* PMM-4172 Deps.

* PMM-4172 Endpoint.

* PMM-4172 Add params to request.

* PMM-4172 Changes.

* [PMM-4172] Updated pt-mysql-summary functions comments.

* Use tabs, not spaces (#511)

* [PMM-4172] Updated structures from pmm.

* [PMM-4172] Changed to use DSN.

* [PMM-4172] Removed forgotten debug line.

* [PMM-4172] dep ensure.

* PMM-4172 dep ensure.

* PMM-4172 Functionality to process the pt-mysql-summary request with split params.

* PMM-4172 Changed structure members names.

* PMM-4172 Changed referrence to PMM-2.0.

* PMM-4172 Updated exporters agent filter structure.

* PMM-4172 Dep ensure.

* PMM-4172 Changed function comment.

* PMM-4172 Code review change request changes.

* PMM-4172 Lint complaints.

* PMM-4172 Variable name.

* PMM-4172 Updated comment.

* PMM-4172 Updated Gopkg.toml

* PMM-4172 dep ensure.

* PMM-4172 Dep ensure. Link to revert-623-PMM-7026-revert-pt-mysql-summary.

* PMM-4172 Condition rewritten.

* PMM-4172 Dep ensure.

* Update services/management/grpc/actions_server.go

Co-authored-by: Nurlan Moldomurov <[email protected]>

* PMM-4172 Suggested change.

* PMM-4172 Dep ensure.

* PMM-4172 Suggested changes.

* PMM-4172 dep ensure.

* PMM-4172 dep ensure.

* PMM-4172 Fixed supervisord.go.

* PMM-6874 Add template rule converter (#544)

* PMM-6874 Add template rule converter

* PMM-6874 fix tests

* PMM-6874 use yaml v3

* PMM-6874 remove parallel

* PMM-6874 fix tests

* PMM-6874 fix spellings

* PMM-6874 remove duplicate cutset characters

* PMM-6874 remove whitespace from cutset

* PMM-6874 remove duplicate char

* PMM-6874 Use template

* PMM-6874 err check

* PMM-6874 refactoring

* PMM-6874 Add more testcases

* PMM-6874 rename var

* PMM-6874 fix comment

* PMM-6874 check rule before creating filename

* PMM-7069 Add TODO

* PMM-6874 Use a slice of values

Co-authored-by: Alexey Palazhchenko <[email protected]>

* PMM-7024 Add built-in IA templates (#552)

* Extract global logger setup (#556)

* PMM-4172 Fix comment

* PMM-4172 Add default timeout

* PMM-6891 Extract alerting rule validation function (#557)

* PMM-6891 Extract alerting rule validation function

We need for both integrated and external alerting,
but vmalert.AlertingRules is very external-specific.

* PMM-6891 Format

* PMM-6909 Rule templates API (#541)

* PMM-6908 Add rule templates models

* PMM-6909 Fix table definition

* PMM-6909 Update template model

* PMM-6909 Fix migration

* PMM-6909 Add some helpers for rule templates, update model

* PMM-6909 Rename table

* PMM-6909 Update templates model

* PMM-6909 Implement List and Create templates API methods

* PMM-6909 Add license headers

* PMM-6909 Implelment Delete rule template API method

* PMM-6909 Implelment Change rule template API method

* PMM-6909 Add tests

* PMM-6909 Fix template update

* PMM-6909 Refactoring

* PMM-6909 Update template model, add comments

* PMM-6909 Save original yaml file on template cration/updation

* PMM-6909 Make template parameter range optional

* PMM-6909 Remove debug code

* PMM-6909 Improve template model

* PMM-6909 Cleanup after tests

* PMM-6909 Format

* PMM-6909 Fix linters

* PMM-6909 Add constraints to ia_templates table

* PMM-6909 Refactoring

* Update saas dependency, refactoring

* PMM-6909 Refactoring

* PMM-6909 Refactoring

* PMM-6909 Update API branch

* PMM-6909 Tweak comments

* PMM-6909 Tweak formatting

Co-authored-by: Alexey Palazhchenko <[email protected]>
Co-authored-by: Alexey Palazhchenko <[email protected]>

* PMM-6982 Remove PERCONA_TEST_IA. (#564)

* PMM-7108 Update reform to make it work with Go 1.15 (#561)

* PMM-7108 Update reform to make it work with Go 1.15

* PMM-7108 Use release/2.13

* PMM-6827 DB cluster monitoring. (#515)

* PMM-6827 Add method to get public IP, add new field to settings.

* PMM-6827 Changes in settings API.

* PMM-6827 Add verification of public URL.

* PMM-6827 Deps.

* PMM-6827 Changes.

* PMM-6827 Deps.

* PMM-6827 Changes.

* PMM-6827 Deps.

* PMM-6827 Check for empty public URL.

* PMM-6827 Deps.

* Revert "PMM-6827 Changes."

This reverts commit 5daacc50e0c152d561f1547a77cb937665c8b2ff.

* PMM-6827 Fix.

* PMM-6827 Fix.

* PMM-6827 Rerun.

* Revert "PMM-6827 Rerun."

This reverts commit 1b76d039de2a50ec700976a8c024fcdc4c0149b5.

* PMM-6827 Deps.

* PMM-6827 Fix build.

* PMM-6827 Deps.

* PMM-6827 Remove public address URL check on this side.

* PMM-6827 Deps.

* PMM-6827 Deps.

* PMM-6827 Changes.

* PMM-6827 Deps.

* PMM-6827 Change field name.

* PMM-6827 Remove forget lines.

* PMM-6827 PMM Public address in Settings API.

* update dependencies.

* PMM-6827 Update dependencies.

* PMM-6827 Update dependencies.

* PMM-6827 Update dependencies.

Co-authored-by: Nurlan Moldomurov <[email protected]>

* Log original error (#566)

* PMM-6983 Add force for unregister k8s (#555)

* PMM-6983 Deps.

* PMM-6983 If force is true then register cluster only.

* PMM-6983 Small fix.

* PMM-6983 Deps.

* PMM-6983 Deps.

* PMM-6983 Deps.

* PMM-6983 Deps.

* PMM-4172 Renamed address to host.

* PMM-4172 Fixed merge errors, dep ensure.

* PMM-4172 Merge fix.

* PMM-4172 Merge.

* PMM-4172 Deps.

* PMM-4172 Dep ensure.

* PMM-4172 Created PT timout of 30 s.

* PMM-4172 Removed unwanted file.

* PMM-7383 Remove backup location

* PMM-7383 Remove backup location - service test

* PMM-7383 Remove backup location - helpers test

* PMM-7383 Simplify assert not found

* PMM-7383 assert len 0 -> empty

* PMM-7383 update API

* PMM-7383 Check if location exists before removal

* PMM-4172 Deps.

* PMM-4172 Refactor.

* PMM-4172 Deps.

* PMM-4172 Fix.

* PMM-4172 Fix for REMOTE.

* PMM-4172 Deps.

* PMM-4172 Nice error for remotes nodes.

* PMM-4172 Deps.

* PMM-4172 Lint.

* PMM-4172 Lint.

* PMM-4172 Deps.

* PMM-7383 deps

* PMM-4172 Fix for older clients.

* PMM-4172 Deps.

* PMM-4172 Deps.

* PMM-4172 Deps before merge.

Co-authored-by: Jaroslav Musil <[email protected]>
Co-authored-by: Nurlan Moldomurov <[email protected]>
Co-authored-by: Alexey Palazhchenko <[email protected]>
Co-authored-by: Aliaksandr Valialkin <[email protected]>
Co-authored-by: f41gh7 <[email protected]>
Co-authored-by: Palash Nigam <[email protected]>
Co-authored-by: Alexey Palazhchenko <[email protected]>
Co-authored-by: Vasyl Yurkovych <[email protected]>
Co-authored-by: Artem Gavrilov <[email protected]>
Co-authored-by: Andrii Skomorokhov <[email protected]>
Co-authored-by: Jaroslav Musil <[email protected]>
Co-authored-by: Nurlan Moldomurov <[email protected]>
Co-authored-by: Dasio <[email protected]>

services/agents/registry.go

@@ -1273,7 +1273,7 @@ func (r *Registry) StartMongoDBQueryGetCmdLineOptsAction(ctx context.Context, id
 func (r *Registry) StartPTSummaryAction(ctx context.Context, id, pmmAgentID string) error {
 	aRequest := &agentpb.StartActionRequest{
 		ActionId: id,
-		// Need pass params, even empty, because othervise request's marshal fail.
+		// Requires params to be passed, even empty, othervise request's marshal fail.
 		Params: &agentpb.StartActionRequest_PtSummaryParams{
 			PtSummaryParams: &agentpb.StartActionRequest_PTSummaryParams{},
 		},
@@ -1343,6 +1343,33 @@ func (r *Registry) StartPTMongoDBSummaryAction(ctx context.Context, id, pmmAgent
 	return nil
 }
 
+// StartPTMySQLSummaryAction starts pt-mysql-summary action on the pmm-agent.
+// The pt-mysql-summary's execution may require some of the following params: host, port, socket, username, password.
+func (r *Registry) StartPTMySQLSummaryAction(ctx context.Context, id, pmmAgentID, address string, port uint16, socket, username, password string) error {
+	actionRequest := &agentpb.StartActionRequest{
+		ActionId: id,
+		// Proper params that'll will be passed to the command on the agent's side.
+		Params: &agentpb.StartActionRequest_PtMysqlSummaryParams{
+			PtMysqlSummaryParams: &agentpb.StartActionRequest_PTMySQLSummaryParams{
+				Host:     address,
+				Port:     uint32(port),
+				Socket:   socket,
+				Username: username,
+				Password: password,
+			},
+		},
+		Timeout: defaultPtActionTimeout,
+	}
+
+	pmmAgent, err := r.get(pmmAgentID)
+	if err != nil {
+		return err
+	}
+	pmmAgent.channel.SendRequest(actionRequest)
+
+	return nil
+}
+
 // StopAction stops action with given given id.
 // TODO: Extract it from here: https://jira.percona.com/browse/PMM-4932
 func (r *Registry) StopAction(ctx context.Context, actionID string) error {

services/management/grpc/actions_server.go

@@ -461,6 +461,71 @@ func (s *actionsServer) StartPTMongoDBSummaryAction(ctx context.Context, req *ma
 	return &managementpb.StartPTMongoDBSummaryActionResponse{PmmAgentId: pmmAgentID, ActionId: res.ID}, nil
 }
 
+// StartPTMySQLSummaryAction starts pt-mysql-summary action and returns the pointer to the response message
+//nolint:lll
+func (s *actionsServer) StartPTMySQLSummaryAction(ctx context.Context, req *managementpb.StartPTMySQLSummaryActionRequest) (*managementpb.StartPTMySQLSummaryActionResponse, error) {
+	service, err := models.FindServiceByID(s.db.Querier, req.ServiceId)
+	if err != nil {
+		return nil, err
+	}
+
+	node, err := models.FindNodeByID(s.db.Querier, service.NodeID)
+	if err != nil {
+		return nil, err
+	}
+
+	var pmmAgentID string
+	switch node.NodeType {
+	case models.RemoteNodeType:
+		// Remove this error after: https://jira.percona.com/browse/PMM-7562
+		return nil, status.Errorf(codes.FailedPrecondition, "PTMySQL Summary doesnt working with remote node yet")
+
+		//pmmAgentID = models.PMMServerAgentID
+	default:
+		pmmAgents, err := models.FindPMMAgentsRunningOnNode(s.db.Querier, service.NodeID)
+		if err != nil {
+			return nil, status.Errorf(codes.NotFound, "No pmm-agent running node %s", service.NodeID)
+		}
+		pmmAgents = models.FindPMMAgentsForVersion(s.l, pmmAgents, pmmAgent2150)
+		if len(pmmAgents) == 0 {
+			return nil, status.Error(codes.FailedPrecondition, "all available agents are outdated")
+		}
+		pmmAgentID, err = models.FindPmmAgentIDToRunAction(req.PmmAgentId, pmmAgents)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	res, err := models.CreateActionResult(s.db.Querier, pmmAgentID)
+	if err != nil {
+		return nil, err
+	}
+
+	agentFilter := models.AgentFilters{PMMAgentID: "", NodeID: "",
+		ServiceID: req.ServiceId, AgentType: pointerToAgentType(models.MySQLdExporterType)}
+	mysqldExporters, err := models.FindAgents(s.db.Querier, agentFilter)
+	if err != nil {
+		return nil, err
+	}
+
+	exportersCount := len(mysqldExporters)
+	if exportersCount < 1 {
+		return nil, status.Errorf(codes.FailedPrecondition, "No mysql exporter")
+	}
+	if exportersCount > 1 {
+		return nil, status.Errorf(codes.FailedPrecondition, "Found more than one mysql exporter")
+	}
+
+	err = s.r.StartPTMySQLSummaryAction(ctx, res.ID, pmmAgentID, pointer.GetString(service.Address), pointer.GetUint16(service.Port),
+		pointer.GetString(service.Socket), pointer.GetString(mysqldExporters[0].Username),
+		pointer.GetString(mysqldExporters[0].Password))
+	if err != nil {
+		return nil, err
+	}
+
+	return &managementpb.StartPTMySQLSummaryActionResponse{PmmAgentId: pmmAgentID, ActionId: res.ID}, nil
+}
+
 // CancelAction stops an Action.
 func (s *actionsServer) CancelAction(ctx context.Context, req *managementpb.CancelActionRequest) (*managementpb.CancelActionResponse, error) {
 	ar, err := models.FindActionResultByID(s.db.Querier, req.ActionId)