[Bug]: Pagerduty Contact Point reports invalid routing key

[cavila-evoliq]

Terraform Version

1.9.7

Terraform Grafana Provider Version

3.9.0

Grafana Version

10.0.3

Affected Resource(s)

grafana_contact_point

Terraform Configuration Files

resource "pagerduty_service" "service" {
  name                    = var.service_name
  auto_resolve_timeout    = 14400
  acknowledgement_timeout = 600
  escalation_policy       = data.pagerduty_escalation_policy.policy.id
  alert_creation          = "create_alerts_and_incidents"
}


resource "pagerduty_service_integration" "apiv2" {
  name    = "API V2"
  type    = "events_api_v2_inbound_integration"
  service = pagerduty_service.service.id
}

data "pagerduty_escalation_policy" "policy" {
  name  = var.escalation_policy_name
}

resource "grafana_contact_point" "my_contact_point" {
  name = var.service_name

  pagerduty {
    integration_key = pagerduty_service_integration.apiv2.integration_key
  }
}

Expected Behavior

• Contact Point is created
• Testing the contact point sends a page

Actual Behavior

• Contact point is created
• Testing the page returns 400 Bad Request. Logs indicate the returned error is that the issue is with the routing key

Steps to Reproduce

1. terraform apply
2. Go to the Contact Point in Grafana
3. Test the contact point

Important Factoids

We are hosting Grafana on GKE.
If we take the integration key that was created by PagerDuty via TF and paste it into Grafana via the UI, it works fine.

TF_LOG=DEBUG seems to incidate some errors

2024-10-14T13:55:36.954-0400 [WARN]  Provider "registry.terraform.io/grafana/grafana" produced an invalid plan for grafana_contact_point.my_contact_point, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .disable_provenance: planned value cty.False for a non-computed attribute
grafana_contact_point.my_contact_point: Creating...
2024-10-14T13:55:36.954-0400 [INFO]  Starting apply for grafana_contact_point.my_contact_point
2024-10-14T13:55:36.955-0400 [DEBUG] grafana_contact_point.my_contact_point: applying the planned Create change
2024-10-14T13:55:36.958-0400 [DEBUG] provider.terraform-provider-grafana_v3.9.0: 2024/10/14 13:55:36 [DEBUG] Waiting for state to become: [success]
2024-10-14T13:55:37.308-0400 [WARN]  Provider "provider[\"registry.terraform.io/grafana/grafana\"]" produced an unexpected new value for grafana_contact_point.my_contact_point, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .org_id: was null, but now cty.StringVal("0")
      - .pagerduty: planned set element cty.ObjectVal(map[string]cty.Value{"class":cty.StringVal(""), "client":cty.StringVal(""), "client_url":cty.StringVal(""), "component":cty.StringVal(""), "details":cty.NullVal(cty.Map(cty.String)), "disable_resolve_message":cty.False, "group":cty.StringVal(""), "integration_key":cty.StringVal("redacted"), "settings":cty.NullVal(cty.Map(cty.String)), "severity":cty.StringVal(""), "source":cty.StringVal(""), "summary":cty.StringVal(""), "uid":cty.UnknownVal(cty.String), "url":cty.StringVal("")}) does not correlate with any element in actual
grafana_contact_point.my_contact_point: Creation complete after 0s [id=0:devops-debug]
2024-10-14T13:55:37.314-0400 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2024-10-14T13:55:37.315-0400 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-10-14T13:55:37.318-0400 [INFO]  provider: plugin process exited: plugin=.terraform/providers/registry.terraform.io/grafana/grafana/3.9.0/linux_amd64/terraform-provider-grafana_v3.9.0 id=21552

References

No response

[cavila-evoliq]

As an update to this, we stood up Grafana 11 in a sandbox and I tested my the contact point resource on the off chance it was fixed. Everything works with Grafana 11! That will be our path forward as we had planned to upgrade anyway.