terraform-google-pubsub-topic

A Terraform module for creating and managing a Cloud Pub/Sub resource on Google Cloud

This module supports Terraform version 1 and is compatible with the Terraform Google Provider version 5

This module is part of our Infrastructure as Code (IaC) framework that enables our users and customers to easily deploy and manage reusable, secure, and production-grade cloud infrastructure.

• Module Features
• Getting Started
• Module Argument Reference
  • Main Resource Configuration
  • Extended Resource Configuration
  • Module Configuration
• Module Outputs
• External Documentation
  • Google Documentation
  • Terraform GCP Provider Documentation
• Module Versioning
  • Backwards compatibility in 0.0.z and 0.y.z version
• About Mineiros
• Reporting Issues
• Contributing
• Makefile Targets
• License

Module Features

This module implements the following Terraform resources

• google_pubsub_topic
• google_pubsub_schema

and supports additional features of the following modules:

• mineiros-io/terraform-google-pubsub-topic-iam
• mineiros-io/terraform-google-pubsub-subscription

Getting Started

Most common usage of the module:

module "terraform-google-pubsub-topic" {
  source = "github.com/mineiros-io/terraform-google-pubsub-topic?ref=v1.2.0"

  name    = "pub-sub-topic-name"
  project = "id-of-project"
}

Module Argument Reference

See variables.tf and examples/ for details and use-cases.

Main Resource Configuration

• name: (Required string)

  The name of the Cloud Pub/Sub Topic.

• project: (Required string)

  The project in which the resource belongs. If it is not provided, the provider project is used.

• labels: (Optional map(string))

  A map of labels to assign to the Pub/Sub topic.

  Default is {}.

• kms_key_name: (Optional string)

  The resource name of the Cloud KMS CryptoKey to be used to protect access to messages published on this topic. Your project's PubSub service account (service-{{PROJECT_NUMBER}}@gcp-sa-pubsub.iam.gserviceaccount.com) must have roles/cloudkms.cryptoKeyEncrypterDecrypter to use this feature. The expected format is projects/*/locations/*/keyRings/*/cryptoKeys/*

• message_retention_duration: (Optional string)

  Indicates the minimum duration to retain a message after it is published to the topic. If this field is set, messages published to the topic in the last messageRetentionDuration are always available to subscribers. For instance, it allows any attached subscription to seek to a timestamp that is up to messageRetentionDuration in the past. If this field is not set, message retention is controlled by settings on individual subscriptions. Cannot be more than 7 days or less than 10 minutes.

• allowed_persistence_regions: (Optional set(string))

  A list of IDs of GCP regions where messages that are published to the topic may be persisted in storage. Messages published by publishers running in non-allowed GCP regions (or running outside of GCP altogether) will be routed for storage in one of the allowed regions. An empty list means that no regions are allowed, and is not a valid configuration.

  Default is [].

Extended Resource Configuration

• iam: (Optional list(iam))

  List of IAM access roles to grant to a set of identities on the topic.

  Default is [].

  Each iam object in the list accepts the following attributes:

  • role: (Optional string)

    The role that should be applied. Only one google_pubsub_subscription_iam_binding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

  • roles: (Optional list(string))

    A list of roles that should be applied. This is a shortcut if a set of members should be assigned to the same roles. If role is set in addition, the list will be extended by the role. Each role can only be defined once withing the list of IAM objects. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

  • members: (Optional set(string))

    Identities that will be granted the privilege in role. Each entry can have one of the following values:

    • allUsers: A special identifier that represents anyone who is on the internet; with or without a Google account.
    • allAuthenticatedUsers: A special identifier that represents anyone who is authenticated with a Google account or a service account.
    • user:{emailid}: An email address that represents a specific Google account. For example, [email protected] or [email protected].
    • serviceAccount:{emailid}: An email address that represents a service account. For example, [email protected].
    • group:{emailid}: An email address that represents a Google group. For example, [email protected].
    • domain:{domain}: A G Suite domain (primary, instead of alias) name that represents all the users of that domain. For example, google.com or example.com.
    • computed:{identifier}: An existing key from var.computed_members_map.

    Default is [].

  • authoritative: (Optional bool)

    Whether to exclusively set (authoritative mode) or add (non-authoritative/additive mode) members to the role.

    Default is true.

• computed_members_map: (Optional map(string))

  A map of members to replace in members of various IAM settings to handle terraform computed values.

  Default is {}.

• schema: (Optional object(schema))

  A schema is a format that messages must follow, creating a contract between publisher and subscriber that Pub/Sub will enforce.

  Example:

  schema = {
    name       = "example"
    type       = "AVRO"
    encoding   = "JSON"
    definition = jsonencode({
      type = "record"
      name = "Avro"
      fields =  [
        {
          name = "StringField"
          type = "string"
        },
        {
          name = "IntField"
          type = "int"
        }
      ]
    })
  }

  The schema object accepts the following attributes:

  • id: (Optional string)

    The resource ID of an existing schema to use. Conflicts with schema.name and will be ignored if schema.name is set.

  • encoding: (Optional string)

    The encoding of messages validated against schema. Possible values are ENCODING_UNSPECIFIED, JSON, and BINARY.

    Default is "ENCODING_UNSPECIFIED".

  • name: (Optional string)

    The ID to use for the schema, which will become the final component of the schema's resource name. Conflicts with schema.id. If set a new schema resource will be created.

  • type: (Optional string)

    The type of the schema definition Default value is TYPE_UNSPECIFIED. Possible values are TYPE_UNSPECIFIED, PROTOCOL_BUFFER, and AVRO. Ignored if schema.name is not set.

  • definition: (Optional string)

    The definition of the schema. This should contain a string representing the full definition of the schema that is a valid schema definition of the type specified in type. Ignored if schema.name is not set.

• policy_bindings: (Optional list(policy_binding))

  A list of IAM policy bindings.

  Example:

  policy_bindings = [{
    role    = "roles/viewer"
    members = ["user:[email protected]"]
  }]

  Each policy_binding object in the list accepts the following attributes:

  • role: (Required string)

    The role that should be applied.

  • members: (Required set(string))

    Identities that will be granted the privilege in role.

  • condition: (Optional object(condition))

    An IAM Condition for a given binding.

    Example:

    condition = {
      expression = "request.time < timestamp(\"2022-01-01T00:00:00Z\")"
      title      = "expires_after_2021_12_31"
    }

    The condition object accepts the following attributes:

    • expression: (Required string)

      Textual representation of an expression in Common Expression Language syntax.

    • title: (Required string)

      A title for the expression, i.e. a short string describing its purpose.

    • description: (Optional string)

      An optional description of the expression. This is a longer text which describes the expression, e.g. when hovered over it in a UI.

• subscriptions: (Optional list(subscription))

  The list of the subscriptions that will be created for the PubSub topic.

  Default is [].

  Example:

  subscriptions = [
    {
      name = "my-subs"
  
      # subscription settings
      labels                     = {..}
      ack_deadline_seconds       = null
      message_retention_duration = "604800s"
      retain_acked_messages      = null
      filter                     = null
      enable_message_ordering    = null
  
      # iam access (roles to grant identities on the subscription)
      iam = [
        {
          role          = "roles/pubsub.subscriber"
          members       = [...]
          authoritative = true
        }
      ]
    },
  ]

  Each subscription object in the list accepts the following attributes:

  • name: (Required string)

    Name of the subscription.

  • labels: (Optional map(string))

    A set of key/value label pairs to assign to this Subscription.

  • ack_deadline_seconds: (Optional number)

    This value is the maximum time after a subscriber receives a message before the subscriber should acknowledge the message. After message delivery but before the ack deadline expires and before the message is acknowledged, it is an outstanding message and will not be delivered again during that time (on a best-effort basis). For pull subscriptions, this value is used as the initial value for the ack deadline. To override this value for a given message, call subscriptions.modifyAckDeadline with the corresponding ackId if using pull. The minimum custom deadline you can specify is 10 seconds. The maximum custom deadline you can specify is 600 seconds (10 minutes). If this parameter is 0, a default value of 10 seconds is used. For push delivery, this value is also used to set the request timeout for the call to the push endpoint. If the subscriber never acknowledges the message, the Pub/Sub system will eventually redeliver the message.

  • message_retention_duration: (Optional string)

    How long to retain unacknowledged messages in the subscription's backlog, from the moment a message is published. If retainAckedMessages is true, then this also configures the retention of acknowledged messages, and thus configures how far back in time a subscriptions.seek can be done. Defaults to 7 days. Cannot be more than 7 days (604800s) or less than 10 minutes (600s). A duration in seconds with up to nine fractional digits, terminated by s.

  • retain_acked_messages: (Optional bool)

    Indicates whether to retain acknowledged messages. If true, then messages are not expunged from the subscription's backlog, even if they are acknowledged, until they fall out of the messageRetentionDuration window.

  • filter: (Optional string)

    The subscription only delivers the messages that match the filter. Pub/Sub automatically acknowledges the messages that don't match the filter. You can filter messages by their attributes. The maximum length of a filter is 256 bytes. After creating the subscription, you can't modify the filter.

  • enable_message_ordering: (Optional bool)

    If true, messages published with the same orderingKey in PubsubMessage will be delivered to the subscribers in the order in which they are received by the Pub/Sub system. Otherwise, they may be delivered in any order.

  • expiration_policy_ttl: (Optional string)

    A policy that specifies the conditions for this subscription's expiration. A subscription is considered active as long as any connected subscriber is successfully consuming messages from the subscription or is issuing operations on the subscription. If expirationPolicy is not set, a default policy with ttl of 31 days will be used. If it is set but ttl is "", the resource never expires. The minimum allowed value for expirationPolicy.ttl is 1 day.

  • dead_letter_policy: (Optional object(dead_letter_policy))

    A policy that specifies the conditions for dead lettering messages in this subscription. If dead_letter_policy is not set, dead lettering is disabled. The Cloud Pub/Sub service account associated with this subscription's parent project (i.e., service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com) must have permission to Acknowledge() messages on this subscription.

    The dead_letter_policy object accepts the following attributes:

    • dead_letter_topic: (Optional string)

      The name of the topic to which dead letter messages should be published. Format is projects/{project}/topics/{topic}. The Cloud Pub/Sub service account associated with the enclosing subscription's parent project (i.e., service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com) must have permission to Publish() to this topic. The operation will fail if the topic does not exist. Users should ensure that there is a subscription attached to this topic since messages published to a topic with no subscriptions are lost.

    • max_delivery_attempts: (Optional number)

      The maximum number of delivery attempts for any message. The value must be between 5 and 100. The number of delivery attempts is defined as 1 + (the sum of number of NACKs and number of times the acknowledgement deadline has been exceeded for the message). A NACK is any call to ModifyAckDeadline with a 0 deadline. Note that client libraries may automatically extend ack_deadlines. This field will be honored on a best effort basis. If this parameter is 0, a default value of 5 is used.

  • retry_policy: (Optional object(retry_policy))

    A policy that specifies how Pub/Sub retries message delivery for this subscription. If not set, the default retry policy is applied. This generally implies that messages will be retried as soon as possible for healthy subscribers. RetryPolicy will be triggered on NACKs or acknowledgement deadline exceeded events for a given message.

    The retry_policy object accepts the following attributes:

    • minimum_backoff: (Optional string)

      The minimum delay between consecutive deliveries of a given message. Value should be between 0 and 600 seconds. A duration in seconds with up to nine fractional digits, terminated by s.

      Example:

      3.5s

    • maximum_backoff: (Optional string)

      The maximum delay between consecutive deliveries of a given message. Value should be between 0 and 600 seconds. A duration in seconds with up to nine fractional digits, terminated by s.

      Example:

      3.5s

  • push_config: (Optional object(push_config))

    If push delivery is used with this subscription, this field is used to configure it. An empty pushConfig signifies that the subscriber will pull and ack messages using API methods.

    The push_config object accepts the following attributes:

    • oidc_token: (Optional object(oidc_token))

      If specified, Pub/Sub will generate and attach an OIDC JWT token as an Authorization header in the HTTP request for every pushed message.

      The oidc_token object accepts the following attributes:

      • service_account_email: (Required string)

        Service account email to be used for generating the OIDC token. The caller (for subscriptions.create, subscriptions.patch, and subscriptions.modifyPushConfig RPCs) must have the iam.serviceAccounts.actAs permission for the service account.

      • audience: (Optional string)

        Audience to be used when generating OIDC token. The audience claim identifies the recipients that the JWT is intended for. The audience value is a single case-sensitive string. Having multiple values (array) for the audience field is not supported. More info about the OIDC JWT token audience here: https://tools.ietf.org/html/rfc7519#section-4.1.3 Note: if not specified, the Push endpoint URL will be used.

    • push_endpoint: (Required string)

      A URL locating the endpoint to which messages should be pushed. For example, a Webhook endpoint might use "https://example.com/push".

    • attributes: (Required string)

      Endpoint configuration attributes. Every endpoint has a set of API supported attributes that can be used to control different aspects of the message delivery. The currently supported attribute is x-goog-version, which you can use to change the format of the pushed message. This attribute indicates the version of the data expected by the endpoint. This controls the shape of the pushed message (i.e., its fields and metadata). The endpoint version is based on the version of the Pub/Sub API. If not present during the subscriptions.create call, it will default to the version of the API used to make such call. If not present during a subscriptions.modifyPushConfig call, its value will not be changed. subscriptions.get calls will always return a valid version, even if the subscription was created without this attribute. The possible values for this attribute are:

      • v1beta1: uses the push format defined in the v1beta1 Pub/Sub API.
      • v1 or v1beta2: uses the push format defined in the v1 Pub/Sub API.

    • bigquery_config: (Optional object(bigquery_config))

      If delivery to BigQuery is used with this subscription, this field is used to configure it. Either push_config or bigquery_config can be set, but not both. If both are empty, then the subscriber will pull and ack messages using API methods."

      The bigquery_config object accepts the following attributes:

      • table: (Required string)

        The name of the table to which to write data, of the form {projectId}.{datasetId}.{tableId}

      • use_topic_schema: (Optional bool)

        When true, use the topic's schema as the columns to write to in BigQuery, if it exists. Only one of use_topic_schema and use_table_schema can be set.

      • use_table_schema: (Optional bool)

        When true, use the BigQuery table's schema as the columns to write to in BigQuery. Messages must be published in JSON format. Only one of use_topic_schema and use_table_schema can be set.

      • write_metadata: (Optional bool)

        When true, write the subscription name, messageId, publishTime, attributes, and orderingKey to additional columns in the table. The subscription name, messageId, and publishTime fields are put in their own columns while all other message properties (other than data) are written to a JSON object in the attributes column.

      • drop_unknown_fields: (Optional bool)

        When true and use_topic_schema is true, any fields that are a part of the topic schema that are not part of the BigQuery table schema are dropped when writing to BigQuery. Otherwise, the schemas must be kept in sync and any messages with extra fields are not written and remain in the subscription's backlog.

      • service_account_email: (Optional string)

        The service account to use to write to BigQuery. If not specified, the Pub/Sub service agent, service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com, is used.

  • cloud_storage_config: (Optional object(cloud_storage_config))

    If delivery to Cloud Storage is used with this subscription, this field is used to configure it. Either pushConfig, bigQueryConfig or cloudStorageConfig can be set, but not combined. If all three are empty, then the subscriber will pull and ack messages using API methods.

    The cloud_storage_config object accepts the following attributes:

    • bucket: (Required string)

      User-provided name for the Cloud Storage bucket. The bucket must be created by the user. The bucket name must be without any prefix like "gs://".

    • filename_prefix: (Optional string)

      (Optional) User-provided prefix for Cloud Storage filename.

    • filename_suffix: (Optional string)

      (Optional) User-provided suffix for Cloud Storage filename. Must not end in "/".

    • max_duration: (Optional string)

      (Optional) The maximum duration that can elapse before a new Cloud Storage file is created. Min 1 minute, max 10 minutes, default 5 minutes. May not exceed the subscription's acknowledgement deadline. A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".

    • max_bytes: (Optional number)

      (Optional) The maximum bytes that can be written to a Cloud Storage file before a new file is created. Min 1 KB, max 10 GiB. The maxBytes limit may be exceeded in cases where messages are larger than the limit.

    • avro_config: (Optional object(avro_config))

      If set, message data will be written to Cloud Storage in Avro format.

      The avro_config object accepts the following attributes:

      • write_metadata: (Optional bool)

        When true, write the subscription name, messageId, publishTime, attributes, and orderingKey as additional fields in the output.

  • iam: (Optional list(iam))

    List of IAM access roles to grant to a set of identities on the subscription.

    Default is [].

    Each iam object in the list accepts the following attributes:

    • role: (Optional string)

      The role that should be applied. Only one google_pubsub_subscription_iam_binding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

    • members: (Optional set(string))

      Identities that will be granted the privilege in role. Each entry can have one of the following values:

      • allUsers: A special identifier that represents anyone who is on the internet; with or without a Google account.
      • allAuthenticatedUsers: A special identifier that represents anyone who is authenticated with a Google account or a service account.
      • user:{emailid}: An email address that represents a specific Google account. For example, [email protected] or [email protected].
      • serviceAccount:{emailid}: An email address that represents a service account. For example, [email protected].
      • group:{emailid}: An email address that represents a Google group. For example, [email protected].
      • domain:{domain}: A G Suite domain (primary, instead of alias) name that represents all the users of that domain. For example, google.com or example.com.

      Default is [].

    • authoritative: (Optional bool)

      Whether to exclusively set (authoritative mode) or add (non-authoritative/additive mode) members to the role.

      Default is true.

Module Configuration

• module_enabled: (Optional bool)

  Specifies whether resources in the module will be created.

  Default is true.

• module_depends_on: (Optional list(dependency))

  A list of dependencies. Any object can be assigned to this list to define a hidden external dependency.

  Default is [].

  Example:

  module_depends_on = [
    null_resource.name
  ]

Module Outputs

The following attributes are exported in the outputs of the module:

• topic: (object(topic))

  The created pub sub resource.

• iam: (list(iam))

  The iam resource objects that define access to the GCS bucket.

• subscription: (object(subscription))

  All attributes of the created subscription.

• module_enabled: (bool)

  Whether this module is enabled.

External Documentation

Google Documentation

• https://cloud.google.com/pubsub

Terraform GCP Provider Documentation

• https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic

Module Versioning

This Module follows the principles of Semantic Versioning (SemVer).

Given a version number MAJOR.MINOR.PATCH, we increment the:

1. MAJOR version when we make incompatible changes,
2. MINOR version when we add functionality in a backwards compatible manner, and
3. PATCH version when we make backwards compatible bug fixes.

Backwards compatibility in 0.0.z and 0.y.z version

• Backwards compatibility in versions 0.0.z is not guaranteed when z is increased. (Initial development)
• Backwards compatibility in versions 0.y.z is not guaranteed when y is increased. (Pre-release)

About Mineiros

Mineiros is a remote-first company headquartered in Berlin, Germany that solves development, automation and security challenges in cloud infrastructure.

Our vision is to massively reduce time and overhead for teams to manage and deploy production-grade and secure cloud infrastructure.

We offer commercial support for all of our modules and encourage you to reach out if you have any questions or need help. Feel free to email us at [email protected] or join our Community Slack channel.

Reporting Issues

We use GitHub Issues to track community reported issues and missing features.

Contributing

Contributions are always encouraged and welcome! For the process of accepting changes, we use Pull Requests. If you'd like more information, please see our Contribution Guidelines.

Makefile Targets

This repository comes with a handy Makefile. Run make help to see details on each available target.

License

This module is licensed under the Apache License Version 2.0, January 2004. Please see LICENSE for full details.

Copyright © 2020-2022 Mineiros GmbH