[sublime_security] Initial release of the sublime security (#10805)

Create New integration package sublime_security. Added data stream. Added data collection logic for all data streams. Added the ingest pipeline for all data streams. Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files. Added dashboards and visualizations. Added test for pipeline for all data streams. Added system test cases for all data streams.
elastic · Aug 29, 2024 · c223a56 · c223a56
1 parent 3a8db73
commit c223a56
Show file tree

Hide file tree

Showing 63 changed files with 15,123 additions and 0 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -326,6 +326,7 @@
 /packages/stan @elastic/obs-infraobs-integrations
 /packages/statsd_input @elastic/obs-infraobs-integrations
 /packages/stormshield @elastic/sec-deployment-and-devices
+/packages/sublime_security @elastic/security-service-integrations
 /packages/suricata @elastic/sec-deployment-and-devices
 /packages/symantec_edr_cloud @elastic/security-service-integrations
 /packages/symantec_endpoint @elastic/security-service-integrations

diff --git a/packages/sublime_security/_dev/build/build.yml b/packages/sublime_security/_dev/build/build.yml
@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]
diff --git a/packages/sublime_security/_dev/build/docs/README.md b/packages/sublime_security/_dev/build/docs/README.md
@@ -0,0 +1,120 @@
+# Sublime Security
+
+Sublime Security is a programmable, AI-powered, cloud email security platform for Microsoft 365 and Google Workspace environments. It is used to block email attacks such as phishing, BEC, malware, threat hunt, and auto-triage user reports.
+
+The Sublime Security integration collects data for Audit, Email Message(MDM Schema) and Message Event logs using REST API and AWS-S3 or AWS-SQS:
+
+- REST API mode - Sublime Security integration collects and parses data from the Sublime Security REST APIs.
+- AWS S3 polling mode - Sublime Security writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files.
+- AWS S3 SQS mode - Sublime Security writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.
+
+## Data streams
+
+The Sublime Security integration collects three types of logs:
+
+**[Audit](https://docs.sublime.security/reference/listeventsinauditlog)** - Captures detailed records of all significant actions and changes within the platform, including changes to email security policies, user access to email data, and modifications to email configurations, ensuring traceability and compliance for all operations.
+
+**[Email Message](https://docs.sublime.security/docs/export-message-mdms)** - Represents the flow of individual emails through the platform, including sender and recipient details, spam filtering outcomes, and overall email disposition, helping to secure and analyze email communication.
+
+**[Message Event](https://docs.sublime.security/reference/getmessage-1)** - Represents document specific actions taken on emails, like spam detection or rule applications, providing detailed insights into how the platform processes and protects email communications.
+
+## Requirements
+
+Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+
+### Installing and managing an Elastic Agent:
+
+You have a few options for installing and managing an Elastic Agent:
+
+### Install a Fleet-managed Elastic Agent (recommended):
+
+With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier.
+
+### Install Elastic Agent in standalone mode (advanced users):
+
+With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only.
+
+### Install Elastic Agent in a containerized environment:
+
+You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
+
+There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#_minimum_requirements).
+
+## Setup
+
+### To collect data from the Sublime Security API:
+
+#### Step 1: Go to Platform
+- Visit the [Sublime Security Platform](https://platform.sublime.security/) and select `API` in Developers section.
+
+#### Step 2: Generating the API Key
+- Retrieve your `API Key`. This key will be used further in the Elastic integration setup to authenticate and access different Sublime Security Logs.
+- `Base URL` of Sublime Security is also required for configuring integration.
+
+**Note**: Users with the `Admin` role are allowed to access `Audit` logs. For more information, refer [here](https://docs.sublime.security/docs/role-based-access-control-rbac).
+
+### To collect data from AWS S3 Bucket or AWS SQS:
+
+#### For AWS S3 Bucket, follow the below steps:
+- Create an Amazon S3 bucket. Refer to the link [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html).
+- User can set the parameter "Bucket List Prefix" according to the requirement.
+
+#### For AWS SQS, follow the below steps:
+1. If data forwarding to an AWS S3 Bucket hasn't been configured, then first set up an AWS S3 Bucket as mentioned in the above documentation.
+2. To set up an SQS queue, follow "Step 1: Create an Amazon SQS queue" mentioned in the [Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html).
+  - While creating an SQS Queue, please provide the same bucket ARN that has been generated after creating an AWS S3 Bucket.
+3. Set up event notifications for a S3 bucket. Follow this [link](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html).
+  - Users have to set the prefix parameter the same as the S3 Bucket List Prefix as created earlier. (for example, `exports/sublime_platform_audit_log/` for a audit data stream).
+  - Select the event type as s3:ObjectCreated:*, select the destination type SQS Queue, and select the queue that has been created in Step 2.
+
+**Note**:
+  - Credentials for the above AWS S3 and SQS input types should be configured using the [link](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#aws-credentials-config).
+  - Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case.
+  - You can configure a global SQS queue for all data streams or a local SQS queue for each data stream. Configuring data stream specific SQS queues will enable better performance and scalability. Data stream specific SQS queues will always override any global queue definitions for that specific data stream.
+
+### Enabling the integration in Elastic:
+
+1. In Kibana go to Management > Integrations.
+2. In "Search for integrations" search bar, type Sublime Security.
+3. Click on the "Sublime Security" integration from the search results.
+4. Click on the "Add Sublime Security" button to add the integration.
+5. Enable the Integration to collect logs via AWS S3 or API input.
+6. Under the AWS S3 input, there are two types of inputs: using AWS S3 Bucket or using SQS.
+7. Add all the required integration configuration parameters, including API Key, Interval, Initial Interval and Page Size for API input and Access Key, Secret Key and Session Token for AWS input type to enable data collection.
+8. Click on "Save and continue" to save the integration.
+
+**Note**:
+- The Base URL for Sublime Security cloud customers is `https://api.platform.sublimesecurity.com`. Depending on your type of deployment, yours may be different.
+- For SSO users, in addition to access key ID and secret access key, the session token is required to configure integration. For IAM users, the session token is optional and not required.
+
+## Logs reference
+
+### Audit
+
+This is the `audit` dataset.
+
+#### Example
+
+{{event "audit"}}
+
+{{fields "audit"}}
+
+### Email Message
+
+This is the `email_message` dataset.
+
+#### Example
+
+{{event "email_message"}}
+
+{{fields "email_message"}}
+
+### Message Event
+
+This is the `message_event` dataset.
+
+#### Example
+
+{{event "message_event"}}
+
+{{fields "message_event"}}
diff --git a/packages/sublime_security/changelog.yml b/packages/sublime_security/changelog.yml
@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: 0.1.0
+  changes:
+    - description: Initial release.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/10805
diff --git a/packages/sublime_security/data_stream/audit/_dev/deploy/docker/docker-compose.yml b/packages/sublime_security/data_stream/audit/_dev/deploy/docker/docker-compose.yml
@@ -0,0 +1,15 @@
+version: '2.3'
+services:
+  sublime_security:
+    image: docker.elastic.co/observability/stream:v0.15.0
+    hostname: sublime_security
+    ports:
+      - 8090
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: '8090'
+    command:
+      - http-server
+      - --addr=:8090
+      - --config=/files/config.yml
diff --git a/packages/sublime_security/data_stream/audit/_dev/deploy/docker/files/config.yml b/packages/sublime_security/data_stream/audit/_dev/deploy/docker/files/config.yml
@@ -0,0 +1,128 @@
+rules:
+  - path: /v0/audit-log/events
+    methods: ['GET']
+    query_params:
+      limit: 1
+      offset: 0
+    request_headers:
+      Authorization:
+        - 'Bearer xxxx'
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {{ minify_json `
+          {
+            "events": [
+              {
+                "id": "bd49af79-0cfb-4184-bd18-b0401d69ac61",
+                "type": "message_group.search",
+                "created_at": "2024-08-12T06:04:03.714126Z",
+                "data": {
+                  "request": {
+                    "id": "6ad202de-0def-423d-a0f2-549402e1a9c9",
+                    "path": "/v0/message-groups",
+                    "method": "GET",
+                    "body": "",
+                    "authentication_method": "api_key",
+                    "ip": "1.128.0.0",
+                    "user_agent": "Go-http-client/1.1",
+                    "api_key_name": "demo mode local"
+                  }
+                },
+                "created_by": {
+                  "id": "6e6eca05-4fea-406b-86d4-b40177e25474",
+                  "active": true,
+                  "first_name": "Demo",
+                  "last_name": "User",
+                  "email_address": "[email protected]",
+                  "phone_number": null,
+                  "created_at": "2024-07-12T05:13:47.879426Z",
+                  "updated_at": "2024-07-12T05:13:47.879426Z",
+                  "role": "admin",
+                  "is_enrolled": true,
+                  "google_oauth_user_id": "d83rb8et4-refe-fe7t4f8efe",
+                  "microsoft_oauth_user_id": "fhe7t4bgf8-freu-ebfur94ref"
+                }
+              }
+            ],
+            "count": 1,
+            "total": 2
+          }
+          `}}
+  - path: /v0/audit-log/events
+    methods: ['GET']
+    query_params:
+      limit: 1
+      offset: 1
+    request_headers:
+      Authorization:
+        - "Bearer xxxx"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {{ minify_json `
+          {
+            "events": [
+              {
+                "id": "bd49af79-0cfj-4184-bd18-b0401d69ac61",
+                "type": "message_group.search",
+                "created_at": "2024-08-12T06:04:03.714126Z",
+                "data": {
+                  "request": {
+                    "id": "6ad202de-0def-423d-a0f2-549402e1a9c9",
+                    "path": "/v0/message-groups",
+                    "method": "GET",
+                    "body": "",
+                    "authentication_method": "api_key",
+                    "ip": "175.16.199.0",
+                    "user_agent": "Go-http-client/1.1",
+                    "api_key_name": "demo mode local-2"
+                  }
+                },
+                "created_by": {
+                  "id": "6e6eca05-4fea-406b-86d4-b40177e25474",
+                  "active": true,
+                  "first_name": "User",
+                  "last_name": "Doe",
+                  "email_address": "[email protected]",
+                  "phone_number": null,
+                  "created_at": "2024-07-12T05:13:47.879426Z",
+                  "updated_at": "2024-07-12T05:13:47.879426Z",
+                  "role": "admin",
+                  "is_enrolled": true,
+                  "google_oauth_user_id": "",
+                  "microsoft_oauth_user_id": ""
+                }
+              }
+            ],
+            "count": 1,
+            "total": 2
+          }
+          `}}
+  - path: /v0/audit-log/events
+    methods: ['GET']
+    query_params:
+      limit: 1
+      offset: 2
+    request_headers:
+      Authorization:
+        - "Bearer xxxx"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {{ minify_json `
+          {
+            "events": [],
+            "count": 0,
+            "total": 2
+          }
+          `}}
diff --git a/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-audit.log b/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-audit.log
@@ -0,0 +1 @@
+{"id":"26704b44-d1b0-4362-8221-579e604f40cb","type":"message_group.search","created_at":"2024-07-30T05:33:47.725649Z","data":{"request":{"id":"ca817b01-cfaa-40ea-ab80-30b6a8e6ef08","path":"/v1/messages/groups/search","method":"GET","query":{},"body":"","api_key_name":"demo mode key","authentication_method":"api_key","ip":"81.2.69.142","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"}},"created_by":{"id":"6e6eca05-4fea-406b-86d4-b40177e25474","active":true,"first_name":"Bob","last_name":"User","email_address":"[email protected]","phone_number":null,"created_at":"2024-07-12T05:13:47.879426Z","updated_at":"2024-07-12T05:13:47.879426Z","role":"admin","is_enrolled":true,"google_oauth_user_id":"","microsoft_oauth_user_id":""}}