OpenTelemetry to OpenSearch Exporter #1

MaxKsyunz · 2023-02-07T19:30:06Z

Description: An exporter of logs and traces from OpenTelemetry to OpenSearch.

Testing: unit tests

Documentation: README.md with overview and description of configuration options.

Signed-off-by: MaxKsyunz <[email protected]>

exporter/opensearchexporter/config_test.go

Signed-off-by: MaxKsyunz <[email protected]>

builder-config.yaml are config file for collector generator. go.mod/go.sum files needed to be updated as a consequence. Signed-off-by: MaxKsyunz <[email protected]>

YANG-DB

plz read the next PR's before addressing the comments

YANG-DB · 2023-02-07T20:46:37Z

exporter/opensearchexporter/README.md

+| Supported pipeline types | traces      |
+| Distributions            | [contrib]   |
+
+This exporter supports sending OpenTelemetry logs to [OpenSearch](https://www.opensearch.org).


rename logs to signals as logs represent one type of the scope

YANG-DB · 2023-02-08T19:41:33Z

exporter/opensearchexporter/README.md

+- `traces_index`: The
+  [index](https://opensearch.org/docs/latest/opensearch/index-data/)
+  or [datastream](https://opensearch.org/docs/latest/opensearch/data-streams/)
+  name to publish trace events to. The default value is `traces-generic-default`. Note: To better differentiate between log indexes and traces indexes, `index` option are deprecated and replaced with below `logs_index`


replace traces-generic-default with `sso_traces-default-namespace'

So SSO will be the default way that they store data? Should we have some kind of opt in variable? Will we be mapping to SSO by default?

what does namespace mean in this context? Do you mean ${namespace} like qal, e2e, prod, etc.?

So SSO will be the default way that they store data? Should we have some kind of opt in variable? Will we be mapping to SSO by default?

Good question. When exporter supports SSO, there will be a configuration setting. However, do we need to support the current schema? I created Bit-Quill/opentelemetry-collector-contrib#2 to discuss this question.

YANG-DB · 2023-02-08T19:46:41Z

exporter/opensearchexporter/README.md

+- `endpoints`: List of OpenSearch URLs. If endpoints is missing, the
+  OPENSEARCH_URL environment variable will be used.
+- `num_workers` (optional): Number of workers publishing bulk requests concurrently.
+- `traces_index`: The


add similar notion for logs_index & metrics_index

Do we need traces_index, or logs_index, or metrics_index configuration options?

These options are unnecessary for SSO mode, because the destination is either derived from (type, dataset, namespace) or traces-default-namespace.

IMO If we support otel mode, then we keep traces_index & co options. If we remove otel mode, then these options should be removed as well.

YANG-DB · 2023-02-08T19:47:41Z

exporter/opensearchexporter/README.md

+exporters:
+  opensearch/trace:
+    endpoints: [https://opensearch.example.com:9200]
+    traces_index: trace_index


add additional support for metrics

Readme will be updated as part of Bit-Quill/opentelemetry-collector-contrib#3.

YANG-DB · 2023-02-08T19:54:19Z

exporter/opensearchexporter/config_test.go

+			id: component.NewIDWithName(typeStr, "trace"),
+			expected: &Config{
+				Endpoints:   []string{"https://opensearch.example.com:9200"},
+				LogsIndex:   "logs-generic-default",


change to traces-default-namespace

sso_traces-default-${namespace}?

Created Bit-Quill/opentelemetry-collector-contrib#4 to track decision.

YANG-DB · 2023-02-08T20:01:55Z

exporter/opensearchexporter/factory.go

+const (
+	// The value of "type" key in configuration.
+	typeStr            = "opensearch"
+	defaultLogsIndex   = "logs-generic-default"


logs-default-namespace
traces-default-namespace
metrics-default-namespace

do we want sso_ prefix? Coming from question in readme about whether we want sso to be default

Created Bit-Quill/opentelemetry-collector-contrib#4 to track decision.

YANG-DB · 2023-02-08T20:02:11Z

exporter/opensearchexporter/factory.go

+			Timeout: 90 * time.Second,
+		},
+		LogsIndex:   defaultLogsIndex,
+		TracesIndex: defaultTracesIndex,


add metrics support

Yes in Bit-Quill/opentelemetry-collector-contrib#3

exporter/opensearchexporter/model.go

exporter/opensearchexporter/testdata/config.yaml

derek-ho · 2023-02-09T14:43:39Z

exporter/opensearchexporter/factory.go

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package opensearchexporter // import "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opensearchexporter"


remove comment?

saw this in a few places

It's for Go tooling.

TLDR: forces package users to use that URL to import this package.

derek-ho · 2023-02-09T14:47:13Z

exporter/opensearchexporter/config.go

+}
+
+type HTTPClientSettings struct {
+	Authentication AuthenticationSettings `mapstructure:",squash"`


is there a typo here?

Sorry, what are you referring to? I don't see it.

",squash" -> "squash", not sure why there is a leading comma

derek-ho · 2023-02-09T14:50:33Z

exporter/opensearchexporter/config_test.go

+				},
+			},
+		},
+	}


minor nit: do we want to add some test cases where mapping is not sso and assert error? Not too concerned though

Supported mapping modes are "sso" and ("no", "none" or no value). In the later case it uses Otel-like schema.

I'll add a unit test for this case and created #8 to make sure unit test coverage is improved.

derek-ho · 2023-02-09T14:51:59Z

exporter/opensearchexporter/factory.go

+// createLogsExporter creates a new exporter for logs.
+//
+// Logs are directly indexed into OpenSearch.
+func createLogsExporter(


metrics exporter?

Yes, when we add support for metrics -- Bit-Quill/opentelemetry-collector-contrib#3.

derek-ho · 2023-02-09T14:53:51Z

exporter/opensearchexporter/factory_test.go

+func TestFactory_CreateMetricsExporter_Fail(t *testing.T) {
+	factory := NewFactory()
+	cfg := factory.CreateDefaultConfig()
+	params := exportertest.NewNopCreateSettings()
+	_, err := factory.CreateMetricsExporter(context.Background(), params, cfg)
+	require.Error(t, err, "expected an error when creating a traces exporter")
+}
+
+func TestFactory_CreateTracesExporter_Fail(t *testing.T) {
+	factory := NewFactory()
+	cfg := factory.CreateDefaultConfig()
+	params := exportertest.NewNopCreateSettings()
+	_, err := factory.CreateTracesExporter(context.Background(), params, cfg)
+	require.Error(t, err, "expected an error when creating a traces exporter")
+}


Little confused here, but why is the trace exporter creation expecting error? Is it just because no endpoint? Can we add a test with trace exporter without error?

Yes, it fails because configuration lacks endpoints and fail validation.

I'll add the success test.

exporter/opensearchexporter/logs_exporter.go

exporter/opensearchexporter/model.go

derek-ho

Looks good to me pending some readme items and small questions I had

MaxKsyunz · 2023-02-09T20:17:55Z

@derek-ho and @YANG-DB thank you for your feedback.

I'll create an issue in this repo for any comment that needs to be implemented. The readme describes the code as it currently is. It will be updated as the code changes.

Signed-off-by: MaxKsyunz <[email protected]>

MaxKsyunz added 8 commits February 7, 2023 11:44

Init.

7867456

Signed-off-by: MaxKsyunz <[email protected]>

updating names, part 1

b567388

Signed-off-by: MaxKsyunz <[email protected]>

Formatting and versioning changes

7b82b7d

Signed-off-by: MaxKsyunz <[email protected]>

More dependency updates -- otelcontribcol now builds.

e97abe1

Update variable name.

e97acd6

Update names and remove unused code.

7475fad

Signed-off-by: MaxKsyunz <[email protected]>

Remove redundant aliases.

cda0180

Signed-off-by: MaxKsyunz <[email protected]>

Update README.md and remove unused configuration options.

76e6c54

Signed-off-by: MaxKsyunz <[email protected]>

Yury-Fridlyand approved these changes Feb 7, 2023

View reviewed changes

exporter/opensearchexporter/config_test.go Outdated Show resolved Hide resolved

MaxKsyunz force-pushed the dev/opensearch-exporter_revB branch from f743ba4 to 76e6c54 Compare February 7, 2023 21:37

MaxKsyunz requested review from forestmvey and GumpacG and removed request for forestmvey February 7, 2023 21:39

MaxKsyunz assigned YANG-DB and MaxKsyunz and unassigned YANG-DB Feb 7, 2023

MaxKsyunz requested review from derek-ho and YANG-DB and removed request for GumpacG and derek-ho February 7, 2023 21:39

MaxKsyunz added 2 commits February 7, 2023 14:04

Updating dependencies

37240a4

Signed-off-by: MaxKsyunz <[email protected]>

Add opensearchexporter to builder-config.yaml's

b782e00

builder-config.yaml are config file for collector generator. go.mod/go.sum files needed to be updated as a consequence. Signed-off-by: MaxKsyunz <[email protected]>

YANG-DB suggested changes Feb 8, 2023

View reviewed changes

derek-ho reviewed Feb 9, 2023

View reviewed changes

exporter/opensearchexporter/logs_exporter.go Show resolved Hide resolved

derek-ho reviewed Feb 9, 2023

View reviewed changes

exporter/opensearchexporter/model.go Show resolved Hide resolved

derek-ho approved these changes Feb 9, 2023

View reviewed changes

MaxKsyunz mentioned this pull request Jun 2, 2023

What's the name for default indices? Bit-Quill/opentelemetry-collector-contrib#3

Open

Addressed PR feedback.

bcc452b

Signed-off-by: MaxKsyunz <[email protected]>

MaxKsyunz merged commit 3717e07 into integ/opensearch-exporter Feb 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenTelemetry to OpenSearch Exporter #1

OpenTelemetry to OpenSearch Exporter #1

MaxKsyunz commented Feb 7, 2023

YANG-DB left a comment

YANG-DB Feb 7, 2023

YANG-DB Feb 8, 2023

derek-ho Feb 9, 2023

derek-ho Feb 9, 2023

MaxKsyunz Feb 9, 2023

YANG-DB Feb 8, 2023

MaxKsyunz Feb 9, 2023

YANG-DB Feb 8, 2023

MaxKsyunz Feb 9, 2023

YANG-DB Feb 8, 2023

derek-ho Feb 9, 2023

MaxKsyunz Feb 9, 2023

YANG-DB Feb 8, 2023

derek-ho Feb 9, 2023

MaxKsyunz Feb 9, 2023

YANG-DB Feb 8, 2023

MaxKsyunz Feb 10, 2023

derek-ho Feb 9, 2023

derek-ho Feb 9, 2023

MaxKsyunz Feb 10, 2023

derek-ho Feb 9, 2023

MaxKsyunz Feb 10, 2023 •

edited

Loading

derek-ho Feb 10, 2023

derek-ho Feb 9, 2023

MaxKsyunz Feb 10, 2023 •

edited

Loading

derek-ho Feb 9, 2023

MaxKsyunz Feb 10, 2023

derek-ho Feb 9, 2023

MaxKsyunz Feb 10, 2023

derek-ho left a comment

MaxKsyunz commented Feb 9, 2023

OpenTelemetry to OpenSearch Exporter #1

OpenTelemetry to OpenSearch Exporter #1

Conversation

MaxKsyunz commented Feb 7, 2023

YANG-DB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxKsyunz Feb 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxKsyunz Feb 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derek-ho left a comment

Choose a reason for hiding this comment

MaxKsyunz commented Feb 9, 2023

MaxKsyunz Feb 10, 2023 •

edited

Loading

MaxKsyunz Feb 10, 2023 •

edited

Loading