[BUG] Performance regression for Dev Tools /api/console/proxy?path=_mapping requests

[tronboto]

Describe the bug

When Dev Tools is open, requests are made periodically and upon each API command to 3 endpoints, one of which is:

/api/console/proxy?path=_mapping&method=GET&dataSourceId=

After building a cluster and filling it up as per the repro steps below, this request takes:

Opensearch Dashboards version	Time taken to complete request (approx.)
2.14.0	5s
2.15.0	15s
2.17.0	16-17s

For the tests above, the Opensearch and Opensearch Dashboards versions were the same each time. However, even if I keep Opensearch on 2.17.0 for all 3 versions of Dashboards, I see the exact same slowdown, suggesting that the issue is with Dashboards rather than Opensearch itself.

To Reproduce

1. Set up a 2-node development cluster as per the docs here:

https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/#sample-docker-compose-file-for-development

Note that I updated the heap size to 2g for each node and I'm running this on a c4.m8 VM.

2. Export the index template for filebeat 7.12.1:

docker run docker.elastic.co/beats/filebeat-oss:7.12.1 filebeat export template --noilm > template.json

3. Run the following python script that creates the index template and 600 indices based on this template (requires elasticsearch==7.12.1 to be installed):

import json
import logging

from elasticsearch import Elasticsearch

logging.basicConfig(level=logging.INFO)

es = Elasticsearch()

# load template from disk
with open("template.json") as f:
    template = json.load(f)

# create template
es.indices.put_template("filebeat-7.12.1-template", template)

# create indices
for i in range(1, 601):
    es.indices.create(f"filebeat-7.12.1-{i:06d}")

4. With the Chrome Developer Tools Network tab open, run a command in Dashboards Dev Tools. Check how long the triggered request to /api/console/proxy?path=_mapping&method=GET&dataSourceId= takes.

5. Repeat the test for versions 2.14.0, 2.15.0 and 2.17.0

Expected behavior

The requests shouldn't take 3 times as long as in 2.14.0. What's more, the impact is multiplicative since every request made in Dev Tools triggers this request - see: #5654

OpenSearch Version
2.17.0

Dashboards Version
2.17.0

Plugins

Using the stock opensearchproject/opensearch-dashboards docker images.

Screenshots

N/A

Host/Environment (please complete the following information):

• Windows 10
• Chrome 128.0.6613.120

Additional context

N/A

[vinylen]

We are experiencing the exact same issue, cluster versions 2.16.0 and 2.17.0 and 2.17.1.

[ananzh]

From the network tab, requests are made to three endpoints for 3 diff meta data including _mapping, _aliases, and _template.

These frequent call and fetching meta data are expected and it is crucial for features like autocomplete in the Dev Console. Regarding the performance, I check the fetch method and don't think we made any changes that can cause this but we shoudl compare the response sizes and contents.

Should we assign someone or ask for help? CC: @ashwin-pc @joshuali925

[dblock]

[Catch All Triage - 1, 2]

[Hailong-am]

I did some investigation on this, here are the findings.

the most time consuming part of api call is sending request to OpenSearch cluster and waiting for response.

OpenSearch-Dashboards/src/plugins/console/server/routes/api/console/proxy/create_handler.ts

Lines 114 to 117 in e9c9983

	opensearchResponse = await client.transport.request(
	{ path: toUrlPath(path), method, body: bufferedBody },
	{ headers: requestHeaders }
	);

going deeper on this method call on opensearch-js lib

try {
    // add logs to show the how long it will take
    console.log('deserialize start: ' + start)
    result.body = this.serializer.deserialize(payload);
    console.log('deserialize end: takes ' + (Date.now() - start) + 'ms')
  } catch (err) {
    this.emit('response', err, result);
    return callback(err, result);
  }

this.serializer.deserialize will take about 15-20s seconds on my test env with command, it total take 30s to get and download the response, deserialize takes 22s.

curl -XPOST -u admin:$password! http://localhost:5602/api/console/proxy\?path\=_mapping\&method\=GET\&dataSourceId\=   -H 'osd-xsrf: osd-fetch' > ods_mapping.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 34.8M  100 34.8M    0     0  1185k      0  0:00:30  0:00:30 --:--:-- 9294k

deserialize start: 1730214785882
JSON11.parse start: 1730214787107
JSON11.parse end: takes 18551ms
deserialize end: takes 19776ms

by looking at commit history about deserialize changes, Use JSON11 to parse and serialize long numerals might be the cause of performance issue. @AMoo-Miki

[Hailong-am]

found a typo that will cause JSON11 parse will execute every time when json string has number with it. For the json string mentioned in the issue, there is no large number with it.

By fixing this, JSON11 parse will be skipped, and total api time will down to 5-6 seconds that matches with 2.14.

raised a PR to fix issue at opensearch-js side opensearch-project/opensearch-js#889 @AMoo-Miki would you mind to take a look?