Bedrock Embeddings ThrottlingException

[Jtewen]

When attempting to generate a vector store using the fromDocuments method on approximately 10,000 documents (specifically on the HNSWLib class) , it reaches the rate limit quickly. Most likely due to the BedrockEmbeddings.embedDocuments method just being a simple Promise.all on every document at once.

[dosubot]

🤖

Hello,

I understand you're encountering a ThrottlingException when trying to generate a vector store using the fromDocuments method on the HNSWLib class for a large number of documents. This is likely due to the BedrockEmbeddings.embedDocuments method running all promises at once.

A similar issue was encountered and resolved in the past with the OpenAIEmbedding. The solution was to add concurrency settings to control the number of promises that can be pending at the same time. This was implemented in version 0.0.130 of langchainjs. You can find more details about this solution in the issue #2140 and the update #2294.

To apply a similar solution to your case, you can modify the fromDocuments method to limit the number of concurrent promises. Here's how you can do it:

First, install the 'bluebird' library:

npm install bluebird

Then, modify the 'fromDocuments' method in the 'HNSWLib' class:

import * as Bluebird from 'bluebird';

// ...

static async fromDocuments(docs: Document[], embeddings: Embeddings, concurrency: number = 1000): Promise<HNSWLib> {
    const vectorStore = new HNSWLib();
    await Bluebird.map(docs, async (doc) => {
        const vector = await embeddings.embedDocument(doc);
        vectorStore.add(doc.id, vector);
    }, {concurrency});
    return vectorStore;
}

In this code, 'Bluebird.map' is used to process the documents in parallel but with a limit on the number of concurrent operations. The 'concurrency' parameter controls the number of promises that can be pending at the same time. You can adjust this value based on your rate limit.

Finally, use the modified 'fromDocuments' method:

const vectorStore = await HNSWLib.fromDocuments(docs, new OpenAIEmbeddings(), 500);

In this example, the concurrency level is set to 500. This means that at most 500 documents will be processed at the same time.

As for the current rate limit for the 'BedrockEmbeddings.embedDocuments' method in the langchainjs codebase, I wasn't able to find this information in the repository. You might want to check the Bedrock Embeddings documentation or contact their support for this information.

I hope this helps! Let me know if you have any other questions.

Sources

• I hope to add concurrency settings in OpenAIEmbedding
• examples/src/indexes/vector_stores/hnswlib_fromdocs.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[jacoblee93]

I'll look into fixing this to better use concurrency, thank you for opening this!

[bracesproul]

@Jtewen fix should be merged soon!