Improved Build Efficiency for Large Blog with Batch Processing: Prevent Javascript heap out of memory

[sr2ds]

This pull request addresses a critical issue encountered during the build process of my blog, which contains over 2,000 posts. The problem manifested itself particularly when deploying on Netlify, where the build process was failing consistently due to memory constraints. Locally, even with attempts to allocate 8GB of RAM to Node.js, the build process was still unfeasible.

Changes Made

My goal was to be simple and easy to maintain in the future, with no additional resources, allowing the process to be simple with a definitive solution that could be easily changed if necessary. It also allows us for the build to happen regardless of the computational power involved.

To overcome this challenge, I implemented a batch processing approach to handle the rendering of pages in smaller batches (batchSize). The original code processed all pages simultaneously, leading to excessive memory consumption and build failures for large datasets. By dividing the rendering tasks into manageable batches, we significantly reduced memory usage during the build process.

Benefits

The primary benefit of this change is a substantial reduction in memory consumption during the build process.
The build now successfully completes on Netlify, even with a blog containing over 2,000 posts.
Locally, the build process is now feasible with significantly less memory usage (less than 1GB of max_old_space_size) compared to the previous attempts.

How It Works:

The implementation involves iterating through the array of pages in batches of a specified size (batchSize). Each batch is then processed asynchronously, allowing for a more efficient use of memory resources.

The issue

Here I'm trying to build my blog with 4GB of max_old_space_size in the current code and no success.
I tried with 8GB also and not success. Memory is not enough.

The solution

In this print we can to see the build with 2186 posts and NODE_OPTIONS max_old_space_size=800 in the new code changed and works fine:

Extra Information

I was using the vuepress but was impossible maintain my blog there. With vitepress was working fine before 1800 posts but right now I needed try something to improve my build and make it possible to run inside the Netlify as free mode.
You can see this comment reported in vuepress some months ago.

Help me if I made something wrong

I encourage all reviewers to provide constructive feedback to enhance the overall quality of the codebase. Your insights and suggestions are invaluable, and I am open to making any necessary adjustments to ensure the best possible solution.

Thanks so much