Katana high memory usage when running with high concurrency and large lists

[Ice3man543]

Please describe your feature request:

Describe the use case of this feature:

[ca5ua1]

I guess my issue is relevant to this.

When crawling pretty big gitea repository (don't ask me why), katana may take more then 1GB of RAM and 100% CPU depending on threads/concurrency settings.
That's the example for scan with more then 1GB of RAM:
katana -u 'https://gitea.com/gitea/blog' -d 2

The key here is size of repo. With larger repos it take more RAM. But my problem that I'm running with -d 2 which should crawl only one URL basically extracting URLs from it, right?!

[moscowchill]

Here also issues with large lists however I kept lower conc but the ram runs full (32gb) after a couple hundred domains