Benchmark your AiiDA setup

[ltalirz]

Dear AiiDA users,

we recently added a simple benchmark script that you can use to estimate the overhead introduced by AiiDA's workflow engine per process. Bottlenecks can be the database (if remote) or the file system (if shared / otherwise slow).

Let's use this thread to post our hardware setup and benchmark results & discuss about them

[ltalirz]

Seb's setup: AMD Ryzen 5 3600 6-Core processor (3.6 GHz, 4.2 GHz turbo boost) using AiiDA v2.2.0, and RabbitMQ and PostgreSQL running on the same machine.

0.25 seconds per process.

[ltalirz]

Cloud setup (not optimized): Intel Xeon E5-2690 v3 2.6GHz using AiiDA v2.1.2, local RabbitMQ & PostgreSQL (DB on OS volume), with network file system (NFSv3, sub-ms latency)

2.2-2.3 seconds per process.

This goes down to 1.7-1.9s per process when moving the PostgresDB to a managed service.

Will investigate file system side next.

[atztogo]

AiiDA v2.1.2, RabbitMQ & PostgreSQL on Ubuntu VM (KVM) 8core, 16GB memory assigned.
VM host: Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz (26 core, 200GB memory), file system on iSCSI volume via 10GbE ethernet, iSCSI target QNAP TS-873 (HDD, Memory 64GB)

Performance: 0.75 s / process (No other AiiDA processes are running)

Another similar VM on the same VM host
Performance: 1.14 s / process (~50 AiiDA processes are running, almost waiting state, simultaneously with the benchmark)

[zhubonan]

AiiDA 2.2.1 RMQ (3.8.2) & PostgreSQL (12) on WSL 2.0 Ubuntu 20.04, 16 GB memory assigned.
The host: 12th Gen Intel(R) Core(TM) i7-12700H 2.30 GHz. NVMe SSD 1TB.

@ltalirz I think the performance depends heavily on the size of the database - running the script on my "production" profile I get

Performance: 3.05 s / process

This profile has 729014 nodes.

In [1]: QueryBuilder().append(Node).count()
Out[1]: 729014

On a newly created emtpy profile, I get:

Performance: 0.67 s / process

I think we should report database sizes in addition to the hardware setup.

[ltalirz]

Thanks, Bonan! Indeed, size of the database matters (my tests were done on a new, empty profile).
Thanks for providing your timings; having these baselines is also useful in order to see where optimizations on the AiiDA side might to bring down the timings of the most frequent queries.