Understand measurement of NOPM

[perdelt]

Hi @sm-shaw

I am interested in understanding the metric NOPM better. As I have found here #197

To be more specific NOPM can be queried from the schema using select sum(d_next_o_id) from district

So the value is queried at the beginning of benchmarking and at the end, and the difference divided by the number of minutes gives NOPM?

Could you please document a bit more about how the metric is derived? Is the above correct? What is sum(d_next_o_id)?

Many thanks!

[sm-shaw]

Hi,
the TPROC-C workload is derived from the TPC-C benchmark, with an overview here https://www.tpc.org/tpcc/detail5.asp
As described in the overview, "The performance metric reported by TPC-C measures the number of orders that can be fully processed per minute and is expressed in tpm-C.". The term "tpmC" is a trademark of the TPC-C and therefore cannot be used except for fully audited and approved benchmarks published at www.tpc.org only. (If anyone is using the term tpmC for a non-audited benchmark then this is a trademark violation). For this reason, for the TPROC-C workload we use a metric called NOPM that is measuring the number of new orders per minute, this is unique to HammerDB. As d_next_o_id is incremented by 1 in the new order transaction, this is a good measure of the new orders we are processing queried by select sum(d_next_o_id) from district. However, as the district table is being updated during the test, we don't want to query it at the same time as that could impact the test itself and therefore capture the value at the start and the end of the test.

We also measure the TPM value, which is the database specific transaction rate that is stored in an in-memory table and can be sampled without impact. It is not a requirement of the TPC-C workload that each user transaction should result in one database transaction, nor that the ratio between NOPM and TPM should be constant. For example, some databases may prefer to commit more frequently (resulting in a higher TPM). Therefore, you can use TPM to compare the same database but NOPM should be used to compare between them as the number of new orders is the key workload metric.

[perdelt]

Hi, thanks for the explanation!
I wonder why summing up. Wouldn't max() return the latest id?
Is it correct that the difference in values is divided by the runtime in minutes for NOPM?

[sm-shaw]

Yes for runtime that is correct. To see why sum instead of max you can check it by testing a build.
Here is an example, I built 20 warehouses in SQL Server and the following query returns.

select sum(d_next_o_id), max(d_next_o_id)from district

sum 600200
max 3001

I then ran a test with 1 virtual user for 1 minute with the result as follows: 43195 NOPM 97973 TPM

I then re-run the query.

sum 642398
max 7318

Looking in the district table I can see my workload selected warehouse 15

d_id	d_w_id	d_next_o_id
1	15	7152
2	15	7249
3	15	7201
4	15	7122
5	15	7145
6	15	7318
7	15	7316
8	15	7301
9	15	7237
10	15	7167

So for 1 warehouse, the max gives the max for one district. The sum gives the total for all districts in that warehouse.
Therefore we need to use sum to get all of the new orders, max will just give us the highest district value.

[perdelt]

Yes, that makes sense. The order_id is not unique, but is assigned per warehouse / district. Thanks very much!