Add Nextorm to benchmarks

[AlexeyShirshov]

Please, take a look at this PR. I add Nextorm lib and make benchmark project to support net8.0. Also fix sql server connection issue for EF core.

[mgravell]

not sure we want to change this - what's the intent here?

[AlexeyShirshov]

you're right, removed

[mgravell]

I'd rather use a #if if this is limited to certain frameworks

[AlexeyShirshov]

Done

[mgravell]

why do we need all these version-overrides? that seems unlikely

[AlexeyShirshov]

I've got this
error NU1010: The PackageReference items Microsoft.Extensions.Logging.Console;Microsoft.Extensions.ObjectPool do not have corresponding PackageVersion

[mgravell]

that probably means it needs adding to Directory.Packages.props

[AlexeyShirshov]

Done

[mgravell]

is this actually realistic usage? usually, you don't want to Compile() anything per-context-instance, as the context-instance is going to be transient - is this different here?

[AlexeyShirshov]

_queryBufferedCompiled is not static variable, it's bound to dbcontext and live with him in the same scope (members on NextormBenchmarks class). Nextorm is not yet supports static compiled queries as EF Core.

[mgravell]

OK, but that makes this a very questionable metric; I'm not familiar with Nextorm to know what Compile() does internally here, and whether it is optional/mandatory/recommended, but: by default I'd question anything named Compile() that occurs per connection/context

[AlexeyShirshov]

Maybe it's better to name the function Prepare, since it build sql stmt, create DbCommand and so on. I'll think about it when start implementing static compiled queries (as in EF Core). Thank you, it is good remark!

[mgravell]

That's fine, but: I think at that point, the entirety of that needs to go into the per-invoke code path. The question ultimately is: in a real world usage, what parts are going to be reused? It doesn't sound like any of this is ever going to be reused,l between real world calls, in which case: don't reuse it. It doesn't represent a meaningful scenario if you measure reusing something that can't usefully be reused.

[mgravell]

They don't. It is form of optimization, so it is optional.

If this does additional work, I would guess (I haven't measured) that it only actually becomes an optimization in the N+1 case for some non-trivial number of invokes - for example, for 10 rows it may be more efficient not to do this; it might be interesting to investigate (for your own purposes I mean, not really useful for Dapper); I had a moment, so I threw together:

    [Params(1, 2, 3, 4, 5, 10, 25, 50, 75, 100)]
    public int Iterations { get; set; } = 1;

    [Benchmark(Description = "Basic")]
    public Post First_Basic()
    {
        Step();
        int iterations = Iterations;
        Post first = null!;
        for (int i = 0; i  < iterations;i++)
        {
            first = _repository.Posts.Where(it => it.Id == i).FirstOrDefault();
        }
        return first;
    }

    [Benchmark(Description = "Parameterized")]
    public Post First_Parameterized()
    {
        Step();
        var getPostById = _repository.Posts.Where(it => it.Id == NORM.Param<int>(0)).FirstOrFirstOrDefaultCommand();
        int iterations = Iterations;
        Post first = null!;
        for (int i = 0; i < iterations; i++)
        {
            first = getPostById.FirstOrDefault(i);
        }
        return first;
    }

    [Benchmark(Description = "Compiled")]
    public Post First_Compiled()
    {
        Step();
        var getPostById = _repository.Posts.Where(it => it.Id == NORM.Param<int>(0)).FirstOrFirstOrDefaultCommand().Compile();
        int iterations = Iterations;
        Post first = null!;
        for (int i = 0; i < iterations; i++)
        {
            first = getPostById.FirstOrDefault(i);
        }
        return first;
    }

which gives us:

| ORM     | Method        | Return | Iterations | Mean        | StdDev     | Error      | Gen0    | Gen1    | Gen2   | Allocated  |
|-------- |-------------- |------- |----------- |------------:|-----------:|-----------:|--------:|--------:|-------:|-----------:|
| Nextorm | Basic         | Post   | 1          |    57.24 us |   0.555 us |   0.839 us |  0.5000 |       - |      - |    9.62 KB |
| Nextorm | Parameterized | Post   | 1          |    57.70 us |   1.464 us |   2.461 us |  0.5000 |       - |      - |    9.65 KB |
| Nextorm | Parameterized | Post   | 2          |   112.85 us |   1.196 us |   1.809 us |  1.2500 |  1.0000 |      - |   22.92 KB |
| Nextorm | Basic         | Post   | 2          |   114.07 us |   0.827 us |   1.390 us |  1.5000 |  1.2500 |      - |   25.23 KB |
| Nextorm | Parameterized | Post   | 3          |   163.62 us |   1.570 us |   2.374 us |  2.0000 |  1.5000 | 0.5000 |    36.2 KB |
| Nextorm | Basic         | Post   | 3          |   179.27 us |   0.767 us |   1.466 us |  2.5000 |  0.5000 |      - |   40.85 KB |
| Nextorm | Parameterized | Post   | 4          |   224.53 us |   2.869 us |   4.337 us |  3.0000 |  0.5000 |      - |   49.47 KB |
| Nextorm | Basic         | Post   | 4          |   234.03 us |   1.113 us |   1.870 us |  3.0000 |  0.5000 |      - |   56.47 KB |
| Nextorm | Parameterized | Post   | 5          |   274.08 us |   1.876 us |   2.837 us |  3.5000 |  0.5000 |      - |   62.74 KB |
| Nextorm | Basic         | Post   | 5          |   307.74 us |   2.753 us |   4.626 us |  4.0000 |  1.0000 |      - |   72.09 KB |
| Nextorm | Compiled      | Post   | 1          |   469.97 us |  38.541 us |  58.268 us |  2.0000 |  1.0000 |      - |   33.23 KB |
| Nextorm | Compiled      | Post   | 2          |   521.50 us |  45.720 us |  69.122 us |  2.0000 |  1.0000 |      - |   43.43 KB |
| Nextorm | Parameterized | Post   | 10         |   538.14 us |   6.070 us |  10.201 us |  7.0000 |  1.0000 |      - |  129.11 KB |
| Nextorm | Compiled      | Post   | 3          |   543.80 us |   4.044 us |   7.733 us |  3.0000 |  2.0000 |      - |   53.65 KB |
| Nextorm | Compiled      | Post   | 4          |   593.31 us |   4.308 us |   8.237 us |  3.0000 |  1.0000 |      - |   63.86 KB |
| Nextorm | Basic         | Post   | 10         |   625.08 us |   5.606 us |   9.420 us |  8.0000 |  2.0000 |      - |  150.17 KB |
| Nextorm | Compiled      | Post   | 5          |   652.65 us |   6.002 us |  10.085 us |  4.0000 |  1.0000 |      - |   74.07 KB |
| Nextorm | Compiled      | Post   | 10         |   936.95 us | 122.213 us | 205.370 us |  6.0000 |  4.0000 |      - |  125.12 KB |
| Nextorm | Parameterized | Post   | 25         | 1,369.06 us |  18.753 us |  28.351 us | 20.0000 |  2.0000 |      - |  328.21 KB |
| Nextorm | Basic         | Post   | 25         | 1,535.49 us |  26.749 us |  40.440 us | 22.0000 |  2.0000 |      - |  384.43 KB |
| Nextorm | Compiled      | Post   | 25         | 1,628.25 us |  37.002 us |  55.942 us | 16.0000 | 10.0000 |      - |  278.43 KB |
| Nextorm | Parameterized | Post   | 50         | 2,726.71 us |  58.572 us |  88.552 us | 40.0000 |  2.0000 |      - |  660.05 KB |
| Nextorm | Compiled      | Post   | 50         | 2,766.69 us |  61.124 us |  92.411 us | 32.0000 | 10.0000 |      - |  533.57 KB |
| Nextorm | Basic         | Post   | 50         | 3,116.97 us |  82.664 us | 124.976 us | 46.0000 |  2.0000 |      - |  774.87 KB |
| Nextorm | Compiled      | Post   | 75         | 4,016.48 us |  94.250 us | 142.492 us | 48.0000 |       - |      - |  788.84 KB |
| Nextorm | Parameterized | Post   | 75         | 4,076.27 us |  77.770 us | 117.577 us | 60.0000 |  2.0000 |      - |  991.89 KB |
| Nextorm | Basic         | Post   | 75         | 4,681.44 us |  91.542 us | 138.398 us | 70.0000 |  2.0000 |      - |  1165.3 KB |
| Nextorm | Parameterized | Post   | 100        | 5,403.15 us | 138.371 us | 209.198 us | 80.0000 |  2.0000 |      - | 1323.73 KB |
| Nextorm | Compiled      | Post   | 100        | 5,496.56 us | 123.691 us | 236.494 us | 62.0000 | 10.0000 |      - | 1044.12 KB |
| Nextorm | Basic         | Post   | 100        | 6,096.84 us |  90.875 us | 137.390 us | 94.0000 |  4.0000 |      - | 1555.73 KB |

from which we can observe that:

• parameterized over basic starts being worthwhile for anything more than one iteration
• compiled over parameterized starts being worthwhile somewhere between 50 and 75 iterations

that's the thing that the static approach in EF attempts to resolve, i.e. so that the compilation overhead can be amortized

[AlexeyShirshov]

if a prepared thing is only useful on a single connection

No, but in this version it is not supported. So let's return to this question after next version (should be in 2-3 weeks, I hope).

[AlexeyShirshov]

we would also want to compare DapperAOT, since that excels at cold start scenarios

I will be happy to join such benchmark

[AlexeyShirshov]

Thank you very mush for the investigation and benchmarking this problem! Can you submit this code into my repo? I'll include it in future PR.
I also trying to measure cold start performance for EF and Dapper. There is PurgeQueryCache in Dapper, but I didn't find method to clear cache in EF. The code should look something like

[Benchmark()]
    public void NextormCached()
    {
        using var ctx = _builder.CreateDbContext();
        var repo = new TestDataRepository(ctx);
        repo.LargeEntity.Where(it => it.Id == 1).Select(it => new { it.Id, it.Str, it.Dt }).ToList();
    }
    [Benchmark()]
    public void NextormNonCached()
    {
        using var ctx = _builder.CreateDbContext();
        var repo = new TestDataRepository(ctx);
        var cmd = repo.LargeEntity.Where(it => it.Id == 1).Select(it => new { it.Id, it.Str, it.Dt });
        cmd.Cache = false;
        cmd.ToList();
    }
    [Benchmark()]
    public void EFcore()
    {
        using var ctx = new EFDataContext(_efBuilder.Options);
        var cmd = ctx.LargeEntities.Where(it => it.Id == 1).Select(it => new { it.Id, it.Str, it.Dt });
        var _ = cmd.ToList();
        // pseudo code
        //Microsoft.EntityFrameworkCore.DbContext.ClearCache();
    }
    [Benchmark()]
    public void Dapper()
    {
        _conn.Query<LargeEntity>("select id, someString as str, dt from large_table where id=@id", new { id = 1 });
        SqlMapper.PurgeQueryCache();
    }

[mgravell]

The "query cache" in dapper is not the same thing - in most reasonable scenarios, it should never be useful (let alone desirable) to call that API; it mostly just deals with how to handle <LargeEntity> and new { id: int } - at the type level - and those things will be identical for every similar usage, i.e. the entire point is that it doesn't change per call. It doesn't even exist in DapperAOT, because the entire point is that we emit the handling of <LargeEntity> and new { id: int }

[mgravell]

Unless I'm misunderstanding, nextorm is brand new, barely seen - is that right? In that case, I doubt it has enough maturity to be worth adding here yet. It might be better if you fork things and do this locally for now, and then if nextorm gets some momentum, we can see about merging it in. However, I do think we've had some worthwhile discussion here, that might help influence design choices (especially around how you measure performance in terms of reusable objects, which could have significant impact on the design)