PERF: datetimelike addition

[jbrockmendel]

In [1]: import pandas as pd

In [2]: dti = pd.date_range("2016-01-01", periods=10_000)

In [3]: td = pd.Timedelta(days=1)

In [4]: %timeit dti + td
108 µs ± 1.85 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  # <- main
94.3 µs ± 3.27 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  # <- PR

[WillAyd]

Interesting idea!

[WillAyd]

Are these possible to specialize to int64?

[jbrockmendel]

not without specifying an ndim.

[WillAyd]

Pretty cool

[WillAyd]

Are these considered superfluous?

[jbrockmendel]

they use a no-longer-used mask argument. actually these are all pretty superfluous. will update to remove

[WillAyd]

By the way I think there's another way of doing this without a loop that could also be more performant. The inspiration from this comes from a book called Hacker's Delight by Henry S Warren in section 4-2

The code would look something like:

uleft = left.view("uint64")
uright = right.view("uint64")
summed = uleft + uright
neg_overflow = uleft & uright & ~summed
pos_overflow = ~uleft & ~uright & summed
overflows = (neg_overflow | pos_overflow) >= 0x8000000000000000

The essence of the algorithm is that you use unsigned arithmetic (which has defined behavior) to do the summation, and then inspect the sign bit to see if "overflow" would have happened

With this would need neither the cython.overflow check nor the try...except block

[WillAyd]

Here's a quick demo

In [42]: left = np.array([2 ** 63 - 1, 2 ** 63 - 1, 2 ** 63 - 1, -(2 ** 63), -(2 ** 63), -(2 ** 63)])
    ...: right = np.array([42, 1, 0, -42, -1, 0])
    ...: uleft = left.view("uint64")
    ...: uright = right.view("uint64")
    ...: summed = uleft + uright
    ...: neg_overflow = uleft & uright & ~summed
    ...: pos_overflow = ~uleft & ~uright & summed
    ...: overflows = (neg_overflow | pos_overflow) >= 0x8000000000000000
    ...: overflows
Out[42]: array([ True,  True, False,  True,  True, False])

[jbrockmendel]

Neat. Does this play nicely with the NPY_NAT check?

[WillAyd]

Good callout. I suppose you would just mask that as well. So something like:

In [5]: left = np.array([2 ** 63 - 1, 2 ** 63 - 1, 2 ** 63 - 1, -(2 ** 63), -(2 ** 63), -(2 ** 63)])
   ...: right = np.array([42, 1, 0, -42, -1, 0])
   ...: is_nat = (left == 2 ** 63 - 1) | (right == 2 ** 63 - 1)
   ...: uleft = left.view("uint64")
   ...: uright = right.view("uint64")
   ...: summed = uleft + uright
   ...: neg_overflow = uleft & uright & ~summed
   ...: pos_overflow = ~uleft & ~uright & summed
   ...: overflows = ((neg_overflow | pos_overflow) >= 0x8000000000000000) & ~is_nat
   ...: overflows
Out[5]: array([False, False, False,  True,  True, False])

I just hardcoded the above to 2 ** 63 - 1 but can replace with macro or python variable, whichever is available

[jbrockmendel]

does that actually out-perform? (maybe SIMD or something?) seems like a lot of additional memory allocations

[WillAyd]

Hard to say. In theory yes this would leverage SIMD better if numpy uses it under the hood (not well versed in if NumPy uses that or not). Also the overflow checks on gcc/clang are pretty fast because they use the compiler builtin, but I think MSVC falls back to a slow implementation (there is a builtin for MSVC too but I don't recall Cython using it - would have to double check)

It requires more research so not something that needs to be done in this PR. Just food for future thought

[mroeschke]

Thanks @jbrockmendel