Implement all of the getitem cases for NumpyArray/ListArray in Numba. (…

…#15)

Numba implementations are only missing `newaxis` and `Ellipsis`.

* Get started on this PR.

* Bring basic getitems of ListOffsetArray up to speed.

* Removed underscores from _numba.util functions.

* Infrastructure for ListOffsetArray.getitem(tuple).

* ListOffsetArray and ListArray's Numba implementations are synchronized.

* Pass other getitem types through the tuple-handler.

* A 64-bit ListArray-NumpyArray advanced array index works.

* Identities that have been tested are correct, but some id assignments are wrong.

* Fixing wrong .id assignments.

* Now all the .id assignments should be correct.

* Cleaned up all TODOs.

* Transfer to laptop.

* Use arraydecl to type NumpyArray slices.

* Regularize NumpyArray's slice, just like any other.

* Slicing 2-dimensional NumpyArray.

* test_current() is the platform for developing NumpyArray.getitem_next.

* [skip ci] in progress.

* Fancy indexes pass from ListArray to NumpyArray in Numba.

* Try to fix 32-bit error.

* ListArray.getitem_next(int) works.

* [skip ci] in progress.

* [skip ci] fixed a carry-issue in type, but there's badly formed LLVM IR.

* ListArray32/64 and ListOffsetArray32/64 are all at the same level: getitem_next(int*) works.

* Probably fixed 32-bit issue.

* Try the test without negative-correction on 32-bit.

* More diagnostics for 32-bit.

* Yet more diagnostics for 32-bit.

* Yet yet more diagnostics for 32-bit.

* Replace 'zext' with 'sext'.

* Cleaned up all debugging code.

* [skip ci] in progress.

* Partially implemented ListArray.getitem_next(range).

* Homogenized ListArray and ListOffsetArray; ensured that nextoffsets follows bit width of old starts or offsets.

* Implemented ListOffsetArray.getitem_next(range, not advanced).

* All of the Numba getitems seem to be done.

* Updated README; ready to squash and merge.

README.md

@@ -58,8 +58,9 @@ The following features of awkward 0.x will be features of awkward 1.x.
    * 2019-09-21 (PR [#9](../../pull/9)): `Identity` is correctly passed through `NumpyArray` slices and `__getitem__` uses `get`, `slice`, or the full `getitem`, depending on argument complexity.
    * 2019-09-26 (PR [#11](../../pull/11)): fully implemented `ListArray` and `ListOffsetArray`'s `__getitem__`.
    * 2019-10-02 (PR [#12](../../pull/12)): implemented `ListArray.__getitem__(array)` in Numba, setting the pattern for all the other cases.
-   * 2019-10-04 (PR [#13](../../pull13)): turned the `Error` type into a struct and `handle_error` into a function that provides `Identity` (if available) and attempted index information. Numba also handles errors, but with messages that must be known at compile-time.
-   * 2019-10-04 (PR [#14](../../pull14)): brought `NumpyArray`, `ListArray`, and `ListOffsetArray` up-to-date on `getitem` for all cases (except `newaxis`) and tested them all as well. (Not including Numba.)
+   * 2019-10-04 (PR [#13](../../pull/13)): turned the `Error` type into a struct and `handle_error` into a function that provides `Identity` (if available) and attempted index information. Numba also handles errors, but with messages that must be known at compile-time.
+   * 2019-10-04 (PR [#14](../../pull/14)): brought `NumpyArray`, `ListArray`, and `ListOffsetArray` up-to-date on `getitem` for all cases (except `newaxis`) and tested them all as well. (Not including Numba.)
+   * 2019-10-10 (PR [#15](../../pull/15)): brought `NumpyArray`, `ListArray`, and `ListOffsetArray` up-to-date on `getitem` in Numba (except `newaxis` and `Ellipsis`).
 
 ## Roadmap
 

VERSION_INFO

@@ -1 +1 @@
-0.1.11
+0.1.12

awkward1/_numba/content.py

@@ -36,7 +36,7 @@ def generic(self, args, kwargs):
                 if not isinstance(wheretpe, numba.types.BaseTuple):
                     wheretpe = numba.types.Tuple((wheretpe,))
 
-                wheretpe = util._typing_regularize_slice(wheretpe)
+                wheretpe = util.typing_regularize_slice(wheretpe)
 
                 if any(not isinstance(t, (numba.types.Integer, numba.types.SliceType, numba.types.EllipsisType, type(numba.typeof(numpy.newaxis)), numba.types.Array)) for t in wheretpe.types):
                     raise TypeError("only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`), and integer or boolean arrays (possibly jagged) are valid indices")

awkward1/_numba/cpu.py

@@ -58,7 +58,7 @@ def __init__(self, dmm, fe_type):
     if xfile.find("./compounddef/location").attrib["file"].startswith("src/cpu-kernels"):
         for xfcn in xfile.findall(".//memberdef[@kind='function']"):
             name = xfcn.find("./name").text
-            if hasattr(kernels, name):
+            if hasattr(kernels, name) and name not in ("success", "failure"):
                 rettype = xfcn.find("./type").text
                 params = [(x.find("./declname").text, x.find("./type").text) for x in xfcn.findall("./param")]
                 getattr(kernels, name).name = name

awkward1/_numba/identity.py

@@ -80,6 +80,29 @@ def lower_len(context, builder, sig, args):
     proxyin = numba.cgutils.create_struct_proxy(tpe)(context, builder, value=val)
     return numba.targets.arrayobj.array_len(context, builder, numba.types.intp(tpe.arraytpe), (proxyin.array,))
 
+def lower_getitem_any(context, builder, idtpe, wheretpe, idval, whereval):
+    proxyin = numba.cgutils.create_struct_proxy(idtpe)(context, builder, value=idval)
+
+    if isinstance(wheretpe, numba.types.Integer):
+        proxyslice = numba.cgutils.create_struct_proxy(numba.types.slice2_type)(context, builder)
+        proxyslice.start = util.cast(context, builder, wheretpe, numba.intp, whereval)
+        proxyslice.stop = builder.add(proxyslice.start, context.get_constant(numba.intp, 1))
+        proxyslice.step = context.get_constant(numba.intp, 1)
+        wheretpe = numba.types.slice2_type
+        whereval = proxyslice._getvalue()
+
+    proxyout = numba.cgutils.create_struct_proxy(idtpe)(context, builder)
+    proxyout.ref = proxyin.ref
+    proxyout.fieldloc = proxyin.fieldloc
+    if isinstance(wheretpe, numba.types.BaseTuple):
+        proxyout.array = numba.targets.arrayobj.getitem_array_tuple(context, builder, idtpe.arraytpe(idtpe.arraytpe, wheretpe), (proxyin.array, whereval))
+    elif isinstance(wheretpe, numba.types.Array):
+        proxyout.array = numba.targets.arrayobj.fancy_getitem_array(context, builder, idtpe.arraytpe(idtpe.arraytpe, wheretpe), (proxyin.array, whereval))
+    else:
+        proxyout.array = numba.targets.arrayobj.getitem_arraynd_intp(context, builder, idtpe.arraytpe(idtpe.arraytpe, wheretpe), (proxyin.array, whereval))
+
+    return proxyout._getvalue()
+
 @numba.typing.templates.infer_getattr
 class type_methods(numba.typing.templates.AttributeTemplate):
     key = IdentityType