Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting integer type within ArrayBuilder #3313

Open
TiniTinyTerminator opened this issue Nov 21, 2024 · 1 comment
Open

Setting integer type within ArrayBuilder #3313

TiniTinyTerminator opened this issue Nov 21, 2024 · 1 comment
Labels
feature New feature or request

Comments

@TiniTinyTerminator
Copy link

Description of new feature

It seems that the arraybuilder always assumes int64 when setting up the data. It would be nice to set the integer type like uint8 or uint64. This could help reduce memory load when building huge datasets that only contain smaller numbers. the same goes for floats for example

@TiniTinyTerminator TiniTinyTerminator added the feature New feature or request label Nov 21, 2024
@jpivarski
Copy link
Member

That's true about ArrayBuilder, but ArrayBuilder is only for ingesting data from untyped sources, such as Python objects or JSON. Those integers don't have any fixed-width type, like int32 or int64.

There are a few functions for setting the type of Awkward Array, after it has been built from an ArrayBuilder. ak.values_astype sets the numeric type of all numbers in an array, wherever they are, and ak.enforce_type provides more control by forcing a whole array to fit a given type description. (You can get the type from one array, either as a ak.types.Type object or as a string, change the int64 to something else, and then enforce-type.)

I think it makes sense to change the type after the fact because the arrays of Python objects or JSON can't be too big if you're able to process them at all. And if you're coming from NumPy arrays ak.from_numpy, the dtype is preserved.)

Is it okay if I turn this from an issue/feature request into a Discussion?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants