Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functionality to split RawImage into channels; Update slice documentation and tests #978

Merged
merged 14 commits into from
Nov 26, 2024

Conversation

BritishWerewolf
Copy link
Contributor

@BritishWerewolf BritishWerewolf commented Oct 16, 2024

Some models, like U2Net, require images to be processed with all the reds, greens and blue pixels grouped together.

As a result, this PR provides the ability to do something like this.

// Get our image, then split into individual channels.
const image = await RawImage.fromURL("https://picsum.photos/600/400");

// ... snip ...
// Preprocess the image by resizing and padding to a square of 320x320
// See these PRs for functionality to do this: #971, #972

// Image pixel values are an array of [R, G, B, R, G, B,…]
// Use this to separate out each individual channel.
const [R, G, B] = image.toChannels();
// Convert from an array of Uint8Array into regular array, then flatten.
// We now have an array of [R, R, …, G, G, …, B, B, …]
const image_data = [R, G, B].map(d => Array.from(d)).flat();

// Create the tensor for this image
const tensor = new Tensor('float32', image_data, [1, image.channels, image.width, image.height]);
const model = await AutoModel.from_pretrained('u2netp');
const outputs = await model({ "input": tensor });

@BritishWerewolf BritishWerewolf changed the title Add functionality to split RawImage into channels; Add functionality to return a vertical slice of a Tensor Add functionality to split RawImage into channels; Update slice documentation and tests Nov 22, 2024
@xenova xenova changed the base branch from v3 to main November 26, 2024 12:59
Copy link
Collaborator

@xenova xenova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR! Thanks! :) I've just applied some code suggestions to improve performance.

src/utils/image.js Outdated Show resolved Hide resolved
tests/utils/utils.test.js Outdated Show resolved Hide resolved
tests/utils/tensor.test.js Outdated Show resolved Hide resolved
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@xenova xenova merged commit 8896dc7 into huggingface:main Nov 26, 2024
@xenova
Copy link
Collaborator

xenova commented Nov 26, 2024

Thanks again @BritishWerewolf! 🔥 To align better with PIL's split() function, we now return new RawImage's from the function. However, if you only need the channel data, you can simply do R.data (for example)

@BritishWerewolf BritishWerewolf deleted the add-vslice branch November 26, 2024 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants