Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize the camera so that phones don't heat up and drain battery #1047

Closed
Tracked by #1255
M123-dev opened this issue Jan 31, 2022 · 14 comments
Closed
Tracked by #1255

Optimize the camera so that phones don't heat up and drain battery #1047

M123-dev opened this issue Jan 31, 2022 · 14 comments
Assignees
Labels
camera 🤳 MLKit P2 performance 🤳🥫 Scan We need to be able to scan on low-end, old devices, even with a bad camera, connexion… speed
Milestone

Comments

@M123-dev
Copy link
Member

M123-dev commented Jan 31, 2022

What

Possible solutions

@PrimaelQuemerais

You can hash the feed and make a delta with the last image passed to ML Kit if it's lower than a certain threshold you don't process the image.

You can also reduce the resolution of the image passed to ML Kit while displaying the higher resolution one

Part of

@M123-dev
Copy link
Member Author

I just played around with hashing the image.
Each image generated from the AVD consists of 1843200 bytes
I tried hashing each of these Uint8Lists with MD5 a fairly simple algoritm and it looked promising since it's not secure anymore but

Wikipedia:

It can still be used as a checksum to verify data integrity

The results weren't good, I only got the same results when putting the virtual device camera directly against the wall otherwise it would always be completely different. In addition to that after a few seconds of running the flutter process got autonomous and wasn't able to controll it in any way until after a while it crashed.
Therefor I don't think hashing is worth it since the improvement is presumably less than the added computing power.

@jasmeet0817
Copy link
Contributor

A quick Google search leads to this library that we may be able to use to compress images before passing to ML Kit

https://pub.dev/packages/flutter_image_compress

@jasmeet0817
Copy link
Contributor

I just played around with hashing the image. Each image generated from the AVD consists of 1843200 bytes I tried hashing each of these Uint8Lists with MD5 a fairly simple algoritm and it looked promising since it's not secure anymore but

Wikipedia:

It can still be used as a checksum to verify data integrity

The results weren't good, I only got the same results when putting the virtual device camera directly against the wall otherwise it would always be completely different. In addition to that after a few seconds of running the flutter process got autonomous and wasn't able to controll it in any way until after a while it crashed. Therefor I don't think hashing is worth it since the improvement is presumably less than the added computing power.

Maybe give this a shot, looks promising

https://pub.dev/packages/diff_image

@PrimaelQuemerais
Copy link
Member

Looking back at it hashing doesn't preserve image delta.

Perhaps the best way to do it is first compress the image and the use the image difference library proposed by @jasmeet0817 (looking at the code the plugin compares pixels one by one, you might aswell write the function yourself). Then if the difference percentage exceeds your threshold pass the new compressed image to ML Kit.

Do your testing as compressing the image may reduce the barcode detection accuracy.

@PrimaelQuemerais
Copy link
Member

PrimaelQuemerais commented Jan 31, 2022

Just checked your code, instead of feeding the _processCameraImage with the camera feed why not reducing the frame rate first? Let's say the camera feed produces 30 frames per second, you could process 1/10 of it arbitrarily and still have a quick response time.

@teolemon teolemon added the 🤳🥫 Scan We need to be able to scan on low-end, old devices, even with a bad camera, connexion… label Jan 31, 2022
@M123-dev
Copy link
Member Author

M123-dev commented Jan 31, 2022

I tried around resizing the photo earlier and it turned out to be harder than it might seem But when we look at the Ml Kit documentation, they recommend a minimum resolution of 2MP (1.920 x 1.080) which is actually higher than the one we currently have.

https://developers.google.com/ml-kit/vision/barcode-scanning/android

Don't capture input at the camera’s native resolution. On some devices, capturing input at the native resolution produces extremely large (10+ megapixels) images, which results in very poor latency with no benefit to accuracy. Instead, only request the size from the camera that's required for barcode detection, which is usually no more than 2 megapixels.

However, we could only scan the upper area which not covered by the cards, but when I look at the complexity of the pure resizing, I suspect that this is not going to be so easy.
The step with the less frames per second seems like a good idea, I am looking into that.


@PrimaelQuemerais , you said you also use ML kit for barcode scanning, I am curious did you take any steps to improve performance

and @jasmeet0817 one thing which just came to my mind, can you maybe keep your stock camera on for a while and check how the temprature is evolving in there perhaps this will help us to further narrow down the performance factor

@jasmeet0817
Copy link
Contributor

I tried around resizing the photo earlier and it turned out to be harder than it might seem But when we look at the Ml Kit documentation, they recommend a minimum resolution of 2MP (1.920 x 1.080) which is actually higher than the one we currently have.

https://developers.google.com/ml-kit/vision/barcode-scanning/android

Don't capture input at the camera’s native resolution. On some devices, capturing input at the native resolution produces extremely large (10+ megapixels) images, which results in very poor latency with no benefit to accuracy. Instead, only request the size from the camera that's required for barcode detection, which is usually no more than 2 megapixels.

However, we could only scan the upper area which not covered by the cards, but when I look at the complexity of the pure resizing, I suspect that this is not going to be so easy. The step with the less frames per second seems like a good idea, I am looking into that.

@PrimaelQuemerais , you said you also use ML kit for barcode scanning, I am curious did you take any steps to improve performance

and @jasmeet0817 one thing which just came to my mind, can you maybe keep your stock camera on for a while and check how the temprature is evolving in there perhaps this will help us to further narrow down the performance factor

you mean you want to check if basic camera usage heats up the phone? In that case, I can confirm that it doesn't, also no other app that uses the camera (example whatsapp, telegram etc) heats up the phone since they don't process each image.

I just tried doing as you suggested with "QR Scanner" app, and it didn't heat up the phone either, but the app closed itself after a while on it's own. I think they realized the camera was idle and closed the app. Maybe we could do something similar but instead of closing we could just move to history page (or something similar)

@teolemon teolemon changed the title Camera optimization Optimize the camera so that phones don't heat up and drain battery Feb 5, 2022
@M123-dev M123-dev self-assigned this Feb 20, 2022
@M123-dev M123-dev removed their assignment Mar 3, 2022
@teolemon teolemon added this to the V 1 milestone Mar 31, 2022
@g123k g123k self-assigned this Apr 11, 2022
@olivierdrouard olivierdrouard moved this from Todo (ready 2 dev) to In Progress in 🤳🥫 The Open Food Facts mobile app (Android & iOS) Apr 11, 2022
@g123k
Copy link
Collaborator

g123k commented May 4, 2022

Here are my proposals on how we could improve battery usage.

1) Improve the frequency and how barcodes are detected

Currently, in the app, we scan for a barcode every 10 frames.

I think this approach can be improved:

  • If the decoding takes more than 10 frames, we will stack computations (-> may cause an out of memory)
  • 10 frames is not a good unit, as it changes from one device to another depending on its refresh rate (eg: a 60Hz display will mean every ~160 ms, while a 90Hz display [Pixel 6 for example] ~= 110 ms)
  • there is no correlation between the computation power and the refresh rate

What I suggest instead:

  • Don't use fps (frames per second) as a unit
  • Don't start a computation until the previous is done
  • Detect how long the first computation took and base the next computation on this unit (I think an empirical approach is required here, before stating on the frequency)
  • Detect barcodes in an Isolate (or maybe a pool of Isolates?) to "smooth" the UI

2) Only decode barcodes when necessary

If the user has scrolled to an item in the carousel, we can still continue to show the camera feed (= to prevent an empty background). But at the same time, the barcode detection is paused: the visor is replaced by a "resume scanning" button.

When this button is pressed, the scanning is restarted.
And obviously, if the user goes back to the "home" card of the carousel, the decoding is restarted automatically

This way, I think we won't impact the quality of the detection but will reduce instead the number of unnecessary computations.

@M123-dev
Copy link
Member Author

M123-dev commented May 4, 2022

Heyyy @g123k good proposals, I especially like the idea of calculating some delta time for the frequency.

Don't start a computation until the previous is done

Already in place

if (isBusy || barcodeScanner == null) {

Detect barcodes in an Isolate (or maybe a pool of Isolates?) to "smooth" the UI

The real detection is running native, so this is already on a different threat, but definitely makes sense for cropping.

Only decode barcodes when necessary

We should do some more benchmarking before to analyze what needs the most computing power, the scanning, the camera maybe even the cropping.

@monsieurtanuki
Copy link
Contributor

@g123k @M123-dev I remember working on the "half-cropping" for optimization, then said to @M123-dev now you can test/profile and got as a reply: "Actually I don't have a heat/drain problem". The key is: we need someone that actually experience that problem, for the test part, not just theory.

there is no correlation between the computation power and the refresh rate

Is that a fact? If the device is constantly working and has no time to gc() or something like that, or is moving Mb of data while computing, that's not very idle.

Detect barcodes in an Isolate

I don't believe in Isolate, but maybe I could have a good surprise.

Only decode barcodes when necessary

That's what I already suggested. Of course that would mean a slightly different UX, but I think that the "Scan now!/Resume scanning" button is an elegant solution.

@g123k
Copy link
Collaborator

g123k commented May 9, 2022

For the battery drain, on the OFF team, we have some iOS users complaining about this.
We will check if this somehow improves things.

One optimization I have also in mind -> switch to a JPEG format (instead of YUV), as it will require less memory and should also quick the cropping process. But in that case, I think some changes are needed in image's croppers.

@monsieurtanuki
Copy link
Contributor

@g123k We would need iOS developers with battery drain, for tests during development. If the scan process causes the battery drain, an Isolate would just run the battery draining process in parallel, while adding code complexity: that would not solve the problem at all.

JPEG format could be a good idea - is that available on all platforms? Of course that would probably mean changes in the croppers. Or we could just ignore the crop process in the end, depending on the performances.

But wait, the only problem is that we need developers with battery draining problems. Of course we can have good theoretical ideas regarding how to solve it, but currently we are playing rubik's cube blindfolded. I've wasted enough time coding the croppers only to realize that anyway no one I know had the battery draining problem.

Could any tester with battery draining problems lend a device?

@teolemon teolemon modified the milestones: V1, V1.1 May 24, 2022
@teolemon teolemon added P2 and removed 🎯 P0 labels Jul 9, 2022
@g123k
Copy link
Collaborator

g123k commented Oct 24, 2022

Should we consider that this issue is now resolved?

@M123-dev
Copy link
Member Author

Sounds good performance isn't really a problem anymore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
camera 🤳 MLKit P2 performance 🤳🥫 Scan We need to be able to scan on low-end, old devices, even with a bad camera, connexion… speed
Development

No branches or pull requests

6 participants