Possible problem with Hamamatsu camera

[ilyasdc]

We have a Hamamatsu C11440-36U which sometimes irrevocably crashes and requires a hard restart (turned on and off) irrespectively of the restart of MM or even the PC with this kind of message displayed:

A script as simple as that can reproduce the issue by repeateadly taking images:

import java.util.Date;
import java.text.SimpleDateFormat;

public class NowString {
    public static String getTime() { 
          SimpleDateFormat df = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss.SSS");//Set the date format
          String date = df.format(new Date());//new Date() is to get the current system time
          return date;
    }
}

mm.getScriptController().clearMessageWindow();
mm.scripter().message("Starting...");

mmc.setAutoShutter(false); // disable auto shutter 
mmc.setShutterOpen(true);  // keeps the shutter open
mm.live().setLiveMode(false);

long i = 0;
for (;;i++) {
		image = mm.live().snap(true).get(0);
		print(i+"\t"+NowString.getTime());
}

It can trigger the bug more or less randomly (might take hours, might take days or even weeks).

After discussion with Hamamatsu, it appears that something /might/ be off on micro-manager side (not sure but that's what /they/ claim). I provide the decoded camera log captured by hamamatsu software (don't remember the name on top of my head but thyey directly provided it during the debug process)
: 230617_camera_logs_crash.pdf

According to the engineer, when capturing an image with snapImage() a call to dcamcap_start is made on the DCAM-API side but is not followed by dcamcap_stopwhich /could/ be the source of the problem. I'm not sure that's totally true since it can be that the buffer only has 1 image, or maybe not ? Anyway, since there's no public access to the camera device adapter I cannot look further.

The real question here, maybe: is this a normal usecase to use this snap() function repeatedly ? Any alternative ? it doesn't seem to do anything else than to call MMCore.snapImage() in our case (single channel image)

Thanks !

[ilyasdc]

Some extra info (the mm logs when this happens, can provide the full log upon request):

2023-06-16T17:49:13.854860 tid8188 [dbg,Core] Will snap image from current camera
2023-06-16T17:49:13.866828 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
2023-06-16T17:49:13.872811 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:13.928662 tid8188 [dbg,Core] Did snap image from current camera
2023-06-16T17:49:13.973542 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
2023-06-16T17:49:14.007451 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:14.014433 tid8188 [dbg,Core] Will snap image from current camera
2023-06-16T17:49:14.108182 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
2023-06-16T17:49:14.144085 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:14.244816 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
2023-06-16T17:49:14.252795 tid8188 [dbg,Core] Did snap image from current camera
2023-06-16T17:49:14.264763 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:14.365493 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
2023-06-16T17:49:14.385440 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:14.486170 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
2023-06-16T17:49:14.520079 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:14.620810 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
2023-06-16T17:49:14.640757 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:14.741487 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
[... 2 previous lines repeat many times ...]

2023-06-16T17:49:30.698783 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:30.793529 tid8188 [ERR,Core] Failed to snap image from current camera
2023-06-16T17:49:30.793529 tid8188 [ERR,Core] Error occurred in device CMMCore::snapImage: Error in device "HamamatsuHam_DCAM": Unknown error in the device (1)
2023-06-16T17:49:30.793529 tid8188 [IFO,App] Failed to snap image
                                   [       ] java.lang.Exception: Error in device "HamamatsuHam_DCAM": Unknown error in the device (1) in Thread[Thread-11,6,main]
                                   [       ]   at mmcorej.MMCoreJJNI.CMMCore_snapImage(Native Method)
                                   [       ]   at mmcorej.CMMCore.snapImage(CMMCore.java:928)
                                   [       ]   at org.micromanager.acquisition.internal.DefaultAcquisitionManager.snap(DefaultAcquisitionManager.java:262)
                                   [       ]   at org.micromanager.internal.SnapLiveManager.snap(SnapLiveManager.java:848)
                                   [       ]   at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
                                   [       ]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                   [       ]   at java.lang.reflect.Method.invoke(Method.java:498)
                                   [       ]   at bsh.Reflect.invokeMethod(Reflect.java:131)
                                   [       ]   at bsh.Reflect.invokeObjectMethod(Reflect.java:77)
                                   [       ]   at bsh.BSHPrimarySuffix.doName(BSHPrimarySuffix.java:170)
                                   [       ]   at bsh.BSHPrimarySuffix.doSuffix(BSHPrimarySuffix.java:114)
                                   [       ]   at bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:74)
                                   [       ]   at bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:41)
                                   [       ]   at bsh.BSHAssignment.eval(BSHAssignment.java:71)
                                   [       ]   at bsh.BSHBlock.evalBlock(BSHBlock.java:125)
                                   [       ]   at bsh.BSHBlock.eval(BSHBlock.java:75)
                                   [       ]   at bsh.BSHBlock.eval(BSHBlock.java:41)
                                   [       ]   at bsh.BSHForStatement.eval(BSHForStatement.java:105)
                                   [       ]   at bsh.Interpreter.eval(Interpreter.java:659)
                                   [       ]   at bsh.Interpreter.eval(Interpreter.java:750)
                                   [       ]   at bsh.Interpreter.eval(Interpreter.java:739)
                                   [       ]   at org.micromanager.internal.script.BeanshellEngine$EvalThread.run(BeanshellEngine.java:36)
2023-06-16T17:49:30.799513 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
2023-06-16T17:49:30.815471 tid8188 [IFO,App] Error: Line 24: Null Pointer Exception
2023-06-16T17:49:30.819460 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:30.920191 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
2023-06-16T17:49:30.954100 tid5684 [dbg,dev:COM3] GetAnswer <- CSSAXF000BXF000CXF000\r\n
2023-06-16T17:49:31.054830 tid5684 [dbg,dev:COM3] SetCommand -> CSS?\r
[the last two line repeat until I close MM]

Micro-manager version (I guess what matter is the Device API version)

MM Studio version: 2.0.1 20220720
MMCore version 10.2.0
Device API version 70, Module API version 10
User: User
Host: DESKTOP-LRLD4CI

[marktsuchida]

The real question here, maybe: is this a normal usecase to use this snap() function repeatedly ? Any alternative ? it doesn't seem to do anything else than to call MMCore.snapImage() in our case (single channel image)

Yes, that usage should not pose a problem, but it is not the most typical either (more often, there will be a fixed time interval between repeated snaps). If you think the crash is caused because of it, you could try inserting a brief sleep (say, 10 ms) between snaps, or use a sequence acquisition (or live mode, if displaying the images is all that is needed) instead.

[ilyasdc]

Hi @marktsuchida and thank you for replying

Yes, that usage should not pose a problem, but it is not the most typical either (more often, there will be a fixed time interval between repeated snaps).
If you think the crash is caused because of it, you could try inserting a brief sleep (say, 10 ms) between snaps, or use a sequence acquisition (or live mode, if displaying the images is all that is needed) instead.

Actually, the simplified version of the script omits this sleep delay which exists in another script (100ms in our case) which is slighly more involved since it processes the images for which such failure already occured.

I wanted to simplify it as much as possible to find the source of the problem. If I find some time, I will try to trigger a camera crash again with the simple script and an added delay (it's kind of tedious because it can take a week to occur).

I still quite don't understand how it is possible that a hardware failure happens if pictures are taken too quickly.

[ilyasdc]

Hello @nicost, is it possible to reopen the thread ? I know that you have more important bugs to fix but I want to be sure it's not a hardware problem.
As I stated previously, adding a delay doesn't fix the problem. However I have to say that since I don't have access to the camera adapter nor have the necessary knowledge to dig into it, I'm not sure I can fix it without your help. If you can have a look at this camera adapter code, can you tell me if, when using MM snapImage() method, the dcam_start uses the argument DCAM_START_SNAP, would be a huge help !

[palmada]

Have you considered using the camera sequence function from the core object instead? That starts the camera acquiring frames in a continuous sequence. You then have to extract them from the buffer and add them to an acquisition so a tiny bit more code but should do what you need without any issues.

[ilyasdc]

Hello @palmada and thank you for your follow-up on this issue. Is setting a 1 frame sequenceAcquisition (https://valelab4.ucsf.edu/~MM/doc-2.0.0-gamma/mmcorej/mmcorej/CMMCore.html#startSequenceAcquisition-int-double-boolean-) different from using the snapImage from the device adapter point of view ?

For context I am computing sharpness of a z-stack, like the AutoFocus plugin, but with an imposed delay (100ms) since we have also problem with the stage (see micro-manager/micro-manager@f9c6e7a#commitcomment-95719958 ).

[palmada]

Yes it is indeed different, but not immensely so. If you want to use a delay, then it's not appropriate, as it starts a stream with no delay.

One trick that might work for you instead is calling the StopSequenceAcquisition core method (here), which may call stop after a snap. A wild guess though as the device adapter code isn't open source, last I checked.

[ilyasdc]

Yes it is indeed different, but not immensely so. If you want to use a delay, then it's not appropriate, as it starts a stream with no delay.

One trick that might work for you instead is calling the StopSequenceAcquisition core method (here), which may call stop after a snap.
Ok I will try to check if the call to that method is shown in the logs of this usb monitoring tool that I was provided with.
A wild guess though as the device adapter code isn't open source, last I checked.
It's not, indeed.

[nicost]

Your best bet is to contact Hamamatsu support. Hamamatsu maintains the code to the device adapter. The bug is surely either in the device adapter, the Hamamatsu driver/SDK, or the camera firmware, and there is nothing we can (or should) do about any of those. If you can use a sequence acquisition (as suggested by @palmada), that is surely a good workaround the problem.

[ilyasdc]

Your best bet is to contact Hamamatsu support. Hamamatsu maintains the code to the device adapter. The bug is surely either in the device adapter, the Hamamatsu driver/SDK, or the camera firmware, and there is nothing we can (or should) do about any of those. If you can use a sequence acquisition (as suggested by @palmada), that is surely a good workaround the problem.

Hi @nicost, thanks for your input. I will contact Hamamatsu, try to use StopSequenceAcquisition and see if it still triggers the issue.