Demos without user interface are not working (Solved: Use only TWAIN drivers, not WIA drivers)

[gerardcarbo]

In the C demos like SimpleFileAcquireToBMPNoUserInterface (32 bits) the source can be selected and the output is:

The selected source is "WIA-HP Deskjet 3050 J610 series"
Acquiring to file "SimpleFileAcquireToBMPNoUserInterface.bmp"
Acquired "SimpleFileAcquireToBMPNoUserInterface.bmp" successfully

But no image is captured, and the scanner does not scan (does not move). Tested with a HP and Fujitsu scanners.

In the C++ wrapper demo SimpleFileAcquireToBMPNoUserInterface 32 bits the response is:

WIA-HP Deskjet 3050 J610 series
Canceled

With demo AdvancedBufferedAcquire the output is:

The selected source is "WIA-HP Deskjet 3050 J610 series"
 UI is opened
 UI is closing
 UI is closed
Images could not be acquired: -1109
status code: 1002

With DTWDemo it is working properly.

[dynarithmic]

Please see if the issue occurs in the upcoming version of DTWAIN in the master-staging branch.

There are several major changes in this version regarding the internal message processing. See this discussion.

[gerardcarbo]

Same error using AdvancedBufferedAcquire demo and using new dll:

The selected source is "WIA-HP Deskjet 3050 J610 series"
 UI is opened
 UI is closing
 UI is closed
Images could not be acquired: -1109
status code: 1002

[dynarithmic]

ok. I will take a look at this and update further.

[dynarithmic]

One question: Are the scanners in duplex mode when you are acquiring the pages without the UI?

Once difference in DTWDEMO is that the DTWAIN_AcquireBufferEx call has the scanner retrieving all the pages (DTWAIN_ACQUIREALL), while the other demo program only asks for a single page.

[gerardcarbo]

They are single side scanners.

I've tested to specify DTWAIN_ACQUIREALL in AdvancedBufferedAcquire with the same result (Images could not be acquired: -1109)

DTWAIN_AcquireBuffered(theSource, /* The TWAIN Data Source that was selected */
                                     DTWAIN_PT_DEFAULT,  /* Default pixel (color) type*/
                                     DTWAIN_ACQUIREALL,       /* Acquire a single page*/
                                     TRUE,   /* Show the user interface */
                                     TRUE,    /* Close the source when DTWAIN_AcquireFileA returns*/
                                     &status); /* return status */

[dynarithmic]

The -1109 indicates a TWAIN triplet sequence error. You mentioned that the DTWDEMO program works when specifying to acquire to buffered mode, is that correct? If so, then the difference between the two programs is

1. the number of pages to acquire (1 in AdvanceBufferedAcquire, as opposed to DTWAIN_ACQUIREALL in dtwdemo).
2. The user-defined buffer in AdvancedBufferedAcquire.
3. The DTWAIN_SetTwainMode is set to DTWAIN_MODELESS in dtwdemo.

Since a lot of changes were done in the unreleased 5.3.0.2 version of the DLL's, can you comment out (in DTWDEMO.c) the call to DTWAIN_SetTwainMode, rebuild that app, and see if it still works? If it does, that takes the focus away from the new message loop processing that 5.3.0.2 has introduced.

I tested on a Plustek 286 scanner, and didn't run into issues when scanning in simplex mode. For duplex mode, I do see one potential issue which I will have to address shortly in the 5.3.0.2 update.

What brand of Fujitsu scanner are you testing with? Since I may not have the brand of scanners you are testing on (I don't have the HP model you are using), the next step is to turn on the TWAIN logging and produce a log file to see which set of TWAIN calls is causing the issue. See DTWAIN_SetTwainLog() -- you can look at DTWDEMO.c in the sample source to see how to call this function.

[gerardcarbo]

Hi Paul, DTWDEMO works properly in the tree cases (bufered, native and file). Here you have the logs using 'file' successfully:

DTWDemo.txt

On the other side using SimpleNativeAcquire, AdvancedBufferedAcquire or SimpleFileAcquireToBMPNoUserInterface it does not work producing the logs:

SimpleFileAcquireToBMPNoUserInterface.txt

The Fujitsu is a Fi-65f and the HP a Deskjet 3050 J610.

[dynarithmic]

I looked at the DTWDEMO logs. The DTWAIN_SetTwainMod is still set to 1.

[2023-03-04 10:58:02]    ===>>>Entering DTWAIN_SetTwainMode(lMode=1)
[2023-03-04 10:58:02]    <<<===Exiting DTWAIN_SetTwainMode returning value of 1

I would like to see if the demo program works if you comment out the line

DTWAIN_SetTwainMode(DTWAIN_MODELESS);

rebuild, and rerun DTWDEMO.

[dynarithmic]

Also, in the DTWDEMO program, please test without showing the Source UI. Uncheck the option in the Acquire Test main menu (Use Source UI).

[dynarithmic]

I believe I've simulated the error. The problem seems to be that for some odd reason, when the UI is turned off, the Source is not sending the MSG_XFERREADY to initiate the scanning process. Instead, the source is sending the request to close down the scanning operation, as if the scanning already took place.

This is very strange behavior, since it seems to be intermittent (DTWDEMO it works, but simpler applications, it is giving the issue).

I know this sounds strange, but what happens if you selected the Source again, immediately after selecting it the first time? For example:

DTWAIN_SOURCE source = DTWAIN_SelectSource(); // First time
source = DTWAIN_SelectSource();  // Second time.

and make sure you select the same source both times. Does the scanning go through?

[dynarithmic]

OK, I went through my inventory of scanners, and got a hold of a Fujitsu 5120C, and it seems to show the same issue. I tested some changes that may be a workaround to the issue. I will probably put out test DLLs in the experimental branch when ready.

The problem seems to be that the TWAIN driver sources for the problem scanners may be written in a way so as to turn the user-interface off automatically as soon as a scan has occurred, instead of having the user manually hit an "Exit" or "Cancel" button. This may account for the spurious requests to close the user interface in no-UI mode before the scan has occurred. That's the experience I had with the 5120C.

The Plustek 286 and HP 7000 S3, and the TWAIN sample source all keep their user interfaces open, even after a scan has occurred -- the user has to interact with the UI for the UI to close down. Also, in no-UI mode, these drivers properly (and consistently) send the transfer request before the close UI request, so there were no issues with those scanners. This doesn't seem to be the case with the Fujitsu driver, at least the WIA driver.

[dynarithmic]

Just another question -- do you have drivers that are not WIA? It looks like it may be an issue with the WIA drivers (you can tell it's WIA by looking at the Product Name that shows up in the TWAIN select source dialog and seeing if the name starts with "WIA").

There has always been an issue with the WIA drivers -- that's why many manufacturers ship both "real" TWAIN drivers, and the stripped down WIA drivers. If you do have non-WIA drivers for the scanners, can you select those drivers and see if the issue persists?

Just to let you know, the scanners that I am using that are having no issues are all non-WIA TWAIN drivers, not WIA drivers.

[dynarithmic]

I think I see the root cause of the issue.

The problem is that these sources/drivers do eventually post a MSG_XFERREADY request to trigger the start of the scan, however they send this request after they have sent the request to close the user-interface. I did some further debugging/investigation, and indeed this is what's happening, at least with the Fujitsu scanner.

So basically, this will throw off a lot of the logic that depends on sources to get it right. The sources that get it right follow this basic algorithm:

1. If a request to transfer occurs, send the transfer request(s) (there could be more than one, depending if the device has a feeder)

2. Close the scanning down only in the scenarios:
   a) automatically in the case of no-UI,
   b) in the case of the user pressing the "Cancel", "Done", "Exit" etc. buttons on the UI,
   c) the application (your app/DTWAIN) explicitly demanding the UI to be closed down.

Note that no transfer requests are sent if the user cancelled without scanning pages, or if the app wants to close the UI down for whatever reason before scanning any pages.

If the driver decides on their own to do step 2) before step 1) (in other words, without knowledge of the user or your application) this will throw off DTWAIN causing the scan to fail (as in your case).

A fix for this would require changes and thorough testing, since a lot of current logic is tied into the UI being closed down (such as logic to simulate a document feeder for flatbed scanners, so as to produce multi-page image files). I will have the upcoming 5.3.0.2 DLL's for release, but they will not include any changes to address this issue. The 5.3.0.2 version address a much more broader issue with the general TWAIN message processing.

I will keep you posted on progress on this issue -- so far I believe I know exactly what the problem is, it's just that the solution requires time to implement.

[dynarithmic]

Just to let you know --

I tested VueScan drivers with DTWAIN, and they work correctly for the Fujitsu 5120C (VueScan shows up as a TWAIN device in the Select Source dialog).

Note that VueScan claimed to have reverse engineered the Fujitsu drivers, so they are not working with the original driver that I'm using that is giving the issue. The VueScan engineers more than likely addressed this issue with their driver rewrite.

The thing is that the product (VueScan) is not free, so you will be paying for drivers that actually work correctly. In the meantime, I will try to work with the original drivers to come up with a solution.

[gerardcarbo]

Ok, great. Thank you very much!
You are right, problem is with the WIA drivers. I installed the native drivers of the scanners and with them the non IU demos are working properly.

[dynarithmic]

That sounds good.

The WIA drivers, even though they sound like a good idea in theory, provide limited functionality when compared to a true TWAIN driver. One issue I remember is that no-UI mode was a problem to deal with, as well as not being able to use a document feeder for more than a single page.

I will probably keep this issue opened for a couple of days before closing it out.

[gerardcarbo]

Do you think there is any hack to make it work with WIA drivers (put a sleep somewhere...)?

[dynarithmic]

The only thing I can think of is to record the close request and not do anything until a message for the "start scanning" is sent. Probably wait for 100 or so messages to go by, and if no "start scanning" is detected, just close out the scanner (the user actually wanted to quit scanning).

[dynarithmic]

I commited new 5.3.0.3 DLLs in the experimental branch. Also, you will need the updated .ini and .txt files. Whenever you are ready, you can test these to see if the issue with the WIA drivers have been reduced, if not solved.

Basically, the .INI files contain a section that lists all of the devices that have an issue with the closure of the scan window before the request to scan has been sent. The [SourceXferWaitInfo] section in the .INI file indicates which devices will have specialized processing done to mitigate the WIA issue.

Note that wildcards can be used in the setting -- in this case, any device that has a name that starts with WIA- is to be included as one that needs to be handled in this way.

The =1000 setting tells the maximum number of messages that can be sent before a "transfer ready" message is seen. So basically if after 1000 messages after the initial "close UI" has been received, if we don't get a "transfer ready", then the UI closes down (this allows the user to actually hit the cancel button and scan no images). This is similar to what you mentioned about having a sleep, but we are sleeping for 1000 messages, waiting for a transfer ready to be sent by the driver.

This has not been finalized (which is why it is in the experimental branch), but from the little testing that I did, it seems to work for the Fujitsu scanner.

[gerardcarbo]

Great! tested with WIA drivers for HP and Fujitsu scanners and working properly with GUI and non GUI demos. Also tested with TWAIN drivers and working OK.

[dynarithmic]

That sounds good. As long as the scanning occurs, that's about as much as can be asked for with the WIA drivers.

The only issue is that those WIA drivers that behave this way can't be used for things like manual feeder mode (basically the ability to produce multi-page image files using a flatbed scanner). The closing of the UI automatically will prevent those DTWAIN features from being realized.

[gerardcarbo]

You are right I've tested with WIA drivers and I only get one image even if the scanner is duplex.
On the other side, playing with the sources (changing from one driver to another) sometimes on the DTWAIN_SysDestroy call I get the exception:

ntdll.dll!77207073()    Unknown
ntdll.dll![Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]    Unknown
[External Code]   
[Inline Frame] dtwain32.dll!boost::dll::detail::shared_library_impl::unload() Line 120    C++
[Inline Frame] dtwain32.dll!boost::dll::shared_library::unload() Line 286    C++
dtwain32.dll!dynarithmic::CTL_TwainAppMgr::UnloadSourceManager() Line 314    C++
dtwain32.dll!dynarithmic::CTL_TwainAppMgr::Destroy() Line 113    C++
dtwain32.dll!DTWAIN_EndTwainSession() Line 1562    C++
dtwain32.dll!DTWAIN_SysDestroy() Line 1623    C++

[dynarithmic]

Unloading the source manager should eventually call the FreeLibrary() API function for TWAIN32_DLL. So not sure why an exception would be thrown there.

Are you getting this when using the WIA drivers, or simply using the TWAIN drivers?

[gerardcarbo]

This exception happens very occasionally when changing from one type of driver (WIA or TWAIN) and scanner (Fujitsu fi-65/Fujitsu 800r) to another, after scanning. Maybe it's a driver problem that surfaces on DTWAIN_SysDestroy(). I'm trying to further reproduce the problem, and I will get back to you if I can find an easy way to reproduce it.

[dynarithmic]

I will be closing this issue, since version 5.3.0.3 was introduced that bypassed the WIA issue. #47

[dynarithmic]

Opened a new issue for the changing of the drivers and the occasional crash. #50