Issues with ÅÄÖ Swedish characters #79
Comments
|
Specifying the CCSID on the connection string should override any CCSIDs in the Try setting the CCSID to 1208 and see if that helps. That should make the driver convert from 278 (what your system/database are problem on) to UTF-8 encoding, so Node.js can convert it correctly. |
|
Hey! I tried 1208 with connection string, and also a DSN, but it still doesn't work. DSN=MYDSN;Password=MYPASSWORD;CCSID=1208; When I have CCSID 278 I get this error;
When I use 1208 the Express server crashes with no information. I'll try to debug it. |
|
The CCSID attribute for the ODBC driver doesn't set the job CCSID, it sets the ODBC's client CCSID. This is understandably confusing as CCSIDs are primarily associated with the IBM i itself and are usually EBCDIC. Even when running on IBM i, the driver and Node.js run in PASE, which is an ASCII environment. For the ODBC driver, you have to set an ASCII or UTF-8 CCSID (unless somehow you have a strange Windows or Linux that runs EBCDIC 😉). As Mark said, the Node.js interfaces used by node-odbc only supports converting to/from UTF-8, so CCSID should be set to 1208. |
|
I am now using CCSID=1208 in my connection string. This is my code: getSession never reaches the catch statement. statement.execute() crashes the node process it seems. BTW, my table is CCSID 278. |
|
@markdirish I wonder if this is related to SQL_NO_TOTAL or other length issues that are being worked on in the 2.3 branch. |
|
@MrTeapot can you build with
Then |
|
@MrTeapot does CCSID 1208 work with |
|
Here it it, it aint pretty: I also see this in my terminal: [Error: Invalid argument] |
odbc@latest gave me 2.2.2 I have the same issue with that version. |
|
Aha! Well I have a hunch what the issue is (ignore this technical stuff if you want, its more for me to have something to refresh my memory later): When converting from C data to a JavaScript string, the package calls: The length is gotten from the value returned in the StrLen_or_IndPtr. In the case of character conversions, it returns The fix should be to just NOT pass a size, which will make JavaScript assume that it is a null-terminated string. You could maybe test this yourself by changing this line: to and see if that fixes it. This function will read the char_data as a null-terminated string, not a string of a given length. I will try to create a CCSID 278 table on my system and see if I can recreate the error, then see if I can fix it by generating a Napi::String from a null-terminated C string. |
|
^ My hunch above is right, but you can't just make the change yourself. They are dependent on changes in this PR: #77. Will work on landing all of this soon. |
|
Great. Thank you! Looking forward to it. |
Actually, since the length is a size_t, it gets converted to unsigned, ie. -4 converted to a size_t (on 64-bit) is 0xFFFFFFFFFFFFFFFC or 1.84467440737e+19 (that's 18 exabytes...) |
|
@MrTeapot as a workaround for now, you can set DEBUG=524288 on the connection string or your DSN. This prevents the IBM i Access ODBC driver from returning SQL_NO_TOTAL. Note that this will cause a performance reduction as the the total data will have to be converted to calculate the actual length. On second thought, this might result in errors as #77 also fixes the length calculation when the returned length does not match the available length. |
|
@kadler Adding DEBUG=524288 stopped the crashing and allows ÅÄÖ. But, it adds a weird character at the end of strings that contain ÅÄÖ: This is the hex value: |
|
I have a PR open that should fix all of this. Try to install from my Hopefully that branch works, but if not let me know so I can fix it before landing the PR |
|
Yeah, as I mentioned later I had second thoughts this would work right. The problem is the bug in the node-odbc code that uses the indicator as the returned string length, when actually the indicator contains the length of the string that could be returned. Turning on the debug flag will cause the calculated length to be returned, but when you have expansion going to UTF-8 this could be bigger than the size of the column. So in your example, the column was maybe a CHAR(40) and a 40 byte buffer was bound for it, but the total length was 41, since Å takes 2 bytes in UTF-8. This causes a buffer over-read of random data to be returned in the string. |
It is working! Amazing :D |
|
The solutions works even for german Umlaute (äöüÄÖÜß)! Do you have an idea when 2.3.0 will be released? |
|
The solution works if i use ODBC but does not work in a loopback project that uses loopback-connector-ibmi ... any suggestion? tx |
|
Go into |
|
i have changed the version into node_modules/loopback-connector-ibmi/, then ran npm install, but it does not work |
|
@serininformatica I will continue to look into this. I'm not sure why loopback isn't playing nice with 1208. @MrTeapot and @nedi-dev would you mind downloading the latest |
|
I had to remove 2 dev modules to run "npm install" into loopback-connector-ibmi, now it's working |
odbc@beta is working! It gives me "odbc": "^2.3.0-beta.1" in package.json |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey!
I'm running node-odbc (2.3.0-beta.0) against IBMI DB2.
All SQL queries have issues with ÅÄÖ being replaced with �.
After reading #46 I figured it has something to do with CCSID; being the root of all evil.
I therefore added CCSID=278 to my connection string, but to no avail.
After adding CCSID I can't connect at all.
Where else can I specify the CCSID to get it to work properly?
The documentations talks about odbc.ini and odbcinst.ini. Do I specify it in those files? But where do I put the files, I currently don't have them anywhere.
Thanks in advance.
The text was updated successfully, but these errors were encountered: