possibility to set up manual connection charset on connect

[tangor86]

Hello

if there any possibility to setup manual connection CHARSET on connect?
It seems by default it is UTF8, that doesn't go for some projects...

best wishes!

[xdenser]

As for now it is not possible.
What for do you need other CHARSET? You will not be able to work correctly with strings in JavaScript.
libfbclient makes transcoding for you.
Other connetion charset support will require to return node buffers for all char, varchar fields.

[tangor86]

Client's default FB database charset is set to WIN1251 (cyrilic),
Also columns in tables have win1251 charset, and the collate also :)

But, the fact, that data in DB is in win1257 (baltic) charset.

(to change the charset of all DB, on all tables in DB is NOT possible because it is big and they use it like that, I mean their software is done to work with win1257 charset...)

So, I have to deal with 3 charsets:

1. win1251 - the default charset of DB,
2. win1257 - the charset of data stored in that DB,
3. utf8 - the charset of my website, that is meant to get data from nodejs and show them :).
   (nodejs uses your module to get data from FB DB)

In PHP, to work all fine, I do the following:

1. connecting to DB using win1251
2. converting received data from win1257 to utf8
3. showing the data on the page!
   It works fine!

After long hour of investigation, I found out, that libfbclient module using utf8 by default.
In fact, it means, that by the moment nodejs server sends data taken from FB — it is ALREADY broken (because of "transcoding" and/or utf8).
So in my page I see invalid characters.

I found the sting nr. 53 in your software:
char lc_type = "WIN1251"; / UTF8 */

Changed utf8 to win1251 with hope that it will help me.
It didn't help. I suppose because all your other code is written with utf8 in mind :)

I also tried "iconv" module to convert data taken from FB with your module, but now I thing it was no sense because at the moment it was get by module — it was broken by transcoding, because of difference of charsets of DB and Data (or maybe simply because and is not utf8).

P.S: iconv also didn't help.

In my situation, may you give some advice how to deal with charset...?

Thank you very much!

[xdenser]

Well to read non utf8 charsets fetch, fetchSync methods should return Buffers not strings.
That need major changes in fb-bindings-fbresult.cc.

For example for varchar field type line 392-394 may be changed to

vary2 = (PARAMVARY*) var->sqldata;
node::Buffer *slowBuffer = node::Buffer::New(vary2->vary_length);
memcpy(node::Buffer::Data(slowBuffer), (const char*)(vary2->vary_string), vary2->vary_length);
v8::Local<v8::Object> globalObj = v8::Context::GetCurrent()->Global();
v8::Local<v8::Function> bufferConstructor = v8::Local<v8::Function>::Cast(globalObj->Get(v8::String::New("Buffer")));
v8::Handle<v8::Value> constructorArgs[3] = { slowBuffer->handle_, v8::Integer::New(vary2->vary_length), v8::Integer::New(0) };
js_field = bufferConstructor->NewInstance(3, constructorArgs);

that should return you buffers for varchar fields. You may apply iconv to buffers then. For char() fields something similar should be done. But things become more complex when you need updates. Your only option is to use prepared statements and to patch FBResult::set_params method to use buffers instead of strings.

But anyway it will be custom patch - It will never go into master branch, Better way is to provide whole record buffer to JavaScript code and parse it in javascript. But this requires many changes to API - it will not be done fast.

And offcourse all this needs string nr. 53 to be patched too
char lc_type = "WIN1251"; / UTF8 */

[tangor86]

Hello,

I changed the code like so:
case SQL_VARYING:
/*
vary2 = (PARAMVARY_) var->sqldata;
vary2->vary_string[vary2->vary_length] = '\0';
js_field = String::New((const char_)(vary2->vary_string));
/
vary2 = (PARAMVARY) var->sqldata;
node::Buffer slowBuffer = node::Buffer::New(vary2->vary_length);
memcpy(node::Buffer::Data(slowBuffer), (const char)(vary2->vary_string), vary2->vary_length);
v8::Localv8::Object globalObj = v8::Context::GetCurrent()->Global();
v8::Localv8::Function bufferConstructor = v8::Localv8::Function::Cast(globalObj->Get(v8::String::New("Buffer")));
v8::Handlev8::Value constructorArgs[3] = { slowBuffer->handle_, v8::Integer::New(vary2->vary_length), v8::Integer::New(0) };
js_field = bufferConstructor->NewInstance(3, constructorArgs);
break;

        case SQL_SHORT:

Old code 392-394 commented, and a patch-code added after, but while compiling it gaves error:

web@webcaurlaides:/var/www/protected/nodejs/node_modules/firebird$ node-waf configure build
Checking for program g++ or c++ : /usr/bin/g++
Checking for program cpp : /usr/bin/cpp
Checking for program ar : /usr/bin/ar
Checking for program ranlib : /usr/bin/ranlib
Checking for g++ : ok
Checking for node path : not found
Checking for node prefix : ok /usr
Checking for program fb_config : not found
'configure' finished successfully (0.130s)
Waf: Entering directory /var/www/protected/nodejs/node_modules/firebird/build' [5/8] cxx: src/fb-bindings-fbresult.cc -> build/Release/src/fb-bindings-fbresult_1.o ../src/fb-bindings-fbresult.cc: In static member function āstatic v8::Local<v8::Value> FBResult::GetFieldValue(XSQLVAR*, Connection*)ā: ../src/fb-bindings-fbresult.cc:406:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:407:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:408:11: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:450:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:454:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:458:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:480:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:516:17: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:539:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:540:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:27: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:29: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:27: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:19: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc: In destructor āvirtual FBResult::~FBResult()ā: ../src/fb-bindings-fbresult.cc:886:131: warning: format ā%dā expects argument of type āintā, but argument 3 has type āISC_STATUS {aka long int}ā [-Wformat] ../src/fb-bindings-fbresult.cc:886:131: warning: format ā%dā expects argument of type āintā, but argument 4 has type ālong intā [-Wformat] ../src/fb-bindings-fbresult.cc:891:17: warning: converting to non-pointer type āisc_stmt_handle {aka unsigned int}ā from NULL [-Wconversion-null] Waf: Leaving directory/var/www/protected/nodejs/node_modules/firebird/build'
Build failed: -> task failed (err #1):
{task: cxx fb-bindings-fbresult.cc -> fb-bindings-fbresult_1.o}
web@webcaurlaides:/var/www/protected/nodejs/node_modules/firebird$

Unfortunately my C++ knowledge is poor to fix that error...

[tangor86]

But things become more complex when you need updates.

For the luck, in my node server I need only SELECT statements, so if the custom patch will finally be compilated, it would be a great save for me))

[xdenser]

Try w/o namespaces

vary2 = (PARAMVARY*) var->sqldata;
Buffer *slowBuffer = Buffer::New(vary2->vary_length);
memcpy(Buffer::Data(slowBuffer), (const char*)(vary2->vary_string), vary2->vary_length);
Local<Object> globalObj = Context::GetCurrent()->Global();
Local<Function> bufferConstructor = Local<Function>::Cast(globalObj->Get(String::New("Buffer")));
Handle<Value> constructorArgs[3] = { slowBuffer->handle_, Integer::New(vary2->vary_length), Integer::New(0) };
js_field = bufferConstructor->NewInstance(3, constructorArgs);

[xdenser]

Ok I'll try to make the changes, In saparate branch.

[tangor86]

New code:

case SQL_VARYING:
/*
vary2 = (PARAMVARY_) var->sqldata;
vary2->vary_string[vary2->vary_length] = '\0';
js_field = String::New((const char_)(vary2->vary_string));
/
vary2 = (PARAMVARY) var->sqldata;
Buffer slowBuffer = Buffer::New(vary2->vary_length);
memcpy(Buffer::Data(slowBuffer), (const char)(vary2->vary_string), vary2->vary_length);
Local globalObj = Context::GetCurrent()->Global();
Local bufferConstructor = Local::Cast(globalObj->Get(String::New("Buffer")));
Handle constructorArgs[3] = { slowBuffer->handle_, Integer::New(vary2->vary_length), Integer::New(0) };
js_field = bufferConstructor->NewInstance(3, constructorArgs);
break;

Gave the following on compilation:

node-waf configure build
Checking for program g++ or c++ : /usr/bin/g++
Checking for program cpp : /usr/bin/cpp
Checking for program ar : /usr/bin/ar
Checking for program ranlib : /usr/bin/ranlib
Checking for g++ : ok
Checking for node path : not found
Checking for node prefix : ok /usr
Checking for program fb_config : not found
'configure' finished successfully (1.234s)
Waf: Entering directory /var/www/protected/nodejs/node_modules/firebird/build' [5/8] cxx: src/fb-bindings-fbresult.cc -> build/Release/src/fb-bindings-fbresult_1.o ../src/fb-bindings-fbresult.cc: In static member function āstatic v8::Local<v8::Value> FBResult::GetFieldValue(XSQLVAR*, Connection*)ā: ../src/fb-bindings-fbresult.cc:406:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:407:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:408:11: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:450:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:454:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:458:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:480:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:516:17: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:539:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc:540:18: error: jump to case label [-fpermissive] ../src/fb-bindings-fbresult.cc:402:19: error: crosses initialization of āv8::Handle<v8::Value> constructorArgs [3]ā ../src/fb-bindings-fbresult.cc:401:21: error: crosses initialization of āv8::Local<v8::Function> bufferConstructorā ../src/fb-bindings-fbresult.cc:400:19: error: crosses initialization of āv8::Local<v8::Object> globalObjā ../src/fb-bindings-fbresult.cc:398:13: error: crosses initialization of ānode::Buffer* slowBufferā ../src/fb-bindings-fbresult.cc: In destructor āvirtual FBResult::~FBResult()ā: ../src/fb-bindings-fbresult.cc:886:131: warning: format ā%dā expects argument of type āintā, but argument 3 has type āISC_STATUS {aka long int}ā [-Wformat] ../src/fb-bindings-fbresult.cc:886:131: warning: format ā%dā expects argument of type āintā, but argument 4 has type ālong intā [-Wformat] ../src/fb-bindings-fbresult.cc:891:17: warning: converting to non-pointer type āisc_stmt_handle {aka unsigned int}ā from NULL [-Wconversion-null] Waf: Leaving directory/var/www/protected/nodejs/node_modules/firebird/build'
Build failed: -> task failed (err #1):
{task: cxx fb-bindings-fbresult.cc -> fb-bindings-fbresult_1.o}
web@webcaurlaides:/var/www/protected/nodejs/node_modules/firebird$

I compared some peace of that output just to find out if it is different from previous (just to make sure I made changes in source, and the compiler doe not cache anything) — they are different.

[xdenser]

Try this commit

fc6a54b

it compiles for me.

[tangor86]

Thank you very much, it was compiled for me also (with some "warnings")!
Now it works, so that symbols are normally converted to utf8 AND on the clients it is desired results!!

But unfortunately, every time, after 1-2 min. of work — nodejs fails with "Segmentation fault", after that patch...

What I do in my server?

1. Doing THE SAME select query to FB server, that returns an object (FBResult?) every 3 seconds
2. Converting pObj.FFIRST and pObj.FLAST fields from win1257 to utf8 with my func "myConv"

function myConv( pObj, from, to ) {

var obj = {};

obj.dir = from + ' -> ' + to;

try {
    var iconv = new Iconv(from, to);

    //var body = new Buffer(objCH.FFIRST);
    var FFIRST = pObj.FFIRST;
    obj.name = iconv.convert(FFIRST).toString();

    var FLAST = pObj.FLAST;
    obj.surname = iconv.convert(FLAST).toString();

} catch (err) {
    console.log( "myConv error:" + err.message );
    console.log( "myConv direc:" + from + ' -> ' + to );
}

return obj;

}

setInterval(function() {
var chData = getCardHoldersData(5); // it does SQL query by Id=5 to FB and returns object (FBResult?) as result
var myVar = 'Это тест русского!';

arr.push( myConv( chData, 'CP1257', 'UTF-8' ) );

//... broadcasting to clients after all work done
broadcast({
channel: REDIS_CHANNEL,
code: 300,
act: 'RefreshBlock',
name: chData.FFIRST,
surname: chData.FLAST,
poRusski: myVar,
convertations: arr
});

}, 3000);

All works fine instead of "Segmentation fault".
Maybe this will help:
http://stackoverflow.com/questions/9762616/why-does-this-crash-nodejs

Also, I can check (test) your new patches, if needed.

[xdenser]

You may try to remove/comment HandleScope declarations.

HandleScope scope1;

lines 392,412
in fb-bindings-fbresult.cc

I am unsure if I am using them correctly.

[tangor86]

I tried that, but the same "Seg fault" appears.

Also, I removed the call on iconv, just to be sure it is not a fault of iconv.

The logs are empty also...

[xdenser]

try this commit
530c7e1

[tangor86]

Started my server yesterday in the evening — still working!

thank you very much!

Can you please leave that issue for users who will have the same need as me (not to have "utf8" charset at connection) ? They will use that code, branch for patching...

[xdenser]

You also may now try a very perspective pure JS solution from here

https://github.com/hgourvest/node-firebird

thought it has no synchronous versions of methods and possibly should be patched too for your case.