possibility to set up manual connection charset on connect #9
Comments
|
As for now it is not possible. |
|
Client's default FB database charset is set to WIN1251 (cyrilic), But, the fact, that data in DB is in win1257 (baltic) charset. (to change the charset of all DB, on all tables in DB is NOT possible because it is big and they use it like that, I mean their software is done to work with win1257 charset...) So, I have to deal with 3 charsets:
In PHP, to work all fine, I do the following:
After long hour of investigation, I found out, that libfbclient module using utf8 by default. I found the sting nr. 53 in your software: Changed utf8 to win1251 with hope that it will help me. I also tried "iconv" module to convert data taken from FB with your module, but now I thing it was no sense because at the moment it was get by module — it was broken by transcoding, because of difference of charsets of DB and Data (or maybe simply because and is not utf8). P.S: iconv also didn't help. In my situation, may you give some advice how to deal with charset...? Thank you very much! |
|
Well to read non utf8 charsets fetch, fetchSync methods should return Buffers not strings. For example for varchar field type line 392-394 may be changed to vary2 = (PARAMVARY*) var->sqldata;
node::Buffer *slowBuffer = node::Buffer::New(vary2->vary_length);
memcpy(node::Buffer::Data(slowBuffer), (const char*)(vary2->vary_string), vary2->vary_length);
v8::Local<v8::Object> globalObj = v8::Context::GetCurrent()->Global();
v8::Local<v8::Function> bufferConstructor = v8::Local<v8::Function>::Cast(globalObj->Get(v8::String::New("Buffer")));
v8::Handle<v8::Value> constructorArgs[3] = { slowBuffer->handle_, v8::Integer::New(vary2->vary_length), v8::Integer::New(0) };
js_field = bufferConstructor->NewInstance(3, constructorArgs);that should return you buffers for varchar fields. You may apply iconv to buffers then. For char() fields something similar should be done. But things become more complex when you need updates. Your only option is to use prepared statements and to patch FBResult::set_params method to use buffers instead of strings. But anyway it will be custom patch - It will never go into master branch, Better way is to provide whole record buffer to JavaScript code and parse it in javascript. But this requires many changes to API - it will not be done fast. And offcourse all this needs string nr. 53 to be patched too |
|
Hello, I changed the code like so: Old code 392-394 commented, and a patch-code added after, but while compiling it gaves error: web@webcaurlaides:/var/www/protected/nodejs/node_modules/firebird$ node-waf configure build Unfortunately my C++ knowledge is poor to fix that error... |
For the luck, in my node server I need only SELECT statements, so if the custom patch will finally be compilated, it would be a great save for me)) |
|
Try w/o namespaces vary2 = (PARAMVARY*) var->sqldata;
Buffer *slowBuffer = Buffer::New(vary2->vary_length);
memcpy(Buffer::Data(slowBuffer), (const char*)(vary2->vary_string), vary2->vary_length);
Local<Object> globalObj = Context::GetCurrent()->Global();
Local<Function> bufferConstructor = Local<Function>::Cast(globalObj->Get(String::New("Buffer")));
Handle<Value> constructorArgs[3] = { slowBuffer->handle_, Integer::New(vary2->vary_length), Integer::New(0) };
js_field = bufferConstructor->NewInstance(3, constructorArgs); |
|
Ok I'll try to make the changes, In saparate branch. |
|
New code: case SQL_VARYING: Gave the following on compilation: node-waf configure build I compared some peace of that output just to find out if it is different from previous (just to make sure I made changes in source, and the compiler doe not cache anything) — they are different. |
|
Try this commit it compiles for me. |
|
Thank you very much, it was compiled for me also (with some "warnings")! But unfortunately, every time, after 1-2 min. of work — nodejs fails with "Segmentation fault", after that patch... What I do in my server?
function myConv( pObj, from, to ) { } setInterval(function() { //... broadcasting to clients after all work done }, 3000); All works fine instead of "Segmentation fault". Also, I can check (test) your new patches, if needed. |
|
You may try to remove/comment HandleScope declarations. HandleScope scope1; lines 392,412 I am unsure if I am using them correctly. |
|
I tried that, but the same "Seg fault" appears. Also, I removed the call on iconv, just to be sure it is not a fault of iconv. The logs are empty also... |
|
try this commit |
|
Started my server yesterday in the evening — still working! thank you very much! Can you please leave that issue for users who will have the same need as me (not to have "utf8" charset at connection) ? They will use that code, branch for patching... |
|
You also may now try a very perspective pure JS solution from here https://github.com/hgourvest/node-firebird thought it has no synchronous versions of methods and possibly should be patched too for your case. |
|
hello! testing a new patch on my server, it is find out that it also fails with "Segmentation fault" after 20-30 min of work... :( |
|
Please show your db related code as gist. It has syntax high lighting. From code I have seen in other issue - you are creating connections constantlym but not colsing them. Either use single cnnection and transactions or connect every time with new connection, but close it after. If you do not close connections - you have memory leak. |
|
https://gist.github.com/2876204 My new code establish 1 connection (actually 2, because I have to take data from 2 different DB). And after quering I do: What is interesting, before that changes (connected with starting and committing transactions), I was leaving server for 1 day and night and it was totally fine: So, it seams seg fault started after "transaction changes"... Maybe I should:
And on the next iteration (I mean: "setTimeout(fbGetAndPushEvents, fbGetAndPushEventsTimeout);") — I will do a new connections again and so on? Is it critical to create a 2 new connections every 900 ms and close them after some 100 ms?? If creating the connection to FB does not take much time and CPU, I will try, because have something to do with seg fault... |
|
меня смущает этот комментарий // если не делать эту проверку то при комите выдает ошибку чтото типа "cannot handle transaction..." очень странно а все понял сразу после var rows = rs.fetchSync("all", true); там же внутри цикла перебора идут еще обращения к базе. совсем правильно начать транзакцию в начале вызова большой функции странно что падает с SegFault попробую повторить у себя |
|
Там есть СКЛ которые выполняются когда новый клиент присоединяется к серверу — это одно. А есть общий цикл который каждые Х секунд выполняет некоторые СКЛ по разным таблицам — это другое (об этом цикле вы писали). Я сейчас сделал следующее — оставил только создание и комит транзакций в главном цикле, и оставил сервер крутится на ночь, посмотрим, что будет на утро. |
|
Ничего не будет тоже дает SegFault |
|
Только что проверил, да, меня тоже вылетел с Segmentation fault... Попробую сейчас сделать с conn.disconnect();
А что значит "forever" ? |
|
Модуль для "демонизации", заодно рестарт приложения при падении npm install forever -g https://github.com/nodejitsu/forever а с SEGFAULT проблема однозначно в патче, потому что мастер ветка не падает |
|
В варианте без транзакций и с conn.disconnect(); он проработал 4 часа и потом вылетел с SEGFAULT, то есть по моему мнению стабильней чем с транзакциями и постоянным открытым соединением.. запустил еще раз.. |
|
Да, падать в дисконект-варианте он стал немного поменьше, но все равно падает. |
|
Все это работает нормально. Более 100 тыс. итераций ни одного падения. Спасибо Nathan Rajlich. |
|
Спасибо Вам большое Денис! За последнюю ночь и за сегоднешний день еще ни разу не вылетел мой сервер. На всякий случай, для надежности запущу его под forever в продакшене! Удачного дня! |
|
We have similar problem, trying to run Interbase with library from firebird, it's working in UTF8 but ours database is in WIN1250, and there are errors on field that are in Polish chars (about field to long). We have tried to run branch cp1251 - but it's not working with current version of node , is there any chance to update this branch to current version of node ? |
|
best way is to migrate database to UTF-8 with fbclone https://github.com/zedalaye/fbclone , also this lib is for Firebird |
|
Подскажите как собрать код из ветки cp1251, на node.js 4.6.2x32? Сборщик за пару лет изменился на node-gyp пробовал изменить сбоку, по аналогии с тем что в последнем релизе модуля, но не помогает. |
|
Sorry for late answer Check out char *lc_type = const_cast<char *>("UTF8"); to char *lc_type = const_cast<char *>("WIN1251"); and rebuild. |
|
I'm with a similar problem from a database that is used here, and it worked with the lc_type change, I think that we can add a charset (default 'utf8') on the connection functions. and other thing I see is the:
On the fbresult SQL_VARYING should be something like: That should remove the zeroing from the buffer on JS code. I'm using iconv-lite for decode the results and is working fine, except from a zeroing that is coming from the buffer and something like a 16-bit code on the start of the buffer (string size I guess). |
Hello
if there any possibility to setup manual connection CHARSET on connect?
It seems by default it is UTF8, that doesn't go for some projects...
best wishes!
The text was updated successfully, but these errors were encountered: