[Error] - 500 HTTP Error Response with Debug output '[ERROR] BBRF server error: os_process_error'

[xqd-ai]

Hi, I started receiving 500 HTTP error responses today when performing bbrf domains / urls / programs ...etc, however bbrf show example.com still works normally, I had more than 3 Mil urls and domains
bellow is the debug ouput:

root@ ~ # bbrf domain add www.slack.com -p @INFER
/usr/lib/python3/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.6) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
[DEBUG] getting program blacklist
[DEBUG] adding documents in bulk
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): IP:PORT
DEBUG:urllib3.connectionpool:https://IP:PORT "POST /bbrf/_all_docs?include_docs=true HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://IP:PORT "GET /bbrf/_design/bbrf/_view/programs?reduce=false&key=false HTTP/1.1" 500 73
[ERROR] BBRF server error: os_process_error
root@ ~ # bbrf urls -p slack
/usr/lib/python3/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.6) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): IP:PORT
DEBUG:urllib3.connectionpool:https://IP:PORT "GET /bbrf/_design/bbrf/_view/urls_by_program?reduce=false&key=%22slack%22 HTTP/1.1" 500 73
[ERROR] 'rows'

Bellow is the last output of docker logs container-name output, there's multiple errors, one is 'out of memory' however I have 64GB of memory, and the utilization now is not more than 4GB:

[info] 2021-08-19T00:40:02.061306Z nonode@nohost <0.512.0> -------- Starting index update for db: shards/80000000-ffffffff/bbrf.1621544843 idx: _design/bbrf
out of memory
[error] 2021-08-19T00:40:03.534489Z nonode@nohost <0.11558.0> -------- OS Process Error <0.11577.0> :: {os_process_error,{exit_status,1}}
[info] 2021-08-19T00:40:03.534617Z nonode@nohost <0.231.0> -------- couch_proc_manager <0.11577.0> died normal
[error] 2021-08-19T00:40:03.534968Z nonode@nohost emulator -------- Error in process <0.11558.0> with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}

[error] 2021-08-19T00:40:03.535004Z nonode@nohost <0.11554.0> 394c758296 rexi_server: from: nonode@nohost(<0.11410.0>) mfa: fabric_rpc:map_view/5 throw:{os_process_error,{exit_status,1}} [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,138}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,82}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,262}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,138}]}]
[error] 2021-08-19T00:40:03.535293Z nonode@nohost <0.11410.0> 394c758296 req_err(3722911048) os_process_error : {exit_status,1}
    [<<"couch_mrview_util:get_view_index_state/5 L138">>,<<"couch_mrview_util:get_view/4 L82">>,<<"couch_mrview:query_view/6 L262">>,<<"rexi_server:init_p/3 L138">>]
[notice] 2021-08-19T00:40:03.535732Z nonode@nohost <0.11410.0> 394c758296 IP:PORT ::ffff:172.17.0.1 bbrf GET /bbrf/_design/bbrf/_view/programs?reduce=false&key=false 500 ok 1475

I suspect this is something with the couchdb, but don't have an idea what it's, I tried multiple times to restart the container, but with the error still persist, anyone have an idea how to solve this?

[xqd-ai]

I figured out the problem, it was due to couchjs default heap memory limit of 64MB,
The issue was resolved by increasing couchjs heap memory to 512MB
Steps

• login inside the bbrf-server docker

docker exec -it container-name bash

• Change the value of export COUCHDB_QUERY_SERVER_JAVASCRIPT inside the file /opt/couchdb/bin/couchdb from

export COUCHDB_QUERY_SERVER_JAVASCRIPT="${COUCHDB_QUERY_SERVER_JAVASCRIPT:-./bin/couchjs  ./share/server/main.js}"

To

export COUCHDB_QUERY_SERVER_JAVASCRIPT="${COUCHDB_QUERY_SERVER_JAVASCRIPT:-./bin/couchjs -S 536870912 ./share/server/main.js}"

(we add the option -S 536870912)

References
https://docs.couchdb.org/en/main/config/query-servers.html

[xqd-ai]

This error happened again after my data grow more, I had to increase CouchJS heap memory again to 1024MB