Can't override heap configuration #159

csvance · 2018-12-08T17:09:50Z

I have tried everything at this point:

-v /docker-store/neo4j/conf:/conf
-v /docker-store/neo4j/conf:/var/lib/neo4j/conf
-v /docker-store/neo4j/conf/neo4j.conf:/conf/neo4j.conf
-v /docker-store/neo4j/conf/neo4j.conf:/var/lib/neo4j/conf/neo4j.conf
Edit /conf/neo4j.conf in vi, commit container and launch the new container

Nothing seems to stick, and this works for pretty much every other docker image I have ever used.

I saw another person had the issue, but was not able to successfully use their solution:
#95

System: Ubuntu 18.04 x86_64
Docker Version:

ii  docker-ce                                  5:18.09.0~3-0~ubuntu-bionic                 amd64        Docker: the open-source application container engine
ii  docker-ce-cli                              5:18.09.0~3-0~ubuntu-bionic                 amd64        Docker CLI: the open-source application container engine

Docker Launch Script:

#!/bin/bash
docker run --name neo4j --restart=unless-stopped -p 7474:7474 -p 7687:7687 \
        -v /docker-store/neo4j/conf:/conf \
        -v /docker-store/neo4j/data:/data \
        --user 1000 -d neo4j

The text was updated successfully, but these errors were encountered:

csvance · 2018-12-08T23:26:30Z

So, I was mistaken because inside the docker root /config is a real directory, and config is a symlink to /var/lib/neo4j/conf and somehow in docker land these can both coexist. Simply doing -v /docker-store/neo4j/conf:/conf is adequate.

csvance · 2018-12-09T16:38:29Z

So I just checked whether configuration was being applied and its not:

name,value
dbms.active_database,graph.db
dbms.allow_format_migration,false
dbms.allow_upgrade,false
dbms.checkpoint,periodic
dbms.checkpoint.interval.time,900000ms
dbms.checkpoint.interval.tx,100000
dbms.checkpoint.iops.limit,300
dbms.config.strict_validation,false
dbms.connector.bolt.enabled,true
dbms.connector.bolt.listen_address,0.0.0.0:7687
dbms.connector.http.enabled,true
dbms.connector.http.listen_address,0.0.0.0:7474
dbms.connector.https.enabled,true
dbms.connector.https.listen_address,0.0.0.0:7473
dbms.connectors.default_advertised_address,localhost
dbms.connectors.default_listen_address,0.0.0.0
dbms.db.timezone,UTC
dbms.directories.certificates,/var/lib/neo4j/certificates
dbms.directories.data,/var/lib/neo4j/data
dbms.directories.import,/var/lib/neo4j/import
dbms.directories.lib,/var/lib/neo4j/lib
dbms.directories.logs,/var/lib/neo4j/logs
dbms.directories.plugins,/plugins
dbms.directories.run,/var/lib/neo4j/run
dbms.directories.tx_log,/var/lib/neo4j/data/databases/graph.db
dbms.filewatcher.enabled,true
dbms.import.csv.buffer_size,2097152
dbms.import.csv.legacy_quote_escaping,true
dbms.index.default_schema_provider,native-btree-1.0
dbms.index.fulltext.default_analyzer,standard
dbms.index.fulltext.eventually_consistent,false
dbms.index.fulltext.eventually_consistent_index_update_queue_max_length,10000
dbms.index_sampling.background_enabled,true
dbms.index_sampling.buffer_size,67108864
dbms.index_sampling.sample_size_limit,8388608
dbms.index_sampling.update_percentage,5
dbms.index_searcher_cache_size,2147483647
dbms.jvm.additional,-Dunsupported.dbms.udc.source=tarball
dbms.lock.acquisition.timeout,0ms
dbms.logs.debug.level,INFO
dbms.logs.debug.path,/var/lib/neo4j/logs/debug.log
dbms.logs.debug.rotation.delay,300000ms
dbms.logs.debug.rotation.keep_number,7
dbms.logs.debug.rotation.size,20971520
dbms.logs.gc.enabled,false
dbms.logs.gc.options,-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:+PrintTenuringDistribution
dbms.logs.gc.rotation.keep_number,5
dbms.logs.gc.rotation.size,20971520
dbms.logs.http.enabled,false
dbms.logs.http.path,/var/lib/neo4j/logs/http.log
dbms.logs.http.rotation.keep_number,5
dbms.logs.http.rotation.size,20971520
dbms.logs.query.allocation_logging_enabled,false
dbms.logs.query.enabled,false
dbms.logs.query.page_logging_enabled,false
dbms.logs.query.parameter_logging_enabled,true
dbms.logs.query.path,/var/lib/neo4j/logs/query.log
dbms.logs.query.rotation.keep_number,7
dbms.logs.query.rotation.size,20971520
dbms.logs.query.runtime_logging_enabled,false
dbms.logs.query.threshold,0ms
dbms.logs.query.time_logging_enabled,false
dbms.logs.timezone,UTC
dbms.logs.user.path,/var/lib/neo4j/logs/neo4j.log
dbms.logs.user.rotation.delay,300000ms
dbms.logs.user.rotation.keep_number,7
dbms.logs.user.rotation.size,0
dbms.logs.user.stdout_enabled,true
dbms.memory.heap.initial_size,512M
dbms.memory.heap.max_size,512M
dbms.memory.pagecache.size,512M
dbms.memory.pagecache.swapper,""
dbms.netty.ssl.provider,JDK
dbms.procedures.kill_query_verbose,true
dbms.query_cache_size,1000
dbms.read_only,false
dbms.record_format,""
dbms.relationship_grouping_threshold,50
dbms.rest.transaction.idle_timeout,60000ms
dbms.security.allow_csv_import_from_file_urls,true
dbms.security.auth_enabled,true
dbms.security.auth_lock_time,5000ms
dbms.security.auth_max_failed_attempts,3
dbms.security.http_access_control_allow_origin,*
dbms.security.http_authorization_classes,[]
dbms.security.http_strict_transport_security,""
dbms.security.procedures.unrestricted,algo.*
dbms.security.procedures.whitelist,*
dbms.shutdown_transaction_end_timeout,10000ms
dbms.ssl.policy.<policyname>.allow_key_generation,false
dbms.ssl.policy.<policyname>.base_directory,""
dbms.ssl.policy.<policyname>.ciphers,""
dbms.ssl.policy.<policyname>.client_auth,REQUIRE
dbms.ssl.policy.<policyname>.private_key,private.key
dbms.ssl.policy.<policyname>.public_certificate,public.crt
dbms.ssl.policy.<policyname>.revoked_dir,revoked
dbms.ssl.policy.<policyname>.tls_versions,[TLSv1.2]
dbms.ssl.policy.<policyname>.trust_all,false
dbms.ssl.policy.<policyname>.trusted_dir,trusted
dbms.ssl.policy.<policyname>.verify_hostname,false
dbms.threads.worker_count,8
dbms.track_query_allocation,false
dbms.track_query_cpu_time,false
dbms.transaction.bookmark_ready_timeout,30000ms
dbms.transaction.monitor.check.interval,2000ms
dbms.transaction.timeout,0ms
dbms.tx_log.rotation.retention_policy,100M size
dbms.tx_log.rotation.size,262144000
dbms.tx_state.max_off_heap_memory,2147483648
dbms.tx_state.memory_allocation,ON_HEAP
dbms.tx_state.off_heap.block_cache_size,128
dbms.tx_state.off_heap.max_cacheable_block_size,524288
dbms.udc.enabled,true
dbms.unmanaged_extension_classes,[]
dbms.windows_service_name,neo4j

csvance · 2018-12-09T17:00:11Z

So quick update, the memory settings in neo4j.conf are not used. You need to pass this to docker in order to increase the maximum heap size: --env=NEO4J_dbms_memory_heap_max__size=12G

Maybe there is a better way to handle this?

chrstnb · 2019-01-14T07:57:05Z

Didn't find this when I created #12130; apologies! What's happening is when you download the docker release, configurations that are normally commented out get added to neo4j.conf (code here), which diverges from the documentation afaik. The setup_heap function, which checks for an environment variable setting HEAP_SIZE is still called in the scripts, however there are "hidden" configurations in the conf file, those configurations get added second and take precedence when passed into Java because they appear later in the arguments. See the issue for more details!

slothkong · 2019-03-30T08:00:51Z

lol I wasted a good 6 hours becuase of this. Guess Ill just try rollingback version until my configs are applied...
EDIT: By the way this seems to be an issue not only for the inial and max heap size but also for the cache page size.

jennyowen · 2019-07-09T08:03:09Z

@csvance @chrstnb @slothkong Sorry for taking so long to tackle this issue.
It should have been fixed by: #180
This fix is in the 3.5.7 release. Do you still have the problem if you use a neo4j 3.5.7 image?

Kiollpt · 2019-07-22T11:04:27Z

@jennyowen
It still does not work even upgrade the latest image.
I mount conf file to my local and override it .
but after I restart my docker, it was gone.

jennyowen · 2019-07-22T11:43:06Z

@Kiollpt
We fixed it so that dbms.memory.heap.initial_size and dbms.memory.heap.max_size settings from the conf file don't get overridden any more, but there are still other settings that get overridden. We're still working on fixing those too.

The settings that are still being overridden are:

dbms.tx_log.rotation.retention_policy
dbms.memory.pagecache.size
wrapper.java.additional
dbms.memory.heap.initial_size
dbms.memory.heap.max_size
dbms.connectors.default_listen_address
dbms.connector.http.listen_address
dbms.connector.https.listen_address
dbms.connector.bolt.listen_address
ha.host.coordination
ha.host.data
causal_clustering.discovery_listen_address
causal_clustering.discovery_advertised_address
causal_clustering.transaction_listen_address
causal_clustering.transaction_advertised_address
causal_clustering.raft_listen_address
causal_clustering.raft_advertised_address

I have some questions:

Can you tell me what conf settings you're overriding in your code?
How are you mounting the conf file? Specifically what's the docker run command you're using to start Neo4j?
When you say

but after I restart my docker, it was gone.

how are you restarting docker?
When you say gone, do you mean that the file goes missing from your host folder? Or are the settings you have in the conf file not read in correctly?

Kiollpt · 2019-07-22T12:25:25Z

I override dbms.memory.heap.max_size=1024M
and used docker-compose up to start again. The settings went back to 512M

The following is my docker-compose.yml

version: '3'
services:
    jupyter:
        image: "jupyter/pyspark-notebook"
        volumes:
         - .:/work
        ports:
         - 8888:8888
    neo4j:
        image: "neo4j:3.5.7"
        user: "151:20"
        volumes: 
        - $HOME/neo4j/data:/data
        - $HOME/neo4j/conf:/var/lib/neo4j/conf
        - $HOME/neo4j/log:/var/lib/neo4j/logs
        ports:
         - 7474:7474
         - 7687:7687

jennyowen · 2019-07-22T12:37:18Z

@Kiollpt ok that's interesting. What happens if you change the conf and log mounting to this?

        - $HOME/neo4j/conf:/conf
        - $HOME/neo4j/log:/logs

Kiollpt · 2019-07-23T06:07:24Z

I could override neo4j.conf at /conf in the container.
And I replace it at /var/lib/neo4j/neo4j.conf, then restart it. but it still not read correctly

jennyowen · 2019-07-23T09:19:15Z

@Kiollpt
The documentation about mounting volumes is that you should mount the logs output to /logs and the configuration file folder to /conf.

See:
https://neo4j.com/docs/operations-manual/current/docker/introduction/#docker-volumes

And this section about the /conf volume:
https://neo4j.com/docs/operations-manual/current/docker/configuration/#docker-conf-volume

If I understood what you wrote correctly, your neo4j.conf is read properly if you mount it to /conf? This is the best (and tested) way to override conf settings in a Neo4j docker container. There should be no need to copy it over to /var/lib/neo4j/conf/neo4j.conf and restart. It should be ready to go on start up with the settings you want.

Kiollpt · 2019-07-24T10:07:12Z

I write dbms.memory.heap.max_size=1024M to $HOME/neo4j/conf/neo4j.conf in local

and then run docker-compose-up.

I go into the container and /conf/neo4j.conf with dbms.memory.heap.max_size=1024M

But /var/lib/conf/neo4j.conf still remain dbms.memory.heap.max_size=512

In addition to , I check the browser localhost:7474 and run

CALL dbms.listConfig()
YIELD name, value
WHERE name CONTAINS "heap"
RETURN name, value
ORDER BY name
LIMIT 4;

it shows dbms.memory.heap.max_size=512M

Does It really override my setting?

jennyowen · 2019-07-25T14:05:59Z

@Kiollpt
I made the docker-compose.yml:

version: '3'
services:
    neo4j:
        container_name: "neo"
        image: "neo4j:3.5.7"
        user: "1000:1000"
        volumes: 
        - $HOME/temp/data:/data
        - $HOME/temp/conf:/conf
        - $HOME/temp/log:/logs
        ports:
         - 7474:7474
         - 7687:7687

Then ran: docker exec neo grep heap /var/lib/neo4j/conf/neo4j.conf, which returned:

# Java Heap Size: by default the Java heap size is dynamically
# heap size.
dbms.memory.heap.initial_size=512m
dbms.memory.heap.max_size=1024M

I ran your cypher query in the browser but it still returned 1024M as the max heap size.

I got the same results if I mounted the host conf folder to /conf or /var/lib/neo4j/conf.

Could you provide some reproduction steps?

Kiollpt · 2019-07-26T03:54:08Z

@jennyowen It is weird. I did almost like you.

version: '3'
services:
    jupyter:
        image: "jupyter/pyspark-notebook"
        volumes:
         - .:/work
        ports:
         - 8888:8888
    neo4j:
        image: "neo4j:3.5.7"
        user: "neo4j"
        volumes:
        - $HOME/neo4j/data:/data
        - $HOME/neo4j/conf:/conf
        - $HOME/neo4j/logs:/logs
        ports:
         - 7474:7474
         - 7687:7687

I revised memory at local file $HOME/neo4j/conf/neo4j.conf to 1024M.
below is my conf

dbms.directories.import=import
dbms.connector.bolt.enabled=true
dbms.connector.http.enabled=true
dbms.connector.https.enabled=true
dbms.jvm.additional=-XX:+UseG1GC
dbms.jvm.additional=-XX:-OmitStackTraceInFastThrow
dbms.jvm.additional=-XX:+AlwaysPreTouch
dbms.jvm.additional=-XX:+UnlockExperimentalVMOptions
dbms.jvm.additional=-XX:+TrustFinalNonStaticFields
dbms.jvm.additional=-XX:+DisableExplicitGC
dbms.jvm.additional=-Djdk.tls.ephemeralDHKeySize=2048
dbms.jvm.additional=-Djdk.tls.rejectClientInitiatedRenegotiation=true
dbms.windows_service_name=neo4j
dbms.jvm.additional=-Dunsupported.dbms.udc.source=tarball
dbms.security.procedures.unrestricted=algo.*
wrapper.java.additional=-Dneo4j.ext.udc.source=docker
ha.host.data=c31074b36139:6001
ha.host.coordination=c31074b36139:5001
dbms.tx_log.rotation.retention_policy=100M size
dbms.memory.pagecache.size=512M
dbms.memory.heap.max_size=1024M
dbms.memory.heap.initial_size=512M
dbms.directories.logs=/logs
dbms.connectors.default_listen_address=0.0.0.0
dbms.connector.https.listen_address=0.0.0.0:7473
dbms.connector.http.listen_address=0.0.0.0:7474
dbms.connector.bolt.listen_address=0.0.0.0:7687
causal_clustering.transaction_listen_address=0.0.0.0:6000
causal_clustering.transaction_advertised_address=c31074b36139:6000
causal_clustering.raft_listen_address=0.0.0.0:7000
causal_clustering.raft_advertised_address=c31074b36139:7000
causal_clustering.discovery_listen_address=0.0.0.0:5000
causal_clustering.discovery_advertised_address=c31074b36139:5000
HOME=/var/lib/neo4j
EDITION=community

I ran docker-compose up
and then ran docker exec neo4j_neo4j_1 grep heap /var/lib/neo4j/conf/neo4j.conf , which returned:

dbms.memory.heap.max_size=512M
dbms.memory.heap.initial_size=512M

These are my steps.

manojlds · 2019-11-14T18:57:40Z

Not able to do remote debugging as per this as well - https://neo4j.com/docs/java-reference/current/server-debugging/

The dbms.jvm.additional is overridden in the final conf.

jennyowen · 2019-11-28T11:11:14Z

@manojlds this should be fixed by #205
~~The fix will be in 3.5.13 which will be released when this pull request is accepted by docker: docker-library/official-images#7045~~
Argh sorry, my mistake it didn't make it into the 3.5.13 release after all. I don't want to bother the nice docker people with another pull request so soon but it will be in the one afterwards.

rjurney · 2023-09-30T19:56:10Z

I had similar issues and just want to take a moment to politely curse Neo4j for the double __ in this value: NEO4J_dbms_memory_heap_max__size :D

csvance closed this as completed Dec 8, 2018

csvance reopened this Dec 9, 2018

csvance changed the title ~~Can't override configuration~~ Can't override heap configuration Dec 10, 2018

chrisvest mentioned this issue Jan 11, 2019

Heap size environment variable is overwritten by default config setting in docker neo4j/neo4j#12130

Closed

jennyowen closed this as completed Jul 22, 2019

jennyowen reopened this Jul 22, 2019

jennyowen closed this as completed Apr 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't override heap configuration #159

Can't override heap configuration #159

csvance commented Dec 8, 2018

csvance commented Dec 8, 2018

csvance commented Dec 9, 2018

csvance commented Dec 9, 2018 •

edited

Loading

chrstnb commented Jan 14, 2019 •

edited

Loading

slothkong commented Mar 30, 2019

jennyowen commented Jul 9, 2019

Kiollpt commented Jul 22, 2019 •

edited

Loading

jennyowen commented Jul 22, 2019

Kiollpt commented Jul 22, 2019

jennyowen commented Jul 22, 2019

Kiollpt commented Jul 23, 2019

jennyowen commented Jul 23, 2019

Kiollpt commented Jul 24, 2019

jennyowen commented Jul 25, 2019

Kiollpt commented Jul 26, 2019

manojlds commented Nov 14, 2019

jennyowen commented Nov 28, 2019 •

edited

Loading

rjurney commented Sep 30, 2023

Can't override heap configuration #159

Can't override heap configuration #159

Comments

csvance commented Dec 8, 2018

csvance commented Dec 8, 2018

csvance commented Dec 9, 2018

csvance commented Dec 9, 2018 • edited Loading

chrstnb commented Jan 14, 2019 • edited Loading

slothkong commented Mar 30, 2019

jennyowen commented Jul 9, 2019

Kiollpt commented Jul 22, 2019 • edited Loading

jennyowen commented Jul 22, 2019

Kiollpt commented Jul 22, 2019

jennyowen commented Jul 22, 2019

Kiollpt commented Jul 23, 2019

jennyowen commented Jul 23, 2019

Kiollpt commented Jul 24, 2019

jennyowen commented Jul 25, 2019

Kiollpt commented Jul 26, 2019

manojlds commented Nov 14, 2019

jennyowen commented Nov 28, 2019 • edited Loading

rjurney commented Sep 30, 2023

csvance commented Dec 9, 2018 •

edited

Loading

chrstnb commented Jan 14, 2019 •

edited

Loading

Kiollpt commented Jul 22, 2019 •

edited

Loading

jennyowen commented Nov 28, 2019 •

edited

Loading