Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mnesia crashes and recovers on secondary nodes in clustered configuration #2

Open
beardedeagle opened this issue Jul 30, 2018 · 1 comment
Open

Mnesia crashes and recovers on secondary nodes in clustered configuration #2

beardedeagle opened this issue Jul 30, 2018 · 1 comment
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@beardedeagle
Copy link
Owner

beardedeagle commented Jul 30, 2018 •
edited
Loading

When mnesiac is configured to run in a clustered env, mnesia crashes on any node brought up after the first node. mnesia does recover and this lib is STILL usable in production (I use it in production), but this should be solved. It LOOKS like a issue with how the schema is being initialized. Trace output below.

config:

config :libcluster,
  topologies: [
    vanguard: [
      strategy: Cluster.Strategy.Epmd,
      config: [
        hosts: [
          :"test01@127.0.0.1",
          :"test02@127.0.0.1"
        ]
      ]
    ]
  ]

config :mnesia, dir: to_charlist(Path.join(File.cwd!(), System.get_env("DB_DIR") || "priv/data"))

config :mnesiac,
  stores: [
    Vanguard.Db.ProcStore,
    Vanguard.Db.OwnerStore,
    Vanguard.Db.HandlerStore,
    Vanguard.Db.StatusStore,
    Vanguard.Db.CheckStore
  ],
  schema_type: :disc_copies,
  table_load_timeout: 600_000

test01:

~> PORT=4001 DB_DIR=priv/test01 iex --erl '-mnesia debug trace verbose' --name test01@127.0.0.1 --cookie test -S mix phx.server
Erlang/OTP 21 [erts-10.0.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Mnesia('test01@127.0.0.1'): mnesia_monitor starting: <0.267.0>
Mnesia('test01@127.0.0.1'): Version: "4.15.4"
Mnesia('test01@127.0.0.1'): Env access_module: mnesia
Mnesia('test01@127.0.0.1'): Env allow_index_on_key: false
Mnesia('test01@127.0.0.1'): Env auto_repair: true
Mnesia('test01@127.0.0.1'): Env backup_module: mnesia_backup
Mnesia('test01@127.0.0.1'): Env debug: trace
Mnesia('test01@127.0.0.1'): Env dir: "/Users/rlthompson/GitHub/vanguard/priv/test01"
Mnesia('test01@127.0.0.1'): Env dump_disc_copies_at_startup: true
Mnesia('test01@127.0.0.1'): Env dump_log_load_regulation: false
Mnesia('test01@127.0.0.1'): Env dump_log_time_threshold: 180000
Mnesia('test01@127.0.0.1'): Env dump_log_update_in_place: true
Mnesia('test01@127.0.0.1'): Env dump_log_write_threshold: 1000
Mnesia('test01@127.0.0.1'): Env event_module: mnesia_event
Mnesia('test01@127.0.0.1'): Env extra_db_nodes: []
Mnesia('test01@127.0.0.1'): Env ignore_fallback_at_startup: false
Mnesia('test01@127.0.0.1'): Env fallback_error_function: {mnesia,lkill}
Mnesia('test01@127.0.0.1'): Env fold_chunk_size: 100
Mnesia('test01@127.0.0.1'): Env max_wait_for_decision: infinity
Mnesia('test01@127.0.0.1'): Env schema_location: opt_disc
Mnesia('test01@127.0.0.1'): Env core_dir: false
Mnesia('test01@127.0.0.1'): Env pid_sort_order: false
Mnesia('test01@127.0.0.1'): Env no_table_loaders: 2
Mnesia('test01@127.0.0.1'): Env dc_dump_limit: 4
Mnesia('test01@127.0.0.1'): Env send_compressed: 0
Mnesia('test01@127.0.0.1'): Env schema: []
Mnesia('test01@127.0.0.1'): Mnesia debug level set to trace
Mnesia('test01@127.0.0.1'): mnesia_subscr starting: <0.268.0>
Mnesia('test01@127.0.0.1'): mnesia_locker starting: <0.269.0>
Mnesia('test01@127.0.0.1'): mnesia_recover starting: <0.270.0>
Mnesia('test01@127.0.0.1'): mnesia_tm starting: <0.271.0>
Mnesia('test01@127.0.0.1'): Schema initiated from: default
Mnesia('test01@127.0.0.1'): mnesia_controller starting: <0.273.0>
Mnesia('test01@127.0.0.1'): mnesia_late_loader starting: <0.274.0>
Mnesia('test01@127.0.0.1'): Intend to load tables: []
[info] [swarm on test01@127.0.0.1] [tracker:init] started
[warn] [libcluster:vanguard] unable to connect to :"test02@127.0.0.1"
[info] [mnesiac:test01@127.0.0.1] mnesiac starting...
Mnesia('test01@127.0.0.1'): Mnesia already started, 0 seconds
Mnesia('test01@127.0.0.1'): Transaction log dump initiated by schema_prepare: already_dumped
Mnesia('test01@127.0.0.1'): Transaction log dump initiated by schema_update: {needs_dump,1}
Mnesia('test01@127.0.0.1'): write performed by {tid,3,<0.318.0>} on record:
        {schema,schema,
                [{name,schema},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,schema},
                 {attributes,[table,cstruct]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224666330000,-576460752303423483,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test01@127.0.0.1',{1532,929224,771269}}}}]}
Mnesia('test01@127.0.0.1'): write performed by {tid,4,<0.334.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['test01@127.0.0.1']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test01@127.0.0.1'): write performed by {tid,4,<0.334.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['test01@127.0.0.1']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Getting table 'Elixir.ProcStore' (ram_copies) from disc: {dumper,
                                                                                      create_table}
Mnesia('test01@127.0.0.1'): Creating index for 'Elixir.ProcStore' 3 ordered
Mnesia('test01@127.0.0.1'): write performed by {tid,5,<0.338.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test01@127.0.0.1'): write performed by {tid,5,<0.338.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Getting table 'Elixir.OwnerStore' (disc_copies) from disc: {dumper,
                                                                                        create_table}
Mnesia('test01@127.0.0.1'): write performed by {tid,6,<0.344.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test01@127.0.0.1'): write performed by {tid,6,<0.344.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Getting table 'Elixir.HandlerStore' (disc_copies) from disc: {dumper,
                                                                                          create_table}
Mnesia('test01@127.0.0.1'): Creating index for 'Elixir.HandlerStore' 3 ordered
Mnesia('test01@127.0.0.1'): Creating index for 'Elixir.HandlerStore' 7 ordered
Mnesia('test01@127.0.0.1'): write performed by {tid,7,<0.350.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test01@127.0.0.1'): write performed by {tid,7,<0.350.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Getting table 'Elixir.StatusStore' (disc_copies) from disc: {dumper,
                                                                                         create_table}
Mnesia('test01@127.0.0.1'): Creating index for 'Elixir.StatusStore' 3 ordered
Mnesia('test01@127.0.0.1'): Creating index for 'Elixir.StatusStore' 4 ordered
Mnesia('test01@127.0.0.1'): write performed by {tid,8,<0.356.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test01@127.0.0.1'): write performed by {tid,8,<0.356.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test01@127.0.0.1'): Getting table 'Elixir.CheckStore' (disc_copies) from disc: {dumper,
                                                                                        create_table}
Mnesia('test01@127.0.0.1'): Creating index for 'Elixir.CheckStore' 3 ordered
Mnesia('test01@127.0.0.1'): Creating index for 'Elixir.CheckStore' 4 ordered
Mnesia('test01@127.0.0.1'): Creating index for 'Elixir.CheckStore' 10 ordered
[info] [mnesiac:test01@127.0.0.1] mnesiac started
Interactive Elixir (1.7.1) - press Ctrl+C to exit (type h() ENTER for help)

test02:

 ~> PORT=4000 DB_DIR=priv/test02 iex --erl '-mnesia debug trace verbose' --name test02@127.0.0.1 --cookie test -S mix phx.server
Erlang/OTP 21 [erts-10.0.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Mnesia('test02@127.0.0.1'): mnesia_monitor starting: <0.267.0>
Mnesia('test02@127.0.0.1'): Version: "4.15.4"
Mnesia('test02@127.0.0.1'): Env access_module: mnesia
Mnesia('test02@127.0.0.1'): Env allow_index_on_key: false
Mnesia('test02@127.0.0.1'): Env auto_repair: true
Mnesia('test02@127.0.0.1'): Env backup_module: mnesia_backup
Mnesia('test02@127.0.0.1'): Env debug: trace
Mnesia('test02@127.0.0.1'): Env dir: "/Users/rlthompson/GitHub/vanguard/priv/test02"
Mnesia('test02@127.0.0.1'): Env dump_disc_copies_at_startup: true
Mnesia('test02@127.0.0.1'): Env dump_log_load_regulation: false
Mnesia('test02@127.0.0.1'): Env dump_log_time_threshold: 180000
Mnesia('test02@127.0.0.1'): Env dump_log_update_in_place: true
Mnesia('test02@127.0.0.1'): Env dump_log_write_threshold: 1000
Mnesia('test02@127.0.0.1'): Env event_module: mnesia_event
Mnesia('test02@127.0.0.1'): Env extra_db_nodes: []
Mnesia('test02@127.0.0.1'): Env ignore_fallback_at_startup: false
Mnesia('test02@127.0.0.1'): Env fallback_error_function: {mnesia,lkill}
Mnesia('test02@127.0.0.1'): Env fold_chunk_size: 100
Mnesia('test02@127.0.0.1'): Env max_wait_for_decision: infinity
Mnesia('test02@127.0.0.1'): Env schema_location: opt_disc
Mnesia('test02@127.0.0.1'): Env core_dir: false
Mnesia('test02@127.0.0.1'): Env pid_sort_order: false
Mnesia('test02@127.0.0.1'): Env no_table_loaders: 2
Mnesia('test02@127.0.0.1'): Env dc_dump_limit: 4
Mnesia('test02@127.0.0.1'): Env send_compressed: 0
Mnesia('test02@127.0.0.1'): Env schema: []
Mnesia('test02@127.0.0.1'): Mnesia debug level set to trace
Mnesia('test02@127.0.0.1'): mnesia_subscr starting: <0.268.0>
Mnesia('test02@127.0.0.1'): mnesia_locker starting: <0.269.0>
Mnesia('test02@127.0.0.1'): mnesia_recover starting: <0.270.0>
Mnesia('test02@127.0.0.1'): mnesia_tm starting: <0.271.0>
Mnesia('test02@127.0.0.1'): Schema initiated from: default
Mnesia('test02@127.0.0.1'): mnesia_controller starting: <0.273.0>
Mnesia('test02@127.0.0.1'): mnesia_late_loader starting: <0.274.0>
Mnesia('test02@127.0.0.1'): Intend to load tables: []
[info] [swarm on test02@127.0.0.1] [tracker:init] started
[info] [swarm on test02@127.0.0.1] [tracker:ensure_swarm_started_on_remote_node] nodeup test01@127.0.0.1
[info] [swarm on test02@127.0.0.1] [tracker:cluster_wait] joining cluster..
[info] [swarm on test02@127.0.0.1] [tracker:cluster_wait] syncing with test01@127.0.0.1
[info] [swarm on test02@127.0.0.1] [tracker:awaiting_sync_ack] received sync acknowledgement from test01@127.0.0.1, syncing with remote registry
[info] [swarm on test02@127.0.0.1] [tracker:awaiting_sync_ack] local synchronization with test01@127.0.0.1 complete!
[info] [swarm on test02@127.0.0.1] [tracker:resolve_pending_sync_requests] pending sync requests cleared
[info] [libcluster:vanguard] connected to :"test01@127.0.0.1"
[info] [mnesiac:test02@127.0.0.1] mnesiac starting...
Mnesia('test02@127.0.0.1'): mnesia_controller terminated: shutdown
Mnesia('test02@127.0.0.1'): mnesia_tm terminated: shutdown
Mnesia('test02@127.0.0.1'): mnesia_recover terminated: shutdown
Mnesia('test02@127.0.0.1'): mnesia_locker terminated: shutdown
Mnesia('test02@127.0.0.1'): mnesia_subscr terminated: shutdown
Mnesia('test02@127.0.0.1'): mnesia_monitor terminated: shutdown
[info] Application mnesia exited: :stopped
Mnesia('test02@127.0.0.1'): mnesia_monitor starting: <0.329.0>
Mnesia('test02@127.0.0.1'): Version: "4.15.4"
Mnesia('test02@127.0.0.1'): Env access_module: mnesia
Mnesia('test02@127.0.0.1'): Env allow_index_on_key: false
Mnesia('test02@127.0.0.1'): Env auto_repair: true
Mnesia('test02@127.0.0.1'): Env backup_module: mnesia_backup
Mnesia('test02@127.0.0.1'): Env debug: trace
Mnesia('test02@127.0.0.1'): Env dir: "/Users/rlthompson/GitHub/vanguard/priv/test02"
Mnesia('test02@127.0.0.1'): Env dump_disc_copies_at_startup: true
Mnesia('test02@127.0.0.1'): Env dump_log_load_regulation: false
Mnesia('test02@127.0.0.1'): Env dump_log_time_threshold: 180000
Mnesia('test02@127.0.0.1'): Env dump_log_update_in_place: true
Mnesia('test02@127.0.0.1'): Env dump_log_write_threshold: 1000
Mnesia('test02@127.0.0.1'): Env event_module: mnesia_event
Mnesia('test02@127.0.0.1'): Env extra_db_nodes: []
Mnesia('test02@127.0.0.1'): Env ignore_fallback_at_startup: false
Mnesia('test02@127.0.0.1'): Env fallback_error_function: {mnesia,lkill}
Mnesia('test02@127.0.0.1'): Env fold_chunk_size: 100
Mnesia('test02@127.0.0.1'): Env max_wait_for_decision: infinity
Mnesia('test02@127.0.0.1'): Env schema_location: opt_disc
Mnesia('test02@127.0.0.1'): Env core_dir: false
Mnesia('test02@127.0.0.1'): Env pid_sort_order: false
Mnesia('test02@127.0.0.1'): Env no_table_loaders: 2
Mnesia('test02@127.0.0.1'): Env dc_dump_limit: 4
Mnesia('test02@127.0.0.1'): Env send_compressed: 0
Mnesia('test02@127.0.0.1'): Env schema: []
Mnesia('test02@127.0.0.1'): Mnesia debug level set to trace
Mnesia('test02@127.0.0.1'): mnesia_subscr starting: <0.330.0>
Mnesia('test02@127.0.0.1'): mnesia_locker starting: <0.331.0>
Mnesia('test02@127.0.0.1'): mnesia_recover starting: <0.332.0>
Mnesia('test02@127.0.0.1'): mnesia_tm starting: <0.333.0>
Mnesia('test02@127.0.0.1'): Schema initiated from: default
Mnesia('test02@127.0.0.1'): mnesia_controller starting: <0.335.0>
Mnesia('test02@127.0.0.1'): mnesia_late_loader starting: <0.336.0>
Mnesia('test02@127.0.0.1'): Intend to load tables: []
Mnesia('test02@127.0.0.1'): Mnesia started, 0 seconds
Mnesia('test02@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test02@127.0.0.1'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['test01@127.0.0.1']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          'test01@127.0.0.1'}},
                 {version,{{2,0},[]}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,3,<0.340.0>} on record:
        {schema,schema,
                [{name,schema},
                 {type,set},
                 {ram_copies,['test02@127.0.0.1']},
                 {disc_copies,['test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,schema},
                 {attributes,[table,cstruct]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224666330000,-576460752303423483,1},
                          'test01@127.0.0.1'}},
                 {version,{{4,0},{'test02@127.0.0.1',{1532,929235,924960}}}}]}
Mnesia('test02@127.0.0.1'): Table 'Elixir.ProcStore' is loaded on 'test01@127.0.0.1'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('test02@127.0.0.1'): Table 'Elixir.OwnerStore' is loaded on 'test01@127.0.0.1'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('test02@127.0.0.1'): Table 'Elixir.HandlerStore' is loaded on 'test01@127.0.0.1'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('test02@127.0.0.1'): Table 'Elixir.StatusStore' is loaded on 'test01@127.0.0.1'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('test02@127.0.0.1'): Table 'Elixir.CheckStore' is loaded on 'test01@127.0.0.1'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('test02@127.0.0.1'): Logging mnesia_up 'test01@127.0.0.1'
Mnesia('test02@127.0.0.1'): Logging mnesia_up 'test02@127.0.0.1'
Mnesia('test02@127.0.0.1'): Transaction log dump initiated by schema_prepare: already_dumped
Mnesia('test02@127.0.0.1'): Transaction log dump initiated by schema_update: {needs_dump,1}
Mnesia('test02@127.0.0.1'): write performed by {tid,5,<0.342.0>} on record:
        {schema,schema,
                [{name,schema},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,schema},
                 {attributes,[table,cstruct]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224666330000,-576460752303423483,1},
                          'test01@127.0.0.1'}},
                 {version,{{5,0},{'test02@127.0.0.1',{1532,929235,977055}}}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,6,<0.358.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,93952}}}}]}
Mnesia('test02@127.0.0.1'): Getting table 'Elixir.ProcStore' (ram_copies) from node 'test01@127.0.0.1': {dumper,
                                                                                                         {add_table_copy,
                                                                                                          {tid,
                                                                                                           6,
                                                                                                           <0.358.0>}}}
Mnesia('test02@127.0.0.1'): Creating index for 'Elixir.ProcStore' 3 ordered
Mnesia('test02@127.0.0.1'): Table 'Elixir.ProcStore' copied from 'test01@127.0.0.1' to 'test02@127.0.0.1'
Mnesia('test02@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test02@127.0.0.1'): write performed by {tid,6,<0.358.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,93952}}}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,7,<0.363.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,135578}}}}]}
Mnesia('test02@127.0.0.1'): Getting table 'Elixir.OwnerStore' (disc_copies) from node 'test01@127.0.0.1': {dumper,
                                                                                                           {add_table_copy,
                                                                                                            {tid,
                                                                                                             7,
                                                                                                             <0.363.0>}}}
Mnesia('test02@127.0.0.1'): Table 'Elixir.OwnerStore' copied from 'test01@127.0.0.1' to 'test02@127.0.0.1'
Mnesia('test02@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test02@127.0.0.1'): write performed by {tid,7,<0.363.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,135578}}}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,8,<0.370.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,188492}}}}]}
Mnesia('test02@127.0.0.1'): Getting table 'Elixir.HandlerStore' (disc_copies) from node 'test01@127.0.0.1': {dumper,
                                                                                                             {add_table_copy,
                                                                                                              {tid,
                                                                                                               8,
                                                                                                               <0.370.0>}}}
Mnesia('test02@127.0.0.1'): Creating index for 'Elixir.HandlerStore' 3 ordered
Mnesia('test02@127.0.0.1'): Creating index for 'Elixir.HandlerStore' 7 ordered
Mnesia('test02@127.0.0.1'): Table 'Elixir.HandlerStore' copied from 'test01@127.0.0.1' to 'test02@127.0.0.1'
Mnesia('test02@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test02@127.0.0.1'): write performed by {tid,8,<0.370.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,188492}}}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,9,<0.377.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,239534}}}}]}
Mnesia('test02@127.0.0.1'): Getting table 'Elixir.StatusStore' (disc_copies) from node 'test01@127.0.0.1': {dumper,
                                                                                                            {add_table_copy,
                                                                                                             {tid,
                                                                                                              9,
                                                                                                              <0.377.0>}}}
Mnesia('test02@127.0.0.1'): Creating index for 'Elixir.StatusStore' 3 ordered
Mnesia('test02@127.0.0.1'): Creating index for 'Elixir.StatusStore' 4 ordered
Mnesia('test02@127.0.0.1'): Table 'Elixir.StatusStore' copied from 'test01@127.0.0.1' to 'test02@127.0.0.1'
Mnesia('test02@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test02@127.0.0.1'): write performed by {tid,9,<0.377.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,239534}}}}]}
Mnesia('test02@127.0.0.1'): write performed by {tid,10,<0.384.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,292209}}}}]}
Mnesia('test02@127.0.0.1'): Getting table 'Elixir.CheckStore' (disc_copies) from node 'test01@127.0.0.1': {dumper,
                                                                                                           {add_table_copy,
                                                                                                            {tid,
                                                                                                             10,
                                                                                                             <0.384.0>}}}
Mnesia('test02@127.0.0.1'): Creating index for 'Elixir.CheckStore' 3 ordered
Mnesia('test02@127.0.0.1'): Creating index for 'Elixir.CheckStore' 4 ordered
Mnesia('test02@127.0.0.1'): Creating index for 'Elixir.CheckStore' 10 ordered
Mnesia('test02@127.0.0.1'): Table 'Elixir.CheckStore' copied from 'test01@127.0.0.1' to 'test02@127.0.0.1'
Mnesia('test02@127.0.0.1'): Transaction log dump skipped (optional): schema_prepare
Mnesia('test02@127.0.0.1'): write performed by {tid,10,<0.384.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['test02@127.0.0.1','test01@127.0.0.1']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          'test01@127.0.0.1'}},
                 {version,{{3,0},{'test02@127.0.0.1',{1532,929236,292209}}}}]}
[info] [mnesiac:test02@127.0.0.1] mnesiac started
Interactive Elixir (1.7.1) - press Ctrl+C to exit (type h() ENTER for help)
@beardedeagle beardedeagle added bug Something isn't working help wanted Extra attention is needed labels Jul 30, 2018
@beardedeagle beardedeagle changed the title Mnesia crashes on secondary nodes in clustered configuration Mnesia crashes and recovers on secondary nodes in clustered configuration Jul 31, 2018
@beardedeagle beardedeagle mentioned this issue Aug 8, 2019
Closed
5 tasks
@beardedeagle
Copy link
Owner Author

I believe I have a solution to this, but will need to test with the #53 release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@beardedeagle