[Bug] Query Causes Segmentation Fault #806
Labels
type: Bug
Something isn't working
Comments
|
Tagging @foreyes per request |
antoniopetrole
changed the title
CloudBerry Database Interconnect Crash AnalysisOverviewA widespread interconnect crash occurred across a CloudBerry Database 1.6.0 cluster affecting multiple segments. This analysis examines the crash patterns across the affected segments and hosts. Would it be possible to share a couple of representative core files from this crash? The analysis shows a consistent pattern:
Having access to two core files would help with:
Core Analysis Script Output and SourceAnalysis OutputAnalysis Script#!/usr/bin/env python3
import os
import json
import datetime
from collections import defaultdict
from pathlib import Path
def load_core_analysis(json_file):
"""Load and parse a core analysis JSON file."""
with open(json_file) as f:
return json.load(f)
def extract_process_info(file_output):
"""Extract process info from core file output string."""
info = {}
try:
parts = file_output.split("from 'postgres: ")[1].split("'")[0].strip().split()
info['pid'] = parts[0].strip(',')
conn_details = ' '.join(parts[1:])
info['details'] = conn_details
for part in conn_details.split():
if 'seg' in part:
info['segment'] = part
elif 'slice' in part:
info['slice'] = part
elif 'cmd' in part:
info['cmd'] = part
except:
info['details'] = 'Parse failed'
return info
def analyze_stack_trace(stack_trace):
"""Analyze stack trace for key patterns."""
if not stack_trace:
return {}
top_functions = [frame.get('function', 'unknown') for frame in stack_trace[:5]]
return {
'top_functions': top_functions,
'crash_function': top_functions[0] if top_functions else 'unknown'
}
def analyze_cores(base_dir):
results = {
'cores_by_host': defaultdict(list),
'stack_patterns': defaultdict(int),
'segments': defaultdict(list),
'timing': [],
'signals': defaultdict(int)
}
for host_dir in Path(base_dir).iterdir():
if not host_dir.is_dir():
continue
host = host_dir.name
for json_file in host_dir.glob("*.json"):
analysis = load_core_analysis(json_file)
proc_info = extract_process_info(analysis['file_info']['file_output'])
timestamp = datetime.datetime.fromisoformat(analysis['timestamp'])
results['timing'].append({
'host': host,
'timestamp': timestamp,
'segment': proc_info.get('segment', 'unknown'),
'file': json_file.name
})
stack_info = analyze_stack_trace(analysis.get('stack_trace', []))
stack_key = tuple(stack_info.get('top_functions', ['unknown']))
results['stack_patterns'][stack_key] += 1
signal_info = analysis.get('signal_info', {})
signal_name = signal_info.get('signal_name', 'Unknown')
results['signals'][signal_name] += 1
results['cores_by_host'][host].append({
'file': json_file.name,
'timestamp': timestamp,
'process_info': proc_info,
'signal': signal_name,
'crash_function': stack_info.get('crash_function', 'unknown')
})
if 'segment' in proc_info:
results['segments'][proc_info['segment']].append({
'host': host,
'timestamp': timestamp,
'file': json_file.name
})
return results
def print_analysis(results):
print("\nCore File Analysis Summary")
print("=" * 50)
print("\nHosts Affected:", len(results['cores_by_host']))
for host, cores in sorted(results['cores_by_host'].items()):
print(f"\n{host}: {len(cores)} core files")
for core in sorted(cores, key=lambda x: x['timestamp']):
print(f" {core['timestamp'].strftime('%H:%M:%S')} - {core['process_info'].get('segment', 'unknown')} "
f"({core['signal']})")
print("\nStack Trace Patterns")
print("-" * 30)
for stack, count in sorted(results['stack_patterns'].items(), key=lambda x: x[1], reverse=True):
print(f"\nPattern occurred {count} times:")
for i, func in enumerate(stack, 1):
print(f" {i}. {func}")
print("\nSignal Distribution")
print("-" * 30)
for signal, count in sorted(results['signals'].items(), key=lambda x: x[1], reverse=True):
print(f"{signal}: {count}")
timestamps = [t['timestamp'] for t in results['timing']]
if timestamps:
print("\nTiming Analysis")
print("-" * 30)
min_time = min(timestamps)
max_time = max(timestamps)
duration = max_time - min_time
print(f"First core: {min_time.strftime('%H:%M:%S.%f')}")
print(f"Last core: {max_time.strftime('%H:%M:%S.%f')}")
print(f"Duration: {duration.total_seconds():.3f} seconds")
print("\nSegment Distribution")
print("-" * 30)
for segment, occurrences in sorted(results['segments'].items()):
print(f"{segment}: {len(occurrences)} cores")
if __name__ == "__main__":
print("Starting core file analysis...")
results = analyze_cores('.')
print_analysis(results) |
|
can you provide core and postgres binary ? @antoniopetrole And table ddl of |
|
@yjhjstz Wow I can't believe I forgot to upload the ddl, I've attached it to this comment. Also I shared the coredump with @edespino directly, I don't want to share it publicly since it can contain sensitive data. |
|
I have looked through corefiles analysis. It seems that the segmentation fault if caused by another thread with stack: {
"thread_id": "1",
"lwp_id": "unknown",
"backtrace": [
{
"frame_num": "0",
"location": "0x00007ff0b041cbaf",
"function": "raise",
"args": ""
},
{
"frame_num": "1",
"location": "0x00007ff0b11187bb",
"function": "StandardHandlerForSigillSigsegvSigbus_OnMainThread",
"args": ""
},
{
"frame_num": "3",
"location": "0x00007ff0b116da24",
"function": "comparetup_heap",
"args": ""
},
{
"frame_num": "4",
"location": "0x00007ff0b1169dc2",
"function": "qsort_tuple",
"args": ""
},
{
"frame_num": "5",
"location": "0x00007ff0b116d1f4",
"function": "dumptuples",
"args": ""
},
{
"frame_num": "6",
"location": "0x00007ff0b1170bf6",
"function": "tuplesort_performsort",
"args": ""
},
{
"frame_num": "7",
"location": "0x00007ff0b0ddfbf7",
"function": "ExecSort",
"args": ""
},
{
"frame_num": "8",
"location": "0x00007ff0b0dac77b",
"function": "ExecProcNodeFirst",
"args": ""
},
{
"frame_num": "9",
"location": "0x00007ff0b0de5098",
"function": "begin_partition",
"args": ""
},
{
"frame_num": "10",
"location": "0x00007ff0b0de9278",
"function": "ExecWindowAgg",
"args": ""
},
{
"frame_num": "11",
"location": "0x00007ff0b0dac77b",
"function": "ExecProcNodeFirst",
"args": ""
},
{
"frame_num": "12",
"location": "0x00007ff0b0df1b1c",
"function": "ExecMotion",
"args": ""
},
{
"frame_num": "13",
"location": "0x00007ff0b0dac77b",
"function": "ExecProcNodeFirst",
"args": ""
},
{
"frame_num": "14",
"location": "0x00007ff0b0da347a",
"function": "ExecutePlan.part.1",
"args": ""
},
{
"frame_num": "15",
"location": "0x00007ff0b0da3f89",
"function": "standard_ExecutorRun",
"args": ""
},
{
"frame_num": "16",
"location": "0x00007ff0b0da4135",
"function": "ExecutorRun",
"args": ""
},
{
"frame_num": "17",
"location": "0x00007ff0b0fbe95e",
"function": "PortalRunSelect",
"args": ""
},
{
"frame_num": "18",
"location": "0x00007ff0b0fc022c",
"function": "PortalRun",
"args": ""
},
{
"frame_num": "19",
"location": "0x00007ff0b0fba5df",
"function": "exec_mpp_query",
"args": ""
},
{
"frame_num": "20",
"location": "0x00007ff0b0fbdebd",
"function": "PostgresMain",
"args": ""
},
{
"frame_num": "21",
"location": "0x00007ff0b0f19ddf",
"function": "ServerLoop",
"args": ""
},
{
"frame_num": "22",
"location": "0x00007ff0b0f1af1f",
"function": "PostmasterMain",
"args": ""
},
{
"frame_num": "23",
"location": "0x00000000004017ae",
"function": "main",
"args": ""
}
]
}
],I will continue to analyze this issue anyway, and it will be helpful if you can provide any one of these corefile and your postgres binary. @antoniopetrole |
|
@foreyes & @my-ship-it - Here is what I have been able to retrieve from the core files. Given the schema, I was not able to reproduce the issue on a dev environment (single-node). |
|
@antoniopetrole - I can provide a debug version (with symbols) of the pre-Apache Cloudberry 1.6.0 release in rpm format. If you have a dev environment that can reproduce this issue, this should be a drop-in-place binary swap. |
|
Okay, I am analyzing the location of the error and trying to reproduce the problem with these informations. |
|
So I've gotten the green light to go ahead and try to recreate this issue in our dev environment. If I manage to do this then the next step will be swapping out the binaries in this environment with a build that has debug flags enabled so we have more information for troubleshooting. I'll keep everyone in the loop as I go about recreating this |
|
Excellent ... I'll be ready with an pre-Apache Cloudberry v1.6.0 debug rpm (el8). |
|
@antoniopetrole - Here is a pre-Apache Cloudberry v1.6.0 debug rpm (el8): RPM: Let us know if you reproduce the issue in your dev environment. |
@foreyes Here is some additional information from the cores. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Apache Cloudberry version
Cloudberry 1.6.0 (this is a pre-Apache release)
What happened
We had a number of segments go down both on 12/16 (multiple waves of segments going down) as well as 12/18 due to a problem query that was run. The log messages show this query along with a segmentation fault error in each segment log that crashed. I've obfuscated the table and column names for the sake of privacy in the below error message as well as the provided DDL and problem query, but everything else is exactly the same as it was when it was executed
As far as I can tell it doesn't seem like this query was ever successful, however it's hard to tell from the logs since they're pretty verbose. However I did confirm with an engineer that this query was first executed on 12/16 (the day of our segments going down the first time with this query), there's no entry of it being executed at all on 12/17, and then it was executed again on 12/18 causing the same error and crash. This is a very large table with a lot of partitions and just from taking a quick sampling each partition is around 100GB+ in size, however I can get exact sizes if that's helpful for analysis.
I've attached the core dump analysis files for all 31 segments that went down during this time (this is directly related to issue #803) provided by @edespino in his comment here #803 (comment). I've also provided the DDL to create the table that I captured using gpbackup and then obfuscated as well as the query itself in files that are attached. Happy to provide any more detail upon request. I've tested this in my local Cloudberry docker environment but haven't been able to recreate the segfault. Mind you though that this test didn't include any data in the table so I imagine that would influence what steps the query execution plan actually takes.
segfault_artifacts.zip
What you think should happen instead
This query shouldn't cause a segmentation fault and should just return the results when it's finished
How to reproduce
I've tested this locally and haven't been able to reproduce the segfault
Operating System
Rocky Linux 8.10 (Green Obsidian)
Anything else
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: