Failure to symbolize binary

[danipozo]

Parca can't symbolize addresses for ClickHouse binaries.

How to reproduce

First, download and decompress ClickHouse binary:

curl -LO https://s3.amazonaws.com/clickhouse-builds/24.10/2e054d555719f509b96e0cfea4d5f5819c69a61f/package_release/clickhouse
chmod +x clickhouse
./clickhouse --version

Normal flow reproducer

Then, execute ClickHouse server:

mkdir ch-data && cd ch-data
../clickhouse server

Try to profile it with Parca (you will need to compile parca-agent with the latest version of the upstream OTel eBPF profiler to get the fix for this). Also use --debuginfo-upload-max-size=2244458216 to be able to upload the binary debuginfo. You will observe that stacks are missing symbols in the UI.

Smaller reproducer

Compile and run this as a binary in this repo (pardon my poor Go):

package main

import (
	"context"
	"debug/elf"
	"os"

	"github.com/go-kit/log"
	"github.com/go-kit/log/level"
	"github.com/parca-dev/parca/pkg/symbol/addr2line"
	"github.com/parca-dev/parca/pkg/symbol/demangle"
)

func main() {
	ctx := context.Background()

  path := "clickhouse"
  f, err := elf.Open(path)
  if err != nil {
    print(err)
  }

  demangler := demangle.NewDemangler("simple", false)
  writer := log.NewSyncWriter(os.Stdout)
  logger := log.NewLogfmtLogger(writer)
  liner, err := addr2line.DWARF(logger, path, f, demangler)
  if err != nil {
    print(err)
  }

  lines, err := liner.PCToLines(ctx, 0x1192c860)
  if err != nil {
    level.Error(logger).Log("err", err)
  }
}

Error logs

These come ultimately from debug/dwarf/entry.go, method dwarf5Ranges.

Parca:

level=debug name=parca ts=2024-11-15T12:39:31.37757694Z caller=symbolizer.go:172 component=symbolizer msg="failed to get lines" err="liner pctolines: failed to extract dwarf tree: invalid rnglist offset 409180350 (max 636512)"

Reproducer bin:

level=error err="failed to extract dwarf tree: invalid rnglist offset 2858131463 (max 636512)"

Cross-checking with other tools

Using gimli's addr2line (0x1192c860 is one of the addresses it fails to symbolize):

$ ./addr2line-gimli -e clickhouse -fC 0x1192c860
DB::AggregateUtils::isAggregateFunction(DB::ASTFunction const&)
./build_docker/./src/AggregateFunctions/AggregateFunctionFactory.h:119

Also to note, I've checked some of the generated stacks with this addr2line implementation and they seem to make sense, so stack walking seems to be producing valid addresses.

Other comments

Any tips to debug or cross-check this with dwarfdump or other tools are welcome. For example, how could I check if the offsets that the entry offsets the code is parsing at each step are valid?