GLIBC version mismatch in Promtail binary

[iljalukin]

Describe the bug
When attempting to run Promtail, the following error occurs:
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /usr/bin/promtail)
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/promtail)

To Reproduce
Install and run promtail 3.1.0 from apt.grafana.com using instructions from https://apt.grafana.com

Expected behavior
Promtail should run without GLIBC version errors.

Environment:

• local install on Debian 11 (bullseye)

Screenshots, Promtail config, or terminal output
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /usr/bin/promtail)
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/promtail)

[mrlhansen]

I can confirm that I am seeing the same issue with the RPM repository when installing on AlmaLinux 8.10.

[jostrasser]

Can confirm the same on Ubuntu 20.04.6 LTS (Focal Fossa):

Jul 3 10:52:36 loki promtail[377]: /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/promtail)

A rollback/downgrade to promtail 3.0 is fixing it temporarily.

[Smith4545]

Please roll back any rpm+apt repositories distributing this binary. How can something like this just be distributed to a wide audience? Is there no testing for this stuff? This just breaks log aggregation for whole companies.

[jytx]

I have the same problem

[anhdowastaken]

Same with Rocky Linux 8.8

[Vitaliy155]

the same issue on AlmaLinux release 8.9

[jacobamey]

Same issue on Amazon Linux 2

[Aethylred]

same issue on Rocky8

# promtail --version
promtail: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by promtail)
promtail: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by promtail)

[adamw-linadm]

almalinux 8.10 also :(

[jostrasser]

@JStickler any updates when this will be fixed?

[mmagdy88]

Same issue here on Ubuntu 20

Edit: Working on Ubuntu 22 because GLIBC is version 2.35, but on Ubuntu 20, GLIBC is version 2.31

[bursan85]

Good evening,

Issue is present with promtail-3.1.1-1.x86_64, so i went for an older version this package for the moment.
In my case, for AlmaLinux release 8.10 (Cerulean Leopard) i have installed an older version:
dnf install promtail-2.9.9-1.x86_64

[Alex4Codes]

CentOS 8 stream problem version 3.1.1 - /lib64/libc.so.6: version `GLIBC_2.32' not found

[bpfoster]

3.0.1 is also affected

[shantanugadgil]

This affects Amazon Linux 2 as well (CentOS 7 derivative + updated kernel etc.)

Amazon Linux 2023 works ok.

EDIT: Downgrading to 2.9.9 works on older OSes.

[boite]

Tried downgrading to 3.0.1 and 2.9.10 and finally got it working with 2.9.9 on Ubuntu 20

$ grep -i pretty /etc/os-release
PRETTY_NAME="Ubuntu 20.04.6 LTS"
$ dpkg -s promtail | grep Version
Version: 2.9.9

[shantanugadgil]

I assume the fix would be to use a build container/os of a lower glibc version!

[adamw-linadm]

Or build trully static go binary.

[fauust]

Same here, install from binaries, downgrade from 2.9.10 to 2.9.9 as a workaround (Debian 11, x86_64).

[bkkoli]

Same Issue occurs to me.

My OS is Debian 11.7
My Kernel version is 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux

[bkkoli]

I check my GLIBC Version as belows.

ldd --version

My GLIBC Version is ldd (Debian GLIBC 2.31-13+deb11u6) 2.31

If you use a version of Promtail that is compatible with the GLIBC version of your OS, this may fix the problem.

[bkkoli]

@iljalukin
In my case, Promtail v2.9.9, It works well.

[iljalukin]

@iljalukin In my case, Promtail v2.9.9, It works well.

This does not solve my problem, as I want to install the latest available version via the APT repository as described above.
My workaround is to install version 3.0.0 with apt install promtail=3.0.0 and pin it with apt-mark hold promtail.
Debian 11 is the current LTS version, which will be maintained until 2026 and it would be nice if promtail could be installed there in the current version.

I hope that the developers will pay attention to this issue and fix it.

[mrlhansen]

It's honestly a little concerning that this has not been fixed after two months. This breaks log aggregation on most enterprise Linux distributions used in production. I have decided to disabled the Grafana RPM repository completely and moved working versions of the RPMs that I need into a private repository to avoid these things affecting production. But if this is not fixed in a reasonable time, then Grafana Loki is not the right solution.

[BlackMoon2020]

Indeed concerning, hopefully due to holiday, meanwhile running old version. If not fix in long term, will need to look for alternative

[bkkoli]

I think there is no different way except updating OS which support target GLIBC library.

In production, Because It is hard to update OS version, It is better to make binary file less dependent on libraries like GLIBC closely related to the operating system.

[shantanugadgil]

#13393 (comment)

#13393 (comment)

I am not a GitHub workflow expert, so I am not able to submit the relevant PR to downgrade the "build environment", but I believe that the workflow is building on the latest Ubuntu (based on a rudimentary search inside the .github/ directory) which would have a glibc >= 2.32

My use case is to support Amazon Linux 2, which uses glibc 2.26

$ cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
Amazon Linux release 2 (Karoo)

$ ldd --version
ldd (GNU libc) 2.26
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

$ rpm -qa | grep glibc-2
glibc-2.26-64.amzn2.0.2.x86_64

[shantanugadgil]

@cyriltovena could you help with this?

[shantanugadgil]

Another workaround is to build the binary on the OS you want to run it on (lowest GLIBC versioned OS).

Needs an installed version of golang and gcc

git clone from main or the tag v3.2.0

# make promtail
# ./clients/cmd/promtail/promtail --version
# promtail, version 3.2.0 (branch: HEAD, revision: 659f5421d)
  build user:       user@machine
  build date:       2024-09-27T11:31:41Z
  go version:       go1.23.1
  platform:         linux/arm64
  tags:             unknown

[SckyzO]

Hello, same bug here, So i just build promtail from source.

In WSL, just install goland

sudo apt install golang gcc

Download Loki 3.2.0 from github, and build promtail with CGO_ENABLED=0

cd /tmp
git clone --depth 1 --branch v3.2.0 https://github.com/grafana/loki
cd loki
CGO_ENABLED=0 go build ./clients/cmd/promtail
ls -al promtail

[TomiCheng]

redhat 8.10 also -_-

[x4e-jonas]

git clone from main or the tag v3.2.0

# make promtail
# ./clients/cmd/promtail/promtail --version

Download Loki 3.2.0 from github, and build promtail with CGO_ENABLED=0

cd /tmp
git clone --depth 1 --branch v3.2.0 https://github.com/grafana/loki
cd loki
CGO_ENABLED=0 go build ./clients/cmd/promtail
ls -al promtail

You need to set PROMTAIL_JOURNAL_ENABLED="true" or build with -tags promtail_journal_enabled otherwise promtail is not linked to libsystemd/ glibc at all.

[Jayclifford345]

Hi all, we are extremely sorry this issue has gone under the radar for so long. We should have identified this issue faster. The Loki team is now aware of the issue and is working on a fix within the build process.

[digdilem]

Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.

[cstyan]

Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.

Promtail is in essence deprecated in the sense that all of our long term efforts are behind Grafana Alloy. For Promtail we're continuing to accept bug fixes, security patches, and minor updates/fixes such as this one. However, Promtail will not be getting new features.

[slim-bean]

Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.

As @cstyan mentioned, Alloy has superseded Promtail in every way at this point and we would consider it the superior product.

We forked Promtail code into Alloy some time ago making maintenance a challenge for us.

We've not been in any particular hurry to announce long term plans around Promtail because the overall maintenance burden is not that high. That being said, we've decided not to add new features to it and rather only focus new features on Alloy and announcing deprecation of Promtail is a logical next step.

All that being said, I would love to understand your sentiment better though @digdilem, I think our opinion is that moving to Alloy is an improvement but if that's not everyone's perception we would love to learn more to hopefully make sure our decisions are not alienating users.

[digdilem]

All that being said, I would love to understand your sentiment better though @digdilem, I think our opinion is that moving to Alloy is an improvement but if that's not everyone's perception we would love to learn more to hopefully make sure our decisions are not alienating users.

@slim-bean - thanks for the opportunity. My sentiment is fairly simple and it's not that strong. I'm a sysadmin who helps maintain a couple of hundred linux vms running literally hundreds of different pieces of software, of which promtail is one. My devs asked for it to be installed on around 40 machines, I did so, and I maintain it. I'm not particularly angry or emotional, but I can identify my concerns and, I hope, express them clearly.

Grafana made it easy in providing repos for our target distro (EL - specifically Rocky 8 and 9), and quite well behaved software. It didn't take much effort to please the devs, and happy devs is one of our KPIs. All good.

Then we noticed this error - no problem, software is complicated and breaks. A dependency issue is relatively common and understandable. We found this thread and our generally positive view of Grafana was slightly dented. Why was it being ignored? Why was Grafana not fixing a pretty major issue reported in the official way, that looks easy to do? It's clear from comments that I'm not alone in being puzzled and when there's nobody listening, your mind starts to wander.

Ignored and uninformed, I (speaking personally) started wondering if there was an agenda to let promtail die an ignoble death to force people to adopt alloy. We've all seen this type of cynical behaviour before and it's damaging to the users, to the software company's reputation, and to foss generally. I wasn't convinced Grafana would do this - I like the company and it does good stuff - and I'm glad that niggling voice in my head wasn't correct. This was an oversight, nothing more.

In my experience, people - either as users of software or in life generally - want tomorrow to be much like today. Waking up one morning and finding out you've now got to spend some unplanned hours to migrate from one piece of software to another upsets the balance of the day and makes sysadmins grumpy. We like warning, we like clear communications, we like to plan ahead. The IT world is more full of dodgy practices than ever before (see recent price hikes and model changing (eg vmware/prtg - and they won't be the last)) and we're probably more quick to see shapes in the shadows where they don't actually exist. I'm guilty of that here, I think.

Sorry, written too much already - but the basis of my sentiment is that I have no issues with alloy replacing promtail. Provided it's as well behaved from a system's point of view, great. Change is the one constant in IT. We don't need alloy - promtail gets logs from our java services and sends them to loki. That's what we need it for and it does that well. Whatever new features alloy has, we almost certainly don't need or want them - but I'm not emotionally attached to promtail so if we have to change because you can't support both forever, so be it.

My sentiment was stirred because this issue wasn't being responded to - nobody likes to be ignored. I'm glad it was just an accidental oversight and not deliberate. No big drama here.

[Jayclifford345]

Hey all, thank you for your patience on this issue. We have a PR ready to change the build image causing the problem. We wanted to do some testing to make sure doing this wouldn't invalidate for new OS's. Keep us in the loop with your own testing once this release it cut so we can keep an eye on things.

#14368

Once again thank you for bringing this to out attention.

[chaudum]

Re-opening, because the issue was automatically closed by #14368 but new versions haven't been released yet.

[skazazes-work]

Is there a timeline for a working release? I am currently in the process of setting up promtail and do not want to manually install 2.9.9 if a new release will be out in the coming days.

Thanks!

[chaudum]

Is there a timeline for a working release? I am currently in the process of setting up promtail and do not want to manually install 2.9.9 if a new release will be out in the coming days.

Thanks!

There is no timeline for 3.2.x and 3.1.x patch release yet.

I understand that you would wanna wait for them, but tbh, there hasn't been many changes in Promtail between 2.9.x and 3.2.x

[smaddock]

It would be helpful if this documentation was updated to refer to Alloy; the Loki Getting Started docs are updated but not the Loki Install docs.

[Jayclifford345]

Great catch @smaddock! Will get that updated now

<-- Update -->
Docs Update:
#14404

[AlexeyDemidov]

2.9.11 released yesterday is still broken on Ubuntu 20.04 LTS.

promtail --version                                                                         
promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by promtail)
promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by promtail)

[Jayclifford345]

@AlexeyDemidov just looking into this and testing and will get back to you soon. Apologies for the ongoing issue

[Jayclifford345]

Hi @AlexeyDemidov can you check version: 3.3.1
I downloaded it via the release page and I cannot reproduce the issue. Let me know if you see different:

jay_clifford@jay-ubuntu-test:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
jay_clifford@jay-ubuntu-test:~$ sudo systemctl status promtail
● promtail.service - Promtail service
     Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-12-09 11:41:07 UTC; 19min ago
   Main PID: 2311 (promtail)
      Tasks: 7 (limit: 4680)
     Memory: 14.8M
     CGroup: /system.slice/promtail.service
             └─2311 /usr/bin/promtail -config.file /etc/promtail/config.yml

Dec 09 11:41:07 jay-ubuntu-test systemd[1]: Started Promtail service.
Dec 09 11:41:08 jay-ubuntu-test promtail[2311]: level=info ts=2024-12-09T11:41:08.01703617Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=33c9b6c3668b0fd158ce912ea159eca3
Dec 09 11:41:08 jay-ubuntu-test promtail[2311]: level=info ts=2024-12-09T11:41:08.018715371Z caller=server.go:351 msg="server listening on addresses" http=[::]:9080 grpc=[::]:35259
Dec 09 11:41:08 jay-ubuntu-test promtail[2311]: level=info ts=2024-12-09T11:41:08.018918022Z caller=main.go:173 msg="Starting Promtail" version="(version=3.3.1, branch=release-3.3.x, revision=60f2af32)"
Dec 09 11:41:08 jay-ubuntu-test promtail[2311]: level=warn ts=2024-12-09T11:41:08.019106095Z caller=promtail.go:263 msg="enable watchConfig"
Dec 09 11:41:13 jay-ubuntu-test promtail[2311]: level=info ts=2024-12-09T11:41:13.018745993Z caller=filetargetmanager.go:372 msg="Adding target" key="/var/log/messages:{job=\"varlogs\"}"
jay_clifford@jay-ubuntu-test:~$ ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.16) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
jay_clifford@jay-ubuntu-test:~$ promtail --version
promtail, version 3.3.1 (branch: release-3.3.x, revision: 60f2af32)
  build user:       root@f884b8078e69
  build date:       2024-12-04T17:10:01Z
  go version:       go1.23.1
  platform:         linux/amd64
  tags:             promtail_jour

[AlexeyDemidov]

Hi @Jayclifford345 , I'm still on the 2.* branch, the report is about the latest version on this branch - 2.9.11.

[Jayclifford345]

Hi @AlexeyDemidov, I was chatting with the Loki engineers there shouldn't be a reason not to update to the latest release of Promtail as this is compatible with Loki 2.9.11 as the write API has not changed. Let me know if you have a strong need to stay on 2.9.11 and why so I can see if we can get the fix backported.

[shantanugadgil]

This affects Amazon Linux 2 as well (CentOS 7 derivative + updated kernel etc.)

Amazon Linux 2023 works ok.

EDIT: Downgrading to 2.9.9 works on older OSes.

Just to confirm: v3.3.1 doesn't exhibit this error anymore on the (older) Amazon Linux 2 OS.
We have upgraded to using Promtail v3.3.1 and things seem to be working for now.

After the above comment, I checked v2.9.11, it does indeed have the the same problem as before.

[Jayclifford345]

Thanks for confirming @shantanugadgil! We advice everyone to move to the latest version of Promtail since this contains the image fix. If there is a strong need to stay on v2.9.11 make sure you feed it back to us and we can look into it :). For now I am going to close out this issue but feel free to reopen it.