Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Thanos Store v0.32.0 draining node memory until (OOM) #6644

Closed
rgarcia89 opened this issue Aug 24, 2023 · 3 comments
Closed

Thanos Store v0.32.0 draining node memory until (OOM) #6644

rgarcia89 opened this issue Aug 24, 2023 · 3 comments

Comments

@rgarcia89
Copy link
Contributor

rgarcia89 commented Aug 24, 2023 •
edited
Loading

Thanos, Prometheus and Golang version used:
Thanos:

thanos, version 0.32.0 (branch: HEAD, revision: 041eb41a9628e8123c9dabf20c2e35e1328d3a8b)
  build user:       root@0eb82e8bb550
  build date:       20230823-13:38:23
  go version:       go1.20.7
  platform:         linux/amd64
  tags:             netgo

Deployed using kube-thanos: v0.29.0 (no memory limit)
Prometheus: 2.46.0

Object Storage Provider:
Azure

What happened:
After updating to v0.32.0, all but the Thanos Store component started properly. The Thanos-Store ended up in a crashloopbackoff due to running the node out of memory on which it was running.

image

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
I am not sure. I have downgraded the Thanos Store to v0.31.0 for now - which works properly.

Full logs to relevant components:

Logs 

ts=2023-08-24T06:38:27.479455108Z caller=factory.go:53 level=info msg="loading bucket configuration"
ts=2023-08-24T06:38:27.511151985Z caller=inmemory.go:179 level=info msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
ts=2023-08-24T06:38:27.511893952Z caller=options.go:26 level=info protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
ts=2023-08-24T06:38:27.512791817Z caller=store.go:537 level=info msg="starting store node"
ts=2023-08-24T06:38:27.512866733Z caller=store.go:418 level=info msg="initializing bucket store"
ts=2023-08-24T06:38:27.512910128Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
ts=2023-08-24T06:38:27.512955655Z caller=http.go:73 level=info service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
ts=2023-08-24T06:38:27.513158533Z caller=tls_config.go:232 level=info service=http/server component=store msg="Listening on" address=[::]:10902
ts=2023-08-24T06:38:27.513175857Z caller=tls_config.go:235 level=info service=http/server component=store msg="TLS is disabled." http2=false address=[::]:10902
ts=2023-08-24T06:38:27.898132927Z caller=fetcher.go:487 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=385.229139ms duration_ms=385 cached=290 returned=195 partial=0
ts=2023-08-24T06:38:27.909888217Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=11.437031ms id=01H2QCX23FBQCAEGJPMHW58KNJ
ts=2023-08-24T06:38:27.911702359Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=1.768434ms id=01H8GK2T3R8Y1SGW0E8M8WXWV3
ts=2023-08-24T06:38:27.915241344Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=17.015247ms id=01H4MVE4SSNX8NWNJP2QP7F97A
ts=2023-08-24T06:38:27.918704277Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=4.546254ms id=01H8GK2TP1ER452RMQGVGPCFK5
ts=2023-08-24T06:38:27.92183508Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=7.237761ms id=01H2QBT9DXDBKZKSGA2RZ365FD
ts=2023-08-24T06:38:27.925630445Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=10.357534ms id=01H8B9C7N9BJZ1Z9CY61V5G481
ts=2023-08-24T06:38:27.933804413Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=19.874588ms id=01H5RXN8WNQBDVN2S2JF60D1C5
ts=2023-08-24T06:38:27.934217059Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=12.351163ms id=01H4MVTCGQVHWJ0ZH4JRANXXMT
ts=2023-08-24T06:38:27.939600787Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=5.353119ms id=01H2QBKTA1WD65S504KC8PT1G5
ts=2023-08-24T06:38:27.93984185Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=41.30552ms id=01H6X0EPETNEC3QW5CPH9FR5SX
ts=2023-08-24T06:38:27.940523985Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=5.946928ms id=01H8H7P08KSHYWS35835JAWKG8
ts=2023-08-24T06:38:27.946584081Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=5.601612ms id=01H8HEHPNRHCGXMM2YTKT0H362
ts=2023-08-24T06:38:27.963448188Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=4.567087ms id=01H8JQR25RZR7A72Q3HKJJTQEC
ts=2023-08-24T06:38:27.968361494Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=42.633396ms id=01H8HW952FDE9KMSTCD0G0PRBF
ts=2023-08-24T06:38:27.968439874Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=4.958653ms id=01H8HEHPSHV51Y9YPCAFNXAMXK
ts=2023-08-24T06:38:27.974739274Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=63.00112ms id=01H3GW2RTJR8XH61WH4ABSGJ15
ts=2023-08-24T06:38:27.975255965Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=5.204279ms id=01H8JQR361WGVMRF3E8VNSQSNE
ts=2023-08-24T06:38:27.980437289Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=9.551884ms id=01H5RXVM02CC8PEXWXE3M78QA6
ts=2023-08-24T06:38:27.983106343Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=64.287701ms id=01H864GZR86VC44HR6SYFDMZZP
ts=2023-08-24T06:38:27.986361597Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=15.237898ms id=01H8JYKTE13G17JBTSJHD52Z82
ts=2023-08-24T06:38:27.986816866Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=52.97994ms id=01H2QBDGS34S5TNXHEZNA63F0S
ts=2023-08-24T06:38:27.98840426Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=41.733838ms id=01H2QDC8EPFCABA3Y6Y1Y5R13W
ts=2023-08-24T06:38:27.991250419Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=19.926984ms id=01H8H0T89ETC2M7CC32BMECYQW
ts=2023-08-24T06:38:27.99727367Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=25.764349ms id=01H8JA0KNRFZDQY5R4S243CAAR
ts=2023-08-24T06:38:28.000121853Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.758234ms id=01H864FXYB5PPQ7RDMSRK425K4
ts=2023-08-24T06:38:28.000750107Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=32.287311ms id=01H4MX3ERNWWSK2186GTT8CG1K
ts=2023-08-24T06:38:28.004886091Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=106.472994ms id=01H5RZ7M22P5X5AQ940YASK52P
ts=2023-08-24T06:38:28.007723194Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=37.426762ms id=01H811FCE56128Q0AZF9938CR2
ts=2023-08-24T06:38:28.008998511Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=37.146718ms id=01H3GVA7F44E25BWCVQ9FKE4AM
ts=2023-08-24T06:38:28.013009236Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=40.90527ms id=01H8K0DHSCJ2767HAPSQ8JGQCQ
ts=2023-08-24T06:38:28.013441861Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=43.978468ms id=01H864Q13WEW9WV6D93DD3X27A
ts=2023-08-24T06:38:28.014457934Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=9.54541ms id=01H4MV92G4GF8PRSX8V8R7GZ55
ts=2023-08-24T06:38:28.015925052Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=29.042803ms id=01H864F5KPJNQJC69VR37XAJRR
ts=2023-08-24T06:38:28.017647324Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=47.58607ms id=01H811S3HP3FFCMSB4PK8B2MZP
ts=2023-08-24T06:38:28.018542092Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.593839ms id=01H8HNDER90184CP10X4P7E6ZZ
ts=2023-08-24T06:38:28.020478986Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=32.043865ms id=01H8HNDDT8MATDS696F1C2VGWP
ts=2023-08-24T06:38:28.023966005Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=5.401893ms id=01H8B9JB6MFFSDPV0SJEPXKCMX
ts=2023-08-24T06:38:28.025224556Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=8.906735ms id=01H5RYEGZZVB9HT5ZVM4B34VZB
ts=2023-08-24T06:38:28.065914338Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=65.101086ms id=01H8H0T85RJJH5GX5AJHZSAMDN
ts=2023-08-24T06:38:28.066491738Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=94.77951ms id=01H8K0D9AKFCWV9B46Q97NK303
ts=2023-08-24T06:38:28.068436736Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.817588ms id=01H8JGWBY0H3DVX94MX86AAVWG
ts=2023-08-24T06:38:28.069009016Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.055548ms id=01H864RFZ84H54BEXTGPZ5ZHMT
ts=2023-08-24T06:38:28.069702381Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=99.481902ms id=01H8GE8DFQKDMQB14M31EADW4Y
ts=2023-08-24T06:38:28.071921614Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=80.644001ms id=01H8GE3TJ0TYYCW3AT9PVDR1VR
ts=2023-08-24T06:38:28.07289097Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.162194ms id=01H8B98DT49GJJC91CTHCH1CJJ
ts=2023-08-24T06:38:28.075881844Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.961764ms id=01H8GE51K5CCDK8MGSZ4QPA1V9
ts=2023-08-24T06:38:28.077941374Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=77.730866ms id=01H810FXGZNZ8HJKXEHXQYDBEA
ts=2023-08-24T06:38:28.079612161Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=110.31535ms id=01H8H0T8THM9WXZDTZGXD2R03Z
ts=2023-08-24T06:38:28.080513875Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=110.156482ms id=01H81153FS43TBZVSQJ4NM9Z35
ts=2023-08-24T06:38:28.111087752Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=96.598181ms id=01H8G5BB9E29C6YFC16SHAG0R9
ts=2023-08-24T06:38:28.113896281Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.10851ms id=01H8JGWBR83SRM0MPQZM197Y91
ts=2023-08-24T06:38:28.11698451Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.992ms id=01H8GEB7SRRWGGZJ9EBD29CS0T
ts=2023-08-24T06:38:28.122535001Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=46.559104ms id=01H8J34X846V6VHTN8K9MN905W
ts=2023-08-24T06:38:28.125272339Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=100.028679ms id=01H8HNDEY0K85NAM6NBPG2FFWC
ts=2023-08-24T06:38:28.126403738Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.840196ms id=01H8J34WHEE1YGH0BHE7TRF5CN
ts=2023-08-24T06:38:28.127361442Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=8.786647ms id=01H8H7P0E1NN1DEQZXJ5VFM1Y0
ts=2023-08-24T06:38:28.165101527Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=190.256819ms id=01H812ACRCQJ9NTYC5966BG94E
ts=2023-08-24T06:38:28.169724267Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=91.753211ms id=01H8HW9661TMVP8H3BG1FSNVM2
ts=2023-08-24T06:38:28.176178324Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=11.006829ms id=01H8HW95ZY41ZQ3QJ2FJGHVM2V
ts=2023-08-24T06:38:28.178327999Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=97.792951ms id=01H8GECKG1CY688PAXKQNYRWM7
ts=2023-08-24T06:38:28.1792909Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=192.447501ms id=01H3GVWYJDC3JQ37VDM43EYYMM
ts=2023-08-24T06:38:28.273392187Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=208.315322ms id=01H8GK2SSE94T191NPRCBPQC7F
ts=2023-08-24T06:38:28.279117289Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=210.012928ms id=01H6WYBY7A9YZVESAB5E6MYTA0
ts=2023-08-24T06:38:28.282405436Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=155.96048ms id=01H2QDFWRJVK9CGVEH33QP52FZ
ts=2023-08-24T06:38:28.285147548Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=172.287519ms id=01H3GVJV0691CQGG5K2RYPR686
ts=2023-08-24T06:38:28.285704094Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=174.589564ms id=01H864SYJ282HNC85JGK7D3J5C
ts=2023-08-24T06:38:28.286704035Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=61.441462ms id=01H8GK2TG5AY3DTVY5RDVP1HFH
ts=2023-08-24T06:38:28.287620768Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=128.92837ms id=01H8GE7FNEFRJSXZEG570F24RZ
ts=2023-08-24T06:38:28.288427432Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=163.105919ms id=01H8B9E9KF2Y8KT88QFDE7NHEA
ts=2023-08-24T06:38:28.322074325Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=204.61748ms id=01H80ZZ680VR02466T60K7PMKM
ts=2023-08-24T06:38:28.322483175Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=92.799652ms id=01H8B9B12W4S7CNZNWEPH05HP2
ts=2023-08-24T06:38:28.324844754Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=166.189105ms id=01H3GTQ20660M2QPAXX5ZQ1G0X
ts=2023-08-24T06:38:28.325371078Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.270649ms id=01H8GSYHY1VK4Y3N313A4KQ2N8
ts=2023-08-24T06:38:28.325441161Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=46.289892ms id=01H8G5BC07PXFWJAB09GM9EAQ8
ts=2023-08-24T06:38:28.328203922Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.740574ms id=01H8H0T903DNK0K4E66KYYVQBQ
ts=2023-08-24T06:38:28.330419247Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.186021ms id=01H8GC72C9Q46Z7PFHQ5CVQ5E2
ts=2023-08-24T06:38:28.331898015Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=204.520253ms id=01H6X0QZ8RGB0K0R61A42H812W
ts=2023-08-24T06:38:28.33211415Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=12.522305ms id=01H6WZTD5M1Z5Y8JDW930VBD4K
ts=2023-08-24T06:38:28.332561932Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=7.158062ms id=01H8J4Z9DF2Y0XE7CKRQXE90RN
ts=2023-08-24T06:38:28.370905212Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=1.967646ms id=01H8H7NZHETCNKZADJQDY0V8TS
ts=2023-08-24T06:38:28.371679928Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.016755ms id=01H8H0T84M9AW8X2E4TR9ZJN06
ts=2023-08-24T06:38:28.372967037Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=1.990725ms id=01H8GC72HE9X1X13T4AG1ZM83F
ts=2023-08-24T06:38:28.373716083Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=150.905157ms id=01H2QDMS98TJ1G9VXE9NTVZ62E
ts=2023-08-24T06:38:28.374095152Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=6.583453ms id=01H2QD7JRSMNAG7TCADTM7NW1C
ts=2023-08-24T06:38:28.375611191Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.610422ms id=01H8GE21QK4CJXZKYR8MTEJ7A1
ts=2023-08-24T06:38:28.377903706Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=6.132908ms id=01H3GTGSRWRB0NQ77W5GQ928SY
ts=2023-08-24T06:38:28.380630808Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=300.998503ms id=01H5RWWE9CF017RZ6PFYSMR0CS
ts=2023-08-24T06:38:28.382948364Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=306.234806ms id=01H5RX8EQ1XDP8CFZD90ZWGQW7
ts=2023-08-24T06:38:28.383536464Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=15.895601ms id=01H8GSYGXQ0425RKFC7ST684TF
ts=2023-08-24T06:38:28.387393911Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=58.722609ms id=01H6WZ59N223KVRF0GJDNTHQNM
ts=2023-08-24T06:38:28.419723866Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=89.27329ms id=01H8GC73E058NBWQ42NYX79F8J
ts=2023-08-24T06:38:28.421560495Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=1.801265ms id=01H8HW957G4QG3J6E9QZZE7HSW
ts=2023-08-24T06:38:28.468251101Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=94.507742ms id=01H4MV4ZS7S048AVR6Z05Y7A41
ts=2023-08-24T06:38:28.470969528Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.59484ms id=01H8GC7381CFMFFZJA7H3977VA
ts=2023-08-24T06:38:28.47106528Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=53.750182ms id=01H6WZCPAYZ0MJ6ZYW343RQK9B
ts=2023-08-24T06:38:28.47222686Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=52.136809ms id=01H8126NXDYNVT68M4FTVDMKDC
ts=2023-08-24T06:38:28.472488864Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=8.76317ms id=01H4MWT2XQZR841JG468PB0CWH
ts=2023-08-24T06:38:28.472920737Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=1.878724ms id=01H8H9FF0XB4PQFDQ5EE4H5Z2B
ts=2023-08-24T06:38:28.474684827Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.569689ms id=01H8B9FDEGTG0PRV8M4FTARNV3
ts=2023-08-24T06:38:28.476483038Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=1.72571ms id=01H8HW95478076V624A9CJ5MFR
ts=2023-08-24T06:38:28.480274733Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=321.383378ms id=01H3GTZMJRYWEPA52AR2Q6VNNE
ts=2023-08-24T06:38:28.480307741Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=92.89196ms id=01H4MXVD7AKCVPEB3B4M5YGQCY
ts=2023-08-24T06:38:28.488401972Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=155.825718ms id=01H4MXKKKC53XCMS53075NAN7P
ts=2023-08-24T06:38:28.49036415Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=312.01332ms id=01H8JYKSHE53GRKXD7K0DXJ512
ts=2023-08-24T06:38:28.511638723Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=491.055865ms id=01H5RX22BA91ASZQX5TSXP9QJC
ts=2023-08-24T06:38:28.512155358Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=394.660708ms id=01H8JQR30AFXP14WHYVK7C1GZT
ts=2023-08-24T06:38:28.512908371Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.451092ms id=01H8HNDE1F1Z9DHBJS3XEGNP52
ts=2023-08-24T06:38:28.515662357Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.72491ms id=01H8K0DVNP9DJ2CCECA51FF0JV
ts=2023-08-24T06:38:28.518721988Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=42.180795ms id=01H6X1CBATH694Q8ZHC71QGXGY
ts=2023-08-24T06:38:28.575974584Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.128946ms id=01H8JGWAW5RMVGC7MY00S7KV34
ts=2023-08-24T06:38:28.623667465Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.465423ms id=01H8JGWB1E6SMK8C42HQ0F82FR
ts=2023-08-24T06:38:28.624342575Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=202.758215ms id=01H8GSYHR63Y448PSTX28RY1M0
ts=2023-08-24T06:38:28.6259376Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.232029ms id=01H8JA0KN3S961MPZ4PP4YA338
ts=2023-08-24T06:38:28.626163169Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.940421ms id=01H8JA0MP1FHK8J1XFT79XW1HA
ts=2023-08-24T06:38:28.627597476Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.150265ms id=01H864MKJ55GWZGY6YWZ1SPE56
ts=2023-08-24T06:38:28.62826093Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=252.62521ms id=01H6X1G1GBPPGGJ0X4YF3B6Z8B
ts=2023-08-24T06:38:28.629043086Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.864385ms id=01H864NMK81NW9XZX5886JW3VH
ts=2023-08-24T06:38:28.631847427Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=455.626489ms id=01H5RYQATN14KXHB3RVDQHCCN0
ts=2023-08-24T06:38:28.632883175Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=5.190443ms id=01H2QD9SE0H7VQ2PJ2KFT5FZPS
ts=2023-08-24T06:38:28.632988242Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.883477ms id=01H8HEHQP1XXWE6ZGN69XBE3H7
ts=2023-08-24T06:38:28.634866892Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=8.580079ms id=01H6WYYBYXHM5V7S92CJD2QHGX
ts=2023-08-24T06:38:28.635087273Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=146.658517ms id=01H8HEHQFT4GZ2MT190YXKCR0P
ts=2023-08-24T06:38:28.635847634Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.887862ms id=01H8J4YM96SSMADADW14FY1KSD
ts=2023-08-24T06:38:28.636917098Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=1.805361ms id=01H8J34WDQDJMMDV5MBNHXTYTZ
ts=2023-08-24T06:38:28.637269731Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.360331ms id=01H8GSYGX4F1W6KMFDS3Y4HV6N
ts=2023-08-24T06:38:28.637517483Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=121.764193ms id=01H4MXD72T092KH0WXR844KP0M
ts=2023-08-24T06:38:28.641694844Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=129.989285ms id=01H6X183VYZ3B1KJ7FMWB6FKWK
ts=2023-08-24T06:38:28.673071497Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.156948ms id=01H8JYKSCE0BJKX95CZVV6V5K7
ts=2023-08-24T06:38:28.673868072Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=98.841123ms id=01H864K67K463P1DERDY9HXNFZ
ts=2023-08-24T06:38:28.675778677Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=38.828735ms id=01H8GC72DSPH5E5HH0RAVJ9R3F
ts=2023-08-24T06:38:28.678174336Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=158.265579ms id=01H4MXFS91FN1GVW1WTX0RKJ16
ts=2023-08-24T06:38:28.724534057Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.914043ms id=01H8HEHPMQNDBYXZZDD56EC8FV
ts=2023-08-24T06:38:28.736879605Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=58.678815ms id=01H8H9GEQPWXDED2N4DHHCR0M1
ts=2023-08-24T06:38:28.742323321Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=73.309391ms id=01H8JYKT825PPT2T9J3BB46DAH
ts=2023-08-24T06:38:28.74419537Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=71.082783ms id=01H3GW6ST501WQJK42TH6Q29BM
ts=2023-08-24T06:38:28.748719361Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=172.703477ms id=01H6WYGBGHXAW3RDQY1T32BN87
ts=2023-08-24T06:38:28.774652356Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=6.64518ms id=01H8B99ZWASZHQP2PCN7NRACWQ
ts=2023-08-24T06:38:28.776623694Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=9.817926ms id=01H8B9GYM7MDVPM1Q1S2B4PVFW
ts=2023-08-24T06:38:28.780417009Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=506.987179ms id=01H81047594QV1NJGZ4F2MPBDT
ts=2023-08-24T06:38:28.781112491Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=6.423875ms id=01H8J34WC26ANTE748GBQVT45V
ts=2023-08-24T06:38:28.783280349Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=56.265884ms id=01H8G5BC60V1EFTEZRNNVMWFSB
ts=2023-08-24T06:38:28.783781292Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=17.473965ms id=01H8GSYH1FQVBR72VXDXZXAJM0
ts=2023-08-24T06:38:28.784121355Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=59.425325ms id=01H8JYKSDQT6F1C0TG6VQ6Q9V9
ts=2023-08-24T06:38:28.785399053Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=148.106709ms id=01H8J4Y4FPFMAKW6R3FAF4PJWM
ts=2023-08-24T06:38:28.791175649Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=155.243618ms id=01H4MWFH1P9R6K4GMBNWR4RW94
ts=2023-08-24T06:38:28.816913071Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=89.740468ms id=01H5RZFYBP9HG3WWNKWVDRET3W
ts=2023-08-24T06:38:28.81880216Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.641881ms id=01H8HNDDXQY3A679VRQZ3HYMM6
ts=2023-08-24T06:38:28.819271988Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=4.284342ms id=01H8H9G3GGVRM7QKQ6HVXBQTRQ
ts=2023-08-24T06:38:28.82037107Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=82.718473ms id=01H80ZVQPRKWDEBY7FG5C84EXA
ts=2023-08-24T06:38:28.822166118Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=6.837578ms id=01H8B9D9VHGD6DES877SFQDT6A
ts=2023-08-24T06:38:28.823184135Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=56.636839ms id=01H810WTJPNKXNCZ47T2AN5JP8
ts=2023-08-24T06:38:28.823205155Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=7.944403ms id=01H8G5BB5QYBP07BZFQ608HMEM
ts=2023-08-24T06:38:28.834663553Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=51.356287ms id=01H4MVMKG2H8KQXTXGR628Z1ZA
ts=2023-08-24T06:38:28.83545451Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=19.57896ms id=01H8JQR249MR3J33N3E35HP745
ts=2023-08-24T06:38:28.864576596Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=83.37085ms id=01H5RZ14NMD3CX7F5WB0GNW34S
ts=2023-08-24T06:38:28.868226658Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=32.754486ms id=01H6X11T0GZGK54EX0HP02B0YR
ts=2023-08-24T06:38:28.868614004Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=53.088079ms id=01H5RZC6E2PK048W7H7GM4YAZZ
ts=2023-08-24T06:38:28.86912726Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=53.413109ms id=01H8GE9NRQR0YVSRDZJ9FF3PBE
ts=2023-08-24T06:38:28.873176896Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=294.672572ms id=01H6WYQ2ASW713ZP83EN900Q8J
ts=2023-08-24T06:38:28.919180893Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=86.994012ms id=01H8GE2X4QW5FR8F3CQS31ARHT
ts=2023-08-24T06:38:28.924372748Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=92.811296ms id=01H3GVZE2JY6HPTJ2XJMB210KC
ts=2023-08-24T06:38:28.925036346Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=93.028956ms id=01H3GWAKJHNWNW4VGZ3B7BJNJE
ts=2023-08-24T06:38:28.976798548Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.151543ms id=01H8J4YYAK26XZWM7V60TYYG1D
ts=2023-08-24T06:38:28.980620593Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=148.899307ms id=01H4MW799MVM15Z448JBK8NRQX
ts=2023-08-24T06:38:28.982432965Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=150.634359ms id=01H81241S83G3EYK225FM5AN9Y
ts=2023-08-24T06:38:28.984476767Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=168.981877ms id=01H8K0E6DDRFKF8Z35MZV68S1X
ts=2023-08-24T06:38:29.023212428Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=204.385197ms id=01H8H7NZW4N9MAPTSRECGKS13E
ts=2023-08-24T06:38:29.024936435Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=190.251976ms id=01H8JGWAXQF4B7D6X93ADNY0SB
ts=2023-08-24T06:38:29.026552923Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=5.10341ms id=01H8GE6CPCWY7J9136J0FKEHEW
ts=2023-08-24T06:38:29.027126461Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=195.566874ms id=01H8H9F801ZR0X94KAM5632CHM
ts=2023-08-24T06:38:29.029374295Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=108.679114ms id=01H6X14EFW6HFTBPANT44K132G
ts=2023-08-24T06:38:29.069884026Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.042536ms id=01H8H7NZE1MR3HPH507C4XSW5X
ts=2023-08-24T06:38:29.070469092Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.717576ms id=01H8B993A9VNMGE0Y9VRE82901
ts=2023-08-24T06:38:29.074597771Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=155.37934ms id=01H5RXEJCFGWWRWAHTJFHWN93C
ts=2023-08-24T06:38:29.074804366Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=260.255707ms id=01H8G5BBM3060F8MBWG4TDSQZB
ts=2023-08-24T06:38:29.077113273Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=50.531605ms id=01H864J1FATWHFR4PXK1MMV2FW
ts=2023-08-24T06:38:29.120096044Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=1.909036ms id=01H8JQR29E520WNFQY8JEWWGDK
ts=2023-08-24T06:38:29.1202252Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=96.8722ms id=01H4MXQMT2GC4K493M9YX6AMMV
ts=2023-08-24T06:38:29.126593182Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=309.641563ms id=01H6X03PECC8J1Y5Z83QG2QGW0
ts=2023-08-24T06:38:29.129926964Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=59.901708ms id=01H810AEXXF3XZJ54T8B88DKHZ
ts=2023-08-24T06:38:29.133708649Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=63.219031ms id=01H2QCFN43NBQ5AWMSSAKMJKEY
ts=2023-08-24T06:38:29.165748614Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=334.109651ms id=01H8JA0KSE4Y1KFB6Y3DRM8EKY
ts=2023-08-24T06:38:29.166855284Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.344552ms id=01H8K0D1X17M8M1N2X9VYZE9Q6
ts=2023-08-24T06:38:29.169115757Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.200826ms id=01H8GK2SNQC4RTJE57TVZVCQES
ts=2023-08-24T06:38:29.17116381Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=306.542239ms id=01H812EWMC47PK8QEM5CG61XCS
ts=2023-08-24T06:38:29.023317179Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=105.213878ms id=01H8H9FR3WF73KF0RC7P4YTZ68
ts=2023-08-24T06:38:29.171450822Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=194.612103ms id=01H810PG7JCR8A7JF5HAMYN10Y
ts=2023-08-24T06:38:29.171863283Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=2.715999ms id=01H8JA0MG2G9FQGNQWYWZYDEKK
ts=2023-08-24T06:38:29.1732658Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=151.854993ms id=01H5RZ3SFENDS4G64SGSRB4EYK
ts=2023-08-24T06:38:29.174045784Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=40.075674ms id=01H8J4YBQ7CGC0KJGCM3H5RRQR
ts=2023-08-24T06:38:29.174987287Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=3.104674ms id=01H8J34XE102ZE3XF4CFR5HNQQ
ts=2023-08-24T06:38:29.178895843Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=361.602651ms id=01H5RWRQ7816YPXD6S9S2D5E76
ts=2023-08-24T06:38:29.181035786Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=399.883635ms id=01H5RY3SZGYFH5QSXQJPMVBW1R
ts=2023-08-24T06:38:29.182527121Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=365.462352ms id=01H812JQ66Y8EJHN5NBFRAP8FA
ts=2023-08-24T06:38:29.184681187Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=13.481053ms id=01H4MW0V42WT38MRNJ6TT7JXQ7
ts=2023-08-24T06:38:29.185142447Z caller=store.go:435 level=info msg="bucket store ready" init_duration=1.672248945s
ts=2023-08-24T06:38:29.185264838Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready
ts=2023-08-24T06:38:29.185318524Z caller=grpc.go:131 level=info service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901
ts=2023-08-24T06:38:29.360586571Z caller=fetcher.go:487 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=175.420503ms duration_ms=175 cached=290 returned=195 partial=0

Anything else we need to know:
@rgarcia89 rgarcia89 mentioned this issue Aug 24, 2023
Closed
@rgarcia89
Copy link
Contributor Author

duplicate

@rgarcia89 rgarcia89 closed this as not planned Won't fix, can't repro, duplicate, stale Aug 24, 2023
@farodin91
Copy link
Contributor

We hit the same issue.

@antikilahdjs
Copy link

antikilahdjs commented Aug 24, 2023 •
edited
Loading

I have the same issue and upgraded to v31, all the other components works perfectly in 0.32 version but the store consume 1tb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@farodin91 @rgarcia89 @antikilahdjs