Need way to poll sources (from redis)

[StevenCTimm]

When everything is now concentrated into one channel and that channel goes offline for some reason, then it is not possible to poll any of the data blocks via the de-client --print-product functionality. In this case it would be nice to be able to poll the data from the running sources in the redis facility, because the sources all keep running.

[StevenCTimm]

Discussion in today's meeting brought up that there is likely a way already in redis-cli to do this, we just have to figure out the right commands. redis-cli can be run via podman exec against the container.

[knoepfel]

@StevenCTimm, we can likely find a way to print the value of enqueued data products. However, now that message-bloat problem with DE 2.0 has been resolved, printing the value of enqueued data products will only be useful when no channel is able to consume the products--e.g., the channel has gone offline for some reason. So...is this facility still desired?

[StevenCTimm]

The case where the channel has gone offline is exactly the place where we need to examine the contents of the source products. One of the key reasons that channels do go offline is because of either the absence of info that should be there, or extra junk coming in that should not be there. Source polling in this case is important.. and the read_redis.py temporary utility that was made I've found to be quite useful. Steve

…

________________________________ From: Kyle Knoepfel ***@***.***> Sent: Monday, March 28, 2022 11:31 AM To: HEPCloud/decisionengine ***@***.***> Cc: Steven C Timm ***@***.***>; Mention ***@***.***> Subject: Re: [HEPCloud/decisionengine] Need way to poll sources (from redis) (Issue #627) @StevenCTimm<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_StevenCTimm&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=10BCTK25QMgkMYibLRbpYg&m=etnG_Pc54mEO5vpJNtYFZPVhjp0sHLbhUcIHLjc7oVTeLmQZamQezYal_oevFugW&s=mlZYJErCQdJdQXLfiwvJxXpohyQcOkZ3Rl3h2WHt2xk&e=>, we can likely find a way to print the value of enqueued data products. However, now that message-bloat problem with DE 2.0 has been resolved, printing the value of enqueued data products will only be useful when no channel is able to consume the products--e.g., the channel has gone offline for some reason. So...is this facility still desired? — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_HEPCloud_decisionengine_issues_627-23issuecomment-2D1080876428&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=10BCTK25QMgkMYibLRbpYg&m=etnG_Pc54mEO5vpJNtYFZPVhjp0sHLbhUcIHLjc7oVTeLmQZamQezYal_oevFugW&s=0Z7gninVZIANW87A32MmfAofV1kg3zeQtxbXLGufKVU&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AGG4SOC7OYZ7TEUNITC7Q53VCHNHNANCNFSM5QHTQDJA&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=10BCTK25QMgkMYibLRbpYg&m=etnG_Pc54mEO5vpJNtYFZPVhjp0sHLbhUcIHLjc7oVTeLmQZamQezYal_oevFugW&s=B656u_cHO-YZftxYPujabsKtWJK7MagK8PpUPjs-rn4&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>