Can someone help with a minimal clustered config #55
Replies: 12 comments 18 replies
-
|
Hi, I was asked by email to comment on this, but this looks like it is pretty Pelican specific (i.e. the |
Beta Was this translation helpful? Give feedback.
-
|
Hi @amadio - Actually, I think the problem is more about plain ol' xrootd clustering (it just happens that the OSS backend is the S3 one). I think that's why we were struggling within the Pelican team to get someone to answer (besides me, not a lot of folks know how to setup the cmsd). @osschar -- do you recall any of the cmsd stuff? Brian |
Beta Was this translation helpful? Give feedback.
-
|
Well ... I sure do recall some of it :) You guys are setting up a cache cluster over S3? Hmmh ... how does that make sense? You want to have multiple caches all writing into the same S3 space as DFS? I guess we should really meet in person and talk about it. |
Beta Was this translation helpful? Give feedback.
-
|
If you have some time Matevž, either tomorrow, or on wednesday afternoon, we could meet on Zoom?
Micke
Den 22 november 2024 18:53:23 CET, "Matevž Tadel" ***@***.***> skrev:
…Well ... I sure do recall some of it :) You guys are setting up a cache cluster over S3? Hmmh ... how does that make sense? You want to have multiple caches all writing into the same S3 space as DFS? I guess we should really meet in person and talk about it.
--
Reply to this email directly or view it on GitHub:
#55 (comment)
You are receiving this because you authored the thread.
Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Sure, let's try, shoot me an email. I'm on Pacific time. |
Beta Was this translation helpful? Give feedback.
-
|
I sent you an invite for today, hopefully it works! |
Beta Was this translation helpful? Give feedback.
-
|
Just to let everyone know, I got further thanks to @osschar ! This is what I have now: Managers:
Servers:
They are each running two docker containers with a shared volume for ipc This is the config for the managers: And almost the same for the servers: This is sufficient to get all the containers to start. This is what the managers say: The cmsd container does not even complain :) The xrootd container however has errors in the log: I have not yet tried to enable authentication, so maybe not surprising, that it fails to login. For the servers, the situation is reversed, xrootd does not complain: but cmsd does: I see alot of potential issues here though, related to ip addresses, I see a mix of ipv6 and ipv4 and internal docker ip addresses, and also fqdn, vs host names. What do you guys think, what should be my next step? |
Beta Was this translation helpful? Give feedback.
-
|
Do you have ports open? Do the files for xrd/cms communication get created correctly -- and have the correct permissions / access across containers? I'm not exactly sure how multiple manager setup works if configured as you have it. What we do in CMS, for load balancing and redundancy -- we use DNS alias to register two managers together and then use the and, on machines below those managers (in our case this is a bit more complicated as we have another level in manager hierarchy) you would do: Since you use default ports -- you do not need the Are you able to 'talk' to the servers themselves? List "directories", copy files in and out? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. This was helpful. I noticed some bugs in my config, I had 1094 as port in the all.manager stanza instead of 1213 and I also needed to move the I also changed the the all.manager to use the Now both containers/processes for xrootd and cmsd starts up without complaints in the log. I will now move on to see if I can interact with the setup by looking at files and adding/removing them and such. If that works, I will start looking in to tls and authenication. |
Beta Was this translation helpful? Give feedback.
-
|
Some amount of success: But also something is not working quite right with the s3-connection: I can see that it exists in s3 though One interesting thing, is that when I try to access a directory that I know does not exist, I see an error message in the logs: But nothing in the logs if I try to access the directory that I know exists. I do get different answers from the cli depending on if the directory actually exists. For existing dir: For non-existing dir: Obviously I checked that the credentials for s3 are correct, they are the same as I use with rclone. I do know that this is fundamentally insecure and accessible to anyone at this point, so you should even be able to access this set up from wherever without authentication. I will totally scrap these servers when I have something working and do a complete reinstall... |
Beta Was this translation helpful? Give feedback.
-
|
What happens if you talk directly to the server, not the manager? After that we'd need to bump up the trace levels -- one way is to add various xxx.trace options into your config ... but for your test it's probably enough to just run xrootd with Once we figure out what is going on Andy should be able to recommend the trace options, I'd have to go read the docs and scratch my head a bit :) |
Beta Was this translation helpful? Give feedback.
-
|
I suspect something is wrong with the xrootd config on the manager servers. When I try to access the file via the round robin dns of the manager address, it does not redirect, but instead can not find the file, from xrdm1: This is what I seen in cmsd log at the same time, this time on xrdm2: |
Beta Was this translation helpful? Give feedback.
-
|
I also see this: Not exactly sure what that is about, since they both get their config from puppet, so I am guessing it might have something to do with the generated config in /var/spool/xrootd? |
Beta Was this translation helpful? Give feedback.
-
|
Huh! I think I see the issue now. On the manager servers cmsd starts up and binds to port 1213, just fine: However on the server side, cmsd starts up using a random port: And as such I suppose the managers cant talk to them... What I cant figure out is why this happens: That is litteraly the only difference in the config for them. |
Beta Was this translation helpful? Give feedback.
-
|
Seems unrelated though. I opened all the ports between 32768 - 60999 in the firewall, but no change. |
Beta Was this translation helpful? Give feedback.
-
|
@osschar should I revert the firewall changes or do we still need them? |
Beta Was this translation helpful? Give feedback.
-
|
I got it working now! to: |
Beta Was this translation helpful? Give feedback.
-
|
But not fully: |
Beta Was this translation helpful? Give feedback.
-
|
You can close the ports. Manager cmsds listen on 1213 for connections from servers. xrootd and cmsd on the sam machine communicate through FS sockets. Server-side cmsd shouldn't even need to listen on any port, I think, hmmh. With @abh3, am I lying here about something? :) |
Beta Was this translation helpful? Give feedback.
-
|
Hi Matevz,
Everything you said was true. Indeed, the server cmsd doesn't need to listen on a port as its only incoming connection is the named pipe. The -d option will not increase the debugging level as I don't think that code checks if the XRDDEBUG envar is set. Is it our S3 plugin or pelican's plugin? Wei is the person who knows the details of how a listing works in S3 and I know it's rather complicated and unusually baroque involving a lot of xml. Send him a ping.
Andy
…________________________________
From: Matevž Tadel ***@***.***>
Sent: Friday, December 6, 2024 10:00 AM
To: PelicanPlatform/xrootd-s3-http ***@***.***>
Cc: Andrew Hanushevsky ***@***.***>; Mention ***@***.***>
Subject: Re: [PelicanPlatform/xrootd-s3-http] Can someone help with a minimal clustered config (Discussion #55)
BEWARE: This email originated outside of our organization. DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.
You can close the ports. Manager cmsds listen on 1213 for connections from servers. xrootd and cmsd on the sam machine communicate through FS sockets.
Server-side cmsd shouldn't even need to listen on any port, I think, hmmh.
With all.export / and without authorization this should go through the xrootd layer. What do the logs say if you bump up ofs and s3 trace levels (or run with -d -- not sue if this will also increase debug for s3)? I have no clue how read and list permissions are done / handled in s3.
@abh3<https://github.com/abh3>, am I lying here about something? :)
—
Reply to this email directly, view it on GitHub<#55 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAUIW55DN5R6BDSGCTVQLB32EHQ5PAVCNFSM6AAAAABR56VGNGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNBYG43TQMA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
@jhiemstrawisc -- please advise what to do here :) |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, In the meantime, if your S3 endpoint supports virtual/DNS-style buckets, the immediate fix is as trivial as switching |
Beta Was this translation helpful? Give feedback.
-
|
In general, the S3 permissions are decoupled from XRootD permissions. If XRootD accepts whatever form of authz it's given, the S3 plugin will pick up the S3 access/secret keys and use them to sign an HTTP request to the S3 endpoint. It's this signature generation step that isn't yet working with path-style list operations. |
Beta Was this translation helpful? Give feedback.
-
|
Alright, thanks. If this is expected behavior that is great for me, since it is not a mistake on my part then 😆 I can move on to adding auth and tls to my set up, and I'll try a new version of the plugin when you have shipped it :) |
Beta Was this translation helpful? Give feedback.
-
|
@mickenordin, we created a new |
Beta Was this translation helpful? Give feedback.
-
I want to set up a small cluster that have:
2 redirectors
3 xrootd-s3-http proxies
This is for transferring large amounts of data into our s3 storage/
I have successfully set up a very simple single xrootd-s3-http proxy that can read and write to a bucket using this config (in erb template format):
However, now with a more complex config I am struggeling.
For example, I understand that I need to run a
cmsdinstance on each of the nodes. Is this correct? Do I understand correctly that thexrootdandcmsduse the same config file?This is my current, non working try:
A specific problem is the role "proxy manager" and "proxy server" seems wrong, but this is what I see in: https://xrootd.slac.stanford.edu/doc/dev53/cms_config.htm
Obviously I have misunderstood something, and if someone can point these misunderstandings to me or otherwise point me in the right direction I would be very grateful. I am a total noob when it comes to xrootd.
Beta Was this translation helpful? Give feedback.
All reactions