From 396438444a5f1a1084e6797666cfd19edfa3df07 Mon Sep 17 00:00:00 2001 From: Matyas Selmeci Date: Fri, 16 Aug 2024 17:33:33 -0500 Subject: [PATCH 01/13] Let's Encrypt CA cert docs: use different paths for Pelican/OSDF The Pelican/OSDF configuration does not look for the cert/key in the usual /etc/grid-security location, so update the Let's Encrypt docs to mention the difference. --- docs/security/host-certs/lets-encrypt.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/security/host-certs/lets-encrypt.md b/docs/security/host-certs/lets-encrypt.md index 41bb1a5f5..70298a88b 100644 --- a/docs/security/host-certs/lets-encrypt.md +++ b/docs/security/host-certs/lets-encrypt.md @@ -34,6 +34,13 @@ Installation and Obtaining the Initial Certificate 1. Set up hostcert/hostkey links: + If using host certificates for Pelican/OSDF: + :::console + root@host # ln -sf /etc/letsencrypt/live/*/cert.pem /etc/pki/tls/certs/pelican.crt + root@host # ln -sf /etc/letsencrypt/live/*/privkey.pem /etc/pki/tls/private/pelican.key + root@host # chmod 0600 /etc/letsencrypt/archive/*/privkey*.pem + + If using host certificates for other software: :::console root@host # ln -sf /etc/letsencrypt/live/*/cert.pem /etc/grid-security/hostcert.pem root@host # ln -sf /etc/letsencrypt/live/*/privkey.pem /etc/grid-security/hostkey.pem From 54741367d26f661dd4aa27c770687ac13dc5ef7c Mon Sep 17 00:00:00 2001 From: Matyas Selmeci Date: Fri, 16 Aug 2024 17:56:58 -0500 Subject: [PATCH 02/13] WIP: Instructions for installing a Pelican/OSDF Origin via RPM (SOFTWARE-5962) - Some of the configuration is still up in the air as we learn more from supporting JLab/GLBRC/others, and as we revamp the config we ship - Registration instructions could use improvement - No validation instructions yet --- docs/data/osdf/install-origin-rpm.md | 171 +++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 docs/data/osdf/install-origin-rpm.md diff --git a/docs/data/osdf/install-origin-rpm.md b/docs/data/osdf/install-origin-rpm.md new file mode 100644 index 000000000..7fadcfd39 --- /dev/null +++ b/docs/data/osdf/install-origin-rpm.md @@ -0,0 +1,171 @@ +title: Installing the OSDF Origin by RPM + +Installing the OSDF Origin +========================== + +This document describes how to install a Pelican-based Open Science Data Federation (OSDF) Origin service via RPMs. +This service allows an organization to export its data to the Data Federation. + +!!! note + The origin must be registered with the OSG prior to joining the data federation. + You may start the registration process prior to finishing the installation by [using this link](#registering-the-origin) + along with information like: + + * Resource name and hostname + * VO associated with this origin server (which will be used to determine the origin's namespace prefix) + * Administrative and security contact(s) + * Who (or what) will be allowed to access the VO's data + * Which caches will be allowed to cache the VO data + + +Before Starting +--------------- + +Before starting the installation process, consider the following requirements: + +* __Operating system:__ A RHEL 8 or RHEL 9 or compatible operating systems. +* __User IDs:__ If they do not exist already, the installation will create the Linux user ID `xrootd` for running daemons. +* __Host certificate:__ Required for authentication. See note below. +* __Network ports:__ The origin service requires the following ports open: + * Inbound TCP port 8443 for file access via the HTTP(S) and XRoot protocols. + * (Optional) Inbound TCP port 8444 for access to the web interface for monitoring and configuration; + if enabled, this should be restricted to the LAN or management network. +* __Hardware requirements:__ We recommend that an origin has at least 1Gbps connectivity and 12GB of RAM. + We suggest that several gigabytes of local disk space be available for log files, + although some logging verbosity can be reduced. + +As with all OSG software installations, there are some one-time steps to prepare in advance: + +* Obtain root access to the host +* Prepare [the required Yum repositories](../../common/yum.md) + + +!!! note "OSG 23" + In OSG 23, the Pelican-based OSDF RPMs are only available in the "osg-upcoming" repositories. + +!!! note "Host certificates" + Origins should use a CA that is accepted by major browsers and operating systems, + such as InCommon RSA or [Let's Encrypt](../../security/host-certs/lets-encrypt). + IGTF certs are not recommended because clients are not configured to accept them by default. + + The following locations should be used (note that they are in separate directories): + + * **Host Certificate**: `/etc/pki/tls/certs/pelican.crt` + * **Host Key**: `/etc/pki/tls/private/pelican.key` + + +Installing the Origin +--------------------- + +The origin service is provided by the `osdf-origin` RPM. +Install it using the following command: + + +```console +root@host # yum install --enablerepo=osg-upcoming osdf-origin +``` + + +Configuring the Origin Server +----------------------------- + +Configuration for a Pelican-based OSDF Origin is located in `/etc/pelican/osdf-origin.yaml`. + +You must configure the following: +``` +XRootD: + Sitename: +``` + +In addition, you must tell Pelican the data to export to the federation. +An origin may export one or more directory trees, or one or more S3 buckets -- follow one of the sections below. +(An single origin cannot export both a bucket and a directory tree.) + + + +### Configuring POSIX (directory) export + +Set these options to export one or more directory trees to the federation. + +``` +Origin: + StorageType: "posix" + Exports: + # You may have one or more of the following block: + - FederationPrefix: "" + StoragePrefix: "" + Capabilities: # Add or remove as desired + - Reads # Enable authenticated reading of objects from under the directory tree through a cache + - PublicReads # Enable unauthenticated reading of objects from under the directory tree through a cache + - DirectReads # Enable reading objects from under the directory tree + # without going through a cache + - Listings # Enable directory listings + - Writes # Enable writing to files in the directory tree +``` + +### Configuring S3 export + +Set these options to export one or more S3 buckets to the federation + +``` +Origin: + StorageType: "s3" + S3Region: "" + S3ServiceUrl: "" + S3UrlStyle: "path" + Exports: + # You may have one or more of the following block: + - FederationPrefix: "" + S3Bucket: "" + S3AccessKeyfile: "" + S3SecretKeyfile: "" + Capabilities: # Add or remove as desired + - Reads # Enable authenticated reading of objects from the bucket through a cache + - PublicReads # Enable unauthenticated reading of objects from the bucket through a cache + - DirectReads # Enable reading objects from the bucket + # without going through a cache + - Listings # Enable listing bucket items + - Writes # Enable writing to objects in the bucket +``` + + +Preparing for Initial Startup +----------------------------- + +1. The origin identifies itself to the federation via public key authentication; +before starting the origin for the first time, it is recommended to generate a keypair. + + :::command + root@host$ cd /etc/pelican + root@host$ pelican generate keygen + + + The newly created files, `issuer.jwk` and `issuer-pub.jwks` are the private and public keys, respectively. + **Save these files**; if you lose them, you will have to re-register the origin. + +1. Contact OSG Staff and let them know that you are about to start your origin, + and what namespace(s) the origin will serve. + OSG Staff will need to approve the origin's registration. + + +Managing the Origin Service +--------------------------- +Use the following SystemD commands as root to start, stop, enable, and disable the OSDF Origin. + +| To... | Run the command... | +| :--------------------------------------- | :--------------------------------- | +| Start the origin | `systemctl start osdf-origin` | +| Stop the origin | `systemctl stop osdf-origin` | +| Enable the origin to start on boot | `systemctl enable osdf-origin` | +| Disable the origin from starting on boot | `systemctl disable osdf-origin` | + + +Registering the Origin +---------------------- +To be part of the Open Science Data Federation, your origin must be +[registered with the OSG](../../common/registration.md). The service type is `Pelican origin`. + + +Getting Help +------------ +To get assistance, please use the [this page](../../common/help.md). From 2015290194f7c6d7348ba2c6990fef448c6c3925 Mon Sep 17 00:00:00 2001 From: Matyas Selmeci Date: Fri, 11 Oct 2024 15:28:44 -0500 Subject: [PATCH 03/13] Let's Encrypt: mention that Pelican needs the full cert chain --- docs/security/host-certs/lets-encrypt.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/security/host-certs/lets-encrypt.md b/docs/security/host-certs/lets-encrypt.md index 70298a88b..27bab39cb 100644 --- a/docs/security/host-certs/lets-encrypt.md +++ b/docs/security/host-certs/lets-encrypt.md @@ -36,7 +36,7 @@ Installation and Obtaining the Initial Certificate If using host certificates for Pelican/OSDF: :::console - root@host # ln -sf /etc/letsencrypt/live/*/cert.pem /etc/pki/tls/certs/pelican.crt + root@host # ln -sf /etc/letsencrypt/live/*/fullchain.pem /etc/pki/tls/certs/pelican.crt root@host # ln -sf /etc/letsencrypt/live/*/privkey.pem /etc/pki/tls/private/pelican.key root@host # chmod 0600 /etc/letsencrypt/archive/*/privkey*.pem @@ -46,6 +46,10 @@ Installation and Obtaining the Initial Certificate root@host # ln -sf /etc/letsencrypt/live/*/privkey.pem /etc/grid-security/hostkey.pem root@host # chmod 0600 /etc/letsencrypt/archive/*/privkey*.pem + + Note that Pelican requires the full certificate chain, not just the certificate, + so the pelican.crt symlink needs to point to fullchain.pem, not cert.pem. + 1. Restart services running on port 80 if there were any. From ee75e3039e17b8a5e525a2b55585389edf0afceb Mon Sep 17 00:00:00 2001 From: Matyas Selmeci Date: Fri, 11 Oct 2024 15:48:04 -0500 Subject: [PATCH 04/13] Mention that you need the cert change in install-origin-rpm as well --- docs/data/osdf/install-origin-rpm.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/data/osdf/install-origin-rpm.md b/docs/data/osdf/install-origin-rpm.md index 7fadcfd39..4870e65f1 100644 --- a/docs/data/osdf/install-origin-rpm.md +++ b/docs/data/osdf/install-origin-rpm.md @@ -47,10 +47,11 @@ As with all OSG software installations, there are some one-time steps to prepare Origins should use a CA that is accepted by major browsers and operating systems, such as InCommon RSA or [Let's Encrypt](../../security/host-certs/lets-encrypt). IGTF certs are not recommended because clients are not configured to accept them by default. + Note that you will need the full certificate chain, not just the certificate. The following locations should be used (note that they are in separate directories): - * **Host Certificate**: `/etc/pki/tls/certs/pelican.crt` + * **Host Certificate Chain**: `/etc/pki/tls/certs/pelican.crt` * **Host Key**: `/etc/pki/tls/private/pelican.key` From 08e7249b8fdda74359fdb427f781c7a91efe6e68 Mon Sep 17 00:00:00 2001 From: Brian Lin Date: Mon, 28 Oct 2024 16:36:30 -0500 Subject: [PATCH 05/13] Apply changes from PR feedback (SOFTWARE-5962) --- docs/data/osdf/install-origin-rpm.md | 36 +++++----------------------- 1 file changed, 6 insertions(+), 30 deletions(-) diff --git a/docs/data/osdf/install-origin-rpm.md b/docs/data/osdf/install-origin-rpm.md index 4870e65f1..5c6cb3755 100644 --- a/docs/data/osdf/install-origin-rpm.md +++ b/docs/data/osdf/install-origin-rpm.md @@ -4,7 +4,7 @@ Installing the OSDF Origin ========================== This document describes how to install a Pelican-based Open Science Data Federation (OSDF) Origin service via RPMs. -This service allows an organization to export its data to the Data Federation. +This service allows an organization to export its data to the OSDF. !!! note The origin must be registered with the OSG prior to joining the data federation. @@ -12,10 +12,7 @@ This service allows an organization to export its data to the Data Federation. along with information like: * Resource name and hostname - * VO associated with this origin server (which will be used to determine the origin's namespace prefix) * Administrative and security contact(s) - * Who (or what) will be allowed to access the VO's data - * Which caches will be allowed to cache the VO data Before Starting @@ -29,7 +26,7 @@ Before starting the installation process, consider the following requirements: * __Network ports:__ The origin service requires the following ports open: * Inbound TCP port 8443 for file access via the HTTP(S) and XRoot protocols. * (Optional) Inbound TCP port 8444 for access to the web interface for monitoring and configuration; - if enabled, this should be restricted to the LAN or management network. + if enabled, consider restricting access from your LAN * __Hardware requirements:__ We recommend that an origin has at least 1Gbps connectivity and 12GB of RAM. We suggest that several gigabytes of local disk space be available for log files, although some logging verbosity can be reduced. @@ -80,7 +77,7 @@ XRootD: In addition, you must tell Pelican the data to export to the federation. An origin may export one or more directory trees, or one or more S3 buckets -- follow one of the sections below. -(An single origin cannot export both a bucket and a directory tree.) +A single origin cannot export both a bucket and a directory tree. @@ -98,36 +95,15 @@ Origin: Capabilities: # Add or remove as desired - Reads # Enable authenticated reading of objects from under the directory tree through a cache - PublicReads # Enable unauthenticated reading of objects from under the directory tree through a cache - - DirectReads # Enable reading objects from under the directory tree - # without going through a cache + - DirectReads # Enable reading objects from under the directory tree without going through a cache - Listings # Enable directory listings - Writes # Enable writing to files in the directory tree ``` ### Configuring S3 export -Set these options to export one or more S3 buckets to the federation - -``` -Origin: - StorageType: "s3" - S3Region: "" - S3ServiceUrl: "" - S3UrlStyle: "path" - Exports: - # You may have one or more of the following block: - - FederationPrefix: "" - S3Bucket: "" - S3AccessKeyfile: "" - S3SecretKeyfile: "" - Capabilities: # Add or remove as desired - - Reads # Enable authenticated reading of objects from the bucket through a cache - - PublicReads # Enable unauthenticated reading of objects from the bucket through a cache - - DirectReads # Enable reading objects from the bucket - # without going through a cache - - Listings # Enable listing bucket items - - Writes # Enable writing to objects in the bucket -``` +To configure your origin to serve objects from an S3 endpoint, see the +[upstream documentation](https://docs.pelicanplatform.org/federating-your-data/s3-backend). Preparing for Initial Startup From fd86f7a8bbc191bc2f7bb568a04e4202b7fec53a Mon Sep 17 00:00:00 2001 From: Brian Lin Date: Mon, 28 Oct 2024 17:24:19 -0500 Subject: [PATCH 06/13] Various improvements to the intro and requirements sections --- docs/data/osdf/install-origin-rpm.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/data/osdf/install-origin-rpm.md b/docs/data/osdf/install-origin-rpm.md index 5c6cb3755..c260b3e6e 100644 --- a/docs/data/osdf/install-origin-rpm.md +++ b/docs/data/osdf/install-origin-rpm.md @@ -1,10 +1,14 @@ title: Installing the OSDF Origin by RPM -Installing the OSDF Origin -========================== +Installing the OSDF Origin by RPM +================================= -This document describes how to install a Pelican-based Open Science Data Federation (OSDF) Origin service via RPMs. -This service allows an organization to export its data to the OSDF. +!!! warning "OSG 24" + This installation guide requires OSG 24 + +This document describes how to install an Open Science Data Federation (OSDF) Origin service via RPM. +This service, based on the [Pelican Platform](https://docs.pelicanplatform.org/federating-your-data), allows an +administrator to serve data from a POSIX filesystem or S3 endpoint through the global OSDF infrastructure. !!! note The origin must be registered with the OSG prior to joining the data federation. @@ -20,7 +24,7 @@ Before Starting Before starting the installation process, consider the following requirements: -* __Operating system:__ A RHEL 8 or RHEL 9 or compatible operating systems. +* __Operating system:__ A RHEL 8 or RHEL 9 or [compatible operating system](../../release/supported_platforms.md). * __User IDs:__ If they do not exist already, the installation will create the Linux user ID `xrootd` for running daemons. * __Host certificate:__ Required for authentication. See note below. * __Network ports:__ The origin service requires the following ports open: @@ -34,11 +38,8 @@ Before starting the installation process, consider the following requirements: As with all OSG software installations, there are some one-time steps to prepare in advance: * Obtain root access to the host -* Prepare [the required Yum repositories](../../common/yum.md) - - -!!! note "OSG 23" - In OSG 23, the Pelican-based OSDF RPMs are only available in the "osg-upcoming" repositories. +* Prepare [the required Yum repositories](../../common/yum.md), + including the [OSG 24 repositories](../../common/yum.md#install-the-osg-repositories) !!! note "Host certificates" Origins should use a CA that is accepted by major browsers and operating systems, From 80e50975723a84b8d1d58c2586ffd854c29674b0 Mon Sep 17 00:00:00 2001 From: Brian Lin Date: Mon, 28 Oct 2024 17:24:34 -0500 Subject: [PATCH 07/13] WIP upgrade outline --- docs/data/osdf/install-origin-rpm.md | 32 ++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/data/osdf/install-origin-rpm.md b/docs/data/osdf/install-origin-rpm.md index c260b3e6e..b0ca31c2e 100644 --- a/docs/data/osdf/install-origin-rpm.md +++ b/docs/data/osdf/install-origin-rpm.md @@ -52,6 +52,38 @@ As with all OSG software installations, there are some one-time steps to prepare * **Host Certificate Chain**: `/etc/pki/tls/certs/pelican.crt` * **Host Key**: `/etc/pki/tls/private/pelican.key` +Upgrading a Non-Pelican Origin +------------------------------ + +If you are running a non-Pelican origin, e.g. one that was installed before OSG 24, there are special consideratiosn for +the upgrade to ensure minimal downtime for your users. + +1. Verify that you are not already running a Pelican-based origin, run the following on your origin host: + + :::console + root@host # systemctl status osdf-origin + Unit osdf-origin.service could not be found. + + If you see the following, then you are not running a Pelican-based origin and should proceed with the rest of the + instructions in this section + +1. Install the origin + +1. Configure the origin + +1. Directly verify the origin + +1. Register the origin in the Director and Topology + + !!! danger "" + +1. Verify the origin through the OSDF director + +1. Uninstall the old service: + + :::console + root@host # yum remove stash-origin + Installing the Origin --------------------- From f110e00c30d4e89660603e18cd74d7bbf69d7352 Mon Sep 17 00:00:00 2001 From: Matyas Selmeci Date: Tue, 10 Dec 2024 14:00:34 -0600 Subject: [PATCH 08/13] Copy some of the language used in install-cache-rpm to install-origin-rpm --- docs/data/osdf/install-cache-rpm.md | 6 ++--- docs/data/osdf/install-origin-rpm.md | 33 +++++++++++++--------------- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/docs/data/osdf/install-cache-rpm.md b/docs/data/osdf/install-cache-rpm.md index 70d30960b..a521e5a2a 100644 --- a/docs/data/osdf/install-cache-rpm.md +++ b/docs/data/osdf/install-cache-rpm.md @@ -1,7 +1,7 @@ title: Installing the OSDF Cache by RPM -Installing the OSDF Cache -========================= +Installing the OSDF Cache by RPM +================================ This document describes how to install an Open Science Data Federation (OSDF) Cache service via RPMs. This service allows a site or regional network to cache data frequently used in Open Science Pool jobs, @@ -45,7 +45,7 @@ As with all OSG software installations, there are some one-time steps to prepare !!! note "Host certificates" Caches are accessed by users through browsers, meaning caches need a certificate from a CA acceptable to a standard browser. - Examples include [Let's Encrypt](../../security/host-certs/lets-encrypt.md) or the InCommon IGTF CA. + Examples include [Let's Encrypt](../../security/host-certs/lets-encrypt.md) or the InCommon RSA CA. Caches without a valid certificate for the browser cannot be added to the OSDF. Note that, unlike legacy grid software, the public certificate file will need to contain the "full chain", including any intermediate CAs (if you're unsure about your setup, try accessing your cache from your browser). diff --git a/docs/data/osdf/install-origin-rpm.md b/docs/data/osdf/install-origin-rpm.md index b0ca31c2e..742c963c6 100644 --- a/docs/data/osdf/install-origin-rpm.md +++ b/docs/data/osdf/install-origin-rpm.md @@ -6,18 +6,10 @@ Installing the OSDF Origin by RPM !!! warning "OSG 24" This installation guide requires OSG 24 -This document describes how to install an Open Science Data Federation (OSDF) Origin service via RPM. +This document describes how to install an Open Science Data Federation (OSDF) Origin service via RPMs. This service, based on the [Pelican Platform](https://docs.pelicanplatform.org/federating-your-data), allows an administrator to serve data from a POSIX filesystem or S3 endpoint through the global OSDF infrastructure. -!!! note - The origin must be registered with the OSG prior to joining the data federation. - You may start the registration process prior to finishing the installation by [using this link](#registering-the-origin) - along with information like: - - * Resource name and hostname - * Administrative and security contact(s) - Before Starting --------------- @@ -25,27 +17,32 @@ Before Starting Before starting the installation process, consider the following requirements: * __Operating system:__ A RHEL 8 or RHEL 9 or [compatible operating system](../../release/supported_platforms.md). -* __User IDs:__ If they do not exist already, the installation will create the Linux user ID `xrootd` for running daemons. +* __User IDs:__ If it does not exist already, the installation will create the Linux user named `xrootd` for running daemons. * __Host certificate:__ Required for authentication. See note below. * __Network ports:__ The origin service requires the following ports open: * Inbound TCP port 8443 for file access via the HTTP(S) and XRoot protocols. * (Optional) Inbound TCP port 8444 for access to the web interface for monitoring and configuration; - if enabled, consider restricting access from your LAN -* __Hardware requirements:__ We recommend that an origin has at least 1Gbps connectivity and 12GB of RAM. + if enabled, access to this port should be restricted to the LAN. +* __Service requirements:__ + * An origin in the OSDF should have at least: + * 1 core + * 1 Gbps connectivity + * 12 GB of RAM We suggest that several gigabytes of local disk space be available for log files, although some logging verbosity can be reduced. As with all OSG software installations, there are some one-time steps to prepare in advance: * Obtain root access to the host -* Prepare [the required Yum repositories](../../common/yum.md), - including the [OSG 24 repositories](../../common/yum.md#install-the-osg-repositories) +* Prepare [the required Yum repositories](../../common/yum.md); + the [OSG 24 repositories](../../common/yum.md#install-the-osg-repositories) should be used. !!! note "Host certificates" - Origins should use a CA that is accepted by major browsers and operating systems, - such as InCommon RSA or [Let's Encrypt](../../security/host-certs/lets-encrypt). - IGTF certs are not recommended because clients are not configured to accept them by default. - Note that you will need the full certificate chain, not just the certificate. + Origins are accessed by users through browsers, meaning origins need a certificate from a CA acceptable to a standard browser. + Examples include [Let's Encrypt](../../security/host-certs/lets-encrypt.md) or the InCommon RSA CA. + Origins without a valid certificate for the browser cannot be added to the OSDF. + Note that, unlike legacy grid software, the public certificate file will need to contain the "full chain", including any + intermediate CAs (if you're unsure about your setup, try accessing your origin from your browser). The following locations should be used (note that they are in separate directories): From 4405918007968211872b466d0d0a8bec22a1d037 Mon Sep 17 00:00:00 2001 From: Matyas Selmeci Date: Tue, 10 Dec 2024 14:05:38 -0600 Subject: [PATCH 09/13] Move "Upgrading a Non-Pelican Origin" section to the end and comment it out for now --- docs/data/osdf/install-origin-rpm.md | 66 +++++++++++++++------------- 1 file changed, 35 insertions(+), 31 deletions(-) diff --git a/docs/data/osdf/install-origin-rpm.md b/docs/data/osdf/install-origin-rpm.md index 742c963c6..f188f3dbd 100644 --- a/docs/data/osdf/install-origin-rpm.md +++ b/docs/data/osdf/install-origin-rpm.md @@ -49,37 +49,6 @@ As with all OSG software installations, there are some one-time steps to prepare * **Host Certificate Chain**: `/etc/pki/tls/certs/pelican.crt` * **Host Key**: `/etc/pki/tls/private/pelican.key` -Upgrading a Non-Pelican Origin ------------------------------- - -If you are running a non-Pelican origin, e.g. one that was installed before OSG 24, there are special consideratiosn for -the upgrade to ensure minimal downtime for your users. - -1. Verify that you are not already running a Pelican-based origin, run the following on your origin host: - - :::console - root@host # systemctl status osdf-origin - Unit osdf-origin.service could not be found. - - If you see the following, then you are not running a Pelican-based origin and should proceed with the rest of the - instructions in this section - -1. Install the origin - -1. Configure the origin - -1. Directly verify the origin - -1. Register the origin in the Director and Topology - - !!! danger "" - -1. Verify the origin through the OSDF director - -1. Uninstall the old service: - - :::console - root@host # yum remove stash-origin Installing the Origin @@ -154,6 +123,41 @@ before starting the origin for the first time, it is recommended to generate a k and what namespace(s) the origin will serve. OSG Staff will need to approve the origin's registration. + Managing the Origin Service --------------------------- From 3d9991f9e6aac3004d3e9fae18c115e0dc891461 Mon Sep 17 00:00:00 2001 From: Matyas Selmeci Date: Tue, 10 Dec 2024 15:21:15 -0600 Subject: [PATCH 10/13] Fix grammar --- docs/data/osdf/install-cache-rpm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/data/osdf/install-cache-rpm.md b/docs/data/osdf/install-cache-rpm.md index a521e5a2a..2afa04e36 100644 --- a/docs/data/osdf/install-cache-rpm.md +++ b/docs/data/osdf/install-cache-rpm.md @@ -93,7 +93,7 @@ Preparing for Initial Startup ----------------------------- 1. The cache identifies itself to the federation via public key authentication; -before starting the cache for the first time, it is generate a keypair. +before starting the cache for the first time, it is recommended to generate a keypair. :::console root@host$ cd /etc/pelican From 902d76e39c29c5b1c8553c8d83ba6eb2e57a7686 Mon Sep 17 00:00:00 2001 From: Matyas Selmeci Date: Tue, 10 Dec 2024 15:21:19 -0600 Subject: [PATCH 11/13] Update OSDF origin RPM installation doc to make it match the OSDF cache RPM installation doc in terms of structure and phrasing --- docs/data/osdf/install-origin-rpm.md | 81 ++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 16 deletions(-) diff --git a/docs/data/osdf/install-origin-rpm.md b/docs/data/osdf/install-origin-rpm.md index f188f3dbd..b6d494f22 100644 --- a/docs/data/osdf/install-origin-rpm.md +++ b/docs/data/osdf/install-origin-rpm.md @@ -58,23 +58,22 @@ The origin service is provided by the `osdf-origin` RPM. Install it using the following command: +OSG 24: ```console -root@host # yum install --enablerepo=osg-upcoming osdf-origin +root@host # yum install osdf-origin ``` +!!! note "osdf-cache 7.11.1" + This document covers versions 7.11.1 and later of the `osdf-cache` package; ensure the above installation + results in an appropriate version. + Configuring the Origin Server ----------------------------- -Configuration for a Pelican-based OSDF Origin is located in `/etc/pelican/osdf-origin.yaml`. +Create a file named `/etc/pelican/config.d/20-origin.yaml` -You must configure the following: -``` -XRootD: - Sitename: -``` - -In addition, you must tell Pelican the data to export to the federation. +You must tell Pelican the data to export to the federation. An origin may export one or more directory trees, or one or more S3 buckets -- follow one of the sections below. A single origin cannot export both a bucket and a directory tree. @@ -111,17 +110,67 @@ Preparing for Initial Startup 1. The origin identifies itself to the federation via public key authentication; before starting the origin for the first time, it is recommended to generate a keypair. - :::command - root@host$ cd /etc/pelican - root@host$ pelican generate keygen + :::console + root@host$ cd /etc/pelican + root@host$ pelican generate keygen The newly created files, `issuer.jwk` and `issuer-pub.jwks` are the private and public keys, respectively. - **Save these files**; if you lose them, you will have to re-register the origin. -1. Contact OSG Staff and let them know that you are about to start your origin, - and what namespace(s) the origin will serve. - OSG Staff will need to approve the origin's registration. +1. **Save these files**; if you lose the `issuer.jwk`, your origin will need to be re-approved. + + +Validating the Origin Installation +---------------------------------- + +Do the following steps to verify that the cache is functional: + +1. Start the origin using the following command: + + :::console + root@host$ systemctl start osdf-origin + +1. Download a test file (POSIX) or object (S3) from your origin (replacing `ORIGIN_HOSTNAME` with the host name of your origin, + and TEST_PATH with the OSDF path to the test file or object + + :::console + user@host$ curl -L https://ORIGIN_HOSTNAME:8443/TEST_PATH -o /tmp/testfile + + Verify the contents of `/tmp/testfile` match the test file or object your origin was serving. + + If the download fails, debugging information is located in `/var/log/pelican/osdf-origin.log`. + See [this page](../../common/help.md) for requesting assistance; please include the log file + in your request. + + +Joining the Origin to the Federation +------------------------------------ + +The origin must be registered with the OSG prior to joining the data federation. +Send mail to requesting registration; provide the following information: + +* Origin hostname +* Administrative and security contact(s) +* Institution that the cache belongs to + +OSG Staff will register the origin and respond with the Resource Name that the origin was registered as. + +Once you have that information, edit `/etc/pelican/config.d/15-osdf.yaml`, and set `XRootD.Sitename`: +``` +XRootD: + Sitename: +``` + +Then, restart the origin by running + +```console +root@host$ systemctl restart osdf-origin +``` + +Let OSG Staff know that you have restarted the origin with the updated sitename, +so they can approve the new origin. + +