Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Director based cache health tests #960

Merged
merged 25 commits into from
Mar 26, 2024

Conversation

haoming29
Copy link
Contributor

@haoming29 haoming29 commented Mar 18, 2024

Closes #545

This PR has two major changes. One is to add unauthenticated director-based health tests for caches. Second is to have a large overhaul of project structure to 1. minimize chances for cyclic import. 2. Establish a convention on internal dependencies and naming practice.

The first task happens over the most of the commits. The second task happens in 3c17dd8, dd13fc3, 8192935, 952229b, a8dce19, 4f444ba, where 4f444ba should contain most of the changes to the whole project while others are small changes to a specific directory.

The task two implemented proposed ideas in Pelican Folder Structure v7.7.0, including:

  • Renamed cache_ui --> cache, origin_ui --> origin
  • Renamed common --> server_structs
  • Renamed director.go --> director_ui.go, redirect.go --> director in director package
  • Renamed server_ui --> launcher_utils
  • Removed cache dependency on director by moving NamespaceV1/V2 conversion functions from director to server_structs (old name common)
  • Moved XRootDServer structs from server_utils to server_structs
  • Moved origin export related code from common to server_utils

@haoming29 haoming29 added this to the v7.7.0 milestone Mar 18, 2024
@haoming29 haoming29 added enhancement New feature or request cache Issue relating to the cache component labels Mar 18, 2024
director/cache_monitor.go Dismissed Show dismissed Hide dismissed
director/monitor.go Dismissed Show dismissed Hide dismissed
@haoming29 haoming29 marked this pull request as ready for review March 18, 2024 16:00
@haoming29 haoming29 requested a review from jhiemstrawisc March 18, 2024 16:42
cache/cache.go Outdated Show resolved Hide resolved
cache/cache.go Outdated
return nil
}
for idx, item := range dirItems {
if idx <= len(dirItems)-1-2 { // For all but the latest two files (test file and its .cinfo file)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, do we have some underlying guarantee that this slice is ordered? Or how do we know we're leaving the "right" two files? Also, the logic here might be a more clear at first glance if you do if idx < len(dirItems)-2 to get rid of the extra -1.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReadDir sorts the dirEntry in order. The filenames are in format of cache-test-<date>.txt where date is a comparable string, so here I'm looking for the last two files, which have the largest date as its filename (or the latest files). I'll add a comment here to explain a bit

cmd/origin.go Show resolved Hide resolved
cmd/plugin_stage.go Show resolved Hide resolved
director/cache_monitor.go Show resolved Hide resolved
director/monitor.go Outdated Show resolved Hide resolved
director/monitor.go Show resolved Hide resolved
director/monitor.go Show resolved Hide resolved
origin/origin.go Outdated Show resolved Hide resolved
origin/origin.go Outdated Show resolved Hide resolved
@haoming29 haoming29 requested a review from jhiemstrawisc March 26, 2024 15:45
Copy link
Member

@jhiemstrawisc jhiemstrawisc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jhiemstrawisc jhiemstrawisc merged commit e60b5d3 into PelicanPlatform:main Mar 26, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cache Issue relating to the cache component enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Periodic director-based health tests for cache
2 participants