Fix output_root #855

stcui007 · 2024-07-17T13:57:49Z

This PR fixes the problem of directory that output_root pointed to does not exist. If this is the case, the code creates a sub-directory based in the name given by the output_root in the realization. The code defaults to writing the outputs to the project directory if output_root does not exist as in the current realization.

This is an attempt to solve the problem raised in issues #857 and #848, where large number of output files are written. This is a common problem for running ngen on any large basin.

Additions

Removals

Changes

get_output_root() function and unit test.
Two realizations with output_root for demonstration. One realization in its original form, i.e., w/o output_root for demonstration

Testing

Local run ngen in serial and parallel modes, and unit test

Screenshots

Notes

Todos

Checklist

Testing checklist (automated report can be put here)

Target Environment support

Linux

include/realizations/catchment/Formulation_Manager.hpp

aaraney · 2024-07-17T20:26:46Z

include/realizations/catchment/Formulation_Manager.hpp

                const auto output_root = this->tree.get_optional<std::string>("output_root");
+                std::string str;


@stcui007, I think this should fix your issue.

Suggested change

std::string str;

std::string str = "./";

You mean the failing tests or something else. I don't want to write to "./" though.

I'm fairly certain the call to stat will fail if you provide it the empty string like you are right now in the case when output_root is not present. I think that is why the Build_and_Run_Model action is failing.

I think you should return "./" if output_root is not present (boost::none) like in the previous code. We can always assume that the directory ngen is run from, ./ , exists.

One solution is:

if (output_root != boost::none && *output_root != "") { // Check if the path ends with a trailing slash, // otherwise add it. output_root->back() == '/' ? *output_root : *output_root + "/"; //use C system function to check if there is a dir match that defined in realization struct stat sb; if (stat(output_root->c_str(), &sb) == 0 && S_ISDIR(sb.st_mode)) return *output_root; else { throw std::runtime_error("output_root directory does not exist, please create one matching that in realization"); } } return "./";

Aside, I would think we would want to check that the calculated output_root can be written to by the current user. That is out of the scope of this PR and should be handled elsewhere if deemed necessary.

(1) It looks like this code would return "./" if output_root directory does not exist.
(2) This code would seem not get a chance to throw based on the outer if, which is not what we want.
(3) This block of code is a bit tricky:

output_root->back() == '/' ? *output_root : *output_root + "/";

where does the value land?
(4) The failed test looks like come from this line in Example_model_run.yml:
./ngen data/catchment_data.geojson '' data/nexus_data.geojson '' data/example_realization_config.json
It takes realization file from the master, not from the PR, right?
(5) In the latest realization file, I set output_root to "./", so it automatically exists.

This code handles the output root for ngen, so if there is one in a realization config file under the key, output_root, that one will be used, otherwise, the default is to write to the directory where ngen was called, "./".

I don't think you pushed an updated data/example_realization_config.json file. Looks like you pushed up data/example_bmi_multi_realization_config.json and data/example_bmi_multi_realization_config_w_noah_pet_cfe.json.

program-- · 2024-07-18T20:15:28Z

Based on the comments, I think this PR is partially solving #848 but needs a bit more discussion, since it feels like an XY problem.

There's 2 separate issues here:

(1) output_root fails to write outputs if the specified path is invalid (doesn't exist, isn't writable, etc.) as outlined in #848 -- There are two approaches to solving this I think: either try to create the directory path (i.e. emulate mkdir -p) if possible, or always throw an error and have the user create the directory. This PR currently performs the latter via stat.

(2) If no output_root is specified, potentially lots of files will be outputted into the project directory since output_root defaults to ./, as outlined in #857. I don't think this is actually an issue as the solution is to just specify an output_root in the config file.

I think the question still unanswered for this PR is: do we want output_root to be optional or required? Or rather, if output_root has a non-existent path, should we try to make the directories or should we always throw an error?

stcui007 · 2024-07-18T20:43:06Z

The issue is, if we can still call it an issue, if someone runs a big job and forgot to create the directory or entered wrong format in realization, then when the job finishes, he ends up having a lot of problems. This PR tries to stop that happening at the start. It seems to me in the long run, requireing an output_root doesn't have any downside. The decision needs to be made by Nels. Not sure if there is any ripple effect to calibration if we change realization. Maybe as you alluded to in (1). We could create a directory programmatically.

…

On Thu, Jul 18, 2024 at 3:15 PM Justin Singh-M. - NOAA < ***@***.***> wrote: Based on the comments, I think this PR is partially solving #848 <#848> but needs a bit more discussion, since it feels like an XY problem. There's 2 separate issues here: (1) output_root fails to write outputs if the specified path is invalid (doesn't exist, isn't writable, etc.) as outlined in #848 <#848> -- There are two approaches to solving this I think: either create the directory path (i.e. emulate mkdir -p) if possible, or throw an error. This PR currently performs the latter via stat. (2) If no output_root is specified, potentially lots of files will be outputted into the project directory since output_root defaults to ./, as outline in #857 <#857>. I don't think this is actually an issue as the solution is to just specify an output_root in the config file. I think the real question for this PR as is: do we want output_root to be *optional* or *required*? Or rather, if output_root has an invalid path, should we try to make the directories or should we always throw an error? — Reply to this email directly, view it on GitHub <#855 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA4SRK325UA36A37C3532TZNAO7NAVCNFSM6AAAAABLAWMQD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZXGQ4DINZYGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

aaraney · 2024-07-18T20:44:11Z

Great, summary @program--. I share the same feelings.

I think the real question for this PR is: do we want output_root to be optional or required? Or rather, if output_root has a non-existent path, should we try to make the directories or should we always throw an error?

My preference would be that output_root stayed optional. For instance in calibration, we take advantage of the absence of output_root so that the output is written to where ever we call ngen from. Right, of course we could add functionality to account for this, but this is just my personal preference.

Likewise, if possible, emulating mkdir -p is desirable IMO. I would expect that a program would create the output directory I specified if it didn't already exist. For sure a matter of option, so I am going to ping @PhilMiller, @hellkite500, and @donaldwj to get their thoughts as well.

From a practicality stance, we are already including sys/stat.h, so we have access to mkdir but there are likely other issues that I might not be considering.

stcui007 · 2024-07-18T20:47:11Z

@austin Raney - NOAA Affiliate ***@***.***> You are right, I didn't. I thought that test used the multiple bmi realization. On Thu, Jul 18, 2024 at 3:42 PM Shengting Cui - NOAA Affiliate < ***@***.***> wrote:

…

The issue is, if we can still call it an issue, if someone runs a big job and forgot to create the directory or entered wrong format in realization, then when the job finishes, he ends up having a lot of problems. This PR tries to stop that happening at the start. It seems to me in the long run, requireing an output_root doesn't have any downside. The decision needs to be made by Nels. Not sure if there is any ripple effect to calibration if we change realization. Maybe as you alluded to in (1). We could create a directory programmatically. On Thu, Jul 18, 2024 at 3:15 PM Justin Singh-M. - NOAA < ***@***.***> wrote: > Based on the comments, I think this PR is partially solving #848 > <#848> but needs a bit more > discussion, since it feels like an XY problem. > > There's 2 separate issues here: > > (1) output_root fails to write outputs if the specified path is invalid > (doesn't exist, isn't writable, etc.) as outlined in #848 > <#848> -- There are two > approaches to solving this I think: either create the directory path (i.e. > emulate mkdir -p) if possible, or throw an error. This PR currently > performs the latter via stat. > > (2) If no output_root is specified, potentially lots of files will be > outputted into the project directory since output_root defaults to ./, > as outline in #857 <#857>. I > don't think this is actually an issue as the solution is to just specify an > output_root in the config file. > > I think the real question for this PR as is: do we want output_root to > be *optional* or *required*? Or rather, if output_root has an invalid > path, should we try to make the directories or should we always throw an > error? > > — > Reply to this email directly, view it on GitHub > <#855 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACA4SRK325UA36A37C3532TZNAO7NAVCNFSM6AAAAABLAWMQD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZXGQ4DINZYGM> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

aaraney · 2024-07-18T20:47:14Z

The issue is, if we can still call it an issue, if someone runs a big job and forgot to create the directory or entered wrong format in realization, then when the job finishes, he ends up having a lot of problems. This PR tries to stop that happening at the start.

Yeah, that's a good point. Like you have started to do in this PR, transitioning our examples in data to use output_root I think is a great middle ground. Thoughts?

aaraney · 2024-07-18T20:48:39Z

@stcui007, no worries!

stcui007 · 2024-07-18T20:59:00Z

@austin Raney - NOAA Affiliate ***@***.***> I assume you know quite bit about calibration. So changing the Examples would not affect them? We probably need to ask Nels. At the same time, we need to carefully check our code so it doesn't have effect on that.

…

On Thu, Jul 18, 2024 at 3:49 PM Austin Raney ***@***.***> wrote: @stcui007 <https://github.com/stcui007>, no worries! — Reply to this email directly, view it on GitHub <#855 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA4SROKIXCXHVUDP2IGE4LZNAS35AVCNFSM6AAAAABLAWMQD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZXGU2DSNJSGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

include/realizations/catchment/Formulation_Manager.hpp

stcui007 · 2024-07-23T12:52:26Z

This link: https://stackoverflow.com/questions/55994752/adding-multiple-directories-to-mkdir

…

On Mon, Jul 22, 2024 at 6:02 PM Justin Singh-M. - NOAA < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In include/realizations/catchment/Formulation_Manager.hpp <#855 (comment)>: > + const char* dir = str.c_str(); + + //use C++ system function to check if there is a dir match that defined in realization + struct stat sb; + if (stat(dir, &sb) == 0 && S_ISDIR(sb.st_mode)) + return dir; + else { + //throw std::runtime_error("output_root directory does not exist, please create one matching that in realization"); + mkdir(dir, 0755); For (1), as long as we throw an error, that's fine. For (2), where are you seeing that mkdir can create multi-level directories? This example results in an error: #include <stdio.h>#include <string.h>#include <sys/stat.h>#include <errno.h>#include <unistd.h> void test_path(const char* path) { errno = 0; int result_code = mkdir(path, 0775); if (result_code == -1) printf("[%s] error: %s\n", path, strerror(errno)); unlink(path); } int main(int argc, const char* argv[]) { // This passes (unless current directory is not writable) test_path("example_path"); // This fails unless 'example_nested_path/' is created test_path("example_nested_path/subdir"); // This fails unless '/opt/ngen/outputs/' is created test_path("/opt/ngen/outputs/22July2024_1"); return 0; } For your second point in (2), the purpose is for consumers. It's a reasonable assumption that if output_root is set to a directory, and it can create a top-level directory, then it should be able to create multi-level directories. However, worst-case, as long as we're throwing the errors, then the consumer will be informed. — Reply to this email directly, view it on GitHub <#855 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA4SRMW7TIPOHRKWFHJCBTZNWFO5AVCNFSM6AAAAABLAWMQD6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCOJSGY3TSNBVGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

stcui007 · 2024-07-23T13:33:07Z

A more careful reading of the site suggests it might have to be done recursively. On Tue, Jul 23, 2024 at 7:52 AM Shengting Cui - NOAA Affiliate < ***@***.***> wrote:

…

This link: https://stackoverflow.com/questions/55994752/adding-multiple-directories-to-mkdir On Mon, Jul 22, 2024 at 6:02 PM Justin Singh-M. - NOAA < ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In include/realizations/catchment/Formulation_Manager.hpp > <#855 (comment)>: > > > > + const char* dir = str.c_str(); > + > + //use C++ system function to check if there is a dir match that defined in realization > + struct stat sb; > + if (stat(dir, &sb) == 0 && S_ISDIR(sb.st_mode)) > + return dir; > + else { > + //throw std::runtime_error("output_root directory does not exist, please create one matching that in realization"); > + mkdir(dir, 0755); > > For (1), as long as we throw an error, that's fine. > > For (2), where are you seeing that mkdir can create multi-level > directories? > > This example results in an error: > > #include <stdio.h>#include <string.h>#include <sys/stat.h>#include <errno.h>#include <unistd.h> > void test_path(const char* path) { > errno = 0; > int result_code = mkdir(path, 0775); > if (result_code == -1) > printf("[%s] error: %s\n", path, strerror(errno)); > > unlink(path); > } > int main(int argc, const char* argv[]) { > // This passes (unless current directory is not writable) > test_path("example_path"); > > // This fails unless 'example_nested_path/' is created > test_path("example_nested_path/subdir"); > > // This fails unless '/opt/ngen/outputs/' is created > test_path("/opt/ngen/outputs/22July2024_1"); > return 0; > } > > For your second point in (2), the purpose is for consumers. It's a > reasonable assumption that if output_root is set to a directory, and it > can create a top-level directory, then it should be able to create > multi-level directories. However, worst-case, as long as we're throwing the > errors, then the consumer will be informed. > > — > Reply to this email directly, view it on GitHub > <#855 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACA4SRMW7TIPOHRKWFHJCBTZNWFO5AVCNFSM6AAAAABLAWMQD6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCOJSGY3TSNBVGI> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

stcui007 · 2024-07-23T14:59:08Z

"However, worst-case, as long as we're throwing the errors, then the consumer will be informed." I agree. If users are warned, then create a multi-level directory manually, it should always work. On Tue, Jul 23, 2024 at 8:32 AM Shengting Cui - NOAA Affiliate < ***@***.***> wrote:

…

A more careful reading of the site suggests it might have to be done recursively. On Tue, Jul 23, 2024 at 7:52 AM Shengting Cui - NOAA Affiliate < ***@***.***> wrote: > This link: > https://stackoverflow.com/questions/55994752/adding-multiple-directories-to-mkdir > > On Mon, Jul 22, 2024 at 6:02 PM Justin Singh-M. - NOAA < > ***@***.***> wrote: > >> ***@***.**** commented on this pull request. >> ------------------------------ >> >> In include/realizations/catchment/Formulation_Manager.hpp >> <#855 (comment)>: >> >> > >> + const char* dir = str.c_str(); >> + >> + //use C++ system function to check if there is a dir match that defined in realization >> + struct stat sb; >> + if (stat(dir, &sb) == 0 && S_ISDIR(sb.st_mode)) >> + return dir; >> + else { >> + //throw std::runtime_error("output_root directory does not exist, please create one matching that in realization"); >> + mkdir(dir, 0755); >> >> For (1), as long as we throw an error, that's fine. >> >> For (2), where are you seeing that mkdir can create multi-level >> directories? >> >> This example results in an error: >> >> #include <stdio.h>#include <string.h>#include <sys/stat.h>#include <errno.h>#include <unistd.h> >> void test_path(const char* path) { >> errno = 0; >> int result_code = mkdir(path, 0775); >> if (result_code == -1) >> printf("[%s] error: %s\n", path, strerror(errno)); >> >> unlink(path); >> } >> int main(int argc, const char* argv[]) { >> // This passes (unless current directory is not writable) >> test_path("example_path"); >> >> // This fails unless 'example_nested_path/' is created >> test_path("example_nested_path/subdir"); >> >> // This fails unless '/opt/ngen/outputs/' is created >> test_path("/opt/ngen/outputs/22July2024_1"); >> return 0; >> } >> >> For your second point in (2), the purpose is for consumers. It's a >> reasonable assumption that if output_root is set to a directory, and it >> can create a top-level directory, then it should be able to create >> multi-level directories. However, worst-case, as long as we're throwing the >> errors, then the consumer will be informed. >> >> — >> Reply to this email directly, view it on GitHub >> <#855 (comment)>, or >> unsubscribe >> <https://github.com/notifications/unsubscribe-auth/ACA4SRMW7TIPOHRKWFHJCBTZNWFO5AVCNFSM6AAAAABLAWMQD6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCOJSGY3TSNBVGI> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> >

include/realizations/catchment/Formulation_Manager.hpp

program--

@aaraney If you could do a final review just to see if this satisfies #848 that'd be great! Otherwise, this looks just about done

stcui007 · 2024-07-23T21:22:11Z

Actually, it soesn't make any difference.

…

On Tue, Jul 23, 2024 at 4:10 PM Justin Singh-M. - NOAA < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In include/realizations/catchment/Formulation_Manager.hpp <#855 (comment)>: > + const char* dir = str.c_str(); + + //use C++ system function to check if there is a dir match that defined in realization + struct stat sb; + if (stat(dir, &sb) == 0 && S_ISDIR(sb.st_mode)) + return dir; + else { + int result = mkdir(dir, 0755); + if (result == 0) + return dir; + else + throw std::runtime_error("mkdir " + std::string(dir) + " failed, please create " + std::string(dir)); Oops, I think it actually should be std::strerror <https://en.cppreference.com/w/cpp/string/byte/strerror>, my bad. — Reply to this email directly, view it on GitHub <#855 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA4SRKIONXSXU4BSJLH54TZN3BF5AVCNFSM6AAAAABLAWMQD6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCOJVGA4TQMBSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

stcui007 · 2024-07-24T14:07:35Z

Changed the included header.

include/realizations/catchment/Formulation_Manager.hpp

PhilMiller · 2024-07-24T18:54:59Z

The logic looks fine to me.

my comments were resolved

hellkite500 · 2024-07-25T01:59:59Z

Looks like this adds some reasonable QoL logic to help with output management. Thanks!

stcui007 added 2 commits July 17, 2024 13:29

Fix output_root

900a517

Add output_root dir in realization

a731e62

PhilMiller reviewed Jul 17, 2024

View reviewed changes

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

program-- reviewed Jul 17, 2024

View reviewed changes

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

stcui007 added 2 commits July 17, 2024 15:34

Incorporate reviewer's suggestion

c829a5b

Revision after reviewer's comments

549628b

aaraney reviewed Jul 17, 2024

View reviewed changes

Using a placeholder output_root dir

9fcdbed

stcui007 added 3 commits July 18, 2024 21:15

Add example_realization_config.json

fbfc39c

Create the directory in code instead of manually

ed97ff8

Generalize to include the case no output_root in realization

79853a8

stcui007 force-pushed the output_root branch from 3af9b0e to 79853a8 Compare July 22, 2024 15:20

stcui007 added 2 commits July 22, 2024 15:56

Remove the modified realization files

74160a6

Add back the realization files w/ or w/o output_root

35f90b8

stcui007 requested review from program--, PhilMiller, aaraney and hellkite500 July 22, 2024 17:00

stcui007 marked this pull request as ready for review July 22, 2024 17:03

program-- previously requested changes Jul 22, 2024

View reviewed changes

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

program-- reviewed Jul 22, 2024

View reviewed changes

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

First pass to address reviewer's comments

1342d2b

program-- reviewed Jul 23, 2024

View reviewed changes

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

Revise throw message

c372080

program-- reviewed Jul 23, 2024

View reviewed changes

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

program-- reviewed Jul 23, 2024

View reviewed changes

stcui007 added 2 commits July 23, 2024 21:33

Minor revision

14a3798

Change a C to C++ header

6b5be12

PhilMiller reviewed Jul 24, 2024

View reviewed changes

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

PhilMiller reviewed Jul 24, 2024

View reviewed changes

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

PhilMiller reviewed Jul 24, 2024

View reviewed changes

include/realizations/catchment/Formulation_Manager.hpp Outdated Show resolved Hide resolved

Incorporate reviewer's suggestions

f0bd6c7

hellkite500 approved these changes Jul 25, 2024

View reviewed changes

PhilMiller approved these changes Jul 25, 2024

View reviewed changes

PhilMiller merged commit f15b0b1 into NOAA-OWP:master Jul 30, 2024
19 checks passed

stcui007 mentioned this pull request Aug 7, 2024

No output written if output_root is specified in realization config and does not exist or is a file #848

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix output_root #855

Fix output_root #855

stcui007 commented Jul 17, 2024 •

edited by PhilMiller

Loading

aaraney Jul 17, 2024 •

edited

Loading

stcui007 Jul 17, 2024

aaraney Jul 18, 2024

stcui007 Jul 18, 2024

aaraney Jul 18, 2024

program-- commented Jul 18, 2024 •

edited

Loading

stcui007 commented Jul 18, 2024 via email

aaraney commented Jul 18, 2024

stcui007 commented Jul 18, 2024 via email

aaraney commented Jul 18, 2024

aaraney commented Jul 18, 2024

stcui007 commented Jul 18, 2024 via email

stcui007 commented Jul 23, 2024 via email

stcui007 commented Jul 23, 2024 via email

stcui007 commented Jul 23, 2024 via email •

edited

Loading

program-- left a comment

stcui007 commented Jul 23, 2024 via email •

edited

Loading

stcui007 commented Jul 24, 2024

PhilMiller commented Jul 24, 2024

hellkite500 commented Jul 25, 2024

		const auto output_root = this->tree.get_optional<std::string>("output_root");
		std::string str;

Fix output_root #855

Fix output_root #855

Conversation

stcui007 commented Jul 17, 2024 • edited by PhilMiller Loading

Additions

Removals

Changes

Testing

Screenshots

Notes

Todos

Checklist

Testing checklist (automated report can be put here)

Target Environment support

aaraney Jul 17, 2024 • edited Loading

Choose a reason for hiding this comment

stcui007 Jul 17, 2024

Choose a reason for hiding this comment

aaraney Jul 18, 2024

Choose a reason for hiding this comment

stcui007 Jul 18, 2024

Choose a reason for hiding this comment

aaraney Jul 18, 2024

Choose a reason for hiding this comment

program-- commented Jul 18, 2024 • edited Loading

stcui007 commented Jul 18, 2024 via email

aaraney commented Jul 18, 2024

stcui007 commented Jul 18, 2024 via email

aaraney commented Jul 18, 2024

aaraney commented Jul 18, 2024

stcui007 commented Jul 18, 2024 via email

stcui007 commented Jul 23, 2024 via email

stcui007 commented Jul 23, 2024 via email

stcui007 commented Jul 23, 2024 via email • edited Loading

program-- left a comment

Choose a reason for hiding this comment

stcui007 commented Jul 23, 2024 via email • edited Loading

stcui007 commented Jul 24, 2024

PhilMiller commented Jul 24, 2024

hellkite500 commented Jul 25, 2024

stcui007 commented Jul 17, 2024 •

edited by PhilMiller

Loading

aaraney Jul 17, 2024 •

edited

Loading

program-- commented Jul 18, 2024 •

edited

Loading

stcui007 commented Jul 23, 2024 via email •

edited

Loading

stcui007 commented Jul 23, 2024 via email •

edited

Loading