Error checking out submodules due to non-existent remote

[bartgol]

On September 20th 2024, one of the submodules of E3SM changed its location. In particular, yaml-cpp (itself a submodule of EKAT), was originally forked and hosted in the SNLComputation organization. That organization was deleted, leaving the clones of e3sm broken. We recreated a fork of yaml-cpp in the E3SM-Project organization, and changed master (as well as the v3 maint branch) to use that remote. However, old sha's of E3SM that have EKAT as a submodule will NOT correctly check out. To mitigate this, we resurrected the SNLComputation organization, but changed the name (to avoid misleading the public to believe that Sandia National Labs owned it or was somewhat responsible for its content). Due to GitHub features, cloning SNLComputation/yaml-cpp will still work for some time, but GitHub does not guarantee for how long.

If you are encounter an error that looks like the following:

ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:SNLComputation/yaml-cpp.git' into submodule path '/home/circleci/project/externals/ekat/extern/yaml-cpp' failed

it may mean that the redirect SNLComputation->E3SMLegacyRepos is no longer supported by GitHub. Fortunately, you can easily fix the error. First of all, it may be that you don't really need a submodule, in which case you can locally tell git to ignore it:

$ git submodule deinit path/to/submodule

In the case of yaml-cpp, it is used inside ekat, which is both a submodule of e3sm itself, as well as a submodule of Haero. So you would have to deinit ekat and haero. Another approach is to run the following:

git -c submodule.extern/yaml-cpp.update=none submodule  update --init --recursive

which updates all the submodules, but sets a temporary rule (for this run only) to skip the yaml-cpp submodule when running the submodule update command.

On the other hand, if you do need EKAT, the simplest solution is to rebase your local branch on master (or the v3 maint branch). If that's not possible, you can take the long path... You need to become familiar with the following set of steps, for submodule X

$ cd path/to/submodule
$ git remote set-url origin git@github.com:new-org-or-user/repo-name # if you prefer https, you can use that instead
$ git remote -v # to verify the remote is now set correctly
$ cd -
$ git checkout -b update-X-remote
$ git add /path/to/submodule
$ git commit -m "Update remote for X"
$ git config -f .gitmodules submodule.X.url <new-repo-url>
$ git add .gitmodules
$ git commit -m "Updated submodule URL in .gitmodules"
$ git push origin update-X-remote

This step, which we will refer to as "change-X-remote, needs to be done inside every repo where X is a submodule. This means that if X is a submodule of E3SM, you run this from main E3SM repo, but if X is a submodule of Y (another submodule) you need to run change-X-remote from inside submodule Y, and then update submodule Y in the main repo. This is where it gets tricky: if you need to change remote of X inside submodule Y, you need to be able to push to Y's remote. If that's not possible (you don't have write permissions in Y's remote), you first need to run change-Y-remote inside E3SM.

Back to yaml-cpp, right now it appears as a submodule of EKAT, which (as of Sept 2024) is a submodule of E3SM, as well as a submodule of Haero. Assuming you do not have write permissions in the Ekat and Haero repos, you need to:

1. fork Ekat and Haero in your user/org
2. do the "change-Ekat-remote" and "change-Haero-remote" step in E3SM
3. from externals/ekat, do the "change-yaml-cpp-remote" step
4. from externals/haero, do the "change-ekat-remote" step
5. update haero to make its ekat submodule be the correct branch from your fork:

$ cd externals/haero
$ cd /path/to/ekat/submodule/inside/haero
$ git checkout origin/update-yaml-cpp-remote
$ cd -
$ git add /path/to/ekat/submodule/in/haero
$ git checkout -b update-ekat-submodule
$ git commit -m "Update ekat submodule"
$ git push origin update-ekat-submodule

6. update e3sm to make its ekat and haero submodules point to the branch from your forks

$ cd externals/haero
$ git checkout origin/update-ekat-submodule
$ cd -
$ cd externals/ekat
$ git checkout origin/change-yaml-cpp-remote
$ cd -
$ git add externals/ekat externals/haero
$ git commit -m "Update ekat and haero branches"

As you can see, changing the remotes takes quite a few steps, so we STRONGLY recommend that you simply rebase on master (or v3 maint).