Typical project workflow and setup
This document will provide information to help get started using the new MPAS git repository.
To begin, the image below gives a visual representation of the topography of the repositories and forks. In addition, it provides some examples for creating a feature branch.
This branching strategy is important, as it will make working in the repositories easier. The reason work is not carried out in master (or in a branch of master) is because master will be tied to the release repository eventually.
All work should be carried out in a branch. Within the shared MPAS-Dev/MPAS repository, we have the following "permanent" branches:
- master
- develop
- atmosphere/develop
- landice/develop
- ocean/develop
- ocean/private
The master branch is always synchronized with the MPAS-Dev/MPAS-Release repository, to create public releases.
The develop branch represents the most up-to-date version of framework, and shared models (like the shallow water model). All code on develop will be released during the next release of MPAS. All framework and operators branches should branch off of develop, with the intent of merging back into develop.
Each core has their own develop branch. These branches represent the most up-to-date version of each component model. In order for a core to release, they must merge their core/develop branch into the develop branch prior to a release. Feature branches that modify code within the src/core_${CORE} directory should branch off of core/develop with the intent of merging back into core/develop.
Each core is also allowed to have a core/next or core/private branch to represent a version of the model that can include features that are not yet releasable, for testing.
In order to begin a new project, the first step is to create a fork of the main remote MPAS development repository. To create a fork, simply visit the github page for the repository of interest (e.g. MPAS) and click the fork button in the top right hand part of the web page.
Pressing the fork button causes github to create a personal copy of the original repository. You gain elevated permissions over the fork, in the sense that you can read, write, and administer this fork. The fork contains all branches that existed in the original repository at the time of creation. It also contains all of the history that was present in the original repository when the fork was created.
If you visit the page for the fork, it will give you the address for the remote repository. For example, my personal fork of the main git@github.com:MPAS-Dev/MPAS.git repository would be git@github.com:douglasjacobsen/MPAS.git.
Your fork is your main conduit for contributions to the main developer repository.
Once you have the fork created, and the address for the specific for (we'll use git@github.com:douglasjacobsen/MPAS.git in this example)
To being working, you must create a local repository which is a clone of the remote repository. This is done through the following command:
git clone git@github.com:douglasjacobsen/MPAS.git
When this command finishes, a new directory is created called MPAS. This directory is a copy of the remote repository. It contains a single branch, called develop. This develop branch is (by default) setup to track with the remote branch named develop that lives in your fork.
In addition to a develop branch, the git clone command sets up an alias named origin. origin in this case points to the remote repository that you cloned. It is used in several commands as a means of communicating with the remote repository.
As per the branching strategy at the beginning of this page, master is reserved to releases. Develop is the name for the main development branch. New "feature branches" should be created from origin/develop. Before describing how to create a branch, we will talk about choosing a name for a branch.
It's very important to create descriptive and useful branch names, so when other developers need to find related projects they can do so easily. For example, if you were working on a project to implement a new history attribute in output files, this would largely affect the shared framework. Because of this, a good name for the branch would be
framework/history_attribute
This way, other developers can search for branches with the name framework/ to see which branches affect framework. A template for the name of branches could be:
directory_name/project_name
Where directory_name refers to the name of the directory under src/ the majority of the work will take place in. NOTE: Although this appears to setup some sort of directory structure, this directory structure is not actually present in the repository.
There are several ways to setup this example branch. Each has it's own pros and cons. The easiest and most useful way, would be the following:
git branch framework/history_attribute origin/develop
The specifics of this command will hopefully be understandable by the end of this document. But as a brief description, this creates a local branch named framework/history_attribute that branches off of the current state of the remote branch named develop in the remote repository origin.
When creating a branch, it's important to branch off of the correct place. This will make merging and rebasing easier in the future.
Although the branch has been created, the current working branch as not been changed yet. In order to start working on the new branch, the following command can be used:
git checkout framework/history_attribute
While git checkout seems to act different to svn checkout, it actually is quite similar. In svn checkout, you are simply telling svn to pull some version of the remote repository into your local working directory. While in git checkout, you are simply telling git to pull some version of the local repository into your working directory.
In order to work with branches, you first need to find out what branch you're working on. Typing
git branch
lists all local branches, and places a star next to the one you are currently working on.
This can be expanded with a -a flag to list all branches that exists locally, or on remote repositories you are connected with.
As explained earlier, the git checkout command is used to change which branch you are working on. For example:
git checkout develop
Will change the current working directory to be on the local branch named master.
Changing the branch changes the contents and state of the working directory to be that of the branch you are switching to. Although it will not be explained in this document, this script can be used to create multiple working directories from a single clone of a git repository. This can be useful in testing multiple branches at the same time, for example comparing a run from the master branch with a run from a feature branch.
As you work on your branch, you can choose to stage any of the changes you make and commit them to your local repository. Files can be staged in a few different ways, but only the most basic methods will be covered here. Staging modifications can be done through
git add filename
This is done on any file that has been modified since the previous commit. Once the changes are staged, they can be reviewed with:
git status
To see which files contain changes that will be committed to the local repository. To commit these changes to the local repository, one can issue a:
git commit
Which will use the $EDITOR variable to edit the commit message.
Alternatively, the git add and git commit commands can be combined into:
git commit -a
Which will stage and commit all modifications to currently tracked files. This will start tracking new files.
###Please Read all about pulling, pushing, fetching, merging, and rebasing before doing any of them
The only way to propose changes to the main developer repository, and the easiest way to share changes with other developers is through the use of your fork, or remote repository. One of the methods of communicating with a remote repository is through the git push command.
git push takes the branch you are currently working on an pushes the changes to a remote branch on a remote repository. If the remote branch does not exist, it will be created.
As arguments, git push requires the alias to a remote repository, and the name of the branch on the remote repository. As an example, using the default origin remote repository alias and the branch name we've been working with in this document, the following command can be used to push our changes to our fork.
git push origin framework/history_attribute
If the local branch and remote branch will have different names, the following command can be used:
git push origin local_branch:remote_branch
git push can also be used to delete a remote branch, through the following command:
git push origin :framework/history_attribute
Once your branch exists on your fork, pull requests can be used to propose changes to other repositories.
Another method of communicating with a remote repository is through the use of the git pull command. git pull is used for reading changes from a remote repository and "pulling" them into your local repository.
NOTE: The git pull command can be detrimental to certain workflows. Please use it with caution.
As arguments, git pull needs again the name of a remote repository, and the name of a remote branch. For example:
git pull origin framework/history_attribute
Will pull the remote version of the branch framework/history_attribute on the remote repository origin into the local repository. By default and changes that have occurred on origin's version of framework/history_attribute since the last pull get overlaid on the working directory. This is because a pull is actually the combination of two separate commands. The first being a fetch and the second being a merge. In the coming sections, we will discuss these two additional commands.
In order to pull a remote branch into a new local branch the following syntax can be used:
git pull origin framework/history_attribute:framework/history_attribute
or, more generally:
git pull remote_repo remote_branch:local_branch
git pull is not used to delete local branches, but this is discussed later.
Another method for communicating with remote repositories is through the use of git fetch. In contrast to the git pull command, git fetch simply fetches the changes from the remote repository, and stores them in the local repository. These changes are not overlaid on any of your currently existing local branches.
git fetch can be used to "fetch" all changes in all branches on a remote repository through the use of:
git fetch origin or more generally git fetch remote_repo
Or it can fetch specific branches from the remote repository through the use of:
git fetch origin framework/history_attribute
To refer to the changes on this remote branch, the local branch name would be origin/framework/history_attribute or more generally remote_repo/remote_branch.
Because git fetch does not overlay changes onto the current working directory you must do this manually. These will be discussed in the two following sections, but they are git merge and git rebase. They have very different use cases and the use of each affects the repository in a different way.
git merge is probably the most straight forward way of combining changes between branches. A merge will attempt to automatically integrate the changes between two or more branches into a single branch. By default git merge assumes one of the branches it should try to merge is the current working branch (listed in git branch with a star next to it). It then takes as arguments any branches you want to merge into the current one. For example:
git merge framework/history_attribute
Will attempt to merge the local framework/history_attribute branch into the current working branch. It will then update the branch pointer for the current working branch, after which the framework/history_attribute branch can be deleted.
After a git fetch, there may be new changes on a remote branch that are not yet incorporated into the local branch with the same name. In order to combine these changes, one can issue the following commands:
git checkout framework/history_attribute
git merge origin/framework/history_attribute
Because framework/history_attribute and origin/framework/history_attribute diverged, and were merged together their history will show a similar branch/merge occur.
As a developer, you should not merge develop or core/develop into your feature branch if you can accurately describe why you need to perform the merge. Merges are fixed points in history, and can increase the amount of work you need to perform in order to get the code merged into the relevant branch upon completion.
The git rebase command is an alternative way to combine changes between two branches. It is a bit more intimidating, but might be closer to what you are looking for (especially in the example given above about merging a remote branch into a local branch).
Using the previous example, if there were changes on origin's version of framework/history_attribute that I wanted to incorporate into my local framework/history_attribute branch, but I wanted the history to remain linear, I could make use of rebase.
As with git merge rebase assumes one of the arguments is the current working branch. In addition, it takes another branch as an argument. The branch listed in the git rebase command will be the branch that is untouched.
For example, if you want to "rebase" your local modifications to framework/history_attribute on top of origin's version of framework/history_attribute you could use the following two commands.
git checkout framework/history_attribute
git rebase origin/framework/history_attribute
or, more generally
git rebase remote_repo/remote_branch
If the remote_repo/ is left off of the branch name, it is assumed to be a local branch.
This command takes the two branches (the current working branch, and the "rebase" branch) and determines their most recent common ancestor. Then, it takes all changes on the current working branch, and overlays them on the HEAD (or tip) of the rebase branch.
This causes the history to look as if you had just created your branch, and applied all of your changes as you had before. So it looks like a nice clean line.
After a project is complete, the branches need to be deleted. The deletion of a remote branch was covered earlier, so this section will only cover the deletion of a local branch. To delete a local branch, the following command can be used:
git branch -d framework/history_attribute
This will only remote the branch if it has been merged. If it has not been merged, and you are sure you want to delete it the following command can be used:
git branch -D framework/history_attribute
| Name | Definition |
|---|---|
| Remote Repository | A repository that lives at a different location |
| Remote Branch | A branch that lives on a remote repository |
| Local Repository | The clone of a remote repository onto a local machine |
| Local Branch | A branch that exists in a local repository |
| Fork | A copy of another repository. Both the original and the fork live on github |
| Clone | A copy of another git repository. Creates a local repository from a remote repository. Also creates a link between the two. |
| Origin | An "alias" in a local repository that points to the remote repository it was created from. Formally called a remote. |
| Push | A method of writing changes to a remote repository. |
| Pull | A method of reading changes from a remote repository. |
| HEAD (or tip) | The latest commit on a branch |