New Bioc-GitHub Source Control

BiocGitHub.md

Bioconductor-GitHub Package Sync

This page outlines how to maintain and sync R packages to both GitHub and Bioconductor's git source control system.

Note: the master branch was renamed to devel branch following the instructions here (Mar-2023).

---

Table of Content

1. Commit and push your work
2. Clone new repos instance to local
3. Steps after new Bioc release
4. First time sync

---

1. Commit and push your work

The following instructions are from here. If this is the first sync of a GitHub/Bioc repository, follow the instructions in the First time sync section first. Note, the weekly build schedule for the different types of Bioc packages is here. In order for changes to be included, they should be committed the morning before the listed day with a valid version bump.

1.1. Do some work in devel branch

git checkout devel # switch to devel branch 
git fetch --all # download changes from both GitHub and Bioc without immediately updating your local files. Note: if there are changes in remotes you can investigate them with commands below under 1.3. 
git merge upstream/devel # sync with Bioc. Result is same as one step pull which provides less control: git pull upstream devel
git merge origin/devel # sync with GitHub. Result is same as one step pull which provides less control: git pull origin devel
## -> Do coding work here. When finished, build/check package with 'R CMD build/check <MyPackage>'. Note if there are tests that do not work on your testing machine, then run CMD check with "--no-tests"
git commit -am "some meaningful message"
git push origin devel # push to github
git push upstream devel # push to Bioc

1.2. If necessary make changes to release branch

Example for RELEASE_3_22

git checkout RELEASE_3_22 # Switch to RELEASE_3_22 branch. If you get a 'detached HEAD message', see Section 2 for a fix.
git fetch --all # download changes from both GitHub and Bioc without immediately updating your local files. Note: if there are changes in remotes you can investigate them with commands below under 1.3.
git merge upstream/RELEASE_3_22 # sync with Bioc. Result is same as one step pull which provides less control: git pull upstream RELEASE_3_22
git merge origin/RELEASE_3_22 # sync with GitHub. Result is same as one step pull which provides less control: git pull origin RELEASE_3_22
## -> Do coding work here. When finished, build/check package with 'R CMD build/check <MyPackage>'.
git commit -am "some meaningful message"
git push origin RELEASE_3_22 # push to github
git push upstream RELEASE_3_22 # push to Bioc

1.3 Inspect changes after git fetch --all

Below is for devel branch and Bioc remote. To apply to release branch and/or GitHub, use RELEASE_X and origin as needed.

git log HEAD..upstream/devel --oneline # list new commits in remote that are not in local
git diff HEAD..upstream/devel # compare content changes in files that changed
git diff HEAD..upstream/devel --name-only # list only names of files that changed
git diff HEAD..upstream/devel -- path/to/file # compare changes of specific files that changed

1.4. Sync devel with release branch

Note, it is important here to include the correct range of commits starting from earliest from where the two branches started to diverge.

git fetch --all # ensure local devel and release branches are up-to-date
git checkout devel
git log # find the commit ID range to include in sync
git checkout RELEASE_3_22
git cherry-pick A^..B # A being first 7 characters of oldest and B newest commit ID identified under 'git log' step.
vim DESCRIPTION # correct version number in Description file to expected value in release and/or devel. 
## -> now address conflicts if any

1.5. Useful utilities

Create a copy of a local branch for major changes and then merge back to source branch

git checkout old_branch
git branch new_branch
# -> do some work (edit/add/commit)
git checkout old_branch
git merge new_branch

Clone package from Bioconductor (e.g. to check updates)

git clone -b <my_branch> git@git.bioconductor.org:packages/<MyPackage>

Check differences among two directories with basic Linux diff command

diff -r --exclude=".git" dir1/ dir2/ # shows differences among the content of files
diff -rw --exclude=".git" dir2/ dir2/ # excludes difference in white space and line endings
diff -rwq --exclude=".git" dir2/ dir2/ # lists only the names of files that have differences  

For comparing two branches of same repos, use git diff. The argument setting -w --ignore-cr-at-eol ignores differences in the encodings of white spaces and line endings. For more argument options, see section 1.3 above.

git diff -w --ignore-cr-at-eol devel RELEASE_3_22
git diff devel RELEASE_3_22 -- <path/to/file> # show differences in specific file between two branches

2. Clone new repos instance to local

Instructions adopted from here

Clone repos from GitHub

git clone git@github.com:<developer>/<MyPackage>.git

Add Bioc/upstream remote

cd <MyPackage>
git remote add upstream git@git.bioconductor.org:packages/<MyPackage>.git

Fetch content from Bioc/upstream

git fetch upstream

Merge devel branch from Bioc/upstream with GitHub/origin

git merge upstream/devel

Push any changes to GitHub

git push origin devel 

If syncing the release branch for the first time, you will have to create a local copy for it first. Without this you will end up in 'HEAD detached' state.

git checkout -b RELEASE_3_19 upstream/RELEASE_3_19

Next check whether remotes are set up correctly.

git remote -v 

Correct output is given here.

If necessary add new branch (e.g. from Bioc) to GitHub

git checkout -b RELEASE_3_19 upstream/RELEASE_3_19 # Checkout new branch with name and tracking from Bioc (here RELEASE_3_19)
git push -u origin RELEASE_3_19 # Push new branch to GitHub

Now do some work as described in Secton 1.

3. Steps after new Bioc release

The following creates first a backup/freezed copy of the previous devel branch; then fetches the new release branch created by Bioc; and syncs any changes in upstream/Bioc and origin/GitHub with local devel instance. For additional details see corresponding section in BiocDevel Book here.

git checkout devel
git checkout -b devel_3_22 # Creates copy of devel branch where extension in name is number of corresponding Bioc release. Usually, you don't push this copy to GitHub
git fetch upstream # Fetch content from Bioc/upstream
git checkout -b RELEASE_3_22 upstream/RELEASE_3_22 # Checkout new branch with name and tracking from Bioc (here RELEASE_3_22)
git push -u origin RELEASE_3_22 # Push new branch to GitHub; then repeat for other missing branches if any
git checkout devel
git pull upstream devel # Sync with new devel on Bioc
git pull origin devel # Sync with GitHub
git push origin devel # push to github

After this contine under Commit/Push Section 1

4. First time sync

Follow these instructions when syncing a GitHub repos with a Bioc repo for the first time.

Instructions from here

4.1. Clone from GitHub

git clone https://github.com/MyUsername/MyPackage.git
cd MyPackage

4.2. Configure the “remotes” of the GitHub clone

git remote add upstream git@git.bioconductor.org:packages/<MyPackage>.git

4.3. Fetch updates from all (Bioconductor and GitHub) remotes

git fetch --all

4.4. Merge updates from the GitHub (origin) remote

git checkout devel # Make sure you are on devel branch
git merge origin/devel 

4.5. Merge updates from the Bioconductor (upstream) remote

git merge upstream/devel # resolve potential conflicts, e.g. in DESCRIPTION
git commit -am "resolved conflicts"
git merge upstream/devel 
# git merge --allow-unrelated-histories upstream/devel # use this line instead of previous one with git version >= 2.9 

4.6 Check for duplicated commits

The following command returns a sorted summary of git commit logs.

git log --oneline

Shell command to identify duplicate commits based on patch-id. Commits with identical patch ids are very likely to have identical content (derived from last comment here).

git rev-list devel | xargs -r -L1 git diff-tree -m -p | git patch-id | sort | uniq -w40 -D | cut -c42-80  | xargs -r git log --no-walk --pretty=format:"%h %ad %an (%cn) %s" --date-order --date=iso

After duplicates have been identified one can write their log messages to files and check with diff or vimdiff whether their content is identical. Note, argument --no-pager turns paging off which allows to redirect output to file.

git --no-pager show <commit_id1> > zzz1
git --no-pager show <commit_id2> > zzz2
vimdiff zzz1 zzz2

If there are duplicated commits then a brach swap, as recommended here, may be the easiest solution, meaning the current branch will be replaced with the one from Bicoonductor. If it is preferred to remove the duplicated commits (e.g. duplicates have been committed to Bioconductor already, which should not be possible in the future) then one can eliminate them via git merge --squash as follows.

git checkout devel
git pull upstream devel # just in case
git reset --hard <commit_id> # Reset the current branch to the commit right before dups started
git merge --squash HEAD@{1} # Squashes duplicated commits from chosen <commit_id> to HEAD@{1} (state right before previous reset step)
git commit -am "fixed github/bioc sync problem" # Commit squashed changes
git push upstream devel # Push to bioc

After the removal of duplicated commits continue with the branch swap:

git checkout -b devel_backup upstream/devel # Checkout new branch with temp name and tracking from Bioc devel
git branch -m devel devel_deprecated # Assign to local branch some archive name
git branch -m devel_backup devel # Rename branch with temp name to the one you wish to use (here devel)
git push -f origin devel # Force push to github

4.7. Push to both Bioconductor and GitHub repositories

git push upstream devel # Pushes to Bioconductor
git push origin devel # Pushes to GitHub

4.8. Repeat steps A2/5/6/7 for current release branch

git checkout RELEASE_3_17
git merge upstream/RELEASE_3_17 # -> Already up-to-date
git merge origin/RELEASE_3_17 # -> Nothing to merge since 3_5 didn't exist on GitHub yet
git push upstream RELEASE_3_17
git push origin RELEASE_3_17

4.9. Check whether remotes are set up correctly.

If so the output of git remote -v should look like this

origin  https://github.com/MyUsername/MyPackage.git (fetch)
origin  https://github.com/MyUsername/MyPackage.git (push)
upstream        git@git.bioconductor.org:packages/MyPackage.git (fetch)
upstream        git@git.bioconductor.org:packages/MyPackage.git (push)