Maintaining a Patch Series with Mercurial Queues

This topic has been written about quite a bit (there are links to some additional resources at the end of the article), but I'm going to describe how I handle patches using Mercurial Queues. Some existing articles are either out of date or describe a different workflow.

I normally use Mercurial Queues to maintain patches on top of source trees for applications that I need to modify before compiling for use for whatever reason (compiler fixes, system tweaks, non-mainstream features, etc). Although Mercurial Queues can be used to manage thousands of patches, the number of patches I work with is usually less than ten in this situation, so this work flow is optimized for a smaller number of patches. It also requires the rebase extension (bundled with "modern" versions of Mercurial) to be enabled, as well as Mercurial Queues itself.


Initial Setup
We're going to assume that the official application source is not contained in a Mercurial repository. If it is, you can just clone that and skip the steps in this section. The commands below are based on the workflow I use, just to convey the general idea of what to do.
First, grab the latest version of the application you need to modify, and extract it into a directory. In this case I'll be modifying some_application, because I have no imagination. Since this will be the repository in which you'll be making your changes, you may want to remove the version from the folder name, if present.

cd custom_builds
tar xvf some_application-version.tar.bz2
mv some_application-version some_application
cd some_application

Create a Mercurial repository here.

hg init

Custom Changes
Now you can make your changes, adding as many patches as necessary. Please see the MQ official documentation for more information on these commands,

hg qnew [patch name] # create a new patch
hg qrefresh          # refresh the current patch
hg qrecord           # refresh the current patch (at the patch hunk level)
hg qdelete           # remove the current patch
hg qpush             # push the next patch onto the stack
hg qpop              # pop the current patch off of the stack
hg qseries           # list all of the patches in the series
hg qapplied          # list all of the patches that are currently applied

After your changes are set up to your liking, you can compile and enjoy the application.
Please note that there are many more Mercurial Queues commands; to get a full listing you can type hg help mq.

Handling Updates to the Core Source Code
Say it's been a few months and there have been a few significant changes to some_application, and a new version is out. The kind of changes you would like to be able to use.

Clone your existing repository (without the patches) to create an area in which to apply the core updates.

hg clone some_application some_application-new_version

Extract the source code for the new version over the files in your cloned repository, and update it.

tar xvf some_application-new_version
cd some_application-new_version
hg addremove # tell Mercurial to add new files and remove ones that were deleted
hg commit -m 'Updated to new version.'

Now you can go back to your main repository, and pull in the updated changes from the clone. Thanks to the rebase extension, the pull command can be used to update the changes to the core code, then merge in your managed patches on top using the regular merge tools.

hg qpush -a # apply all of your patches
hg pull --rebase ../some_application-new_version
...do any manual merging necessary...

After this process is complete your patches will apply cleanly onto the updated source tree. You can delete the some_application-new_version folder if you wish, or keep it around for archival purposes.

This process makes maintaining a series of patches relatively painless in terms of merging and keeping the code up to date. Based on the experiences of others, similar processes scale well, so I assume this one does as well.

For more information and additional work flows, check out the links below:
Mercurial Definitive Guide - Managing Change with Mercurial Queues (a little outdated, but useful).
Mozilla's MQ Notes
David Herron's Introduction to MQ
Git User's Guide to Mercurial Queues
Random Notes on Mercurial Queues

Emacs - Clear Current Buffer

There are a few cases where clearing the current buffer is useful. For example,when using the *scratch* buffer a lot, or running a shell inside Emacs, being able to clear the screen can help the user stay organized. Since I'm often stuck in a Windows environment without Cygwin, running a shell inside Emacs is a great boost to productivity.

I recently found a method that works great. I'm sure there other, (possibly more elegant) solutions, but this gets the job done.

There are two functions to define for this solution; the best place for them is probably your .emacs file.

The first function, beginning-of-buffer(), from rattlesnake.com simply moves the point to the beginning of the buffer, leaving the mark wherever it happens to currently be.

(defun beginning-of-buffer()
 "Move point to the beginning of the buffer."
 (interactive)
 (push-mark)
 (goto-char (point-min))

The second function, clear-buffer(), from Audacity Init-Nyquist, deletes everything in the buffer and moves the point to the beginning of the buffer.

(defun clear-buffer()
 "Kill all of the text in the current buffer."
 (interactive)
 (clipboard-kill-region 1 (point-max))
 (beginning-of-buffer)

It is of course possible to combine these two functions together if there is no reason to re-use beginning-of-buffer(). Once this is in your .emacs file, or has been run in Emacs, you can type M-x clear-buffer to clear the current buffer. It can also be mapped to a key combination like any other function.

Mercurial Relink Extension and hg-relink.py

In newer versions of Mercurial (greater than 1.3.1), a great extension called relink comes bundled with the system. For larger projects, this can significantly reduce the amount of disk space consumed when you have a large number of cloned repositories.

When a repository is locally cloned, Mercurial will create hardlinks to the original repository's files in order to save space, if the operating system and file system support hardlinking. However, when a pull is done into either repository (the original or the cloned), those hardlinks are broken, regardless of whether the files are still identical. The relink extension allows Mercurial to re-create those hardlinks. For those of you stuck with 1.3.1 or a prior version, see the last section of this post for the solution used before this extension.

Since the relink extension is bundled with Mercurial, to activate it, just add the standard lines to your .hgrc.

[extensions]
.
.
.
relink =

Using the extension is very straightforward. Execute one of the following commands from somewhere within the cloned repository's directory structure.

# Re-link with the stored origin (the original repository this was cloned from.)
hg relink

# Relink with a specific ORIGIN repository.
hg relink /path/to/origin

Mercurial will print out which repositories were relinked and how many files were pruned. See the official extension page, or type hg help relink in a terminal for more information on usage and output.

For users of version 1.3.1 and prior:
There is a python script (contrib/hg-relink.py, or sometimes contrib/relink.py) in the source tree that provides this same functionality, and can be used with versions 1.3.1 and prior. I used it myself until I found out about the relink extension.

The usage is just as easy as the extension, but requires some extra setup. First, copy the hg-relink.py (or relink.py) file to a directory in your PATH. You can then call it as shown below:

python hg-relink.py /path/to/source /path/to/destination

The script will then hardlink the appropriate files from the source repository to the destination repository. There used to be a page on the Mercurial wiki describing this script in more detail, but it seems to have disappeared.

Changing the Terminal Titlebar Text for Each Application

I have been using the awesome window manager, and it has been, well, awesome. I have also tried ratpoison, xmonad, and ion but awesome is my current favorite. Anyway, as you may have gathered from the title, window managers are not the topic of this post.

I have found myself using more terminal applications because it is easy to manage them with a tiling window manager, and I am a fan of minimizing mouse use when doing text-related tasks. However, when there are several terminal windows open, it would be nice to see what is currently running in each one, instead of having to cycle through all of the windows. I imagine that this is the case with all window managers, and it has just become more obvious with my current setup.

This technique is based on a tip found here, which is for displaying the hostname in the titlebar of an xterm window. I use rxvt-unicode as my terminal, but this technique should work with any xterm compatible terminal emulator running a bash terminal. The following is an example of the section in my .bashrc.

if [ "$TERM" = "rxvt-unicode" ] && [ "$0" = "-bash" ]
then
   # Escape sequence for setting xterm title
   label () { echo -ne "\e]0;$*\a"; }

   # Titles for each application
   alias s1='label urxvt - ${PWD#$HOME/}'
   alias s2='label urxvt - vim $*'
   alias s3='label urxvt - top $*'

   # Commands for each application
   cd_t () { "cd" $*; eval s1; }
   vims_t () { eval s2; "vim" $*; eval s1;}
   top_t () { eval s3; "top" $*; eval s1;} 

   # Assign the aliases
   alias cd=cd_t
   alias vim=vim_t
   alias top=top_t

   # Set title to home directory
   eval s1
fi

Let's take a look and (hopefully) shed some light on what is going on here.

The first line is checking for the name of the terminal. Replace this with whatever terminal you are using (e.g. xterm, aterm, rxvt, etc) so that the configuration will take effect.

The next section defines a function named label that will change the title of the terminal window.

label () { echo -ne "\e]0;$*\a"; }

Next, we set up some aliases to rename the window depending on the application. The $* after each application represents the list of arguments it was called with. For example, the command vim bob.txt would name the window vim bob.txt. Shocking, no? Here we define three aliases; the first sets the window title to the name of the home directory, the second to "vim", and the third to "top", all with the accompanying arguments.

alias s1='label urxvt - ${PWD#$HOME/}'
alias s2='label urxvt - vim $*'
alias s3='label urxvt - man $*'

Now we need some functions to actually change the titles. These are defined in the next section, one for each application. One downside of this technique is that your .bashrc will become long and messy if you need to do this for many applications. The applications have been named with the scheme [application name]_t. Each function does the following:

1. Changes the title for the current application


2. Runs the application


3. Resets the title to the current directory after the application closes

cd_t () { "cd" $*; eval s1; }
vim_t () { eval s2; "vim" $*; eval s1;}
top_t () { eval s3; "top" $*; eval s1;}

The final section assigns aliases for the commands actually used to the functions defined in the previous section. This allows the title window text to be changed transparently in the background. For example, when "vim" is typed, the "vim_t" function will be called.

alias cd=cd_t
alias vim=vim_t
alias top=top_t

The last line of the configuration section simply sets the title to the home directory after the terminal window is opened.

This setup has made identifying windows more convenient for me, and I imagine it would with any window manager, not just the tiling variety. Remember that this should work for any terminal application. Enjoy!

Creating Unrelated Branches in Mercurial

Earlier I watched Scott Chacon's excellent GitCast about using "Empty Branches", or branches that are not derivations of your main development flow. In other words, they are not descendants of your first revision. Obviously his tutorial was about how this is done in Git, but it is possible in Mercurial as well.

It's important to remember that while Git ignores commits that it cannot reach via a branch head or tag, Mercurial follows all commits. Therefore some of the steps described below are not necessary if you wish to refer to a branch head anonymously. Another caveat is that when I refer to a branch, I mean a fork in the revision history graph (you can automatically track these with bookmarks, similar to Git's branches), not what Mercurial sees as a named branch.

Starting an Unrelated Empty Branch


First let's bookmark the tip revision for easier reference later:

$ hg bookmark -r tip MainBranch

Make sure your working directory is clean, with no changes. If there are changes, either revert or commit them.

$ hg status
M test.txt
$ hg revert --all

Now set the parents of the current directory to null to start the root of a new tree.

$ debugsetparents null

Update: Some readers have pointed out that there is a much cleaner (and potentially safer) way to set the parents to null.

$ hg update -C null

Go ahead and add some files and make your first commit.

$ echo "test file" >> f1.txt
$ echo "test file" >> f2.txt
$ hg addremove
adding f1.txt
adding f2.txt
$ hg ci -m "A new separate branch."

Let's bookmark this head so it's easier to review separately from the main development tree.

$ hg bookmark NewBranch

That's it! You can now switch between these unrelated branches using hg update -C. Remember that the bookmarks are simply for convenience, you can always refer to the heads anonymously if you would like. These separate branches have several uses. Scott Chacon mentions creating one for documentation, or for the website for the project. While these branches are part of the project, they are unrelated to the main code base. You could also use these branches to store an alternate version of the project that doesn't share much of the same code.

Using Unrelated Branches

Here are a few usage examples.

# Check out the main project tree
$ hg update -C MainBranch
# Check out the alternate tree
$ hg update -C NewBranch
# View only the history for the main tree
$ hg view MainBranch
# View only the history for the alternative tree
$ hg view NewBranch

Enjoy!

Mercurial vs Git: Tags

The implementation of tags is one design idea that is fundamentally different between Mercurial and Git. In this post I aim to summarize how each system handles tagging revisions, and hopefully enlighten the reader.

Mercurial
Both systems have two types of tags, long-lived tags that are meant to mark significant points in project history, and local or lightweight tags for temporarily marking revisions.

Mercurial brands the two types as public tags and local tags. Public tags are stored in a text file called .hgtags that lives in the root directory of the repository and is versioned along with the rest of the files.

However, the latest committed version is always used, regardless of the changeset currently checked out. This means that any tag can be referenced at any time, even between named branches.

The idea is that changes to tags should be tracked over time, so that we can look back and see when a tag changed, and who made the change. A custom commit message can be added to the changeset adding the tag, allowing for annotated tags. Mercurial also allows GPG signing of changesets with the GPG Extension.
The disadvantage of this is that all of the tags must be shared among clones of the repository. There are some workarounds, as described below. In addition, there are a few caveats with the .hgtags file:

• Each named branch in Mercurial has a separate set of tags.
  
• Occasionally the .hgtags file introduces merge conflicts.
  

Example Usage

hg tag -r 78hfvea v1.08

The other type of tagging Mercurial supports involves local tags. These are stored in a plain text file called .hg/localtags. These tags are, as the name implies, local to the repository in which they were created and are not propagated during a push, pull or a clone. These types of tags are ideal for temporary markers for finding bugs, or simply tags that you do not wish to share.

Example Usage

hg tag -l -r 78hfvea "Bug51 sighted"

However, if you wish to propogate these tags to certain repositories, you can always add a hook for a push event (or just do it manually) to trigger scp or rsync to copy the .hg/localtags file.

Git
Git brands its two types of tags as normal tags and lightweight tags. Normal tags are actual objects in Git (like commits, trees, and blobs), and can be annotated and GPG signed. Unlike Mercurial, Git treats all tags completely independent of history and manages them outside of the file tree.

Example Usage

# create an unsigned annotated tag
git tag -a v1.01 78h2gv

Git's lightweight tags are similar to Mercurial's local tags. They exist as text files in .git/refs/tags and are not annotated. However, unlike Mercurial's local tags, these are separate objects and can be shared between repositories.

Example Usage

git tag PossibleBug 78h2gv

Summary
Both systems have capable tagging capabilities that work well with most workflows. Mercurial's global tags are version controlled, which can be advantageous if your project workflow involves lots of tag changes. The local tags provide an alternative for throw-away tags that are not part of project history, at the expense of not being able to directly share them between repositories.

Git's tag objects have the advantage that they can be annotated directly in the tag, which is advantageous if you often need to store longer comments about particular revisions. In addition, both normal and lightweight tags can be selectively shared between repositories (via push/pull/etc). This feature adds some flexibility, allowing someone to publish only the tags that make sense for a public repository.

For more information concerning how each system handles tags, check the manual for the respective system: Mercurial, Git.

Uses of Mercurial's "bookmarks" Extension

The bookmarks extension that is now bundled with Mercurial makes it easy to keep track of multiple heads in a repository, similar to Git's "lightweight" branches.

While Mercurial has always supported lightweight branches via multiple repository heads, they are somewhat difficult to keep track of because they are unnamed. Moving tags manually (or via a script) to keep track of each head becomes unwieldy, as does tracking by the last commit message. In Git, unnamed repository heads simply become inaccessible if they are not tagged. A branch in Git is basically a pointer to the head of a fork in the tree.

Similar to Git's branches, bookmarks are markers that move forward with each commit automatically, allowing you to track different lines of work in one repository. This feature is perfect for small experimental features or bugfixes that will either later be thrown away or merged into the main line of development.

NOTE: To take advantage of having a "current" bookmark, make sure you have the latest version of the extension, available here.
You will also need to add the following section to your .hgrc or Mercurial.ini.

...
[extensions]
hgext.bookmarks =
...

[bookmarks]
track.current = True

Here are a few example uses of bookmarks to consider:

Let's say you're working on a relatively small project and only have two clones, one for main development and a "stable" clone. You decide to start work on a new feature based off of version 1.0 of your project. Fortunately, thanks to your crazy awesome software development skills, you tagged v1.0 after it's release.

...create a bookmark for your new feature
hg bookmark -r v1.0 awesome_feature
...check out the code and set as the current bookmark
hg update -C awesome_feature

Those who are familiar with Git's branches will see the similarity. The current bookmark is set in .hg/bookmarks.current. Now you start on your feature.

...some good work going on
hg commit -m "A good feature."

You will notice that the bookmark moves along with your commit. With the extension set to track the "current" bookmark, only the one currently set will move forward. This means that you can set two bookmarks to the same revision, and as long as you update appropriately only one will move forward. This is different from how the bookmarks extension operates by default, which entails all bookmarks moving forward when a commit is done on their parent.

Now you start another feature from the v1.0 revision and make some commits.

hg bookmark -r v1.0 even_cooler_feature
...set as current bookmark
hg update -C even_cooler_feature
...work
hg commit -m "some work."
...work
hg commit -m "some more work."

*** soda break ***

...work
hg commit -m "almost done."

Someone wants to help with one of your new features and clones the repository. Since they also want the bookmarks, they have to copy the .hg/bookmarks file over manually. This is important: bookmarks are not part of the repository (yet). They are not pushed/pulled or cloned. However, since they refer to the hash of the changeset ID, they can be used in any clone of the repository.

The anonymous helpful person does some work on the "awesome_feature", making a few commits and asks you to pull and review them. However, he has also made some commits to "even_cooler_feature" that are not ready yet, so you need to pull only the changes from the awesome_feature branch. This is a two step process, although it can be easily automated.

# update and set the current bookmark to awesome_feature so it is fowarded
hg update -C awesome_feature
# find the tip of the awesome_feature branch 
# (the location of the remote bookmark)
hg id -r awesome_feature http://some.remote.repo
74b4cb5hs4b2
# pull all of the ancestor changesets of that tip
hg pull -r 74b4cb5hs4b2 http://some.remote.repo

pulling from http://some.remote.repo
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 6 changes to 2 files
(run 'hg update' to get a working copy)

The awesome_feature bookmark will have moved to the tip of your now updated branch, and you can review to your heart's content. No unrelated changesets were pulled. You can also push bookmarks just as you can with any other revision identifier.

Now say you want to add an experimental feature that you may not keep.

hg bookmark -r v1.0 experiment
# locally tag the base of the branch so it can be deleted later if necessary
hg tag -l experiment_base
...some work
hg commit -m "here goes nothing."
...some more work.
hg commit -m "work."
...realization that this is a horrible idea

You can delete the bookmark and leave the head and changesets intact (which is what Git does), or remove the changesets with the strip command from Mercurial Queues.

# delete the reference to the tip
hg bookmark -d experiment
# now no one will ever know, muwahaha
hg strip experiment_base

There are many more things you can do with bookmarks, but this should get you started. This is a great extension and people who use both Git and Mercurial will enjoy having more flexibility with lightweight branches in Mercurial. I'm sure as the extension evolves we'll have more features to play with. I...err...of course mean "optimize our productivity". Have fun!