Fun at the Bazaar sprint

Last week was the Bazaar sprint, which was fantastic and tiring. Somehow even the people who’d been at UDS just before made it through five packed days of fixing bugs, preparing releases, and debugging package imports. We were most hospitably hosted at the Canonical offices a long way up Millbank tower. But even those who couldn’t be there in person to enjoy the view were part of the experience. At home in the Ukraine Alexander wore his Bazaar shirt in support during the first day. On IRC larstiq and santagada ran the test suite on pypy and investigated incompatibilities. And all week we had a small robot John sitting in the middle of the table on the line from the Netherlands, working on performance bugs and offering helpful advice.

There were two new faces introduced. Max has been a stalwart maintaining the ~bzr PPAs and getting daily builds working. Jonathan is joining the Bazaar team on rotation from Kubuntu, which is very exciting for fans of qbzr. He started getting to know bzrlib by taking on some bugs tagged ‘easy’ and pair programming on harder ones. It was a bit tough to keep track of everything going on, but good progress was made on the Ubuntu Distributed Development front, the translation framework branches Naoki put together were landed, and lots of pet bugs were fixed. Download bzr 2.4b3 now to see the rest of the results for yourself.

After these long days in front of screens a nice meal out was a welcome treat. Over dinner we even managed to get on to topics other than code on occasion. On Thursday evening everyone went to As You Like It at the Globe as groundlings. Even with the language barrier to overcome for some of the sprinters, the comedy lived up to the categorisation. Trying to use the cycle hire scheme to travel there and back proved more of an obstacle. The bikes themselves were fine, provided you could get past the terrible computer interface and persuade the system to let you rent them. Now, if only they took patches for that…

bzr-2.4 faster large tree handling

The last of my patches is queued up to land, so I figured I’d post an update about the performance improvements I’ve been working on. I’m also just excited about how well it has all come together.

There were essentially 3 changes that mattered for performance on large trees.

1. Fixing iter_entries_by_dir() to preload the data in Repository- optimal ordering rather than by-request ordering. In large trees this was causing us to thrash and become pathologically slow. In the 70,000-file test tree, thrashing took about 3 minutes, the preloading version takes about 15s. This affected a lot of our commands, though I guess the next two fixes would actually reduce the number of commands affected by this.
2. Fixing several code paths to use optimized iter_changes() rather than the generic iter_changes(). The generic path walks both inventories iter_entries_by_dir() and compares them. Our 2a format Repository can do iter_changes without loading the whole tree. (It internally uses a hash_trie to store the inventory, and so nodes with matching sub-trees can be skipped for comparison.) This generally shows up as something that was taking 15s (to load the whole inventory) dropping to <2s for the improved comparison. (bzr revert and bzr pull were both directly impacted here)
3. Changing WT.set_parent_trees([one_tree]) to update itself using current_basis.iter_changes(one_tree), rather than setting the state from scratch. This basically adds another case where we can avoid reading the whole inventory state again, which is another 15s to <2s sort of change. This only showed up after fixing (2), because once the tree is loaded, the other actions are generally pretty quick. (bzr up, bzr pull)

This is the chart I put together for “whats-new-in-2.4.txt”. bzr-2.3.2 will have fix (1), but not (2) or (3), to give a feel for how much of an impact different fixes have had.

    bzr-2.3.1 bzr-2.3.2 bzr-2.4  action
    3m39s         1m08s   1m03s  bzr co --lightweight
      38s            8s      2s  bzr revert (in a clean tree)
    4m47s         3m56s     15s  bzr merge
    4m45s           20s      3s  bzr pull
    4m58s         3m00s      2s  bzr up
    9m33s           21s     19s  bzr uncommit (including a merge)
    4m44s           17s      2s  bzr uncommit (simple commit)

So yes, some operations that were taking almost 5 minutes have now dropped down to taking <3s.

You won’t see that dramatic of an improvement for smaller trees, though most cases will have a pleasant improvement. Here is a short list for the ‘Launchpad‘ tree (with ~8k items).

    bzr-2.3.1   bzr-2.4     action
    5.3s        5.2s        bzr co --lightweight
    0.9s        0.3s        bzr revert
    1.4s        0.4s        bzr pull
    3.9s        3.7s        bzr uncommit (with merge)
    0.9s        0.3s        bzr uncommit (without merge)

Anyway, I’m quite happy about how much better bzr-2.4 will be in large trees.

update:Add graphs…

nice fast grep

Parth announced bzr-grep 0.2.0.  Amongst other things there are performance enhancements such that Eli says:

Thanks, this looks great. I just tried it on Windows XP searching for a fixed string in the Emacs repository — took 28 seconds with a cold cache and only 7.5 with a warm cache, which is impressive.

By contrast, “grep -F -R” (with suitable –exclude patterns, to prevent it from searching binary files and inside .bzr) took about 12.5 seconds with a warm cache.

Ten reasons to switch to Bazaar

Are you a Bazaar fan and need some help explaining to others why Bazaar is cool? I published a document last week called Why switch to Bazaar? that may help. I’ve tried hard to present the big picture together with some concrete examples, explaining what we stand for and what that means to users, teams and communities in reality. Furthermore, if you tried Bazaar 1.x but found it too slow or inefficient, I’m sure you’ll find the Bazaar 2.0 benchmarks included in the document great news.

I hope you find the document interesting and food for thought. If there are any mistakes or you’ll like to translate the document to another language, please let me know.

Update on brisbane-core

[I drafted this post a couple of weeks ago, so there is some later news. It’s still useful so I’ll post it anyhow, with more to come later — mbp]

The short story: brisbane-core has some good size and speed numbers; it’s now merged in to bzr 1.14rc1, so we can get wider testing across Launchpad and from users.

We hope soon to have an “edge” Launchpad code server, which would be working on live data but running release-candidate bzr code for better testing.

As fairly typical numbers for python3.0: fastimport time (similar to repeated commits) is down from 168m to 51m and the repository size is down from 160M to 77.4M. For MySQL the numbers are better, 501MB down to 170MB (nearly 3x). ‘log -v’ which was a particular point for emacs is now about 20x faster than in the 1.9 format.

We don’t have to do much new development as such but we do have to do some more risk reduction, in checking for performance or functionality problems, and some further review before it merges. This testing and tuning will continue after it lands. There are still some operations that are known to be slower than in current formats, such as branching the whole repository, but they should be fairly shallow and are being worked on. (To be clear, they’re only cases where the new format is not as good as it will be, not regressions on current formats.) It makes sense to continue working on these in parallel with letting Launchpad and other people integrate it.

The largest question mark is over how smooth we can make upgrading of stacked branches on Launchpad. This is important but probably something that we should tackle during the beta period, when other things are tied down. We will have a reasonable answer, the question is just how automatic will it be and where will it be done.

Aaron is also now working on nested trees since the sprint and that seems to be going well, with the -subtree variant now obsolete. We’re aiming to do this for 2.0 too but won’t block finishing brisbane-core on it.

Brisbane sprint report

All of the meeting rooms in the Ibis Brisbane are named after
British/Australian explorers. It was probably a good sign that ours
was named after one of the most successful, Matthew Flinders. At least he didn’t get lost and die horribly…

We had a sprint for Canonical people interested in or working on
Bazaar: Aaron Bentley, Andrew Bennetts, Ian Clatworthy, James Westby, John Arbash Meinel, Jonathan Lange, Martin Pool, Robert Collins, and Vincent Ladeuil. I’m glad all came, particularly those who had to travel a long way.

The main point of the sprint was to develop and solidify the
brisbane-core format.
This was accomplished: we settled some open questions (especially on
using the groupcompress compressor), and showed it performing well in
many situations. Aaron is resuming work on finishing off nested tree
support, allowing separate branches to be linked in as subdirectories.

Some more random notes:

• merged the groupcompress plugin into
  the brisbane-core branch, so now that
  branch is all you need for the new formats
• finished a patch to kill autodetection of nested trees, which should
  mean we can use just one format that supports nested-trees and they’ll
  only be active if the user actually turns them on
• log -v is now roughly twice the speed it was before the sprint.
• measured brisbane-core storing the large Launchpad tree in roughly
  6x less space than 1.9
• worked on content filtering integration with dirstate, so that we
  can handle newline conversion; finished the main patch to support it
• talked about workflows for merge proposals for package branches, and settled how to link source branches to packages officially, and almost completed work to expose this through the api (which is the most useful bit)
• fixed the tilde bug in urls
• got a stacking bug fixed which was a regression from 1.12
• bzr now does streaming push and pull, including for stacked
  repositories, and the server will no longer try to open stacked-on
  repositories itself; many fewer roundtrips; so the largest part of
  outstanding network problems are done. (To see the best performance
  you’ll need 1.14dev on both client and server.)
• added a new conflict-diff command to bzrtools, and released bzrtools
• fixed a bug in unicode symlink handling
• mentored Bob Tanner to do the 1.13 release, the first in some time
  done by a non-Canonical contributor
• settled that we’ll have the next one-week sprint in Barcelona at the
  same time and place as UDS
• worked on a mysql-affecting bug

It was good.

Going to these things always makes me wonder what Canonical would be like if we had just one big office…