Bazaar developers’ blog

June 2, 2011

fault-line: bzr plugin to guess relevant test modules

Filed under: Uncategorized — Martin Pool @ 1:36 am
Tags: , ,

Aaron has announced a bzr plugin fault-line:

It works by looking at previous revisions where the file was changed,
and seeing what test files were changed at the same time. You can
specify the files, or it will autodetect them by looking at your working
tree.

e.g.

4 Comments

  1. What use is this? As QA/testing methods go, guessing is the least effective and should be ruled out altogether.

    Comment by Carter — June 2, 2011 @ 4:48 am

    • The point is that guessing which tests are most likely to fail may make your development cycle faster. Obviously it is no substitute for running the whole suite before concluding a particular branch is finished and ready for integration.

      Comment by Martin Pool — June 2, 2011 @ 4:52 am

      • But guessing yields non-definitive results, leading in some cases to a prolonged development cycle. A tool that sometimes helps but sometimes hinders is not very useful; it’s not too different from having tests which are themselves less than 100% correct. Certainty is always desired where tests are concerned.

        For each changeset being merged every test that could possibly be broken should be run. In side-effectful languages (i.e. pretty much every language except Haskell), for non-trivial programs with non-trivial test suites, determining which tests could possibly be broken is an intractable problem, short of actually running every test. So unfortunately there aren’t any shortcuts — the only way to be certain is to run all tests every time.

        Comment by Carter — June 2, 2011 @ 7:03 am

  2. You’re right that this should not used when merging a branch into the mainline – and Launchpad still runs the full testsuite in that case.

    The point of faultline is to help developers figure out what tests to run when they’re doing development on their local machine, before they submit their branch for inclusion in the mainline. The Launchpad test suite takes a couple of hours to run – that’s too much to run the full test suite every time you refactor a function.

    Comment by Jelmer Vernooij — June 2, 2011 @ 9:07 am


RSS feed for comments on this post.

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: