Valgrind Survey, November 2003

Table of Contents

Overview

Summary

Q1: Project Info

Q2: Skin Use

Q3: How Used?

Q4: Other Tools

Q5,6,9,10: Good and Bad Things

Q7: Other OSs

Q8: Other Architectures

Conclusion and Actions

Overview

The survey was sent to vg-users list, plus 226 people who had contacted me directly in the past 18 months (10 of those bounced).

We received 116 full responses (plus 2 or 3 that gave no useful info). 115 were in English. 1 was in French. Fortunately the French was pretty easy.

Nationalities of the 226 people directly contacted (based on email suffixes; lots of .com ones so some countries might be missed):

• Europe: Belgium, Czech Republic, Denmark, Finland, France, Germany, Greece, Italy, The Netherlands, Norway, Poland, Russia, Sweden, Switzerland, UK
• North/South America: Argentina, Brazil, Canada, USA
• Asia/Oceania: Australia, Japan, New Zealand
• Africa: South Africa
• Middle East: Israel

The survey wasn't well structured -- too verbose, not quantitative enough. I had to interpret written answers to fit them into categories a lot. Collating it all was cumbersome, and I probably made mistakes transcribing lots of things. But the results should hopefully be representative.

Generally, when I say "Valgrind" it usually refers to Memcheck, although judge each mention by the context. It should be obvious when I'm referring to other skins.

I've come up with an improved version that could be used in the future, or hung off the web page.

Summary

Projects:

- huge range of software types, almost everything imagineable
- very roughly, number of language mentions: 45% C, 40% C++, 15% other
- projects ranging from 1 programmer to 10s, even 100s
- project sizes ranging from tiny to 25 MLOC

Skin use:

- Memcheck dominates massively;  partly because it's the most useful,
  but confusion about skins means its default-ness is a huge advantage.
- Addrcheck is holding its weight in the distro
- Calltree is doing very well outside the distro, probably should replace
  Cachegrind
- Cachegrind and Helgrind aren't holding their weight in the distro.

Use of Valgrind:

- vast majority (90%?) run it manually, a few use it as part of automated
  testing
- most commonly used when a bug is found or suspected (40% of repondents)
- also commonly before releases, and when big changes are made

Other tools:

- lots of people have used Purify, Valgrind compares well
- Valgrind's pros: no recompilation, free, no horrible license server
  - Purify's pros: GUI is nicer, and allows some interactivity
  - Equal points: technically seem similar, eg. speed, bugs found
- Valgrind compares well with insure/insure++
- Valgrind thrashes other error-checking tools
- KCachegrind seems to compare pretty well with other profilers

Good things about the software:

- praise about bug-finding:      [42]
- lack of recompilation:         [40]
- free software/Linux-ness:      [21]
- praise for error messages:     [12]

Bad things/improvements for the software:

- no complaints:                 [18]
- slowness:                      [32]
- robustness problems:           [23]
- docs bad:                      [18]
- desired features
  - GUI:                         [8]
  - static binaries:             [3]
  - interactive leak checking:   [2]
  - stack variable checking:     [ ]
- annoying features
  - error messages:              [20]
  - external skins hard to use:  [ 6]
  - suppressions difficult:      [ 5]

Good things about non-software:

- most people generally happy with development:  [71]
- development praised:                           [17]
- mailing list is good:                          [ 7]

Bad things about non-software:

- webpage is bad, doesn't link to skins/GUIs:    [18]
- skins are confusing, poorly understood:        [15]
- releases aren't always done well:              [10]
- Valgrind not marketed well:                    [ 6]
- a few patches have been handled poorly

Money:

Recommendations that we should actively seek donations, etc.

Other OSs:

- Windows most popular, although not sure about which version
- Solaris next, aroused the most number of passionate responses
- FreeBSD the most popular other Unix, just ahead of OS X.

Other architectures:

- PowerPC, SPARC and AMD-64 most popular.
- IA64 a fair way back.

	TOC	Q1: Project Info >>