SOFTWARE DEVELOPMENT
How to manage software
development for start-ups
By Osborne Hardison
Senior embedded engineer
Although software-development
processes are numerous and well
documented, they don’t always fit
the time and resource constraints
of an embedded systems start-up
company. Often a start-up’s software team follows a leaner path to
achieve their goal: get a product
out on time with no major defects.
These teams frequently use publicdomain tools and a mix of hybrid
software development life cycle
techniques, but there are as many
nuanced processes as there are
embedded systems start-ups.
What works? This article provides a set of software-development processes that work well
in the context of an embedded
systems start-up. These include:
• The early stage setup of the
source control and build
policies
• Activities during start of the
development stage—from
the functional requirements
document (FRD) to first customer ship (FCS)
While some of these steps
may seem obvious, each step I
describe is vital to achieving the
stated goal.
Minimal tools required Starting with the assumption that
start-ups are notoriously cheap
organisations, I’ve compiled a list
of tools that start-ups would do
well to have at their disposal:
• Source-code control (SCC)
system, such as CVS and
CVSweb
• Scripting languages for builds
and reports, such as Perl and
make
• Bug-tracking or problemtracking database
Your bug-tracking databases
should preferably have a web
interface. Free bug-tracking databases include Bugzilla (www.
‘Figure 1: Revision-difference report for reviewing check-ins
bugzilla.org) and GNATS (www.
gnu.org/software/gnats), but
licensed databases such as FogBUGZ (www.fogcreek.com/fogbugz) have compelling features
such as reports that make them
worth considering, too.
Source-control policies
Before starting to develop the
product, the engineering organisation needs a foundation on
which to reliably build and release software. This foundation
should include, at a minimum,
support for documentation,
source-code control (SCC), and
release check-ins.
Document in SCC and
on the web
You should make sure that all
documentation is source-code
controlled and available via the
company’s intranet. This step
makes it easy to track, change,
and find the latest design and
module information. The documentation should include some
means of displaying source
changes and histories so that
interested parties (for example,
the quality-assurance and documentation groups) can review
and note code changes. The CVS
web tool, CVSweb, does just this.
It provides a web interface that
displays source history and differences between code revisions.
You should also have a repository online that contains the
overall structure of each major
module within the product. The
repository should point out the
changes to be added to a module
in each release. This information reduces architectural and
release-oriented questions that
might otherwise be foisted on
engineers during the cycle,
thus freeing them up from
low-priority interrupts. One of
the major jobs of engineering
managers is to protect their
engineers from external information and feature requests. By
having all pertinent information
online, a manager will reduce
some of these requests; others
can usually be handled by the
manager himself.
Plan the release and the
merge
A major release should be represented as a branch off of the current CVS trunk. Minor and patch
releases should be branches
off of existing release branches.
Spell out your policies regarding
merges back to other releases
or to the main branch on a
web page and via e-mails to
the engineers. If possible you
should plan to check merge
progress using an automation
tool; with CVS you can peruse
the CVSROOT/commitlog to
check comments for bug reference and branch targets.
Include a reference to a bug
ID in release check-ins
It’s relatively easy to tweak the
CVS loginfo script to call a Perl
script that can munge the checkin information into a webified
version using the MIME::Lite
CPAN Perl module. The script can
then make the source-file reference and new revision-number
links into CVSweb’s pages for the
file and the revision-difference
report respectively. Additional
points are possible for parsing
out the bug reference and making it a link into the web interface
of the bug database.
The net effect is that the
MIME e-mail is sent to your
software organisation for every
check-in. If you have a policy
of code reviewing check-ins,
the revision-difference report
makes that task much easier.
Figure 1 shows an example of
a source change report.
The comment field has a link to
the bug or problem report (1621
in this case), and below that are
the CVSweb links to the source
file and a revision-difference
report (from 1.12.2.1 to 1.12.2.2).
Finally, readers can see that the
check-in was a file modification
for the 3.1 release branch (with
the name “Release_3_1_branch”
as it exists in CVS). Note that if
someone fails to add the bug ID
on a check-in, you can go back
and use “cvs admin” to fix the
comment so that later reports
work properly.
Build policies
With the source controlled, it’s
time to tame the build. How
often will you build? What is the
format for the version of the release? What type of notification
will take place each time a build
is completed (or fails)?
EE Times-India | December 2004 | eetindia.com
Figure 2: An e-mail notification for release 2.5.0 QA build 15.0
Create a versioning format
You’re going to be firing up many
builds over the course of a release
lifecycle. At the same time, your
team might create other release
builds or top of tree (trunk)
builds. You will most assuredly
need a versioning scheme of
some sort. To maintain a logical
versioning system, you could
use a scheme that includes major, minor, patch, QA, and nightly
build numbers. For instance, if
a developer wants to check the
2.5.0 release nightly build #3 off of
QA build 2, he would download
build version 2.5.0.2.3. The build
directory should be named
something akin to its version,
so that in the last example the
build directory might be named
“020500-002-003.” In this way
nightly, QA, and release candidate (RC) builds can all peacefully
coexist in a logical fashion.
Mandate e-mail notification
of builds
For any build completed, an email notification should be sent
to the entire software group. This
e-mail should once again be a
MIME-enabled web page that
includes all of the files that have
changed since the last build of
the same type (nightly or QA).
The e-mail should include a
similar listing of all files checked
into the branch and their new
revision—and once again, this
e-mail should have links to
bugs, source files in CVSweb, and
a revision difference section in
CVSweb. In this way, engineers
know what parts of the system
have changed or what bugs
were potentially fixed. To increase
the report’s usefulness to QA
especially, the generating script
should pull differences between
build tags for QA or RC builds.
eetindia.com | December 2004 | EE Times-India
Figure 2 is an example e-mail
notification for release 2.5.0 QA
build 15.0. Note that a CVS tag
with this version information
has also been created (“QA_
BTAG_V2_5_0_15_0”). The first
nightly build after this build milestone would have the version
2.5.0.15.1. The QA department,
on receipt of this build, can tell
that the pesky foobar device
has a fix applied against an outstanding bug and that the state
of the bug is now “fixed.”
Schedule nightly builds
for current releases
and the trunk
You should schedule nightly
builds for all releases that are currently in the works until they reach
beta. This process ensures that
first the release has not been broken (breaking a build is a serious
offense with a dreaded penalty:
the offender has to buy doughnuts for the rest of the organisation), and allows for a nightly
“smoke test” or automated-test
procedure to run. In addition to
releases in the works, a nightly
build of the main trunk (or HEAD
branch in CVS) should be done
to ensure its health. Nightly
builds can devour a great deal
of disc space, so pruning the
builds to keep the latest five
can maintain your free space.
If no check-ins are detected for
the branch or trunk, the build
shouldn’t run (and the e-mail
notification should indicate that
no check-ins were found so the
build was not done).
QA and RC builds are usually
done out-of-band to the nightly
builds, but maintaining a valid,
smoke-tested build can go far
towards deriving a product that’s
ready for testing or release.
Figure 3: Graph of open versus fixed problems
Engineering and QA policies
Developers are buzzing away on
features and fixes, you’ve got the
nightly builds firing, so now it’s
time to engage the QA people.
Some organisations promote
builds to QA fairly early in the cycle; others wait for a preordained
code-freeze milestone before
sending a build in for testing.
Your mileage may vary, and you
might be short a tyre if QA is
working on other releases concurrently. Once you do decide
to send QA a release, here are
some tips on what to expect.
QA and later-cycle RC builds
Engineers use the nightly builds
when testing embedded systems
on their target processor. The
builds can be loaded onto the
target and debugged from that
point. QA builds are done on
a regular basis so that QA may
perform their test suites and validate and fixed bugs. As a rule of
thumb, a QA build is appropriate once a week so that QA has
enough time to run their suites
and the queue of fixed bugs
doesn’t grow too large. The frequency of QA builds depends on
the size of the project.
RC builds, like QA builds,
should be designated by an
increment in the QA number
that appears in both the version
and the build-directory name. RC
builds are created the same way
that QA builds are, minus whatever internal QA designations
that might appear in the build
information. Avoid deleting RC
and QA builds (especially if they
aren’t backed-up), since they’re
extremely useful for testing older
builds for the existence of some
classes of difficult bugs.
Finally, both QA and RC builds
should be tagged in CVS so they
can be reliably reproduced.
Having tags also helps generate build-difference reports so
that engineering team and QA
can agree on what fixes made it
into a build.
Turning knobs—time
vs. resources
Which knob are you going to
turn? Do you have the luxury of
time in your schedule to take
your project to zero defects, or
will you be forced to settle for
zero fatal and major defects? Can
you move resources either within
QA or into QA to help the time
crunch? The more challenging
part of a release is the trade-off
when confronted with a timeconstrained release or resourceconstrained release. While there
are no short answers to problems
such as these, once you know
where you are and where you
should be, you can gauge what
knobs to turn. The key is understanding your current position.
Hold regular meetings
Hold regular meetings to consider the state of the release.
You should require two types of
meetings: the daily bug meeting
and the weekly product review
meeting.
The first of meeting goes under different names at different
companies but for our purposes
I’ll call it a daily bug meeting.
Whatever you call it, the meeting’s goal is defined as taking a
regular and frequent (daily is the
norm) snapshot of the current
bug state of a release. Early in the
cycle numerous bugs may be on
the list so you should concentrate
on a those with a high priority.
The first part of the daily bug
meeting should be devoted to
the bug review. The meeting
coordinator (either the project
manager or the engineering
manager in charge) runs a projector that displays a summary and
individual bug reports. The latter
part of the meeting should look
at the bug queues (new, open,
fixed) and outstanding issues
to determine how the release is
progressing along. The benefits
of a regular review are many:
• All of the responsible parties are up-to-date on the
state of the release—thus
it makes sense to include
a representative from each
of the stakeholders for the
release—engineering, QA,
product marketing, program
management, manufacturing,
and documentation if possible
• Bug priorities can be assigned—later in the cycle
this becomes a critical determinate of whether a bug will
be fixed before shipping
• Late feature requests can be
judged for merit
The second required meeting
is a product review meeting, during which all of the stakeholders
are required to meet and review
EE Times-India | December 2004 | eetindia.com
the progress of the product's
release as a whole. This weekly
meeting should be called by the
marketing product manager and
should be attended by the same
people as the daily bug meeting plus some senior marketing
and manufacturing heads. The
purpose is to ensure the release
stays on schedule and that the
disparate groups coordinate
their efforts (is the web site up,
is the marketing collateral in
shape, will the product launch
at any trade shows?).
Towards the end game
So you have these wonderful
policies implemented and your
process is humming, new source
developed, and at some point
the QA builds start. What now?
There should be some time in
the schedule for a code freeze.
That point may be determined
by some heuristic like a graph
showing open bugs vs. closed
over time, or it may be the
hard-and-fast requirement of a
predetermined schedule. It all
depends on what knobs you’re
able to turn. At code freeze,
the bar is set higher—software
engineers are advised to check-
in only the bugs at a threshold
priority. QA must turn around
and verify bug fixes with a high
degree of reliability for both the
fixes themselves and for the time
that it takes to test them.
Open vs. fixed graph
A graph that displays open vs.
fixed issues can provide insight
on stability of the release (number of open issues) and QA loading (number of fixed bugs waiting close). For example, consider
the graph in Figure 3; the open
issues are being handled even
though there are still five waiting
for a fix. The to-be-fixed queue is
rising, indicating that QA is lagging closing out bugs. A decision
maker might look at this and decide to move some QA resources
to help close out this release.
Beta
To get to this stage and have a
product ready for customer trial,
only minor issues should be outstanding (or at least one or two
major issues with workarounds).
The open vs. fixed graph should
be trending downward, and
the frequency of bug meetings
should be reduced to once or
eetindia.com | December 2004 | EE Times-India
twice a week, and the majority
of the discussion should be on
coordination issues with the
documentation group on how
to characterise any lingering
issues. Most of the engineering group should have already
started on the next release with
some possibly working on a follow-up maintenance or patch
release and others on the next
major release.
First customer ship
Ah, you are done. Or are you?
Congratulate yourself and your
team and then get ready to start
pushing the rock uphill once
more. Often the release team
should host a post-release
meeting to go over what did
and did not succeed during the
cycle. This is a great time to fine
tune your release process.
Along the way you might
consider some additional questions. Have you started on a
maintenance release? Have
you checked with your sales or
support engineer (quite a valuable and underutilised resource
for internal status checks) on
how the release is being accepted? Has anyone considered
a customer survey to see if the
release produced as it promised
or if it missed the mark?
Apollo 13
Many anxious moments will
occur during any release; some
may seem as bad as NASA Fight
Director Gene Krantz had it during Apollo 13. He heroically rose
above the issues and led a seemingly doomed space mission
back to Earth. He and the rest
of Mission Control were able to
detach themselves emotionally
from the problems to search for
solutions.
Engineering release managers must be made of the
same mettle and must summon
some introspection during the
tough times to be able to wade
through a mess and see your
organisation’s “finest hour.” The
recommendations within this
article cannot in and of themselves help forge your mettle,
but they can provide you with
the tools that give you release
status information, process, and
controls in place.
Email Send inquiry