SOFTWARE DEVELOPMENT How to manage software development for start-ups By Osborne Hardison Senior embedded engineer Although software-development processes are numerous and well documented, they don’t always fit the time and resource constraints of an embedded systems start-up company. Often a start-up’s software team follows a leaner path to achieve their goal: get a product out on time with no major defects. These teams frequently use publicdomain tools and a mix of hybrid software development life cycle techniques, but there are as many nuanced processes as there are embedded systems start-ups. What works? This article provides a set of software-development processes that work well in the context of an embedded systems start-up. These include: • The early stage setup of the source control and build policies • Activities during start of the development stage—from the functional requirements document (FRD) to first customer ship (FCS) While some of these steps may seem obvious, each step I describe is vital to achieving the stated goal. Minimal tools required Starting with the assumption that start-ups are notoriously cheap organisations, I’ve compiled a list of tools that start-ups would do well to have at their disposal: • Source-code control (SCC) system, such as CVS and CVSweb • Scripting languages for builds and reports, such as Perl and make • Bug-tracking or problemtracking database Your bug-tracking databases should preferably have a web interface. Free bug-tracking databases include Bugzilla (www. ‘Figure 1: Revision-difference report for reviewing check-ins bugzilla.org) and GNATS (www. gnu.org/software/gnats), but licensed databases such as FogBUGZ (www.fogcreek.com/fogbugz) have compelling features such as reports that make them worth considering, too. Source-control policies Before starting to develop the product, the engineering organisation needs a foundation on which to reliably build and release software. This foundation should include, at a minimum, support for documentation, source-code control (SCC), and release check-ins. Document in SCC and on the web You should make sure that all documentation is source-code controlled and available via the company’s intranet. This step makes it easy to track, change, and find the latest design and module information. The documentation should include some means of displaying source changes and histories so that interested parties (for example, the quality-assurance and documentation groups) can review and note code changes. The CVS web tool, CVSweb, does just this. It provides a web interface that displays source history and differences between code revisions. You should also have a repository online that contains the overall structure of each major module within the product. The repository should point out the changes to be added to a module in each release. This information reduces architectural and release-oriented questions that might otherwise be foisted on engineers during the cycle, thus freeing them up from low-priority interrupts. One of the major jobs of engineering managers is to protect their engineers from external information and feature requests. By having all pertinent information online, a manager will reduce some of these requests; others can usually be handled by the manager himself. Plan the release and the merge A major release should be represented as a branch off of the current CVS trunk. Minor and patch releases should be branches off of existing release branches. Spell out your policies regarding merges back to other releases or to the main branch on a web page and via e-mails to the engineers. If possible you should plan to check merge progress using an automation tool; with CVS you can peruse the CVSROOT/commitlog to check comments for bug reference and branch targets. Include a reference to a bug ID in release check-ins It’s relatively easy to tweak the CVS loginfo script to call a Perl script that can munge the checkin information into a webified version using the MIME::Lite CPAN Perl module. The script can then make the source-file reference and new revision-number links into CVSweb’s pages for the file and the revision-difference report respectively. Additional points are possible for parsing out the bug reference and making it a link into the web interface of the bug database. The net effect is that the MIME e-mail is sent to your software organisation for every check-in. If you have a policy of code reviewing check-ins, the revision-difference report makes that task much easier. Figure 1 shows an example of a source change report. The comment field has a link to the bug or problem report (1621 in this case), and below that are the CVSweb links to the source file and a revision-difference report (from 1.12.2.1 to 1.12.2.2). Finally, readers can see that the check-in was a file modification for the 3.1 release branch (with the name “Release_3_1_branch” as it exists in CVS). Note that if someone fails to add the bug ID on a check-in, you can go back and use “cvs admin” to fix the comment so that later reports work properly. Build policies With the source controlled, it’s time to tame the build. How often will you build? What is the format for the version of the release? What type of notification will take place each time a build is completed (or fails)? EE Times-India | December 2004 | eetindia.com Figure 2: An e-mail notification for release 2.5.0 QA build 15.0 Create a versioning format You’re going to be firing up many builds over the course of a release lifecycle. At the same time, your team might create other release builds or top of tree (trunk) builds. You will most assuredly need a versioning scheme of some sort. To maintain a logical versioning system, you could use a scheme that includes major, minor, patch, QA, and nightly build numbers. For instance, if a developer wants to check the 2.5.0 release nightly build #3 off of QA build 2, he would download build version 2.5.0.2.3. The build directory should be named something akin to its version, so that in the last example the build directory might be named “020500-002-003.” In this way nightly, QA, and release candidate (RC) builds can all peacefully coexist in a logical fashion. Mandate e-mail notification of builds For any build completed, an email notification should be sent to the entire software group. This e-mail should once again be a MIME-enabled web page that includes all of the files that have changed since the last build of the same type (nightly or QA). The e-mail should include a similar listing of all files checked into the branch and their new revision—and once again, this e-mail should have links to bugs, source files in CVSweb, and a revision difference section in CVSweb. In this way, engineers know what parts of the system have changed or what bugs were potentially fixed. To increase the report’s usefulness to QA especially, the generating script should pull differences between build tags for QA or RC builds. eetindia.com | December 2004 | EE Times-India Figure 2 is an example e-mail notification for release 2.5.0 QA build 15.0. Note that a CVS tag with this version information has also been created (“QA_ BTAG_V2_5_0_15_0”). The first nightly build after this build milestone would have the version 2.5.0.15.1. The QA department, on receipt of this build, can tell that the pesky foobar device has a fix applied against an outstanding bug and that the state of the bug is now “fixed.” Schedule nightly builds for current releases and the trunk You should schedule nightly builds for all releases that are currently in the works until they reach beta. This process ensures that first the release has not been broken (breaking a build is a serious offense with a dreaded penalty: the offender has to buy doughnuts for the rest of the organisation), and allows for a nightly “smoke test” or automated-test procedure to run. In addition to releases in the works, a nightly build of the main trunk (or HEAD branch in CVS) should be done to ensure its health. Nightly builds can devour a great deal of disc space, so pruning the builds to keep the latest five can maintain your free space. If no check-ins are detected for the branch or trunk, the build shouldn’t run (and the e-mail notification should indicate that no check-ins were found so the build was not done). QA and RC builds are usually done out-of-band to the nightly builds, but maintaining a valid, smoke-tested build can go far towards deriving a product that’s ready for testing or release. Figure 3: Graph of open versus fixed problems Engineering and QA policies Developers are buzzing away on features and fixes, you’ve got the nightly builds firing, so now it’s time to engage the QA people. Some organisations promote builds to QA fairly early in the cycle; others wait for a preordained code-freeze milestone before sending a build in for testing. Your mileage may vary, and you might be short a tyre if QA is working on other releases concurrently. Once you do decide to send QA a release, here are some tips on what to expect. QA and later-cycle RC builds Engineers use the nightly builds when testing embedded systems on their target processor. The builds can be loaded onto the target and debugged from that point. QA builds are done on a regular basis so that QA may perform their test suites and validate and fixed bugs. As a rule of thumb, a QA build is appropriate once a week so that QA has enough time to run their suites and the queue of fixed bugs doesn’t grow too large. The frequency of QA builds depends on the size of the project. RC builds, like QA builds, should be designated by an increment in the QA number that appears in both the version and the build-directory name. RC builds are created the same way that QA builds are, minus whatever internal QA designations that might appear in the build information. Avoid deleting RC and QA builds (especially if they aren’t backed-up), since they’re extremely useful for testing older builds for the existence of some classes of difficult bugs. Finally, both QA and RC builds should be tagged in CVS so they can be reliably reproduced. Having tags also helps generate build-difference reports so that engineering team and QA can agree on what fixes made it into a build. Turning knobs—time vs. resources Which knob are you going to turn? Do you have the luxury of time in your schedule to take your project to zero defects, or will you be forced to settle for zero fatal and major defects? Can you move resources either within QA or into QA to help the time crunch? The more challenging part of a release is the trade-off when confronted with a timeconstrained release or resourceconstrained release. While there are no short answers to problems such as these, once you know where you are and where you should be, you can gauge what knobs to turn. The key is understanding your current position. Hold regular meetings Hold regular meetings to consider the state of the release. You should require two types of meetings: the daily bug meeting and the weekly product review meeting. The first of meeting goes under different names at different companies but for our purposes I’ll call it a daily bug meeting. Whatever you call it, the meeting’s goal is defined as taking a regular and frequent (daily is the norm) snapshot of the current bug state of a release. Early in the cycle numerous bugs may be on the list so you should concentrate on a those with a high priority. The first part of the daily bug meeting should be devoted to the bug review. The meeting coordinator (either the project manager or the engineering manager in charge) runs a projector that displays a summary and individual bug reports. The latter part of the meeting should look at the bug queues (new, open, fixed) and outstanding issues to determine how the release is progressing along. The benefits of a regular review are many: • All of the responsible parties are up-to-date on the state of the release—thus it makes sense to include a representative from each of the stakeholders for the release—engineering, QA, product marketing, program management, manufacturing, and documentation if possible • Bug priorities can be assigned—later in the cycle this becomes a critical determinate of whether a bug will be fixed before shipping • Late feature requests can be judged for merit The second required meeting is a product review meeting, during which all of the stakeholders are required to meet and review EE Times-India | December 2004 | eetindia.com the progress of the product's release as a whole. This weekly meeting should be called by the marketing product manager and should be attended by the same people as the daily bug meeting plus some senior marketing and manufacturing heads. The purpose is to ensure the release stays on schedule and that the disparate groups coordinate their efforts (is the web site up, is the marketing collateral in shape, will the product launch at any trade shows?). Towards the end game So you have these wonderful policies implemented and your process is humming, new source developed, and at some point the QA builds start. What now? There should be some time in the schedule for a code freeze. That point may be determined by some heuristic like a graph showing open bugs vs. closed over time, or it may be the hard-and-fast requirement of a predetermined schedule. It all depends on what knobs you’re able to turn. At code freeze, the bar is set higher—software engineers are advised to check- in only the bugs at a threshold priority. QA must turn around and verify bug fixes with a high degree of reliability for both the fixes themselves and for the time that it takes to test them. Open vs. fixed graph A graph that displays open vs. fixed issues can provide insight on stability of the release (number of open issues) and QA loading (number of fixed bugs waiting close). For example, consider the graph in Figure 3; the open issues are being handled even though there are still five waiting for a fix. The to-be-fixed queue is rising, indicating that QA is lagging closing out bugs. A decision maker might look at this and decide to move some QA resources to help close out this release. Beta To get to this stage and have a product ready for customer trial, only minor issues should be outstanding (or at least one or two major issues with workarounds). The open vs. fixed graph should be trending downward, and the frequency of bug meetings should be reduced to once or eetindia.com | December 2004 | EE Times-India twice a week, and the majority of the discussion should be on coordination issues with the documentation group on how to characterise any lingering issues. Most of the engineering group should have already started on the next release with some possibly working on a follow-up maintenance or patch release and others on the next major release. First customer ship Ah, you are done. Or are you? Congratulate yourself and your team and then get ready to start pushing the rock uphill once more. Often the release team should host a post-release meeting to go over what did and did not succeed during the cycle. This is a great time to fine tune your release process. Along the way you might consider some additional questions. Have you started on a maintenance release? Have you checked with your sales or support engineer (quite a valuable and underutilised resource for internal status checks) on how the release is being accepted? Has anyone considered a customer survey to see if the release produced as it promised or if it missed the mark? Apollo 13 Many anxious moments will occur during any release; some may seem as bad as NASA Fight Director Gene Krantz had it during Apollo 13. He heroically rose above the issues and led a seemingly doomed space mission back to Earth. He and the rest of Mission Control were able to detach themselves emotionally from the problems to search for solutions. Engineering release managers must be made of the same mettle and must summon some introspection during the tough times to be able to wade through a mess and see your organisation’s “finest hour.” The recommendations within this article cannot in and of themselves help forge your mettle, but they can provide you with the tools that give you release status information, process, and controls in place. Email Send inquiry
© Copyright 2024