30
CHAPTER 3
RESEARCH METHODOLOGY
Our research methodology is classified as exploratory. Its main
objectives were to determine the extent of adoption of software engineering
best practice and to identify appropriate organizations to involve in the field
experiments. This stage of the study can further be classified as correlational,
a form of descriptive study which involves collecting data in order to
determine whether and to what degree, an association exists between two or
more quantifiable variables.
3.1
JUSTIFICATION FOR THE PROPOSED METHODOLOGY
The proposed research was initiated after analyzing the available
literature and published data by various researchers. Literature relating to the
underlying theories of process improvement for small software organizations
and diffusion of innovation were reviewed and then current research about
software process, the best practice and improvement was summarized. From
the literature review, a gap was identified which necessitated more research
about improving software processes at low cost in small software
development firms.
The study indicates that Small and Medium Enterprises can benefit
from a low cost process improvement program with a restricted scope, a short
time frame to evaluation, and mentoring from external assessors and
consultants. It is also crucial that the firm is not disrupted by internal or
external events during the course of the software process improvement
program. Furthermore, this study provides a contribution to assessment
31
methods by validating the CMMI model and method and providing
recommendations to adopt the same by the SME. The outcomes from this
research have the potential to better equip practitioners and consultants to
undertake software process improvement at lesser cost, hence increasing the
success of small software development firms in domestic and global markets.
3.2
RESEARCH DESIGN
The research design enables to explain the research paradigm,
describe the approach and introduce the methods used to collect and analyse
the empirical measures. Also, the adopted research approach is justified.
The research design consists of three main stages : (1) a multiple
survey of five software organizations (2) formulation of the model (3) testing
the hypotheses of the model using scientific approaches like fuzzy logic and
cuckoo search optimization. The designed framework is depicted in figure 3.1.
DATA COLLECTION USING QUESTIONNAIRE
ANALYSIS OF DATA
CURRENT STATE OF
PROCESS MATURITY
DATA VALIDATION
MATLAB
RANK ORDERING METHOD
OPTIMIZATION
CUCKOO'S SEARCH
FINAL OUTPUT
TECHNIQUE
Figure 3.1 Research design
32
3.3
POPULATION AND SAMPLE
3.3.1
Unit of Analysis
In this study, a software organization is defined as a whole
company that has software development as its primary business and the data
obtained are used for the analysis.
The unit of analysis for the survey were five Indian software
organizations with an employee strength less than 50 numbers undertaking
software development. Table 3.1 presents the list of organizations selected for
the study. It describes their domain in which projects are being developed and
indicates employee strength of corresponding organization.
Table 3.1 Organization and number of employees
Domain expertise
Organization
of the organization
Technologies
used
Employee
Strength
Annual
turnover in
million
A
IT Software
services
JAVA
10
6
B
IT- Telecom
/Mobile services
Object oriented
technologies
49
35
C
IT- Network
security
JAVA and .NET
Framework
30
30
D
IT-ERP/CRM
PHP
37
40
E
IT Software
services
Object oriented
technologies
49
43
These five organizations were chosen based on the employee size,
turnover, nature of business involved like service, development or testing.
The main criterion taken for consideration was the number of employees,
which should not exceed 50 persons. Five persons in each organization were
considered in the data collection. They can be categorized as development
executives, team leaders and project leaders. The same questionnaire was
33
given to all and their responses were collected in a stipulated time. A model
was formulated to evaluate the collected data and the intermediate results
were arrived with the help of the model.
3.3.2
Profiles of the Participating Companies
Altogether five software enterprises have participated in the
assessments. The following background information is collected by using
questionnaire forms which company representatives independently filled and
returned to the author. The companies represent different sizes, ages and
application domains of software industry.
The total number of software work-related employees in
participating companies is 175. The smallest size of personnel is 10 and the
largest company has 49 employees. During the period of research work
carried out, the organizations had typically 5 software projects and the
average team size of the projects was 3-5 employees. Most of the
organizations and projects had one physical location. The total percentages of
production activities of organizations were, new development 44%,
maintenance 30% and other 26%.
Typically the companies key customers operated in the sectors like
education, municipal services and construction industry, but companies also
had markets from state authorities, agriculture and food products, military and
customer services (retail) sectors. The most typical development model used
was rapid prototyping and incremental or evolutionary model, but classic
waterfall model also existed widely.
34
3.4
DATA COLLECTION PROCEDURE
In this part of the study, the researcher used questionnaire as the
data collection instrument. The results of reliability and validity analyses
showed that the instrument has desirable psychometric properties. The
primary intent of the survey was thus, to test the existence of statistical
associations among the identified factors and SPI success, thus providing
quantitative evidence for the key factors of success in SPI, and the
generalizability of results.
3.4.1
Characteristics of the Questionnaire
Questionnaires may be used to collect regular or infrequent routine
data and data for specialized studies. Some of the data often obtained through
questionnaires include demographic characteristics, opinions of stakeholders,
organizational issues or management, general information etc.
A questionnaire requires respondents to fill out the form themselves
and so requires a high level of literacy. Where multiple languages are
common, questionnaires should be prepared using the major languages of the
target group. Special care needs to be taken in these cases to ensure accurate
translations.
In order to maximize return rates, questionnaires should be
designed to be as simple and clear as possible, with targeted sections and
questions. Most importantly, questionnaires should also be as short as
possible. If the questionnaire is being given to a sample population, then it
may be preferable to prepare several smaller, more targeted questionnaires,
each provided to a sub-group. If the questionnaire is used for a complete
enumeration, then special care needs to be taken to avoid overburdening the
35
respondent. If, for instance, several agencies require the same data, attempts
should be made to co-ordinate its collection to avoid duplication.
The information that can be obtained through questionnaires
consists of almost any data variable. Any socio-economic data can also be
obtained through questionnaires from a variety of sources.
Questionnaires, like interviews, can contain either structured
questions with blanks to be filled in, multiple choice questions, or they can
contain open-ended questions where the respondent is encouraged to reply at
length and choose their own focus to some extent.
To facilitate filling out forms and data entry in a structured format,
the form should ideally be machine-readable, or atleast laid out with data
fields clearly identifiable and responses pre-coded. In general, writing should
be reduced to a minimum (e.g. tick boxes, multiple choices), preferably being
limited to numerals. In an open-ended format, keywords and other structuring
procedures should be imposed later to facilitate database entry and analysis, if
necessary.
3.4.2
Types of Questionnaire
Questionnaires can be paper-based, or electronic. Structured
questionnaires are based predominantly on closed questions which produce
data that can be analysed quantitatively for patterns and trends. The agenda is
entirely predetermined by the evaluator and provides little flexibility for
respondents to qualify their answers.
Unstructured questionnaires, whilst still having a structured
sequence and focus predetermined by the evaluator, are based on open
questions allowing respondents the freedom to answer in their own words and
36
therefore to provide greater qualification in their response. Semi-structured
questionnaires take a mixed approach.
3.4.3
Application of Questionnaires
Questionnaires are used to get the feedback from a large number of
respondents, where it is impractical to collect feedback using other more
resource intensive methods. Questionnaires allow each of the participants an
opportunity to provide anonymous feedback on their experience.
Structured questionnaires allow for the exploration of patterns and
trends which help to describe the real time context and provide a measure of
respondents’ opinions, attitudes, feelings and perceptions about issues of
particular concern to the evaluator. They also help to identify patterns and
trends that merit further exploration using qualitative methods.
Unstructured questionnaires allow for richer feedback that may
provide insight into explanations for what is happening and participants’
opinions, attitudes, feelings, perceptions etc. They also allow for issues to
emerge that are not necessarily foreseen by the evaluator.
3.4.4
Strengths of Using Questionnaires
x
x
Questionnaires can be used to collect data quite quickly.
All participants can be given the opportunity to provide
feedback.
x
x
Feedback
is
generally
anonymous,
which
encourages
openness and honesty.
Structured questionnaire data can be processed by software
packages such as Excel and SPSS.
37
3.4.5
Limitations
x
Questions could be interpreted differently by respondents. It
can be difficult to design questionnaires to minimize this
x
effect.
Data processing and analysis for large samples can be time
consuming.
x
To motivate the potential respondents to complete the
questionnaires.
3.4.6
Practicalities
x
It is advisable to test questionnaires before using them, to
ensure that participants understand them and interpret them in
x
x
the way expected.
The respondents should not take much time to complete the
questionnaire .
It can be difficult to achieve an appropriate balance between
asking sufficient questions to get useful feedback, but not so
many questions that respondents get bored or feel they do not
have sufficient time to devote for completing them.
x
Complex questionnaire formats and structures can be difficult
for respondents to complete.
x
Long questionnaires can be answered more quickly if the
format of the question is fairly similar and respondents do not
need to keep learning how to complete different types of
questions, or have to make too many decisions about which
sections apply to them.
38
x
Different approaches to phrasing questions may have strengths
and weaknesses for collecting the data needed.
x
Questions will need to be in accordance with the relevant
guidance for the ethical handling of data. This involves not
collecting unnecessary personal data, and if the questionnaire
is intended to be anonymous, any personal data that would
enable respondents to be identified.
The following information are generally given to the respondents:
x
x
x
How to complete the questionnaire
How feedback will be used in the evaluation
How respondents can find out about the evaluation findings
and any action that will be taken as a consequence of the
x
x
3.5
evaluation findings
How confidentiality, anonymity, data protection etc will be
assured
How to return the form, to whom, where and by when
SELF DIAGNOSIS METHODS
Some of the self diagnosis methods are Goal Activities Response
Matrix, Extended Maturity Questionnaire and Directed Questionnaire.
3.5.1
Merits of Extended Maturity Questionnaire (EMQ)
The purpose of EMQ is to inform the status of the organization
with the answers given by the responder. It is also easy for the organization to
identify the goals and commitments that are partially fulfilled and which can
be included in the improvement plan in the self diagnostic process .Hence an
39
EMQ is designed as an instrument for data collection. Maturity questionnaires
are based on the Maturity Questionnaire developed by the SEI. This
questionnaire poses questions about specific KPAs, allowing for four possible
answers like “YES”, “NO”, “I DON’T KNOW”, and “NOT APPLIED”. In
the Maturity Questionnaire, all questions are related directly to goals and
commitments for each KPA, but not to specific activities that should be
performed to achieve such goals. The direct application of the Maturity
Questionnaire in small and micro organizations results in a great majority of
questions answered with a "NO", the main reason for this tendency being that
many of the goals proposed in the questionnaire are only partially achieved.
Therefore, modification of the Maturity Questionnaire with the addition of an
answer option will permit describing incomplete goals.
Extended questionnaire will guide the participants of a small
organization so that they can answer the questions in a way that reflects more
accurately the status of the organization. With the questionnaire the
participant is capable of identifying the goals and commitments that are being
only partially fulfilled.
An Extended Maturity Questionnaire is designed for assessing the
maturity level of the organization. Extended Maturity Questionnaire is used
to collect data about the organization. Out of 22 Key Process Areas (KPA)
under the four category process management, engineering, process
management and support, 13 process areas were selected and based on these
the questionnaire was framed.
40
3.6
QUESTIONNAIRE-BASED ASSESSMENT (QBA) METHOD
The
data
collection
is
done
by
Questionnaire
Based
Assessment(QBA) method. QBA is based primarily on the quantitative data
obtained from the questionnaire. These data are used to construct various
maturity profiles and a detailed report covering every question.
QBA is based on 13 process areas of level 2 of the CMMI. The
questions are answered by systems developers and project managers from the
organizations. Some key practices were augmented with additional questions.
Data collection starts with a meeting with all the respondents. At
this meeting the respondents are briefed about the organization’s ongoing
software process improvement project, the rationale of CMMI and the
assessment method. This is done in order to make sure that the respondents
understand the questions they will be answering, thus increasing the quality of
the answers.
The data collection then proceeds with the questionnaire, which is
answered by project managers and systems developers. This is accomplished
in a single session with everybody present in the same room, so that the
assessors can answer any question concerning the questionnaire. The aim is to
involve at least five employees in each organization. Answering the
questionnaire generally takes between 45 minutes and 2 hours.
QBA has two assessment instruments; the questionnaire and a
database for storing and analyzing the responses. The questionnaire is divided
into 7 sections. One section contains the introduction to the questionnaire and
the 6 remaining sections contain the questions related to the 6 key process
areas (KPA) at CMMI level 2.
41
3.7
SURVEY PRE-TEST
Prior to the data collection, the survey instrument was pre-tested to
enable clarification of constructs, to provide the means of operationalising
selected constructs, because pre-tests can be useful in ‘qualitatively
establishing the reliability, construct validity, and content validity of
measure’. In order to locate and correct weaknesses in the questionnaire, the
questionnaire is pre-tested critically.
During the pre-test, the respondents completed the questionnaire in
the presence of the researcher, and identified any difficulties with
interpretation of words or questions. Besides testing the reliability and
construct validity, the pre-tests served as ‘dry runs’ for the final
administration of the instrument. Emphasis was placed on identifying
questions that respondents may misunderstand, misinterpret or find difficult to
answer, as well as format or design problems.
The format of the questionnaire was changed to appear more
compact, and the few double-barrel questions were split to reduce ambiguity.
Finally, the format was changed so that the questionnaire fitted A4-sized
booklet. This was done to appeal to respondents as the booklet style which
could be completed quickly. Research indicates shorter questionnaires are
associated with higher response rates. Included in the survey were instructions
for respondents, presented in italic font. These comments were used to give
direction for using the scale, or were specific to the particular construct,
providing a common frame of reference.
3.8
QUESTIONNAIRE FORMAT
A total of 50 questions were formulated
to cover the thirteen
process areas of CMMI maturity level 2. The process areas and the number of
questions for each are given below:
42
The number of questions was determined beforehand, since the
number of questions reflects the importance of each process area. The
questions are based on the guidelines given by Software Engineering Institute
(SEI). It can be further increased to accommodate the other possible aspects
of specific and generic practices. Therefore this design choice can easily
accommodate further expansion in input system. Addition and deletion of
questions to cover other aspects of CMMI are also possible.
An Extended Maturity Questionnaire (EMQ) was designed with 5
point scale rating with options namely,
x
x
Yes
x
No
x
x
Partially Yes
Does Not Apply
Don’t Know
Based on the individual answers about each process area, marks
were allotted and a final score was calculated for each process area from the
five questionnaires using fuzzy logic. The two variable approach of fuzzy
logic is based on associative property of fuzzy sets.
3.9
EXECUTION OF SURVEY
The five organizations were chosen based on their employee
strength. Small software organizations with employee strength of 1-50
persons are taken for study. In the pre- talk for fifteen minutes, the aim of the
survey was clearly explained to the participants. Questions raised by the
participants were clarified so that they did not have any confusion or bias
while answering the questionnaire. Just like feedback form, the questionnaires
were given and collected in an anonymous method. Details like their
43
qualification, years of experience in programming and the domain they
currently work were collected, along with their designations.
First the participants were
seated comfortably so that they could
sit for a long duration without any discomfort. It was seen that room was
properly ventilated and lighted, so that they did not feel any discomfort for a
long stay. Secondly, they were asked to relax for a period of two minutes and
ease their mind. They were advised to pay much attention to fully understand
the question and then to answer without the omission of any question.
While collecting the questionnaire after completion, it was verified
that there was no omission of questions. An hour was given to each
participant to answer the fifty questions. Doubts raised during the survey were
clarified to ensure that the participants proceeded without any bias.
Refreshments were given to participants to encourage their long sit for an
hour. Participants were requested to do their work attentively and slowly so
that they finished
by the end of the hour. Earlier, they were informed they
would not be permitted to leave the survey before the end of the hour. It was
done to make certain that they gave legitimate answers for the given
questions.
3.10
RECORDING SURVEY RESPONSES
The filled questionnaire was collected
from the respondents. All
of this information was used to test for statistical significance. Information
was recorded in many ways. Charts and graphs were clear, visual ways to
record findings in many cases. A system was developed to facilitate data entry
of the responses. Data files of responses were then verified and imported into
MS Excel for analysis. This feedback enabled respondents to benchmark their
development practices against the aggregated responses of all the survey
responses.
44
3.11
DATA EVALUATION
Many evaluation methods for software development process are
available. While each of these methods has specific features, they share in
common the requirement for substantial time commitment (days or weeks)
and a trained team for the engagement. As a result, they are budgeted as
costly activities that interfere not only with daily activities for the
development team but the whole organization as well. The level of
interference and costs can be assimilated by organizations with a great
number of developers (because they are more risk averse) or those with
tradition in software development (because they are more familiar with the
risks in this activity). Direct and indirect costs and annoyances in the
organization workplace are the key factors that make the process evaluation
activity a tough decision for many companies.
In spite of a growing recognition, these factors inhibit the
dissemination of methods for software processes evaluation in small
organizations. The demand for software process improvement exists in both
large and small organizations. While the large organizations are reasonably
well served by the existing solutions, the opposite for the small organizations
is felt. The requirement for good software development processes for all
organization sizes inspired this research, but with our eyes set on the small
organizations.
This section presents the concepts related to the evaluation process
execution and short descriptions from methods found in the literature.
Evaluation is a formal and structured method to examine the way how
software projects are built and maintained.