30 CHAPTER 3 RESEARCH METHODOLOGY Our research methodology is classified as exploratory. Its main objectives were to determine the extent of adoption of software engineering best practice and to identify appropriate organizations to involve in the field experiments. This stage of the study can further be classified as correlational, a form of descriptive study which involves collecting data in order to determine whether and to what degree, an association exists between two or more quantifiable variables. 3.1 JUSTIFICATION FOR THE PROPOSED METHODOLOGY The proposed research was initiated after analyzing the available literature and published data by various researchers. Literature relating to the underlying theories of process improvement for small software organizations and diffusion of innovation were reviewed and then current research about software process, the best practice and improvement was summarized. From the literature review, a gap was identified which necessitated more research about improving software processes at low cost in small software development firms. The study indicates that Small and Medium Enterprises can benefit from a low cost process improvement program with a restricted scope, a short time frame to evaluation, and mentoring from external assessors and consultants. It is also crucial that the firm is not disrupted by internal or external events during the course of the software process improvement program. Furthermore, this study provides a contribution to assessment 31 methods by validating the CMMI model and method and providing recommendations to adopt the same by the SME. The outcomes from this research have the potential to better equip practitioners and consultants to undertake software process improvement at lesser cost, hence increasing the success of small software development firms in domestic and global markets. 3.2 RESEARCH DESIGN The research design enables to explain the research paradigm, describe the approach and introduce the methods used to collect and analyse the empirical measures. Also, the adopted research approach is justified. The research design consists of three main stages : (1) a multiple survey of five software organizations (2) formulation of the model (3) testing the hypotheses of the model using scientific approaches like fuzzy logic and cuckoo search optimization. The designed framework is depicted in figure 3.1. DATA COLLECTION USING QUESTIONNAIRE ANALYSIS OF DATA CURRENT STATE OF PROCESS MATURITY DATA VALIDATION MATLAB RANK ORDERING METHOD OPTIMIZATION CUCKOO'S SEARCH FINAL OUTPUT TECHNIQUE Figure 3.1 Research design 32 3.3 POPULATION AND SAMPLE 3.3.1 Unit of Analysis In this study, a software organization is defined as a whole company that has software development as its primary business and the data obtained are used for the analysis. The unit of analysis for the survey were five Indian software organizations with an employee strength less than 50 numbers undertaking software development. Table 3.1 presents the list of organizations selected for the study. It describes their domain in which projects are being developed and indicates employee strength of corresponding organization. Table 3.1 Organization and number of employees Domain expertise Organization of the organization Technologies used Employee Strength Annual turnover in million A IT Software services JAVA 10 6 B IT- Telecom /Mobile services Object oriented technologies 49 35 C IT- Network security JAVA and .NET Framework 30 30 D IT-ERP/CRM PHP 37 40 E IT Software services Object oriented technologies 49 43 These five organizations were chosen based on the employee size, turnover, nature of business involved like service, development or testing. The main criterion taken for consideration was the number of employees, which should not exceed 50 persons. Five persons in each organization were considered in the data collection. They can be categorized as development executives, team leaders and project leaders. The same questionnaire was 33 given to all and their responses were collected in a stipulated time. A model was formulated to evaluate the collected data and the intermediate results were arrived with the help of the model. 3.3.2 Profiles of the Participating Companies Altogether five software enterprises have participated in the assessments. The following background information is collected by using questionnaire forms which company representatives independently filled and returned to the author. The companies represent different sizes, ages and application domains of software industry. The total number of software work-related employees in participating companies is 175. The smallest size of personnel is 10 and the largest company has 49 employees. During the period of research work carried out, the organizations had typically 5 software projects and the average team size of the projects was 3-5 employees. Most of the organizations and projects had one physical location. The total percentages of production activities of organizations were, new development 44%, maintenance 30% and other 26%. Typically the companies key customers operated in the sectors like education, municipal services and construction industry, but companies also had markets from state authorities, agriculture and food products, military and customer services (retail) sectors. The most typical development model used was rapid prototyping and incremental or evolutionary model, but classic waterfall model also existed widely. 34 3.4 DATA COLLECTION PROCEDURE In this part of the study, the researcher used questionnaire as the data collection instrument. The results of reliability and validity analyses showed that the instrument has desirable psychometric properties. The primary intent of the survey was thus, to test the existence of statistical associations among the identified factors and SPI success, thus providing quantitative evidence for the key factors of success in SPI, and the generalizability of results. 3.4.1 Characteristics of the Questionnaire Questionnaires may be used to collect regular or infrequent routine data and data for specialized studies. Some of the data often obtained through questionnaires include demographic characteristics, opinions of stakeholders, organizational issues or management, general information etc. A questionnaire requires respondents to fill out the form themselves and so requires a high level of literacy. Where multiple languages are common, questionnaires should be prepared using the major languages of the target group. Special care needs to be taken in these cases to ensure accurate translations. In order to maximize return rates, questionnaires should be designed to be as simple and clear as possible, with targeted sections and questions. Most importantly, questionnaires should also be as short as possible. If the questionnaire is being given to a sample population, then it may be preferable to prepare several smaller, more targeted questionnaires, each provided to a sub-group. If the questionnaire is used for a complete enumeration, then special care needs to be taken to avoid overburdening the 35 respondent. If, for instance, several agencies require the same data, attempts should be made to co-ordinate its collection to avoid duplication. The information that can be obtained through questionnaires consists of almost any data variable. Any socio-economic data can also be obtained through questionnaires from a variety of sources. Questionnaires, like interviews, can contain either structured questions with blanks to be filled in, multiple choice questions, or they can contain open-ended questions where the respondent is encouraged to reply at length and choose their own focus to some extent. To facilitate filling out forms and data entry in a structured format, the form should ideally be machine-readable, or atleast laid out with data fields clearly identifiable and responses pre-coded. In general, writing should be reduced to a minimum (e.g. tick boxes, multiple choices), preferably being limited to numerals. In an open-ended format, keywords and other structuring procedures should be imposed later to facilitate database entry and analysis, if necessary. 3.4.2 Types of Questionnaire Questionnaires can be paper-based, or electronic. Structured questionnaires are based predominantly on closed questions which produce data that can be analysed quantitatively for patterns and trends. The agenda is entirely predetermined by the evaluator and provides little flexibility for respondents to qualify their answers. Unstructured questionnaires, whilst still having a structured sequence and focus predetermined by the evaluator, are based on open questions allowing respondents the freedom to answer in their own words and 36 therefore to provide greater qualification in their response. Semi-structured questionnaires take a mixed approach. 3.4.3 Application of Questionnaires Questionnaires are used to get the feedback from a large number of respondents, where it is impractical to collect feedback using other more resource intensive methods. Questionnaires allow each of the participants an opportunity to provide anonymous feedback on their experience. Structured questionnaires allow for the exploration of patterns and trends which help to describe the real time context and provide a measure of respondents’ opinions, attitudes, feelings and perceptions about issues of particular concern to the evaluator. They also help to identify patterns and trends that merit further exploration using qualitative methods. Unstructured questionnaires allow for richer feedback that may provide insight into explanations for what is happening and participants’ opinions, attitudes, feelings, perceptions etc. They also allow for issues to emerge that are not necessarily foreseen by the evaluator. 3.4.4 Strengths of Using Questionnaires x x Questionnaires can be used to collect data quite quickly. All participants can be given the opportunity to provide feedback. x x Feedback is generally anonymous, which encourages openness and honesty. Structured questionnaire data can be processed by software packages such as Excel and SPSS. 37 3.4.5 Limitations x Questions could be interpreted differently by respondents. It can be difficult to design questionnaires to minimize this x effect. Data processing and analysis for large samples can be time consuming. x To motivate the potential respondents to complete the questionnaires. 3.4.6 Practicalities x It is advisable to test questionnaires before using them, to ensure that participants understand them and interpret them in x x the way expected. The respondents should not take much time to complete the questionnaire . It can be difficult to achieve an appropriate balance between asking sufficient questions to get useful feedback, but not so many questions that respondents get bored or feel they do not have sufficient time to devote for completing them. x Complex questionnaire formats and structures can be difficult for respondents to complete. x Long questionnaires can be answered more quickly if the format of the question is fairly similar and respondents do not need to keep learning how to complete different types of questions, or have to make too many decisions about which sections apply to them. 38 x Different approaches to phrasing questions may have strengths and weaknesses for collecting the data needed. x Questions will need to be in accordance with the relevant guidance for the ethical handling of data. This involves not collecting unnecessary personal data, and if the questionnaire is intended to be anonymous, any personal data that would enable respondents to be identified. The following information are generally given to the respondents: x x x How to complete the questionnaire How feedback will be used in the evaluation How respondents can find out about the evaluation findings and any action that will be taken as a consequence of the x x 3.5 evaluation findings How confidentiality, anonymity, data protection etc will be assured How to return the form, to whom, where and by when SELF DIAGNOSIS METHODS Some of the self diagnosis methods are Goal Activities Response Matrix, Extended Maturity Questionnaire and Directed Questionnaire. 3.5.1 Merits of Extended Maturity Questionnaire (EMQ) The purpose of EMQ is to inform the status of the organization with the answers given by the responder. It is also easy for the organization to identify the goals and commitments that are partially fulfilled and which can be included in the improvement plan in the self diagnostic process .Hence an 39 EMQ is designed as an instrument for data collection. Maturity questionnaires are based on the Maturity Questionnaire developed by the SEI. This questionnaire poses questions about specific KPAs, allowing for four possible answers like “YES”, “NO”, “I DON’T KNOW”, and “NOT APPLIED”. In the Maturity Questionnaire, all questions are related directly to goals and commitments for each KPA, but not to specific activities that should be performed to achieve such goals. The direct application of the Maturity Questionnaire in small and micro organizations results in a great majority of questions answered with a "NO", the main reason for this tendency being that many of the goals proposed in the questionnaire are only partially achieved. Therefore, modification of the Maturity Questionnaire with the addition of an answer option will permit describing incomplete goals. Extended questionnaire will guide the participants of a small organization so that they can answer the questions in a way that reflects more accurately the status of the organization. With the questionnaire the participant is capable of identifying the goals and commitments that are being only partially fulfilled. An Extended Maturity Questionnaire is designed for assessing the maturity level of the organization. Extended Maturity Questionnaire is used to collect data about the organization. Out of 22 Key Process Areas (KPA) under the four category process management, engineering, process management and support, 13 process areas were selected and based on these the questionnaire was framed. 40 3.6 QUESTIONNAIRE-BASED ASSESSMENT (QBA) METHOD The data collection is done by Questionnaire Based Assessment(QBA) method. QBA is based primarily on the quantitative data obtained from the questionnaire. These data are used to construct various maturity profiles and a detailed report covering every question. QBA is based on 13 process areas of level 2 of the CMMI. The questions are answered by systems developers and project managers from the organizations. Some key practices were augmented with additional questions. Data collection starts with a meeting with all the respondents. At this meeting the respondents are briefed about the organization’s ongoing software process improvement project, the rationale of CMMI and the assessment method. This is done in order to make sure that the respondents understand the questions they will be answering, thus increasing the quality of the answers. The data collection then proceeds with the questionnaire, which is answered by project managers and systems developers. This is accomplished in a single session with everybody present in the same room, so that the assessors can answer any question concerning the questionnaire. The aim is to involve at least five employees in each organization. Answering the questionnaire generally takes between 45 minutes and 2 hours. QBA has two assessment instruments; the questionnaire and a database for storing and analyzing the responses. The questionnaire is divided into 7 sections. One section contains the introduction to the questionnaire and the 6 remaining sections contain the questions related to the 6 key process areas (KPA) at CMMI level 2. 41 3.7 SURVEY PRE-TEST Prior to the data collection, the survey instrument was pre-tested to enable clarification of constructs, to provide the means of operationalising selected constructs, because pre-tests can be useful in ‘qualitatively establishing the reliability, construct validity, and content validity of measure’. In order to locate and correct weaknesses in the questionnaire, the questionnaire is pre-tested critically. During the pre-test, the respondents completed the questionnaire in the presence of the researcher, and identified any difficulties with interpretation of words or questions. Besides testing the reliability and construct validity, the pre-tests served as ‘dry runs’ for the final administration of the instrument. Emphasis was placed on identifying questions that respondents may misunderstand, misinterpret or find difficult to answer, as well as format or design problems. The format of the questionnaire was changed to appear more compact, and the few double-barrel questions were split to reduce ambiguity. Finally, the format was changed so that the questionnaire fitted A4-sized booklet. This was done to appeal to respondents as the booklet style which could be completed quickly. Research indicates shorter questionnaires are associated with higher response rates. Included in the survey were instructions for respondents, presented in italic font. These comments were used to give direction for using the scale, or were specific to the particular construct, providing a common frame of reference. 3.8 QUESTIONNAIRE FORMAT A total of 50 questions were formulated to cover the thirteen process areas of CMMI maturity level 2. The process areas and the number of questions for each are given below: 42 The number of questions was determined beforehand, since the number of questions reflects the importance of each process area. The questions are based on the guidelines given by Software Engineering Institute (SEI). It can be further increased to accommodate the other possible aspects of specific and generic practices. Therefore this design choice can easily accommodate further expansion in input system. Addition and deletion of questions to cover other aspects of CMMI are also possible. An Extended Maturity Questionnaire (EMQ) was designed with 5 point scale rating with options namely, x x Yes x No x x Partially Yes Does Not Apply Don’t Know Based on the individual answers about each process area, marks were allotted and a final score was calculated for each process area from the five questionnaires using fuzzy logic. The two variable approach of fuzzy logic is based on associative property of fuzzy sets. 3.9 EXECUTION OF SURVEY The five organizations were chosen based on their employee strength. Small software organizations with employee strength of 1-50 persons are taken for study. In the pre- talk for fifteen minutes, the aim of the survey was clearly explained to the participants. Questions raised by the participants were clarified so that they did not have any confusion or bias while answering the questionnaire. Just like feedback form, the questionnaires were given and collected in an anonymous method. Details like their 43 qualification, years of experience in programming and the domain they currently work were collected, along with their designations. First the participants were seated comfortably so that they could sit for a long duration without any discomfort. It was seen that room was properly ventilated and lighted, so that they did not feel any discomfort for a long stay. Secondly, they were asked to relax for a period of two minutes and ease their mind. They were advised to pay much attention to fully understand the question and then to answer without the omission of any question. While collecting the questionnaire after completion, it was verified that there was no omission of questions. An hour was given to each participant to answer the fifty questions. Doubts raised during the survey were clarified to ensure that the participants proceeded without any bias. Refreshments were given to participants to encourage their long sit for an hour. Participants were requested to do their work attentively and slowly so that they finished by the end of the hour. Earlier, they were informed they would not be permitted to leave the survey before the end of the hour. It was done to make certain that they gave legitimate answers for the given questions. 3.10 RECORDING SURVEY RESPONSES The filled questionnaire was collected from the respondents. All of this information was used to test for statistical significance. Information was recorded in many ways. Charts and graphs were clear, visual ways to record findings in many cases. A system was developed to facilitate data entry of the responses. Data files of responses were then verified and imported into MS Excel for analysis. This feedback enabled respondents to benchmark their development practices against the aggregated responses of all the survey responses. 44 3.11 DATA EVALUATION Many evaluation methods for software development process are available. While each of these methods has specific features, they share in common the requirement for substantial time commitment (days or weeks) and a trained team for the engagement. As a result, they are budgeted as costly activities that interfere not only with daily activities for the development team but the whole organization as well. The level of interference and costs can be assimilated by organizations with a great number of developers (because they are more risk averse) or those with tradition in software development (because they are more familiar with the risks in this activity). Direct and indirect costs and annoyances in the organization workplace are the key factors that make the process evaluation activity a tough decision for many companies. In spite of a growing recognition, these factors inhibit the dissemination of methods for software processes evaluation in small organizations. The demand for software process improvement exists in both large and small organizations. While the large organizations are reasonably well served by the existing solutions, the opposite for the small organizations is felt. The requirement for good software development processes for all organization sizes inspired this research, but with our eyes set on the small organizations. This section presents the concepts related to the evaluation process execution and short descriptions from methods found in the literature. Evaluation is a formal and structured method to examine the way how software projects are built and maintained.
© Copyright 2024