6 Reasons Why Firms Should Enhance Quality Assurance Testing. Market participants are under heavy pressure to ensure their technology is stable and performing as intended. But often participants rush to launch services to the marketplace, and are cutting corners on performance testing. In their push to quickly release new functionality – whether in response to regulations, exchange updates or customer demand, firms may be putting themselves in a dangerous position. Poorly performing systems can cause disastrous errors or trading outages that can damage a firm’s reputation— possibly even jolting investors enough that they take their business elsewhere. That’s why thorough quality assurance (QA) testing is essential, giving developers the opportunity to fix minor glitches before new technology is unleashed on customers. Proper QA practices can also help firms obtain a competitive edge in a highly complex trading ecosystem. It can improve customer acquisition and retention rates through helping deliver cuttingedge services to customers faster and more reliably. 1. Speed Time to Market for Improved Competitiveness In this fiercely competitive landscape, financial firms need to be evervigilant of and responsive to evolving market and customer demands. In large part, this means being able to deliver innovative offerings in a reasonable timeframe. But often the process of delivering new solutions, new functionality or major upgrades is involved and lengthy – both with extensive development requirements and thorough QA. Automation is a viable way for firms to improve time to market and keep systems quality high by speeding up regression testing. Using automating testing, organizations can deliver quickly while preventing major defects that could have a negative impact on customer relationships. www.camerontecgroup.com CameronTec Group Authors Josh Tolman Deputy CEO CameronTec Group 6 Reasons Why Firms Should Enhance Quality Assurance Testing Direct Edge, a US-based stock exchange, is one firm using automating testing for just this purpose. They automated regression testing not only to speed up the overall process, but also so they could add more sophisticated test cases. Rene Ovalle, Head of Quality Assurance at Direct Edge commented, “Introducing greater sophistication into QA has helped us reduce the regression testing cycle from months down to just 33 hours. Not only is the process exponentially faster now, but it is also far more thorough.” Improved efficiency is a prime area where automated testing shows its value. It’s not uncommon for firms to see a 40% reduction in time spent on the end-to-end testing process. This means they can perform platform migrations faster, giving customers an uninterrupted user experience. Josh Nardo Global Head of Services CameronTec Group 2. Lower TCO to Free-up Budget Rene Ovalle Head of Quality Assurance Direct Edge Some firms are skeptical about implementing automated testing technology, considering the initial upfront investment involved. They think it’s more affordable to simply hire a team of testers to handle the job. “It’s true that in the short–term, manual testing is the lower cost option. However, the overhead involved in sustaining this approach over time is significantly higher when stacked up against automation,” said Josh Tolman, Deputy CEO at CameronTec Group. ...do you want the same people who are building your software to also be building your testing tools? Likewise, building testing tools inhouse can present considerable longterm expenses. “For example,” said Tolman, “IT staff will need to devote a lot of time to not only building, but also keeping these systems up-to-date as regulations and other requirements change. That can become quite arduous. Furthermore, if building testing software is not within their area of expertise, then the quality of the system may suffer. Think about it like this— do you want the same people who are building your software to also be building your testing tools?” According to Tolman, the average firm will see a 155% return on investment (ROI) after the first year using a third party testing tool, and a 327% ROI by the end of year three. The total cost of ownership (TCO) attached to automation is not only lower because it demands fewer employee resources, but it also minimizes the possibility of costly mistakes related to lack of domain knowledge or human error. Reduced TCO can also freeup budget, allowing firms to redirect money to other value-added areas of the business. Tolman used an ROI calculator developed by Greenline to arrive at these numbers. Of course, results will vary by firm. Direct Edge’s experience with automated QA testing proves Tolman’s point. “A lot of our ROI can be seen in our improved productivity rates and how much more efficiently we’re handling testing. Prior to automation, all testing had to be completed during our business hours, but now it can be done 24 hours per day, 7 days per week. As a result, we’re now able to deliver new releases to our clients in a matter of weeks, not months,” said Ovalle. 3. Improve Testing Accuracy for Better Stability Firms that use automated testing systems stand to dramatically enhance quality assurance. If they’re designed well, automated processes can be far more accurate, reliable and consistent than manual processes. Furthermore, sophisticated testing tools can introduce consistent, repeatable processes into a testing environment that can improve systems stability. “With automation, many firms have reported a 75% reduction in production defects when compared to manual methods. Minimizing such defects can save sell-sides a significant amount of money – both in terms of resources needed to fix errors and pay-outs to clients in the event of trading errors,” commented Josh Nardo, Global Head of Services at CameronTec Group. In essence, an automated approach will help make the testing process less prone to error, and thereby more cost-efficient. 4. Respond Nimbly to Regulatory Pressures Today’s regulatory landscape is in a state of flux, forcing firms to repeatedly modify their front, middle and back office systems to meet evolving compliance requirements, often with short deadlines. Every change introduces new system integrity risks. In addition, regulators are putting added pressure on these firms to establish thorough testing practices and to ensure the operational integrity of their systems. www.camerontecgroup.com Compliance is further complicated by the principles-based rather than prescriptive regulations. “We’ve observed that while regulating bodies do set forth some guidelines for compliance, they have not supplied precise rules. This leaves room for interpretation on the part of exchanges that are striving to comply with the latest standards,” said Ovalle. “At Direct Edge, we’ve made it a point to be a model in this space. Across our organization, we’re trying to create repeatable, sustainable, adaptable processes that help simplify the compliance burden.” Automated testing is an essential ingredient that can help organizations stay on top of regulatory pressures. “Automation can help you tackle the compliance challenge in a few different ways,” said Tolman. “It allows you to create best practices in testing and auditable processes. It can also establish and enforce consistent testing procedures. Additionally, automation can make testing more comprehensive, while simultaneously making the process less manually intensive and time-consuming. Therefore, more testing gets done, and the tests are less prone to error.” 5. Reduce Testing Costs and Improve Testing Maturity Some sell-side firms think outsourcing technology to a vendor would be more expensive than handling it internally. But this is largely a misconception. Testing complex financial technology requires specialized knowledge and experience. It’s neither simple nor inexpensive to find this skilled labor pool. When you compare it to the cost of recruiting, training and retaining this type of talent; leveraging a vendor with proven knowledge in this space and skilled resources can ultimately be more affordable. Nardo comments, “Firms that choose to outsource automated testing to an experienced provider will typically see an overall 30 to 60% process savings. Not only is the process less expensive, but leveraging subject matter experts can also make tests more sophisticated.” ...we’ve been able to get our testing environment up and running quicker and more cost-effectively than if we had managed the entire process in-house. Direct Edge agrees. Reducing overhead and getting access to domain experts were the primary reasons they chose to outsource. “When setting up our automated testing environment, we elected to use Greenline’s support resources. As a result, we’ve been able to get our testing environment up and running quicker and more costeffectively than if we had managed the entire process in-house. Greenline’s support team has also been very helpful in sharing their knowledge with our internal staff,” said Ovalle. 6. Combine Automation and Processes for Enhanced Testing Organizations everywhere are looking to enhance the quality of their processes in order to lower risk and keep operating costs in check. But when it comes to testing, a manual approach does not offer the level of sophistication necessary to create standardized, enforceable processes. • Level 5: Those that use full automation and test scripts According to Tolman, there are five primary categories organizations fall under in terms of testing processes: “I’ve found that most organizations are testing at the level of four or below. But they really need to be at level five, using full automation combined with standard processes,” said Tolman. “Sophisticated testing lets developers optimize test cases from the very beginning of a project, instead of waiting until just prior to release. This way firms are able to constantly optimize and refine their tests, ultimately leading to a much higher ROI.” • Level 0: Those that don’t test at all, or do so only in production • Level 1: Those where there is no QA department, and the developers handle testing • Level 2: Those where there is a QA department, but they do testing manually and are typically not testing all cases as a result of time restraints and limitations in skillsets • Level 3: Those that build point testing tools in-house (Since these are built internally, problems can exist in the testing tools themselves.) • Level 4: Those that use one or two third-party tools, but only use them at a basic level Direct Edge has found a good deal of success utilizing a more intensive testing approach. “Engineering uniform testing processes has helped us mitigate much of the risk involved in delivering new functionality to the market. Currently, we’ve been able to automate a total of 20k regression test cases, and insert standardization into the process of how we move through production,” commented Ovalle. Meeting Today’s Demands In this unpredictable environment characterized by constant change and unrelenting competition, firms need to find new ways of attracting and retaining trading flow. One way of doing this is through introducing automation into the quality assurance process, which can help organizations enhance their performance, while meeting the pressures of offering improved functionality to clients. “We’re proud that our firm has one of the best uptimes of any other exchange in the industry. A large part of this we attribute to having rigorous testing practices in place that help ensure the quality and reliability of the service we offer our clients,” said Ovalle. www.camerontecgroup.com