How do we fix testing
Download as PPTX, PDF•0 likes•378 views
The document details a presentation by Peter Varhol and Gerie Owen on improving software testing practices, highlighting the disconnect between theoretical knowledge and practical application. It emphasizes the need for testers to adopt a growth mindset and move beyond merely testing to requirements, advocating for a more comprehensive approach that includes understanding the context and evaluating risks. The presentation also outlines a roadmap for enhancing the value of testing in software development processes.
How do we fix testing
- 1. Slide 1Logo design: Sarah Cole Design SQGNE is made possible by the support of our sponsors: Oracle and Sun Integrated Systems Dec 2014
- 2. All-volunteer non-profit with no membership dues! Supported entirely by our sponsors… Over 1,275 members on LinkedIn, 875 Constant Contact Monthly meetings - Sept to July on 2nd Wed of month SQGNE Web site: www.sqgne.org Dec 2014
- 3. Officers / Hosts / Mission Mission To promote use of engineering and management techniques that lead to delivery of high quality software To disseminate concepts and techniques related to software quality engineering and software engineering process To provide a forum for discussion of concepts and techniques related to software quality engineering and the software engineering process To provide networking opportunities for software quality professionals Dec 2014 Current Officers: John Pustaver – Founder Steve Rakitin – President Stan Wrobel – Vice President Barbara Wioncek – Treasurer David Sullivan– Clerk Our gracious Hosts: Paul Ratty Tom Arakel Marge Shinkle Jack Guilderson At-large Directors: Robin Goldsmith Marge Shinkle Jim Turner
- 4. SQGNE 2014-15 Schedule Speaker Affiliation Date Topic Jim Parson Nick Olivo Checkpoint Tech. SmartBear Sept 10 Test Tool Bake-off featuring HP WebInspect Checkpoint Tech. and Test Complete Harish Narayan Oct 8 Leveraging lean systems development and testing to maximize business value Michael Mah QSM Associates Nov 12 Passion: What Software Teams Might Learn About Creating Lasting Change from Eco-pirates Peter Varhol Gerie Owen Dec 10 How do we fix testing? Stan Wrobel Mass. Dept of Education Jan 14 Agile Development and the changing roles for Business Analysts and QA Robin Goldsmith GoPro Management Feb 11 Is Exploratory Testing Really Better? Mario Moreira Mar 11 Agile – Bringing Testing to the Forefront Andrew Ambroise Apr 8 Shifting Quality to the Left; Negotiating Cleaner Results Gerie Owen NSTAR May 13 Testing the Internet of Things: Human Experience Testing David Heimann June 10 IEEE Standard 730-2014 Software Quality Assurance Processes Annual Election of Officers Everyone July 8 Annual Hot Topics Night… Dec 2014
- 5. Tonight’s Topic How do we fix testing? Peter Varhol and Gerie Owen Perhaps in no other professional field is the dichotomy between theory and practice more starkly different than in the realm of software testing. Researchers and thought leaders claim that testing requires a high level of cognitive and interpersonal skills, in order to make judgments about the ability of software to fulfill its operational goals. In their minds, testing is about assessing and communicating the risks involved in deploying software in a specific state. However, in many organizations, testing remains a necessary evil, and a cost to drive down as much as possible. Testing is merely a measure of conformance to requirements, without regard to the quality of requirements or how conformance is measured. This is certainly an important measure, but tells an incomplete story about the value of software in support of our business goals. We as testers often help to perpetuate the status quo. Although in many cases we realize we can add far more value than we do, we continue to perform testing in a manner that reduces our value in the software development process. This presentation looks at the state of the art as well of the state of common practice, and attempts to provide a rationale and roadmap whereby the practice of testing can be made more exciting and stimulating to the testing professional, as well as more valuable to the product and the organization. Bios Peter Varhol is a well-known writer and speaker on software and technology topics, having authored dozens of articles and spoken at a number of industry conferences and webcasts. He has advanced degrees in computer science, applied mathematics, and psychology, and is currently Principal at Technology Strategy Research. His past roles include technology journalist, software product manager, software developer, and university professor. Gerie Owen is a Test Architect who specializes in developing and managing test teams. She has implemented various Quality Assurance methodology models, and has developed, trained and mentored new teams from their inception. Gerie manages large, complex projects involving multiple applications, coordinates test teams across multiple time zones and delivers high quality projects on time and within budget. In her everyday working life, Gerie brings a cohesive team approach to testing. She has also presented at several conferences and authored articles on testing and quality assurance at www.qsma.com. Dec 2014
- 6. HOW DO WE FIX TESTING? WHY WE HAVE A GREAT OPPORTUNITY TO MAKE IMPROVEMENTS TO OUR CRAFT
- 7. THOUGHT LEADER Peter Varhol peter@petervarhol.com International speaker on technology topics Technology Evangelist and Writer, Product Manager, University Professor Someone who cares about getting testing right
- 8. PRACTITIONER Gerie Owen gerie@gerieowen.com Quality Assurance Consultant Speaker and Writer on Testing topics Experienced Tester, Test Lead, & Test Architect Marathon Runner & Running Coach
- 10. AGENDA • What we do • Why it’s not right • What thought leaders say we should do • What we need to do differently • How we get there • Summary and conclusions
- 11. OUR APPLICATIONS HAVE PROBLEMS
- 12. EVEN THE IMPORTANT ONES
- 13. WHAT HAPPENED TO TESTING? • We don’t have time for it • We have to get to market • We don’t have funding to test • We haven’t shown sufficient value • So we let our users test • That seems to work
- 14. INCREASINGLY THE RESULT IS FAILURE •And everyone knows it’s going to fail • But no one wants to say so •So we are all complicit • Because we know better
- 15. WHAT WE DO • Write test plans and procedures • Analyze and deconstruct requirements • Write test cases • Execute test cases and compare results to requirements • Make a determination on whether requirements have been met
- 16. WHAT WE DO • Testing finds bugs • Bugs are a mismatch between requirements and implementation • It’s an investigation with a yea or nay answer • If only it were that simple
- 17. WHAT WE DO • Classify bugs by priority and severity • Allowing room for negotiation and compromise • Provide metrics on test cases executed • Use test cases and metrics to make decisions on quality
- 18. WHAT WE DO • Many organizations call this Quality Assurance • But we can’t assure quality • Quality is the result of lifecycle processes • And we can’t build in quality • But we can test and evaluate • If not quality, then what?
- 19. AND WHAT IS QUALITY? • Is it really a mismatch of requirements and reality? • That makes several assumptions • The requirements are accurate and complete • The requirements are unambiguous • The requirements are all we care about • We talk about 100 percent test coverage • But do we really know?
- 20. WHAT THOUGHT LEADERS SAY WE SHOULD DO • Think • Act • Go beyond the requirements • Look at the context • Evaluate and make judgments • Communicate and advise
- 21. AND WHAT IS QUALITY? • Does testing to requirements enable us to find • Inaccurate or incomplete requirements? • Inefficient database calls? • Memory or object leaks? • Just plain bad design? • External or internal
- 22. AND WHAT IS QUALITY? • So we have to do more • Yes, it is our problem
- 23. WHAT THOUGHT LEADERS SAY WE SHOULD DO • Testing to requirements is a necessary part of the process • But . . . • It’s necessary but not sufficient
- 24. RULES OF THE ROAD • Testing requires understanding the problem domain intimately • Domain experts must be part of the team • Testing requires understanding the requirements • Testing requires thinking like a user • And being willing to take unpopular stands
- 25. AND BY THE WAY • A lot of testing today is done by users • Whether they know it or not • Whether they’re qualified or not • And they probably have no idea of the purpose or requirements • Let alone context
- 26. AND THAT’S A PROBLEM • Software that should be better isn’t • Often not even tested to requirements • And many software organizations don’t seem to care • We should care about that
- 27. WHAT SHOULD WE DO DIFFERENTLY? • Understand our own Mindsets
- 28. WHAT ARE MINDSETS? • Developed by psychologist Carol Dweck • How we mentally approach life and its challenges • Why brains and talent don’t bring success • How they can stand in the way of it • Why praising brains and talent doesn’t foster self-esteem and accomplishment, but jeopardizes them
- 29. WHAT IS A MINDSET •Broadly categorized in one of two ways • Fixed • Growth
- 30. IMPLICATIONS OF A FIXED MINDSET • I did my job right; someone else screwed up • That’s not really a bug, so I didn’t miss it • My test cases are complete • I test to requirements • You can’t ship this software
- 31. IMPLICATIONS OF A GROWTH MINDSET • I always have more to learn • I missed that bug, and I understand why • I can adapt my testing approach to new trends • I advise decision makers on risks of releasing software • I don’t ensure quality, whatever my title is
- 32. HOW DO WE FIX TESTING? •Personally Adopt a “Growth Mindset” •Evangelize Testing within the Context of Your Organization
- 33. YOUR GROWTH MINDSET • Use your intelligence • If something seems wrong to you, it probably does to others • Test Beyond Requirements • Exploratory Testing • Experiment with new test approaches • Field testing for mobile devices • Study your craft • Learn about new technologies and apply your knowledge
- 34. EVANGELIZE TESTING • APM is for testers too • Goes beyond the IT ops people • Do root cause analysis • Know why the problem exists • You’re the expert; let’s hear your opinion
- 35. UNDERSTAND YOUR ORGANIZATION • Depending on your organization • You may be thanked for going the extra mile • You may be criticized for exceeding your portfolio • When you feel burned out, discouraged or bored • Participate in the professional community
- 36. SUMMARY • Make it personal while remaining professional • Think beyond the requirements • Does it feel right? • Experiment • Work with development early on • You are all on the same side • Make recommendations
Editor's Notes
- #14: This implies that something has changed in the testing realm. But we’ve never had sufficient time or funding. I don’t think it isn’t that we haven’t shown sufficient value. It’s just that when thing go bad, everyone notices; when things go well, that’s the expectation and no one notices. I think there’s something to be said for that. But I think the time pressures are more today, too.
- #15: Tester’s aren’t necessarily all complicit or complacent. Our role as I have always understood it. The role of the tester is to provide clear information about the functionality, stability, performance, usability etc of the application under test and to assess and advise on the risk of deploying in its current condition. Then explain to me the role of testing in healthcare.gov. This should have been the most tested application on the planet. Instead, everyone just shut their eyes and mumbled.
- #19: Well as we move beyond Agile and into Continuous delivery and continuous deployment, the testers role actually shifts to quality assurance. In continuous delivery, the entire team from including developers, systems administrators and operations must be responsible for testing. Since the tester is the expert, it becomes the testers role to manage the testing process. And that’s what I think I say on this slide – that quality is the result of process.
- #23: Isn’t this old news? Given all the disruptive trends in testing including cloud, mobile, "human experience", Agile, and continuous delivery, aren't we already testing more than just requirements, sometimes without requirements and sometimes creating requirements.
- #33: However, you need to understand and evangelize your role in the context of your own organization. In some organizations, testers are involved testing post production defects and with APM. In other organizations, the testers are solely pre-production and the business takes over with post production testing. Some of this is dictated by SOX requirements. In TCoE's with offshore test teams, the TCoE experts might be involved post production whereas the offshore teams are tasked with pre prod testing only. Testers be able to compare notes by suggesting a weekly meeting or sharing through a collaboration tool etc.