Testing page refresh

practices/testing.md

@@ -9,49 +9,76 @@
   * [Governance as a side effect](../patterns/governance-side-effect.md)
   * [Quality Metrics](../quality-checks.md)
 
-## Details
-* Quality is the whole team's responsibility
-  * Software testing is the responsibility of the whole team.
-  * Education on testing and testing principles is important to drive the right cultural behaviours.
+## General Testing Priniciples
+
+* **Quality is the whole team's responsibility**
+  * Education on testing and testing principles should be important to the whole team.
   * Quality approaches should be driven as a team and implemented by everyone.
-* Combining business knowledge with testing knowledge yields better quality outcomes
+
+* **Combining business knowledge with testing knowledge yields better quality outcomes**
   * Include business knowledge and critical thinking as part of assurance
   * Intrinsic knowledge and mindset of the team is key to driving quality outcomes
-  * Use Behaviour Driven Development (BDD) as an approach to align testing across the whole team and customer
-  * Consider BDD to encode acceptance criteria in business terms as automated tests where appropriate.
-    * This can be an effective communication and collaboration tool where the product owner is involved in reviewing detailed acceptance criteria (expressed as BDD tests).
-    * But if the PO is not involved in this way then well written unit and integration tests are often sufficient and avoid the extra layer of abstraction which BDD frameworks introduce.
-* Testing is prioritised based on risk
+
+* **Testing is prioritised based on risk**
   * A testing risk profile is defined and understood by the whole team, including the customer
   * Risk appetite should be worked across the whole team, including customers and/or users
   * Solution Assurance risks and Clinical Safety hazards must also be considered when prioritising risks
-* Use experiental / experimental / exploratory testing as well as formal methods
-  * Gamify testing - try multiple techniques and learn as you go
-  * Integrate chaos engineering / resilence testing into your pipelines e.g. using AWS Fault Injection Simulator (see [AWS FIS](../tools/aws-fis) for sample code)
-* Testing is assisted by automation with automated tests as a default (see [automate everything](../patterns/automate-everything.md)).
+
+* **Testing is context driven**
+  * Context should be considered when deciding on test techninques and tools to use.
+
+* **Testing is assisted by automation**
   * Appreciate that not everything can be automated
   * Identify good candidates for automation - particular focus on high risk and repeatable areas
-* Look to continually improve testing
+  * Automated tests should be used to provide confidence to all stakeholders.  This includes test analysts themselves who should be familiar with what the tests are doing to allow them to make decisions on what they want to test.
+  * Automated test packs should be maintained regularly to ensure they have suitable coverage, are efficient and providing correct results.
+  * Consider using testing tools to enhance other test techniques.
+    * Eg. using record and play tools to aid exploratory UI testing
+    * Eg. using API testing tools to aid exploratory API testing
+
+* **Testing should be continually improved**
+
+* **Testing is continuous**
   * Testing is a continuous activity, not a phase of delivery
+  * Testing can be performed in [all phases of a product's DevOps cycle](https://danashby.co.uk/2016/10/19/continuous-testing-in-devops/)
+
+
+
+## Test Automation Theory
+* Test should be added at the most appropriate level of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html) — in general, the lower the better and avoid testing the same thing in multiple levels.
+* Consider using the [agile testing quadrant](https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/) to help support a whole team approach around test planning and test automation.
+
+
+## Guidelines for different types of testing
+### Unit Testing Guidence
 * Tests act as documentation of the system's behaviour and should be clear enough to do this effectively.
 * Tests are written alongside or before whatever they are testing.
-* Tests are code (see [everything as code](../patterns/everything-as-code.md)).
 * Tests and test suites must be executable from the command line via a single command.
 * Tests do not rely on other tests (i.e. each test can be executed in isolation, tests can be run concurrently and in any order).
 * Tests only check a single condition.
 * Tests are idempotent (i.e. give the same result when re-run against the same code).
 * Test data is automatically generated when needed.
-* New functionality must be covered by passing automated tests (unless there is a legitimate reason not to).
-* Consider where best to add tests within the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html) — in general, the lower the better.
-* Avoid testing the same thing at multiple levels of the test pyramid.
-* Consider using the [agile testing quadrant](https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/) to help support a whole team approach around test planning and test automation.
-* Use stubs to test the behaviour of components in isolation — focus tests of multiple services on validating the interactions between services e.g. using consumer-driven contract testing.
 * Do not add tests of no value (e.g. tests of default getters/setters) just to increase the coverage level — 100% coverage is not a good use of time.
 * Tests must pass before code can be merged.
-* Non-functional requirements should also be covered by automated tests (e.g. security, performance, accessibility).
+* Use stubs to test the behaviour of components in isolation — focus tests of multiple services on validating the interactions between services e.g. using consumer-driven contract testing.
+### API Testing Guidence
+
+### UI Testing Guidence
+### Browser/OS Testing Guidence
 * Website user interfaces should be tested against an appropriate set of browsers and versions — in particular, no effort should be spent testing against unsupported browsers or unsupported versions of browsers. See supported browsers for [Staff](https://aalto.digital.nhs.uk/#/document/viewer/8c039de1-eec0-49cd-8af3-a97fed6a8bff?library=5464c07f-daf1-4eee-b9b6-22e6c4dfbbd0) and [Citizens](https://aalto.digital.nhs.uk/#/document/viewer/465e6d1b-f107-49eb-ad25-e72c0299d3a6?library=5464c07f-daf1-4eee-b9b6-22e6c4dfbbd0)
 
-## Further reading
 
+
+## Other Tools To Consider
+* BDD tools to encode acceptance criteria in business terms as automated tests where appropriate.
+* Chaos engineering / resilence testing e.g. using AWS Fault Injection Simulator (see [AWS FIS](../tools/aws-fis) for sample code)
+* Performance tools to check load, volume, soak and stress limits
+
+
+
+## Further reading and resources
 * [Test Driven Development: By Example](https://learning.oreilly.com/library/view/test-driven-development/0321146530/)
 * [xUnit Test Patterns: Refactoring Test Code](https://learning.oreilly.com/library/view/xunit-test-patterns/9780131495050/)
+* [Setting a foundation for successfull test automation](https://testautomationu.applitools.com/setting-a-foundation-for-successful-test-automation/), a short course by Angie Jones, who also has a talk on test automation strategy [based on game design](https://applitools.com/event/level-up-playing-the-automation-game/)
+* [Test Automation University](https://testautomationu.applitools.com/)
+* [Ministry of Testing](https://www.ministryoftesting.com/)