Adapting testing standards for the world of
world of data science:
Best practices for
a reliable
AI testing strategy
Let’s face the new challenges in the testing world together with solid knowledge of data management – from pre-processing to corner cases
Contact AI team
Follow us
Develop a robust AI testing strategy with a focus on precise data analysis
Optimize your AI implementation through extensive testing to ensure that your models perform reliably even in challenging scenarios.
We advise you on the development of a comprehensive test strategy for AI and machine learning-based systems:
- Joint design of test strategies
- Identify potential exceptions (corner cases)
- Excellent test experts for explorative practical testing
- Comprehensive tool evaluation
podcast
What data can be used to test and validate an AI?
QualityHeroes Podcast Episode 18: Our test expert Bettina Stühle-Stein reports on a major IT project for autonomous driving.
AI protection project:
Corner cases
The complexity of testing artificial intelligence is particularly evident in our largest AI project, a project funded by the German government, which aims to develop a generally valid validation methodology for AI in autonomous vehicles. We are responsible for systematically taking into account so-called corner cases, i.e. critical exceptional situations both within road traffic and within the AI system and sensor technology.
You can find more information about the project here: Completion of the project: AI safeguarding – safety argumentation for autonomous driving in Berlin.
Improved pedestrian detection with computer vision
A reliable process for the holistic evaluation of the various AI functions was developed for a verifiable safety argumentation.
Targeted training of the AI with modern simulation technologies has improved pedestrian recognition even under difficult conditions.
Example of pedestrian detection to set up and illustrate the chain of reasoning
Testing of existing algorithms for pedestrian detection
Identification of critical scenarios (corner cases), e.g. altered incidence of light or temporarily obscured view
The test strategy is the most important building block for effectively and efficiently testing AI- and ML-based systems.
While the testing of “traditional systems” (web, mobile, API, etc.) can be based on various existing test strategies, this is not the case with machine learning-based systems. Instead, the focus must be placed on the data, its structure, semantics, etc. as well as the data generation techniques. Test concepts form the basis for cross-project standards in the area of quality and testing. They form the basis for the project-related derivation of test strategies and are therefore one of the cornerstones of testing activities. We advise product teams on how such a test strategy can be designed and how data preparation and validation pipelines can be created. Our consulting is carried out in an agile manner using methods such as the OnePager, the 10-minute test strategy and risk storming.
AI test strategy
Exceptional cases
Corner cases
When testing AI- and ML-based systems, the systematic identification of potential exceptions, the “corner cases”, is extremely relevant.
We advise product teams on the identification, design and testing of corner cases and are involved in the training and testing process as well as during the application phase.
Workshop
In 5 steps to your
AI test strategy
Learning journey AI test strategy for companies
The learning content is sharpened through an initial discovery workshop and can be supplemented and adapted as required during the course of the journey. To this end, we will involve all relevant roles in the development process on an interdisciplinary basis.
Process derivation: From the requirement via the test data to the production system.
Risk analysis along the derived process.
A cross-content aspect here will be the creation and use of the Operational Design Domain (ODD) of the application, particularly with regard to content, metrics and monitoring, risks and mitigation strategies, shared knowledge and the structure of the test pyramid.
Individual development of an AI test concept for the project.
Derivation of a modular AI test strategy.
Quality of the test data
When testing ML-based systems, the focus is much more on data than is the case with “traditional” test approaches. The quality assessment of the training, test and validation data used requires a solid knowledge of both data science and data management.
We offer our customers a combination of both worlds by providing methods and tools for designing and building a pipeline for test data in projects.
Hands-on testing
Every successful test strategy is a mixture of carefully crafted test cases, test automation and exploratory testing. The latter in particular requires a very good understanding of the domain and excellent testing skills.
We offer our customers a range of excellent test experts for exploratory field testing to identify the most important exceptional situations for ML-based systems.
ML-based test tools
The various techniques known from data science and machine learning can also be used to improve the testing process.
The market for ML-based testing tools is slowly evolving. We provide our customers with an overview and tool evaluation based on their requirements and our knowledge of this market.
Revolutionize your
product quality with testing experts:
We have been successfully active in quality assurance for 12 years – our roots are in testing!
We are an agile organization that adapts quickly to new technologies and market needs.
Our lifeblood is testing, especially by testers for testers: SKILLUP – AI testing for testers.
We offer testing for NON-Testers and ML Engineers
podcast
How can you test machine learning?
QualityHeroes Podcast Episode 36: Our guests Namrata Gurung and Michael Mlynarski discuss different methods of testing AI- and ML-based systems.
Projects
Introducing our customers
Project AI safeguarding in autonomous driving
This is a research project funded by the Federal Ministry for Economic Affairs and Climate Protection. A methodology for AI safety in connection with automated driving is being developed. QualityMinds led the work package on the topic of corner cases.
ATTENTION funding project
ATTENTION is a research project funded by the German government. In cooperation with a consortium of 5 German project partners (OEMs, suppliers, technology partners, research institutes), methods are being developed in the field of autonomous driving to predict the risk of accidents in real time and to limit damage.
Test ML
Automotive
Consulting and development of test planning software using supervised machine learning. QualityMinds optimized test planning for a major automotive manufacturer using machine learning and developed efficient tests for the control software.
Do you have any questions?
Get in touch with us! We look forward to exchanging knowledge with you.
