In response to demands from broadcasters and other organizations that process large volumes of A/V content, the EBU has launched and coordinates a framework for benchmarking Artificial Intelligence (AI) and Machine Learning (ML) services. The project is spearheaded by ‘BenchmarkSTT’, a tool designed to facilitate the benchmarking of speech-to-text systems and services.
2023
Development of a facial recognition system for video- Development of metrics to evaluate facial recognition systems for video
- Deploy an Open API for facial recognition to benchmark state of the art systems
- Write a report on best practices and state of the art on facial recognition
2022
Development of a facial recognition system for video- Development of metrics to evaluate facial recognition systems for video
- Deploy an Open API for facial recognition to benchmark state of the art systems
- Write a report on best practices and state of the art on facial recognition
2021
Publish BenchmarkSTT tool 1.1- Engage the development of a facial recognition system for video
2020
- Add the Levenshtein distance to the STT benchmarking code (Q1 2020)
- Test the first STT benchmarking software release with the three metrics (Q1 2020)
- Test the STT benchmarking API and Docker image (Q1 2020)
- Publish STT benchmarking release 1.0.0 on PyPi (Q2 2020)
- Update the STT benchmarking documentation on ReadTheDocs (Q2 2020)
- Organise a Webinar (Q4 2020)
- Add the Levenshtein distance to the STT benchmarking code (Q1 2020)
- Develop the STT benchmarking new metrics for 1.1 on Github (Q3 2020)
BenchmarkSTT
Unlike tools used by ML experts in academic settings, BenchmarkSTT targets non-specialists in production environments. It does not require meticulous preparation of test data, and it prioritises simplicity, automation and relative ranking over scientific precision and absolute scores.
With a single command, the tool calculates the accuracy of Automatic Speech Recognition (ASR) transcripts against a reference. Optionally, the user can apply normalization rules to remove non-significant differences such as case or punctuation. Supporting multiple languages and user-defined normalizations, this CLI tool can be integrated into production workflows to perform real-time benchmarking.
Open Source
This collaborative project is open source.
- BenchmarkSTT is available on GitHub: github.com/ebu/benchmarkstt
- The first release has been published on PyPi: pypi.org/project/benchmarkstt/
- It is fully documented on ReadTheDocs: benchmarkstt.readthedocs.io/en/latest/
Webinar
Contributors and users of the opensource 'STT Benchmarking' explain the tool's principles, useful metrics and applications.
The second part of the webinar addresses developers and provides an overview of the code and guidance for its integration.