Is there an algorithm for Speaker Error Rate for speech-to-text diarization?

Question

Some speech-to-text services, like Google Speech-to-Text, offer speaker differentiation via diarization which attempts to identify and separate multiple speakers on a single audio recording. This is often needed when multiple speakers are in a meeting room sharing a single microphone.

Is there an algorithm and implementation to calculate the correctness of speaker separation?

This would be used in conjunction with Word Error Rate which is often used to test correctness of baseline transcription.

Answer 1

The commonly used approach for this appears to be the Diarization Error Rate (DER) defined by NIST in the NIST-RT projects.

A newer evaluation metric is the Jaccard Error Rate (JER) introduced in DIHARD II: The Second DIHARD Speech Diarization Challenge.

Two projects for measuring these include:

• https://github.com/nryant/dscore
• https://github.com/wq2012/SimpleDER

DER is referenced in these papers:

• A Comparison of Neural Network Feature Transforms for Speaker Diarization
• The ICSI RT-09 Speaker Diarization System