Article: Measuring Data Drift with the Unstable Population Indicator
has assertion: assertion
has provenance: provenance
has publication info: pubinfo
type: Nanopublication
created: 2024-02-29T09:53:47.853Z
creator: Tobias Kuhn
license: https://creativecommons.org/licenses/by/4.0/
has nanopub type: ScholarlyWork
has nanopub type: DataScienceNanopub
introduces: Measuring Data Drift with the Unstable Population Indicator
was created at: nanodash.petapico.org

Measuring Data Drift with the Unstable Population Indicator
abstract: Measuring data drift is essential in machine learning applications where model scoring (evaluation) is done on data samples that differ from those used in training. The Kullback-Leibler divergence is a common measure of shifted probability distributions, for which discretized versions are invented to deal with binned or categorical data. We present the Unstable Population Indicator, a robust, flexible and numerically stable, discretized implementation of Jeffrey's divergence, along with an implementation in a Python package that can deal with continuous, discrete, ordinal and nominal data in a variety of popular data types. We show the numerical and statistical properties in controlled experiments. It is not advised to employ a common cut-off to distinguish stable from unstable populations, but rather to let that cut-off depend on the use case.
date: 2024
is part of: Data Science
authored by: Marcel R. Haas
authored by: Lisette Sibbald
type: ResourcePaper

Marcel R. Haas
affiliation: Business Intelligence, University of Amsterdam, Spui 21, 1012WX Amsterdam, The Netherlands
affiliation: Public Health and Primary Care, Leiden University Medical Center, Albinusdreef 2, The Netherlands
type: Person

Lisette Sibbald
affiliation: Department of Methodology and Statistics and Department of Cognitive Neuropsychology, Tilburg University, Prof. Cobbenhagenlaan 125, 5037 DB Tilburg, The Netherlands
affiliation: Business Intelligence, University of Amsterdam, Spui 21, 1012WX Amsterdam, The Netherlands
type: Person

Department of Methodology and Statistics and Department of Cognitive Neuropsychology, Tilburg University, Prof. Cobbenhagenlaan 125, 5037 DB Tilburg, The Netherlands
type: Organization

Business Intelligence, University of Amsterdam, Spui 21, 1012WX Amsterdam, The Netherlands
type: Organization

Public Health and Primary Care, Leiden University Medical Center, Albinusdreef 2, The Netherlands
type: Organization

sig
has algorithm: RSA
signed by: Tobias Kuhn