Sunday, January 26, 2020

Handling Imbalanced Dataset in Machine Learning

  • Imbalanced dataset can affect machine learning decision boundaries.
  • Distribution of target class matters!
  • Why is imbalanced class bad - source 1
  • Sometimes imbalanced dataset is "good" such as in fraud detection. It is important to find those anomalies. 
  • "imbalancec class puts accuracy out of business". It is important to not choose accuracy as metric because the model can cheat by just guessing the majority class and achieve high accuracy. The high accuracy in this case is an illusion. It is about using a trade-off of precision and recall, confusion matrix.

Related Concepts:
  • Imbalanced datasets
  • Confusion matrix
  • Resampling
  • Random under-sampling
  • Random over-sampling
  • Python imbalanced-learn module
  • Random under-sampling and over-sampling with imbalanced-learn
  • Under-sampling: Tomek links
  • Under-sampling: Cluster Centroids
  • Over-sampling: SMOTE
Source 4


Flask 2022

 increase jinja2 template cache limit from 50 to above. to speed up flask app significantly source: One line of code cut our Flask page load...