In this post, the automatic quality evaluation of Machine Learning (ML) based algorithms is discussed. Recently I and my team have taken 2nd place in Sberbank AIJ competition 2019 which is one of the largest ML contests in Russia (100 teams, 45k$ prizes).