Validation

ByAdmin July 10, 2021

Week – 2 Validation >>> How to Win a Data Science Competition: Learn from Top Kagglers

Performance increase on a fixed cross-validation split guaranties performance increase on any cross-validation split.

The model, that performs best on the validation set is guaranteed to be the best on the test set.

We use validation to estimate the quality of our model

The logic behind validation split should mimic the logic behind train-test split.

Underfitting refers to not capturing enough patterns in the data

Generally, this approach is based on the assumption that people rarely tend to overfit to the Public LB. Almost always you have a lot of data in the test set and it is quite hard to overfit. Indeed, this render validation useless.

Generally, this approach is based on the assumption that validation is rarely valid in competitions. Often it is hard to trust your validation and thus you should account for both cases if the validation will succeed and if the validation will fail.

Generally, this approach is based on the assumption that the test data may have a different target distribution compared to the train data. If that would be the true, the submission which was chosen based on Public LB, will perform better. If, otherwise, the above distributions will be similar, the submission which was chosen based on validation scores, will perform better.

Campaign_id, Date, {some features}, Number_of_new_customers

Test set consists of multiple campaigns. For each of them we are given several first days in train data. For example, if a campaign runs for two weeks, we could have three first days in train set, and all next days will be present in the test set. For another campaign, running for weeks, we could have the first 6 days in the train set, and the remaining days in the test set.

Identify train/test split in a competition.

2 points

Time-based split

Id-based split

Random split

Combined split

Public leaderboard score will be unreliable because of too little data

Different scores/optimal parameters between folds

Train and test target distribution are from different distributions

Train and test data are from different distributions

How to Win a Data Science Competition

Mean encodings
ByAdmin July 11, 2021July 11, 2021

Week – 3 Mean encodings >>> How to Win a Data Science Competition: Learn from Top Kagglers 1. What can be an indicator of usefulness of mean encodings? 1 point…

Read More Mean encodings
How to Win a Data Science Competition

Graded Advanced Features II Quiz
ByAdmin July 11, 2021

Week – 4 Graded Advanced Features II Quiz >>> How to Win a Data Science Competition Learn from Top Kagglers 1. Imagine that we apply X = PCA(n_components=5).fit_transform(data) and data…

Read More Graded Advanced Features II Quiz
How to Win a Data Science Competition

Graded quiz of How to Win a Data Science Competition
ByAdmin July 11, 2021

Week – 4 Graded quiz How to Win a Data Science Competition Learn from Top Kagglers 1. Which hyperparameters are first to tune in sklearn’s RandomForest? 1 point n_estimators, max_depth,…

Read More Graded quiz of How to Win a Data Science Competition
How to Win a Data Science Competition

Data leakages
ByAdmin July 11, 2021

Week – 2 Data leakages >>> How to Win a Data Science Competition Learn from Top Kagglers 1. Suppose that you have a credit scoring task, where you have to…

Read More Data leakages
How to Win a Data Science Competition

Exploratory data analysis
ByAdmin July 10, 2021

Week – 2 Exploratory data analysis >>> How to Win a Data Science Competition Learn from Top Kagglers 1. Suppose we are given a data set with features XX, YY,…

Read More Exploratory data analysis
How to Win a Data Science Competition

Feature preprocessing and generation with respect to models
ByAdmin July 10, 2021

Week – 1 Feature preprocessing and generation with respect to models 1. Suppose we have a feature with all the values between 0 and 1 except few outliers larger than…

Read More Feature preprocessing and generation with respect to models

Week – 2 Validation >>> How to Win a Data Science Competition: Learn from Top Kagglers

Similar Posts