Tune Hyperparameters Without Overfitting

Parameters vs. hyperparameters

The model learns parameters; you set hyperparameters (tree depth, learning rate, regularization strength). Good ones can lift performance a lot.

from sklearn.model_selection import GridSearchCV
grid = {"max_depth": [3, 5, 10], "n_estimators": [100, 300]}
search = GridSearchCV(model, grid, cv=5, scoring="f1")
search.fit(X_train, y_train)
print(search.best_params_)

Don't fool yourself

Tune using cross-validation on the training data, and only touch the test set once at the very end. For big search spaces, RandomizedSearchCV or Bayesian tools (Optuna) find good settings far faster than exhaustive grids.