Repository navigation
Early stop of optimization for hyperparameter tuning #373
Description
Problem Description.
I am trying to tune the hyperparameters of my ML models. I would like to stop optimization when the target values exceed a certain threshold to save time.
Here is the minimal reproducible example:
from sklearn.datasets import load_iris
from sklearn.svm import SVC
from sklearn.metrics import f1_score
from bayes_opt import BayesianOptimization
from bayes_opt.event import DEFAULT_EVENTS, Events
data = load_iris()
kernels = ['rbf', 'poly']
def svm_cl_bo(kernel, C, degree):
params_svm = {}
params_svm['kernel'] = kernels[1 if kernel > 0.5 else 0]
params_svm['C'] = C
params_svm['degree'] = degree
model = SVC(**params_svm)
model.fit(data['data'], data['target'])
weighted_f1 = f1_score(model.predict(data['data']), data['target'], average='weighted')
return weighted_f1
params_svm ={
'kernel':(0,1),
'C':(1e-1, 1e+1),
'degree':(1, 3),
}
gbm_bo = BayesianOptimization(svm_cl_bo, params_svm, random_state=42, verbose=0)
def my_callback(event, instance):
if instance.max['target'] > 0.8:
instance.dispatch(Events.OPTIMIZATION_END)
print(f"Event `{event}` was observed with {instance.max['target']}.")
gbm_bo.subscribe(
event=Events.OPTIMIZATION_STEP,
subscriber=" " ,
callback=my_callback,
)
gbm_bo.maximize(init_points=2, n_iter=4)
Result:
Event
optimization:stepwas observed with 0.9800180162145932. -- Why it doesn't stop here?
Eventoptimization:stepwas observed with 0.9933340000666734.
Eventoptimization:stepwas observed with 0.9933340000666734.
Eventoptimization:stepwas observed with 0.9933340000666734.
Eventoptimization:stepwas observed with 0.9933340000666734.
Eventoptimization:stepwas observed with 0.9933340000666734.
Describe the solution you tried
I followed the examples from the notebook Advanced tour of the Bayesian Optimization package to use callback or Observers but I wasn't able to solve the problem. Here are my attempts and the related issues about these attempts:
- Attempt 1 - Callback:
gbm_bo.subscribe(
event=Events.OPTIMIZATION_STEP,
subscriber="Any hashable object",
callback= my_callback,
)
def my_callback(event, instance)
if instance.max['target'] > 0.8:
instance.dispatch(Events.OPTIMIZATION_END)
It didn't stop the optimization process.
def my_callback(event, instance)
if instance.max['target'] > 0.8:
raise StopIteration
It stopped the process but it returns nothing.
- Attempt 2 - Observer:
class BasicObserver:
def update(self, event, instance):
"""Does whatever you want with the event and `BayesianOptimization` instance."""
if instance.max['target'] > 0.8:
instance.dispatch(Events.OPTIMIZATION_END)
my_observer = BasicObserver()
gbm_bo.subscribe(
event=Events.OPTIMIZATION_STEP,
subscriber=my_observer,
callback=None,
)
It didn't stop the optimization process.
I also tried the combination of callback and Observer but I couldn't solve it.
Any help on this issue would be appreciated.
Activity
t-muser commentedon Nov 1, 2022
Hi @doganMemory,
this is easier if you run in suggest-probe-register, i.e. manually perform the steps of the maximization. Let me know if this helps:
Note that I had to increase your threshold, otherwise the loop would simply break after this first iteration.
To my understanding -- maybe @bwheelz36 can chime in here -- this Event logs that the optimization has ended but is not responsible for ending the optimization. Notably, it used internally as signal for the
ScreenLoggerto print the bottom rule of the logging table.doganMemory commentedon Nov 1, 2022
Hi @Till-M,
Thanks for the quick reply and the solution - it helps and I can continue with this one.
Okay, it kind of makes sense, I thought if I would dispatch the event or somehow change it, it would affect the process, not just the log.
The solution that you provided helps me, we can close the issue if it's suitable.