OptimizationCallbacks

This package contains a bunch of callable objects that can be useful as callbacks in optimization, such as progress logging (LogProgress), check point saving (CheckPointSaver), or evaluations on other functions, e.g. for tracking validation losses (Evaluator). These callbacks can be triggered with different mechanisms, either based on iteration step (IterationTrigger), on time (TimeTrigger), or on special events, e.g. at the end of optimization (EventTrigger). The package is tested with the popular Optimization.jl package, but it does not depend on it and can also be used in custom optimization procedures, or with other packages.

OptimizationCallbacks.Callback — Type
Callback(trigger, function; t = 0, extra = nothing, stop = (_, _, _, _) -> false)

Callback((trigger1, trigger2, ...), function)

See triggers IterationTrigger, TimeTrigger, EventTrigger. For callback functions see LogProgress, CheckPointSaver.

The callback function has arguments Optimization.OptimizationState, value, t, extra.

Example

julia> using Optimization

julia> using OptimizationCallbacks

julia> import ForwardDiff

julia> function rosenbrock(x, p)
           (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
       end
rosenbrock (generic function with 1 method)

julia> optf = OptimizationFunction(rosenbrock, AutoForwardDiff());

julia> prob = OptimizationProblem(optf, [0., 0.], [1., 100.]);

julia> callback = Callback(IterationTrigger(5), LogProgress());

julia> sol = solve(prob, Optimization.LBFGS(); callback)
 eval   | current     | lowest      | highest
_________________________________________________
      5 |    0.460215 |    0.460215 |    0.460215
     10 |    0.162607 |    0.162607 |    0.460215
     15 |   0.0257404 |   0.0257404 |    0.460215
     20 | 0.000911646 | 0.000911646 |    0.460215
     25 | 1.04339e-13 | 1.04339e-13 |    0.460215
retcode: Success
u: 2-element Vector{Float64}:
 0.9999997057368228
 0.999999398151528
OptimizationCallbacks.CheckPointSaver — Type
CheckPointSaver(filename; overwrite = false)

Saves checkpoints as JLD2 files.

Examples

julia> using Optimization

julia> using OptimizationCallbacks

julia> import ForwardDiff

julia> function rosenbrock(x, p)
           (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
       end;

julia> optf = OptimizationFunction(rosenbrock, AutoForwardDiff());

julia> prob = OptimizationProblem(optf, [0., 0.], [1., 100.]);

julia> filename = tempname() * ".jld2";

julia> callback = Callback((IterationTrigger(5), EventTrigger((:end,))),
                           CheckPointSaver(filename));

julia> sol = solve(prob, Optimization.LBFGS(); callback);

julia> OptimizationCallbacks.trigger!(callback, :end);

julia> callback(Optimization.OptimizationState(u = sol.u, objective = sol.objective),
                sol.objective);

julia> using JLD2

julia> checkpoint_dict = load(filename);

julia> checkpoint_dict["15"].u
2-element Vector{Float64}:
 0.8834203727171949
 0.7694090396265355
OptimizationCallbacks.Evaluator — Type
Evaluator(f; T = Float64, label = :evaluation)

Evaluate function f on state and store it in evaluations.

Example

julia> using Optimization

julia> using OptimizationCallbacks

julia> import ForwardDiff

julia> function rosenbrock(x, p)
           (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
       end
rosenbrock (generic function with 1 method)

julia> optf = OptimizationFunction(rosenbrock, AutoForwardDiff());

julia> prob = OptimizationProblem(optf, [0., 0.], [1., 100.]);

julia> callback = Callback(IterationTrigger(5), Evaluator(x -> rosenbrock(x.u, [1., 90.])));

julia> sol = solve(prob, Optimization.LBFGS(); callback);

julia> callback.func.evaluations
5-element Vector{Float64}:
 0.45989873460740843
 0.16216539624081874
 0.024525435426304077
 0.0009100921964470193
 1.0256411872737028e-13
OptimizationCallbacks.EventTrigger — Type
EventTrigger(events)

Triggers at given events using the trigger! function.

Example

julia> using OptimizationCallbacks

julia> callback = Callback(EventTrigger((:start, :end)), (_, value,_ ,_) -> @info( "Current value: " * string(value)));

julia> begin
           @info "Start."
           OptimizationCallbacks.trigger!(callback, :start)
           callback(nothing, 10.)
           callback(nothing, 9.)
           callback(nothing, 7.)
           OptimizationCallbacks.trigger!(callback, :end)
           callback(nothing, 6.)
       end;
[ Info: Start.
[ Info: Current value: 10.0
[ Info: Current value: 6.0
OptimizationCallbacks.TimeTrigger — Type
TimeTrigger(Δt)

Triggers every Δt seconds.

Example

julia> using Optimization

julia> using OptimizationCallbacks

julia> import ForwardDiff

julia> function rosenbrock(x, p)
           sleep(.1)
           (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
       end
rosenbrock (generic function with 1 method)

julia> optf = OptimizationFunction(rosenbrock, AutoForwardDiff());

julia> prob = OptimizationProblem(optf, [0., 0.], [1., 100.]);

julia> callback = Callback(TimeTrigger(2.0), (_,_,_,_) -> @info("Hi"));

julia> sol = solve(prob, Optimization.LBFGS(); callback)
[ Info: Hi
[ Info: Hi
retcode: Success
u: 2-element Vector{Float64}:
 0.9999997057368228
 0.999999398151528
OptimizationCallbacks.reset! — Method
reset!(callback)

Resets internal states, like iteration counters, in a Callback.

Example

julia> using Optimization

julia> using OptimizationCallbacks

julia> callback = Callback(IterationTrigger(5), LogProgress());

julia> for _ in 1:6
           callback(Optimization.OptimizationState(), 17.); # arbitrary calls to callback
       end
 eval   | current     | lowest      | highest
_________________________________________________
      5 |          17 |          17 |          17

julia> callback.t
6

julia> callback.func.lowest
17.0

julia> OptimizationCallbacks.reset!(callback);

julia> callback.t
0

julia> callback.func.lowest
Inf