profile fails when OptmizationProblem algorithm uses Hessian

[sebapersson]

I have tried to make PEtab.jl work together with ProfileLikelihood for user provided gradients (see #95 ).

It works well if I use an algorithm that only uses gradient (like BFGS from NLopt) (I can later make a PR to create LikelihoodProblem from OptimizationProblem):

using PEtab
using Optimization
using ProfileLikelihood
using Ipopt
using OptimizationMOI
using OptimizationNLopt

alg = Ipopt.Optimizer()
alg = NLopt.LD_LBFGS

petab_model = PEtabModel(joinpath(@__DIR__, "Test_model2", "Test_model2.yaml"); write_to_file=false)
petab_problem = PEtabODEProblem(petab_model)
optimization_problem = PEtab.OptimizationProblem(petab_problem)

_likelihood_problem = LikelihoodProblem{length(petab_problem.θ_nominalT),
                                        typeof(optimization_problem),
                                        typeof(SciMLBase.NullParameters()),
                                        typeof(optimization_problem.f),
                                        typeof(petab_problem.θ_nominalT),
                                        typeof(petab_problem.θ_names)}(optimization_problem,
                                                                       SciMLBase.NullParameters(),
                                                                       optimization_problem.f,
                                                                       petab_problem.θ_nominalT,
                                                                       petab_problem.θ_names)
sol = mle(_likelihood_problem, alg)
prof = profile(_likelihood_problem, sol)

However, with alg = Ipopt.Optimizer() (where the Hessian is computed) the code fails with a very long error-message. The following MVE reproduces the error (I get the same error on IPNewton from Optim.jl):

using Optimization
using ProfileLikelihood
using Ipopt
using OptimizationMOI
using OptimizationNLopt
using ForwardDiff
rosenbrock(x, p) = (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2
_∇f = (G, u, p) ->  begin 
    p = Float64[]
    _f = (x) -> rosenbrock(x, p)
    ForwardDiff.gradient!(G, _f, u)

end
_Δf = (H, u, p) ->  begin 
    p = Float64[]
    _f = (x) -> rosenbrock(x, p)
    ForwardDiff.hessian!(H, _f, u)
end

x0 = zeros(2) .+ 0.5
xnames = [:x1, :x2]
optimization_function = Optimization.OptimizationFunction(rosenbrock;
                                                          grad = _∇f,
                                                          hess = _Δf,
                                                          syms=xnames)
prob = OptimizationProblem(optimization_function, x0, lb = [-1.0, -1.0], ub = [0.8, 0.8])

_likelihood_problem = LikelihoodProblem{length(x0),
                                        typeof(prob),
                                        typeof(SciMLBase.NullParameters()),
                                        typeof(prob.f),
                                        typeof(x0),
                                        typeof(xnames)}(prob,
                                                        SciMLBase.NullParameters(),
                                                        prob.f,
                                                        x0,
                                                        xnames)
sol = mle(_likelihood_problem, Ipopt.Optimizer())
prof = profile(_likelihood_problem, sol)

With error message:

ERROR: BoundsError: attempt to access 1-element Vector{ForwardDiff.Dual{ForwardDiff.Tag{var"#20#22"{Vector{Float64}}, Float64}, Float64, 1}} at index [2]
Stacktrace:
  [1] getindex
    @ ./essentials.jl:13 [inlined]
  [2] rosenbrock
    @ ~/.julia/dev/PEtab/test/MVE.jl:31 [inlined]
  [3] #20
    @ ~/.julia/dev/PEtab/test/MVE.jl:34 [inlined]
  [4] vector_mode_dual_eval!
    @ ~/.julia/packages/ForwardDiff/PcZ48/src/apiutils.jl:24 [inlined]
  [5] vector_mode_gradient!(result::Vector{Float64}, f::var"#20#22"{Vector{Float64}}, x::Vector{Float64}, cfg::ForwardDiff.GradientConfig{ForwardDiff.Tag{var"#20#22"{Vector{Float64}}, Float64}, Float64, 1, Vector{ForwardDiff.Dual{ForwardDiff.Tag{var"#20#22"{Vector{Float64}}, Float64}, Float64, 1}}})
    @ ForwardDiff ~/.julia/packages/ForwardDiff/PcZ48/src/gradient.jl:96
  [6] gradient!
    @ ~/.julia/packages/ForwardDiff/PcZ48/src/gradient.jl:37 [inlined]
  [7] gradient!(result::Vector{Float64}, f::var"#20#22"{Vector{Float64}}, x::Vector{Float64}, cfg::ForwardDiff.GradientConfig{ForwardDiff.Tag{var"#20#22"{Vector{Float64}}, Float64}, Float64, 1, Vector{ForwardDiff.Dual{ForwardDiff.Tag{var"#20#22"{Vector{Float64}}, Float64}, Float64, 1}}})
    @ ForwardDiff ~/.julia/packages/ForwardDiff/PcZ48/src/gradient.jl:35
  [8] gradient!(result::Vector{Float64}, f::var"#20#22"{Vector{Float64}}, x::Vector{Float64})
    @ ForwardDiff ~/.julia/packages/ForwardDiff/PcZ48/src/gradient.jl:35
  [9] #19
    @ ~/.julia/dev/PEtab/test/MVE.jl:35 [inlined]
 [10] (::Optimization.var"#20#27"{OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#6#7"{Int64, Float64, PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}})(::Vector{Float64}, ::Vector{Float64})
    @ Optimization ~/.julia/packages/Optimization/fPVIW/src/function.jl:77
 [11] eval_objective_gradient(evaluator::OptimizationMOI.MOIOptimizationNLPEvaluator{Float64, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#6#7"{Int64, Float64, PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Optimization.var"#20#27"{OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#6#7"{Int64, Float64, PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Optimization.var"#21#28"{OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#6#7"{Int64, Float64, PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}, Vector{Float64}, Vector{Float64}, Nothing, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Nothing, Nothing}, G::Vector{Float64}, x::Vector{Float64})
    @ OptimizationMOI ~/.julia/packages/OptimizationMOI/Ip1EA/src/nlp.jl:194
 [12] eval_objective_gradient(model::Ipopt.Optimizer, grad::Vector{Float64}, x::Vector{Float64})
    @ Ipopt ~/.julia/packages/Ipopt/GMR7n/src/MOI_wrapper.jl:697
 [13] (::Ipopt.var"#eval_grad_f_cb#4"{Ipopt.Optimizer})(x::Vector{Float64}, grad_f::Vector{Float64})
    @ Ipopt ~/.julia/packages/Ipopt/GMR7n/src/MOI_wrapper.jl:782
 [14] _Eval_Grad_F_CB(n::Int32, x_ptr::Ptr{Float64}, #unused#::Int32, grad_f::Ptr{Float64}, user_data::Ptr{Nothing})
    @ Ipopt ~/.julia/packages/Ipopt/GMR7n/src/C_wrapper.jl:54
 [15] IpoptSolve(prob::IpoptProblem)
    @ Ipopt ~/.julia/packages/Ipopt/GMR7n/src/C_wrapper.jl:442
 [16] optimize!(model::Ipopt.Optimizer)
    @ Ipopt ~/.julia/packages/Ipopt/GMR7n/src/MOI_wrapper.jl:924
 [17] __solve(cache::OptimizationMOI.MOIOptimizationNLPCache{OptimizationMOI.MOIOptimizationNLPEvaluator{Float64, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#6#7"{Int64, Float64, PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Optimization.var"#20#27"{OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#6#7"{Int64, Float64, PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Optimization.var"#21#28"{OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#6#7"{Int64, Float64, PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}, Vector{Float64}, Vector{Float64}, Nothing, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Nothing, Nothing}, Ipopt.Optimizer})
    @ OptimizationMOI ~/.julia/packages/OptimizationMOI/Ip1EA/src/nlp.jl:422
 [18] solve!
    @ ~/.julia/packages/SciMLBase/ynHlA/src/solve.jl:177 [inlined]
 [19] #solve#614
    @ ~/.julia/packages/SciMLBase/ynHlA/src/solve.jl:94 [inlined]
 [20] solve
    @ ~/.julia/packages/SciMLBase/ynHlA/src/solve.jl:91 [inlined]
 [21] add_point!(param_vals::Vector{Float64}, profile_vals::Vector{Float64}, other_mles::Vector{Vector{Float64}}, θₙ::Float64, param_range::LinRange{Float64, Int64}, restricted_prob::OptimizationProblem{true, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, n::Int64, cache::PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}, alg::Ipopt.Optimizer, sub_cache::Vector{Float64}, ℓmax::Float64, normalise::Bool, threshold::Float64, min_steps::Int64, mles::Vector{Float64}; next_initial_estimate_method::Val{:prev}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ProfileLikelihood ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:452
 [22] add_point!
    @ ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:443 [inlined]
 [23] find_endpoint!(param_vals::Vector{Float64}, profile_vals::Vector{Float64}, other_mles::Vector{Vector{Float64}}, param_range::LinRange{Float64, Int64}, restricted_prob::OptimizationProblem{true, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, n::Int64, cache::PreallocationTools.DiffCache{Vector{Float64}, Vector{Float64}}, alg::Ipopt.Optimizer, sub_cache::Vector{Float64}, ℓmax::Float64, normalise::Bool, threshold::Float64, min_steps::Int64, mles::Vector{Float64}; min_steps_fallback::Val{:replace}, next_initial_estimate_method::Val{:prev}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ProfileLikelihood ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:428
 [24] find_endpoint!
    @ ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:423 [inlined]
 [25] #_find_endpoint_serial!#103
    @ ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:399 [inlined]
 [26] _find_endpoint_serial!
    @ ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:395 [inlined]
 [27] #find_endpoint!#102
    @ ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:382 [inlined]
 [28] find_endpoint!
    @ ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:377 [inlined]
 [29] profile_single_parameter!(θ::Dict{Int64, Vector{Float64}}, prof::Dict{Int64, Vector{Float64}}, other_mles::Dict{Int64, Vector{Vector{Float64}}}, splines::Dict{Int64, Interpolations.Extrapolation{Float64, 1, Interpolations.MonotonicInterpolation{Float64, Float64, Float64, Float64, Float64, Interpolations.FritschCarlsonMonotonicInterpolation, Vector{Float64}, Vector{Float64}}, Union{Interpolations.FritschCarlsonMonotonicInterpolation, Interpolations.NoInterp, Tuple{Vararg{Union{Interpolations.FritschCarlsonMonotonicInterpolation, Interpolations.NoInterp}}}}, Interpolations.Line{Nothing}}}, confidence_intervals::Dict{Int64, ProfileLikelihood.ConfidenceInterval{Float64, Float64}}, n::Int64, num_params::Int64, param_ranges::Vector{Tuple{LinRange{Float64, Int64}, LinRange{Float64, Int64}}}, mles::Vector{Float64}, shifted_opt_prob::OptimizationProblem{true, OptimizationFunction{true, SciMLBase.NoAD, ProfileLikelihood.var"#10#11"{Float64, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, alg::Ipopt.Optimizer, ℓmax::Float64, normalise::Bool, threshold::Float64, min_steps::Int64, spline_alg::Val{Interpolations.FritschCarlsonMonotonicInterpolation}, extrap::Type, confidence_interval_method::Val{:spline}, conf_level::Float64; min_steps_fallback::Val{:replace}, next_initial_estimate_method::Val{:prev}, parallel::Val{false}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ProfileLikelihood ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:259
 [30] profile(prob::LikelihoodProblem{2, OptimizationProblem{true, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, SciMLBase.NullParameters, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, Vector{Symbol}}, sol::ProfileLikelihood.LikelihoodSolution{2, Vector{Float64}, LikelihoodProblem{2, OptimizationProblem{true, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, SciMLBase.NullParameters, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, Vector{Symbol}}, Float64, SciMLBase.ReturnCode.T, Ipopt.Optimizer}, n::UnitRange{Int64}; alg::Ipopt.Optimizer, conf_level::Float64, confidence_interval_method::Symbol, threshold::Float64, resolution::Int64, param_ranges::Vector{Tuple{LinRange{Float64, Int64}, LinRange{Float64, Int64}}}, min_steps::Int64, min_steps_fallback::Symbol, normalise::Bool, spline_alg::Type, extrap::Type, parallel::Bool, next_initial_estimate_method::Symbol, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ProfileLikelihood ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:84
 [31] profile(prob::LikelihoodProblem{2, OptimizationProblem{true, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, SciMLBase.NullParameters, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, Vector{Symbol}}, sol::ProfileLikelihood.LikelihoodSolution{2, Vector{Float64}, LikelihoodProblem{2, OptimizationProblem{true, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, SciMLBase.NullParameters, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, Vector{Symbol}}, Float64, SciMLBase.ReturnCode.T, Ipopt.Optimizer}, n::UnitRange{Int64})
    @ ProfileLikelihood ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:49
 [32] profile(prob::LikelihoodProblem{2, OptimizationProblem{true, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, SciMLBase.NullParameters, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, Vector{Symbol}}, sol::ProfileLikelihood.LikelihoodSolution{2, Vector{Float64}, LikelihoodProblem{2, OptimizationProblem{true, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, SciMLBase.NullParameters, OptimizationFunction{true, SciMLBase.NoAD, typeof(rosenbrock), var"#19#21", var"#27#29", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, Vector{Symbol}}, Float64, SciMLBase.ReturnCode.T, Ipopt.Optimizer})
    @ ProfileLikelihood ~/.julia/packages/ProfileLikelihood/Zepnj/src/profile_likelihood.jl:49
 [33] top-level scope
    @ ~/.julia/dev/PEtab/test/MVE.jl:63

[DanielVandH]

Ah. The issue is in construct_fixed_optimisation_problem -it's only changing the objective function and resizing u0. It needs to be updated so that it (1) uses the user-provided gradients/Hessians and (2) modifies them so that the fixed variable's derivatives are zero. I'm not sure the best way to do that at the moment.

[sebapersson]

As I understood the code, for each parameter that is profiled you remake the OptmizationProblem to run the optimization over fewer parameters?

Given this, I think the one approach would be to allow construct_fixed_optimisation_problem to return f, grad and hess, where f is handled like now, and grad is handled as (similar holds for hess just a bit more work with indices):

if isnothing(prob.grad) # If autodiff nothing by default 
    grad = nothing
else
   grad = (grad, θ, p) -> begin
        _grad_full = get_tmp(cache_grad, grad)
        _θ = get_tmp(cache, θ)
        _θ[iest] .= θ
        prob.f.grad(_grad_full, _θ)
        grad .= _grad_full[iest]
    end
end

Would something like this work?

[DanielVandH]

Yeah, that looks like it would work. The same could be done for the Hessian. (I'm not sure that accessing the gradient like that in Optimization.jl is in the API, but it should be fine since I already have to use its internals [hence why its version is fixed in the Project.toml]).

There are other fields for an OptimizationProblem to take care of https://docs.sciml.ai/Optimization/stable/API/optimization_function/, e.g. hv and hess_prototype. It might be a bit annoying to directly take care of those, not sure. Or we could just leave those off for now, focusing only on f, grad, and hess.

[sebapersson]

I think accessing the gradient like above should work (even though I might be slightly off with the exact syntax).

Maybe at the moment letting hv and hess_prototype be is best, they are rare options, for example, I only think hv is used for Krylov-Trust-Region method which seldom is used. A good solution at the moment could be to throw an error if hv and hess_prototype are not equal to nothing?

[DanielVandH]

Yeah, throwing errors for those other fields would be a good idea. Those are rarely used from what I can tell, and probably even rarer for profiling, so I don't see it being a problem.

[sebapersson]

Closed with #97