Probe generialized implementation

[nicolaefilat]

We ask for code review to check if the implementation is up to standards.

It requires HerbGrammar to be on branch fix-normalizing to pass tests.

[THinnerichs]

I do not get this. Why would you do this exactly?

[nicolaefilat]

It could be that the code that actually returns the outcome of the iteration never returns a program because the programs are skipped because of observational equivalence.

if return_type(grammar, prog.ind) == start_symbol
    eval_observation, correct_examples = evaluate_program(prog, grammar, iter.spec, iter.symboltable)

    if eval_observation in state.eval_cache # program already cached
        continue # maybe this skips all programs...
    end
    prog._val = (eval_observation, correct_examples)
    
    push!(state.eval_cache, eval_observation) # add result to cache
    push!(state.bank[state.level+1], prog) # add program to bank
    return (prog, state) # return program
end

The while true forces the function to keep executing until a program is returned. We could have called the function recursively but using an iterative approach is better I think.

[nicolaefilat]

I did a bit of a hack to add the HerbGrammar fix-normalizing branch to the CI as a dependency to make the CI pass to prove that the code works :)

[ReubenJ]

I did a bit of a hack to add the HerbGrammar fix-normalizing branch to the CI as a dependency to make the CI pass to prove that the code works :)

Please switch this back, @nicolaefilat. We don't want to mess with the CI to make it pass. We'll get the HerbGrammar branch merged and released before merging this.

[nicolaefilat]

@ReubenJ I reverted the CI hack :)

[ReubenJ]

Hackathon Plan

Steps

• Run a budgeted search
  • Gets an iterator with some limit (the low-level budget)
  • This is a normal iterator
• Collect (partial) successes
  • Function to define success, which is just % of examples solved by default
  • Function takes any kind of specification
  • Boolean output: should the program be saved or not?
• Modify the grammar
  • Function to modify the grammar based on the successes
  • Takes the successes (saved programs)
  • Modify the grammar: in place or return a new one, TBD
• Rinse and repeat
  • Function defining high-level budget
  • For now, just a parameter defining the number of overall runs of budgeted search

Optimization

• When evaluating a program during search, store the output of a rulenode on the specification in the rulenode's _val field
• Keep queue across grammar updates to avoid re-enumerating the search space that was already explored earlier