UI stuck when applying define_user_type for a large number (~400k)

[xfoxfu]

I have a script calling define_user_type on a background thread for a large number of types, which is about 400k. I have found the UI occasionally stuck for a pretty long time when reaching some count of types, such as ~100k, ~233k and ~310k, and at this time the memory usage of Binary Ninja drastically increases. I would like to know if this could be improved.

[plafosse]

This could depend on which windows visible. Would it be possible for you to share your script? There are also some performance issues if you're using type parsing instead of constructing types through the API that might help with performance.

[xfoxfu]

The active window is linear view and function list.

Yes, I am doing parsing, but the performance problem happens after the parsing has finished. The entire script is a little bit long, so I am posting the related part. When parsing, the UI is always interactive, but after logging parse success, it still take very long time (~1hrs), and UI sometimes stuck.

let responses = bn::interaction::FormInputBuilder::new()
    .open_file_field(
        "Select script.json",
        Some("script.json"),
        Some("script.json"),
    )
    .open_file_field(
        "Select il2cpp_binja.h",
        Some("il2cpp_binja.h"),
        Some("il2cpp_binja.h"),
    )
    .get_form_input("script.json from Il2CppDumper");
let bv = bv.to_owned();
std::thread::spawn(move || {
    let FormResponses::String(ref script_file) =  responses[0] else {unreachable!();};
    let FormResponses::String(ref header_file) = responses[1] else {unreachable!();};
    info!("loading {} and {}", script_file, header_file);

    let bg_task = BackgroundTask::new("Il2Cpp start", true).unwrap();
    bg_task.set_progress_text("Il2Cpp types (1/3)");
    let header_content = std::fs::read_to_string(header_file).unwrap();
    info!("platform: {}", bv.default_platform().unwrap().name());
    info!("start parsing at {:?}", Utc::now());

    let types = match bv.default_platform().unwrap().parse_types_from_source(
        header_content.as_str(),
        "il2cpp.h",
        &["."],
        "",
    ) {
        Ok(types) => types,
        Err(e) => {
            error!("parse faild: {}", e);
            return;
        }
    };
    info!("parse success at {:?}", Utc::now());
    let types_len = types.types.len();
    for (i, (name, ty)) in types.types.into_iter().enumerate() {
        if bv.get_type_by_name(&name).is_some() {
            continue;
        }
        bv.define_user_type(&name, &ty);
        if i % 1000 == 0 {
            bg_task.set_progress_text(&format!("Il2Cpp types: {}/ {}", i, types_len));
        }
    }
    info!("type defined");

[CouleeApps]

Defining many types like this is a known slow operation, since every new definition needs to check and update references in all types in case anything existing has changed. There's an operation to do this faster in C++: DefineUserTypes (plural), although it does not look like the Rust API has a function for this yet. But the plural version of this function takes advantage of knowing all 400k types to update ahead of time, and only needs to run through the list once, instead of O(n) times. I can go add a rust binding for that API, since it seems like you need it.

One thing to be warned about is that the UI Types View gets rather slow if you have 400k types in your binary. We've seen this when looking at large PDBs (like chrome), and while improving it is on our roadmap, it's still a performance sinkhole and you should be aware of it too.

[CouleeApps]

An API for this has been added to dev: BinaryView::define_auto_types / BinaryView::define_user_types. Hopefully these have the improved performance you need.