add concurrency to the discover

[keyn4]

Description of change

add concurrency to the discover to make it faster

[hsyyid]

Shouldn't we get rid of the for loop now since the work is being done above?

[keyn4]

old loop deleted

[butkeraites-hotglue]

Excellent work! Left two suggestions feel free to apply them or not 👍

[butkeraites-hotglue]

Suggested change

	results = []
	for future in as_completed(all_tasks):
	(index, result) = future.result()
	# Insert the result in the right index of the list
	results.insert(index, result)
	results = [None] * len(fn_args_list) # Preallocate list for correct ordering
	for future in as_completed(all_tasks):
	index, result = future.result()
	results[index] = result

It's just a suggestion as it's not necessary to do this memory allocation dynamically.

[butkeraites-hotglue]

Suggested change

	objects_list = sorted(objects_to_discover)
	start_counter = 0
	concurrency_limit = 25
	while start_counter < len(objects_list):
	end_counter = start_counter + concurrency_limit
	if end_counter >= len(objects_list):
	end_counter = len(objects_list)
	chunk = objects_list[start_counter:end_counter]
	chunk_args = [
	[
	sf,
	sobject_name,
	entries,
	sf_custom_setting_objects,
	object_to_tag_references,
	]
	for sobject_name in chunk]
	run_concurrently(discover_stream, chunk_args)
	start_counter = end_counter
	objects_list = sorted(objects_to_discover)
	concurrency_limit = 25
	for start_counter in range(0, len(objects_list), concurrency_limit):
	chunk = objects_list[start_counter:start_counter + concurrency_limit]
	chunk_args = [
	(sf, sobject_name, entries, sf_custom_setting_objects, object_to_tag_references)
	for sobject_name in chunk
	]
	run_concurrently(discover_stream, chunk_args)

Seems a little bit clearer like that