Update /generate to not split classes & functions across cells

[srdas]

Fixes #1111

/generate will sometimes split a class or a function across multiple code cells. After tracing the code in generate.py the error arises from unanticipated LLM behavior in the curation of the resultant Jupyter notebook in the creation of the outline object. This is best handled by post-processing the notebook to merge "hanging" code cells with the preceding code cell.

The error looks like this:

After the fix, the error is no longer present. See this example:

Here is another example of the rectified notebook generation:

if necessary to check the log, add print statements in lines 236, 240, 245 to the code as shown below.

The before and after versions of the logged notebook can be compared for lines with ***** CELL 1 ***** as shown here:

[dlqqq]

@srdas Awesome work! Love the creativity applied here. Few minor comments below.

[dlqqq]

It may be worth renaming this variable to avoid confusion with nb["cells"].

Suggested change

	nb_cells = []
	merged_cells = []

[dlqqq]

The True branch requires nb_cells[-1] to exist, otherwise it will throw an exception. The condition cell["source"][0] == " " also requires cell["source"] to have at least 1 char, which is not guaranteed.

The condition can be updated to avoid these edge cases:

Suggested change

	if (cell["cell_type"] == "code") and (cell["source"][0] == " "):
	follows_code_cell = nb_cells and nb_cells[-1]["cell_type"] == "code"
	is_incomplete = cell["cell_type"] == "code" and cell["source"].startswith(" ")
	if follows_code_cell and is_incomplete:

Note that nb_cells should be renamed to merged_cells if the previous suggestion is accepted.

[srdas]

Made all suggested changes! Thanks for the help.

[dlqqq]

@srdas One other question: with this change, are you able to run every notebook generated via /generate? It would be helpful to identify & document what issues still exist with /generate so that they can be fixed in the future.

[srdas]

During testing across several different /generate use cases, one more issue seemed to occur (though somewhat rarely) -- there are a few cells that contains markdown text but the cell_type is "code" instead. Therefore, added another sweep through the cells to detect these cases and flip the cell_type to markdown.

Additional code to handle this is a new function:

And a second post-processing pass through the notebook to fix cells that should be markdown:

By adding print statements, this was checked as shown below, the cell_type before and after the rectification is shown:

See also the corresponding fix in the notebook:

The last example:

Notebooks usually run from end to end in most cases. The two cases when they do not are unrelated to the /generate code. These are:

1. A missing package
2. The function in the package being used is deprecated and the generated code is still using the old version. This is simply the fact that the LLM is not up to date.