Text area (#2931)

* Add docstring and switch to tree-sitter-languages wheels - although the wheels arent working * Adding highlights files * Fix index error on SyntaxAwareDocument * Narrowing highlighting scope * Adding basic highlights for Markdown * Using utf-8 byte length instead of codepoint count in syntax aware doc * Start creating an ABC defining functionality required by Document impls * Simplify tree-sitter logic * Extracting more ABC * Fix width calculation, add SyntaxTheme * Ensure the highlight line style goes right to the very end * Updating a docstring * Renaming, and adding document width guide * Ensuring that line number column toggling refreshes virtual size * Ensuring that line number column toggling refreshes virtual size * Width guide * Fix focus event stopping * Use release_mouse * Improving a docstring * Remove bash * TextArea language snapshot testing * Updating snapshots for TextArea since we now highlight more nodes * Typing fixes * Testing * Adding tests * Fixing language selection * Refresh size on indent width change * Testing, renaming, fixing display of selection * Fix multibyte highlight glitch * Fix deleting right with selection at end of document in TextArea * Fixing utf-8 multibyte character issues * Default location of text insertion is cursor position, add cursor_location properties * Removing some debugging code * Cursor location tests * Updating snapshots * Cached utf8 encoding * TextArea selection snapshot testing * Tidying docstrings and queries * Updating selection snapshot output * Binding for ESC to shift focus * Only build the tree-sitter query once! * Expand cursor scroll horizontal leeway in TextArea * Property setter for cursor_location in TextArea shouldnt return value * Avoiding NamedTuple subclassing - using type aliasing instead * Tidying API, docstrings etc. * Tidying the API and docstrings * TextArea additional cursor tests * Testing pageup and pagedown in TextArea * Fix a faulty test * Docstring in a test for TextArea edit * Stop using DEFAULT_SYNTAX_THEME * Docstrings * Change cursor_destination to move_cursor, add more tests * Remove faulty assertion * Tidying cursor movement * Tidying up, adding docstrings for component classes * Fix a broken selection test * Remove some unused highlighting machinery * Fix some Python highlighting issues * Make HTML syntax highlight nicely * Create tag name for mismatching HTML end tag * Add styling for YAML, update boolean styling * Stylising toml types * Styling floats * JSON syntax highlighting * Updating snapshots * Syntax highlighting datetimes in TOML * Namespace TOML errors in highlighting * Add a move_cursor_relative method * Update TOML TextArea snapshot for datetime highlighting support * Adjusting selections * At TextArea widget level, delete_range is insert_range of empty string * Refactoring * Dunder all, docstring fix * Fix XFAIL * Remove unused import * More tests, tidying up * Cleaning the API * Docstrings for TextArea * A bunch of docstrings, delete unused code * More tidying and docstrings * Cursor origin on document load, correctly handle delete word left/right when selection is non-empty, fix delete_line when selection spans multiple lines and is in reverse direction * Moving things around * Fixing dunder all to export DocumentBase * Add docstring * Record cursor width on programmatic insert since it can result in the cursor moving * Typing fixes * Fixing remaining typing issues with TextArea * Add tree-sitter-languages stubs and fix typing issues in documents * Fixing remaining typing issues with document * Updating Syntax themes * Improve highlighting, add initial TextArea docs page * Add TextArea indent note * Start TextArea guide inside reference * Add TextArea to widget gallery * Fleshing out TextArea docs * Add note * Fix TextArea programmatic insert/cursor interaction * Improve a test * Testing replacement within selection * Testing double-width character keyboard navigation and deletion keybinds with active selections * Testing "delete to start of line" TextArea binding * Testing TextArea delete line methods and delete to end of line * Testing shift selecting using keyboard in vertical direction * Expand tests for home and end keybinds in TextArea * Renaming tests, testing empty replace and insert * Testing delete word left via API * Testing delete word left via API * Testing delete_word_left with tabs, and delete_word_right * Remove unused variables * Remove debugging width guide * Fix snapshot report path * Deleting word left/right interaction with line ends fixes, ensure cursor width recorded on all edits * Docstring fixes * Unpin textual snapshot library dependency (issue is fixed) * Docstring fixes * Fix recording cursor width * Fix a docstring * Add select_all to TextArea * Remove unused tree-sitter stuff from .gitignore * Line select * Make word pattern private in TextArea * Add blinking cursor to TextArea * Renaming, adding missing return typing * Add selection bindings * Moving cursor left/right by word while selecting * Change escape keybind description, TextArea * Stripping whitespace when going word left/right * Add missing annotation * Cursor word right and left parity with PyCharm * Use repaint=False for cursor blink * Improve focus/blur styling * A whole bunch of TextArea testing * Simplify delete_left and delete_right * Testing hiding line numbers in snapshot * Adding snapshot test for unfocus styling * Create initial snapshot for text-area unfocused * Support shift+home, shift+end * Document shift+home, shift+end * Add Dracula syntax highlighting theme * Small change to delete_line behaviour when multiple lines selected to match vscode/pycharm behaviour * Add test for new delete line logic * Delete line improvement * Add extra test for delete_line multiple selection * Test cursor "smart" home behaviour * Fix typo * Highlight matching brackets * Update snapshot * Update snapshot * Fix xfails * Simplify delete_word_left * Catch correct exception to ensure support for Python 3.7 * Add styling for Markdown * Add styles for Dracula for Markdown * Remove unused _fix_direction.py * Add docstring to EditResult * Use default=0 in max inside Document * Remove redundant actions * Use cell-width aware expand tabs implementation from @willmcgugan * Construct strip with cell length * Some TextArea keyword-only arguments * Begin moving over to TextAreaTheme #skipci * Prepare queries inside document #skip-ci * Add comment * Refactoring * TextAreaTheme styling * Setting width of blank selected lines * Building the highlight map in the text area * Remove unused default css from TextArea * Moving highlighting stylize into widget * Moving syntax highlighting into TextArea widget * Remove unused code * Optimise imports * Fix highlighting when initial text supplied to TextArea * Rebuild highlight map when the theme changes * Extending * Restore themes * Remove old comment, fix docstring * Fixing docstrings * Fixing mypy * Fixing mypy issues in document * Tidying things * Updating version * Add theme * Fix VSCode theme bracket matching * Only match brackets when theres no selection * Highlighting tidying * Fix markdown header highlighting * Setting theme correctly in background * Tidying module interface * Merging main * Fixing a bunch of typing problems * Fixing more typing problems * Correctly setting theme object * mypy * Small fix to bracket matching * Improve a docstring * Fix docstring * Testing builtin and custom languages * Unit testing theme stuff * Reworking themes * Error handling * Improve error message * Testing new theme setting approach, error handling * Improvements/tests for theme and language setting * Remove unused TextArea unfocus snapshot * Update snapshot file * Adding theme snapshot tests * Add `function.call` style binding in dark vscode theme * Renaming a test file * Making active line clearer on vscode theme * Renaming tests * A whole lot of docs for TextArea * Update wording in docs * A bit more docs * Example on adding Java as a custom language * More custom language docs * Finishing up custom themeing/syntax highlighting guide for TextArea * Add note on potential issue * Fix wording * Add note on Apple Silicon Python 3.7 fallback * Add another note on Apple Silicon Python 3.7 fallback * Fix class names in example files * Add some documentation for useful TextArea APIs * TextArea docs improvements * TextArea docs typo fix * Note about extending TextArea * Tab-stop support when spaces used for indent * Docs update * Text area blog post (#3356) * Start blog post * Add demo script to blog post * Continuing the blog post * Yet more writing for TextArea blog post * Working on closing section * Finishing up * Update docs/blog/posts/text-area-learnings.md Co-authored-by: Dave Pearson <[email protected]> * Update docs/blog/posts/text-area-learnings.md Co-authored-by: Dave Pearson <[email protected]> * Typo fix * Update docs/blog/posts/text-area-learnings.md Co-authored-by: Dave Pearson <[email protected]> --------- Co-authored-by: Dave Pearson <[email protected]> * Remove redundant pass * Add docstring * Docs fix * Simplify docs * Improve docstring * Add links in docstrings --------- Co-authored-by: Dave Pearson <[email protected]>
Textualize · Sep 21, 2023 · bbde62f · bbde62f
1 parent d204ba8
commit bbde62f
Show file tree

Hide file tree

Showing 65 changed files with 12,060 additions and 1,125 deletions.
diff --git a/.github/workflows/pythonpackage.yml b/.github/workflows/pythonpackage.yml
@@ -55,4 +55,4 @@ jobs:
         uses: actions/upload-artifact@v3
         with:
           name: snapshot-report-textual
-          path: tests/snapshot_tests/output/snapshot_report.html
+          path: snapshot_report.html
diff --git a/docs/blog/images/text-area-learnings/cursor_position_updating_via_api.png b/docs/blog/images/text-area-learnings/cursor_position_updating_via_api.png
diff --git a/docs/blog/images/text-area-learnings/maintain_offset.gif b/docs/blog/images/text-area-learnings/maintain_offset.gif
diff --git a/docs/blog/images/text-area-learnings/text-area-api-insert.gif b/docs/blog/images/text-area-learnings/text-area-api-insert.gif
diff --git a/docs/blog/images/text-area-learnings/text-area-pyinstrument.png b/docs/blog/images/text-area-learnings/text-area-pyinstrument.png
diff --git a/docs/blog/images/text-area-learnings/text-area-syntax-error.gif b/docs/blog/images/text-area-learnings/text-area-syntax-error.gif
diff --git a/docs/blog/images/text-area-learnings/text-area-theme-cycle.gif b/docs/blog/images/text-area-learnings/text-area-theme-cycle.gif
diff --git a/docs/blog/images/text-area-learnings/text-area-welcome.gif b/docs/blog/images/text-area-learnings/text-area-welcome.gif
diff --git a/docs/blog/posts/text-area-learnings.md b/docs/blog/posts/text-area-learnings.md
@@ -0,0 +1,210 @@
+---
+draft: false
+date: 2023-09-18
+categories:
+  - DevLog
+authors:
+  - darrenburns
+---
+
+# Things I learned while building Textual's TextArea
+
+`TextArea` is the latest widget to be added to Textual's [growing collection](https://textual.textualize.io/widget_gallery/).
+It provides a multi-line space to edit text, and features optional syntax highlighting for a selection of languages.
+
+![text-area-welcome.gif](../images/text-area-learnings/text-area-welcome.gif)
+
+Adding a `TextArea` to your Textual app is as simple as adding this to your `compose` method:
+
+```python
+yield TextArea()
+```
+
+Enabling syntax highlighting for a language is as simple as:
+
+```python
+yield TextArea(language="python")
+```
+
+Working on the `TextArea` widget for Textual taught me a lot about Python and my general
+approach to software engineering. It gave me an appreciation for the subtle functionality behind
+the editors we use on a daily basis — features we may not even notice, despite
+some engineer spending hours perfecting it to provide a small boost to our development experience.
+
+This post is a tour of some of these learnings.
+
+<!-- more -->
+
+## Vertical cursor movement is more than just `cursor_row++`
+
+When you move the cursor vertically, you can't simply keep the same column index and clamp it within the line.
+Editors should maintain the visual column offset where possible,
+meaning they must account for double-width emoji (sigh 😔) and East-Asian characters.
+
+![maintain_offset.gif](../images/text-area-learnings/maintain_offset.gif){ loading=lazy }
+
+Notice that although the cursor is on column 11 while on line 1, it lands on column 6 when it
+arrives at line 3.
+This is because the 6th character of line 3 _visually_ aligns with the 11th character of line 1.
+
+
+## Edits from other sources may move my cursor
+
+There are two ways to interact with the `TextArea`:
+
+1. You can type into it.
+2. You can make API calls to edit the content in it.
+
+In the example below, `Hello, world!\n` is repeatedly inserted at the start of the document via the
+API.
+Notice that this updates the location of my cursor, ensuring that I don't lose my place.
+
+![text-area-api-insert.gif](../images/text-area-learnings/text-area-api-insert.gif){ loading=lazy }
+
+This subtle feature should aid those implementing collaborative and multi-cursor editing.
+
+This turned out to be one of the more complex features of the whole project, and went through several iterations before I was happy with the result.
+
+Thankfully it resulted in some wonderful Tetris-esque whiteboards along the way!
+
+<figure markdown>
+  ![cursor_position_updating_via_api.png](../images/text-area-learnings/cursor_position_updating_via_api.png){ loading=lazy }
+  <figcaption>A TetrisArea white-boarding session.</figcaption>
+</figure>
+
+Sometimes stepping away from the screen and scribbling on a whiteboard with your colleagues (thanks [Dave](https://fosstodon.org/@davep)!) is what's needed to finally crack a tough problem.
+
+Many thanks to [David Brochart](https://mastodon.top/@davidbrochart) for sending me down this rabbit hole!
+
+## Spending a few minutes running a profiler can be really beneficial
+
+While building the `TextArea` widget I avoided heavy optimisation work that may have affected
+readability or maintainability.
+
+However, I did run a profiler in an attempt to detect flawed assumptions or mistakes which were
+affecting the performance of my code.
+
+I spent around 30 minutes profiling `TextArea`
+using [pyinstrument](https://pyinstrument.readthedocs.io/en/latest/home.html), and the result was a
+**~97%** reduction in the time taken to handle a key press.
+What an amazing return on investment for such a minimal time commitment!
+
+
+<figure markdown>
+  ![text-area-pyinstrument.png](../images/text-area-learnings/text-area-pyinstrument.png){ loading=lazy }
+  <figcaption>"pyinstrument -r html" produces this beautiful output.</figcaption>
+</figure>
+
+pyinstrument unveiled two issues that were massively impacting performance.
+
+### 1. Reparsing highlighting queries on each key press
+
+I was constructing a tree-sitter `Query` object on each key press, incorrectly assuming it was a
+low-overhead call.
+This query was completely static, so I moved it into the constructor ensuring the object was created
+only once.
+This reduced key processing time by around 94% - a substantial and very much noticeable improvement.
+
+This seems obvious in hindsight, but the code in question was written earlier in the project and had
+been relegated in my mind to "code that works correctly and will receive less attention from here on
+out".
+pyinstrument quickly brought this code back to my attention and highlighted it as a glaring
+performance bug.
+
+### 2. NamedTuples are slower than I expected
+
+In Python, `NamedTuple`s are slow to create relative to `tuple`s, and this cost was adding up inside
+an extremely hot loop which was instantiating a large number of them.
+pyinstrument revealed that a large portion of the time during syntax highlighting was spent inside `NamedTuple.__new__`.
+
+Here's a quick benchmark which constructs 10,000 `NamedTuple`s:
+
+```toml
+❯ hyperfine -w 2 'python sandbox/darren/make_namedtuples.py'
+Benchmark 1: python sandbox/darren/make_namedtuples.py
+  Time (mean ± σ):      15.9 ms ±   0.5 ms    [User: 12.8 ms, System: 2.5 ms]
+  Range (min … max):    15.2 ms …  18.4 ms    165 runs
+```
+
+Here's the same benchmark using `tuple` instead:
+
+```toml
+❯ hyperfine -w 2 'python sandbox/darren/make_tuples.py'
+Benchmark 1: python sandbox/darren/make_tuples.py
+  Time (mean ± σ):       9.3 ms ±   0.5 ms    [User: 6.8 ms, System: 2.0 ms]
+  Range (min … max):     8.7 ms …  12.3 ms    256 runs
+```
+
+Switching to `tuple` resulted in another noticeable increase in responsiveness.
+Key-press handling time dropped by almost 50%!
+Unfortunately, this change _does_ impact readability.
+However, the scope in which these tuples were used was very small, and so I felt it was a worthy trade-off.
+
+
+## Syntax highlighting is very different from what I expected
+
+In order to support syntax highlighting, we make use of
+the [tree-sitter](https://tree-sitter.github.io/tree-sitter/) library, which maintains a syntax tree
+representing the structure of our document.
+
+To perform highlighting, we follow these steps:
+
+1. The user edits the document.
+2. We inform tree-sitter of the location of this edit.
+3. tree-sitter intelligently parses only the subset of the document impacted by the change, updating the tree.
+4. We run a query against the tree to retrieve ranges of text we wish to highlight.
+5. These ranges are mapped to styles (defined by the chosen "theme").
+6. These styles to the appropriate text ranges when rendering the widget.
+
+<figure markdown>
+  ![text-area-theme-cycle.gif](../images/text-area-learnings/text-area-theme-cycle.gif){ loading=lazy }
+  <figcaption>Cycling through a few of the builtin themes.</figcaption>
+</figure>
+
+Another benefit that I didn't consider before working on this project is that tree-sitter
+parsers can also be used to highlight syntax errors in a document.
+This can be useful in some situations - for example, highlighting mismatched HTML closing tags:
+
+<figure markdown>
+  ![text-area-syntax-error.gif](../images/text-area-learnings/text-area-syntax-error.gif){ loading=lazy }
+  <figcaption>Highlighting mismatched closing HTML tags in red.</figcaption>
+</figure>
+
+Before building this widget, I was oblivious as to how we might approach syntax highlighting.
+Without tree-sitter's incremental parsing approach, I'm not sure reasonable performance would have
+been feasible.
+
+## Edits are replacements
+
+All single-cursor edits can be distilled into a single behaviour: `replace_range`.
+This replaces a range of characters with some text.
+We can use this one method to easily implement deletion, insertion, and replacement of text.
+
+- Inserting text is replacing a zero-width range with the text to insert.
+- Pressing backspace (delete left) is just replacing the character behind the cursor with an empty
+  string.
+- Selecting text and pressing delete is just replacing the selected text with an empty string.
+- Selecting text and pasting is replacing the selected text with some other text.
+
+This greatly simplified my initial approach, which involved unique implementations for inserting and
+deleting.
+
+
+## The line between "text area" and "VSCode in the terminal"
+
+A project like this has no clear finish line.
+There are always new features, optimisations, and refactors waiting to be made.
+
+So where do we draw the line?
+
+We want to provide a widget which can act as both a basic multiline text area that
+anyone can drop into their app, yet powerful and extensible enough to act as the foundation
+for a Textual-powered text editor.
+
+Yet, the more features we add, the more opinionated the widget becomes, and the less that users
+will feel like they can build it into their _own_ thing.
+Finding the sweet spot between feature-rich and flexible is no easy task.
+
+I don't think the answer is clear, and I don't believe it's possible to please everyone.
+
+Regardless, I'm happy with where we've landed, and I'm really excited to see what people build using `TextArea` in the future!
diff --git a/docs/examples/widgets/horizontal_rules.py b/docs/examples/widgets/horizontal_rules.py
@@ -1,6 +1,6 @@
 from textual.app import App, ComposeResult
-from textual.widgets import Rule, Label
 from textual.containers import Vertical
+from textual.widgets import Label, Rule
 
 
 class HorizontalRulesApp(App):

diff --git a/docs/examples/widgets/java_highlights.scm b/docs/examples/widgets/java_highlights.scm
@@ -0,0 +1,140 @@
+; Methods
+
+(method_declaration
+  name: (identifier) @function.method)
+(method_invocation
+  name: (identifier) @function.method)
+(super) @function.builtin
+
+; Annotations
+
+(annotation
+  name: (identifier) @attribute)
+(marker_annotation
+  name: (identifier) @attribute)
+
+"@" @operator
+
+; Types
+
+(type_identifier) @type
+
+(interface_declaration
+  name: (identifier) @type)
+(class_declaration
+  name: (identifier) @type)
+(enum_declaration
+  name: (identifier) @type)
+
+((field_access
+  object: (identifier) @type)
+ (#match? @type "^[A-Z]"))
+((scoped_identifier
+  scope: (identifier) @type)
+ (#match? @type "^[A-Z]"))
+((method_invocation
+  object: (identifier) @type)
+ (#match? @type "^[A-Z]"))
+((method_reference
+  . (identifier) @type)
+ (#match? @type "^[A-Z]"))
+
+(constructor_declaration
+  name: (identifier) @type)
+
+[
+  (boolean_type)
+  (integral_type)
+  (floating_point_type)
+  (floating_point_type)
+  (void_type)
+] @type.builtin
+
+; Variables
+
+((identifier) @constant
+ (#match? @constant "^_*[A-Z][A-Z\\d_]+$"))
+
+(identifier) @variable
+
+(this) @variable.builtin
+
+; Literals
+
+[
+  (hex_integer_literal)
+  (decimal_integer_literal)
+  (octal_integer_literal)
+  (decimal_floating_point_literal)
+  (hex_floating_point_literal)
+] @number
+
+[
+  (character_literal)
+  (string_literal)
+] @string
+
+[
+  (true)
+  (false)
+  (null_literal)
+] @constant.builtin
+
+[
+  (line_comment)
+  (block_comment)
+] @comment
+
+; Keywords
+
+[
+  "abstract"
+  "assert"
+  "break"
+  "case"
+  "catch"
+  "class"
+  "continue"
+  "default"
+  "do"
+  "else"
+  "enum"
+  "exports"
+  "extends"
+  "final"
+  "finally"
+  "for"
+  "if"
+  "implements"
+  "import"
+  "instanceof"
+  "interface"
+  "module"
+  "native"
+  "new"
+  "non-sealed"
+  "open"
+  "opens"
+  "package"
+  "private"
+  "protected"
+  "provides"
+  "public"
+  "requires"
+  "return"
+  "sealed"
+  "static"
+  "strictfp"
+  "switch"
+  "synchronized"
+  "throw"
+  "throws"
+  "to"
+  "transient"
+  "transitive"
+  "try"
+  "uses"
+  "volatile"
+  "while"
+  "with"
+] @keyword