Feat/compiler #483

happytomatoe · 2024-10-03T12:28:08Z

Improved Jack compiler. Underneath the covers it uses ANTLR
List of improvements:

Added more error messages
Closes Jack compiler is lagging with big file. Ofc we can speed it up by calling compiler only once. As you can see that 1 key stroke calls compiler 2 times on screenshot.

Cons:

I've removed the interfaces for built in classes. Though we don't have a validation rule to check if you supplying the right arguments(type) when calling OS classes
If we want to use advanced ANTLR feature better to switch to JS target

Don't get scared because it says 13k LOC. Most of it are tests and test files. The src changes is around 1k LOC. This doesn't include the generated files.

You can check it out on https://happytomatoe.github.io/web-ide/compiler

TODO:

Test lexer errors
Test parser
Test validations
Fix error message if class name doesn't match file name
Validation doesn't work for empty function without return
Fix bug connected to an error when playing Pong

simulator/src/jack/builtins.ts

simulator/src/jack/listener/binder.listener.ts

simulator/src/jack/listener/validator.listener.ts

happytomatoe · 2024-10-11T19:00:26Z

@happytomatoe I hope you're able to keep making progress on this, but I wanted to let you know that I'm unavailable for the coming week. I'll be back October 20th to review any changes.

Have a nice holiday if it's a holiday)

…piler

happytomatoe · 2024-10-20T09:04:39Z

Small update - I was fed up working with official antlr4 typescript target. Migrated to antlr4ng

DavidSouther · 2024-10-21T17:50:51Z

Small update - I was fed up working with official antlr4 typescript target. Migrated to antlr4ng

Have you looked at Treesitter at all? (question, not a suggestion, but I hear great things about it and might be where I'd start if I was going from scratch today).

DavidSouther

Looking great, these are mostly optional thoughts at this point, up to you to incorporate them now or hold off a bit.

DavidSouther · 2024-10-21T18:47:20Z

components/src/stores/compiler.store.ts

    ) {
      state.files[name] = content;
      state.isCompiled = false;
-      this.compile(state);
+    },


Ok. Please do your best in the meantime to keep this as tidy as you can.

DavidSouther · 2024-10-21T18:50:01Z

simulator/src/jack/symbol.ts

@@ -11,6 +11,7 @@ export interface SubroutineInfo {
  type: SubroutineType;
  localVarsCount?: number;
 }
+
 export type GlobalSymbolTable = Record<string, GenericSymbol>;


I see in several places this is used specifically as a map, rather than an ad-hoc object. Should this be Map<string, GenericSymbol>? It'll enforce type checking the possibly null .get response, but will also make it easier to check .size instead of Object.keys(...)

(Edit: this is the current interface with the compile, so that would also need to change. Agreed to do that leter.)

DavidSouther · 2024-10-21T18:50:34Z

simulator/package.json

@@ -22,13 +22,14 @@
    "@nand2tetris/projects": "^1.0.0",
    "@nand2tetris/runner": "^1.0.0",
    "@types/node": "^20.14.2",
-    "antlr4": "^4.13.2",
+    "antlr4ng": "^3.0.7",


Does this change need a change in the README?

I've already changed the README. Both of these libraries implement the same api/interfaces

This library already has a ton of comments in typescript sources so I don't know if makes sense to add anything else in terms of documentation

DavidSouther · 2024-10-21T18:51:45Z

simulator/src/jack/anltr.compiler.ts

+  files: Record<string, string>,
+  cmd: Command,
+): Record<string, string | CompilationError> {
+  if (files.type == "LexerOrParserError") {
+    throw new Error("Expected tree but got a lexer or parser error");
+  }
+  const result: Record<string, string | CompilationError> = {};
+  for (const name of Object.keys(files)) {
+    result[name] = "";
+  }
+  const trees: Record<string, ProgramContext> = {};
+  const errors: Record<string, CompilationError> = {};


For most of these - are these Objects or Maps? (It's OK if they're records, but please leave a comment if that's a specific design choice, and why Record is preferred to Map)

These are maps. I will change it

@DavidSouther Why to choose Map in these cases?

After doing a small research on this topic I don't see any substantial difference other than a little bit better api.

DavidSouther · 2024-10-21T18:53:24Z

simulator/src/jack/anltr.compiler.ts

+      return errors;
+    }
+    const validateTree = treeOrErrors as ProgramContext;
+    const vmWriter = new VMWriter(


Ahh, I added a comment earlier about Record vs Map, but VMWriter wants a Record. Hmm, maybe that should have been a Map too. Up to you to change to using Maps, or acknowledging the comment and moving on.

simulator/src/jack/antlr.compiler.test.ts

simulator/src/jack/listener/vm.writer.listener.ts

happytomatoe · 2024-10-22T07:02:48Z

Small update - I was fed up working with official antlr4 typescript target. Migrated to antlr4ng

Have you looked at Treesitter at all? (question, not a suggestion, but I hear great things about it and might be where I'd start if I was going from scratch today).

What do you hear specifically? It's a pure C library with language bindings. I didn't work with such combination. Probably it has it's own pain points. I don't know.
I've seen this tool when I was building prettier plugin for Jack as it can build syntax tree(Tree with whitespaces and comments.)

If you would build this from scratch, based on my experience, the most go to options are

Antlr - because it's old and mature. There are commercial companies that use this tool - https://strumenta.com/. There are a lot documentation/resources that can help with niche problems. On the main website looks like Python used this library before switching to own implementation.
Chevrotain - I've seen java prettier plugin and probably of others prettier plugins that use chevrotain. If comparing with previous option it probably is faster (according to their benchmark). Though we probably don't need a faster lexer/parser. I don't know how mature is this library. I've seen a PR where they are implementing a feature from ANTLR. I also don't know if there are commercial companies that use it.

happytomatoe · 2024-10-22T17:23:10Z

Testing is done from my subjective point of view.

simulator/src/jack/antlr.compiler.test.ts

simulator/src/jack/symbol.ts

happytomatoe added 27 commits September 24, 2024 22:35

Init

95131df

WIP

53b5a0b

WIP

1f3b3f4

WIP

240aedc

WIP

3d6f8be

WIP

49075d7

Add symbol table. Add validator skeleton

f9dab02

Add more validations

029f124

Add more validations

d07a587

Add unknown subroutine return type error

3a69078

Add unknown type error

b541d22

Add CFG for simple cases

ab1e0c9

Add not all pahts retun validation

59f7507

Add print binary tree function

730f72e

Add validations for subroutine call

d61c6ca

Add builtin functions

2fb01eb

Add function/method called as function/method

81193e2

Add more validations

04d720f

Add int out of range validation

1511c94

Add field cant be referenced in a function validation

c40bfdd

this can't be referenced in a function validation

32b91f7

Fix bug in validation

d612816

WIP

5716258

WIP

aee5ec7

WIP

fec5790

WIP

3c1185a

WIP

af250f2

happytomatoe commented Oct 3, 2024

View reviewed changes

simulator/src/jack/builtins.ts Outdated Show resolved Hide resolved

happytomatoe commented Oct 3, 2024

View reviewed changes

simulator/src/jack/listener/binder.listener.ts Outdated Show resolved Hide resolved

happytomatoe commented Oct 3, 2024

View reviewed changes

simulator/src/jack/listener/validator.listener.ts Outdated Show resolved Hide resolved

happytomatoe added 4 commits October 11, 2024 20:24

WIP

98f696e

WIP

12edbb1

Remove ! signs

1dfc0ce

WIP

70cbcec

David Souther and others added 9 commits October 11, 2024 15:28

Format error.ts file

30f1b9c

WIP

ce31d3a

Fix bug in jack lexer. Fix cli for compiler

bf7acd0

Merge remote-tracking branch 'fork/feat/compilererrors' into feat/com…

179a39c

…piler

Fix bugs

3e35604

Generate antlr files

7604ef5

Migrate to antlr4ng

773cf3d

WIP

7c154b9

WIP

4bcce0a

happytomatoe added 6 commits October 20, 2024 11:05

WIP

8b71fdb

WIP

b573d00

WIP

7924123

WIP

8afba76

Fix formatting and lint errors

2dbf34d

WIP

a68a248

DavidSouther requested changes Oct 21, 2024

View reviewed changes

happytomatoe added 4 commits October 22, 2024 17:26

WIP

a2620ec

WIP

d41d95e

Fix bug with never ending stack increase

078f95b

Formatting

520ce31

DavidSouther approved these changes Oct 23, 2024

View reviewed changes

simulator/src/jack/antlr.compiler.test.ts Outdated Show resolved Hide resolved

simulator/src/jack/symbol.ts Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/compiler #483

Feat/compiler #483

happytomatoe commented Oct 3, 2024 •

edited

Loading

happytomatoe commented Oct 11, 2024 •

edited

Loading

happytomatoe commented Oct 20, 2024

DavidSouther commented Oct 21, 2024

DavidSouther left a comment

DavidSouther Oct 21, 2024

DavidSouther Oct 21, 2024

DavidSouther Oct 21, 2024

happytomatoe Oct 22, 2024 •

edited

Loading

happytomatoe Oct 22, 2024

DavidSouther Oct 21, 2024

happytomatoe Oct 22, 2024

happytomatoe Oct 22, 2024

happytomatoe Oct 22, 2024 •

edited

Loading

DavidSouther Oct 21, 2024

happytomatoe commented Oct 22, 2024 •

edited

Loading

happytomatoe commented Oct 22, 2024

Feat/compiler #483

Are you sure you want to change the base?

Feat/compiler #483

Conversation

happytomatoe commented Oct 3, 2024 • edited Loading

happytomatoe commented Oct 11, 2024 • edited Loading

happytomatoe commented Oct 20, 2024

DavidSouther commented Oct 21, 2024

DavidSouther left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happytomatoe Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happytomatoe Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happytomatoe commented Oct 22, 2024 • edited Loading

happytomatoe commented Oct 22, 2024

happytomatoe commented Oct 3, 2024 •

edited

Loading

happytomatoe commented Oct 11, 2024 •

edited

Loading

happytomatoe Oct 22, 2024 •

edited

Loading

happytomatoe Oct 22, 2024 •

edited

Loading

happytomatoe commented Oct 22, 2024 •

edited

Loading