Tonel doesn't support custom #compilerClass

[syrel]

The ability to override Behavior>>#compilerClass is a great feature that shows the power of Pharo. Unfortunately, Tonel is designed to only handle valid Smalltalk code inside of the method body. Which contradicts custom compilers.

Imagine if ones creates a compiler that changes comment syntax from double quotes "comment" to double slash // comment.

Such method would be written as follows in a Tonel file:

{ #category : #accessing }
MyClass >> method [
	// this example shows the new syntax that's great
	^ 42
]

Notice that there is a single quote ' inside of the comment. Because Tonel matches blocks, strings and comments, the method is not loadable. Here is how to reproduce it:

(TonelSourceScanner on: '[
	// this example shows the new syntax that''s great
	^ 42
]' readStream) scan

What is your vision of the future of Tonel? Will it only handle valid smalltalk code?

A possible solution would be to escape all special characters and only match unescaped [ ] to detect a method body.