Implement Java Interface with Frege Record

[matil019]

This is a proposal which introduces a new syntax that allows Frege programmers to implement Java Interfaces with Frege Record declarations.

Proof of Concept

Supports the "Pure" case only: #361

Short description and Examples

native new functions take Frege records as their arguments and uses their fields to implement the methods of Java interface.

Pure

public interface J {
  public int succ(int i);
  public int succ(int i, int n);
}

data JImpl = JImpl
  { succ :: Int -> Int
  , succN :: Int -> Int -> Int
  }

data J = pure native J where
  pure native new "extends" :: JImpl -> J
  pure native succ :: J -> Int -> Int
  pure native succN succ :: J -> Int -> Int -> Int

main = do
    let j = J.new $ JImpl { succ = (+1) }
    println $ j.succ 1    -- 2
    let j = J.new $ JImpl { succ = (*10) }
    println $ j.succ 1    -- 10
    println $ j.succN 5 2 -- 500

Mutable

public interface Counter {
  public int get();
  public void increment();
}

data Impl = Impl
  { get :: ST s Int
  , increment :: ST s ()
  }

data Counter = native Counter where
  native new "extends" :: Impl -> STMutable s Counter
  native get :: Mutable s Counter -> ST s Int
  native increment :: Mutable s Counter -> ST s ()

main = do
    c <- do
        internal <- Ref.new 0
        Counter.new $ Impl
          { get = internal.get
          , increment = internal.modify succ
          }
    println =<< c.get    -- 0
    c.increment
    println =<< c.get    -- 1

Abstract class

public abstract class AC {
  public AC(int i) {
    // does something with i...
  }
  public abstract void doFoo();
}

data Impl = Impl { doFoo :: ST s () }

data AC = native AC where
  native new "extends" :: Impl -> Int -> AC
  native doFoo :: AC -> ST s ()

-- ...

Long description

Motivation

Java libraries and frameworks tend to provide Java interfaces, which client code must implement to work with them. Currently, in order to implement Java interfaces in Frege, Frege programmers must either (a) use the native module declarations, or (b) write Java source files separately. In either case, Frege programmers must work out the laziness of Frege arguments/return values and write boilerplates such as .call(), TST.performUnsafe(...), Thunk.lazy(...), etc. Since implementing interfaces and extending classes are the essential part of Java programming, having the compiler automate these tasks should greatly help writing Frege against existing huge Java codebase.

Concept

Implementing a Java interface is essentially providing a set of functions that implements its methods. "A set of (named) functions" can be represented with the record syntax in Frege, so currently one can write something like this:

module Foo where

native module where {
  public static I mk_(final TImpl impl) {
    return new I() {
      @Override
      public int getValue() {
        return TImpl.getValue(impl);
      }
    };
  }
}

data Impl = Impl { getValue :: Int }

data I = pure native I

pure native mk "Foo.mk_" :: Impl -> I

For concrete, working, real examples, please see modules such as this.

The native module part in the above example is a boilerplate and is to be automatically generated in the proposal. With the new syntax, it can be rewritten to:

data Impl = Impl { getValue :: Int }

data I = pure native I where
  pure native getValue :: I -> Int
  pure native new "extends" :: Impl -> I

Syntax

The new syntax introduced in this proposal is the Java-name "extends" in native function declarations:

[pure] native fregeName "extends"
  :: RecordType
  [-> constructor arguments (abstract classes only) ...]
  -> [STMutable s] NativeType (LHS of native data declaration)

Specifying "extends" instead of "new" generates an anonymous class declaration instead of a plain new expression.

Field/Method correspondence and Overload resolution

Since Java has method overloading, the syntax must support overload resolution. For this, it relies on the other native member declarations.

data Impl = Impl
  { add2 :: Int -> Int -> Int
  , add3 :: Int -> Int -> Int -> Int
  , ignored :: Int
  , garbage :: Int -> Bool
  }

data JavaI = pure native JavaI where
  pure native new "extends" :: Impl -> JavaI
  pure native add2 add :: JavaI -> Int -> Int -> Int
  pure native add3 add :: JavaI -> Int -> Int -> Int -> Int
  pure native garbage :: JavaI -> Int -> Int

Methods for an anonymous class are generated in the following algorithm:

1. For each fields in a record type (Impl),
2. Members of the target native type (JavaI) are looked up by Frege name
   (add2, add3, etc., but not add).
3. If a matching member is found for the field, a method is generated with the Java name. For example, because Impl.add2 matches JavaI.add2, and JavaI.add2 has Java-name add,

   public int add(int arg1, int arg2) {
     return TImpl.add2(impl, Thunk.lazy(arg1), Thunk.lazy(arg2)).call();
     // where `impl` is the parameter of `JavaI.new`
   }

4. Superfluous fields in the record (Impl.ignored) are ignored.
5. If signatures are incorrect (Impl.garbage), it's a javac-time error.
6. Both record fields and native members must exist for every methods of the interface. otherwise, it's a javac-time error.

[Ingo60]

This looks extremely cool!

Can't wait to try it out on the weekend.

Will prove very useful when we need to pass functions to java, where they will be expected as instances of some functional interface.

[Ingo60]

So, it looks like your branch has not made it past the merge. Anyway.
(Maybe you make yourself branches to work with in the github repo, so I can try something without merging it into master first.)

This looks extremely promising, though the details need, of course, some more love. For example, the following did not work:

frege> :history
data IRun = IRun { run :: IO () }
runPrint = IRun { run = println "Hi" }
import Java.Lang(Runnable)

frege> native newRunnable "extends" :: IRun → Runnable
E <console>.fr:5: newRunnable has an illegal return type
    for a method that is not pure, perhaps ST s Runnable
    would work

frege> native newRunnable "extends" :: IRun → IOMutable Runnable
Exception in thread "main" frege.runtime.NoMatch: TauT.name at line 146 no match for value frege.compiler.types.Types$TTauT$DTApp@712cfb63
	at frege.compiler.types.Types$TTauT.name(Types.java:4554)
	at frege.compiler.gen.java.MethodCall$1Let$18056.lambda$call$14856$14(MethodCall.java:1515)
	at frege.prelude.PreludeList.lambda$map$13(PreludeList.java:1935)
	at frege.run8.Thunk.call(Thunk.java:231)
	at frege.prelude.Maybe$1Let$3598.$lc999$2877(Maybe.java:1085)
	at frege.prelude.Maybe.catMaybes(Maybe.java:1110)
	at frege.prelude.Maybe.mapMaybe(Maybe.java:1115)
	at frege.compiler.gen.java.MethodCall$1Let$18056.lambda$call$14856$15(MethodCall.java:1508)
	at frege.control.monad.State$TState.lambda$run$1(State.java:1774)
	at frege.run8.Thunk.call(Thunk.java:231)
	at frege.compiler.gen.java.MethodCall.evalStG(MethodCall.java:1147)
	at frege.compiler.gen.java.MethodCall$1Let$18056.call$14856(MethodCall.java:1610)
	at frege.compiler.gen.java.MethodCall.nativeCall(MethodCall.java:1949)

[Ingo60]

By the way, I discovered that the code generation emits a bunch of useless code for records. Why, for example, does it generate an add3() method anyway? Can't remember the reason for this. But even if it's useful somewhere, why this

    final Func.U<Integer, Func.U<Integer, Integer>> a4$7632 = arg$1.mem$add2.call();

It's simply unused. I must have been drunk back then.

Anyway. I think we've got a bit of work to do here:

1. make the mutable case work

2. check polymorphism, i.e. we want to extend a

   class/interface Foo<A,B,C>

with

data Impl a b c = Impl { .... }

3. consider the case where the constructor needs arguments. Will the pseudo argument Impl be the first one all the time? or rather the last one, like in Java? (I think there will be good reasons to make it the first argument always, see below).
4. We should do our best to check for errors, rather than relying on the java compiler only. For instance, we can check if someone tries to extend a final class, tries to overwrite final methods, misses methods that must be implemented, etc.

Finally, we'll have a nice encoding for extending classes and interfaces. But, truth be told, it's a bit clumsy for ad hoc extensions. We can reduce the boilerplate by accepting the following expression (taking Runnable as an example):

Runnable { run = println "I'm here!" }

This is the same syntax that creates a record value, but instead of the constructor, we have the type name of a native type. In order to remain backwards compatible, the rule would be that when we see

Foo { name1 = expr1, name2 = expr2, ... }

it will be the construction of a record value if Foo resolves to a constructor name, and a class-extension if Foo resolves to a native type name (and an error otherwise). Then we generate a record type:

data AnonImpl ß = AnonImpl { run :: ß }

and a native function

native anon123 extends :: AnonImpl ß -> IOMuatble Runnable

and rewrite the above expression as:

anon123 (AnonImpl {run = println "I'm here!" })

This looks innocent enough to survive until typechecking.
We have now two possible ways to proceed:

1. we can infer the type of the record, here AnonImpl (IO ()) and check later (in gen/MethodCall) if there is a run method with type Runnable -> IO (). This is nice, but we may get in trouble when the extended class is polymorphic or when there are extra constructor arguments. Especially with extra constructor arguments, we'd like to check, rather than infer them if at all possible. Conversely, if the constructor itself is overloaded, we want to employ the machinery for overload resolution and type directed name resolution that is already in place. Hence

2. We should perhaps modify our scheme a bit and say that our extends function has the following type:

   native anon123 extends :: AnonImpl ß -> IOMutable Runnable -> IOMutable Runnable
   

and we translate

Runnable  {bar = println "Hi" } x y z

to

anon123 (AnonImpl {bar = println "Hi" }) (Runnable.new x y z)

or, if there are no arguments

anon123 (AnonImpl {bar = println "Hi" }) (Runnable.new ())

(disregarding the possibility that such an expression may appear curried, hence the following would not work:

 data J = pure native x.y.J where 
         pure native new :: Int -> J 
         pure native overridable :: J -> ....
 map J{overridable= ...} [1,2,3]

though

map (\x -> J{overridable= ...} x) [1,2,3]

would be ok.

Here, we have it especially easy in code generation: we can just generate the second argument, and change it into a JNewClass, appending the generated class body. We still need to check whether ß matches the type of (one of the overloads of) the extended method.

I guiess all this was a bit of a brain dump. I hope we can clear things up in further discussion.

[matil019]

Do you mean I should push Frege/frege instead of my fork? You can fetch branches in other repository like:

git fetch https://github.com/matil019/frege.git implement-interface:implement-interface

[Ingo60]

You choose what is most convenient for you regarding the branches. Just let me know if you need something.

[matil019]

All right, then I'll push to my fork, as I have always done.

We can reduce the boilerplate by accepting the following expression (taking Runnable as an example):

Runnable { run = println "I'm here!" }

Actually, this is what I had in mind when I came up with the idea. However, it doesn't always work with abstract classes. Class constructors themselves may have side-effects, as well as taking other arguments. I thought we should avoid a record-like construct having IO type.

native anon123 extends :: AnonImpl ß -> IOMutable Runnable -> IOMutable Runnable

and we translate

Runnable  {bar = println "Hi" } x y z

to

anon123 (AnonImpl {bar = println "Hi" }) (Runnable.new x y z)

We have to be careful here, because the correct constructor must be called just once at the time when an object is instantiated. I think we know that already, but the type signature ... -> IOMutable Runnable -> IOMutable Runnable suggests otherwise. well, not necessarily, as IO a doesn't create a until bound to next IO.

[matil019]

In case you missed, I already proposed a case where a Java class constructor takes arguments:

data AC = pure native AC where
  pure native new "extends" :: Impl -> Int -> AC

AC.new Impl{...} 42

would pass 42 to AC's constructor.

The reason why Impl is the first is just because it would be (a bit?) easier for code generation.

[Ingo60]

The point is, in a POC it is ok to have the Impl record types, but in practice it would mean even more boilerplate. Here we do need some kind of anonymous record types. But, unfortunately, we don't have anonymous records in Frege (and I'm not going to even think about trying to implement them ... see the desperate attempts of the Haskell community to solve their record problem).

Hence, the proposed syntax is just a special case (and the only posssible one without introducing new data types in the compiler, as far as I can see) where we can pretend to have anonymous records, but behind the scenes we derive a one-time-use named record type and let the type checker fill in the types. If we do it right, there need not be any traces of this type in the generated code. (Since we know that in anon123 the first argument must be a record construction (and not just some value that happens to be such a record), we can just pick up the code pieces that are interesting for us and then forget the rest so that it never gets actually emitted. Likewise, we can suppress the static inner class that is normally generated for such types (which would create an extra class file).

You are right that we cannot simply give a J.new to anon123 since this would not always appear as JNew java code, especially with mutable things, where it would come back as an invocation of a wrapper function. Thus we would instead have to overload the extends function in the same way the constructors are.

[matil019]

I think understand your point. Surely we don't want to write similar method signatures twice!

What I'm concerned right now is if the shorthand notation gets too magical. I'm continuing my approach to support mutable interfaces/methods to have a more concrete view.

[Ingo60]

There seems to be a problem with type applications:

frege> data PureFunction t r = pure native java.util.function.Function where pure native apply :: PureFunction t r -> t -> r
data type PureFunction :: pure native java.util.function.Function

frege> :java

frege> data Impl t r = Impl { apply :: t -> r }
data type Impl :: *->*->*

frege> pure native mk "extends" :: Impl t r -> PureFunction t r
native function mk :: Impl t r -> PureFunction t r

frege> :java

frege> pf = mk Impl{apply=Int.succ} 
Exception in thread "main" frege.runtime.NoMatch: TauT.name at line 146 no match for value frege.compiler.types.Types$TTauT$DTApp@58178b69

[matil019]

The code currently doesn't check from which data constructor TauT is constructed, and it tries to call TApp.name (which doesn't exist). It happens when Impl takes type variable(s). It's already fixed in my unpushed branch.

[matil019]

Just pushed it. Please see the latest branch if you are interested: https://github.com/matil019/frege/tree/implement-interface

[matil019]

I'm going to add the "receiver" parameter (a.k.a. this) to the fields of Implementation Records (records that implements Java interfaces) so that they have the same signatures as the ordinary native method declarations. This should simplify generating anonymous IRs and enables calling other methods from implementing code:

public abstract class JClass {
  public void concrete(int param) { /* do something ... */ }
  public abstract void abstr(int param);
}

data JClass = native JClass where
  native concrete :: JClass -> Int -> IO ()
  native abstr :: JClass -> Int -> IO ()

JClass { abstr = \this i -> println ("calling concrete(%d);".format i) >> this.concrete i }

[Ingo60]

Still, my Runnable example dies from (the same?) error:

Exception in thread "main" frege.runtime.NoMatch: TauT.name at line 168 no match for value frege.compiler.types.Types$TTauT$DTApp@56afdf9a
	at frege.compiler.types.Types$TTauT.name(Types.java:5061)
	at frege.compiler.gen.java.MethodCall$1Let$18980.lambda$call$15644$14(MethodCall.java:1531)

IMHO, this is because you simply assume that the return type is just some simple type, like Int, but you don't allow for type applications.

Here are some comments on the code in MethodCall.fr, please take them as constructive criticism.

Line 114

This is probably where the Runnable example breaks down.

Line 107

I'd advise against using the symInfo function here. Since it is in StG, it introduces some form of running the state monad behind the scenes in a pure function. But apart from that, the information you want here is already available: in lines 68 to 76 we have everything: the types of the arguments (taus), the base return type of the method (brty - that is if the signature was ... -> ST s (a|Maybe b) you get b here), the java type arguments applicable to this call (targs), the java code for the argument expressions complete with Maybe marshalling, and the java types of the result (jrty)

Also later, when it comes to the methods, why not take the types directly from the symbols?

Line 121

Why not

JNewClass jrty args (evalStG g x)

We do support constructors with arguments, do we not?

Line 350-357

This looks to me like a re-invention of frege.compiler.gen.java.Common.lazyJX

Line 315-338 wrapIRMethod

In methretjt you reinvent parts of what is already there in baserty further above (though it is local to nativeCall it should be easy to lift it to the top-level), but it's not complete. What we need here is, in principle, some counterpart to the things done in wrapCode.

I also suggest to use ST.performUnsafe to run ST actions. Reason is that any ST action can be treated like an IO action, but not the other way around. Hence, the code returned by wrapIRMethod should look like:

 final Return1 val = PreludeBase.TST.<Return1>performUnsafe(   --- only for IO/ST
    IR.mem$method.apply(lazy arg1).call().apply(lazy arg2).call() ....    --- OR
    IR.method(lazy arg1, lazy arg2,...).call()
 ).call();                                                                              --- only for IO/ST
// if Return1 is Either(a|Return2) where a is an exception (or a nested Either with all exceptions)
// throw appropriate exception if it is a Left value
// if Return2 is Maybe Return3, map Just x to x, Nothing to null

[matil019]

No worries :) I gladly take your advices.

[matil019]

@Ingo60 Thank you for your comments. Exactly the kind of comments I needed since I have had difficulty finding existing functions which do what I want. I'll open a WIP PR later so that we can follow code more efficiently.

[Ingo60]

I had that additional idea the other day: it would be generally very useful to have a functionality to "export" a frege function as a method (instance or static) conforming to a given signature.
Would be useful for instance to let the module implement some interface.
And this could then be used by the new - class functionality, too.

[Ingo60]

another hint: there is a JOverride method attribute, that we should set for the generated methods. This will cause the javac to check if we got our method signatures right.