How to extend the midi API

[steveschow]

I would like to extend the MIDI api that is available in LuaProtoPlug. I could do this either in C++ or LUA, as makes sense, but I would appreciate some pointers as to where in the code I could potentially add a couple more midi related features. Mainly I want to create the ability to schedule midi events into the future, but a few more things too. I could just use some pointers about the existing code structure to get my head wrapped around where in the code I could extend these capabilities.

Thanks

[interstar]

Presumably you saw my arpeggiator example. https://www.youtube.com/watch?v=MHo1FXyRvrA

Which obviously does manually schedule events for the future. You could wrap up some code like that into a Lua library.

The only thing is, I think you'd still need to make an explicit call to whatever your library does from within the processBlock() call, which AFAICT is the main and only callback into your plugin from the DAW.

(Unless you actually wanted to get involved implementing your own threading library in C. Which is obviously a lot of work.
If I understand correctly Lua doesn't have its own threading library, or rather its own one is "cooperative" which means you can't just, say, fire off a new thread to play noteOff in the future and forget it in your main thread.)

But yeah, obviously some of what I did in custom form in that arpeggiator example could be packaged into a more general, reusable library.

The implementation of MIDI behind the scenes in Protoplug is, I believe, a thin Lua wrapper around JUCE. So if you want to know more about what's going on at that C++ level, look at JUCE's own documentation and tutorials etc.

https://juce.com/learn

I've found the JUCE forums and community pretty good when I've had questions etc.

[steveschow]

Thanks for your response. Haven't looked at your videos yet. forking a thread to schedule stuff in the future is not how I would handle it. As you said, something has to check a sequence queue once per process block to see if there are any events due that need to be output during that process block. It involves maintaining a long term queue and checking once per process block to do it. As you said, if I do it with LUA, then I'd have to require a "checker" function to be called once per process block in the users's LUA script in order to enable that behavior. Not the end of the world, but possible.

It would be better to to have that happen inside C++ code of the plugin. User sets a flag in LUA, which instructs the plugin to Call the "checker" once per process block up in the C++. No extra threads are needed or warranted.

I am looking for info about where in the C++ I could tap into adding that kind of functionality. It means having also some accessor methods in LUA that can be used to save midi events into C++ queue.

[interstar]

So what are the advantages of doing this in C++ as opposed to Lua?

AFAICT most of the C++ is actually JUCE. So then the question is, do you want to get into making customized changes to JUCE code? And the issue is then whether Pierre Cusa (author of Protoplug), would want to base Protoplug on your custom fork of JUCE.

Or would you hope to get your changes into the main branch of JUCE?

Presumably speed is the main reason for doing stuff in C++ rather than Lua. But with MIDI processing, is speed such a bottleneck?

[steveschow]

I already explained why above.

I don't actually care whether my changes make it into the mainline of ProtoPlug, Pierre used an MIT license and made it clear we are free to "Hack away" for our own purposes.

[interstar]

OK

But either way, I don't think there's a time based event queue for midi in either Protoplug or JUCE so you'll still have to implement it yourself.

Behind the scenes I think we're using https://docs.juce.com/master/classMidiBuffer.html which is kind of time-ordered but I don't think there's any reason to assume that future events persist within a particular MidiBuffer between between one call of processBlock and another. I mean when the DAW calls the JUCE processBlock it's giving it a brand new MidiBuffer with whatever new events just happened, not an old one, preserving existing events.

So your users are going to have to take responsibility for both adding future events to a future events queue, AND explicitly taking events that are to be played now and putting them into the actual current output MidiBuffer.

I don't see there's a way around that, or of hiding that responsibility in the C++ infrastructure, without making a big change to the JUCE framework itself.

Look at this JUCE example that processes midi in the processBlock : https://docs.juce.com/master/tutorial_code_basic_plugin.html It's the same principles as Protoplug

[steveschow]

Yes for that particular thing I am sure I will have to do some similar things. as I said already above, the right approach is to load midi events into a queue, then once per process block you check the queue to see what has to be actually transferred to the REAL process block buffer (of JUCE). You are making a mountain out of a mole hill here and I don't really want to discuss the minutia of what I will do, there are other tasks I want to cover also...

The question is, where in the code is the best place to look, for extending the api available in LUA?

I already understand that there are some standard includes of LUA functions..., which could be expanded, using LUA, but as in the example above, there still could be some need to go into C++ JUCE level for adding some handlers that will happen under the surface, so that is my question....and/or how to bind Lua data to data in the C++ layer...how to add perhaps additional callbacks (Lua functions that will be called if present, the LUA process block function is one example of such a thing.

[steveschow]

To give a little better idea where I am headed with some of this, I have some years of experience using LogicPro's Scripter plugin and am intimately familiar with the API that Apple chose to use for that plugin. It does a few things a little more smartly then all the other midi scripters out there, including ProtoPlug... I can elaborate on that some more if there is interest, but mainly I would just like to extend ProtoPlug's midi API so that it is closer to the same approach employed by LogicPro Scripter....which is done in such a way that many typical user scripts, include time-based scripts...require very little script code...and very simple script code, to accomplish.

[interstar]

OK. I don't know Logic or scripter. So I can't make much comparison.

But my suspicion is that it's not very like the way Lua / Protoplug / JUCE work, so may be leading your intuitions astray.

(And obviously I'm not trying to make a mountain out of a molehill. If this isn't useful to you, say so, and I'll just stop.)

The point I was really getting at is that I suspect the place you say you are looking for ("where in the code is the best place to look, for extending the api available in LUA?") doesn't exist in the way you imagine it does.

It doesn't exist because Protoplug is a thin wrapper that delegates how it handles MIDI to JUCE. And JUCE's own handling of processBlock doesn't do what you want either.

In other words. You can implement your entire "FutureEventQueue" in Lua and store it as an attribute of the Protoplug plugin object, all of which are in Lua controlled memory space. And talk to it directly through Lua calls. This, I think is easy, but doesn't involve going into C++

OR

you can implement it in C++, bypassing JUCE's own processBlock() and writing your own alternative processBlock2() that does the same as processBlock() but ALSO manages its own FutureEventQueue in a C++ controlled memory space. And then you'd write your own Lua wrapper to access that processBlock2() function. And you are then rewriting that whole chunk of the stack.

But what you can't do is have the current existing Protoplug Lua API that talks to JUCE, lightly extended to talk to something extra you've written in C++ which nevertheless still works with the existing stack of JUCE / Protoplug calls.

No, if you are going to implement FutureEventQueue and change things down at the C++ level, you are going to have to make major changes down there and re-implement the whole stack through which Lua Protoplug talks to it. It's not a small intervention.

If you want to know how Lua talks to C then have a look at https://www.lua.org/pil/26.html and maybe http://gamedevgeek.com/tutorials/calling-c-functions-from-lua/

But where does this happen specifically in Protoplug?

Well, @pac-dev could tell you more and better, but, my hunch is that it's everywhere. A lot of the Protoplug source code is just providing these wrappers around specific JUCE library functions.

For example, my guess is that https://github.com/pac-dev/protoplug/blob/master/ProtoplugFiles/include/core/plugin.lua is the file that ultimately defines the Lua version of processBlock() such that it calls the JUCE version of processBlock().

But this is largely a boilerplate template for wrapping a C call in Lua. I don't think you'd try to add any extra functionality there. And if you are talking about a different JUCE function, the wrapper will be somewhere else.

Now maybe what you are really interested in is "I would just like to extend ProtoPlug's midi API so that it is closer to the same approach employed by LogicPro Scripter.."

In which case I'd suggest something else. Possibly what you really want to do is create a new Lua object / class that presents its users with an API closer to the one you find intuitive. And then make that new object wrap the current Protoplug plugin object. Or perhaps inherit from it. ( https://www.lua.org/pil/16.2.html ) Either way it would basically translate from your preferred way of doing things to the way Protoplug currently does them (which is the way JUCE does them)

[steveschow]

As I said, I am asking for pointers about where in the ProtoPlug code to look for places that I can extend what it is capable of...it is perfectly well capable of doing what I am talking about, and its not in conflict with JUCE either. I just need to understand a little better how Lua is generally embedded into this Juce-based plugin paradigm.

You make a lot of assertions above about what you THINK I know or what you THINK I want, but you don't understand what I am trying to say so I find that kind of hyperbole unhelpful.

Lua is embedded into ProtoPlug as JUCE plugin..and I need to understand where in the code to look for that, I can find out more about how Lua binds with functions and data elements in the hosting C++ code. Whether its JUCE or not is irrelevant.

[interstar]

OK. Then maybe I'm not understanding your questions. Sorry.

Perhaps @pac-dev can help

[interstar]

I've had another think about this.

If you just want to know "where is the C++ code of Protoplug that calls into the Lua code?" then I believe that, for example, for processBlock it's in https://github.com/pac-dev/protoplug/blob/master/Source/PluginProcessor.cpp on line 88

I suspect, though maybe @pac-dev can confirm this, that the comment at the top of this file, which says it was auto-generated is misleading. It may have been auto-generated originally, but it now contains the actual authored code that does this call into Lua from C++ .

(Originally, because of that comment, I assumed that this file was just some autogenerated intermediary and the original authored code was elsewhere. But looking through the source I can't find anywhere else, so I'm starting to think this is it. But obviously Pierre may tell us something different.)

[pac-dev]

Well I like where this is going! If you want to add features to the internals of protoplug, I'd say Phil was on the right track, in the sense that the internals are actually mostly Lua.

First, let's imaging that you want to redefine the Lua API exposed to scripts. Here's a trick: make a working protoplug script, for example:

-- plays a 500Hz tone when the host is playing.
require "include/protoplug"
local phase = 0
function plugin.processBlock(samples, smax)
    local freq = 500
    local delta = freq*2*math.pi/plugin.getSampleRate()
    if not plugin.getCurrentPosition().isPlaying then return end
    for i=0,smax do
        local s = math.sin(phase)*0.3
        samples[0][i] = s -- left
        samples[1][i] = s -- right
        phase = phase + delta
    end
end

You can play that script directly, but you can also turn it into a base library. Save that script inside ProtoplugFiles/include/, but make it call some non-existing global functions, to be defined later by scripts:

-- source of new file: ProtoplugFiles/include/sineplug.lua
-- plays a tone defined by the including script.
require "include/protoplug"
local phase = 0
function plugin.processBlock(samples, smax)
    local freq = getFreq() -- <-- script-defined function
    local delta = freq*2*math.pi/plugin.getSampleRate()
    if not plugin.getCurrentPosition().isPlaying then return end
    for i=0,smax do
        local s = math.sin(phase)*0.5
        samples[0][i] = s -- left
        samples[1][i] = s -- right
        phase = phase + delta
    end
end

From now on, you can make scripts that don't require "include/protoplug", but instead they require your custom library file, for example:

-- plays an 800Hz tone when the host is playing.
require "include/sineplug"
function getFreq()
    return 800
end

This is basically how the protoplug Lua library is built. Your library file can also define functions (eg. "scheduleMidiEvent") that can be called by any including script. You could also change the API so that it's not block-based but sample-based, or some other abstraction that makes more sense for MIDI. Also note that scripts can still use the rest of the normal protoplug API.

Now what if you want to only add a scheduleMidiEvent function, without taking over plugin.processBlock? This can be solved with a similar approach, but modifying the internals a bit more. Your modified library file can do something like this:

require "include/protoplug"
local scheduledEvents = {}

-- function that scripts can call
function scheduleMidiEvent(...)
    -- add event to table here
end

script.addHandler("init", function ()
    local original_processBlock = plugin_processBlock
    function plugin_processBlock(nSamples, samples, midiBuf, playHead, _sampleRate)
        original_processBlock(nSamples, samples, midiBuf, playHead, _sampleRate)
        midiBuf = ffi.typeof("pMidiBuffer")(midiBuf)
        -- add scheduled events to midiBuf here
    end
end)

The above script redefines stuff from plugin.lua. In particular, plugin.lua normally looks for a script-defined method called plugin.processBlock, and wraps it into a global function called plugin_processBlock. This global function is directly called from the C++ plugin. As you can see, you can redefine the global function so that it performs additional work before returning to the host.

Now, if after all this, you still want to add Lua functions from C++, I can go through how that's done, but I believe the pure Lua approach is simpler and can give an elegant API (well, at least as elegant as the original one!)

[steveschow]

Thanks for checking in! Yes I am definitely interested in getting under the covers into C++. Several reasons. For one I can make use of JUCE objects directly designed for maintaining a queue of midi events. For another thing I would like to make something such that the user will not have to make sure to add anything to the Lua process block function. I want them to be able to schedule an event in the future and never have to do ANYTHING else in their Lua code to make sure it happens.

I agree, a lot of simpler methods could easily be handled in Lua include, but in this case where I want ProtoPlug to automatically do some things under the covers, I need to tap into the C++

It seems like wherever in the C++ that the Lua process block function is being called is where I would want to go into..but I do not know how that Lua methods are fundamentally bound to C++, etc.. If the Lua process block function itself is actually in the include dir, then I could probably just modify that Lua code to accomplish it also, though I could also make good use of some JUCE data structures.

[pac-dev]

None of the methods I described above require scripts to add anything to their process block functions, or add anything at all for that matter. That said, sure, I will go over adding Lua functions in C++ a bit later.

[steveschow]

Thanks I look forward to hearing about that!

Let me try to explain better. I want user script to be able to say simply something like:

scheduleEventAfterMilliseconds(event)

for example. Then the user should not have to add a process block callback or do anything at all in their script to make sure that event will be sent on schedule. That needs to happen automatically under the covers.

Perhaps the way you have added the process block Lua callback means I can do this in Lua, I am not sure at this moment. Perhaps I would need to insert hooks in the C++. I look forward to hearing more. I would also really like to be able to make more use of some JUCE data structures. Perhaps that is already all exposed to the Lua also though, I do not know. That's why I'm asking you!!

[steveschow]

This looks like what I need to add hooks that will happen...and addHandler... I will need to spend some time digesting this and experimenting...but anyway,..

script.addHandler("init", function ()
    local original_processBlock = plugin_processBlock
    function plugin_processBlock(nSamples, samples, midiBuf, playHead, _sampleRate)
        original_processBlock(nSamples, samples, midiBuf, playHead, _sampleRate)
        midiBuf = ffi.typeof("pMidiBuffer")(midiBuf)
        -- add scheduled events to midiBuf here
    end
end)

The above script redefines stuff from plugin.lua. In particular, plugin.lua normally looks for a script-defined method called plugin.processBlock, and wraps it into a global function called plugin_processBlock. This global function is directly called from the C++ plugin. As you can see, you can redefine the global function so that it performs additional work before returning to the host.

I will look into the above a little more. The one thing I want to make sure is that users will not have to actually write a processBlock Lua callback if they don't want to. I want them to be able to just schedule the event, and then ProtoPlug will make it happen without having them having to do anything in the process block callback at all, and possibly not even provide an empty processblock callback either.

another method I would like to add, this is easier, is an event callback that will be called once per midi event. This is how LogicPro does it. So instead of providing a processBlock Lua callback that loops through the midi buffer, the user will simply provide a function that gets called once for every midi event. (This means, again, I need something under the covers that will be executed in the processBlock...that basically will look through the midi buffer and recall the Lua callback provided by the user for each midi event.

It looks to me like I can modify your processBlock implementation with this additional functionality, but I need to investigate this further to make sure I'm understanding right.. I'm still learning Lua, so it will take me a minute

[interstar]

Cool. @pac-dev. I hadn't quite grokked that you could add handlers to the script like that. But, yeah, that looks convenient for overriding processBlock with your own.

[pac-dev]

Here's a rough draft of how I would implement what you're describing by creating a new library file:

-- new library file: ProtoplugFiles/include/midi-lib.lua
local scheduledEvents = {}

-- add our method to the "midi" global for clarity
function midi.scheduleEvent(...)
    -- add event to table here
end

-- internal use only
local playScheduledEvents = function(midiBuf)
    -- play events by adding them to the block's midiBuf at the right time
    -- use plugin.getCurrentPosition: https://www.osar.fr/protoplug/api/modules/plugin.html#getCurrentPosition
end

-- internal use only: call user-defined "onEvent" for existing MIDI events
local processOnEvent = function(midiBuf)
    if midi.onEvent then
        for ev in midiBuf:eachEvent() do
            midi.onEvent(ev)
        end
    end
end

-- if the script never defines processBlock, make a default one with our features
local defaultProcessBlock = function(samples, smax, midiBuf)
    playScheduledEvents(midiBuf)
    processOnEvent(midiBuf)
end

plugin.processBlock = defaultProcessBlock

-- if the script defines processBlock, make it also run the MIDI features
plugin.addHandler('prepareToPlay', function()
    if plugin.processBlock ~= defaultProcessBlock then
        local original_processBlock = plugin.processBlock
        plugin.processBlock = function(samples, smax, midiBuf)
            original_processBlock(samples, smax, midiBuf)
            playScheduledEvents(midiBuf)
            processOnEvent(midiBuf)
        end
    end
end)

Then you can include this file from ProtoplugFiles/include/protoplug.lua, so scripts can use the new functionality without any changes, and without any processBlock if they don't need it. The parts I've left unimplemented will obviously be the meat of it, but I think it will still be a lot easier than writing it in C++.

[interstar]

OK. Maybe that example answers the question that was still puzzling me.

Namely, what's the scope of the scheduledEvents buffer? And how does a user defined processBlock declared in a user defined plugin file, get to see it, if it's owned and managed in this library?

But if I understand correctly, scheduledEvents lives in a namespace belonging to midi-lib.lua, and the method midi.scheduleEvent() by being defined within that namespace, therefore sees it. Even though that method is then attached to a different object called "midi" (which I guess is the actual midi library)

Or is it that in Lua a file called midi-lib.lua automatically becomes a namespace called midi?

But anyway, if I want to add something to that buffer from within my own processBlock, I'd just need to include "include/midi" into my script and then call

midi.scheduleEvent(e) 

inside my processBlock?

[steveschow]

That's very helpful info thanks! Will take me a bit of time to digest this, still getting into Lua generally. Ironically at this point in time I would actually find it easier to write C++ hehe. but the obvious advantage of this is very fast prototyping and modification and quickly extend the capabilities of ProtoPlug without having to actually build and distribute a different binary...

I see that somehow you have made many things from the JUCE api directly available in ProtoPlug Lua...in order to do Lua gui stuff, etc. Is that a limited set of JUCE or all of JUCE? I'd love to make use of some JUCE data structures if possible...

[steveschow]

Namely, what's the scope of the scheduledEvents buffer? And how does a user defined processBlock declared in a user defined plugin file, get to see it, if it's owned and managed in this library?

This can be managed by having new API functions with accessor methods to set/get the value of such a buffer indirectly. The user script doesn't necessarily need to actually be able to access that buffer directly.

The idea is to make it brain dead simple to write many simple scripts. I want musicians to be able to script things with just bare bones Lua knowledge. That is handled with simple API that hides all the details inside the library.

[steveschow]

There is one small thing that I guess will have to be implemented as an API extension, in JUCE... which is that I need a way to report latency to the host. This is a simple one-liner in JUCE.

So the question is what the best way to go about adding something like this to LuaProtoPlug? I suspect it could be handled either way...as a direct call from LUA, that then calls the JUCE function to report latency..

or...

An optional callback, that LUA can provide, which reports latency (this way LuaProtoPlug can determine the best times to query for that and send it back to JUCE and to the host)...

BlueCatAudio PlugNScript handles this with a callback... I suggest that would be probably be the better way also.

In any case, how would I go about adding something like this to LuaProtoPlug...I guess it will involve both some C/C++ as well as LUA includes..

While We're on this topic, I see that LPP has rudimentary JUCE line drawing API's exposed, but nothing like buttons, faders and knobs...and I am wondering what it would take to add that sort of functionality to LPP?

I have a fork of LPP, I originally created it a few years ago to build the AUMI version of LPP, but I am going to make any LUA includes changes...and these other kinds of additions over there. Looks like LPP hasn't been updated to JUCE6 yet, I actually can't get it to compile as of now, so that will take some effort also to sort through all that and get it all caught up to 2021. I will make some effort on that front, but while I'm at it I'd like to try to get latency reporting added.

[steveschow]

another thing I could really use is an "Idle" callback. LogicPro Scripter has a call back called "Idle" which is called outside of the main process block thread...and managed in some way at a lower priority...where certain tasks can be handled that aren't directly relevant to processing audio or midi.

I typically used the Idle callback in LogicPro Scripter, for flushing buffered console logging...so that logging messages to the console doesn't cause the process_block callback to have to wait.

So anyway, I don't know the best way to accomplish that with Lua ProtoPlug right now. Maybe that's a case for having a seperate low priority Lua thread forked? if that's even possible.. But an actual Idle callback would be useful for many things including updating ProtoPlug's GUI, among other things. Or maybe something like that is in there that I'm just not aware about?

[interstar]

One thing to bear in mind is that Protoplug is still a VST plugin. So it can only communicate with the DAW via the callbacks that are part of the VST specification. I'm guessing LogicPro Scripter is an integral part of Logic. So Apple can define whatever API they like for scripts to talk to it.

So unless VST supports some kind of regular low frequency callback into the plugin, you won't be able to get one from the DAW.

However, VST (at least on Windows) expects plugins to be multithreaded DLLs. So you can launch and run other threads within your VST. And if you want some kind of background "worker" to do something at regular intervals, you can probably do that by writing it in C++ .

Lua, though, doesn't have threads, I believe, so you couldn't do it there.

[steveschow]

lua is capable of threading, but I'd rather keep the Lua more simple...and if it means forking a thread in C++ in order to handle Idle tasks which should be handled outside of the process block, such as GUI updates, etc..then that is the best place to do it... or integrated into the JUCE infrastructure, JUCE already does certain tasks outside of the process block thread for exactly this reason...so that part of ProtoPlug could call an "idle" callback in Lua......

[steveschow]

so protoplug has gui.paint() as a callback that happens outside of processBlock.. but this can only seem to work if and when the GUI tab is selected in the ProtoPlug GUI.. rightly so, because when its closed you want to minimize GUI operations. But makes it unsuitable as a general idle callback. Would be good if there was another handler that gets called pretty much as often as gui.paint() but always gets called even when the GUI window is closed. or something along those lines. Maybe I can add this in the LUA includes I will take a look later.