Slowing down sysex for older midi gear.

[DavidOpgh]

My Rocktron All Access midi foot controller can't handle sysex uploads at full speed. It throws a Buffer Overrun error.

I read the previous discussion concerning splitting the sysex into small data blocks and the issues involved
https://ctrlr.org/forums/topic/splitting-sysex-for-older-synths/

I want to approach the issue from another angle.
Is it possible to configure Ctrlr to insert a delay in between each byte of the sysex?

I successfully configured MIDI-OX for a 1byte output buffer and inserted a delay of 2ms in between. The All Access was able to handle the upload without an error.

If Ctrlr could do the same it might solve the issue for older gear without the problems of trying to break the sysex into small blocks.

[damiensellier]

It is possible to do it.
you need a timer with a 2ms interval and split the message at a byte level and send small packets, but it's a workaround.
Usually on old gears, buffers overload when the packet size exceeds the buffer size, not because of the speed of the transfer. It's possible your unit has a cache failure.
Does it always ever fail or is it intermittent?

[damiensellier]

Unfortunately it doesn't work :(
I don't think it's possible to split midi messages to a byte-level to drip-feed the external gear.
I checked everything in JUCE and it comes from the fact sendMidi and sendMessageNow use "MidiMessage".
MIDI has some very precise norms and the transfer speed is one of them. So it's very strange your pedal board is not 100% midi compliant.
From what I understood "MidiMessage" relies on juce_MidiDataConcatenator.h which compose a MIDI valid only message with F0... F7 before sending it.
So every chunk of data sent via MidiMessage will get automatically the header and EOX everytime. This is why when I send my 505bytes message in 505 packets, We have 505 x F0 F7. and invalid messages.

I can't really see a solution for that for the moment, it's not from Ctrlr but directly from the JUCE Framework.
@DavidOpgh you should post a question on the JUCE forum to see if there's other solutions for slowing down the midi transfer speed or drip-fed the midi output at a byte level with a timer without the F0..F7 for each packet. And we could adapt it to Ctrlr probably.

[damiensellier]

Although one thing someone might want to try: Can Ctrlr create/play a midi sequence? If the sysex can be created as a midi sequence it might be able to provide the timing needed.

Yes Ctrlr can send midi sequences. It's possible with a timer and there are 2 promissing function in JUCE for that, I never heard before called sendBlockOfMessagesNow() and sendBlockOfMessages() which have time related variable in them.
I don't know how to use it but it could be a solution if ever you could share exactly about your "sequence" strategy and what your pedal board is accepting.
I am reading the ALL ACCESS manual and you are probably refering to page 66 "SYSX" (to perform a bulk load/dump)
What you want, in the end, is a panel editor to assign the different midi parameters to your pedal board isn't it?

What would be really helpful is to open your DAW create a midi file, allow SYSEX to be recorded from your pedal play+REC and send us the midi file to check exactly how the sysex dump is performed so that we can replicate the same speed and behaviour while sending a dump TO the All Access. Both in SYSX mode and also in SEQUENCE mode.

Can you please also send us a full dump file of your All Access here as a .syx file?

all_access.pdf

[DavidOpgh]

@damiensellier
Thanks for checking into the issue.

I agree it doesn't look like it's possible to split midi messages. I was hoping there was someone available at a lower level to introduce a delay between transmitted bytes.

The pedal board is 100% midi compliant. It just can't handle uploads at full throughput. This behavior is not unusual for devices made more than 30 years ago.

The sequence strategy is to figure what timing info is being stored when the All Access dumps the sysex when captured by a DAW. Then duplicate that timing info in Ctrlr if it can send sequence data.
Are you aware if anyone posted a panel that sends sequence data?
I agree I need to try to save the sysex dump as a sequence with a DAW and examine the .mid file.

What I want is an editor / librarian for the All Access. There has never been one released.

There two complete sysex files included in this post.
#457

[dnaldoog]

@dnaldoog I gave it a quick try. It appears the data is not being sent. The only thing being sent are two bytes [F0 F7] every time send button is pressed.

Yes, MIDI needs a status byte ( > 0x7f ) then data byte(s) (< 0x80 ), so it would never work, but in the Ctrlr MIDI monitor I could see each byte being sent, but only F0 F7 was received remotely. I just uploaded the panel before retiring for the night without full testing.

[dnaldoog]

Although one thing someone might want to try: Can Ctrlr create/play a midi sequence? If the sysex can be created as a midi sequence it might be able to provide the timing needed.

Yes Ctrlr can send midi sequences. It's possible with a timer and there are 2 promissing function in JUCE for that, I never heard before called sendBlockOfMessagesNow() and sendBlockOfMessages() which have time related variable in them. I don't know how to use it but it could be a solution if ever you could share exactly about your "sequence" strategy and what your pedal board is accepting. ...

They don't seem to be available on lua.

[damiensellier]

Alright I understand exactly what you are looking for ☺️ the midi file would be the best starting point and I'm sure it's going to be the way to our solution. Waiting for your file mate !

[DavidOpgh]

I did a quick test with a free midi program called MidiEditor. It didn't work. I immediately got a buffer overrun error when I tried to resend the sysex back to the All Access. I didn't save the data to a midi file but quick look at it in the timeline didn't show much. Just 309 byte blocks of data sent about 2.3 seconds apart. I was hoping for a little more granularity in the transmission. I had the tick per quarter note set at 1000, the BPM was 120. I'll have to try again with another DAW. Although I'm not hopeful.

[damiensellier]

Sure no worries, keep us updated when you have the files. And try the 2 transmission modes according to the manual. "Sysex" and "midi sequence"

[DavidOpgh]

I did try the two transmission modes in the manual. Sysex didn't work but midi sequence did.

[dnaldoog]

Or maybe MidiMessageSequence but it's not available in lua?

• https://docs.juce.com/master/classMidiMessageSequence.html
• https://github.com/RomanKubiak/ctrlr/search?q=MidiMessageSequence

[DavidOpgh]

Yes, I saw that as well. But from my brief test earlier tonight I'm not sure if the sequence route is going to work. If all the sequence is doing is waiting a specific time to send an entire 309 byte block of data I already know that doesn't work. The All Access won't accept that many bytes at full bandwidth. I'm suspecting the reason the DataDisk works is because it's also is a 30+ year old device that can't transmit a sysex at full bandwidth.

[damiensellier]

One last thing David, I know I will sound like those Internet provider call-center dummies, but did you try another pair of midi cables, because sometime a bad cable or bad contacts can corrupt midi datas and avoid sysex dumps at normal speed. Cleaning/checking the midi port contacts on the pedal could also help.

I thought about that, and If you have success with only 1 byte every 2ms it means 500bytes/sec which is not normal for a midi standard of 31,5Kb/s anyway. Also if you can't send the message at least half the speed of what you received, there's something wrong definitely. here we are talking about x60 slower. It's better to check if the pedal is 100% working before diving in a rabbithole of speed control workarounds... :)

Also check if you have the latest firmware revision in your pedal.
I asked on the JUCE forum how to slow down transfers, let's see if someone there knows. I'll let you know.

[DavidOpgh]

As someone who worked in technical support I can appreciate the question.
The All Access midi pedal is working as designed. It has the latest firmware (but it was release 30 years ago)
The manual of the All Access states that sysex uploads are processed at 66 bytes/sec or 1 byte every 15ms.
So I'll disagree there is something wrong. The All Access just writes to memory very slowly.
Using MIDI-OX I'm able to create various output buffers and timing configurations to get consistent successful upload.
In fact by sending 1 byte with a 2ms delay I'm able to significantly increase the transfer rate from 66 to 400 bytes/sec.
If Ctrlr could create a similar output buffer like MIDI-OX this wouldn't be an issue.

Thanks for asking the question in the JUCE forum. Hopefully someone will be able to help.
If not, I'll create a panel to use Ctrlr to manipulate the sysex file and use MIDI-OX to do the transfers.

[damiensellier]

lol I worked at technical support as well and we were asked to follow the troubleshooting diagram and check everything, the power being ON as the 1st.
i get it, it's made to be slow like a snail. 90s midi gears were usually faster from my own experience.
Yes, worst case scenario, having the sysex file prepared in the panel and send it with something else should do the trick.

[DavidOpgh]

@damiensellier and @dnaldoog
Thanks for all your help.

Do you know if Ctrlr has an existing panel to play *.mid midi files i.e. sequences?

After doing more research into *.mid midi files it looks like a sequence might have the structure needed to accomplish my goal.
The process is described here in Section 2.3
http://www.music.mcgill.ca/~ich/classes/mumt306/StandardMIDIfileformat.html

It appears to show how to transmit smaller blocks of sysex messages at a timed rate. I already know that I can send 16 bytes blocks with a 50ms delay. I'm going to try to manually create a *.mid file with 16 byte data blocks of a partial sysex (probably the first patch of 309 bytes). I should be able to send it with Ctrlr or another midi player to test the upload. A partial sysex upload with hang but as long as it doesn't generate a buffer overrun error it will prove the concept.

[damiensellier]

Can you please record a dump in your DAW as a midi file?
Record (while playing) your midi track from the rocktron output to the DAW with SYSEX filter disabled, then start the dump as midi sequence in the pedal board.
That would be great to see the relation between data and timing and see how rocktron handles it.
Thanks

[DavidOpgh]

Thanks @damiensellier for testing my .mid file.

It's interesting to see how Cubase tried to reformat the midi track data.
On the plus side Cubase does correctly recognizes the byte as x78 = 60ms but it mangles the rest of the data.

Normal sysex message packets inside of a track have an F0 header byte and an F7 ending byte.
But according to the midi spec (Section 2.3)
http://www.music.mcgill.ca/~ich/classes/mumt306/StandardMIDIfileformat.html
you can also have a multi-packet system exclusive message i.e. A larger sysex can be broken down into smaller packets with a delay in between.

In a multi-packet system exclusive message the F0 byte is used as the start of the first packet but the F7 byte is now used to indicate the start of next packet.
The software is supposed to be smart enough to recognize if it sees a F7 byte without a corresponding F0 byte it's part of a multi-packet system exclusive message.

But Cubase doesn't seem to recognize the F7 byte as part of a multi-packet system exclusive message.
Cubase thinks the sysex data is wrong and is deliberately inserting an F0 byte where there isn't one to force the data to be inside a F0-F7 sysex packet. Which is why Cubase shows multiple manufacturers for the packets. If it understood multi-packet system exclusive message it would know only the first packet contains the manufacturer ID.

And in the process of this conversion Cubase is also dropping the 16th byte of each packet.

Unless my mid file isn't crafted correctly according to the midi spec it doesn't seem any DAW recognizes the concept of multi-packet system exclusive track data. And Ctrlr probably doesn't either. I was going to use this functionality to get Ctrlr to break the sysex into smaller pieces so I could effectively slow down the transmission rate.

[damiensellier]

I am not aware about those data packets requirements but I guess everytime the controller receives a F7 byte it reads it like the end of transmission and get ready for the next one like a new message.
Your problem is really unique.

[DavidOpgh]

Yes, My problem is unique. And other than Midi-OX there isn't anything else out there that allows you to slow down the transmission rate enough for a reliable solution. I was hoping the multi-packet system exclusive method built into the midi spec was a possible solution. But it doesn't appear it's widely implemented. Since I have a work-around I think I'm going to abandon trying another solution. Instead of using Ctrlr as a single solution, I'll edit with Ctrlr and transmit the resulting sysex file with Midi-OX.

[damiensellier]

Just send the midi sequence dump here. We will see how we can replicate it, even slowing it down to accomodate the small buffer issue.

[DavidOpgh]

Here is a sequence capture file of the Rocktron played into the Alesis Datadisk and then replayed from the Datadisk to a DAW. I have verified the Datadisk sequence can be successfully played back to the Rocktron without the buffer overflow issue.

The track data is pretty simple. Each patch is defined as a sysex with a F0 header and F7 end byte. Each sysex is sent approx every 2.3 seconds. The first 120 patches are made up of 309 bytes.

But to be clear, I don't need this duplicated because I already know trying to send this sequence from a PC to the Rocktron doesn't work. The PC throughput is too fast for the Rocktron. The Rocktron generates a buffer overflow error. I'm concluding the Datdisk works because the throughput on the Datadisk must be slow enough for the Rocktron to keep up.

I'm trying to find a way to split the larger sysex into small packets with a delay in between. From using MIDI-OX I know that I can successfully the send the sysex from my PC to the Rocktron if I break the sysex into 16 byte packets with a delay of 50ms in between.

The midi spec definition of a multi-packet system exclusive track event sounded like it could give me the needed functionality. If Ctrlr implements this portion of the midi spec I would have my solution. But since I haven't found a DAW that implements this portion of the spec I doubt Ctrlr does either.

But this could be verified if someone knows how to play a sequence file using Ctrlr and then capturing the output.

Thanks for your help.

Alesis DataDisk sequence capture of Rocktron All Access (2).zip

[damiensellier]

Hi David, I opened the .mid file and it's pretty interesting.
I checked the specs in http://www.music.mcgill.ca/~ich/classes/mumt306/StandardMIDIfileformat.html
There's probably something a little bit wrong in the header done by the DAW because if I import it in Cubase or Studio One It just creates an empty "new instrument" track.
I checked the header and it says :

MThd���������èMTrk���$�ÿX������ÿQ��¡ �ÿ��Tempo Track¨�fÿ/�MTrk��«¸�ÿ��New Instrument

All the other datas seems to be perfectly right with the sysex packet with the following related time.

What DAW did you use? I would love to see the data chunks in the timeline but importing the .mid doesn't work. Can you please post the project file .cpr for cubase or else.. as you recorded it?

I'm quite confident we will be able to dripfeed those chunks over the time from Ctrlr without overloading the buffer.

[DavidOpgh]

The DAW I used is more of a midi recorder/playback program called MidiEditor
http://www.midieditor.org/

Sorry, after reviewing it I sent the wrong mid file. I've made a lot of versions and I didn't test that one before I sent it.
I tried loading it into MidiEditor and got the same results you described.

Try this new one. I loaded it into MidiEditor and got the correct behavior. It is a dump of the All Access after it's been initialized, so all the patch data is the same.

There is a single track called Tempo Track and it contains sysex messages for the patches followed by sysex song patches, then sysex configuration patches.

All Access dump 05-02-2022.zip

[damiensellier]

Hi David, I could not import this midi file as well in Cubase or Studio One.
I installed midieditor on my Windows VM and I could not get the sysex to output on midimonitor.
The first time I opened the .mid there were sysex events available but not on the second time I opened it, not sure Midieditor is really reliable.

I don't know what's going on really. Do you have another more serious DAW like cubase etc?
Even a demo version would do the job since limitations are on audio processing usually and not midi.
Recording the sequence from there would be better I guess. Midieditor is really 1984.

Let me know if you need some help with cubase ;) have a nice weekend
Damien

[DavidOpgh]

That's interesting. MidiEditor has worked well for me. I've been using it because it's simple and it's free.
I had an old old version of Cubase LE (2009 vintage) that I tried to install but I was having issues with it.
I know I have Reaper installed on another PC. I'll give that a try.