Ability for ble_gatts_notify_custom() to wait for new packet buffers

[bryghtlabs-richard]

I suggest a new feature where a caller can wait for new packet buffers and specify a timeout limiting how long to wait for a packet buffer or queue entry to become available. I have an idea about how to implement this, but wanted to propose it here first.

The problem I face is caused by two threads calling ble_gatts_notify_custom() on FreeRTOS. One low-priority thread generates packets at up to 250Hz(many will fail to enqueue to the stack, and this is ok). The high-priority thread is generating packets at 1Hz, but when it calls ble_gatts_notify_custom(), BLE_HS_ENOMEM almost always occurs because the other thread has kept the queue full, so the 1Hz packets are usually dropped too.

If I'm understanding the problem right, adding the ability to wait for new packet buffers or queue slots would allow my 1Hz high-priority thread to block a short while until a connection event frees some room, and my 250Hz low-priority thread would continue to behave as it does now, usually keeping the queue full, but never blocking if the queue is already full.

Another use-case would be a serial port to BLE bridge, where the caller of ble_gatts_notify_custom() may want to block to allow the serial port hardware flow control to throttle the other end of the serial link according to BLE buffer availability.

[BOZHENG001]

I am also using NimBLE for BLE development but on RIOT OS, and yes, I agree with this post, since I am facing the same issue now.