tinySA scanraw timing - incremental trace display?

[Ho-Ro]

When scanning a wide frequency range with low RBW the response is very slow - regardless of the number of points!
Some time ago I tried to receive point by point by reading only 3 bytes at a time and found out that the tinySA (at least the tinySA Basic) answers in blocks of 20 points. So a wide scan with 20 points that takes e.g. 1 minute sends the complete response (b'{xXXxXX...xXX}ch> ') after this minute, while the same frequency range with 600 points sends the 1st answer (20 points) already after 2 seconds.
Dunno if it's possible to build the trace step by step like also on the tinySA screen.

[g4ixt]

That's something I was going to look into, or ask Erik about, whether the TinySA sends the whole scan once complete or as it runs.
Yes it's certainly possible to update the trace block by block. I would need to change the way the measurement thread emits the data as well as retrieving the data in blocks.

The data retrieval is based on your NanoVNA/TinySA toolset code so maybe that's something you could look at and I can then copy?

[Ho-Ro]

Yesterday I pushed my experimental version. It has e.g. command line arguments for the frequencies, points, rbw and band, as well as autostart and verbose.
I also like the trace presets, it starts always with one enabled normal trace and the others have useful default types, just click the checkbox.
This made my tests easier (together with --autostart).
Here is the changed measurement routine:

    def measurement(self, f_low, f_high):  # runs in a separate thread
        self.threadrunning = True
        while self.sweeping:
            self.clearBuffer()
            with serial.Serial(self.dev, baudrate=3000000) as serialPort:  # baudrate does nothing for USB cnx
                serialPort.timeout = self.timeout
                while serialPort.inWaiting():
                    serialPort.read_all()  # keep the serial buffer clean
                    time.sleep(0.1)
                logging.debug(f'serial timeout: {self.timeout}')
                logging.debug(f'points: {self.points}')
                scan_command = f'scanraw {int(f_low)} {int(f_high)} {int(self.points)}\r'.encode()
                serialPort.write(scan_command)
                # skip command echoes and leading {
                serialPort.read_until(scan_command + b'\n{')
                raw_data = []
                for n in range( self.points ):
                    if not self.sweeping:
                        self.threadrunning = False
                        self.clearBuffer()
                        return
                    b_data = (serialPort.read(3))
                    if len(b_data) < 3 or b_data[0] != ord(b'x'):
                        self.threadrunning = False
                        exit_handler()
                        raise OSError("TinySA not found")
                    c, data = struct.unpack('<' + 'cH', b_data)
                    raw_data.append( data )
                serialPort.read_until(b'}ch> ') # skip trailing } and prompt
            raw_data = np.array(raw_data, dtype=np.uint16)
            logging.debug(f'raw data: {raw_data}')
            # tinySA: 128  tinySA4: 174
            if self.tinySA4:
                SCALE = 174
            else:
                SCALE = 128
            dBm_power = (raw_data / 32) - SCALE  # scale 0..4095 -> -128..-0.03 dBm
            # store each sweep in an array with most recent at index 0
            self.sweepresults = np.roll(self.sweepresults, 1, axis=0)
            self.sweepresults[0] = dBm_power
            self.signals.result.emit(self.sweepresults)
        self.threadrunning = False

It will be easy to build a line of dBm_power step by step (e.g. starting from a copy of the previous scan) and emit this every 20 points with another signal for the 2D trace, while at the end of a scan sweepresults will be sent for the 3D.
The error handling is not perfect, also the stop in a running scan corrupts the following serial communication, but it is only the 1st step. I do not see the use case for small point numbers at all.
If you could confirm the 20 point block also for the Ultra (I will also doublecheck the FW code) you should set your point sizes as multiples of 20 to simplify the scanning. The tinySA size of 290 comes probably from the screen width of the tinySA (320 minus left margin -> 290).

[g4ixt]

Now I see why you are getting the (sys.argv) error. I am no expert but I don't like GUI programmes with command line arguments, it just feels wrong. I would rather have a 'preferences' menu where startup values could be set and I have thought about doing that: I would probably store then in a SQLite database like I used in my RF Power Meter code.

I also thought about using that same SQLite system to store frequency response/attenuation values for things like attenuators, couplers etc, again like the RF Power Meter. The system would be: measure with NanoVNA, save S2P file, import it and select 'in use' devices. The QtTinySA programme then accounts for the attenuator, coupler etc automatically. See my RF Power Meter code for an example (everything works without the hardware unless you press 'run')

[Ho-Ro]

I see the QSettings::value: Empty key passed error with your original main branch without any command line content. I use python 3.11.2 and PyQt 5.15.9 from Debian stable "bookworm".

[g4ixt]

I have already added a 'scan progress' bar to the GUI but it doesn't do anything yet. It could be driven from the data retrieval blocks.

[Ho-Ro]

Yes, this could also be driven by the incremental scanraw.
If you unselect textVisible the progress bar extends over the complete range:

[erikkaashoek]

Scanraw sends the data for each point as soon as measured, usb buffering groups them together in chunks of about 63 bytes

[Ho-Ro]

Erik, any chance to cancel a running scanraw with a serial command?

[Ho-Ro]

Scanraw sends the data for each point as soon as measured, usb buffering groups them together in chunks of about 63 bytes.

No, scanraw buffers itself, a possibility would be to switch to unbufferd, e.g. with an optional argv[3], e.g.
if (unbuffered || idx >= BUFFER_SIZE - 4) {:

VNA_SHELL_FUNCTION(cmd_scanraw)
{
  freq_t start, stop;
  uint32_t points = sweep_points;
  if (argc < 2 || argc > 3) {
    usage_printf("scanraw {start(Hz)} {stop(Hz)} [points]\r\n");
    return;
  }


  start = my_atoui(argv[0]);
  stop = my_atoui(argv[1]);
  if (start > stop) {
      shell_printf("frequency range is invalid\r\n");
      return;
  }
  if (argc == 3) {
    points = my_atoi(argv[2]);
  }


  if (setting.waterfall)
    disable_waterfall();            // display dma hangs when waterfall is enabled


  freq_t old_step = setting.frequency_step;
  float f_step = (stop-start)/ points;
  setting.frequency_step = (freq_t)f_step;


  operation_requested = false;
  dirty = true;
//  adc_stop_analog_watchdog();
  int oldpos = 0;
  ili9341_set_background(LCD_BG_COLOR);
  ili9341_fill(OFFSETX, CHART_BOTTOM+1, WIDTH, 1);
  ili9341_set_background(LCD_SWEEP_LINE_COLOR);
#define BUFFER_SIZE  64
  uint8_t buf[BUFFER_SIZE];
  int idx = 0;
  buf[idx++] = '{';
  for (uint32_t i = 0; i < points; i++) {
    int val = perform(false, i, start +(freq_t)(f_step * i), false) + float_TO_PURE_RSSI(config.ext_zero_level);
    if (operation_requested || SDU1.config->usbp->state != USB_ACTIVE) // break on operation in perform
      break;
    buf[idx++] = 'x';
    buf[idx++] = (uint8_t)(val & 0xFF);
    buf[idx++] = (uint8_t)((val>>8) & 0xFF);
    if (idx >= BUFFER_SIZE - 4) {
      streamWrite(shell_stream, buf, idx);
      idx = 0;
    }
    int pos = i * (WIDTH+1) / points;
    if (pos - oldpos > 8) {
      ili9341_fill(OFFSETX + oldpos, CHART_BOTTOM+1, pos - oldpos, 1);     // update sweep progress bar
      oldpos = pos;
    }
  }
  buf[idx++] = '}';
  streamWrite(shell_stream, buf, idx);
//  adc_start_analog_watchdog();
  ili9341_set_background(LCD_BG_COLOR);
  ili9341_fill(OFFSETX, CHART_BOTTOM+1, WIDTH, 1);
  setting.frequency_step = old_step;
  dirty = true;
  redraw_request = 0; // disable screen update in this mode
}

[erikkaashoek]

Sending a a single dot ('.') on a line should cancel it if I remember well

[Ho-Ro]

Nope :(
No reaction, echo '.' after completion and '.?'

[Ho-Ro]

Maybe @DiSlord knows a trick to interrupt the scan_raw?

[g4ixt]

I also meant to ask Erik if Scanraw adds scan points if needed by the sweep span, like I read that the TinySA itself does, or if I need to do something like that as well. From looking at how fast it scans with different settings I think it must add points but it would be nice to know.

On the 'number of scan points' I wondered if there were numbers that are impossible, due to synthesiser step size perhaps, or if the TinySA firmware just hops on to the nearest valid frequency. I like the idea of retaining a fairly small number of points because it's very fast. If there are no impossible numbers then it would be neat to change the combo box to an integer spinbox. What limits would you put on it?

[Ho-Ro]

Some comments to the new measurement() routine, your WIP looks fine so far:

serialPort.timeout = 0.2 # anything < 0.12 causes errors (when running in Spyder)
The 0.2 s timeout doesn't work for very slow scans, try e.g. 87.5 .. 108 MHz with RBW 3 kHz -> crash :(

The last read
null = (serialPort.read(3)) # discard the command prompt
blocks until timeout because after the closing three bytes b'}ch' only these two bytes b'> ' remain.

# does 'c' tell us something?

this is the starting 'x' from the 3 byte block ('x', LSB, MSB), it can be used e.g. as another sanity check if we are still in sync:

    buf[idx++] = 'x';
    buf[idx++] = (uint8_t)(val & 0xFF);
    buf[idx++] = (uint8_t)((val>>8) & 0xFF);

Please check also my PR that enables unbuffered scanraw, i.e. the 3 byte blocks are transferred immediately.

[g4ixt]

Hi Martin,

Yes I spotted that it doesn't work for slow scans and had the crashes :-(. I measured the serial port transaction times with 'fast' scans and they are typically 0.1mS but sometimes 120mS. I don't know if this is my PC (which is pretty fast, AMD Ryzen 5) or the TinySA or perhaps both are distracted doing something else.

If I emit the signal from the worker thread every 3 bytes, for higher points / lower RBW it overloads the GUI event loop and it becomes unresponsive, so I am experimenting with bigger chunks.

Something else it might be worth asking Erik about is if 'scanraw' could have a looping option and send some kind of signal when it's starting a fresh loop, or include the frequency as well as the level in the line-by-line output. It appears to take a long time to re-send the scan command and for the measurement to re-start (but I only measured this once). The problem then would be how to stop it looping.

My last read:

null = (serialPort.read(3)) # discard the command prompt

Wsa intended to read in the b'> '. Perhaps I need to set that to null = (serialPort.read(2))?

[Ho-Ro]

If I emit the signal from the worker thread every 3 bytes, for higher points / lower RBW it overloads the GUI event loop and it becomes unresponsive, so I am experimenting with bigger chunks.

That's because of the buffered transfer from tinySA in blocks of 60 bytes / 20 points. If you read in 3 byte blocks you will get suddenly 20 values that are then transferred to the gui without any delay. In this case it would be better to update the gui only every 20 points.

And yes, read(2) is correct as last read.

[Ho-Ro]

so I am experimenting with bigger chunks

Please check my measurement() hack, I update the trace in about 10 parts, the chunksize is a multiple of 20 (the tinySA scanraw chunk size). Also, I keep the previous scan instead of starting with a new -100 dBm for each new trace.
It also works well for very slow scans (rbw=3) with my "unbufferd scanraw" firmware when I set chunksize = 1.

I would reduce the maximum dot size to 3600, at higher values my laptop is very slow to respond with the current "square" sweepresults solution.
Do we really need a 3D history as big as the point size, wouldn't 100 be enough?

    def measurement(self, f_low, f_high):  # runs in a separate thread
        self.threadrunning = True
        # update gui trace in ~10 chunks, chunksize is multiple of 20 (tinySA scanraw chunksize)
        chunksize = ( self.points // 200) * 20
        if chunksize < 20:
            chunksize = 20
        while self.sweeping:
            with serial.Serial(self.dev, baudrate=3000000) as serialPort:  # baudrate does nothing for USB cnx
                #serialPort.timeout = self.timeout
                serialPort.timeout = 1  # anything < 0.12 causes errors (when running in Spyder)
                scan_command = f'scanraw {int(f_low)} {int(f_high)} {int(self.points)}\r'.encode()
                serialPort.write(scan_command)
                index = 0
                serialPort.read_until(scan_command + b'\n{')  # skip command echo
                dataBlock = ''
                samples = 0
                while dataBlock != b'}ch':  # if dataBlock is '}ch' it's reached the end of the scan points
                    dataBlock = (serialPort.read(3))  # read a block of 3 bytes of data
                    logging.debug(f'dataBlock: {dataBlock}\n')
                    if dataBlock != b'}ch':
                        c, data = struct.unpack('<' + 'cH', dataBlock)  # does 'c' tell us something?
                        dBm_power = (data / 32) - self.scale  # scale 0..4095 -> -128..-0.03 dBm
                        # write each measurement into sweepresults
                        self.sweepresults[0, index] = dBm_power
                        samples += 1
                        if samples >= chunksize: #  and emit if chunksize reached
                            samples = 0
                            self.signals.result.emit(self.sweepresults)
                        index += 1
                    logging.debug(f'level = {dBm_power}dBm')
                if samples: # emit also last partly filled block
                    self.signals.result.emit(self.sweepresults)
                null = (serialPort.read(2))  # discard the command prompt
            # # store each sweep in an array with most recent at index 0
            self.sweepresults = np.roll(self.sweepresults, 1, axis=0)
            self.sweepresults[0] = np.copy(self.sweepresults[1]) # init with previous scan
            # self.sweepresults[0] = dBm_power
            # self.signals.result.emit(self.sweepresults)
        self.threadrunning = False

[g4ixt]

I have also adopted updating in chunks and keeping the previous sweep, I just haven't updated GitHub yet. The 'square' sweepresults was just a quick fix. I intend to make 'history' adjustable using a slider - I have just updated the GUI to make space for it.

The 'history' isn't just used for 3D it's also for the average, max and min traces. I have also defaulted in trace types. I will put something in the Wiki about traces and markers.

I will go look for your firmware :-)

[g4ixt]

This is my latest 'measurement' code. I have changed my 'chunksize' to 20 (although I had coded mine differently). I am wondering if there is any value in setting the 'chunksize' based on 'points' as you have done - it seems to me that if it doesn't block the GUI updating it's better to update more often.

def measurement(self, f_low, f_high):  # runs in a separate thread
    self.threadrunning = True
    while self.sweeping:
        with serial.Serial(self.dev, baudrate=3000000) as serialPort:  # baudrate does nothing for USB cnx
            #serialPort.timeout = self.timeout
            serialPort.timeout = 5  # test
            scan_command = f'scanraw {int(f_low)} {int(f_high)} {int(self.points)}\r'.encode()
            serialPort.write(scan_command)
            index = 0
            serialPort.read_until(scan_command + b'\n{')  # skip command echo
            dataBlock = ''
            self.sweepresults[0] = self.sweepresults[1]  # populate each sweep with previous sweep as starting point
            while dataBlock != b'}ch':  # if dataBlock is '}ch' it's reached the end of the scan points
                dataBlock = (serialPort.read(3))  # read a block of 3 bytes of data
                logging.debug(f'dataBlock: {dataBlock}\n')
                if dataBlock != b'}ch':
                    c, data = struct.unpack('<' + 'cH', dataBlock)
                    dBm_power = (data / 32) - self.scale  # scale 0..4095 -> -128..-0.03 dBm
                    # write each measurement into sweepresults and emit
                    self.sweepresults[0, index] = dBm_power
                    if index // 20 == index / 20 or index == (self.points - 1):
                        self.signals.result.emit(self.sweepresults)
                    index += 1
                logging.debug(f'level = {dBm_power}dBm')
            null = (serialPort.read(2))  # discard the command prompt
        # store each sweep in an array with most recent at index 0
        self.sweepresults = np.roll(self.sweepresults, 1, axis=0)
    self.threadrunning = False

[Ho-Ro]

The chunksize based on points should further reduce the load on the gui.

The added app.processEvents() in scan() updates the trace of a slow scan also after stopping:

def scan():
    if tinySA.sweeping:  # if it's running, stop it
        tinySA.sweeping = False  # tells the measurement thread to stop once current scan complete
        ui.scan_button.setEnabled(False)  # prevent repeat presses of 'stop'
        ui.scan_button.setText('Stopping...')  # toggle the 'Stop' button text
        while tinySA.threadrunning:
            app.processEvents() # update the trace
            time.sleep(0.1)  # wait until the measurement thread stops using the serial comms
        ui.scan_button.setEnabled(True)
        activeButtons(True)
        ui.scan_button.setText('Run')  # toggle the 'Stop' button text
        ui.battery.setText(tinySA.battery())
        # tinySA.updateTimeSpectrum()
    else: # start the scan
        ...

[g4ixt]

I just pushed my latest version to 'development'. I think the incremental trace is working quite well now. I have made the 3D graph optional to reduce the graphics load. 'History' is now adjustable too, and 'points' is an integer spinbox with a range of 100 to 3500.

[g4ixt]

Martin,

I think the code would be more efficient if the 'measurement' function read the entire buffer of 20 readings at once, but I haven't figured out how to do that, could you tell me how to do it please? That way there would be no need to count 20 readings before updating the GUI, it could occur once every buffer.

I have changed the way the programme starts up so that it will load if the TinySA isn't connected. I would appreciate it if you could check it still works on the 'original' TinySA, then I will update the main branch from development and post something on the TinySA group. Perhaps I should create a 0.1 'release'.

Thanks for your help.

Ian

[Ho-Ro]

Ian,

On my 1st glance it looks good, my tinySA 'Basic' also is detected and works fine. One point I would like to modify is the different 'Spur' behaviour, the Ultra has the commands spur auto and spur off, while the Basic uses spur on and spur off. I will have a deeper look at the weekend - also for the blocksize 20 change (maybe it doesn't make a big difference due to the buffering by the usb kernel code).

Martin

[g4ixt]

Martin,

Thank you for testing the Basic TinySA. My reasoning for proposing using the blocksize of 20 is because I am limiting the GUI updates from the measurement thread to every 20 readings anyway, otherwise slow PCs just stop responding to mouse clicks (e.g. my Lenovo Ideapad 110s).

Ian

[erikkaashoek]

tinySA ultra has spur auto,on and off. Basic only has on,off Ultra needs auto when scanning above 800MHz

…

On Tue, Jul 18, 2023, 20:40 Martin ***@***.***> wrote: Ian, On my 1st glance it looks good, my tinySA 'Basic' also is detected and works fine. One point I would like to modify is the different 'Spur' behaviour, the Ultra has the commands spur auto and spur off, while the Basic uses spur on and spur off. I will have a deeper look at the weekend - also for the blocksize 20 change (maybe it doesn't make a big difference due to the buffering by the usb kernel code). Martin — Reply to this email directly, view it on GitHub <#6 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDIMRDFWRWFNPFJEVX5N6DXQ3KDJANCNFSM6AAAAAAZZNVPNA> . You are receiving this because you commented.Message ID: ***@***.***>

[g4ixt]

Erik, thank you for the info, I will amend the code.

[g4ixt]

Fixed. I also added code to make attenuator/lna setting mutually exclusive to mimic behaviour of TinySA itself.
I had already coded some of this but not pushed it to 'development' branch when Martin @Ho-Ro sent me some code for it as well and I got a bit confused with merging his and mine, but sorted it out in the end manually with some of Martin's and some of mine.

[Ho-Ro]

Ian, well done, I like your spur button solution.

[g4ixt]

Hi Martin,

The time-out calculation is now too short and is causing the app to hang with 'unpack' errors. For example with these settings, when dragging marker 1

[Ho-Ro]

Well, let me think about a better solution, maybe check the size of received data and see if the communication started is not ready or if it is stalled. In the meantime just increase the timeout, e.g. by adding 5 or 10 more seconds.

[g4ixt]

that was from a long time ago, it seems to be ok as it is so must have been something else

[Ho-Ro]

My fault, I just detected that I answered in this old old thread, dunno why it popped up as new - ???
But nevertheless the scanraw command has got a 4th optional argument:

scanraw f_low f_high points unbuffered

With unbuffered != 0 it sends each point value w/o delay as soon as it is available, this would give a smoother trace buildup.
This feature is available in Erik's FW source since v1.4.115 (12. Aug. 2023)

$ git log  --oneline | grep scanraw_unbuf
2939d1b Merge pull request #71 from Ho-Ro/scanraw_unbuf

$ git log  2939d1b | head -8
commit 2939d1bfbc84e0ae8ef12e0bd3524b9550e21a84
Merge: 37e3a63 feb4cfe
Author: erikkaashoek <[email protected]>
Date:   Sat Aug 12 16:29:28 2023 +0200

    Merge pull request #71 from Ho-Ro/scanraw_unbuf
    
    additional "unbuffered" option for scanraw

$ git describe --tags 2939d1b
v1.4-115-g2939d1b

You can get a numerical version with this modified function version():

    def version(self):
        command = 'version\r'.encode()
        version = self.serialQuery(command) # e.g. tinySA_v1.4-115-g2939d1b
        ver1, ver2, _ = version.split('v')[1].split('-') # 1.4, 115, g2939d1b
        self.verNum = int( 1e4 * float( ver1 ) + int( ver2 ) ) # -> 14115
        print( f'verNum: {self.verNum}' )
        return version

[g4ixt]

Hi Martin,

Thanks for that info. I don't know if using scanraw 'unbuffered' will help because I already have to limit the GUI updates to every 20 points to prevent it freezing with high 'points'. But pyqtgraph should be capable of very fast updates so it's probably due to my other GUI updates and I might be able to just update the traces faster separately - I will look at it.

My development version v0.9.0 now supports settings updates whilst scanning. I am still working on it.

I wonder if it's possible to have scanraw 'loop' instead of having to send a fresh USB command every scan? This would eliminate the USB overhead, but it would also have to be possible to send 'not a query' commands to the TinySA without it corrupting the data being sent from it, like terminal 'no echo' for example, and to read and write to it separately. This would also possibly allow stopping of a slow scan - I have tried resetting the TinySA during a slow scan to stop it, but it just corrupts the trace.

I will update to the latest firmware.

Ian

[g4ixt]

v0.11.7 now uses the new scanraw auto-repeat option from firmware 1.4.177 which speeds up faster sweeps quite noticeably