Reconstructor Outputs Unexpected Or No Content

[florianeckert]

I'm having trouble getting a usable output from the reconstructor. The output file is either empty or much smaller than the compressed iqz file and has no meaningful content.
I'm running SparSDR on a previous stable commit (794d125), because I had trouble builing with gr-iio. Its running on a Docker Container with ubuntu 18.04 and gnuradio 3.7 installed. Dockerfile:

FROM ubuntu:18.04

# install gnuradio
RUN apt-get update
RUN apt-get install -y gnuradio

# clone sparsdr into /opt/
RUN apt-get install -y git
WORKDIR /opt
RUN git clone https://github.com/ucsdsysnet/sparsdr
WORKDIR /opt/sparsdr
RUN git reset --hard 794d125dba34ca8a77410ca3460c4b396ca66360
# install dependencies
RUN apt-get install -y build-essential cmake libuhd-dev libfftw3-3 swig
RUN apt-get install -y qt5-default qttools5-dev-tools
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
# install sparsdr_reconstruct
WORKDIR /opt/sparsdr/reconstruct
RUN cargo install --path .
# install gr-sparsdr
WORKDIR /opt/sparsdr/gr-sparsdr
RUN mkdir build
WORKDIR /opt/sparsdr/gr-sparsdr/build
RUN cmake ..
RUN make
RUN make install
RUN ldconfig 

WORKDIR /root
ENTRYPOINT ["/bin/bash"]

I'm using an N210 for reception. In contrast to the paper mine has an SBX 400-4400 daughterboard build in.
I generate the compressed file using the following options in the CLI:
$ sparsdr_receive --antenna TX/RX --gain 30 --threshold 10000 --frequency 24500000

I reconstruct using:
$ sparsdr_reconstruct --source compressed.iqz --destination out.bin --log-level DEBUG
which gives me the following output

14:22:05 [DEBUG] sparsdr_reconstruct::setup: Opening file compressed.iqz to read compressed samples
14:22:05 [DEBUG] sparsdr_reconstruct::setup: Opening file out.bin for output
14:22:05 [DEBUG] sparsdr_reconstruct::setup: Finished opening files
4% ETA 4s
14:22:05 [INFO] Decimation 1
14:22:05 [INFO] Window sum 1024.0002
99% ETA 0s
100% ETA 0s

Then the file is either empty or smaller by a factor of around 4 compared to the compressed file. If I display the non-empty file in a waterfall sink in grc (using a file source block with repeat set to yes), there is mostly just DC, but nothing that looks like valid output.

The USRP seemed to have no malfunctions when used before with the default firmware. My best guess is that it has maybe something to do with the different daughterboard?

If it helps I can also provide a compressed our decompressed file.

[samcrow]

Thanks for providing a detailed description of the problem. The SBX board should definitely work - it's the main type that we've been testing with.

If you send the compressed file, I can try some things and look for possible problems.

Is the build problem you had something that I can fix? Ideally, I'd like everyone to be able to use the latest gr-sparsdr version so that I can focus on making the latest version work.

[florianeckert]

Thanks for your fast answer. These are two compressed files which produce empty or very small (16K) output respectively.
compressed_empty.iqz.zip
compressed_notempty.iqz.zip

The build issue might be related to me using an maybe outdated version of gr-iio from the ubuntu repos, but I haven't look into it too much yet because getting to run sparsdr at all was more pressing. I get the following error:

[ 35%] Building CXX object lib/CMakeFiles/gnuradio-sparsdr.dir/compressing_plutosdr_source_impl.cc.o
/opt/sparsdr/gr-sparsdr/lib/compressing_plutosdr_source_impl.cc: In constructor 'gr::sparsdr::compressing_plutosdr_source_impl::compressing_plutosdr_source_impl(const string&, long long unsigned int, double)':
/opt/sparsdr/gr-sparsdr/lib/compressing_plutosdr_source_impl.cc:62:59: error: no matching function for call to 'gr::iio::fmcomms2_source::make(const string&, long long unsigned int&, int, int, bool, bool, bool, bool, int, bool, bool, bool, const char [7], double, const char [7], double, const char [11])'
                                               "A_BALANCED"))
                                                           ^
In file included from /opt/sparsdr/gr-sparsdr/lib/compressing_plutosdr_source_impl.h:24:0,
                 from /opt/sparsdr/gr-sparsdr/lib/compressing_plutosdr_source_impl.cc:25:
/usr/include/gnuradio/iio/fmcomms2_source.h:53:19: note: candidate: static gr::iio::fmcomms2_source::sptr gr::iio::fmcomms2_source::make(const string&, long long unsigned int, long unsigned int, long unsigned int, long unsigned int, bool, bool, bool, bool, long unsigned int, bool, bool, bool, const char*, double, const char*, double, const char*, const char*, bool)
       static sptr make(const std::string &uri, unsigned long long frequency,
                   ^~~~
/usr/include/gnuradio/iio/fmcomms2_source.h:53:19: note:   candidate expects 20 arguments, 17 provided
lib/CMakeFiles/gnuradio-sparsdr.dir/build.make:398: recipe for target 'lib/CMakeFiles/gnuradio-sparsdr.dir/compressing_plutosdr_source_impl.cc.o' failed
make[2]: *** [lib/CMakeFiles/gnuradio-sparsdr.dir/compressing_plutosdr_source_impl.cc.o] Error 1
CMakeFiles/Makefile2:139: recipe for target 'lib/CMakeFiles/gnuradio-sparsdr.dir/all' failed
make[1]: *** [lib/CMakeFiles/gnuradio-sparsdr.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

[florianeckert]

Did you find some time to look at the issue?
I tried some more things, maybe it helps to locate the point of failure better.

This time I placed the bluetooth sender right next to the antennas to increase SNR, so maybe something can be seen from the decompessed samples. I confirmed beforehand that Bluetooth frames can clearly be made out in a waterfall diagram with the default FPGA image.
I then captured 15min of traffic once with TX/RX and once with RX2, to rule out something is wrong with the antenna paths. There were no noteworthy differences.
I also fine-tuned the parameters so that they were just below the threshold of an overflow happening during these 15min. I used a gain of 15 and threshold of 25000. I also had a typo in my initial post, I obv listen at 2.45GHz, so:
$ sparsdr_receive --antenna TX/RX --gain 15 --threshold 2500 --frequency 2450000000
I stopped the capture after 15min using CTRL+C. I then get a 43MB file, which seems reasonable for 15min of capture. Here is the file:

compressed.iqz.tar.gz

I then decompress using the CLI, as desribed before. I look at the sample using
this block diagram:

Using 1 bin for decrompession it looks like this:

Using 2048 bins it basically looks the same. But using e.g. 200 bins it will look like this:

I also tried using the SparSDR Reconstruct From File Block in grc, but I get the following error when linked to the compressed.iqz:

gr::log :WARN: file_source0 - file size is not a multiple of item size
thread[thread-per-block[0]: <block file_source(2)>]: fread error

I appreciate your help, thanks.

[florianeckert]

The issues still persist when using exactly the same UHD and gnuradio versions (build from source) you tested sparsdr with and mentioned in the dependencies, which brings me to the assumption that this might not necessarily be a software problem.
I've now seen that there are two SBX daughterboards. I'm using the SBX 40MHz, while there also exists an SBX 120MHz, which by the specification is only compatible with the USRP X-series. In the case studies of the paper you mention using the latter.
Have you also done tests using the SBX 40MHz? If not, can you make an assumption on whether sparsdr should work with the SBX 40MHz as well?