Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to receive frames from Femto Mega after firmware update. #28

Open
Suedocode opened this issue Dec 7, 2024 · 7 comments
Open
Assignees

Comments

@Suedocode
Copy link
Collaborator

Suedocode commented Dec 7, 2024

Updated Femto Mega firmware to v1.2.9 -> v1.3.0 and our sample can no longer stream from them. OrbbecViewer (v1.10.16) can successfully pull frames though, and I'm not sure what the difference is. Here's the code we're using:

#include <libobsensor/ObSensor.hpp>
#include <mutex>
#include <condition_variable>
#include <iostream>

static constexpr uint32_t s_fps = 15;

static constexpr uint32_t s_dw = 640;
static constexpr uint32_t s_dh = 576;
static constexpr uint32_t s_dsz = s_dw*s_dh*2;

static constexpr uint32_t s_cw = 1920;
static constexpr uint32_t s_ch = 1080;
static constexpr uint32_t s_csz = s_dw*s_dh*4;

using namespace std::chrono;
using namespace ob;

int main() {
  ob::Context::setExtensionsDirectory("/opt/lib/ob_extensions");
  ob::Context::setLoggerSeverity(OB_LOG_SEVERITY_INFO);
  ob::Context ctx;
  auto mDev = ctx.createNetDevice("192.168.1.5", 83);
  auto mPipeline = std::make_shared<ob::Pipeline>(mDev);
  auto mConfig = std::make_shared<ob::Config>();

  mConfig->enableStream(
      mPipeline
        ->getStreamProfileList(OB_SENSOR_DEPTH)
        ->getVideoStreamProfile(s_dw, s_dh, OB_FORMAT_Y16, s_fps)
      );
  mConfig->enableStream(
      mPipeline
        ->getStreamProfileList(OB_SENSOR_COLOR)
        ->getVideoStreamProfile(s_cw, s_ch, OB_FORMAT_BGRA, s_fps)
      );
  mConfig->setAlignMode(ALIGN_DISABLE);
  mConfig->setFrameAggregateOutputMode(OB_FRAME_AGGREGATE_OUTPUT_ALL_TYPE_FRAME_REQUIRE);
  mPipeline->enableFrameSync();

  std::cout << "Starting pipeline" << std::endl;
  std::condition_variable cv;
  std::mutex mut;
  std::shared_ptr<ob::FrameSet> frameset;
  mPipeline->start(
      mConfig,
      [&](std::shared_ptr<ob::FrameSet> fs) {
        if (fs == nullptr) return;
        std::unique_lock lk(mut);
        frameset = std::move(fs);
        cv.notify_one();
      });
  //mPipeline->start(mConfig);

  int itrs = 0;
  while(++itrs < 10) {
    std::unique_lock lk(mut);
    frameset = nullptr;
    std::cout << "waiting on frame..." << std::endl;
    cv.wait(lk, [&](){ return frameset != nullptr; });
    //auto frameset = mPipeline->waitForFrames(10000);
    std::cout << "Recieved frame" << std::endl;

    auto df = frameset->depthFrame();
    auto cf = frameset->colorFrame();
  }
}

Here's the output, with extra stuff removed:

Starting pipeline
[12/07 12:42:11.878187][info][1844053][Pipeline.cpp:180] Check and set config done!
[12/07 12:42:11.878208][info][1844053][Pipeline.cpp:218] Try to start streams!
[12/07 12:42:11.878215][info][1844053][VideoSensor.cpp:51] Try to start stream: {type: Depth, format: Y16, width: 640, height: 576, fps: 15}
[12/07 12:42:11.878221][info][1844053][VideoSensor.cpp:84] Start backend stream: {type: Depth, format: Y16, width: 640, height: 576, fps: 15}
[12/07 12:42:12.736421][info][1844053][VideoSensor.cpp:51] Try to start stream: {type: Color, format: BGRA, width: 1920, height: 1080, fps: 15}
[12/07 12:42:12.736500][info][1844053][VideoSensor.cpp:84] Start backend stream: {type: Color, format: MJPG, width: 1920, height: 1080, fps: 15}
[12/07 12:42:13.433189][info][1844053][Pipeline.cpp:234] Start streams done!
[12/07 12:42:13.433270][info][1844053][Pipeline.cpp:202] Pipeline start done!
waiting on frame...
[12/07 12:42:15.094938][warning][1844068][VideoSensor.cpp:137] This frame will be dropped because the data size does not match the expectation! size=40896, expected=737280 @Depth
[12/07 12:42:17.769712][info][1844071][SensorBase.cpp:263] Depth Streaming... frameRate=15.133413fps
[12/07 12:42:18.521925][info][1844075][SensorBase.cpp:263] Color Streaming... frameRate=15.292950fps
[12/07 12:42:22.824011][info][1844071][SensorBase.cpp:263] Depth Streaming... frameRate=15.037594fps
[12/07 12:42:23.572185][info][1844075][SensorBase.cpp:263] Color Streaming... frameRate=15.046525fps
[12/07 12:42:27.890351][info][1844071][SensorBase.cpp:263] Depth Streaming... frameRate=14.999014fps

For some reason, the frames are not getting put into the frameset callback. Similarly, waitForFrames times out in the same way, and never produces a valid frameset.

Here is the relevant directory structure:

$ tree /opt/lib/
/opt/lib/
├── libOrbbecSDK.so
├── ob_extensions
│   ├── filters
│   │   ├── libFilterProcessor.so
│   │   └── libob_priv_filter.so
│   ├── firmwareupdater
│   │   └── libfirmwareupdater.so
│   └── frameprocessor
│       └── libob_frame_processor.so

Environment:
Ubuntu 22.04.5 LTS (x86_64)
g++ 11.4.0
OrbbecSDK_v2 51c2902 (tagged v2.0.24-rc)
Femto: Firmware v1.3.0

@Suedocode Suedocode changed the title Femto Mega v1.3.0 Unable to receive frames from Femto Mega after firmware update. Dec 7, 2024
@Sirudoi Sirudoi assigned Sirudoi and unassigned Sirudoi Dec 9, 2024
@Suedocode
Copy link
Collaborator Author

I confirmed that streaming works with firmware 1.2.7, running the same code with different sensors that hadn't been upgraded yet. There's something incompatible with firmware 1.3.0 and femto mega.

@Sirudoi
Copy link
Contributor

Sirudoi commented Dec 10, 2024

Thank you for your testing and feedback!

Could you confirm whether your network bandwidth is gigabit or megabit? (You can check the bandwidth rate in the 'Show Frame Metadata' section of OrbbecViewer.)

From the logs, the stream appears to be normal. The reason for not being able to fetch frames from the pipeline might be due to the simultaneous activation of enableFrameSync and OB_FRAME_AGGREGATE_OUTPUT_ALL_TYPE_FRAME_REQUIRE.
When the bandwidth is insufficient, the depth frames may experience delays compared to the color frames, causing frame synchronization to fail. As a result, the condition of OB_FRAME_AGGREGATE_OUTPUT_ALL_TYPE_FRAME_REQUIRE will not be met, which prevents frames from being fetched from the pipeline."

@Suedocode
Copy link
Collaborator Author

Suedocode commented Dec 10, 2024

Both tested systems use the same hardware and ethernet cables, and they are gigabit. I disabled enableFrameSync and it still did not produce frames. I also disabled OB_FRAME_AGGREGATE_OUTPUT_ALL_TYPE_FRAME_REQUIRE to see what would happen, and it looks like I just got framesets of null frames.

From OrbbecViewer, the depth frames show 88.72Mbps and color frames are 14.18Mbps
(device framerate is set to 15fps). I attached a snip of the window.
OrbbecFrameMeta

@Suedocode
Copy link
Collaborator Author

1.2.7 and 1.2.9 work,1.3.0 does not. All tested on the same hardware configuration.

@Sirudoi
Copy link
Contributor

Sirudoi commented Dec 11, 2024

Hello! If possible, could you please use firmware version 1.3.0 with the following two configurations and provide a debug-level log? I only encountered this issue at a 100Mbps network speed, not at 1000Mbps, so your feedback would be extremely helpful in addressing this problem.

// Configuration 1
mConfig->setFrameAggregateOutputMode(OB_FRAME_AGGREGATE_OUTPUT_ANY_SITUATION);
mPipeline->enableFrameSync();

// Configuration 2
mConfig->setFrameAggregateOutputMode(OB_FRAME_AGGREGATE_OUTPUT_ALL_TYPE_FRAME_REQUIRE);
mPipeline->enableFrameSync();

mPipeline->start(mConfig);

while(true) {
    auto frameSet = mPipeline->waitForFrameset(1000);
    if (frameSet) { // checck if the frameset has depth and color frames }
}

@Suedocode
Copy link
Collaborator Author

Suedocode commented Dec 11, 2024

Output format:

    ++itrs;
    auto df = frameset->depthFrame();
    auto cf = frameset->colorFrame();
    auto ts = df ? df->getTimeStampUs() : cf->getTimeStampUs();
    std::cout << std::setw(2) << itrs << " d[" << (df ? 'O' : 'X')
              << "] c[" << (cf ? 'O' : 'X') << "] " << ts/1000 << std::endl;

Configuration 1 (C1):

Starting pipeline
[12/11 14:15:57.756337][info][5602][Pipeline.cpp:180] Check and set config done!
[12/11 14:15:57.756359][info][5602][Pipeline.cpp:218] Try to start streams!
[12/11 14:15:57.756367][info][5602][VideoSensor.cpp:51] Try to start stream: {type: Depth, format: Y16, width: 640, height: 576, fps: 15}
[12/11 14:15:57.756373][info][5602][VideoSensor.cpp:84] Start backend stream: {type: Depth, format: Y16, width: 640, height: 576, fps: 15}
[12/11 14:15:58.653182][info][5602][VideoSensor.cpp:51] Try to start stream: {type: Color, format: BGRA, width: 1920, height: 1080, fps: 15}
[12/11 14:15:58.653305][info][5602][VideoSensor.cpp:84] Start backend stream: {type: Color, format: MJPG, width: 1920, height: 1080, fps: 15}
[12/11 14:15:59.355045][info][5602][Pipeline.cpp:234] Start streams done!
[12/11 14:15:59.355115][info][5602][Pipeline.cpp:202] Pipeline start done!
1 d[X] c[O] 4492891
 2 d[O] c[X] 4698861
 3 d[X] c[O] 4493629
 4 d[O] c[X] 4698927
 5 d[X] c[O] 4494370
 6 d[O] c[X] 4698994
 7 d[X] c[O] 4495108
 8 d[O] c[X] 4699061
 9 d[X] c[O] 4495848
10 d[O] c[X] 4699127
11 d[X] c[O] 4496588
12 d[X] c[O] 4497327
13 d[O] c[X] 4699260
14 d[X] c[O] 4498066
15 d[O] c[X] 4699327
16 d[X] c[O] 4498808
17 d[O] c[X] 4699394
18 d[X] c[O] 4499547
19 d[X] c[O] 4500287
20 d[O] c[X] 4699460
[12/11 14:20:07.625254][info][5864][Pipeline.cpp:299] Try to stop pipeline!
[12/11 14:20:07.625315][info][5864][Pipeline.cpp:275] Try to stop streams!
[12/11 14:20:07.829372][info][5864][Pipeline.cpp:289] Sensor stream stopped, sensorType=Color
[12/11 14:20:08.027274][info][5864][Pipeline.cpp:289] Sensor stream stopped, sensorType=Depth
[12/11 14:20:08.027347][info][5864][Pipeline.cpp:295] Stop streams done!
[12/11 14:20:08.027394][info][5864][Pipeline.cpp:319] Stop pipeline done!
[12/11 14:20:08.027418][info][5864][Pipeline.cpp:46] Pipeline destroyed! @0x5596AC7F8150
[12/11 14:20:08.027635][info][5864][SensorBase.cpp:29] SensorBase is destroyed
[12/11 14:20:08.027689][warning][5864][RTSPStreamPort.cpp:95] Stream have not been started!
[12/11 14:20:08.027964][info][5864][SensorBase.cpp:29] SensorBase is destroyed
[12/11 14:20:08.028028][warning][5864][RTSPStreamPort.cpp:95] Stream have not been started!

Configuration 2 (C2):

Starting pipeline
[12/11 14:18:21.062970][info][5711][Pipeline.cpp:180] Check and set config done!
[12/11 14:18:21.062993][info][5711][Pipeline.cpp:218] Try to start streams!
[12/11 14:18:21.063003][info][5711][VideoSensor.cpp:51] Try to start stream: {type: Depth, format: Y16, width: 640, height: 576, fps: 15}
[12/11 14:18:21.063009][info][5711][VideoSensor.cpp:84] Start backend stream: {type: Depth, format: Y16, width: 640, height: 576, fps: 15}
[12/11 14:18:21.967641][info][5711][VideoSensor.cpp:51] Try to start stream: {type: Color, format: BGRA, width: 1920, height: 1080, fps: 15}
[12/11 14:18:21.968187][info][5711][VideoSensor.cpp:84] Start backend stream: {type: Color, format: MJPG, width: 1920, height: 1080, fps: 15}
[12/11 14:18:22.667668][info][5711][Pipeline.cpp:234] Start streams done!
[12/11 14:18:22.667985][info][5711][Pipeline.cpp:202] Pipeline start done!
<no output>
[12/11 14:18:26.995902][info][5728][SensorBase.cpp:263] Depth Streaming... frameRate=15.350878fps
[12/11 14:18:27.752573][info][5732][SensorBase.cpp:263] Color Streaming... frameRate=15.199368fps
[12/11 14:18:32.060680][info][5728][SensorBase.cpp:263] Depth Streaming... frameRate=15.004935fps
[12/11 14:18:32.753780][info][5732][SensorBase.cpp:263] Color Streaming... frameRate=14.997001fps

In C2, there's no test output. The Orbbec log commenst about fps don't appear in C1, presumably because it ends too soon after receiving 20 frames.

As a side note, the time stamps on color and depth frames in C1 are oddly out-of-sync. I don't expect them to be the same, but I expected them to at least be chronological.

@Sirudoi
Copy link
Contributor

Sirudoi commented Dec 12, 2024

Hello, thank you for your cooperation!

I noticed that there is an anomaly in the Color timestamp, and the log output you provided is also different. If you are using the SDK with the tag v2.0.24-rc, the log output related to FPS should include the device name and device SN, like this:

[12/12 10:59:47.736618][info][172282][SensorBase.cpp:265] Femto Mega(CL2C84F000F): Depth Streaming... frameRate=14.656368fps  
[12/12 10:59:48.717941][info][172304][SensorBase.cpp:265] Femto Mega(CL2C84F000F): Color Streaming... frameRate=15.581303fps

When the Mega device uses firmware version v1.3.0 and the color stream is set to an RGB-related format, the issue you encountered may occur. This problem has already been fixed in version v2.0.23.

Could you please confirm the version number of the SDK you are using? The confirmation method is as follows:

  1. Set the log level to DEBUG
ob::Context::setLoggerSeverity(OB_LOG_SEVERITY_DEBUG);
  1. You will find output similar to the following:
[12/12 11:16:15.674690][debug][173398][Context.cpp:36] Context created! Library version: v2.0.24

In the next version, the SDK version number will be embedded in the .so library name to enhance convenience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants