[RSDK-9132] Add (Get)Image to the camera interface #4487

hexbabe · 2024-10-24T19:00:39Z

Overview

https://viam.atlassian.net/browse/RSDK-9132

This PR does not remove Stream from the Camera interface or any gostream logic. It just wraps it all in better English that corresponds with the gRPC method GetImage. Mostly just adding syntactic sugar over stream.Next/ReadImage using a new Image method (for existing builtins that use gostream), and modifying the camera server and client appropriately to utilize Image.

My hopes are that this encourages people to never be tempted to use Stream in modules or even VideoReaderFunc. We will eventually removeStream from the base Camera interface

It will also provide a significant performance boost to the ReadImage data collector since we no longer have to transcode the old image.Image output into bytes anymore in collector.go.

Summary of changes

Add Image to camera interface
Use Image in camera/client.go, camera/server.go and data collector
Use Image in builtins (webcam, replaypcd, ffmpeg, entire transform pipeline, fake etc.) instead of Stream wrappers in preparation for removing Stream entirely from the camera interface

Note

Will this break cli module generator

Manual Tests

Did this list of manual tests already twice at different points in development, but should re-test after these code reviews.

viamrtsp h264 with and without passthrough
Webcam
Webcam hot swap (reconfigure/rebuild due to switching sources)
Transform pipeline on webcam (resize and rotate)
ffmpeg behavior with video0 is on par with the main branch (broken?? yes made ticket to track)
ffmpeg works for a fake mediamtx+ffmpeg rtsp stream though! No hang and stream looks great
Fake camera looks good with and without rtp passthrough toggled
OAK-FFC-3P

Also built the viamrtsp module with local replacement off this branch and the stream is working for both rtp passthrough true and false, indicating that videosourcewrapped modules shouldn't be affected by this change.

Profile for potential performance improvement in data collection

If the PDFs are insanely blurry, try downloading and opening with system viewer

Link

The big thing that Nick noted was alarming was the staggering memory used to alloc image NewYCbCr. This is no longer the case on this feature branch.

Config used for pprof

{
  "components": [
    {
      "name": "nickcam-big-green-1",
      "namespace": "rdk",
      "type": "camera",
      "model": "ncs:camera:nickcam",
      "attributes": {
        "big": true,
        "color": "green",
        "image_type": "jpeg"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "nickcam-big-green-2",
      "namespace": "rdk",
      "type": "camera",
      "model": "ncs:camera:nickcam",
      "attributes": {
        "color": "green",
        "image_type": "jpeg",
        "big": true
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "nickcam-big-green-3",
      "namespace": "rdk",
      "type": "camera",
      "model": "ncs:camera:nickcam",
      "attributes": {
        "image_type": "jpeg",
        "big": true,
        "color": "green"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    }
  ],
  "services": [
    {
      "name": "data_manager-1",
      "namespace": "rdk",
      "type": "data_manager",
      "attributes": {
        "sync_disabled": true,
        "capture_disabled": false
      }
    }
  ],
  "modules": [
    {
      "type": "registry",
      "name": "ncs_nickcam",
      "module_id": "ncs:nickcam",
      "version": "latest"
    }
  ]
}

Comparative performance from an operator's perspective

Config used

{
  "components": [
    {
      "name": "rtsp-1",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "rtsp-2",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "rtsp-3",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "rtsp-4",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    },
    {
      "name": "rtsp-5",
      "namespace": "rdk",
      "type": "camera",
      "model": "viam:viamrtsp:rtsp",
      "attributes": {
        "rtsp_address": "rtsp://10.1.12.67:554"
      },
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "ReadImage",
                "capture_frequency_hz": 60,
                "additional_params": {},
                "capture_queue_size": 0
              }
            ]
          }
        }
      ]
    }
  ],
  "services": [
    {
      "name": "data_manager-1",
      "namespace": "rdk",
      "type": "data_manager",
      "attributes": {
        "sync_disabled": true,
        "capture_disabled": false
      }
    }
  ],
  "modules": [
    {
      "type": "registry",
      "name": "viam_viamrtsp",
      "module_id": "viam:viamrtsp",
      "version": "0.2.0-rc4",
      "disabled": false
    }
  ]
}

v0.50.0 branch (RPI4):

Feature branch (RPI4):

v0.50.0 (Orin Nano):

Feature branch (Orin Nano):

RPI4 memory improvement: I noticed that off the feature branch, memory stabilized around 1.9gb, while on the main branch, memory stabilized around 2.5gb

RPI4 CPU improvement: I learned how to filter for processes in this comparison. It's quite clear that the CPU usage is lower as all figures are lower across the board for the feature branch.

Orin Nano CPU improvement: Though the core usage was about the same between main and feature, I noticed that on the latest release, the viam-server processes were always taking the most CPU. Off the feature branch, viamrtsp were always on top.

I did not notice memory improvements on the Orin Nano. May be missing something. Operator skill issue perhaps

github-actions · 2024-10-24T19:00:54Z

Warning your change may break code samples. If your change modifies any of the following functions please contact @viamrobotics/fleet-management. Thanks!

component	function
base	IsMoving
board	GPIOPinByName
camera	Properties
encoder	Properties
motor	IsMoving
sensor	Readings
servo	Position
arm	EndPosition
audio	MediaProperties
gantry	Lengths
gripper	IsMoving
input_controller	Controls
movement_sensor	LinearAcceleration
power_sensor	Power
pose_tracker	Poses
motion	GetPose
vision	GetProperties

hexbabe · 2024-10-25T18:59:35Z

I'm gonna be out next week but this is ready for a first pass review. Also lmk if we should add Nick or/(and?) Dan to this review. Perhaps for future reviews that are a bit more invasive?

randhid

Since we're breaking go modules by adding another method to the interface, we should make that method follow 1:1 the inputs and returns of the proto api.

This will mean bookkeeping the release of each image stream,should you choose to use ReadImage. This will definitely go away once we get the image from the underlying comms connection like we do in the intel realsense and oak-d camera, I don't know how intense that is.

If changing the returns from GetImage to just be 1:1 the API type cause more problems in changing code outside of camera, we can focus on replacements in the camera packages, and let datamanger and vision packages use Stream and Next for this pr, but I think you're nearly there.

components/camera/camera.go

components/camera/camera_test.go

components/camera/client.go

components/camera/replaypcd/replaypcd.go

components/camera/server.go

components/camera/transformpipeline/mods.go

components/camera/videosource/webcam.go

components/camera/camera.go

seanavery · 2024-11-01T19:48:45Z

components/camera/client.go

@@ -228,6 +228,10 @@ func (c *client) Stream(
 	return stream, nil
 }

+func (c *client) GetImage(ctx context.Context) (image.Image, func(), error) {
+	return c.Read(ctx)


Could we Read and handle release so we do not need to include release func in returns? (perhaps that would make sense in the videosourcewrapper layer).
As mentioned, this would be a larger departure from Stream.Next which assuming will cause major problems.

I'm gonna try to get rid of release in camera components and see what happens

Looks like we just need to move the rimage.EncodeImage step one level in i.e. outBytes should come out of the Image call instead of being handled in the data collector or camera server/client. Currently release is called in the collector/server/client, so we should just move it into the Image implementation. Same logic, just handled more nested in our abstraction.

I guess future module writers should get used to using rimage to encode their output i.e. read from the source bytes and output a newly constructed []byte result? Does that sound okay to everyone?

Looks like we just need to move the rimage.EncodeImage step one level in i.e. outBytes should come out of the Image call instead of being handled in the data collector or camera server/client.

In the client, I think that makes sense to return rimage to match the python sdk functionality -- avoid unnecessary decodes if you just want bytes.

I guess future module writers should get used to using rimage to encode their output i.e. read from the source bytes and output a newly constructed []byte result? Does that sound okay to everyone?

Would this be similar to viamimage in the python sdk, i think that works pretty well.

Looks like the server is already handling release and encoding for the caller.
Are you also suggesting removing encoding step in server GetImage?

This is my WIP in server.go

switch castedCam := cam.(type) { case ReadImager: // RSDK-8663: If available, call a method that reads exactly one image. The default // `ReadImage` implementation will otherwise create a gostream `Stream`, call `Next` and // `Close` the stream. However, between `Next` and `Close`, the stream may have pulled a // second image from the underlying camera. This is particularly noticeable on camera // clients. Where a second `GetImage` request can be processed/returned over the // network. Just to be discarded. // RSDK-9132(sean yu): In addition to what Dan said above, ReadImager is important // for camera components that rely on the `release` functionality provided by gostream's `Read` // such as viamrtsp. // (check that this comment is 100% true before code review then delete this paranthetical statement) img, release, err := castedCam.Read(ctx) if err != nil { return nil, err } defer func() { if release != nil { release() } }() actualMIME, _ := utils.CheckLazyMIMEType(req.MimeType) resp.MimeType = actualMIME outBytes, err := rimage.EncodeImage(ctx, img, req.MimeType) if err != nil { return nil, err } resp.Image = outBytes default: imgBytes, mimeType, err := cam.Image(ctx, req.MimeType, ext) if err != nil { return nil, err } actualMIME, _ := utils.CheckLazyMIMEType(mimeType) resp.MimeType = actualMIME resp.Image = imgBytes }

So I think yes, in the default case we don't encode anymore since the return type is now bytes

Ok cool I like that.

I am little confused on why we still need the ReadImager path here. Shouldnt the camera interface now always have Image defined so we can just use that?

My thinking here is since viamrtsp uses the VideoReaderFunc and Read's release functionality to keep track of pool frames in the frame allocation optimization flow, for the server.go that serves viamrtsp, we need to be able to call release()

I think we could refactor viamrtsp though to just copy out the bytes on return early and use the new .Image pathway... I'm down to remove ReadImager entirely as long as we make it a high priority to refactor viamrtsp to use .Image and []byte

I see, interesting. I think it makes sense to leave in for now for viamrtsp compatibility.

I think we could refactor viamrtsp though to just copy out the bytes on return early and use the new .Image pathway... I'm down to remove ReadImager entirely as long as we make it a high priority to refactor viamrtsp to use .Image and []byte

Yep the whole point to having a memory manager was the issue with passing in a pointer to the avframe in VideoReaderFunc return causing segfaults with the encoding in the server.

Since we are removing encoding from the server and relying on the user to handle this, we should eventually be able to use Image pathway and simplify things quite a bit : )

All our go cameras will need a refactor for the full fix of this interface, I do not anticipate that we will have the best version of this code and go modules after one pr. Eventually we will get rid of Stream and Next completely. The problem is those videosourcewrapper helpers are all over the place in go modules since they're exported convenience functions.

Always export code conscientiously. But we're breaking this interface anyway, so we'll deal with the following breaks.

Okay 👍 will remove ReadImager in this PR

components/camera/server.go

components/camera/transformpipeline/classifier.go

seanavery

I am wondering if it makes sense to do a full removal of Read from client & server, ReadImage, and ReadImager here instead of just aliasing in those cases.

randhid · 2024-11-01T20:50:12Z

I am wondering if it makes sense to do a full removal of Read from client & server, ReadImage, and ReadImager here instead of just aliasing in those cases.

Yes.

seanavery

Should we replace the cemeraservice GetImage call in RenderFrame with Image?

func (s *serviceServer) RenderFrame(
	ctx context.Context,
	req *pb.RenderFrameRequest,
) (*httpbody.HttpBody, error) {
	ctx, span := trace.StartSpan(ctx, "camera::server::RenderFrame")
	defer span.End()
	if req.MimeType == "" {
		req.MimeType = utils.MimeTypeJPEG // default rendering
	}
	resp, err := s.GetImage(ctx, (*pb.GetImageRequest)(req))

hexbabe · 2024-11-06T17:22:50Z

Should we replace the cemeraservice GetImage call in RenderFrame with Image?

func (s *serviceServer) RenderFrame(
	ctx context.Context,
	req *pb.RenderFrameRequest,
) (*httpbody.HttpBody, error) {
	ctx, span := trace.StartSpan(ctx, "camera::server::RenderFrame")
	defer span.End()
	if req.MimeType == "" {
		req.MimeType = utils.MimeTypeJPEG // default rendering
	}
	resp, err := s.GetImage(ctx, (*pb.GetImageRequest)(req))

serviceServers use the gRPC named version, so GetImage is right

nicksanford · 2024-11-14T20:23:47Z

components/camera/server.go

 	}
-	resp.Image = outBytes
-	return &resp, nil
+	actualMIME, _ := utils.CheckLazyMIMEType(resMetadata.MimeType)


Note: This will change behavior.

Lets say you have a viam-server and camera hosted in a module.

Lets say that the camera in the module can only product PNGs but the user ask for a jpeg.

Previously, the server.go in the module would convert the PNG into a JPEG before sending it to viam-server.

Now, with this change, it will return the PNG to the caller.

I think this is still the right change, as implicitly converting between image formats can introduce image quality degridation as well as significantly worse performance (like 15X worse performance), but I want to confirm:

@bhaney & @randhid are we confident no customer robots depend on this implicitly convert between image mime types behavior?

The customers I am aware of rely on video storage and upload and rtsp passthrough where they don't really ask for a specific mime type anyway.

Still trying to track down where gostream actually sets a default for both stream types we use.

nicksanford · 2024-11-14T20:26:38Z

components/camera/transformpipeline/classifier.go

@@ -28,7 +27,7 @@ type classifierConfig struct {

 // classifierSource takes an image from the camera, and overlays labels from the classifier.
 type classifierSource struct {
-	stream             gostream.VideoStream
+	src                camera.VideoSource


why can't this take a camera?

#4487 (comment)

nicksanford · 2024-11-14T20:27:29Z

components/camera/transformpipeline/composed.go

@@ -19,13 +18,13 @@ import (
 // depthToPretty takes a depth image and turns into a colorful image, with blue being
 // farther away, and red being closest. Actual depth information is lost in the transform.
 type depthToPretty struct {
-	originalStream gostream.VideoStream
-	cameraModel    *transform.PinholeCameraModel
+	src         camera.VideoSource


why can't this be a camera?

#4487 (comment)

nicksanford · 2024-11-14T20:27:54Z

components/camera/transformpipeline/depth_edges.go

@@ -23,13 +22,13 @@ type depthEdgesConfig struct {

 // depthEdgesSource applies a Canny Edge Detector to the depth map.
 type depthEdgesSource struct {
-	stream     gostream.VideoStream
+	src        camera.VideoSource


Why can't this be a camera?

#4487 (comment)

components/camera/camera.go

nicksanford · 2024-11-14T20:37:17Z

components/camera/transformpipeline/detector.go

@@ -27,7 +26,7 @@ type detectorConfig struct {

 // detectorSource takes an image from the camera, and overlays the detections from the detector.
 type detectorSource struct {
-	stream       gostream.VideoStream
+	src          camera.VideoSource


why not camera?

#4487 (comment)

nicksanford · 2024-11-14T20:37:23Z

components/camera/transformpipeline/mods.go

-	originalStream gostream.VideoStream
-	stream         camera.ImageType
-	angle          float64
+	src    camera.VideoSource


why not camera?

#4487 (comment)

components/camera/transformpipeline/mods.go

nicksanford · 2024-11-14T20:45:41Z

components/camera/videosourcewrappers.go

+		return nil, ImageMetadata{}, err
+	}
+	defer release()
+	imgBytes, err := rimage.EncodeImage(ctx, img, mimeType)


Isn't this going to force us to do an extra Encode, which is expensive? Doesn't VideoSource already implement Image (Which has the same signature as ReadImageBytes)?

Why do we need this helper?

This helper is necessary for cameras that used adhere to the Read/Stream.Next signature to be able to serve Image. The encode here isn't really an extra encode; it would've happened anyways in server.go before turning into a protobuf message.

robot/client/client_test.go

nicksanford · 2024-11-14T20:48:28Z

robot/client/client_test.go

@@ -512,7 +506,7 @@ func TestStatusClient(t *testing.T) {

 	camera1, err := camera.FromRobot(client, "camera1")
 	test.That(t, err, test.ShouldBeNil)
-	_, _, err = camera.ReadImage(context.Background(), camera1)
+	_, _, err = camera1.Image(context.Background(), rutils.MimeTypeJPEG, nil)


shouldn't we test that this DOESN'T return JPEG bytes and still doesn't error?

I added a check to make sure that the bytes and metadata are nil and empty struct respectively

nicksanford · 2024-11-14T20:49:18Z

robot/impl/local_robot_test.go

@@ -93,7 +93,7 @@ func TestConfig1(t *testing.T) {
 	c1, err := camera.FromRobot(r, "c1")
 	test.That(t, err, test.ShouldBeNil)
 	test.That(t, c1.Name(), test.ShouldResemble, camera.Named("c1"))
-	pic, _, err := camera.ReadImage(context.Background(), c1)
+	pic, err := camera.GoImageFromCamera(context.Background(), rutils.MimeTypeJPEG, nil, c1)


why are we specifying jpeg here, when previously the ReadImage implementation didn't?

Had to specify something because otherwise we hit

/Users/seanyu/Projects/sean-rdk/robot/impl/local_robot_test.go:98: Expected: nil Actual: 'could not get image bytes from camera: do not know how to encode ""'

Stemming from https://github.com/hexbabe/sean-rdk/blob/65b5bad742bf59c60e249d74f22338c06753465f/components/camera/videosourcewrappers.go#L209

…from img; Fix prev test bug

hexbabe · 2024-11-15T15:58:10Z

components/camera/client_test.go

 		}

-		// one kvp created with camera.Extra
-		ext := camera.Extra{"hello": "world"}
-		ctx = camera.NewContext(ctx, ext)


Leaving this test deleted since we no longer have this helper function

Init craziness

a643d22

viambot added the safe to test This pull request is marked safe to test from a trusted zone label Oct 24, 2024

Use camera pkg scoped ReadImage in webcam

41cb592

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Oct 25, 2024

hexbabe marked this pull request as ready for review October 25, 2024 18:56

hexbabe requested review from randhid and seanavery October 25, 2024 18:58

randhid requested changes Oct 31, 2024

View reviewed changes

seanavery reviewed Nov 1, 2024

View reviewed changes

Merge branch 'main' into RSDK-9132

f6e3d69

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 4, 2024

seanavery reviewed Nov 5, 2024

View reviewed changes

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 6, 2024

hexbabe requested review from ethanlookpotts, micheal-parks, DTCurrie and zaporter-work as code owners November 6, 2024 17:59

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 6, 2024

Use agreed upon Image signature

d6439dd

hexbabe force-pushed the RSDK-9132 branch from f90689b to d6439dd Compare November 6, 2024 18:10

viambot removed the safe to test This pull request is marked safe to test from a trusted zone label Nov 6, 2024

nicksanford reviewed Nov 14, 2024

View reviewed changes

Merge branch 'viamrobotics:main' into RSDK-9132

65b5bad

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 14, 2024

If err, then bytes and metadata should be falsy

9a9612f

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 14, 2024

hexbabe force-pushed the RSDK-9132 branch from 72b76ca to 003c379 Compare November 14, 2024 21:53

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 14, 2024

hexbabe force-pushed the RSDK-9132 branch from 003c379 to d31b434 Compare November 15, 2024 14:02

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 15, 2024

hexbabe force-pushed the RSDK-9132 branch from d31b434 to 7e710a4 Compare November 15, 2024 14:04

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 15, 2024

Use imgbuf bytes directly for a robot client test rather than encode …

ab7e736

…from img; Fix prev test bug

hexbabe force-pushed the RSDK-9132 branch from 7e710a4 to ab7e736 Compare November 15, 2024 14:37

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 15, 2024

hexbabe commented Nov 15, 2024

View reviewed changes

Add back ReadImage/Stream tests where it makes sense

8244dfd

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 15, 2024

hexbabe requested a review from nicksanford November 15, 2024 21:04

[RSDK-9132] Add (Get)Image to the camera interface #4487

Are you sure you want to change the base?

[RSDK-9132] Add (Get)Image to the camera interface #4487

Conversation

hexbabe commented Oct 24, 2024 • edited Loading

Overview

Summary of changes

Note

Manual Tests

Profile for potential performance improvement in data collection

Config used for pprof

Comparative performance from an operator's perspective

Config used

v0.50.0 branch (RPI4):

Feature branch (RPI4):

v0.50.0 (Orin Nano):

Feature branch (Orin Nano):

github-actions bot commented Oct 24, 2024

hexbabe commented Oct 25, 2024

randhid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hexbabe Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hexbabe Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hexbabe Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

seanavery Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seanavery left a comment

Choose a reason for hiding this comment

randhid commented Nov 1, 2024

seanavery left a comment

Choose a reason for hiding this comment

hexbabe commented Nov 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hexbabe Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hexbabe Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hexbabe commented Oct 24, 2024 •

edited

Loading

hexbabe Nov 4, 2024 •

edited

Loading

hexbabe Nov 5, 2024 •

edited

Loading

hexbabe Nov 5, 2024 •

edited

Loading

seanavery Nov 5, 2024 •

edited

Loading

hexbabe Nov 14, 2024 •

edited

Loading

hexbabe Nov 14, 2024 •

edited

Loading