-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bad Softbot's calibration results with perfect simulation #950
Comments
This branch The configuration file is updated with the new bagfile. On this link you can transfer the dataset and bagfile. To recreate : Calibrating odometry :
Without calibrating odometry :
This is a completly perfect simulation. We start with reprojection errors of around 3px and finish off at around 0.3px. |
I found out that removing the problematic collection 31from the long dataset solves the first issue. With perfect calibration calibrating odometry : This is 264 parameters
Without calibrating odometry : This is 24 parameters.
The optimizer might be falling a local minima with these many parameters maybe, I'm not sure. @manuelgitgomes told me this does make sense due to overfitting. |
#952 was likely another thing throwing us off. |
@miguelriemoliveira I will use the script I made today in #951 to automate all sorts of insightful plots so we can better analyze the data. Are you available too meet Monday to talk about it? I should have all plots ready by then. |
Here are all the plots computed from the batch. I couldn't plot anything like "without calibrating odom vs calibrating odom" but I don't have enough data as I didn't run a lot of experiments without calibrating odometry. The few I have do prove that calibrating a system with noisy odometry I'm not sure if I should rerun all these experiments without calibrating odometry |
This is better because if the fold lists are defined with strings it's limiting on the lambdas. The user has to user whichever string identifier the fold lists's isnt using and he has no idea until the program crashes when evaluating lambdas
Hi @brunofavs , we can talk Monday morning if you want. 9h or later? |
Sounds good to me. Zoom or at Lar? |
Zoom please
…On Sun, May 12, 2024, 4:13 PM Bruno Silva ***@***.***> wrote:
Sounds good to me.
Zoom or at Lar?
—
Reply to this email directly, view it on GitHub
<#950 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWTHVSJAD4CUGJO2EFDCBDZB6BLHAVCNFSM6AAAAABHOY3CXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBWGI4DMNJUGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Ok zoom it is, |
Hey @miguelriemoliveira @manuelgitgomes I've been running a lot of experiments on the terminal trying to figure this out. The problem of possibly being on the "flat" part of the curve which lead to seeing apparently static results in plots such as the RGB calibration results was confirmed. Thing is I'm getting odd behaviors. When running this command ( x and y as placeholders for nig/ntvf values respectively) clear && rosrun atom_calibration calibrate \
-json $ATOM_DATASETS/softbot/long_train_dataset1/dataset_corrected.json \
-v \
-ss 1 -nig x x \
-ntfv y y \
-ntfl "world:base_footprint" \
-csf 'lambda x: int(x) in [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 17, 18]' \
-ctgt \
-atsf "lambda x : x not in []" \
-ftol 1e-4 -xtol 1e-4 -gtol 1e-4
I'm getting this table for y(ntfv) = 0 and x(nig) varying : Not calibrating odometry here
It's a weak assumption to say it converges to all feasible nig values. This behavior is very odd to me. We won't see any plot other than either a horizontal line for the Adding a fixed noise to odometry and varying again the sensor pose noise (nig), I get a consistent similar behavior : Calibrating odometry here
Note: This experiment isn't the same as the plot we discussed in the morning varying ntfv for a fixed nig, here I am varying nig for a given ntfv value, I'll talk about this case in the next sectionVarying ntfv for a given nig = 0.3 : Calibrating odometry here
Here the limit it more visible. At ntfv = 0.7'ish for nig = 0.3 the results start to deteorate quickly. The residuals paint a similar story. For different values of nig other than 0.3, does the optimizer also break at ntfv = 0.7?Once hitting the threshold, the error curves seem to skyrocket.
I do see that the data with higher nig does reach the ntfv point where the optimizer can't handle it anymore earlier. That's the expected. However I expected this behavior change to be more linear and less abrupt. To make the point that for a certain value of My worriesI could eventually use a logarithmic scale on my plots but that would require a lot more data in the threshold zone. I would need increments like 0.77-0.772-0.774....0.79 to actually have the possibility of seeing something. That would take a absurd amount of data and time to make meaningful conclusions. I could also just make the assumption that for all feasible noise values the optimizer can deal with the problem. I'm not sure what do do |
I will run a batch with
Will post the plots later. |
Going step by step through your comments:
Well, the ctgt is increasing as it should . The residuals are overfitting, I would not worry about those.
I don't think I agree with this comparison, its not apples to apples because before you were not calibrating odometry and now you are. In any case, also here I see nothing wrong. CTGT is increasing as it should.
also nothing strange here
Not sure you have any grounds to be expecting that. It could be abrupt or progressive, there is really not way of saying one is wrong and the other correct.
This is still a bit strange to me, but remember we are seeing residuals, and those are not reliable.
Too complicated.
Right, do not do it.
I think the graph with ctgt values would make sense. The conclusion to draw from these is that we should not trust the residuals, only the ctgt or, if the do not have ground truth, the evaluations (hopefully). |
I mean, they do increase as they should. But they increase so little that it makes me unconfident. Yep I know any past 2pi is "useless", I was really just exhausting possibilities because everything seemed to converge.
It is increasing? It starts at 0.013 and finished at 0.010. From 0.013 went down twice then up thrice to 0.010.
Here I do agree, for a fixed nig (while calibrating odom), increasing ntfv increases CTGT as expected, regardless of the residuals being overfitted. This is a great conclusion.
I really have none to be fair. The model is a black box and I was expecting some linearity due to having some of said linearity for lower values. It was incorrect.
I agree. |
Update on this :
I ran this batch. There was a odd error on the LiDAR evaluation so I took it off ( will create a issue for it --> #957 ). The main things to remark are:
Starting with (1)Note for all the plots below : the Y label should've said "Error (m)" that was my bad.Calibrating OdometryMore nig more error -> Makes total sense More odom noise (ntfv) more noise, up until the point where the optimizer can't handle it anymore, also makes sense Not calibrating OdometryWithout the extra noise on the odom the optimizer can take a lot more nig noise. For all the data points on this plot the calibration results are great. Even for huge noises it wouldn't fail. It failed only at 20m/rad nig, which is outrageous and unrealistic. The oscillation is amplified by the y axis limits, they only vary about 1mm. Here the results seem a little more odd, even though it also spikes upwards at 0.7. I will elaborate more on the next section. Here the error is one entire order of magnitude. I would argue that from the point that the error is way bigger than the starting point, the optimizer is already lost and the results aren't reliable. I would ask @manuelgitgomes to help me on a more scientific explanation for this. We discussed it today but I'm failing to have the right words to properly explain it really. Optimizer spiking at 0.7If we zoom on the second plot of the previous section : We can see that generally the higher degree noise curves spike quicker. I'm not sure this is even a useful conclusion or not. But what I find rather odd is that every curve spikes roughly at the same point (ntfv = 0.7m/rad). I don't have a clear explanation for this. The 'ctgt' tables printed at the end of the calibration always seemed to match. There is something that I will correct for the final version though. Rn the ctgt is saving a file with only the averages. This is a little problematic for 2 reasons. For one there is a anchored sensor with error 0 and there is also the odom error weighing in so I'm not exactly getting the final error of each sensor. Example table to clarify my point.
Final resultsAdding up the calibrations with and without odom, this will easily take 2 to 3 days to compute on my pc. Also I'm not sure whether to do something before this big batch. I wanted it ideally to be the final one to focus the rest on the time on the writting. |
I think there is a bug with the nig or something. This very well defined limit suggest it.
Right, it would be nice to show separately for each sensor.
My suggestion. Wait a bit and lets have the meeting tomorrow, then talk on Friday. Great job by the way... |
Problems
When calibrating the softbot system without any perturbances, we are getting 1px errors instead of something near 0.
Adding noise is also unreliable. Adding more noise adds error until a certain threshold that makes the error magically disappear.
What was tried already :
atom/atom_calibration/scripts/calibrate
Lines 247 to 250 in be9f9bf
What I need to test
-ssf
, might be something to do with the LiDARIf those don't work
I 'll post here a branch with a bagfile + dataset to make a MWE (minimum working example for other people to help out)
Tagging @miguelriemoliveira @manuelgitgomes for visibility
Updates
Using just the RGB cameras like in RRbot doens't help either, leading to believe it has to be a faulty dataset or a bug in
calibrate
Table -> Calibration results with the previous simpler dataset used on atom_examples
The text was updated successfully, but these errors were encountered: