Example of Hybrid Point-Line Localization for own dataset

[Gotta-C]

how to locate from a image or convert a npy like runners/tests/localization_test_data_stairs_1.npy?

[MarkYu98]

Hi, generally you don't need a .npy file like that, it is only prepared so it's easier to run the test.

The main functionality of point-line localization is implemented in limap.estimators.absolute_pose.pl_estimate_absolute_pose() function, which takes the 2D-3D point and line correspondences, the camera model, and returns the estimated absolute camera pose.

The limap.runners.line_localization runner is a wrapper around the pl_estimate_absolute_pose() which first does line detection, extraction, and matching on a collection of images (limap.base.ImageCollection) and then calls pl_estimate_absolute_pose() to localize each query images.

You might also find it helpful to have a glance at the localization scripts for datasets: runners/7scenes/localization.py or runners/cambridge/localization.py, they implement the entire pipeline of point detection, matching, triangulation (using hloc), line detection, matching, and triangulation using LIMAP, and finally do the visual localization for the dataset.

[Gotta-C]

Hi, generally you don't need a .npy file like that, it is only prepared so it's easier to run the test.

The main functionality of point-line localization is implemented in limap.estimators.absolute_pose.pl_estimate_absolute_pose() function, which takes the 2D-3D point and line correspondences, the camera model, and returns the estimated absolute camera pose.

The limap.runners.line_localization runner is a wrapper around the pl_estimate_absolute_pose() which first does line detection, extraction, and matching on a collection of images (limap.base.ImageCollection) and then calls pl_estimate_absolute_pose() to localize each query images.

You might also find it helpful to have a glance at the localization scripts for datasets: runners/7scenes/localization.py or runners/cambridge/localization.py, they implement the entire pipeline of point detection, matching, triangulation (using hloc), line detection, matching, and triangulation using LIMAP, and finally do the visual localization for the dataset.

thanks and I have noticed runners/7scenes/localization.py before and set about to generate own pipeline. And when will you release the documentation that will be great help.

[Gotta-C]

how to get 'depth/7scenes_{args.scene}/train/depth' and '7scenes_densevlad_retrieval_top_10' in runners/7scenes/localization.py

[MarkYu98]

Hi, you could get the rendered depth map and retrieval for 7Scenes following instructions here: https://github.com/cvg/Hierarchical-Localization/tree/master/hloc/pipelines/7Scenes

Sorry for the inconvenience, this info will be included in the upcoming documentations.

[Gotta-C]

it was successed with the retrieval txt and 7scenes_sfm_triangulated models in commit 8db2128

but got error after update a848a58
Start multi-view triangulation...
0%| | 0/2000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/workspace/limap/runners/7scenes/localization.py", line 164, in
main()
File "/workspace/limap/runners/7scenes/localization.py", line 119, in main
linetracks_db = runners.line_triangulation(cfg, imagecols_train, neighbors=neighbors, ranges=ranges)
File "/workspace/limap/limap/runners/line_triangulation.py", line 104, in line_triangulation
matches = limapio.read_npy(os.path.join(matches_dir, "matches{0}.npy".format(img_id))).item()
ValueError: can only convert an array of size 1 to a Python scalar

[B1ueber2y]

Hi. The update from yesterday changed the format of the matching output. Maybe you could try to remove the existing matching folder and re-run the matching?

Thanks!

[Gotta-C]

Hi. The update from yesterday changed the format of the matching output. Maybe you could try to remove the existing matching folder and re-run the matching?

Thanks!

it did not help so I roll-back.I have tried stairs scene and the points result 0.052m/1.473deg/46.80% 5cm, 5deg which is same in paper, but points+lines（sold2）result 0.038m /1.048deg/67.60% 5cm, 5deg, compared to table 16, the trans rot recall was worse, where is the difference?

[MarkYu98]

Hi, do you still have error on the read_npy call with the latest commit on main branch? You'll need to delete the line matchings folder e.g. tmp/7scenes/stairs/line_matchings to use the updated matching output format.

Regarding the p+l localization on Stairs, the best results we achieved were by setting localization.optimize.normalize_weight to True, and using TrivialLoss instead of HuberLoss for localization.optimize.loss_func in the configs (and [] for localization.optimize.loss_func_args), you could try setting them in cfgs/localization/7scenes.yaml or change these settings using command line arguments (e.g. --localization.optimize.loss_func TrivialLoss).

[Gotta-C]

Hi, do you still have error on the read_npy call with the latest commit on main branch? You'll need to delete the line matchings folder e.g. tmp/7scenes/stairs/line_matchings to use the updated matching output format.

Regarding the p+l localization on Stairs, the best results we achieved were by setting localization.optimize.normalize_weight to True, and using TrivialLoss instead of HuberLoss for localization.optimize.loss_func in the configs (and [] for localization.optimize.loss_func_args), you could try setting them in cfgs/localization/7scenes.yaml or change these settings using command line arguments (e.g. --localization.optimize.loss_func TrivialLoss).

Thanks, seems much better. the points+lines result 0.038m /1.027deg/69.40% 5cm, 5deg compare to 0.037m /1.02deg/71.10% 5cm, 5deg in paper.
In my owndata, mainly some indoor scene like rooms and office areas，the points localization is slightly better than points+lines. I haven't try many experiment and adjust hyper-parameter, but seems the scene can be used need contains much more lines which the points can‘t express repetitive texture areas like 7scenes stairs, in another six scenes the improvement is not obvious. Maybe usefull for weak texture areas like walls, floor, glass windows.
In addition to the open source dataset, have you try some big scenes like hundreds square, the Prior line constraint can improve localization effectively by using points map. And some mainly parameters to adjust for desired results.