Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add hota iou calculation along with detection/association accuracy #183

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

Rusteam
Copy link

@Rusteam Rusteam commented Oct 10, 2023

Closes #151

Following the author's blogpost to add HOTA calculation to this package.

Added hota_iou implementation only for a single selected iou threshold. Need some guidance on how to implement HOTA at range(0.05, 0.95, 0.05)

@cheind
Copy link
Owner

cheind commented Oct 10, 2023

Hi, thanks a lot for your contribution!

@cheind
Copy link
Owner

cheind commented Oct 10, 2023

I wonder if you could also add a HOTA metric to this

def test_motchallenge_files():

test? I guess we could derive the necessary GT-values by running the files through the official implementation?

@Rusteam
Copy link
Author

Rusteam commented Oct 11, 2023

I wonder if you could also add a HOTA metric to this

def test_motchallenge_files():

test? I guess we could derive the necessary GT-values by running the files through the official implementation?

should it be added to the default motchallenge metrics?

@cheind
Copy link
Owner

cheind commented Oct 11, 2023

you mean to add it to mm.metrics.motchallenge_metrics?

I believe one could make a separate list of metrics for HOTA? such that we can write

summary = mh.compute_many(
        accs,
        metrics=mm.metrics.motchallenge_metrics + mm.metrics.hota_metrics,
        names=dnames,
        generate_overall=True,
    )

Alternatively, based on https://motchallenge.net/, it seems like MotChallenge is already reporting HOTA. So we could just add the metrics to the mm.metrics.motchallenge_metric list. What do you think?

@Rusteam
Copy link
Author

Rusteam commented Oct 11, 2023

you mean to add it to mm.metrics.motchallenge_metrics?

I believe one could make a separate list of metrics for HOTA? such that we can write

summary = mh.compute_many(
        accs,
        metrics=mm.metrics.motchallenge_metrics + mm.metrics.hota_metrics,
        names=dnames,
        generate_overall=True,
    )

Alternatively, based on https://motchallenge.net/, it seems like MotChallenge is already reporting HOTA. So we could just add the metrics to the mm.metrics.motchallenge_metric list. What do you think?

Since it's already included in the challenge, I can create hota_metrics with these 3 values and include it by default to motchallenge_metrics, however, it will be possible to use it standalone as well.

Also I need some hint how to implement the actual HOTA which is an average of hota_iou across a range of iou thresholds.

@Rusteam
Copy link
Author

Rusteam commented Oct 11, 2023

These are results from the official repo on the same test files:

image

@cheind
Copy link
Owner

cheind commented Oct 12, 2023

Also I need some hint how to implement the actual HOTA which is an average of hota_iou across a range of iou thresholds.

I hopefully can have a look at that in the next week. Do you know if its possible pre-compute some values about Det(alpha) Ass(alpha), so that we don't have to run the entire computation for every alpha?

@Rusteam
Copy link
Author

Rusteam commented Oct 12, 2023

I hopefully can have a look at that in the next week. Do you know if its possible pre-compute some values about Det(alpha) Ass(alpha), so that we don't have to run the entire computation for every alpha?

I doubt because all of them are basically TP, FP and FN which are computed at a specified threshold.

I've noticed that my results of DetA and AssA are different from the orig. repo, need to look more into that.

@mikel-brostrom
Copy link

mikel-brostrom commented Nov 10, 2023

Really looking forward to start using py-motmetrics instead of trackeval for full MOTA, HOTA and IDF1 metrics! 🚀
This will make a huge impact in the accessibility and flexibility when it comes to this specific metric. Using trackeval is TEDIOUS.

@Rusteam
Copy link
Author

Rusteam commented Nov 10, 2023

I have followed the author's blog post in my implementation but the numbers are different from the original repository. Just need to find time to figure out where the catch is and how to compute a metric at multiple IOU thresholds in this package.

@mikel-brostrom
Copy link

mikel-brostrom commented Nov 15, 2023

If it is of any help, here you have the original HOTA implementation @Rusteam :

https://github.com/JonathonLuiten/TrackEval/blob/12c8791b303e0a0b50f753af204249e622d0281a/trackeval/metrics/hota.py#L9

@Rusteam
Copy link
Author

Rusteam commented Nov 15, 2023

Thanks. Yes I've seen it but it requires time to find the difference

@Justin900429
Copy link
Contributor

Justin900429 commented Dec 15, 2023

Hi,

I have tried combining different $\alpha$ by:

import os

import numpy as np

import motmetrics as mm

"""Tests metrics for sequences TUD-Campus and TUD-Stadtmitte."""
dnames = [
    "TUD-Stadtmitte",
]

DATA_DIR = "motmetrics/data"


def compute_motchallenge(dname):
    df_gt = mm.io.loadtxt(os.path.join(dname, "gt.txt"))
    df_test = mm.io.loadtxt(os.path.join(dname, "test.txt"))

    res_file = []

    for th in np.arange(0.05, 0.99, 0.05):
        res_file.append(mm.utils.compare_to_groundtruth(df_gt, df_test, "iou", distth=th))
    return res_file


accs = [compute_motchallenge(os.path.join(DATA_DIR, d)) for d in dnames]
det = []
mh = mm.metrics.create()

for idx in range(len(accs[0])):
    summary = mh.compute_many(
        [accs[0][idx]],
        metrics=["det_acc"],
        names=[dnames[0]],
        generate_overall=False,
    )
    det.append(float(summary["det_acc"].iloc[0]))

print(sum(det) / len(det))

and found the DeT is different: 39.227 (official) v.s. 39.87 (this) in TUD-Stadtmitte. I guess this different is caused by the reweighting before filtering and association implemented in the official repo. Check this line.


Also I need some hint how to implement the actual HOTA which is an average of hota_iou across a range of iou thresholds.

I hopefully can have a look at that in the next week. Do you know if its possible pre-compute some values about Det(alpha) Ass(alpha), so that we don't have to run the entire computation for every alpha?

Computing the iou_matrix (of dist) once and use different $alpha$ for filtering might speed up just like the official does. This require to re-write some of the functions (such as mm.utils.compare_to_groundtruth) to support this kind of operation.


Based on these reasons, maybe we can do the HOTA computation with a indepdent function without changing the current codebase?

@Justin900429 Justin900429 mentioned this pull request Dec 16, 2023
3 tasks
@cheind
Copy link
Owner

cheind commented Dec 18, 2023

@Rusteam , @Justin900429 how does your PR relate to this one?

@Rusteam
Copy link
Author

Rusteam commented Dec 18, 2023

He is right. There's some difference. I wonder how to fix it

@johnnynunez
Copy link

any news?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

About HOTA
5 participants