Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Measurement consumer simulator only works with sampling rate = 1.0 #1743

Open
ple13 opened this issue Aug 8, 2024 · 0 comments
Open

Measurement consumer simulator only works with sampling rate = 1.0 #1743

ple13 opened this issue Aug 8, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@ple13
Copy link
Contributor

ple13 commented Aug 8, 2024

Describe the bug
The measurement consumer simulator gets the measurement results from the MPC systems, then compares them with the ground truth. The ground truth is computed based on the sampled vids instead of the full set. When the sampling rate is not 1.0, the computed ground truth is incorrect.

Steps to reproduce

  1. Update the measurement specs here or here with sampling width interval = 0.27.
  2. Run the accuracy tests and read the log. Search for "round=" to see the discrepancy between estimated reach and actual reach.

Component(s) affected
Integration and accuracy tests.

Version
commit e34610d

Environment
Local testing.

Additional context
Sampling Interval Width = 0.27
Aug 08, 2024 4:27:26 PM org.wfanet.measurement.integration.common.InProcessReachMeasurementAccuracyTest$reach-only hmss results should be accurate with respect to the variance$1 invokeSuspend
INFO: round=1, actual_result=9559, expected_result=1605, interval=(-1412.78, 20530.78), accurate=true

Sampling Interval Width = 1.0
Aug 08, 2024 6:45:06 PM org.wfanet.measurement.integration.common.InProcessReachMeasurementAccuracyTest$reach-only hmss results should be accurate with respect to the variance$1 invokeSuspend
INFO: round=1, actual_result=5446, expected_result=6000, interval=(2484.72, 8407.28), accurate=true

@ple13 ple13 added the bug Something isn't working label Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant