Chunk interpolation to select calibration data #2634

ctoennis · 2024-10-31T11:27:37Z

I will need a method to select calibration data for the strar tracker. I made some slides to decribe how it is supposed to work:
https://docs.google.com/presentation/d/1oxIcYSQvGnU7IQYy3fGdcv0qXiLpvaXR9YtmnDesj4Y/edit?usp=sharing

src/ctapipe/monitoring/interpolation.py

…apipe into ChunkFunction

ctao-dpps-sonarqube · 2024-10-31T12:20:01Z

Analysis Details

1 Issue

0 Bugs
0 Vulnerabilities
1 Code Smell

Coverage and Duplications

98.00% Coverage (94.30% Estimated after merge)
0.00% Duplicated Code (0.70% Estimated after merge)

Project ID: cta-observatory_ctapipe_AY52EYhuvuGcMFidNyUs

View in SonarQube

ctoennis · 2024-11-10T12:08:53Z

@maxnoe @kosack Can you have another look if there is something else to be changed?

mexanick · 2024-11-18T15:52:28Z

@kosack @maxnoe this PR is needed to complete the pointing calibration (for the variance calibration application), can we advance it?

maxnoe · 2024-11-18T15:56:43Z

src/ctapipe/monitoring/interpolation.py

+
+                return np.nan
+
+            self._interpolators[tel_id][column] = interpolate_chunk


I think it's a bit weird here to store closures in a dict instead of just calling a method with tel_id and column that then checks if the necessary table exists.

ok, i'll change it to a method

The current implementation follows the PointingInterpolator approach, but of course can be changed.

so, _check_interpolators in the parent class assumes the current implementation. If i change it to what you describe i will need to make changes to the parent class or forgo this check for the ChunkInterpolator

Maybe keeping it like this is better.

👍 Looks good, the __call__ method is missing on the interface though

I am not sure if putting the private methods you listed here to the base class will work out in the end, but you'll see how much code can be shared between the chunk interpolator and the linear interpolator.

ok, i'll work on the implementation now.

Shall we take an opportunity and stop using interp1d? It is legacy, andscipy suggests this: https://docs.scipy.org/doc/scipy/tutorial/interpolate/1D.html#tutorial-interpolate-1dsection

Also, if we call a base class LinearInterpolator, I guess we should enforce linearity (in the example above, e.g. cubic spline can be applied)

There is no replacement for linear 1d interpolation for interp1d. It is legacy, but not deprecated.

See scipy/scipy#21258

I also got tricked by "may be removed in the future"

ctoennis · 2024-11-22T13:38:49Z

I am a bit stuck here with one of the tests. test_hdf5 is failing in pytest begause the data from the file is not loaded correctly. When i look in the test what x and y values the interpolators have i get some wrong values. However if i try to do the same outside of pytest it works. I used this code to test by myself:

import astropy.units as u
import numpy as np
import tables
from astropy.table import Table
from astropy.time import Time

from functools import partial
from ctapipe.core import Component, traits

from ctapipe.monitoring.interpolation import PointingInterpolator

from ctapipe.io import write_table

t0 = Time("2022-01-01T00:00:00")

table = Table(
    {"time": t0 + np.arange(0.0, 10.1, 2.0) * u.s, "azimuth": np.radians(np.linspace(0.0, 10.0, 6)) * u.rad, "altitude": np.radians(>)

path = "pointing.h5"

write_table(table, path, "/dl0/monitoring/telescope/pointing/tel_001")

with tables.open_file(path) as h5file:
    interpolator = PointingInterpolator(h5file)
    t = t0 + 1 * u.s
    alt, az = interpolator(tel_id=1, time=t)
    print(interpolator._interpolators[1]["alt"].y,interpolator._interpolators[1]["alt"].x)
    print(alt,az)

Has anyone an idea what is wrong here?

maxnoe · 2024-11-22T14:31:39Z

src/ctapipe/monitoring/tests/test_interpolator.py

@@ -57,7 +173,7 @@ def test_invalid_input():
    wrong_unit = Table(
        {
            "time": Time(1.7e9 + np.arange(3), format="unix"),
-            "azimuth": [1, 2, 3] * u.deg,
+            "azimuth": np.radians([1, 2, 3]) * u.rad,


Why did you change these tests here?

The test is explicitly to test the unit conversion, so please don't change to rad

The PointingInterpolator is supposed to use radians for input:

157 expected_units = {"azimuth": u.rad, "altitude": u.rad}

right, and this test checks that correct exception is raised in case the units are not matching. Please revert back

maxnoe · 2024-11-22T14:32:14Z

src/ctapipe/monitoring/tests/test_interpolator.py

@@ -72,8 +188,8 @@ def test_hdf5(tmp_path):
    table = Table(
        {
            "time": t0 + np.arange(0.0, 10.1, 2.0) * u.s,
-            "azimuth": np.linspace(0.0, 10.0, 6) * u.deg,


maxnoe · 2024-11-22T14:32:23Z

src/ctapipe/monitoring/tests/test_interpolator.py

@@ -82,8 +198,8 @@ def test_hdf5(tmp_path):
    with tables.open_file(path) as h5file:
        interpolator = PointingInterpolator(h5file)
        alt, az = interpolator(tel_id=1, time=t0 + 1 * u.s)
-        assert u.isclose(alt, 69 * u.deg)
-        assert u.isclose(az, 1 * u.deg)
+        assert u.isclose(alt, np.radians(69) * u.rad)


mexanick · 2024-11-25T13:35:33Z

@ctoennis the problem you're experiencing with the test is because you have class attributes in the base class that are shared between the class instances, including the _interpolators.

ctoennis · 2024-11-25T13:57:49Z

@mexanick Thanks, i got the issue fixed.

Christoph Toennis and others added 8 commits October 29, 2024 11:32

Adding the chunk function

3d879b9

Reverting to the previous implementation

c2ce42c

Changing some variables

c8f7ef0

Chaning to using scipy functions

bef9677

fixing docustring

cd4864f

Updating docustrings further

fb3a9a4

simplifying chunk interpolation

0f0b948

Refactor ChunkInterpolator and its tests

50f2791

ctoennis requested review from maxnoe and kosack October 31, 2024 11:27

ctoennis added the new functionality label Oct 31, 2024

ctoennis self-assigned this Oct 31, 2024

maxnoe reviewed Oct 31, 2024

View reviewed changes

src/ctapipe/monitoring/interpolation.py Outdated Show resolved Hide resolved

add back base Interpolator to __all__

90115d6

This comment has been minimized.

Sign in to view

Christoph Toennis added 3 commits October 31, 2024 12:55

adding changelog

f772b19

Merge branch 'ChunkFunction' of https://github.com/cta-observatory/ct…

e201356

…apipe into ChunkFunction

renaming changelog

907b6cd

This comment has been minimized.

Sign in to view

ctoennis requested a review from maxnoe October 31, 2024 13:30

ctoennis mentioned this pull request Nov 1, 2024

Handling and applying calibration parameters #2635

Open

mexanick added the calibration label Nov 4, 2024

maxnoe reviewed Nov 18, 2024

View reviewed changes

Changing inheritance scheme

dc3b24c

maxnoe reviewed Nov 22, 2024

View reviewed changes

Christoph Toennis added 2 commits November 22, 2024 15:51

reverting some tests

3c7e6ab

documentation change

cad523e

fixing issue with class variable

4a032ed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chunk interpolation to select calibration data #2634

Chunk interpolation to select calibration data #2634

ctoennis commented Oct 31, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ctao-dpps-sonarqube bot commented Oct 31, 2024

ctoennis commented Nov 10, 2024

mexanick commented Nov 18, 2024

maxnoe Nov 18, 2024

ctoennis Nov 19, 2024

mexanick Nov 19, 2024

ctoennis Nov 19, 2024

ctoennis Nov 19, 2024

maxnoe Nov 21, 2024 •

edited

Loading

ctoennis Nov 21, 2024

mexanick Nov 21, 2024

maxnoe Nov 21, 2024

mexanick Nov 21, 2024

ctoennis commented Nov 22, 2024

maxnoe Nov 22, 2024

maxnoe Nov 22, 2024

ctoennis Nov 22, 2024 •

edited

Loading

mexanick Nov 22, 2024

maxnoe Nov 22, 2024

maxnoe Nov 22, 2024

mexanick commented Nov 25, 2024

ctoennis commented Nov 25, 2024


		return np.nan

		self._interpolators[tel_id][column] = interpolate_chunk

Chunk interpolation to select calibration data #2634

Are you sure you want to change the base?

Chunk interpolation to select calibration data #2634

Conversation

ctoennis commented Oct 31, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ctao-dpps-sonarqube bot commented Oct 31, 2024

Analysis Details

1 Issue

Coverage and Duplications

ctoennis commented Nov 10, 2024

mexanick commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxnoe Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctoennis commented Nov 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctoennis Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mexanick commented Nov 25, 2024

ctoennis commented Nov 25, 2024

maxnoe Nov 21, 2024 •

edited

Loading

ctoennis Nov 22, 2024 •

edited

Loading