Improvement to spatial flexure #1870

debora-pe · 2024-11-13T22:46:59Z

This work makes improvements to the spatial flexure calculation and add a QA plot to check the flexure.
Documentation was also added.

It addresses issue #1843

Dev-Suite PR pypeit/PypeIt-development-suite#339

rcooke-ast

Thanks @debora-pe - Overall this looks good, but it will be good to make sure the dev suite passes. Also, it would be good to check what happens when the slit edges are quite diagonal. I also have a few general questions:

(1) Could you mention the reason why this new algorithm is required?
(2) My main concern is that the spectral smashing might smear out the slit edges. Is there are reason that the ross-correlation needs to be done in 1D instead of 2D (along one dimension)?
(3) I quite like that the cross-correlation is done on the sobel images with this algorithm. Would it also work to use the old algorithm, but instead use the sobel edges? This would avoid the smashing.

rcooke-ast · 2024-11-19T10:10:32Z

pypeit/core/flexure.py

+    sci_sobel, _ = trace.detect_slit_edges(_sciimg, bpm=bpm)
+    slits_sobel, _ = trace.detect_slit_edges(slitmask, bpm=bpm)
+    # collapse both sobel images along the spectral direction
+    sci_smash, _, _ = sigma_clipped_stats(sci_sobel, axis=0, mask=bpm)


Does this work well in cases where the slit edges are significantly tilted wrt detector columns?

Yes, it does seems to work well. I was worried too at first, but if the slits edges are tilted for the trace image they will be tilted also for the science image.

rcooke-ast · 2024-11-19T10:11:38Z

pypeit/core/flexure.py

+    sci_smash, _, _ = sigma_clipped_stats(sci_sobel, axis=0, mask=bpm)
+    slits_smash, _, _ = sigma_clipped_stats(slits_sobel, axis=0, mask=bpm)
+    # remove nan values
+    sci_smash[np.isnan(sci_smash)] = 0


Is it possible to remove the nan values from the 2D image, instead (before smashing)?

The nan values are actually created by sigma_clipped_stats when encounters the masked values. This is why I added it here.

rcooke-ast · 2024-11-19T10:12:55Z

pypeit/core/flexure.py

+    sci_smash[np.isnan(sci_smash)] = 0
+    slits_smash[np.isnan(slits_smash)] = 0
+    # invert the negative values
+    sci_smash[sci_smash < 0] *= -1


Is this needed? I think keeping the left edges and right edges as +ve/-ve might actually help the cross-correlation.

I tested this a lot and it seems that inverting the negative value is still better than keeping them negative. In many cases it does not make much difference, but in some cases the cross-correlation is better with the inversion.

rcooke-ast · 2024-11-19T10:16:56Z

pypeit/core/flexure.py

+
+    """
+    debug = True if outfile is None else False
+    # TODO: should we use initial or tweaked slits in this plot?


I think it's the tweaked slits that you're after. As you noted above on line 85, whatever is used here needs to be consistent with that choice. I think once the tweaked slits are available, they get used.

yes, I believe that is correct.

Well in the code above I'm still using the initial slits (not the tweaked ones) since:
slitmask = slits.slit_img(initial=True, exclude_flag=slits.bitmask.exclude_for_flexure)
It seems to me that the initial ones should work better in the cross-correlation, and therefore this plot should be consistent with that. But I do not want users to get confused when seeing this plot and comparing with what is actually use for reduction, i.e., tweaked slits if tweak_slits=True

rcooke-ast · 2024-11-19T10:40:12Z

pypeit/core/flexure.py


+
+    """


Add a check to make sure that vrange is a tuple (otherwise warn and set to None?)

Done. Thanks!

profxj

a few minor comments from me.

am really happy to see this code advanced.

doc/figures/r230417_01033_DET01_spat_flex_corr.png

pypeit/core/arc.py

profxj · 2024-11-20T03:12:02Z

pypeit/core/flexure.py

+    sci_smash[np.isnan(sci_smash)] = 0
+    slits_smash[np.isnan(slits_smash)] = 0
+    # invert the negative values
+    sci_smash[sci_smash < 0] *= -1


profxj · 2024-11-20T03:14:05Z

pypeit/core/flexure.py

-    #tslits_shift = trace_slits.shift_slits(tslits_dict, lag_max)
-    # Now translate the tilts
+        # 2D plot
+        spat_flexure_qa(sciimg, slits, shift, gpm=np.logical_not(bpm), vrange=qa_vrange)


remove this line? it is duplicated below.
or is the idea to generate 2 PNGs?

No, this one is used only in debug mode and plot in a matplot window the whole image. No PNG is created. I uses the same script, but it does slightly different things.

profxj · 2024-11-20T03:14:50Z

pypeit/core/flexure.py

+
+    """
+    debug = True if outfile is None else False
+    # TODO: should we use initial or tweaked slits in this plot?


yes, I believe that is correct.

profxj · 2024-11-20T03:19:34Z

pypeit/images/combineimage.py

+        if np.sum(no_nan) > 0:
+            if np.any(np.absolute(np.diff(spat_flex[no_nan])) > 0.1):
+                msgs.warn(f'Spatial flexure is not consistent for all images being combined: {spat_flex}.')
+                comb_spat_flex = np.round(np.mean(spat_flex[no_nan]),3)


is there a preference to rounding to the nearest pixel instead of staying a float ?

It is only rounding to the 3rd decimal. It's still a float. But the mean calculation can make it to be a very long number.

pypeit/images/rawimage.py

debora-pe

@rcooke-ast @profxj I have addressed your comments.
I have extensively tested these changes with all the Keck LRIS data in the Dev Suite and all looks good. I will anyway re-run the whole Dev Suite.

@rcooke-ast Here are my answers to your questions:

(1) Could you mention the reason why this new algorithm is required?

The old algorithm was not working well. A peak in the cross-correlation was often not found and the spatial flexure was not corrected.

(2) My main concern is that the spectral smashing might smear out the slit edges. Is there are reason that the ross-correlation needs to be done in 1D instead of 2D (along one dimension)?

As I mentioned in another comment, I was little worried too, but after a lot of testing (also with more tilted slit edges) the correction looks reasonable. I thought about a 2D cross-correlation but I wasn't able to find a way to do it without adding a lot of runtime (it's probably me that I don't know how to deal with it)

(3) I quite like that the cross-correlation is done on the sobel images with this algorithm. Would it also work to use the old algorithm, but instead use the sobel edges? This would avoid the smashing.

I have also tested a lot using the sobel edges, and the correction is not consistently good.

debora-pe · 2024-11-23T00:01:27Z

pypeit/core/flexure.py

+    sci_sobel, _ = trace.detect_slit_edges(_sciimg, bpm=bpm)
+    slits_sobel, _ = trace.detect_slit_edges(slitmask, bpm=bpm)
+    # collapse both sobel images along the spectral direction
+    sci_smash, _, _ = sigma_clipped_stats(sci_sobel, axis=0, mask=bpm)


Yes, it does seems to work well. I was worried too at first, but if the slits edges are tilted for the trace image they will be tilted also for the science image.

debora-pe · 2024-11-23T00:03:12Z

pypeit/core/flexure.py

+    sci_smash[np.isnan(sci_smash)] = 0
+    slits_smash[np.isnan(slits_smash)] = 0
+    # invert the negative values
+    sci_smash[sci_smash < 0] *= -1


I tested this a lot and it seems that inverting the negative value is still better than keeping them negative. In many cases it does not make much difference, but in some cases the cross-correlation is better with the inversion.

debora-pe · 2024-11-23T00:05:13Z

pypeit/core/flexure.py

-    #tslits_shift = trace_slits.shift_slits(tslits_dict, lag_max)
-    # Now translate the tilts
+        # 2D plot
+        spat_flexure_qa(sciimg, slits, shift, gpm=np.logical_not(bpm), vrange=qa_vrange)


No, this one is used only in debug mode and plot in a matplot window the whole image. No PNG is created. I uses the same script, but it does slightly different things.

debora-pe · 2024-11-23T00:05:44Z

pypeit/core/flexure.py


+
+    """


Done. Thanks!

debora-pe · 2024-11-23T00:13:13Z

pypeit/core/flexure.py

+
+    """
+    debug = True if outfile is None else False
+    # TODO: should we use initial or tweaked slits in this plot?


Well in the code above I'm still using the initial slits (not the tweaked ones) since:
slitmask = slits.slit_img(initial=True, exclude_flag=slits.bitmask.exclude_for_flexure)
It seems to me that the initial ones should work better in the cross-correlation, and therefore this plot should be consistent with that. But I do not want users to get confused when seeing this plot and comparing with what is actually use for reduction, i.e., tweaked slits if tweak_slits=True

debora-pe · 2024-11-23T00:15:18Z

pypeit/images/combineimage.py

+        if np.sum(no_nan) > 0:
+            if np.any(np.absolute(np.diff(spat_flex[no_nan])) > 0.1):
+                msgs.warn(f'Spatial flexure is not consistent for all images being combined: {spat_flex}.')
+                comb_spat_flex = np.round(np.mean(spat_flex[no_nan]),3)


It is only rounding to the 3rd decimal. It's still a float. But the mean calculation can make it to be a very long number.

debora-pe · 2024-11-23T00:18:54Z

pypeit/core/flexure.py

+    sci_smash, _, _ = sigma_clipped_stats(sci_sobel, axis=0, mask=bpm)
+    slits_smash, _, _ = sigma_clipped_stats(slits_sobel, axis=0, mask=bpm)
+    # remove nan values
+    sci_smash[np.isnan(sci_smash)] = 0


The nan values are actually created by sigma_clipped_stats when encounters the masked values. This is why I added it here.

debora-pe added 17 commits September 16, 2024 14:41

still nowhere

0e52afa

Merge branch 'develop' into spatflex_improve

aca3e44

final version?

a5c44b3

some stuff for a QA plot

b49a79a

Merge branch 'develop' into spatflex_improve

f06bbd8

finalized QA plot

0a029a8

small fix

c84e3e7

small fix

e526c1e

improved

a447a20

improved

dd06e5c

small fixes

d286930

docs

5cc3f32

docs

3245c8c

Merge branch 'develop' into spatflex_improve

617e550

Merge branch 'release' into spatflex_improve

d82cd5f

added to release doc

3f3237d

forgot one parameters

40cd1d4

debora-pe requested review from profxj and rcooke-ast November 15, 2024 23:29

rcooke-ast changed the base branch from develop to dev2rel November 16, 2024 21:05

Base automatically changed from dev2rel to develop November 18, 2024 16:28

Merge branch 'develop' into spatflex_improve

8ff7bda

rcooke-ast requested changes Nov 19, 2024

View reviewed changes

found a bug an fixed it

8cf8948

debora-pe mentioned this pull request Nov 20, 2024

Spatflex improve tests pypeit/PypeIt-development-suite#339

Open

fix the fix

9f05c66

profxj approved these changes Nov 20, 2024

View reviewed changes

debora-pe added 3 commits November 21, 2024 17:20

small improv

da9e7a1

Merge branch 'develop' into spatflex_improve

80676b4

small fix

279975d

debora-pe commented Nov 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvement to spatial flexure #1870

Improvement to spatial flexure #1870

debora-pe commented Nov 13, 2024 •

edited

Loading

rcooke-ast left a comment

rcooke-ast Nov 19, 2024

debora-pe Nov 23, 2024

rcooke-ast Nov 19, 2024

debora-pe Nov 23, 2024

rcooke-ast Nov 19, 2024

profxj Nov 20, 2024

debora-pe Nov 23, 2024

rcooke-ast Nov 19, 2024

profxj Nov 20, 2024

debora-pe Nov 23, 2024

rcooke-ast Nov 19, 2024

debora-pe Nov 23, 2024

profxj left a comment

profxj Nov 20, 2024

profxj Nov 20, 2024

debora-pe Nov 23, 2024

profxj Nov 20, 2024

profxj Nov 20, 2024

debora-pe Nov 23, 2024

debora-pe left a comment

debora-pe Nov 23, 2024

debora-pe Nov 23, 2024

debora-pe Nov 23, 2024

debora-pe Nov 23, 2024

debora-pe Nov 23, 2024

debora-pe Nov 23, 2024

debora-pe Nov 23, 2024

Improvement to spatial flexure #1870

Are you sure you want to change the base?

Improvement to spatial flexure #1870

Conversation

debora-pe commented Nov 13, 2024 • edited Loading

rcooke-ast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

profxj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

debora-pe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

debora-pe commented Nov 13, 2024 •

edited

Loading