Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Laser test #84

Merged
merged 36 commits into from
Apr 25, 2024
Merged

Laser test #84

Changes from 15 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
3bc1bf2
Reading of uncalibrated reports from Magic lidar and implementation o…
nzywucka Nov 23, 2023
fbd80b1
Removed print.
nzywucka Nov 23, 2023
7a97565
In __init__.py, more parameters were added to ReportLaserContainer() …
nzywucka Nov 29, 2023
3398c84
__init__.py changed regarding Laser test #84 comments in ReportLaser…
nzywucka Dec 4, 2023
867aa73
Unnecessary spaces are removed from the file.
nzywucka Feb 26, 2024
aad168a
Unnecessary spaces are removed
nzywucka Feb 26, 2024
9cebd79
Unnecessary spaces removed
nzywucka Feb 26, 2024
74d5c62
Typo removed
nzywucka Feb 26, 2024
8bc6487
If statement for self.laser = self.parse_laser_info() added
nzywucka Feb 26, 2024
e0c1118
Typo removed
nzywucka Feb 26, 2024
7e0252f
BGSamples parameter added
nzywucka Feb 26, 2024
be6a536
The [0]s have been removed from the arrays while reading the Lidar re…
nzywucka Mar 12, 2024
6c424c8
Unindent that does not match any outer indentation level has been cor…
nzywucka Mar 12, 2024
f5e1669
Filtering for the unique reports has been corrected.
nzywucka Mar 12, 2024
27d8a8f
Unused variable has been removed.
nzywucka Mar 12, 2024
4135a5e
Reading of the unnecesary parameters from the lidar reports are removed.
nzywucka Mar 12, 2024
7d6e7ce
Unused import has been removed
nzywucka Mar 12, 2024
ec4c4b7
Trailing whitespace has been removed
nzywucka Mar 12, 2024
2b2dd17
Reading the unique reports has been improved to keep all the unique p…
nzywucka Mar 12, 2024
08b5101
The definitions of the container parameters have been improved.
nzywucka Mar 12, 2024
f1024fa
Appending the unique parameters to the container was improved.
nzywucka Mar 12, 2024
4625c5d
AttributeError has been fixed.
nzywucka Mar 13, 2024
102aa85
Merge remote-tracking branch 'origin/master' into laser_test
nzywucka Mar 15, 2024
bb14523
Filtering of lidar parameters has been improved.
nzywucka Mar 25, 2024
bcaaef2
Filtering of lidar parameters has been improved
nzywucka Mar 27, 2024
1864e76
test_lidar_parameters was removed
nzywucka Mar 27, 2024
61f4a75
Implementation of some tests of reading lidar parameters
nzywucka Mar 27, 2024
94efd5e
test_lidar_parameters test has been improved.
nzywucka Mar 27, 2024
d46221f
Unnecessary comments and whitespaces have been removed.
nzywucka Mar 27, 2024
fadf6d4
Reading BadReport and State parameters added.
nzywucka Mar 28, 2024
fce7374
Reading unique reports changed.
nzywucka Apr 2, 2024
25136f0
Remove assert from tests in codacy
nzywucka Apr 2, 2024
9f14bb5
Skipping assert checks in the test files.
nzywucka Apr 2, 2024
20091c7
Skipping assert checks in the test files.
nzywucka Apr 2, 2024
4f8597c
Tests for reading lidar parameters have been removed.
nzywucka Apr 3, 2024
399be3b
MJD loop is put outside the try .. except constuction
nzywucka Apr 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 234 additions & 1 deletion ctapipe_io_magic/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,16 @@
import uproot
import logging
import numpy as np
from numpy import nan
from pathlib import Path
from decimal import Decimal
from astropy import units as u
from astropy.time import Time
from astropy.coordinates import Angle
from pkg_resources import resource_filename

from ctapipe.io import EventSource, DataLevel
from ctapipe.core import Provenance
from ctapipe.core import Provenance, Container, Field
from ctapipe.core.traits import Bool, UseEnum
from ctapipe.coordinates import CameraFrame

Expand Down Expand Up @@ -80,6 +82,75 @@
DATA_MAGIC_LST_TRIGGER: EventType.SUBARRAY,
}

class ReportLaserContainer(Container):
""" Container for Magic laser parameters """
UniqueID = Field(None, 'No.')
Bits = Field(None, 'ID')
MJD = Field(None, 'Modified Julian Date')
nzywucka marked this conversation as resolved.
Show resolved Hide resolved
BadReport = Field(bool, 'Bad Report')
State = Field(None, 'State')
IsOffsetCorrection = Field(bool, 'Is Offset Correction')
IsOffsetFitted = Field(bool, 'Is Offset Fitted')
IsBGCorrection = Field(bool, 'Is BG Correction')
IsT0ShiftFitted = Field(bool, 'Is T0 Shift Fitted')
IsUseGDAS = Field(bool, 'Is Use GDAS')
IsUpwardMoving = Field(bool, 'Is Upward Moving')
OverShoot = Field(nan, 'Over Shoot')
UnderShoot = Field(nan, 'Under Shoot')
BGSamples = Field(np.float32(np.nan), 'BG Samples')
nzywucka marked this conversation as resolved.
Show resolved Hide resolved
Transmission3km = Field(np.float32(np.nan), 'Transmission at 3 km')
Transmission6km = Field(np.float32(np.nan), 'Transmission at 6 km')
Transmission9km = Field(np.float32(np.nan), 'Transmission at 9 km')
Transmission12km = Field(np.float32(np.nan), 'Transmission at 12 km')
Zenith = Field(Angle(np.nan, u.deg), 'Zenith angle', unit=u.deg)
Azimuth = Field(Angle(np.nan, u.deg), 'Azimuth angle', unit=u.deg)
CloudFWHM = Field(np.float32(np.nan), 'Cloud FWHM')
jsitarek marked this conversation as resolved.
Show resolved Hide resolved
CloudBase = Field(np.float32(np.nan), 'Cloud Base')
CloudTop = Field(np.float32(np.nan), 'Cloud Top')
CloudTrans = Field(np.float32(np.nan), 'Cloud Trans')
CloudHM = Field(np.float32(np.nan), 'Cloud HM')
CloudHStd = Field(np.float32(np.nan), 'Cloud HStd')
CloudLR = Field(np.float32(np.nan), 'Cloud LR')
FullOverlap = Field(np.float32(np.nan), 'Full Overlap')
EndGroundLayer = Field(np.float32(np.nan), 'End Ground Layer')
GroundLayerTrans = Field(np.float32(np.nan), 'Ground Layer Trans')
Calimaness = Field(np.float32(np.nan), 'Calimaness')
CloudLayerAlt = Field(np.float32(np.nan), 'Altitude of cloud layer')
CloudLayerDens = Field(np.float32(np.nan), 'Density of cloud layer')
Klett_k = Field(np.float32(np.nan), 'Klett k')
PheCounts = Field([], 'Phe Counts')
Offset = Field(np.float32(np.nan), 'Offset')
Offset_Calculated = Field(np.float32(np.nan), 'Offset calculated')
Offset_Fitted = Field(np.float32(np.nan), 'Offset fitted')
Offset2 = Field(np.float32(np.nan), 'Offset 2')
Offset3 = Field(np.float32(np.nan), 'Offset 3')
Background1 = Field(np.float32(np.nan), 'Background 1')
Background2 = Field(np.float32(np.nan), 'Background 2')
BackgroundErr1 = Field(np.float32(np.nan), 'Background error 1')
BackgroundErr2 = Field(np.float32(np.nan), 'Background error 2')
RangeMax = Field(np.float32(np.nan), 'Range max')
RangeMax_Clouds = Field(np.float32(np.nan), 'Range max clouds')
ErrorCode = Field(None, 'Error code')
ScaleHeight_fit = Field(np.float32(np.nan), 'Scale Height fit')
Alpha_fit = Field(np.float32(np.nan), 'Alpha fit')
Chi2Alpha_fit = Field(np.float32(np.nan), 'Chi2 Alpha fit')
Alpha_firstY = Field(np.float32(np.nan), 'Alpha first Y')
Alpha_Junge = Field(np.float32(np.nan), 'Alpha Junge')
PBLHeight = Field(np.float32(np.nan), 'PBL Height')
Chi2Full_fit = Field(np.float32(np.nan), 'Chi2 Full fit')
SignalSamples = Field(np.float32(np.nan), 'Signal Samples')
HWSwitch = Field(np.float32(np.nan), 'HW Switch')
HWSwitchMaxOffset = Field(np.float32(np.nan), 'HW Switch Max Offset')
NCollapse = Field(np.float32(np.nan), 'N Collapse')
Shots = Field(np.float32(np.nan), 'Shots')
T0Shift = Field(np.float32(np.nan), 'T0 Shift')
Interval_0 = Field(np.float32(np.nan), 'Interval 0')
RCS_min_perfect = Field(np.float32(np.nan), 'RCS min perfect')
RCS_min_clouds = Field(np.float32(np.nan), 'RCS min cloud')
RCS_min_mol = Field(np.float32(np.nan), 'RCS min mol')
LIDAR_ratio = Field(np.float32(np.nan), 'LIDAR ratio')
LIDAR_ratio_Cloud = Field(np.float32(np.nan), 'LIDAR ratio cloud')
LIDAR_ratio_Junge = Field(np.float32(np.nan), 'LIDAR ratio Junge')

def load_camera_geometry():
"""Load camera geometry from bundled resources of this repo"""
Expand Down Expand Up @@ -206,6 +277,9 @@ def __init__(self, input_url=None, config=None, parent=None, **kwargs):

self.metadata = self.parse_metadata_info()

if not self.is_simulation:
self.laser = self.parse_laser_info()

# Retrieving the data level (so far HARDCODED Sorcerer)
self.datalevel = DataLevel.DL0

Expand Down Expand Up @@ -889,6 +963,165 @@ def parse_metadata_info(self):

return metadata

def parse_laser_info(self):
laser_info_array_list_runh = [
'MReportLaser.MReport.fUniqueID',
'MReportLaser.MReport.fBits',
'MTimeLaser.fMjd',
'MTimeLaser.fTime.fMilliSec',
nzywucka marked this conversation as resolved.
Show resolved Hide resolved
'MReportLaser.MReport.fBadReport',
'MReportLaser.MReport.fState',
'MReportLaser.fIsOffsetCorrection',
'MReportLaser.fIsOffsetFitted',
'MReportLaser.fIsBGCorrection',
'MReportLaser.fIsT0ShiftFitted',
'MReportLaser.fIsUseGDAS',
'MReportLaser.fIsUpwardMoving',
'MReportLaser.fOverShoot',
'MReportLaser.fUnderShoot',
'MReportLaser.fBGSamples',
'MReportLaser.fTransmission3km',
'MReportLaser.fTransmission6km',
'MReportLaser.fTransmission9km',
'MReportLaser.fTransmission12km',
'MReportLaser.fZenith',
'MReportLaser.fAzimuth',
'MReportLaser.fCloudFWHM[10]',
jsitarek marked this conversation as resolved.
Show resolved Hide resolved
'MReportLaser.fCloudBase[10]',
'MReportLaser.fCloudTop[10]',
'MReportLaser.fCloudTrans[10]',
'MReportLaser.fCloudHM[10]',
'MReportLaser.fCloudHStd[10]',
'MReportLaser.fCloudLR[10]',
'MReportLaser.fFullOverlap',
'MReportLaser.fEndGroundLayer',
'MReportLaser.fGroundLayerTrans',
'MReportLaser.fKlett_k',
'MReportLaser.fPheCounts',
'MReportLaser.fCalimaness',
'MReportLaser.fCloudLayerAlt',
'MReportLaser.fCloudLayerDens',
'MReportLaser.fOffset',
'MReportLaser.fOffset_Calculated',
'MReportLaser.fOffset_Fitted',
'MReportLaser.fOffset2',
'MReportLaser.fOffset3',
'MReportLaser.fBackground1',
'MReportLaser.fBackground2',
'MReportLaser.fBackgroundErr1',
'MReportLaser.fBackgroundErr2',
'MReportLaser.fRangeMax',
'MReportLaser.fRangeMax_Clouds',
'MReportLaser.fErrorCode',
'MReportLaser.fScaleHeight_fit',
'MReportLaser.fChi2Alpha_fit',
'MReportLaser.fChi2Alpha_fit',
'MReportLaser.fAlpha_firstY',
'MReportLaser.fAlpha_Junge',
'MReportLaser.fPBLHeight',
'MReportLaser.fChi2Full_fit',
'MReportLaser.fHWSwitchMaxOffset',
'MReportLaser.fNCollapse',
'MReportLaser.fShots',
'MReportLaser.fT0Shift',
'MReportLaser.fInterval_0',
'MReportLaser.fRCS_min_perfect',
'MReportLaser.fRCS_min_clouds',
'MReportLaser.fRCS_min_mol',
'MReportLaser.fLIDAR_ratio',
'MReportLaser.fLIDAR_ratio_Cloud',
'MReportLaser.fLIDAR_ratio_Junge',
]

laser = ReportLaserContainer()
unique_reports = {}
for rootf in self.files_:
nzywucka marked this conversation as resolved.
Show resolved Hide resolved
try:
aleberti marked this conversation as resolved.
Show resolved Hide resolved
laser_info_runh = rootf['Laser'].arrays(
laser_info_array_list_runh, library="np"
)
laser.UniqueID = laser_info_runh['MReportLaser.MReport.fUniqueID']
laser.Bits = laser_info_runh['MReportLaser.MReport.fBits']
mjd_value = laser_info_runh['MTimeLaser.fMjd']
millisec_value = laser_info_runh['MTimeLaser.fTime.fMilliSec']
laser.BadReport = bool(laser_info_runh['MReportLaser.MReport.fBadReport'])
laser.State = laser_info_runh['MReportLaser.MReport.fState']
laser.IsOffsetCorrection = bool(laser_info_runh['MReportLaser.fIsOffsetCorrection'])
laser.IsOffsetFitted = bool(laser_info_runh['MReportLaser.fIsOffsetFitted'])
laser.IsBGCorrection = bool(laser_info_runh['MReportLaser.fIsBGCorrection'])
laser.IsT0ShiftFitted = bool(laser_info_runh['MReportLaser.fIsT0ShiftFitted'])
laser.IsUseGDAS = bool(laser_info_runh['MReportLaser.fIsUseGDAS'])
laser.IsUpwardMoving = bool(laser_info_runh['MReportLaser.fIsUpwardMoving'])
laser.OverShoot = int(laser_info_runh['MReportLaser.fOverShoot'])
laser.UnderShoot = int(laser_info_runh['MReportLaser.fUnderShoot'])
laser.BGSamples = int(laser_info_runh['MReportLaser.fBGSamples'])
laser.Transmission3km = laser_info_runh['MReportLaser.fTransmission3km']
laser.Transmission6km = laser_info_runh['MReportLaser.fTransmission6km']
laser.Transmission9km = laser_info_runh['MReportLaser.fTransmission9km']
laser.Transmission12km = laser_info_runh['MReportLaser.fTransmission12km']
laser.Zenith = laser_info_runh['MReportLaser.fZenith']* u.deg
laser.Azimuth = laser_info_runh['MReportLaser.fAzimuth']* u.deg
laser.CloudFWHM = laser_info_runh['MReportLaser.fCloudFWHM[10]']
laser.CloudBase = laser_info_runh['MReportLaser.fCloudBase[10]']
laser.CloudTop = laser_info_runh['MReportLaser.fCloudTop[10]']
laser.CloudTrans = laser_info_runh['MReportLaser.fCloudTrans[10]']
laser.CloudHM = laser_info_runh['MReportLaser.fCloudHM[10]']
laser.CloudHStd = laser_info_runh['MReportLaser.fCloudHStd[10]']
laser.CloudLR = laser_info_runh['MReportLaser.fCloudLR[10]']
laser.FullOverlap = laser_info_runh['MReportLaser.fFullOverlap']
laser.EndGroundLayer = laser_info_runh['MReportLaser.fEndGroundLayer']
laser.GroundLayerTrans = laser_info_runh['MReportLaser.fGroundLayerTrans']
laser.Klett_k = laser_info_runh['MReportLaser.fKlett_k']
laser.PheCounts = laser_info_runh['MReportLaser.fPheCounts']
laser.Calimaness = laser_info_runh['MReportLaser.fCalimaness']
laser.CloudLayerAlt = laser_info_runh['MReportLaser.fCloudLayerAlt']
laser.CloudLayerDens = laser_info_runh['MReportLaser.fCloudLayerDens']
laser.Offset = laser_info_runh['MReportLaser.fOffset']
laser.Offset_Calculated = laser_info_runh['MReportLaser.fOffset_Calculated']
laser.Offset_Fitted = laser_info_runh['MReportLaser.fOffset_Fitted']
laser.Offset2 = laser_info_runh['MReportLaser.fOffset2']
laser.Offset3 = laser_info_runh['MReportLaser.fOffset3']
laser.Background1 = laser_info_runh['MReportLaser.fBackground1']
laser.Background2 = laser_info_runh['MReportLaser.fBackground2']
laser.BackgroundErr1 = laser_info_runh['MReportLaser.fBackgroundErr1']
laser.BackgroundErr2 = laser_info_runh['MReportLaser.fBackgroundErr2']
laser.RangeMax = laser_info_runh['MReportLaser.fRangeMax']
laser.RangeMax_Clouds = laser_info_runh['MReportLaser.fRangeMax_Clouds']
laser.ErrorCode = laser_info_runh['MReportLaser.fErrorCode']
laser.ScaleHeight_fit = laser_info_runh['MReportLaser.fScaleHeight_fit']
laser.Alpha_fit = laser_info_runh['MReportLaser.fChi2Alpha_fit']
laser.Chi2Alpha_fit = laser_info_runh['MReportLaser.fChi2Alpha_fit']
laser.Alpha_firstY = laser_info_runh['MReportLaser.fAlpha_firstY']
laser.Alpha_Junge = laser_info_runh['MReportLaser.fAlpha_Junge']
laser.PBLHeight = laser_info_runh['MReportLaser.fPBLHeight']
laser.Chi2Full_fit = laser_info_runh['MReportLaser.fChi2Full_fit']
laser.HWSwitchMaxOffset = laser_info_runh['MReportLaser.fHWSwitchMaxOffset']
laser.NCollapse = laser_info_runh['MReportLaser.fNCollapse']
laser.Shots = laser_info_runh['MReportLaser.fShots']
laser.T0Shift = laser_info_runh['MReportLaser.fT0Shift']
laser.Interval_0 = laser_info_runh['MReportLaser.fInterval_0']
laser.RCS_min_perfect = laser_info_runh['MReportLaser.fRCS_min_perfect']
laser.RCS_min_clouds = laser_info_runh['MReportLaser.fRCS_min_clouds']
laser.RCS_min_mol = laser_info_runh['MReportLaser.fRCS_min_mol']
laser.LIDAR_ratio = laser_info_runh['MReportLaser.fLIDAR_ratio']
laser.LIDAR_ratio_Cloud = laser_info_runh['MReportLaser.fLIDAR_ratio_Cloud']
laser.LIDAR_ratio_Junge = laser_info_runh['MReportLaser.fLIDAR_ratio_Junge']

mjd_values = laser_info_runh['MTimeLaser.fMjd']
millisec_values = laser_info_runh['MTimeLaser.fTime.fMilliSec']
for mjd_value, millisec_value in zip(mjd_values, millisec_values):
millisec_seconds = millisec_value * 1e-3
combined_mjd_value = mjd_value + millisec_seconds / 86400
if (mjd_value, millisec_value) not in unique_reports:
unique_reports[(mjd_value, millisec_value)] = combined_mjd_value
except KeyError as e:
print(f"Required key not found in the file {rootf}: {e}")
continue
for _ , value in unique_reports.items():
nzywucka marked this conversation as resolved.
Show resolved Hide resolved
laser.MJD = value

return laser

def parse_simulation_header(self):
"""
Parse the simulation information from the RunHeaders tree.
Expand Down
Loading