Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Stage 3 NIRCam row-by-row background subtraction is broken #719

Open
1 task done
mithun1996-png opened this issue Nov 11, 2024 · 12 comments
Open
1 task done
Assignees
Labels
bug Something isn't working NIRCam

Comments

@mithun1996-png
Copy link

mithun1996-png commented Nov 11, 2024

Instrument

NIRCam (Stages 1-3)

What happened?

as i am using NIRCam data , so row by row background subtraction is very important , so when i set rorw _by_ror_bg to True , then this type of error is here .

Error traceback output

Starting Stage 3 Reduction

Input directory: /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/output_stage_2/S2_2024-11-05_nircam_wfss_template_run1/
  Found 25 data file(s) ending in calints.fits
Output directory: /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/stage_3/S3_2024-11-11_nircam_wfss_template_run9/ap7_bg11/
Using ap=7, bg=11, expand=1
Copying S3 control file
Starting file 1 of 25
  Reading file 1...
  Performing RxR background subtraction...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 756/756 [00:06<00:00, 125.71it/s]
  Creating figures for background subtraction...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:03<00:00,  1.29it/s]
  Masking NaNs/infs in data arrays...
    FLUX has 54416880 NaNs/infs, which is 100.00% of all pixels.
  WARNING: Your region of interest may be off the edge of the detector subarray.  Masking NaN/inf regions and continuing, but you should really stop and reconsider your choices.
    ERR has 54416880 NaNs/infs, which is 100.00% of all pixels.
  WARNING: Your region of interest may be off the edge of the detector subarray.  Masking NaN/inf regions and continuing, but you should really stop and reconsider your choices.
    V0 has 54416880 NaNs/infs, which is 100.00% of all pixels.
  WARNING: Your region of interest may be off the edge of the detector subarray.  Masking NaN/inf regions and continuing, but you should really stop and reconsider your choices.
  Locating source position...
Traceback (most recent call last):
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/demos/JWST/run_eureka.py", line 27, in <module>
    spec, meta = s3.reduce(eventlabel, ecf_path=ecf_path)
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/s3_reduce.py", line 350, in reduce
    source_pos.source_pos_wrapper(data, meta, log, m)
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/source_pos.py", line 101, in source_pos_wrapper
    meta.src_ypos = source_pos(flux[integ], meta, data.attrs['shdr'],
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/source_pos.py", line 162, in source_pos
    src_ypos, src_ywidth = source_pos_gauss(flux, meta, m, n, plot)
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/source_pos.py", line 410, in source_pos_gauss
    p0 = [np.ma.max(med_row), pos_max, sigma0, np.ma.median(med_row)]
  File "/home/seps3/miniconda3/envs/eureka/lib/python3.10/site-packages/numpy/ma/core.py", line 6795, in max
    return obj.max(axis=axis, fill_value=fill_value, out=out, **kwargs)
  File "/home/seps3/miniconda3/envs/eureka/lib/python3.10/site-packages/numpy/ma/core.py", line 5922, in max
    result = self.filled(fill_value).max(
  File "/home/seps3/miniconda3/envs/eureka/lib/python3.10/site-packages/numpy/core/_methods.py", line 41, in _amax
    return umr_maximum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation maximum which has no identity

What operating system are you using?

linux

What version of Python are you running?

python3.10.14

What Python packages do you have installed?

# packages in environment at /home/seps3/miniconda3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aext-assistant            4.0.15          py312h06a4308_jl4_0  
aext-assistant-server     4.0.15          py312h06a4308_0  
aext-core                 4.0.15          py312h06a4308_jl4_0  
aext-core-server          4.0.15          py312h06a4308_1  
aext-panels               4.0.15          py312h06a4308_0  
aext-panels-server        4.0.15          py312h06a4308_0  
aext-share-notebook       4.0.15          py312h06a4308_0  
aext-share-notebook-server 4.0.15          py312h06a4308_0  
aext-shared               4.0.15          py312h06a4308_0  
alabaster                 0.7.16          py312h06a4308_0  
anaconda-anon-usage       0.4.4           py312hfc0e8ea_100  
anaconda-cloud-auth       0.5.1           py312h06a4308_0  
anaconda-toolbox          4.0.15          py312h06a4308_0  
annotated-types           0.6.0           py312h06a4308_0  
anyio                     4.2.0           py312h06a4308_0  
archspec                  0.2.3              pyhd3eb1b0_0  
argon2-cffi               21.3.0             pyhd3eb1b0_0  
argon2-cffi-bindings      21.2.0          py312h5eee18b_0  
arrow                     1.2.3           py312h06a4308_1  
asdf                      3.5.0                    pypi_0    pypi
asdf-astropy              0.6.1                    pypi_0    pypi
asdf-coordinates-schemas  0.3.0                    pypi_0    pypi
asdf-standard             1.1.1                    pypi_0    pypi
asdf-transform-schemas    0.5.0                    pypi_0    pypi
asdf-wcs-schemas          0.4.0                    pypi_0    pypi
astroid                   2.14.2          py312h06a4308_0  
astropy                   6.1.2                    pypi_0    pypi
astropy-iers-data         0.2024.8.12.0.32.58          pypi_0    pypi
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
async-lru                 2.0.4           py312h06a4308_0  
atomicwrites              1.4.0                      py_0  
attrs                     23.1.0          py312h06a4308_0  
autopep8                  2.0.4              pyhd3eb1b0_0  
babel                     2.11.0          py312h06a4308_0  
bayesicfitting            3.2.1                    pypi_0    pypi
beautifulsoup4            4.12.3          py312h06a4308_0  
binaryornot               0.4.4              pyhd3eb1b0_1  
black                     24.4.2          py312h06a4308_0  
blas                      1.0                    openblas  
bleach                    4.1.0              pyhd3eb1b0_0  
boltons                   23.0.0          py312h06a4308_0  
bottleneck                1.3.7           py312ha883a20_0  
brotli-python             1.0.9           py312h6a678d5_8  
bzip2                     1.0.8                h5eee18b_6  
c-ares                    1.19.1               h5eee18b_0  
ca-certificates           2024.7.4             hbcca054_0    conda-forge
certifi                   2024.8.30       py312h06a4308_0  
cffi                      1.16.0          py312h5eee18b_1  
chardet                   4.0.0           py312h06a4308_1003  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.1.7           py312h06a4308_0  
cloudpickle               3.0.0           py312h06a4308_0  
colorama                  0.4.6           py312h06a4308_0  
comm                      0.2.2              pyhd8ed1ab_0    conda-forge
conda                     24.7.1          py312h06a4308_0  
conda-content-trust       0.2.0           py312h06a4308_1  
conda-libmamba-solver     24.1.0             pyhd3eb1b0_0  
conda-package-handling    2.2.0           py312h06a4308_1  
conda-package-streaming   0.9.0           py312h06a4308_0  
contourpy                 1.2.1                    pypi_0    pypi
cookiecutter              2.6.0           py312h06a4308_0  
crds                      12.0.4                   pypi_0    pypi
cryptography              42.0.5          py312hdda0065_1  
cycler                    0.12.1                   pypi_0    pypi
cyrus-sasl                2.1.28               h52b45da_1  
dbus                      1.13.18              hb2f20db_0  
debugpy                   1.6.7           py312h6a678d5_0  
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd3eb1b0_0  
diff-match-patch          20200713           pyhd3eb1b0_0  
dill                      0.3.8           py312h06a4308_0  
distro                    1.9.0           py312h06a4308_0  
docstring-to-markdown     0.11            py312h06a4308_0  
docutils                  0.18.1          py312h06a4308_3  
drizzle                   2.0.0                    pypi_0    pypi
exceptiongroup            1.2.2              pyhd8ed1ab_0    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
expat                     2.6.2                h6a678d5_0  
filelock                  3.16.1                   pypi_0    pypi
flake8                    7.0.0           py312h06a4308_0  
fmt                       9.1.0                hdb19cb5_1  
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonttools                 4.53.0                   pypi_0    pypi
freetype                  2.12.1               h4a9f257_0  
frozendict                2.4.2           py312h06a4308_0  
future                    1.0.0                    pypi_0    pypi
glib                      2.78.4               h6a678d5_0  
glib-tools                2.78.4               h6a678d5_0  
gst-plugins-base          1.14.1               h6a678d5_1  
gstreamer                 1.14.1               h5eee18b_1  
gwcs                      0.21.0                   pypi_0    pypi
h11                       0.14.0          py312h06a4308_0  
h5py                      3.11.0          py312h34c39bb_0  
hdf5                      1.12.1               h2b7332f_3  
httpcore                  1.0.2           py312h06a4308_0  
httpx                     0.26.0          py312h06a4308_0  
icu                       73.1                 h6a678d5_0  
idna                      3.7             py312h06a4308_0  
imageio                   2.36.0                   pypi_0    pypi
imagesize                 1.4.1           py312h06a4308_0  
importlib-metadata        8.2.0              pyha770c72_0    conda-forge
importlib_metadata        8.2.0                hd8ed1ab_0    conda-forge
inflection                0.5.1           py312h06a4308_1  
intervaltree              3.1.0              pyhd3eb1b0_0  
ipykernel                 6.29.5             pyh3099207_0    conda-forge
ipython                   8.26.0             pyh707e725_0    conda-forge
isort                     5.13.2          py312h06a4308_0  
jaraco.classes            3.2.1              pyhd3eb1b0_0  
jax                       0.4.23          py312h06a4308_0  
jaxlib                    0.4.23          py312h6a678d5_0  
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jeepney                   0.7.1              pyhd3eb1b0_0  
jellyfish                 1.0.1           py312hb02cf49_0  
jinja2                    3.1.4           py312h06a4308_0  
jmespath                  1.0.1                    pypi_0    pypi
jpeg                      9e                   h5eee18b_2  
json5                     0.9.6              pyhd3eb1b0_0  
jsonpatch                 1.33            py312h06a4308_1  
jsonpointer               2.1                pyhd3eb1b0_0  
jsonschema                4.19.2          py312h06a4308_0  
jsonschema-specifications 2023.7.1        py312h06a4308_0  
jupyter-lsp               2.2.0           py312h06a4308_0  
jupyter_client            8.6.2              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.2           py312h06a4308_0  
jupyter_events            0.10.0          py312h06a4308_0  
jupyter_server            2.14.1          py312h06a4308_0  
jupyter_server_terminals  0.4.4           py312h06a4308_1  
jupyterlab                4.0.11          py312h06a4308_0  
jupyterlab-pygments       0.2.2                    pypi_0    pypi
jupyterlab_pygments       0.1.2                      py_0  
jupyterlab_server         2.25.1          py312h06a4308_0  
jwst                      1.16.0                   pypi_0    pypi
keyring                   24.3.1          py312h06a4308_0  
kiwisolver                1.4.5                    pypi_0    pypi
krb5                      1.20.1               h143b758_1  
lazy-loader               0.4                      pypi_0    pypi
lazy-object-proxy         1.10.0          py312h5eee18b_0  
ld_impl_linux-64          2.38                 h1181459_1  
libarchive                3.6.2                h6ac8c49_3  
libclang                  14.0.6          default_hc6dbbc7_1  
libclang13                14.0.6          default_he11475f_1  
libcups                   2.4.2                h2d74bed_1  
libcurl                   8.7.1                h251f7ec_0  
libedit                   3.1.20230828         h5eee18b_0  
libev                     4.33                 h7f8727e_1  
libevent                  2.1.12               hdbd6064_1  
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.4                h6a678d5_1  
libgcc-ng                 14.1.0               h77fa898_0    conda-forge
libgfortran-ng            14.1.0               h69a702a_0    conda-forge
libgfortran5              14.1.0               hc5f4f2c_0    conda-forge
libglib                   2.78.4               hdc74915_0  
libgomp                   14.1.0               h77fa898_0    conda-forge
libiconv                  1.16                 h5eee18b_3  
libllvm14                 14.0.6               hdb19cb5_3  
libmamba                  1.5.8                hfe524e5_2  
libmambapy                1.5.8           py312h2dafd23_2  
libnghttp2                1.57.0               h2d74bed_0  
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.21               h043d6bf_0  
libpng                    1.6.39               h5eee18b_0  
libpq                     12.17                hdbd6064_0  
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsolv                   0.7.24               he621ea3_1  
libspatialindex           1.9.3                h2531618_0  
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libssh2                   1.11.0               h251f7ec_0  
libstdcxx-ng              14.1.0               hc0a3c3a_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcb                    1.15                 h7f8727e_0  
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxkbcommon              1.0.1                h5eee18b_1  
libxml2                   2.10.4               hfdd30dd_2  
libzlib                   1.2.13               h4ab18f5_6    conda-forge
llvmlite                  0.43.0                   pypi_0    pypi
lz4-c                     1.9.4                h6a678d5_1  
markdown-it-py            2.2.0           py312h06a4308_1  
markupsafe                2.1.3           py312h5eee18b_0  
matplotlib                3.9.0                    pypi_0    pypi
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge
mccabe                    0.7.0              pyhd3eb1b0_0  
mdurl                     0.1.0           py312h06a4308_0  
menuinst                  2.0.2           py312h06a4308_1  
mistune                   2.0.4           py312h06a4308_0  
ml_dtypes                 0.4.0           py312h526ad5a_0  
more-itertools            10.1.0          py312h06a4308_0  
mpi                       1.0                       mpich  
mpi4py                    4.0.0           py312h5924d30_1    conda-forge
mpich                     4.2.2              h4a7f18d_101    conda-forge
mypy_extensions           1.0.0           py312h06a4308_0  
mysql                     5.7.24               h721c034_2  
nbclient                  0.8.0           py312h06a4308_0  
nbconvert                 7.10.0          py312h06a4308_0  
nbformat                  5.9.2           py312h06a4308_0  
ncurses                   6.4                  h6a678d5_0  
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
networkx                  3.4.2                    pypi_0    pypi
notebook                  7.0.8           py312h06a4308_2  
notebook-shim             0.2.3           py312h06a4308_0  
nspr                      4.35                 h6a678d5_0  
nss                       3.89.1               h6a678d5_0  
numba                     0.60.0                   pypi_0    pypi
numexpr                   2.8.7           py312he7dcb8a_0  
numpy                     1.26.4          py312h2809609_0  
numpy-base                1.26.4          py312he1a6c75_0  
numpydoc                  1.7.0           py312h06a4308_0  
opencv-python-headless    4.10.0.84                pypi_0    pypi
openssl                   3.3.1                h4bc722e_2    conda-forge
opt_einsum                3.3.0              pyhd3eb1b0_1  
overrides                 7.4.0           py312h06a4308_0  
packaging                 23.2            py312h06a4308_0  
panadas                   0.2                      pypi_0    pypi
pandas                    2.2.2           py312h526ad5a_0  
pandocfilters             1.5.0              pyhd3eb1b0_0  
parsley                   1.3                      pypi_0    pypi
parso                     0.8.4              pyhd8ed1ab_0    conda-forge
pathspec                  0.10.3          py312h06a4308_0  
pcre2                     10.42                hebb0a14_1  
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
photutils                 2.0.2                    pypi_0    pypi
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.3.0                   pypi_0    pypi
pip                       24.0            py312h06a4308_0  
pkce                      1.0.3           py312h06a4308_0  
platformdirs              3.10.0          py312h06a4308_0  
pluggy                    1.0.0           py312h06a4308_1  
ply                       3.11            py312h06a4308_1  
poppy                     1.1.1                    pypi_0    pypi
prometheus_client         0.14.1          py312h06a4308_0  
prompt-toolkit            3.0.47             pyha770c72_0    conda-forge
psutil                    5.9.0           py312h5eee18b_0  
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.3              pyhd8ed1ab_0    conda-forge
pybind11-abi              5                    hd3eb1b0_0  
pycodestyle               2.11.1          py312h06a4308_0  
pycosat                   0.6.6           py312h5eee18b_1  
pycparser                 2.21               pyhd3eb1b0_0  
pydantic                  2.5.3           py312h06a4308_0  
pydantic-core             2.14.6          py312hb02cf49_0  
pydocstyle                6.3.0           py312h06a4308_0  
pyerfa                    2.0.1.4         py312ha883a20_0  
pyflakes                  3.2.0           py312h06a4308_0  
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pyjwt                     2.8.0           py312h06a4308_0  
pylint                    2.16.2          py312h06a4308_0  
pylint-venv               3.0.3           py312h06a4308_0  
pyls-spyder               0.4.0              pyhd3eb1b0_0  
pyparsing                 3.1.2                    pypi_0    pypi
pyqt                      5.15.10         py312h6a678d5_0  
pyqt5-sip                 12.13.0         py312h5eee18b_0  
pyqtwebengine             5.15.10         py312h6a678d5_0  
pysocks                   1.7.1           py312h06a4308_0  
python                    3.12.2          hab00c5b_0_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
python-dotenv             0.21.0          py312h06a4308_0  
python-fastjsonschema     2.16.2          py312h06a4308_0  
python-json-logger        2.0.7           py312h06a4308_0  
python-lsp-black          2.0.0           py312h06a4308_0  
python-lsp-jsonrpc        1.1.2              pyhd3eb1b0_0  
python-lsp-server         1.10.0          py312h06a4308_0  
python-slugify            5.0.2              pyhd3eb1b0_0  
python-tzdata             2023.3             pyhd3eb1b0_0  
python_abi                3.12                    4_cp312    conda-forge
pytoolconfig              1.2.6           py312h06a4308_0  
pytz                      2024.1          py312h06a4308_0  
pyxdg                     0.27               pyhd3eb1b0_0  
pyyaml                    6.0.1           py312h5eee18b_0  
pyzmq                     25.1.2          py312h6a678d5_0  
qdarkstyle                3.2.3              pyhd3eb1b0_0  
qstylizer                 0.2.2           py312h06a4308_0  
qt-main                   5.15.2              h53bd1ea_10  
qt-webengine              5.15.9               h9ab4d14_7  
qtawesome                 1.2.2           py312h06a4308_0  
qtconsole                 5.5.1           py312h06a4308_0  
qtpy                      2.4.1           py312h06a4308_0  
readline                  8.2                  h5eee18b_0  
referencing               0.30.2          py312h06a4308_0  
reproc                    14.2.4               h6a678d5_2  
reproc-cpp                14.2.4               h6a678d5_2  
requests                  2.31.0          py312h06a4308_1  
rfc3339-validator         0.1.4           py312h06a4308_0  
rfc3986-validator         0.1.1           py312h06a4308_0  
rich                      13.7.1          py312h06a4308_0  
rope                      1.12.0          py312h06a4308_0  
rpds-py                   0.10.6          py312hb02cf49_0  
rtree                     1.0.1           py312h06a4308_0  
ruamel.yaml               0.17.21         py312h5eee18b_0  
scikit-image              0.24.0                   pypi_0    pypi
scipy                     1.13.1          py312h2809609_0  
secretstorage             3.3.1           py312h06a4308_1  
semantic-version          2.10.0                   pypi_0    pypi
semver                    3.0.2           py312h06a4308_0  
send2trash                1.8.2           py312h06a4308_0  
setuptools                74.1.2                   pypi_0    pypi
sip                       6.7.12          py312h6a678d5_0  
six                       1.16.0             pyh6c4a22f_0    conda-forge
sniffio                   1.3.0           py312h06a4308_0  
snowballstemmer           2.2.0              pyhd3eb1b0_0  
sortedcontainers          2.4.0              pyhd3eb1b0_0  
soupsieve                 2.5             py312h06a4308_0  
spherical-geometry        1.3.2                    pypi_0    pypi
sphinx                    7.3.7           py312h5eee18b_0  
sphinxcontrib-applehelp   1.0.2              pyhd3eb1b0_0  
sphinxcontrib-devhelp     1.0.2              pyhd3eb1b0_0  
sphinxcontrib-htmlhelp    2.0.0              pyhd3eb1b0_0  
sphinxcontrib-jsmath      1.0.1              pyhd3eb1b0_0  
sphinxcontrib-qthelp      1.0.3              pyhd3eb1b0_0  
sphinxcontrib-serializinghtml 1.1.10          py312h06a4308_0  
spyder                    5.5.1           py312h06a4308_3  
spyder-kernels            2.5.0           py312h06a4308_0  
sqlite                    3.45.3               h5eee18b_0  
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
stcal                     1.9.0                    pypi_0    pypi
stdatamodels              2.1.1                    pypi_0    pypi
stpipe                    0.7.0                    pypi_0    pypi
stsci-imagestats          1.8.3                    pypi_0    pypi
stsci-stimage             0.2.9                    pypi_0    pypi
synphot                   1.4.0                    pypi_0    pypi
tabulate                  0.9.0           py312h06a4308_0  
terminado                 0.17.1          py312h06a4308_0  
text-unidecode            1.3                pyhd3eb1b0_0  
textdistance              4.2.1              pyhd3eb1b0_0  
three-merge               0.1.1              pyhd3eb1b0_0  
tifffile                  2024.9.20                pypi_0    pypi
tinycss2                  1.2.1           py312h06a4308_0  
tk                        8.6.14               h39e8969_0  
tomli                     2.0.1           py312h06a4308_1  
tomlkit                   0.11.1          py312h06a4308_0  
tornado                   6.4.1           py312h5eee18b_0  
tqdm                      4.66.2          py312he106c6f_0  
traitlets                 5.14.3             pyhd8ed1ab_0    conda-forge
truststore                0.8.0           py312h06a4308_0  
tweakwcs                  0.8.9                    pypi_0    pypi
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024a                h04d1e81_0  
ujson                     5.10.0          py312h6a678d5_0  
unidecode                 1.2.0              pyhd3eb1b0_0  
urllib3                   2.1.0           py312h06a4308_1  
watchdog                  4.0.1           py312h06a4308_0  
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
webencodings              0.5.1           py312h06a4308_2  
websocket-client          1.8.0           py312h06a4308_0  
whatthepatch              1.0.2           py312h06a4308_0  
wheel                     0.43.0          py312h06a4308_0  
wiimatch                  0.3.2                    pypi_0    pypi
wrapt                     1.14.1          py312h5eee18b_0  
wurlitzer                 3.0.2           py312h06a4308_0  
xz                        5.4.6                h5eee18b_1  
yaml                      0.2.5                h7b6447c_0  
yaml-cpp                  0.8.0                h6a678d5_1  
yapf                      0.40.2          py312h06a4308_0  
zeromq                    4.3.5                h6a678d5_0  
zipp                      3.19.2             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h4ab18f5_6    conda-forge
zstandard                 0.22.0          py312h2c38b39_0  
zstd                      1.5.5                hc292b87_2  

Code of Conduct

  • I agree to follow this project's Code of Conduct
@jbrande
Copy link
Collaborator

jbrande commented Nov 11, 2024

Can you post your Stage 2 and Stage 3 ECFs? Also, what dataset are you trying to analyze? You may have specified some earlier step incorrectly, as every pixel in the image is marked as bad, so there's no actual data being analyzed after those steps.

@mithun1996-png
Copy link
Author

mithun1996-png commented Nov 12, 2024

Data Specifications (needed to input the MAST JWST Search):
Object name : HD 189733b
Data type : Time series
Instruments : NIRCam
Exposure type : NRC_TSGRISM (Spectrum time series)
Filter : F322W2, F444W
Program (proposal) number : 1633
Principal investigator of the observing program : Deming
Dates of observation : 2022-08-25 (F444W), 2022-08-29 (F322W2)
Total exposure duration : 21,504.2 s (F444W), 21,383.1 s (F322W2) [optional]

# Eureka! Control File for Stage 2: Data Reduction

# Stage 2 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-2

suffix                  rateints     # Data file suffix

# Controls the cross-dispersion extraction
tsgrism_extract_height  None    # Use None to rely on the default parameters

# Note: different instruments and modes will use different steps by default
skip_flat_field         False
skip_photom             True	# Recommended to skip to get better uncertainties out of Stage 3. Set this to False if you want to compute calibrated stellar spectra.
skip_extract_1d         True

# Diagnostics
hide_plots              True    # If True, plots will automatically be closed rather than popping up

# Project directory
topdir                  /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01

# Directories relative to topdir
inputdir                /output_stage_1/
outputdir               /output_stage_2_new/
# Eureka! Control File for Stage 3: Data Reduction

# Stage 3 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-3

ncpu            4           # Number of CPUs
nfiles          1           # The number of data files to analyze simultaneously
max_memory      0.5         # The maximum fraction of memory you want utilized by read-in frames (this will reduce nfiles if need be)
indep_batches   False       # Independently treat each batch of files? Strongly recommended to leave this as False unless you have a clear reason to set it to True.
suffix          calints     # Data file suffix

calibrated_spectra  False   # Set True to generate flux-calibrated spectra/photometry in mJy
                            # Set False to convert to electrons

# Subarray region of interest
ywindow         [5,64]      # Vertical axis as seen in DS9
xwindow         [780,2000]  # Horizontal axis as seen in DS9
src_pos_type    gaussian    # Determine source position when not given in header (Options: header, gaussian, weighted, max, hst, or a numeric value)
record_ypos     True        # Option to record the y position and width for each integration (only records if src_pos_type is gaussian)
poly_wavelength	True		# Use an updated polynomial wavelength solution for NIRCam longwave spectroscopy instead of the linear wavelength solution currently assumed by STScI
dqmask          True        # Mask pixels with an odd entry in the DQ array
expand          1           # Super-sampling factor along cross-dispersion direction

# Outlier rejection along time axis
ff_outlier      True      # Set False to use only background region (recommended for deep transits)
                            # Set True to use full frame (works well for shallow transits/eclipses)
bg_thresh       [10,10]       # Double-iteration X-sigma threshold for outlier rejection along time axis

# Background parameters
bg_hw           11           # Half-width of exclusion region for BG subtraction (relative to source position)
bg_deg          1          # Polynomial order for column-by-column background subtraction, -1 for median of entire frame
bg_method       median        # Options: std (Standard Deviation), median (Median Absolute Deviation), mean (Mean Absolute Deviation)
p3thresh        5           # X-sigma threshold for outlier rejection during background subtraction
bg_row_by_row   True       # Row-by-row BG subtraction (only useful for NIRCam)
bg_x1           0      # Left edge of exclusion region for row-by-row BG subtraction
bg_x2           512    # Right edge of exclusion region for row-by-row BG subtraction

# Spectral extraction parameters
spec_hw         7           # Half-width of aperture region for spectral extraction (relative to source position)
fittype         meddata     # Method for constructing spatial profile (Options: smooth, meddata, poly, gauss, wavelet, or wavelet2D)
median_thresh   10          # Sigma threshold when flagging outliers in median frame, when fittype=meddata and window_len > 1
window_len      7           # Smoothing window length, for median frame or when fittype = smooth or meddata (when computing median frame). Can set to 1 for no smoothing when computing median frame for fittype=meddata.
prof_deg        3           # Polynomial degree, when fittype = poly
p5thresh        10          # X-sigma threshold for outlier rejection while constructing spatial profile
p7thresh        7           # X-sigma threshold for outlier rejection during optimal spectral extraction

# Curvature treatment
curvature       correct     # How to manage the curved trace on the detector (Options: None, correct)

# Diagnostics
isplots_S3      3           # Generate few (1), some (3), or many (5) figures (Options: 1 - 5)
nplots          5           # How many of each type of figure do you want to make per file?
vmin            0.97        # Sets the vmin of the color bar for Figure 3101.
vmax            1.03        # Sets the vmax of the color bar for Figure 3101.
time_axis       'y'         # Determines whether the time axis in Figure 3101 is along the y-axis ('y') or the x-axis ('x')
testing_S3      False       # Boolean, set True to only use last file and generate select figures
hide_plots      True        # If True, plots will automatically be closed rather than popping up
save_output     True        # Save outputs for use in S4
save_fluxdata   Ture      # Save the much larger FluxData.h5 outputs which can be useful for debugging or comparisons between different pipelines
verbose         True        # If True, more details will be printed about steps

# Project directory
topdir        /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01
# Directories relative to topdir
inputdir       /output_stage_2/S2_2024-11-05_nircam_wfss_template_run1/
outputdir      /stage_3/

@kevin218
Copy link
Owner

Does Stage 3 work when you don't run row-by-row background subtraction?

@taylorbell57
Copy link
Collaborator

Are you running F322W2 and F444W observations simultaneously? You can only do one filter at a time, and based on your previous message it seems to me that you're working on both filters at the same time. You'll need to put the F322W2 and F444W files in separate folders. Also make sure that you're only working on the spectroscopy files and not also the simultaneously collected photometry files

@taylorbell57
Copy link
Collaborator

@mithun1996-png, can you confirm whether you're still dealing with this issue?

@mithun1996-png
Copy link
Author

this issue is still happen ,but in stage 3 ecf file when i fix row by row background subtraction is true then this issue actually come up. but if i set row by row background subtraction then their is no issue like that . so can you give a suggestion what can do ,what is the reason for that issue .

@taylorbell57
Copy link
Collaborator

@mithun1996-png, I looked on MAST and there should only be 11 F444W files and 14 F322W2 files; because your log statements up above state Found 25 data file(s) ending in calints.fits, I am quite sure that you are incorrectly trying to work on F322W2 and F444W observations at the same time. Please separate the F322W2 files from the F444W files and only work on one of them at a time. If you are still encountering an issue when working on only one data set at a time, please copy-paste your new error traceback output

@mithun1996-png
Copy link
Author

mithun1996-png commented Nov 21, 2024

we get same error when working on only one data set at a time , i am working using F322W filter
ecf file for this filter :

# Eureka! Control File for Stage 3: Data Reduction

# Stage 3 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-3

ncpu            4           # Number of CPUs
nfiles          1           # The number of data files to analyze simultaneously
max_memory      0.5         # The maximum fraction of memory you want utilized by read-in frames (this will reduce nfiles if need be)
indep_batches   False       # Independently treat each batch of files? Strongly recommended to leave this as False unless you have a clear reason to set it to True.
suffix          calints     # Data file suffix

calibrated_spectra  False   # Set True to generate flux-calibrated spectra/photometry in mJy
                            # Set False to convert to electrons

# Subarray region of interest
ywindow         [5,64]      # Vertical axis as seen in DS9
xwindow         [4,1704]  # Horizontal axis as seen in DS9
src_pos_type    gaussian    # Determine source position when not given in header (Options: header, gaussian, weighted, max, hst, or a numeric value)
record_ypos     True        # Option to record the y position and width for each integration (only records if src_pos_type is gaussian)
poly_wavelength	True		# Use an updated polynomial wavelength solution for NIRCam longwave spectroscopy instead of the linear wavelength solution currently assumed by STScI
dqmask          True        # Mask pixels with an odd entry in the DQ array
expand          1           # Super-sampling factor along cross-dispersion direction

# Outlier rejection along time axis
ff_outlier      True      # Set False to use only background region (recommended for deep transits)
                            # Set True to use full frame (works well for shallow transits/eclipses)
bg_thresh       [10,10]       # Double-iteration X-sigma threshold for outlier rejection along time axis

# Background parameters
bg_hw           11           # Half-width of exclusion region for BG subtraction (relative to source position)
bg_deg          1          # Polynomial order for column-by-column background subtraction, -1 for median of entire frame
bg_method       median        # Options: std (Standard Deviation), median (Median Absolute Deviation), mean (Mean Absolute Deviation)
p3thresh        5           # X-sigma threshold for outlier rejection during background subtraction
bg_row_by_row   True       # Row-by-row BG subtraction (only useful for NIRCam)
bg_x1           1850      # Left edge of exclusion region for row-by-row BG subtraction
bg_x2           2048   # Right edge of exclusion region for row-by-row BG subtraction

# Spectral extraction parameters
spec_hw         7           # Half-width of aperture region for spectral extraction (relative to source position)
fittype         meddata     # Method for constructing spatial profile (Options: smooth, meddata, poly, gauss, wavelet, or wavelet2D)
median_thresh   10          # Sigma threshold when flagging outliers in median frame, when fittype=meddata and window_len > 1
window_len      7           # Smoothing window length, for median frame or when fittype = smooth or meddata (when computing median frame). Can set to 1 for no smoothing when computing median frame for fittype=meddata.
prof_deg        3           # Polynomial degree, when fittype = poly
p5thresh        10          # X-sigma threshold for outlier rejection while constructing spatial profile
p7thresh        7           # X-sigma threshold for outlier rejection during optimal spectral extraction

# Curvature treatment
curvature       correct     # How to manage the curved trace on the detector (Options: None, correct)

# Diagnostics
isplots_S3      3           # Generate few (1), some (3), or many (5) figures (Options: 1 - 5)
nplots          5           # How many of each type of figure do you want to make per file?
vmin            0.97        # Sets the vmin of the color bar for Figure 3101.
vmax            1.03        # Sets the vmax of the color bar for Figure 3101.
time_axis       'y'         # Determines whether the time axis in Figure 3101 is along the y-axis ('y') or the x-axis ('x')
testing_S3      False       # Boolean, set True to only use last file and generate select figures
hide_plots      True        # If True, plots will automatically be closed rather than popping up
save_output     True        # Save outputs for use in S4
save_fluxdata   Ture      # Save the much larger FluxData.h5 outputs which can be useful for debugging or comparisons between different pipelines
verbose         True        # If True, more details will be printed about steps

# Project directory
topdir        /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01
# Directories relative to topdir
inputdir       /output_stage_2_F322W/S2_2024-11-20_nircam_wfss_template_run2/
outputdir     /Output_stage_3/filter_F322W/

Error outcome working with F322W data

Starting Stage 3 Reduction

Input directory: /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/output_stage_2_F322W/S2_2024-11-20_nircam_wfss_template_run2/
  Found 14 data file(s) ending in calints.fits
Output directory: /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Output_stage_3/filter_F322W/S3_2024-11-21_nircam_wfss_template_run1/ap7_bg11/
Using ap=7, bg=11, expand=1
Copying S3 control file
Starting file 1 of 14
  Reading file 1...
  Performing RxR background subtraction...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 756/756 [00:05<00:00, 135.27it/s]
  Creating figures for background subtraction...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:03<00:00,  1.32it/s]
  Masking NaNs/infs in data arrays...
    FLUX has 75826800 NaNs/infs, which is 100.00% of all pixels.
  WARNING: Your region of interest may be off the edge of the detector subarray.  Masking NaN/inf regions and continuing, but you should really stop and reconsider your choices.
    ERR has 75826800 NaNs/infs, which is 100.00% of all pixels.
  WARNING: Your region of interest may be off the edge of the detector subarray.  Masking NaN/inf regions and continuing, but you should really stop and reconsider your choices.
    V0 has 75826800 NaNs/infs, which is 100.00% of all pixels.
  WARNING: Your region of interest may be off the edge of the detector subarray.  Masking NaN/inf regions and continuing, but you should really stop and reconsider your choices.
  Locating source position...
Traceback (most recent call last):
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/demos/JWST/run_eureka.py", line 27, in <module>
    spec, meta = s3.reduce(eventlabel, ecf_path=ecf_path)
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/s3_reduce.py", line 350, in reduce
    source_pos.source_pos_wrapper(data, meta, log, m)
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/source_pos.py", line 101, in source_pos_wrapper
    meta.src_ypos = source_pos(flux[integ], meta, data.attrs['shdr'],
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/source_pos.py", line 162, in source_pos
    src_ypos, src_ywidth = source_pos_gauss(flux, meta, m, n, plot)
  File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/source_pos.py", line 410, in source_pos_gauss
    p0 = [np.ma.max(med_row), pos_max, sigma0, np.ma.median(med_row)]
  File "/home/seps3/miniconda3/envs/eureka/lib/python3.10/site-packages/numpy/ma/core.py", line 6795, in max
    return obj.max(axis=axis, fill_value=fill_value, out=out, **kwargs)
  File "/home/seps3/miniconda3/envs/eureka/lib/python3.10/site-packages/numpy/ma/core.py", line 5922, in max
    result = self.filled(fill_value).max(
  File "/home/seps3/miniconda3/envs/eureka/lib/python3.10/site-packages/numpy/core/_methods.py", line 41, in _amax
    return umr_maximum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation maximum which has no identity

@taylorbell57 taylorbell57 changed the title ValueError: zero-size array to reduction operation maximum which has no identity , when i start to run stage 3 pipeline [Bug]: Stage 3 NIRCam row-by-row background subtraction is broken Nov 21, 2024
@taylorbell57
Copy link
Collaborator

Alright, I just tested Stage 3's NIRCam row-by-row background subtraction, and I can confirm that there does indeed appear to be some bug that results in all pixels becoming masked. While we work to sort this out, I recommend you stick with column-by-column background subtraction only and set bg_row_by_row to False.

On an unrelated note, I saw that you have accidentally set save_fluxdata to Ture rather than True. I also generally recommend leaving save_fluxdata to False unless there is something in particular you want from that file; the FluxData.h5 files are really large and are only really useful for detailed debugging or comparisons between different pipelines

@mithun1996-png
Copy link
Author

in stage 3 i set Row_by_Row background subtraction set it to false ,its working .and i got white light curve after stage 4 . and then i proceed stage 5 for best fit light curve .but fitting not realistic ,is i doing any mistake if i did please some suggestion .
actually i am follow step by step instruction which are mention my paper

@mithun1996-png
Copy link
Author

mithun1996-png commented Nov 25, 2024

hello sir , according to paper correct the 1/f noise correction in each individual pixel in a segment using a 10 sigma threshold , but in ecf file there is no parameter for this ,can you give proper template for this noise correction
ecf file in stage 3 👍

# Eureka! Control File for Stage 3: Data Reduction

# Stage 3 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-3

ncpu            4           # Number of CPUs
nfiles          1           # The number of data files to analyze simultaneously
max_memory      0.5         # The maximum fraction of memory you want utilized by read-in frames (this will reduce nfiles if need be)
indep_batches   False       # Independently treat each batch of files? Strongly recommended to leave this as False unless you have a clear reason to set it to True.
suffix          calints     # Data file suffix

calibrated_spectra  False   # Set True to generate flux-calibrated spectra/photometry in mJy
                            # Set False to convert to electrons

# Subarray region of interest
ywindow         [5,64]      # Vertical axis as seen in DS9
xwindow         [4,1704]  # Horizontal axis as seen in DS9
src_pos_type    gaussian    # Determine source position when not given in header (Options: header, gaussian, weighted, max, hst, or a numeric value)
record_ypos     True        # Option to record the y position and width for each integration (only records if src_pos_type is gaussian)
poly_wavelength	True		# Use an updated polynomial wavelength solution for NIRCam longwave spectroscopy instead of the linear wavelength solution currently assumed by STScI
dqmask          True        # Mask pixels with an odd entry in the DQ array
expand          1           # Super-sampling factor along cross-dispersion direction

# Outlier rejection along time axis
ff_outlier      True      # Set False to use only background region (recommended for deep transits)
                            # Set True to use full frame (works well for shallow transits/eclipses)
bg_thresh       [10,10]       # Double-iteration X-sigma threshold for outlier rejection along time axis

# Background parameters
bg_hw           11           # Half-width of exclusion region for BG subtraction (relative to source position)
bg_deg          1          # Polynomial order for column-by-column background subtraction, -1 for median of entire frame
bg_method       median        # Options: std (Standard Deviation), median (Median Absolute Deviation), mean (Mean Absolute Deviation)
p3thresh        5           # X-sigma threshold for outlier rejection during background subtraction
bg_row_by_row   False       # Row-by-row BG subtraction (only useful for NIRCam)
bg_x1           1850      # Left edge of exclusion region for row-by-row BG subtraction
bg_x2           2048   # Right edge of exclusion region for row-by-row BG subtraction

# Spectral extraction parameters
spec_hw         7           # Half-width of aperture region for spectral extraction (relative to source position)
fittype         meddata     # Method for constructing spatial profile (Options: smooth, meddata, poly, gauss, wavelet, or wavelet2D)
median_thresh   10          # Sigma threshold when flagging outliers in median frame, when fittype=meddata and window_len > 1
window_len      7           # Smoothing window length, for median frame or when fittype = smooth or meddata (when computing median frame). Can set to 1 for no smoothing when computing median frame for fittype=meddata.
prof_deg        3           # Polynomial degree, when fittype = poly
p5thresh        10          # X-sigma threshold for outlier rejection while constructing spatial profile
p7thresh        7           # X-sigma threshold for outlier rejection during optimal spectral extraction

# Curvature treatment
curvature       correct     # How to manage the curved trace on the detector (Options: None, correct)

# Diagnostics
isplots_S3      3           # Generate few (1), some (3), or many (5) figures (Options: 1 - 5)
nplots          5           # How many of each type of figure do you want to make per file?
vmin            0.97        # Sets the vmin of the color bar for Figure 3101.
vmax            1.03        # Sets the vmax of the color bar for Figure 3101.
time_axis       'y'         # Determines whether the time axis in Figure 3101 is along the y-axis ('y') or the x-axis ('x')
testing_S3      False       # Boolean, set True to only use last file and generate select figures
hide_plots      True        # If True, plots will automatically be closed rather than popping up
save_output     True        # Save outputs for use in S4
save_fluxdata   Ture      # Save the much larger FluxData.h5 outputs which can be useful for debugging or comparisons between different pipelines
verbose         True        # If True, more details will be printed about steps

# Project directory
topdir        /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01
# Directories relative to topdir
inputdir       /output_stage_2_F322W/S2_2024-11-20_nircam_wfss_template_run2/
outputdir     /Output_stage_3/filter_F322W/

@taylorbell57
Copy link
Collaborator

in stage 3 i set Row_by_Row background subtraction set it to false ,its working .and i got white light curve after stage 4 . and then i proceed stage 5 for best fit light curve .but fitting not realistic ,is i doing any mistake if i did please some suggestion . actually i am follow step by step instruction which are mention my paper

I'm not sure what you mean by "fitting not realistic". I very strongly recommend you first try your very best to troubleshoot any issues with fitting yourself since these are almost always user error and not caused by a bug with Eureka!. We're available to help resolve bugs and troubleshoot unusual issues, but we can't provide detailed and one-on-one support for everyone's analyses; much of our work on Eureka! is unfunded or underfunded, and we have to focus on our own research. For help on learning how to analyze JWST data, I recommend you look to resources like the Hands-On Sessions I and II that I made for the 2023 Sagan Summer Workshop which come with detailed instructions on how to analyze data using Eureka!. I and some others also publish our Eureka! Control Files for each of our papers which can be used easily reproduce published work (e.g., 1, 2, 3). However, if you become pretty confident that you're encountering a bug with our code, please feel free to open a new issue and we'll look into it.

hello sir , according to paper correct the 1/f noise correction in each individual pixel in a segment using a 10 sigma threshold , but in ecf file there is no parameter for this ,can you give proper template for this noise correction
ecf file in stage 3

I think what you're saying is that 1/f noise correction was applied at the group-level (where 1/f correction is more effective); this group-level work would be part of Stage 1 instead of Stage 3. Since the row-by-row background subtraction code is shared between Stages 1 and 3, I assume that Stage 1 row-by-row background subtraction will also fail in the same way at present. Once we get that code working again, there are similar settings in Stage 1 (e.g. bg-row-by-row) as in Stage 3. As for the 10 sigma threshold you mentioned, I'd assume that would be p3thresh, but can't say for sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working NIRCam
Projects
Development

No branches or pull requests

5 participants