Keeping track of previous reconstruction

[FerreolS]

When doing several reconstructions successively on the same OIfits file. It can be nice to keep track of all the previous reconstruction.
This can be done easily by adding a new HDU containing the current initial image and place the result in the primary.
This may lead to several issues:

• if the number of iteration is 50, then all HDU will have the name OUTPUT50 as stated in OI-Interface. But HDU name must be unique.
• we should keep also track of all parameters used for each reconstruction especially INIT_IMG that can be used to follow the reconstruction chain. I suppose that adding all keywords of OUTPUT_PARAM in the HDU containing the INIT_IMG can be a good solution

[GillesDuvert]

I thought 'keeping track" is something the GUI should manage?
If the input parameter list contains
NUMBER_OF_RECONSTRUCTION_SO_FAR=33
Then it can be incremented to 34 upon return of the imaging software, or something similar?

[FerreolS]

It is something that can be handled by the GUI using a session description that has still to be defined. However, reading the 3.2 section of OI-Interface.pdf . I though that it can be also something that can be handled easily by software (with the GUI patching software that does do it).
In any case, if we decide to stack previous reconstruction result in a single OIFits file, we will face the same two questions. I think that storing the OUTPUT/INPUT PARAM keyword in each image HDU header is OK and easy. But we may have to change HDU names to prevent having several OUTPUT50 name.

[FerreolS]

The solution can be either defining the name with adding iteration number since the beginning or incrementing a new counter such the name become OUTPUT_k_50 for the k-th image reconstructed with 50 iteration.
The only difference I see is if the number of iteration is zero then we will still have duplicate number in the first case. However the image should be identical in that case (but not necessarily as some constraint may have been enforced (like normalisation) before any iteration) .

[jsy1001]

I'm wary of storing input/output parameters in the headers of image HDUs. We originally decided not to do this in case any future algorithm needs parameters that can't straightforwardly be stored as header cards (e.g. vectors or matrices).

[GillesDuvert]

I strongly advocate a clear separation of history memory btw. the caller (GUI) and the softwares. They just need to copy back all the input tables with their headers (and they do it now). What the 'outer world' (=OImaging) does with those headers, keywords, history etc is not our concern. Let's image reconstruction software authors stay focussed on their expertise...

[FerreolS]

I'm not sure to understand. What is the right way to do if the user want to keep track of previous reconstruction:

• first solution build a kind of archive with all the fits files returned by the softwares with some extra file to describe the link between them. In that case, there is no constraints on softwares.
• record previous reconstructed image in other HDU. In that case, the software at least shouldn't destroy those hdu in the output file. We also have to decide where can we store the parameters describing those previous image and ensure that hdu name are unique.

What is your preference? (Mine is on the second solution that seems a bit simpler to manage at the GUI level.)

[emmt]

For me, initial parameters, the selection of data, and the algorithm should (hopegully) uniquely define the result. Intermediate images are useless unless you want to keep the user attention. So it is rather the job of the GUI. It may be a not so good idea to save them in the output file. It may give the impression that "early stopping" results are worth to consider as in the good old times of the RL method ;-)

[FerreolS]

From section 3.2, I understand that the mandatory images are Init image and final image that are identified with their HDUNAME in the keywords INIT_IMG and LAST_IMG respectively. The final image being in the primary HDU. But there is nothing about images in other HDUs that can contains priors or anything needed by the user.
What I want to ensure are:

• software should not alter or destroy other HDUs, blindly copying them in the output file (easy)
• there should be a mechanism preventing unicity of HDU names: let say at the end of 50 iteration of a run, the final image HDU name is IMAGE-OI OUTPUT50 (as stated in sec 3.2). If the user want to continue the run for 50 more iteration, then both INIT_IMG and LAST_IMG will have the same HDU name pointing on different images. This is not possible we should define clearly how to cope with this issue.

I just to make these 2 points fixed, approved by everyone and written it in the OI-Interface document

[FerreolS]

I can make two different issues for the 2 points above if you prefer as the subject differ now for the title

[emmt]

If software blindly copies input HDU's in the output, then if input already has images at different iterates, all iterates will be mixed. So why not having a different output for the intermediate images? For instance, software can save into another FITS file the current iterate every, say n, iterations and the GUI can track this file (or these files) to display a live image. However, there is a pending issue: we need a locking mechanism to avoid overwritting images. This can be just the file name which is numbered and the GUI can do the necessary cleanup of these temporary files.

[FerreolS]

I agree that tracking reconstruction history must be done at OImaging level (it can be just history but it can be more complex e.g. if we make reconstruction decreasing some parameter a each run). Up to now we didn't chose the mechanism to track this: it will be probably a kind of archive storing everything in multiple files but it can also be a single OIfits file with many hdu. As I'm not sure we should settle this mechanism now (it has many implications especially if we want to store it in a database), I would prefer to let the two options possible. So I would like to ensure that all hdus unused by sofwares are all blindly copied. On my opinion, softwares should just write OUTPUT PARAM table, the final image and the model as stated in #12 without destroying other HDUs. This will let everything open for future features of OImaging without bothering you again.

But we still have to find a mechanism preventing HDUNAME duplicates

[jsy1001]

FYI BSMEM rewrites the HDUs containing the input data (OI_VIS2 and OI_T3) in order to add the model values, as proposed in the specification document. It wouldn't be too difficult to implement copying of other HDUs provided we can define a convention to prevent HDUNAME clashes.

You might be interested to read about the grouping convention here: https://fits.gsfc.nasa.gov/registry/grouping/grouping.pdf

[GillesDuvert]

I vote for closing this and open a new issue "How to prevent HDUNAME duplicates"

[FerreolS]

I have split the two points in two issue:

• softwares should not alter or destroy other HDUs -> Software shouldn't change their inputs  #12
• mechanism preventing HDUNAME duplicates -> How to prevent HDUNAME duplicates #17