Timestamp updates for unchanged data complicates usage, especially caching

[aiuto]

The publisher updates timestamps of data files even if no other information has changed.
See this PR for an example:
https://github.com/spdx/license-list-data/commits/6552b858c0f9224be9ed38ad8e941f662a4496ab/json/licenses.json

This creates needless churn in downstream systems that cache this information. It would be more useful to have a modification timestamp.

cc: @danielmachlab

[goneall]

@aiuto - the licenses.json file is regenerated by the application every time the application is run. This may be causing the creation timestamp issue. I don't see an easy way to change this to be a modified timestamp. If you find a way to modify the utility to resolve this problem, please create a pull request or comment on this issue.

[goneall]

@aiuto - Any concern with closing this issue? It doesn't seem there is an easy solution or volunteer to code up the changes.

[aiuto]

The fix would be to

• generate the new list to an in memory struct
• read the old list
• iterate through old
  • if new exists, but is unchanged, preserve old entry
  • if not, drop it
• iterate through the remainder of the new list

I would volunteer to try that, but have no time until December.

[goneall]

@aiuto Another possibility is to look at the timestamps or Git commit information on the input license-list-XML files to determine if they changed then just skip processing - may be simpler than maintaining the list in memory.

The Makefile in the license-list-XML repo that runs the CI running this app deletes the input files before running, so that would also need to change.

If you're willing to help out, I'll leave this issue open - ping me before you start so I can coordinate any other changes.

[aiuto]

using git commits tie truth to the SCM implementation.

…

On Mon, Oct 31, 2022, 2:19 PM goneall ***@***.***> wrote: @aiuto <https://github.com/aiuto> Another possibility is to look at the timestamps or Git commit information on the input license-list-XML files to determine if they changed then just skip processing - may be simpler than maintaining the list in memory. The Makefile <https://github.com/spdx/license-list-XML/blob/main/Makefile> in the license-list-XML repo that runs the CI running this app deletes the input files before running, so that would also need to change. If you're willing to help out, I'll leave this issue open - ping me before you start so I can coordinate any other changes. — Reply to this email directly, view it on GitHub <#136 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHHR67DDGIDN67XTGNLWGAESRANCNFSM57IXX5XA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[goneall]

@aiuto Just pinging you to see if you are interested in creating a fix or if I should close this issue.

[aiuto]

I don't have time for a fix this quarter. That doesn't mean it is not an issue, however.

[goneall]

I don't have time for a fix this quarter. That doesn't mean it is not an issue, however.

I'll leave it open, but mark it as "won't fix" for now. Once you have time to work on it, update the issue or ping me and I'll change it back.