Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move datasets to delete first in line #261

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Commits on Oct 22, 2021

  1. Move datasets to delete first in line

    We have reports of datasets that get re-harvested with an extra `1`
    in the URL. We have confirmed these reports.
    It seems the harvest is doing the best it can to diagnose if
    this is a new dataset or not;
    but still failing in some circumstances.
    This probably won't fix the bug; however it will mitigate it.
    By hopefully running through the datasets removal first,
    if the spatial harvester is essentially doing a "delete and add"
    when it should be replacing, then the name of the new dataset
    won't collide with the one that is marked for deletion
    but still in the system.
    jbrown-xentity committed Oct 22, 2021
    Configuration menu
    Copy the full SHA
    6264488 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2021

  1. Remove package entirely that are marked for delete

    If the harvest is managing the datasets in ckan, it seems that the
    harvest source should be the "source of truth".
    If this is the case, we shouldn't need "revive" capability of soft
    removing packages/datasets in ckan. I propose to actually purge
    the dataset within ckan.
    Since it's difficult/nearly impossible to track these files without a
    unique id, sometimes the harvester will delete and create a new item if
    the waf or files change in any way. This would keep that behind the
    scenes, and allow the end user to get to the same dataset at the old URL
    jbrown-xentity committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    8bea4df View commit details
    Browse the repository at this point in the history