Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add web archive as a final fallback [WIP] #40

Open
wants to merge 5 commits into
base: dev
Choose a base branch
from

Commits on Oct 24, 2024

  1. Add web archive as a final fallback

    This implements the web archive as a potential fallback for when the archive is no longer available during a batch download.  This does not allow for downloading removed archives if you do not already have the image metadata in your local database unless you add that data yourself.
    shinji257 committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    18b29c5 View commit details
    Browse the repository at this point in the history
  2. Wrap internet archive code in an option check

    Add ARCHIVE_ORG optional config to enable this functionality.  If image does not exist on internet archive you will still get a 404.  If they are down you will get a random different error.  The functionality is disabled by default.
    shinji257 committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    14ab00b View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2024

  1. Relocate web archive pull

    Moved the pull so it isn't part of the main loop.  I did duplicate the download image function with the only part removing the server loop and replacing the image download part with the one that rewrites and attempts to download the web archive version.  Previous version of the code tried to download from the archvie 5 times (once for each attempt pass) and I didn't realize this at first.  This one only does it once and only after the 5 tries have been tried with the regular servers.  Ideally I'd like to reduce the code duplicate (or eliminate outright) and have it abort the archive attempt at the first error.  At least it is better about it now.
    shinji257 committed Oct 27, 2024
    Configuration menu
    Copy the full SHA
    a119977 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9f524af View commit details
    Browse the repository at this point in the history
  3. Comment...

    shinji257 committed Oct 27, 2024
    Configuration menu
    Copy the full SHA
    77a4f24 View commit details
    Browse the repository at this point in the history