Issue38: optional purge (#39)

* issue38:
clean up install_update

* issue38:
add purge_dst_dir and exclude_from_purge kwargs to install_update(), and adapt _install_update_win accordingly

* issue38:
add tests for install_update purge and options override

* issue38:
amend client docstring with purge warning

* issue38:
implement optional purge in _install_update_mac

* issue38:
clear src_dir contents in _install_update_mac

* issue38:
add explicit warning to readme about purge option

README.md

@@ -78,13 +78,20 @@ The `tufup.repo` module provides a convenient way to streamline the above proced
 
 ## How updates are applied (client-side)
 
-By default, updates are applied by replacing all files in the current app installation path with files from the latest archive.
-The latest archive is either downloaded in full (as described above), or it is derived from the current archive by applying one or more downloaded patches.
+By default, updates are applied by copying all files and folders from the latest archive to the current app installation directory.
 
-Once the latest archive is available, it is decompressed to a temporary location.
-From there, a script is started that clears the current app installation dir, and moves the new files into place.
-After starting the script, the currently running process will exit. 
-Alternatively, you can specify a custom installation script.
+Here's what happens during the update process:
+
+- The latest archive is either downloaded in full, as described above, or it is derived from the current archive by applying one or more downloaded patches.
+- Once the latest archive is available on disk, it is decompressed to a temporary directory.
+- A default install script is then started, which copies the new files and folders from the temporary directory to the current app installation directory. On Windows, this script is started in a new process, after which the currently running process will exit.
+- Alternatively, you can specify a custom install script to do whatever you want with the new files.
+
+The default install script accepts an optional `purge_dst_dir` argument, which will cause *ALL* files and folders to be deleted from the app installation directory, before moving the new files into place.
+This is a convenient way to remove any stale files and folders from the app installation directory.
+
+>**WARNING**: The `purge_dst_dir` option should *only* be used if the app is properly installed in its *own separate* directory.
+If this is not the case, for example if the app is running from the Windows `Desktop` directory, any *unrelated* files or folders in this directory will also be deleted! 
 
 ## Migrating from other update frameworks
 

src/tufup/client.py

@@ -106,13 +106,25 @@ def download_and_apply_update(
         This downloads the files found by `check_for_updates`, applies any
         patches, and extracts the resulting archive to the `extract_dir`. At
         that point, the update is ready to be installed (i.e. moved into
-        place). This is done by calling `install`.
+        place). This is done by calling `install` with the specified `**kwargs`.
 
-        The default `install` callable purges the `app_install_dir`,
-        moves the files from `extract_dir` to `app_install_dir`, and exits
-        the application (not necessarily in that order).
+        The default `install` callable moves the content of `extract_dir` to
+        `app_install_dir`, and exits the application (not necessarily in that
+        order).
 
-        kwargs are passed on to the 'install' callable
+        The **kwargs are passed on to the 'install' callable
+
+        The default `install` callable accepts two additional arguments:
+
+            `purge_dst_dir` (default False): if True, *ALL* content will be
+            deleted from the `app_install_dir`
+
+            `exclude_from_purge` (default None): list of paths to exclude
+            from purge
+
+            **DANGER**: Only set `purge_dst_dir=True` if your app is
+            installed in its own separate directory, otherwise this will
+            cause unrelated files and folders to be deleted.
         """
         if install is None:
             install = install_update

src/tufup/utils/platform_specific.py

@@ -20,29 +20,62 @@
 def install_update(
         src_dir: Union[pathlib.Path, str],
         dst_dir: Union[pathlib.Path, str],
+        purge_dst_dir: bool = False,
+        exclude_from_purge: List[Union[pathlib.Path, str]] = None,
         **kwargs,
 ):
+    """
+    Installs update files using platform specific installation script. The
+    actual installation script copies the files and folders from `src_dir` to
+    `dst_dir`.
+
+    If `purge_dst_dir` is `True`, *ALL* files and folders are deleted from
+    `dst_dir` before copying.
+
+    **DANGER**:
+
+    ONLY use `purge_dst_dir=True` if your app is properly installed in its
+    own *separate* directory, such as %PROGRAMFILES%\MyApp.
+
+    DO NOT use `purge_dst_dir=True` if your app executable is running
+    directly from a folder that also contains unrelated files or folders,
+    such as the Desktop folder or the Downloads folder, because this
+    unrelated content would be then also be deleted.
+
+    Individual files and folders can be excluded from purge using e.g.
+
+        exclude_from_purge=['path\\to\\file1', r'"path to\file2"', ...]
+
+    If `purge_dst_dir` is `False`, the `exclude_from_purge` argument is
+    ignored.
+    """
     if ON_WINDOWS:
-        return _install_update_win(
-            src_dir=src_dir, dst_dir=dst_dir, **kwargs
-        )
-    if ON_MAC:
-        return _install_update_mac(src_dir=src_dir, dst_dir=dst_dir, **kwargs)
+        _install_update = _install_update_win
+    elif ON_MAC:
+        _install_update = _install_update_mac
     else:
         raise RuntimeError('This platform is not supported.')
+    return _install_update(
+        src_dir=src_dir,
+        dst_dir=dst_dir,
+        purge_dst_dir=purge_dst_dir,
+        exclude_from_purge=exclude_from_purge,
+        **kwargs,
+    )
 
 
 WIN_DEBUG_LINES = """
 rem wait for user confirmation (allow user to read any error messages)
 timeout /t -1
 """
 
-WIN_DEFAULT_ROBOCOPY_OPTIONS = (
-    '/e',  # include subdirs
-    '/move',  # move files and dirs
-    '/v',  # verbose
-    '/purge',  # delete stale files and dirs in destination folder
+WIN_ROBOCOPY_OVERWRITE = (
+    '/e',  # include subdirectories, even if empty
+    '/move',  # deletes files and dirs from source dir after they've been copied
+    '/v',  # verbose (show what is going on)
 )
+WIN_ROBOCOPY_PURGE = '/purge'  # delete all files and dirs in destination folder
+WIN_ROBOCOPY_EXCLUDE_FROM_PURGE = '/xf'  # exclude specified paths from purge
 
 # https://stackoverflow.com/a/20333575
 WIN_MOVE_FILES_BAT = """@echo off
@@ -81,9 +114,11 @@ def run_bat_as_admin(file_path: Union[pathlib.Path, str]):
 def _install_update_win(
         src_dir: Union[pathlib.Path, str],
         dst_dir: Union[pathlib.Path, str],
+        purge_dst_dir: bool,
+        exclude_from_purge: List[Union[pathlib.Path, str]],
         as_admin: bool = False,
         debug: bool = False,
-        extra_robocopy_options: List[str] = None,
+        robocopy_options_override: List[str] = None,
 ):
     """
     Create a batch script that moves files from src to dst, then run the
@@ -92,16 +127,27 @@ def _install_update_win(
     The script is created in a default temporary directory, and deletes
     itself when done.
 
-    Extra options for [robocopy][1] can be specified as a list of strings.
-    For example, to exlude files from being purged:
+    The `as_admin` options allows installation as admin (opens UAC dialog).
+
+    The `debug` option will keep the console open so we can investigate
+    issues with robocopy.
 
-        extra_robocopy_options=['/xf', 'path\\to\\file1', r'"path to\file2"']
+    Options for [robocopy][1] can be overridden completely by passing a list
+    of option strings to `robocopy_options_override`. This will cause the
+    purge arguments to be ignored as well.
 
     [1]: https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy
     """
-    options = list(WIN_DEFAULT_ROBOCOPY_OPTIONS)
-    if extra_robocopy_options:
-        options.extend(extra_robocopy_options)
+    if robocopy_options_override is None:
+        options = list(WIN_ROBOCOPY_OVERWRITE)
+        if purge_dst_dir:
+            options.append(WIN_ROBOCOPY_PURGE)
+            if exclude_from_purge:
+                options.append(WIN_ROBOCOPY_EXCLUDE_FROM_PURGE)
+                options.extend(exclude_from_purge)
+    else:
+        # empty list [] simply clears all options
+        options = robocopy_options_override
     options_str = ' '.join(options)
     debug_lines = ''
     if debug:
@@ -130,15 +176,26 @@ def _install_update_win(
 def _install_update_mac(
         src_dir: Union[pathlib.Path, str],
         dst_dir: Union[pathlib.Path, str],
+        purge_dst_dir: bool,
+        exclude_from_purge: List[Union[pathlib.Path, str]],
         **kwargs,
 ):
     # todo: implement as_admin and debug kwargs for mac
-    logger.debug(f'kwargs not used: {kwargs}')
+    logger.debug(f'Kwargs not used: {kwargs}')
+    if purge_dst_dir:
+        exclude_from_purge = [  # enforce path objects
+            pathlib.Path(item) for item in exclude_from_purge
+        ] if exclude_from_purge else []
+        logger.debug(f'Purging content of {dst_dir}')
+        for path in pathlib.Path(dst_dir).iterdir():
+            if path not in exclude_from_purge:
+                remove_path(path=path)
     logger.debug(f'Moving content of {src_dir} to {dst_dir}.')
-    remove_path(pathlib.Path(dst_dir))
     shutil.copytree(src_dir, dst_dir, dirs_exist_ok=True)
-    logger.debug(f'Removing src directory {src_dir}.')
-    remove_path(pathlib.Path(src_dir))
+    # Note: the src_dir is typically a temporary directory, but we'll clear
+    # it anyway just to be consistent with the windows implementation
+    for path in pathlib.Path(src_dir).iterdir():
+        remove_path(path=path)
     logger.debug(f'Restarting application, running {sys.executable}.')
     subprocess.Popen(sys.executable, shell=True)  # nosec
     sys.exit(0)

tests/test_utils_platform_specific.py

@@ -10,18 +10,59 @@
     ON_WINDOWS, PLATFORM_SUPPORTED, run_bat_as_admin
 )
 
+_reason_platform_not_supported = (
+    'install_update() is only actively supported on windows and mac'
+)
+
 DUMMY_APP_CONTENT = f"""
 import sys
 sys.path.append('{(BASE_DIR.parent / 'src').as_posix()}')
 from tufup.utils.platform_specific import install_update
-install_update(src_dir=sys.argv[1], dst_dir=sys.argv[2], {{extra_kwargs}})
+install_update(src_dir=sys.argv[1], dst_dir=sys.argv[2], {{extra_kwargs_str}})
 """
 
 ON_GITHUB = os.getenv('GITHUB_ACTIONS')
 TEST_RUNAS = os.getenv('TEST_RUNAS')
 
 
 class UtilsTests(TempDirTestCase):
+    def setUp(self) -> None:
+        super().setUp()
+        # create src dir with dummy app file, and dst dir with stale subdir
+        # and a file that must be excluded from purge
+        test_dir = self.temp_dir_path / 'tufup_tests'
+        self.src_dir = test_dir / 'src'
+        self.src_subdir = self.src_dir / 'new'
+        self.src_subdir.mkdir(parents=True)
+        self.dst_dir = test_dir / 'dst'
+        self.dst_subdir = self.dst_dir / 'stale'
+        self.dst_subdir.mkdir(parents=True)
+        (self.dst_subdir / 'stale.file').touch()
+        self.keep_file_path = self.dst_dir / 'keep.file'
+        self.keep_file_path.touch()
+        self.keep_file_str = str(self.keep_file_path).replace('\\', '\\\\')
+        self.src_file_name = 'dummy_app.py'
+        self.src_file_path = self.src_dir / self.src_file_name
+
+    def run_dummy_app(self, extra_kwargs_strings):
+        # write dummy app content to file
+        dummy_app_content = DUMMY_APP_CONTENT.format(
+            extra_kwargs_str=', '.join(extra_kwargs_strings),
+        )
+        print(dummy_app_content)
+        self.src_file_path.write_text(dummy_app_content)
+        # run the dummy app in a separate process, which, in turn, will run
+        # another process that moves the file
+        completed_process = subprocess.run(
+            [sys.executable, self.src_file_path, self.src_dir, self.dst_dir]
+        )
+        print(sys.executable)
+        completed_process.check_returncode()
+        if ON_WINDOWS:
+            # allow some time for the batch file to complete (the batch file
+            # waits a few seconds, so we have to wait longer)
+            sleep(3)
+
     @unittest.skipIf(
         condition=ON_GITHUB or not TEST_RUNAS or not ON_WINDOWS,
         reason='windows only, requires user interaction',
@@ -43,53 +84,64 @@ def test_run_bat_as_admin(self):
         self.assertNotIn(current_user, output)
 
     @unittest.skipIf(
-        condition=not PLATFORM_SUPPORTED,
-        reason='install_update() is only actively supported on windows and mac',
+        condition=not PLATFORM_SUPPORTED, reason=_reason_platform_not_supported
     )
-    def test_install_update(self):
-        # create src dir with dummy app file, and dst dir with stale subdir
-        test_dir = self.temp_dir_path / 'tufup_tests'
-        src_dir = test_dir / 'src'
-        src_dir.mkdir(parents=True)
-        dst_dir = test_dir / 'dst'
-        dst_subdir = dst_dir / 'stale'
-        dst_subdir.mkdir(parents=True)
-        (dst_subdir / 'stale.file').touch()
-        src_file_name = 'dummy_app.py'
-        src_file_path = src_dir / src_file_name
-        keep_file_path = dst_dir / 'keep.file'
-        keep_file_path.touch()
-        # write dummy app content to file
-        extra_kwargs = ''
+    def test_install_update_no_purge(self):
+        extra_kwargs_strings = []
         if ON_WINDOWS:
-            keep_file_str = str(keep_file_path).replace('\\', '\\\\')
-            extra_kwargs = ', '.join(
-                [
-                    'as_admin=False',
-                    'debug=False',
-                    f'extra_robocopy_options=["/xf", "{keep_file_str}"]',
-                ]
-            )
-        dummy_app_content = DUMMY_APP_CONTENT.format(extra_kwargs=extra_kwargs)
-        print(dummy_app_content)
-        src_file_path.write_text(dummy_app_content)
-        # run the dummy app in a separate process, which, in turn, will run
-        # another process that moves the file
-        completed_process = subprocess.run(
-            [sys.executable, src_file_path, src_dir, dst_dir]
-        )
-        print(sys.executable)
-        completed_process.check_returncode()
-        # allow some time for the batch file to complete (it also waits a few
-        # seconds, so we have to wait longer)
-        sleep(3)
+            extra_kwargs_strings.extend(['as_admin=False', 'debug=False'])
+        # run the dummy app in a separate process
+        self.run_dummy_app(extra_kwargs_strings=extra_kwargs_strings)
         # ensure file has been moved from src to dst
-        self.assertTrue(any(dst_dir.iterdir()))
-        self.assertTrue((dst_dir / src_file_name).exists())
+        self.assertTrue(any(self.dst_dir.iterdir()))
+        self.assertTrue((self.dst_dir / self.src_file_name).exists())
+        # new empty subdir has been moved as well
+        self.assertTrue((self.dst_dir / self.src_subdir.name).exists())
         # original src file no longer exists
-        self.assertFalse(src_file_path.exists())
-        # stale dst content has been removed (robocopy /purge)
-        self.assertFalse(dst_subdir.exists())
+        self.assertFalse(self.src_file_path.exists())
+        # stale dst content must still be present
+        self.assertTrue(self.dst_subdir.exists())
         # file to keep must still be present
+        self.assertTrue(self.keep_file_path.exists())
+
+    @unittest.skipIf(
+        condition=not PLATFORM_SUPPORTED, reason=_reason_platform_not_supported
+    )
+    def test_install_update_purge(self):
+        extra_kwargs_strings = [
+            'purge_dst_dir=True', f'exclude_from_purge=["{self.keep_file_str}"]'
+        ]
         if ON_WINDOWS:
-            self.assertTrue(keep_file_path.exists())
+            extra_kwargs_strings.extend(['as_admin=False', 'debug=False'])
+        # run the dummy app in a separate process
+        self.run_dummy_app(extra_kwargs_strings=extra_kwargs_strings)
+        # ensure file has been moved from src to dst
+        self.assertTrue(any(self.dst_dir.iterdir()))
+        self.assertTrue((self.dst_dir / self.src_file_name).exists())
+        # new empty subdir has been moved as well
+        self.assertTrue((self.dst_dir / self.src_subdir.name).exists())
+        # original src file no longer exists
+        self.assertFalse(self.src_file_path.exists())
+        # stale dst content has been removed (robocopy /purge)
+        self.assertFalse(self.dst_subdir.exists())
+        # file to keep must still be present
+        self.assertTrue(self.keep_file_path.exists())
+
+    @unittest.skipIf(condition=not ON_WINDOWS, reason='robocopy is windows only')
+    def test_install_update_robocopy_options_override(self):
+        extra_kwargs_strings = [
+            'as_admin=False', 'debug=False', 'robocopy_options_override=[]'
+        ]
+        # run the dummy app in a separate process
+        self.run_dummy_app(extra_kwargs_strings=extra_kwargs_strings)
+        # ensure file has been copied from src to dst
+        self.assertTrue(any(self.dst_dir.iterdir()))
+        self.assertTrue((self.dst_dir / self.src_file_name).exists())
+        # new subdir has not been copied
+        self.assertFalse((self.dst_dir / self.src_subdir.name).exists())
+        # original src file still exists
+        self.assertTrue(self.src_file_path.exists())
+        # stale dst content must still be present
+        self.assertTrue(self.dst_subdir.exists())
+        # file to keep must still be present
+        self.assertTrue(self.keep_file_path.exists())