v3_tuto_cleanup

Policy tutorial: cleaning old entries

This tutorial explains how to setup a cleanup policy to purge old entries in a filesystem.

Note: in robinhood v2.x this was previously implemented as the purge policy of 'robinhood-tmpfs'.

Table of Contents Configuration Policy declaration Policy rules Policy triggers Running the policy From command line

Configuration

Policy declaration

• Include 'tmpfs.inc' in your config file to define the 'cleanup' policy:

 %include "includes/tmpfs.inc"

Policy rules

• Specify your policy targets as fileclasses. Fileclass definitions must ba based on rather static criteria (like owner, path, group...), but should not be based on time criteria (age of last access, last modification, etc.). Time-based criteria will be specified in policy rules.

Examples:

 fileclass log_files {
     definition { type == file and name == "*.log"
                  and tree == "/path/to/logs" }
 }
 fileclass trash {
     definition { tree == "/path/to/trash/" }
 }

• Then specify cleanup rules:
  • To speed up policy run, you can define entries to be ignored using ignore or ignore_fileclass statements (note: only 'ignore' statements with simple expressions like "crit comp value" result in policy run optimlizations, e.g. last_mod < 1d).
  • In policy rule, specify target fileclasses (using target_fileclass statement) and time-bases conditions (in a condition block).
  • You can optionally define a default policy rule to apply to entries that don't match any target fileclass.

Example:

 cleanup_rules {
    ignore { last_access < 1d }
    ignore_fileclass = empty_files;
 
    rule clean_some_files {
        target_fileclass = log_files;
        target_fileclass = foo_files;
        condition { last_mod > 7d }
    }
 
    rule clean_other_files {
        target_fileclass = user_files;
        target_fileclass = bar;
        condition { last_access > 30d }
    }
 
    # default rule (optional):
    # apply to all other entries
    rule default {
        condition { last_access > 60d }
    }
 }

Policy triggers

If you plan to automatically trigger the cleanup policy (regularly, or when the filesystem is full) you need to define policy triggers.

You can trigger the cleanup policy in the following cases:

• When the filesystem is over a high threshold (in terms of space used or total number of entries). This is done by a global_usage trigger. The cleanup policy stops when the specified low threshold is reached.

Example:

 cleanup_trigger {
     trigger_on = global_usage;
     high_threshold_pct = 80%;
     low_threshold_pct = 75%;
     check_interval = 15min;
 }

• In the case of Lustre, when a given OST usage is over a threshold. In this case, the cleanup policy will only apply to the entries on this OST. This is achieved by defining a ost_usage threshold. The cleanup policy stops when the specified low threshold is reached for the OST.

Example:

 cleanup_trigger {
     trigger_on = ost_usage;
     high_threshold_pct = 80%;
     low_threshold_pct = 75%;
     check_interval = 15min;
 }

• When a user or group exceeds a given usage threshold (in volume or entries). This is done by user_usage and group_usage thresholds. In this case, the policy only applies to entries of the given user or group. The cleanup policy stops when the specified low threshold is reached for the user or group. This trigger can be optionally restricted to a given set of groups (see example below).

 cleanup_trigger {
     trigger_on = group_usage(foo*,project01*);
     high_threshold_vol = 105TB;
     low_threshold_vol = 100TB;
     check_interval = 1d;
 }

Running the policy

From command line

• Run a daemon that regularly check the triggers and apply the policies when necessary:

robinhood --run=cleanup -d

• Check cleanup triggers and eventually apply the policy. The program exits after checking all policy triggers.

robinhood --run=cleanup --once

• Match policy rules for all entries. The program exits when the policy run is complete.

robinhood --run=cleanup --target=all

• Match policy rules for a subset of entries:
  • Example 1: apply policy to user 'foo':

robinhood --run=cleanup --target=user:foo

• • Example 2: apply policy to fileclass 'small':

robinhood --run=cleanup --target=class:small

• Limit the number of deleted entries:
  • Example 1: delete 100TB of data

robinhood --run=cleanup --target=all --max-count=1000

• • Example 2: delete 1000 files in OST 23

robinhood --run=cleanup --target=ost:23 --max-count=1000

• • Example 3: delete entries until the FS usage is 80%

robinhood --run=cleanup --target=all --target-usage=80