Bundle of fixes (4-3-stable)

README.md

@@ -43,13 +43,15 @@ $ ils
 ```
 
 ## Requirements
+
 - iRODS v4.3.0
 - irods-dev package
 - irods-runtime package
 - irods-externals-boost package
 - irods-externals-json package
 
 ## Compiling
+
 ```bash
 $ git clone https://github.com/irods/irods_rule_engine_plugin_logical_quotas
 $ cd irods_rule_engine_plugin_logical_quotas
@@ -64,6 +66,7 @@ irods-rule-engine-plugin-logical-quotas-<plugin_version>-<os>-<arch>.<deb|rpm>
 ```
 
 ## Installing
+
 Ubuntu:
 ```bash
 $ sudo dpkg -i irods-rule-engine-plugin-logical-quotas-*.deb
@@ -79,6 +82,7 @@ should be similar to the following:
 ```
 
 ## Configuration
+
 To enable, prepend the following plugin configuration to the list of rule engines in `/etc/irods/server_config.json`. 
 ```javascript
 "rule_engines": [
@@ -115,13 +119,16 @@ The plugin configuration must be placed ahead of all plugins that define any of
 - pep_api_data_obj_put_pre
 - pep_api_data_obj_rename_post
 - pep_api_data_obj_rename_pre
-- pep_api_data_obj_unlink_ppost
+- pep_api_data_obj_unlink_post
 - pep_api_data_obj_unlink_pre
+- pep_api_mod_avu_metadata_pre
 - pep_api_replica_close_post
 - pep_api_replica_close_pre
 - pep_api_replica_open_pre
 - pep_api_rm_coll_post
 - pep_api_rm_coll_pre
+- pep_api_touch_post
+- pep_api_touch_pre
 
 Even though this plugin will process PEPs first due to its positioning, subsequent Rule Engine Plugins (REP) will 
 still be allowed to process the same PEPs without any issues.
@@ -137,6 +144,7 @@ The _data_size_ specific query may result in an overcount of bytes on an activel
 data object having different sizes. For this situation, consider using slightly larger quota limits.
 
 ## How To Use
+
 **IMPORTANT NOTE:** To invoke rules provided by the plugin, the only requirement is that the user be a *rodsadmin*. The *rodsadmin* user
 does not need permissions set on the target collection.
 
@@ -155,6 +163,7 @@ The following operations are supported:
 - logical_quotas_unset_total_size_in_bytes
 
 ### Invoking operations via the Plugin
+
 To invoke an operation through the plugin, JSON must be passed using the following structure:
 ```javascript
 {
@@ -203,12 +212,14 @@ The JSON output will be printed to the terminal and have the following structure
 The **keys** are derived from the **namespace** and **metadata_attribute_names** defined by the plugin configuration.
 
 ### Invoking operations via the Native Rule Language
+
 Here, we demonstrate how to start monitoring a collection just like in the section above.
 ```bash
 $ irule -r irods_rule_engine_plugin-irods_rule_language-instance 'logical_quotas_start_monitoring_collection(*col)' '*col=/tempZone/home/rods' ruleExecOut
 ```
 
 ## Stream Operations
+
 With previous iterations of this plugin, changes in data were tracked and checked for violations across all
 stream-based operations in real-time. However, with the introduction of intermediate replicas and logical locking
 in iRODS v4.2.9, maintaining this behavior became complex. Due to the complexity, the handling of quotas has been
@@ -220,3 +231,17 @@ relaxed. The most important changes are as follows:
 These changes have the following effects:
 - The plugin allows stream-based writes to violate the maximum bytes quota once.
 - Subsequent stream-based creates and writes will be denied until the quotas are out of violation.
+
+## Questions and Answers
+
+### Sometimes, the total number of bytes for my collection doesn't change when I remove a data object. Why?
+
+When it comes to tracking the total number of bytes in use, only **good** replicas are considered. If the data object being removed has no **good** replicas, the plugin will leave the total number of bytes as is. The reason for this is due to there not being a clear path forward for determining which replica's data size should be used for the update. Therefore, the recommendation is for administrators to recalculate the quota totals periodically.
+
+Remember, the plugin is designed to track the totals of **good** replicas only.
+
+### What are the rules around shared monitored nested collections?
+
+Anytime a user performs an operation that results in a quota update, that user MUST have **modify_object** permissions on ALL monitored parent collections. To reduce the management complexity of this, consider the following:
+- Avoid monitoring collections that have parent collections which are already being monitored
+- Use groups to simplify permission management

packaging/test_rule_engine_plugin_logical_quotas.py

@@ -198,7 +198,7 @@ def test_put_collection(self):
 
             # Test: No quota violations on put of a non-empty collection.
             self.logical_quotas_set_maximum_size_in_bytes(sandbox, '100')
-            self.admin1.assert_icommand(['iput', '-rf', dir_path], 'STDOUT', ['pre-scan'])
+            self.admin1.assert_icommand(['iput', '-rf', dir_path])
             expected_number_of_objects = 3
             expected_size_in_bytes = 60
             self.assert_quotas(sandbox, expected_number_of_objects, expected_size_in_bytes)
@@ -272,7 +272,7 @@ def test_copy_collection(self):
             dir_path = os.path.join(self.admin1.local_session_dir, 'col.a')
             file_size = 1
             self.make_directory(dir_path, ['foo.txt'], file_size)
-            self.admin1.assert_icommand(['iput', '-r', dir_path], 'STDOUT', ['pre-scan'])
+            self.admin1.assert_icommand(['iput', '-r', dir_path])
 
             # Monitor first collection.
             col1 = os.path.join(self.admin1.session_collection, 'col.a')
@@ -831,6 +831,92 @@ def test_quotas_do_not_block_non_administrators_from_creating_or_writing_data_ob
             self.assert_quotas(sandbox, expected_number_of_objects = 1,
                                         expected_size_in_bytes = file_size)
 
+    @unittest.skipIf(test.settings.RUN_IN_TOPOLOGY, "Skip for Topology Testing")
+    def test_data_objects_containing_single_quotes_in_data_name_can_be_removed__issue_94(self):
+        config = IrodsConfig()
+
+        with lib.file_backed_up(config.server_config_path):
+            self.enable_rule_engine_plugin(config)
+
+            col = self.user.session_collection
+            self.logical_quotas_start_monitoring_collection(col)
+
+            # Create a non-empty data object.
+            data_object = f"{col}/test_data_objects_containing_single_quote_can_be_removed__issue_94'"
+            contents = 'test_data_objects_containing_single_quote_can_be_removed__issue_94'
+            self.user.assert_icommand(['istream', 'write', data_object], input=contents)
+            self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging.
+            self.assert_quotas(col, expected_number_of_objects=1, expected_size_in_bytes=len(contents))
+            self.user.assert_icommand(['imeta', 'ls', '-C', col], 'STDOUT') # Debugging.
+
+            # Show the data object was removed.
+            self.user.assert_icommand(['irm', '-f', data_object])
+            self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging.
+            self.assertFalse(lib.replica_exists(self.user, data_object, 0))
+            # TODO(#113): expected_size_in_bytes will need to be updated once issue is resolved.
+            self.assert_quotas(col, expected_number_of_objects=0, expected_size_in_bytes=len(contents))
+            self.user.assert_icommand(['imeta', 'ls', '-C', col], 'STDOUT') # Debugging.
+
+    @unittest.skipIf(test.settings.RUN_IN_TOPOLOGY, "Skip for Topology Testing")
+    def test_data_objects_having_only_stale_replicas_can_be_removed__issue_111(self):
+        config = IrodsConfig()
+
+        with lib.file_backed_up(config.server_config_path):
+            self.enable_rule_engine_plugin(config)
+
+            col = self.user.session_collection
+            self.logical_quotas_start_monitoring_collection(col)
+
+            # Create a non-empty data object.
+            data_object = f'{col}/test_data_objects_having_only_stale_replicas_can_be_removed__issue_111.txt'
+            contents = 'test_data_objects_having_only_stale_replicas_can_be_removed__issue_111'
+            self.user.assert_icommand(['istream', 'write', data_object], input=contents)
+            self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging.
+            self.assert_quotas(col, expected_number_of_objects=1, expected_size_in_bytes=len(contents))
+            self.user.assert_icommand(['imeta', 'ls', '-C', col], 'STDOUT') # Debugging.
+
+            # Mark the replica stale.
+            lib.set_replica_status(self.admin1, data_object, 0, 0)
+            self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging.
+            self.assertEqual(lib.get_replica_status(self.user, os.path.basename(data_object), 0), '0')
+
+            # Show the data object was removed and the total data size was left as is.
+            self.user.assert_icommand(['irm', '-f', data_object])
+            self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging.
+            self.assertFalse(lib.replica_exists(self.user, data_object, 0))
+            self.assert_quotas(col, expected_number_of_objects=0, expected_size_in_bytes=len(contents))
+            self.user.assert_icommand(['imeta', 'ls', '-C', col], 'STDOUT') # Debugging.
+
+    @unittest.skipIf(test.settings.RUN_IN_TOPOLOGY, "Skip for Topology Testing")
+    def test_rodsadmin_does_not_need_permission_on_the_collection_to_start_monitoring_it__issue_76(self):
+        config = IrodsConfig()
+
+        with lib.file_backed_up(config.server_config_path):
+            self.enable_rule_engine_plugin(config)
+
+            test_collection = f'{self.admin1.session_collection}/lq_col76'
+            self.admin1.assert_icommand(['imkdir', test_collection])
+
+            try:
+                test_group = 'lq_group76'
+                self.admin1.assert_icommand(['iadmin', 'mkgroup', test_group])
+
+                # Make it so that the group is the only entity that has permissions on the collection.
+                self.admin1.assert_icommand(['ichmod', 'own', test_group, test_collection])
+                self.admin1.assert_icommand(['ichmod', 'null', self.admin1.username, test_collection])
+                self.admin1.assert_icommand(['ils', '-A', test_collection], 'STDOUT') # Debugging.
+
+                # Show the rodsadmin is able to instruct the plugin to start monitoring the collection
+                # even though they don't have permissions set on the data object.
+                json_string = json.dumps({'operation': 'logical_quotas_start_monitoring_collection', 'collection': test_collection})
+                self.admin1.assert_icommand(['irule', '-r', 'irods_rule_engine_plugin-logical_quotas-instance', json_string, 'null', 'null'])
+                self.admin1.assert_icommand(['imeta', 'ls', '-C', test_collection], 'STDOUT') # Debugging.
+
+            finally:
+                self.admin1.run_icommand(['iadmin', 'rmgroup', test_group])
+                self.admin1.run_icommand(['ichmod', '-M', 'own', self.admin1.username, test_collection])
+                self.admin1.run_icommand(['irmdir', test_collection])
+
     #
     # Utility Functions
     #

src/attributes.hpp

@@ -7,34 +7,34 @@
 
 namespace irods
 {
-    class attributes final
-    {
-    public:
-        attributes(const std::string& _namespace,
-                   const std::string& _maximum_number_of_data_objects,
-                   const std::string& _maximum_size_in_bytes,
-                   const std::string& _total_number_of_data_objects,
-                   const std::string& _total_size_in_bytes)
-            : maximum_number_of_data_objects_{fmt::format("{}::{}", _namespace, _maximum_number_of_data_objects)}
-            , maximum_size_in_bytes_{fmt::format("{}::{}", _namespace, _maximum_size_in_bytes)}
-            , total_number_of_data_objects_{fmt::format("{}::{}", _namespace, _total_number_of_data_objects)}
-            , total_size_in_bytes_{fmt::format("{}::{}", _namespace, _total_size_in_bytes)}
-        {
-        }
+	class attributes final
+	{
+	  public:
+		attributes(const std::string& _namespace,
+		           const std::string& _maximum_number_of_data_objects,
+		           const std::string& _maximum_size_in_bytes,
+		           const std::string& _total_number_of_data_objects,
+		           const std::string& _total_size_in_bytes)
+			: maximum_number_of_data_objects_{fmt::format("{}::{}", _namespace, _maximum_number_of_data_objects)}
+			, maximum_size_in_bytes_{fmt::format("{}::{}", _namespace, _maximum_size_in_bytes)}
+			, total_number_of_data_objects_{fmt::format("{}::{}", _namespace, _total_number_of_data_objects)}
+			, total_size_in_bytes_{fmt::format("{}::{}", _namespace, _total_size_in_bytes)}
+		{
+		}
 
-        // clang-format off
-        const std::string& maximum_number_of_data_objects() const { return maximum_number_of_data_objects_; }
-        const std::string& maximum_size_in_bytes() const          { return maximum_size_in_bytes_; }
-        const std::string& total_number_of_data_objects() const   { return total_number_of_data_objects_; }
-        const std::string& total_size_in_bytes() const            { return total_size_in_bytes_; }
-        // clang-format on
+		// clang-format off
+		const std::string& maximum_number_of_data_objects() const { return maximum_number_of_data_objects_; }
+		const std::string& maximum_size_in_bytes() const          { return maximum_size_in_bytes_; }
+		const std::string& total_number_of_data_objects() const   { return total_number_of_data_objects_; }
+		const std::string& total_size_in_bytes() const            { return total_size_in_bytes_; }
+		// clang-format on
 
-    private:
-        std::string maximum_number_of_data_objects_;
-        std::string maximum_size_in_bytes_;
-        std::string total_number_of_data_objects_;
-        std::string total_size_in_bytes_;
-    }; // class attributes
+	  private:
+		std::string maximum_number_of_data_objects_;
+		std::string maximum_size_in_bytes_;
+		std::string total_number_of_data_objects_;
+		std::string total_size_in_bytes_;
+	}; // class attributes
 } // namespace irods
 
 #endif // IRODS_LOGICAL_QUOTAS_ATTRIBUTES_HPP