feat: Release 3.33 (#2654)

* fix: Non pri cef fix (#2641) * chore(deps): update dependency mkdocs-material to v9.5.42 (#2624) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * chore(deps): update splunk/addonfactory-test-matrix-action action to v2.1.9 (#2620) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix: fix CISE_Alarm messages parsing (#2609) * fix: improve SC4S Dashboard performance (#2592) * docs: Removed reference of Cisco eStreamer for Splunk app from ASA/FTD doc (#2629) * docs: Removed reference of Cisco eStreamer for Splunk app * fix: Updated the regex for non pri cef formated logs such that a '<space>digit' date will be accepted * Updated the regex with some more constraints --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: wojtekzyla <[email protected]> Co-authored-by: mstopa-splunk <[email protected]> * chore(deps): update dependency mkdocs-material to v9.5.47 (#2644) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat: Netapp ontap audit ems support (#2639) * chore(deps): update dependency mkdocs-material to v9.5.42 (#2624) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * chore(deps): update splunk/addonfactory-test-matrix-action action to v2.1.9 (#2620) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix: fix CISE_Alarm messages parsing (#2609) * fix: improve SC4S Dashboard performance (#2592) * docs: Removed reference of Cisco eStreamer for Splunk app from ASA/FTD doc (#2629) * docs: Removed reference of Cisco eStreamer for Splunk app * feat: Added support for ems logs and fixed the existing classification * Updated the documentation and made some changes in the parser * Updating the test file such that all the test cases are passing * Added support in sc4s lite * docs: Added the migration precaution in the upgrade.md file * Removed an addionally created test file and merged my changes to the existing one * Updated the code to maintain the backward compatibility * Updated the test-container workflow to set the newly introduced environment variable's value to 'yes' and made some changes in the test vps parser. --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: wojtekzyla <[email protected]> Co-authored-by: mstopa-splunk <[email protected]> * Added juniper log parsing * Migrated to restricted sqlite --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: wojtekzyla <[email protected]> Co-authored-by: mstopa-splunk <[email protected]> Co-authored-by: sbylica-splunk <[email protected]> Co-authored-by: Szymon Bylica <[email protected]>
splunk · Dec 10, 2024 · 15ed2ce · 15ed2ce
1 parent 6da3060
commit 15ed2ce
Show file tree

Hide file tree

Showing 23 changed files with 348 additions and 104 deletions.
diff --git a/.github/workflows/ci-lite.yaml b/.github/workflows/ci-lite.yaml
@@ -227,6 +227,7 @@ jobs:
           TEST_SC4S_ACTIVATE_EXAMPLES: "yes"
           SC4S_DEBUG_CONTAINER: "yes"
           SC4S_SOURCE_VMWARE_VSPHERE_GROUPMSG: "yes"
+          SC4S_NETAPP_ONTAP_NEW_FORMAT: "yes"
           SC4S_USE_VPS_CACHE: "yes"
     steps:
       - name: Checkout

diff --git a/.github/workflows/ci-main.yaml b/.github/workflows/ci-main.yaml
@@ -227,6 +227,7 @@ jobs:
           TEST_SC4S_ACTIVATE_EXAMPLES: "yes"
           SC4S_DEBUG_CONTAINER: "yes"
           SC4S_SOURCE_VMWARE_VSPHERE_GROUPMSG: "yes"
+          SC4S_NETAPP_ONTAP_NEW_FORMAT: "yes"
           SC4S_USE_VPS_CACHE: "yes"
 
     steps:

diff --git a/docs/sources/vendor/NetApp/ontap.md b/docs/sources/vendor/NetApp/ontap.md
@@ -3,7 +3,7 @@
 ## Key facts
 
 * MSG Format based filter
-* Legacy BSD Format default port 514
+* Netapp Ontap messages are not distinctive. So, either configure known Netapp Ontap hosts in SC4S, or open unique ports for Netapp Ontap devices
 
 ## Links
 
@@ -16,11 +16,49 @@
 
 | sourcetype     | notes                                                                                                   |
 |----------------|---------------------------------------------------------------------------------------------------------|
-| netapp:ems  | None |
+| ontap:ems  | This sourcetype will be assinged only when the environment variable `SC4S_NETAPP_ONTAP_NEW_FORMAT` is not set or is set to 'no'. By default it is unset |
+| netapp:ontap:audit  | This sourcetype will be assinged only when the environment variable `SC4S_NETAPP_ONTAP_NEW_FORMAT` is set to 'yes' |
+| netapp:ontap:ems  | This sourcetype will be assinged only when the environment variable `SC4S_NETAPP_ONTAP_NEW_FORMAT` is set to 'yes' |
 
 ## Sourcetype and Index Configuration
 
 | key            | sourcetype     | index          | notes          |
 |----------------|----------------|----------------|----------------|
-| netapp_ontap      | netapp:ems     | infraops          | none          |
+| netapp_ontap      | ontap:ems     | infraops          | none          |
+| netapp_ontap_audit      | netapp:ontap:audit     | infraops          | none          |
+| netapp_ontap_ems      | netapp:ontap:ems     | infraops          | none          |
 
+## Options
+
+| Variable       | default        | description    |
+|----------------|----------------|----------------|
+| SC4S_NETAPP_ONTAP_NEW_FORMAT      | empty string      | (empty/yes) Set to "yes" for the applying the latest changes. Make sure to configure your system to send the logs to a specific port or have a hostname-based configuration |
+
+## Parser Configuration
+1. Through sc4s-vps
+```c
+#/opt/sc4s/local/config/app-parsers/app-vps-netapp_ontap.conf
+#File name provided is a suggestion it must be globally unique
+
+application app-vps-test-netapp_ontap[sc4s-vps] {
+    filter {
+        host("netapp-ontap-" type(string) flags(prefix))
+        or (
+            message("netapp-ontap-" type(string) flags(prefix))
+            and program("netapp-ontap-" type(string) flags(prefix))
+        )
+    };
+    parser {
+        p_set_netsource_fields(
+            vendor('netapp')
+            product('ontap')
+        );
+    };
+};
+```
+
+2. or through unique port
+```
+# /opt/sc4s/env_file 
+SC4S_LISTEN_NETAPP_ONTAP_UDP_PORT=5005
+```
diff --git a/docs/upgrade.md b/docs/upgrade.md
@@ -18,6 +18,9 @@ For a step by step guide [see here](./v3_upgrade.md).
 
 You may need to migrate legacy log paths or version 1 app-parsers for version 2. To do this, open an issue and attach the original configuration and a compressed pcap of sample data for testing. We will evaluate whether to include the source in an upcoming release.
 
+### Upgrade from <3.33.0
+In NetApp ONTAP, the ontap:ems sourcetype has been updated to netapp:ontap:audit, so old logs are now classified under netapp:ontap:audit. Additionally, a new netapp:ontap:ems sourcetype has been introduced. If you upgrade and want these new changes, ensure that you set `SC4S_NETAPP_ONTAP_NEW_FORMAT` environment variable to `yes` and configure your system to send the logs to a specific port or have a hostname-based configuration in place for proper log onboarding into Splunk.
+
 ### Upgrade from <2.23.0
 
 * In VMware vSphere, update the ESX and vCenter sourcetype for add-on compatibility.

diff --git a/package/etc/conf.d/conflib/netsource/app-netsource-netapp_ontap.conf b/package/etc/conf.d/conflib/netsource/app-netsource-netapp_ontap.conf
@@ -0,0 +1,55 @@
+block parser app-netsource-netapp_ontap() {
+    channel {
+        rewrite {
+            r_set_splunk_dest_default(
+                index("infraops")
+                vendor("netapp")
+                product("ontap")
+            );
+        };
+
+        if {
+            parser {
+                regexp-parser(
+                    prefix(".tmp.")
+                    patterns('^[A-Za-z0-9\-\_\.]+: [0-9a-f]+\.[0-9a-f]+ [0-9a-f]+ [A-Z][a-z][a-z] (?<timestamp>[A-Z][a-z][a-z] \d\d \d\d\d\d \d\d:\d\d:\d\d [+-]?\d{1,2}:\d\d)')
+                );
+                date-parser-nofilter(
+                    format(
+                        '%b %d %Y %H:%M:%S %z',
+                    )
+                    template("${.tmp.timestamp}")
+                );
+            };
+
+            rewrite {
+                set('$PROGRAM: $MESSAGE', value(MESSAGE));
+                set('$PROGRAM', value(HOST));
+                unset(value(PROGRAM));
+            };
+
+            rewrite {
+                r_set_splunk_dest_update_v2(
+                    sourcetype('netapp:ontap:audit')
+                    class('audit')
+                );
+            };
+        } else {
+            rewrite {
+                r_set_splunk_dest_update_v2(
+                    sourcetype('netapp:ontap:ems')
+                    class('ems')
+                );
+            };
+        };
+    };
+};
+
+application app-netsource-netapp_ontap[sc4s-network-source] {
+	filter {
+        match("netapp", value('.netsource.sc4s_vendor'), type(string))
+        and match("ontap", value('.netsource.sc4s_product'), type(string))
+        and "`SC4S_NETAPP_ONTAP_NEW_FORMAT`" eq "yes"
+    };	
+    parser { app-netsource-netapp_ontap(); };
+};
diff --git a/package/etc/conf.d/conflib/raw/app-raw-bsd_nopri.conf b/package/etc/conf.d/conflib/raw/app-raw-bsd_nopri.conf
@@ -53,7 +53,7 @@ block parser app-raw-bsd_nopri() {
 };
 application app-raw-bsd_nopri[sc4s-raw-syslog] {
     filter {
-        message('^\w\w\w \d\d \d\d:\d\d:\d\d ');
+        message('^\w{3} ([0 ][1-9]|[12]\d|3[01]) (0\d|1\d|2[0-3]):([0-5]\d):([0-5]\d) ');
     };
     parser { app-raw-bsd_nopri(); };
 };
diff --git a/package/etc/conf.d/conflib/syslog/app-syslog-juniper_junos_unstructured.conf b/package/etc/conf.d/conflib/syslog/app-syslog-juniper_junos_unstructured.conf
@@ -74,6 +74,7 @@ application app-syslog-juniper_junos_unstructured-pgm[sc4s-syslog-pgm] {
         or program('RT_FLOW' type(string) flags(prefix))
         or program('RT_IDS' type(string) flags(prefix))
         or program('RT_UTM' type(string) flags(prefix))
+        or program('RT_SYSTEM' type(string) flags(prefix))
         or program('Juniper' type(string) flags(prefix))
         or program('rpd' type(string) flags(prefix))
         or program('eswd' type(string) flags(prefix))

diff --git a/package/etc/conf.d/conflib/syslog/app-syslog-netapp_ontap.conf b/package/etc/conf.d/conflib/syslog/app-syslog-netapp_ontap.conf
@@ -37,7 +37,8 @@ block parser app-syslog-netapp_ontap() {
 };
 application app-syslog-netapp_ontap[sc4s-syslog] {
 	filter {
-        program('^[A-Za-z0-9\-\_\.]+$');
+        program('^[A-Za-z0-9\-\_\.]+$')
+        and not "`SC4S_NETAPP_ONTAP_NEW_FORMAT`" eq "yes";
     };	
     parser { app-syslog-netapp_ontap(); };
 };
diff --git a/package/etc/pylib/parser_source_cache.py b/package/etc/pylib/parser_source_cache.py
@@ -2,6 +2,7 @@
 import traceback
 import socket
 import struct
+from sqlitedict import SqliteDict
 
 import time
 
@@ -16,14 +17,15 @@ class LogParser:
     class LogDestination:
         pass
 
+
 def ip2int(addr):
     ip4_to_int = lambda addr: struct.unpack("!I", socket.inet_aton(addr))[0]
-    
+
     def ip6_to_int(addr):
         ip6 = socket.inet_pton(socket.AF_INET6, addr)
         a, b = struct.unpack(">QQ", ip6)
         return (a << 64) | b
-    
+
     try:
         return ip4_to_int(addr)
     except OSError:
@@ -39,7 +41,7 @@ def int_to_ip6(num):
         ip6 = struct.pack(">QQ", a, b)
         addr = socket.inet_ntop(socket.AF_INET6, ip6)
         return addr
-    
+
     try:
         return int_to_ip4(addr)
     except struct.error:
@@ -51,10 +53,8 @@ def int_to_ip6(num):
 
 class psc_parse(LogParser):
     def init(self, options):
-        from sqlite_utils import RestrictedSqliteDict
-
         self.logger = syslogng.Logger()
-        self.db = RestrictedSqliteDict(f"{hostdict}.sqlite")
+        self.db = SqliteDict(f"{hostdict}.sqlite")
         return True
 
     def deinit(self):
@@ -80,11 +80,9 @@ def parse(self, log_message):
 
 class psc_dest(LogDestination):
     def init(self, options):
-        from sqlite_utils import RestrictedSqliteDict
-
         self.logger = syslogng.Logger()
         try:
-            self.db = RestrictedSqliteDict(f"{hostdict}.sqlite", autocommit=True)
+            self.db = SqliteDict(f"{hostdict}.sqlite", autocommit=True)
         except Exception:
             exc_type, exc_value, exc_traceback = sys.exc_info()
             lines = traceback.format_exception(exc_type, exc_value, exc_traceback)
@@ -125,9 +123,7 @@ def flush(self):
 
 
 if __name__ == "__main__":
-    from sqlite_utils import RestrictedSqliteDict
-
-    db = RestrictedSqliteDict(f"{hostdict}.sqlite", autocommit=True)
+    db = SqliteDict(f"{hostdict}.sqlite", autocommit=True)
     db[0] = "seed"
     db.commit()
-    db.close()
+    db.close()
diff --git a/package/etc/pylib/parser_vps_cache.py b/package/etc/pylib/parser_vps_cache.py
@@ -2,6 +2,7 @@
 import traceback
 import socket
 import struct
+from sqlitedict import SqliteDict
 
 import time
 
@@ -16,15 +17,14 @@ class LogParser:
     class LogDestination:
         pass
 
+
 hostdict = str("/var/lib/syslog-ng/vps")
 
 
 class vpsc_parse(LogParser):
     def init(self, options):
-        from sqlite_utils import RestrictedSqliteDict
-
         self.logger = syslogng.Logger()
-        self.db = RestrictedSqliteDict(f"{hostdict}.sqlite")
+        self.db = SqliteDict(f"{hostdict}.sqlite")
         return True
 
     def deinit(self):
@@ -50,11 +50,9 @@ def parse(self, log_message):
 
 class vpsc_dest(LogDestination):
     def init(self, options):
-        from sqlite_utils import RestrictedSqliteDict
-
         self.logger = syslogng.Logger()
         try:
-            self.db = RestrictedSqliteDict(f"{hostdict}.sqlite", autocommit=True)
+            self.db = SqliteDict(f"{hostdict}.sqlite", autocommit=True)
         except Exception:
             exc_type, exc_value, exc_traceback = sys.exc_info()
             lines = traceback.format_exception(exc_type, exc_value, exc_traceback)
@@ -100,4 +98,4 @@ def flush(self):
 
 
 if __name__ == "__main__":
-    pass
+    pass
diff --git a/package/etc/pylib/psc_dump.py b/package/etc/pylib/psc_dump.py
@@ -1,4 +1,3 @@
-
 import sys
 import traceback
 import socket
@@ -9,5 +8,5 @@
 hostdict = str("/var/lib/syslog-ng/cache/hostip")
 db = SqliteDict(f"{hostdict}.sqlite")
 
-for k,v in db.items():
-    print(f"key={k}={v}")
+for k, v in db.items():
+    print(f"key={k}={v}")
diff --git a/package/etc/pylib/sqlite_utils.py b/package/etc/pylib/sqlite_utils.py
diff --git a/package/etc/test_parsers/app-vps-test-netapp_ontap.conf b/package/etc/test_parsers/app-vps-test-netapp_ontap.conf
@@ -0,0 +1,15 @@
+application app-vps-test-netapp_ontap[sc4s-vps] {
+    filter {
+        host("netapp-ontap-" type(string) flags(prefix))
+        or (
+            message("netapp-ontap-" type(string) flags(prefix))
+            and program("netapp-ontap-" type(string) flags(prefix))
+        )
+    };
+    parser {
+        p_set_netsource_fields(
+            vendor('netapp')
+            product('ontap')
+        );
+    };
+};
diff --git a/package/lite/etc/addons/juniper/app-syslog-juniper_junos_unstructured.conf b/package/lite/etc/addons/juniper/app-syslog-juniper_junos_unstructured.conf
@@ -74,6 +74,7 @@ application app-syslog-juniper_junos_unstructured-pgm[sc4s-syslog-pgm] {
         or program('RT_FLOW' type(string) flags(prefix))
         or program('RT_IDS' type(string) flags(prefix))
         or program('RT_UTM' type(string) flags(prefix))
+        or program('RT_SYSTEM' type(string) flags(prefix))
         or program('Juniper' type(string) flags(prefix))
         or program('rpd' type(string) flags(prefix))
         or program('eswd' type(string) flags(prefix))