ChangeLog

nmsg (1.3.1-1)

  * Rely on built-in librdkafka error handling.

nmsg (1.2.0)

  * Add support for Kafka i/o in JSON and binary forms.

  * Add support for prometheus metrics export.

  * Replace mutex protected counters and flags with stdatomic operations.

  * New command line options: --kafkakey, --readkafka, --writekafka, --promport

  * New functions: nmsg_input_open_kafka_endpoint(), nmsg_input_open_kafka_json(),
                   nmsg_output_open_kafka_endpoint(), nmsg_output_open_kafka_json()

nmsg (1.1.2)

  * Now allow nmsgtool --setgroup, --setoperator, and --setsource to
    work with --writepres and --writejson outputs.

  * Permit nmsgtool --setgroup and --setoperator to use a numerical
    value.

  * Unmatched group and operator aliases in presentation format
    now display as the numerical values instead of (null).

  * Ensure mis-running nmsgtool results in failure exit code.

  * Fix output buffering for JSON outputs.

  * Fix magic length calculation in nmsg_json.

  * Terminate strbuf data at initialization.


nmsg (1.1.1)

  * Update estimated container size to include space for sequencing info.

 -- Farsight Security, Inc. <software@farsightsecurity.com>  Mon, 11 Sep 2023 20:21:09 +0000

nmsg (1.1.0)

  * Multiple performance improvements to JSON formatting.

  * Multiple performance and concurrency improvements in nmsg_io loop.

  * Add new dnsobs message type for raw passive DNS observations.

  * Format DNS messages as JSON in base:dnsqr, base:dnstap, and base:dnsobs.

  * Format base:encode payload in human readable form in JSON output.

  * Replace yajl with json-c for JSON parsing.

  * nmsgtool: exit with nonzero status when interrupted by signal.

  * Correct container size calculation when sequencing.

  * Address doxygen deprecations.

 -- Farsight Security, Inc. <software@farsightsecurity.com>  Wed, 16 Aug 2023 18:43:40 +0000

nmsg (1.0.1)

  * nmsgtool: Fix use-after-free bug in statistics debugging
    (seen on Debian 10).

  * nmsg/base/dnsqr.c: Initialize len in dnsqr_hash to squash
    code-checker uninitialized warning.

  * Fix assert conditions with side effects for builds with -DNDEBUG.

nmsg (1.0.0)

  * Replace libxs with zeromq and associated calls.

  * API functions nmsg_input_open_xs(), nmsg_input_open_xs_endpoint(),
    nmsg_io_add_input_xs_channel(), nmsg_output_open_xs(), and
    nmsg_output_open_xs_endpoint() are deprecated and replaced with
    respectively: nmsg_input_open_zmq(), nmsg_input_open_zmq_endpoint(),
    nmsg_io_add_input_zmq_channel(), nmsg_output_open_zmq(), and
    nmsg_output_open_zmq_endpoint().

  * Change format of unknown rrtypes in nmsgtool JSON output to "TYPE" followed
    by number.

  * Update nmsgtool command-line options: --readzchan replaces --readzsock,
    --writezsock replaces --writexsock.

nmsg (0.15.1)

  * Fix output stats collection.

  * Plugins now found by default at $libdir/nmsg instead of lib/nmsg.
    This fixes problem where plugins were not found with recent Debian
    packages built with --libdir configure option (used to install
    libraries in architecture specific directory). And
    ./configure --with-plugindir renamed to --with-pluginsdir.

  * pkgconfig also depend on libprotobuf-c headers (for third-party
    uses of libnmsg with pkgconfig).

 -- Farsight Security, Inc. <software@farsightsecurity.com>  Thu, 23 Jan 2020 13:20:13 +0000

nmsg (0.15.0)

  * Add compile-time and runtime version info to API: NMSG_LIBRARY_VERSION
    and nmsg_get_version(), respectively.

  * Fixed many compiler check warnings.

  * Added multiple unit tests.

  * Added runtime-tunable environment variables NMSG_GRALIAS_FILE and
    NMSG_OPALIAS_FILE.

  * Ensure nmsg_sock_parse_sockspec() returns an error if an invalid
    port range is supplied.

  * Fixed nmsg_io filtering default policy logic. (Previously, if the
    final filter returned "declined", that result was returned in place
    of the configured default policy.)

  * Make nmsg_output_set_buffered() able to unbuffer nmsg_output_type_json
    objects.

  * Fix bugs caught by static analysis (cppcheck).

  * For protobuf compatibility, renamed all protobuf-c generated enum
    names and types, although the wire format of the messages remains the
    same. (Thank you to Chris Morrow.) Compatibility definitions added to
    nmsg/compat.h to allow existing code to continue using the oldnames.

  * Miscellaneous documentation improvements.

 -- Farsight Security, Inc. <software@farsightsecurity.com>  Fri, 01 Nov 2019 14:44:22 -0400

nmsg (0.14.0)

  * Add periodic stats output to nmsgtool.

  * Add .json suffix to json files output by nmsgtool.

 -- Farsight Security, Inc. <software@fsi.io>  Fri, 11 Oct 2019 13:12:46 -0400

nmsg (0.13.2)

  * Restore terminating NUL character in string fields loaded from JSON.

  * Correct display of string fields with missing NUL terminator.

  * Improvements to NUL-termination test cases.

 -- Farsight Security, Inc. <software@fsi.io>  Mon, 22 Jan 2018 13:04:43 -0600

nmsg (0.13.1)

  * Fix crash when formatting empty string fields as JSON.

  * Correct display of empty string fields in presentation output.

  * Remove terminating NUL character in string fields loaded from JSON.

 -- Farsight Security, Inc. <software@fsi.io>  Mon, 23 Oct 2017 14:15:52 -0500

nmsg (0.13.0)

  * Sanitize JSON UTF-8 output. Invalid UTF-8 sequences are replaced
    with the replacement code point U+FFFD.

  * Run msgmod 'load' method when loading a message from raw payload
    or duplicating an existing message.

  * dnstap msgmod: format query_zone in human-readable format.

 -- Farsight Security, Inc. <software@fsi.io>  Thu, 23 Feb 2017 19:31:05 +0000

nmsg (0.12.1)

  * Revert "Don't leak duplicate XS endpoint string in parsing code."

 -- Farsight Security, Inc. <software@fsi.io>  Mon, 28 Nov 2016 18:52:41 +0000

nmsg (0.12.0)

 * Add nmsg_message_get_payload_size() to return the size of the
   serialized payload object.

 * Add dnstap message module support. dnstap is a flexible,
   structured event replication format for DNS software.

 * nmsg_output_set_rate: Do not destroy existing rate as another
   output may be using it. Note the changed behavior: The caller of
   nmsg_output_set_rate() is responsible for reclaiming
   unused nmsg_rate_t objects with nmsg_rate_destroy().

 * Use output lock in nmsg_output_set_rate().

 * Fix sizeof operands in nmsg_chalias_lookup() memory allocations.

 * Fix potential uninitialized filter result in io_run_filters().

 * Replace enum-exhaustive if/else with switch for msgmod.

 * Fix potential nmsg_pres_to_payload crash from unknown fields in msgmod.

 * Remove unnecessary initializations in strbuf code.

 * Add missing cleanup in nmsg_pcap_input_setfilter_raw error returns.

 * Refactor io loop close check to remove unused return value.

 * Fix ignored return value in file input.

 * Don't leak duplicate XS endpoint string in parsing code

 * Fix problems with nmsg_output_set_rate(). It destroys a previously
   provided nmsg_rate. If that rate object is shared with another output,
   activity on the other output will access freed memory formerly
   associated with that nmsg_rate. It does not synchronize setting
   the nmsg_rate object. This can lead to race conditions, e.g. if the
   rate object is set and the old one destroyed while an output is in
   nmsg_rate_sleep().

 -- Farsight Security, Inc. <software@fsi.io> Wed, Nov 23 2016 21:15:55 +0000

nmsg (0.11.2)

  * Fix a build failure when building without YAJL support (#47, #49).

  * dnsqr: Also perform query name filtering for UDP_UNSOLICITED_RESPONSE
    messages (#48).

  * dnsqr: Remove 'icmp' from the generated BPF (#20, #50).

  * dnsqr: Only set 'resolver_address_zeroed' field if addresses were zeroed
    from the underlying query/response packet fields (#51). Resolver address
    zeroing only works for the UDP message types, so we were incorrectly
    setting the 'resolver_address_zeroed' field for TCP and ICMP messages.

  * nmsg-dnsqr2pcap: Also dump ICMP and TCP packets (#52).

 -- Robert Edmonds <edmonds@fsi.io>  Fri, 29 Apr 2016 13:37:40 -0400

nmsg (0.11.1)

  [ Henry Stern ]
  * Use gmtime_r() instead of gmtime() in JSON and text format generators
    (#46).

  [ Robert Edmonds ]
  * Distribute README.FILTERS.md in release tarballs.

 -- Robert Edmonds <edmonds@fsi.io>  Fri, 12 Feb 2016 14:32:07 -0500

nmsg (0.11.0)

  [ Henry Stern ]
  * Add an interval randomization option that randomizes the initial offset
    within the selected time interval. This functionality is exposed via the
    libnmsg nmsg_io_set_interval_randomized() function and the nmsgtool -R /
    --randomize command-line option (#27, #33).

  * Add documentation for nmsgtool -j / --readjson and -J / --write-json
    command-line options (#26, #28).

  * Add PKG_CHECK_MODULES dependency on yajl >= 2.1.0 (#29, #31).

  * Make nmsgtool -k / --kicker work when combined with -c or -t, when
    producing output in JSON format (#25, #38).

  * Fix compiler warning [-Wtautological-compare] in
    _nmsg_msgmod_json_to_payload_load() (#36, #39).

  * Add nmsg_message_get_num_field_values(),
    nmsg_message_get_num_field_values_by_idx() functions (#5, #40).

  [ Robert Edmonds ]
  * Remove the unused enum nmsg_modtype from the internal libnmsg API (#30).

  * Header file cleanups (#14, #34).

  * Rewrite nmsg_res_lookup() to use a switch, which eliminates a Clang
    warning (#14, #35).

  * Add a message filtering capability to the libnmsg I/O loop, including
    external filter module plugin and nmsgtool support (#41, #43, #44).

  [ Mike Schiffman ]
  * Add yajl/ prefix to #include's of yajl headers (#37).

 -- Robert Edmonds <edmonds@fsi.io>  Tue, 02 Feb 2016 18:31:48 -0500

nmsg (0.10.0)

  [ Henry Stern ]
  * Add JSON input and output support, including new libnmsg functions
    nmsg_input_open_json(), nmsg_output_open_json(), nmsg_message_from_json(),
    and nmsg_message_to_json(); new -j and -J command line options for
    nmsgtool; and new per-field 'format' and 'parse' methods in the msgmod
    plugin API.

  [ Chris Mikkelson ]
  * nmsg_chalias_free(): Check for null aliases list.

 -- Robert Edmonds <edmonds@fsi.io>  Thu, 15 Oct 2015 23:12:50 -0400

nmsg (0.9.1)

  * Fix a crash that was caused by incorrect initialization of
    ProtobufCBufferSimple objects.

  * Increase the required verbosity level for several noisy debugging log
    messages in the seqsrc tracking code from 5 (nmsgtool -ddddd) to 6
    (nmsgtool -dddddd).

 -- Robert Edmonds <edmonds@fsi.io>  Fri, 26 Sep 2014 15:20:49 -0400

nmsg (0.9.0)

  * nmsg now depends on protobuf-c version 1.0.1 or newer.

  * The libnmsg msgmod ABI version number has been bumped from 8 to 9. This
    requires external message modules to be recompiled and to be updated for
    the '8' -> '9' msgmod filename change. The msgmod plugin interface now
    verifies that "transparent" (protobuf-based) message modules were compiled
    against the same major version of the protobuf-c library as libnmsg.

  * Internal refactoring of memory allocation and rate limiting functions.

  * Various portability fixes for OS X.

 -- Robert Edmonds <edmonds@fsi.io>  Wed, 13 Aug 2014 13:20:52 -0400

nmsg (0.8.0)

  * Update copyright and license statements as a result of the transition from
    Internet Systems Consortium to Farsight Security.

  * The "ISC" NMSG vendor has been renamed to "base". API/ABI backwards
    compatibility with existing source code and binaries which use the old
    vendor name will be maintained. New code should begin referring to the new
    vendor name and existing code should be updated to refer to the new vendor
    name.

  * Replace the "librsf" submodule with the "libmy" subtree.

  * Fix a double free() which occurred when compression was enabled on a
    libnmsg output and an NMSG payload large enough to cause fragmentation
    before being compressed ended up being small enough to fit in an
    unfragmented NMSG container after compression.

  * The embedded copy of the protobuf-c support library has been removed.
    protobuf-c is now an unconditional build dependency.

  * The generated .pb-c.c/.pb-c.h files that the protobuf-c compiler generates
    have been removed from the nmsg release tarballs. It is now required to
    have the protobuf-c compiler installed in order to build nmsg.

  * The HTML format API documentation generated by Doxygen is no longer
    shipped in the nmsg release tarball, and is now only generated if Doxygen
    was detected during configure.

  * The DocBook manpage documentation is now generated with DocBook 5 and
    xsltproc, rather than with DocBook 4.5 and docbook2x-man.

  * The CRC32C implementation has been replaced with a new version which takes
    advantage of new instructions added to the x86-64 architecture which
    perform the CRC32C algorithm in hardware, if available. On other
    architectures and on x86-64 CPUs where these new instructions are not
    available a fallback software "slicing-by-8" implementation of the CRC32C
    algorithm will be used, which is still faster than the implementation
    shipped in previous nmsg releases. This may speed up NMSG protocol
    decoding significantly on newer CPUs where these hardware instructions are
    available.

  * If the "-T" argument is specified to nmsgtool without specifying "-V",
    nmsgtool now implicitly defaults to assuming "-V base".

  * nmsg_input_read_null(): fix inverted filter logic.
    Patch from Ray Ruvinskiy.

  * nmsg_input_read_null(): fix NULL pointer dereference.
    Patch from Ray Ruvinskiy.

  * Robustness improvements in msgmodset loading.

  * The message module plugin directory is now overridable by setting the
    NMSG_MSGMOD_DIR environment variable.

  * Convert various assertions throughout libnmsg into error returns.

  * Fix various non-portable uses where a NULL character string was passed to
    a printf() family function.

  * nmsg_res_lookup() now returns the string "(unknown libnmsg result code)"
    rather than NULL for unknown result code values.

  * nmsg_output_write() now returns a new result code nmsg_res_errno when an
    underlying call to write() fails and preserves the errno value.

  * Various debugging prints in libnmsg are now only generated when a non-zero
    debugging level is configured. Previously, libnmsg could generate
    debugging output even at the default debug level of 0.

  * nmsgtool's default debugging level is now 1.

  * New function nmsg_get_debug() to retrieve the libnmsg debugging level.

  * Ensure that the "pkt" message type cannot be used with pcap sources that
    do not use the EN10MB pcap datalink type.

  * A new "packet" message type is available which encodes IPv4 and IPv6
    network packets and can process packets from the EN10MB, RAW, LINUX_SLL,
    NULL, and LOOP pcap datalink types. An example program nmsg-packet2pcap
    demonstrates a basic "packet" reader.

  * libnmsg now exposes a new "nmsg_container" interface for marshalling one
    or more nmsg_message_t objects into a serialized NMSG container.

  * The nmsgtool "-L" and "-S" arguments are now documented in the manpage.

  * Fix the autoconf macros that detect DocBook to use the namespaced version
    of the DocBook stylesheets. (Issue #1)

  * New functions nmsg_input_get_count_container_received() and
    nmsg_input_get_count_container_dropped() which retrieve the
    received/dropped container counters for UDP NMSG input streams.
    (Issue #2)

 -- Robert Edmonds <edmonds@fsi.io>  Thu, 13 Mar 2014 12:51:20 -0400

nmsg (0.7.3)

  * The rate-limiting function nmsg_rate_sleep() has been completely
    rewritten in order to sleep much more often and with shorter timeouts.
    At low to moderate event rates, nmsg_rate_sleep() will now result in a
    system-level sleep at almost every invocation.

  * If nmsg_output_set_rate() is used to attach an nmsg_rate_t object to an
    nmsg_output_t object, nmsg_output_close() will now no longer implicitly
    call nmsg_rate_destroy() on the associated nmsg_rate_t object.
    Additionally, internal calls to nmsg_rate_sleep() will now be performed
    while holding the stream output lock. These changes allow library users
    to share an nmsg_rate_t object across more than one output.

  * Rate-limiting in nmsg_output is now performed on a per-packet rather
    than on a per-payload basis.

  * If using rate-limiting on output sockets, nmsgtool will default to a
    "freq" setting of 10 rather than the previous value of 100.

  * If using a port range to specify multiple output sockets while also
    specifying a rate-limit, nmsgtool will use a shared rate-limiter across
    each of the output sockets in the range. Due to this change and the
    switch to per-packet rather than per-payload rate-limiting, the
    semantics of the optional rate parameter to nmsgtool's --writesock
    option have changed.

  * New function nmsg_timespec_add().

 -- Robert Edmonds <edmonds@isc.org>  Fri, 26 Apr 2013 16:50:27 -0400

nmsg (0.7.2)

  * Fix a segfault in nmsgtool when -U (--username) was specified on the
    command line without -P (--pidfile).

  * nmsg can now be built without libxs support by specifying
    "--without-libxs" to configure.

  * Build system: the directory to install the libnmsg pkg-config file to can
    now be overridden by specifying the --with-pkgconfigdir parameter to
    configure.

  * Build system: add "make check" target that runs tests.

  * Build fix: explicitly link examples/nmsg-dnsqr2pcap against libpcap.

  * Portability: use fallback htole32()/le32toh() functions on old systems.

 -- Robert Edmonds <edmonds@isc.org>  Tue, 12 Mar 2013 17:33:22 -0400

nmsg (0.7.1)

  * Fix nmsg_input_open_xs_endpoint() and nmsg_output_open_xs_endpoint() to
    be compatible with newer versions of libxs. libxs >= 1.2.0 is now
    required.

 -- Robert Edmonds <edmonds@isc.org>  Wed, 25 Jul 2012 15:24:40 -0400