OpenTSDB 2.5.0 RC1

What's Changed

• Fix race condition in UID lookup by @neilfordyce in #2176
• Bump Logback version to patch CVE-2021-42550 by @manolama in #2208
• Add logback 1.2.9 jar md5s by @eswdd in #2218
• Enhance range tests and regex and add better default for output validator by @eswdd in #2217
• Update some dependencies for #2262. by @manolama in #2263
• Tighten up the regexes for Gnuplot URI params per multiple by @manolama in #2272
• Improved fix for #2261. by @manolama in #2275
• Fix for #2269 and #2267 XSS vulnerability. by @manolama in #2274
• Trigger callback chain when reached limit. by @ugtar in #2204
• Use ConcurrentHashMap when serializing tags. by @bshakur8 in #2282
• Test for null, avoid NPE by @theultimatequestion in #2281
• Update opentsdb_restart.py by @ddoshy in #1515
• Revert "Update opentsdb_restart.py (#1515)" by @johann8384 in #2298

Noteworthy Changes:

• Bump Jackson to 2.14.1 WARNING: The minimum JDK is now version 8 due to Jackson. (#2263)
• Enhance range tests and regex and add better default for output validator (#2217)
• Converting to using SystemD systemctl (#1515)
• Status API, Start of a new HTTP API (#1742)
• Allow end_time = start_time to be able to query and delete a single datapoint. (#2036)
• Add support for splitting rollup queries (#1853)
• ExplicitTags filtering with FuzzyFilters (#1896)
• Add "check_tsd_v2" script (#1567)
• Do PutDataPointRpc.GroupCB asynchronously (#1472)
• Tag Normalization Plugin Support (#1525)

Bug Fixes:

• Fix race condition in UID lookup (#2176)
• Tighten up the regexes for Gnuplot URI params per multiple security reports. (#2272)
• Trigger callback chain when reached limit. (#2204, #839)
• Use ConcurrentHashMap when serializing tags. (#2282, #1055)
• Test for null, avoid NPE (#2281, #2280)
• Ensure we always add the {} for group_by filters, otherwise the non group_by filters act as group_by filters (#1697)
• Added tracking of metrics which are null due to auto_metric being disabled Fixes (#786,#2042)
• Fixed function description Fixes (#841,#2040)
• Fix concurrent result reporting from scanners (#1753)
• Fix SaltScanner race condition on spans maps (#1651)
• Fix an edge case in TsdbQuery.getScanEndTimeSeconds() (#1581, #1582)
• Fix a compilation error about missing FirstDifference (#1471)

Security Fixes:

• Bump Logback version to patch CVE-2021-42550 (#2208, #2218)
• Make sure the inputs are only plain ASCII printables first. Fixes CVE-2018-12972, CVE-2020-35476 (#2275)

New Contributors

• @ugtar made their first contribution in #2204
• @bshakur8 made their first contribution in #2282
• @theultimatequestion made their first contribution in #2281
• @ddoshy made their first contribution in #1515

** Full Changelog**: https://github.com/OpenTSDB/opentsdb/blob/master/NEWS
Full Diff: v2.4.1...v2.5.0RC1

OpenTSDB 2.4.1

• Version 2.4.1 (2021-09-02)

Noteworthy Changes:

• Add a config flag to enable splitting queries across the rollup and
  raw data tables. Another config determines when to split the queries.
• Fix for CVE-2020-35476 that now validates and limits the inputs for
  Gnuplot query parameters to prevent remote code execution.
• Default log config will log CLI tools at INFO to standard out.
• New check_tsd_v2 script that evaluates individual metric groups given
  a filter.
• Collect stats from meta cache plugins.
• Add a python script to list and pretty-print queries running on a TSD.
• Add a single, standalone TSD systemd startup script and default to that
  instead of the multi-port TSD script.

Bug Fixes:

• Fix the "--sync" flag for FSCK to wait for repairs to execute against
  storage before proceeding.
• Fix expression queries to allow metric only and filterless queries.
• Fix an NPE in /api/query/last when appends are enabled.
• Fix races in the salt scanner and multigets on the storage maps.
• Fix rollup queries that need sum and count for downsampling instead of
  group bys.
• Fix fuzzy row filters for later versions of HBase by using filter pairs.
  And allow it to be combined with a regex filter.
• Fix stats from the individual salt scanners and overall query stats
  concurrency issues.
• Rename the stat "maxScannerUidtoStringTime" to
  "maxScannerUidToStringTime"
• Fix the min initial value for double values in the AggregationIterator
• Fix rollup queries producing unexpected results.
• Fix various UTs
• Support rollups in FSCK

OpenTSDB 2.4.0

• Version 2.4.0 (2018-12-16)

Noteworthy Changes:

• Set default data block encoding to DIFF in the create table script.
• Add callbacks to log errors in the FSCK tool when a call was made to
  fix something.
• Add a sum of squares aggregator "squareSum".
• Add the diff aggregator that computes the difference between the first
  and last values.
• Add a SystemD template to the RPM package.
• Allow tags to be added via HTTP header.
• Add example implementations for the Authorization and Authentication
  plugins.
• Change tsd.storage.use_otsdb_timestamp to default to false.
• Literal or filter now allows single character values.
• Rollup query code now only uses the downsampler value to pick an interval.
• Add jdk 8 in the debian script.
• Setup fill policies in the Nagios check

Bug Fixes:

• Fix rollup scanner filter for single aggregate queries.
• Fix FSCK HBase timestamps when deduping. Sometimes they were negative.
• Fix exception handling when writing data over HTTP with the sync flag enabled.
• Fix missing source files in the Makefile.
• Change UID cache to longs from ints and add hit and miss counters.
• Fix HighestCurrent returning the wrong results.
• Fix running query stats queryStart timestamp to millis.
• Fix TimeShift millisecond bug.
• Fix post remove step in the debian package.

OpenTSDB 2.3.2

• Version 2.3.2 (2018-12-16)

Noteworthy Changes:

• A new Python wrapper script to make FSCK repair runs easier.
• Track performance in the Nagios/Icinga script
• Add a Contributions file.
• Add a config, 'tsd.core.bulk.allow_out_of_order_timestamps' to allow out of
  order timestamps for bulk ingest.
• NOTE: This version also includes a JDK 8 compiled version of Jackson due to
  security patches. If you need to run with an older JDK please replace the
  Jackson JARs with older versions.

Bug Fixes:

• Unwrap NoSuchUniqueIds when writing data points to make it easier to understand
  exceptions.
• Fix an NPE in the PutDataPointRpc class if a data point in the list is null.
• Fix a Makefile error in the clean portion.
• Fix an NPOE in the UIDManager print result.
• Fix a bug in the UI where Y formats may contain a percent sign.
• Allow specifying the data block encoding and TTL in the HBase table creation
  script.
• Change the make and TSDB scripts to use relative paths.
• Fix parsing of use_meta from the URI for the search endpoint.
• Fix the clean cache script to be a bit more OS agnostic.

OpenTSDB 2.3.1

• Version 2.3.1 (2018-04-21)

Noteworthy Changes:

• When setting up aggregators, advance to the first data point equal to or greater
  than the query start timestamp. This helps with calendar downsampling intervals.
• Add support to the Nagios check script for downsampling fill policies.

Bug Fixes:

• Fix expression calculation by avoiding double execution and checking both
  output types for boolean values.
• Fixing missing tools scripts in builds.
• Default HBase 1.2.5 in the OSX install script
• Upgrade AsyncBigtable to 0.3.1
• Log query stats when a channel is closed unexpectedly.
• Add the Java 8 path in the debian init script and remove Java 6.
• Pass the column family name to the get requests in the compaction scheduler.
• Fix a comparison issue in the UI on group by tags.
• Filter annotation queries by the starting timestamp, excluding those in a row that
  began before the query start time.
• Tiny stap at purging backticks from Gnuplot scripts.
• Remove the final annotation from the meta classes so they can be extended.
• Fix the javacc maven plugin version.
• Fix the literal or filter to allow single character filters.
• Fix query start stats logging to use the ms instead of nano time.
• Move Jackson and Netty to newer versions for security reasons.
• Upgrade to AsyncHBase 1.8.2 for compatibility with HBase 1.3 and 2.0
• Fix the Highest Current calculation to handle empty time series.
• Change the cache hits counters to longs.

OpenTSDB 2.4.0 RC2

• Version 2.4.0 RC2 (2017-10-08)

Noteworthy Changes:

• Modify the RPC handler plugin system so that it parses only the first part of
  the URI instead of the entire path. Now plugins can implement sub-paths.
• Return the HTML 5 doctype for built-in UI pages
• Add an optional byte and/or data point limit to the amount of data fetched
  from storage. This allows admins to prevent OOMing TSDs due to massive queries.
• Allow a start time via config when enabling the date tiered compaction in HBase
• Provide the option of using an LRU for caching UIDs to avoid OOMing writers and
  readers with too many strings
• Optionally avoid writing to the forward or reverse UID maps when a specific TSD
  operational mode is enabled to avoid wasting memory on maps that will never be
  used.

Bug Fixes:

• Roll back UTF8 issue with UIDs in RC1 wherein the stored bytes weren't converting
  properly and vice-versa. We'll have to work on full UTF8 support in 3.x
• Fix a build issue for Javacc
• Add Kryo as a dependency to the fat jar
• Javadoc fixes
• Fix an issue with calendar aligned downsampling by seeking to the start time of
  the query when the zone-aligned timestamp may be earlier than the query start time
• Add the missing QueryLimitOverride to the makefile
• Fix compatibility with Bigtable for 2.4
• Enable standard read-only APIs when the TSD is in write only mode

OpenTSDB 2.3.0

• Version 2.3.0 (2016-12-31)

Noteworthy Changes:

• Release of 2.3.0.
• Add example classes for using the Java API.

Bug Fixes:

• Same fixes as in 2.2.2
• Fix a null UID check on decoding metric names from row keys.
• Fix unit tests for JDK 8 and later.

See also 2.3.0RC1 and 2.3.0RC2

OpenTSDB 2.2.2

• Version 2.2.2 (2016-12-29)

Bug Fixes:

• Fix an issue with writing metadata where using custom tags could cause the compare-
  and-set to fail due to variable ordering in Java's heap. Now tags are sorted so the
  custom tag ordering will be consistent.
• Fix millisecond queries that would miss data the top of the final hour if the end
  time was set to 1 second or less than the top of that final hour.
• Fix a concurrent modification issue where salt scanners were not synchronized on the
  annotation map and could cause spinning threads.

OpenTSDB 2.3.0 RC2

• Version 2.3.0 RC2 (2016-10-08)

Noteworthy Changes:

• Added a docker file and tool to build TSD docker containers (#871).
• Log X-Forwarded-For addresses when handling HTTP requests.
• Expand aggregator options in the Nagios check script.
• Allow enabling or disabling the HTTP API or UI.
• TSD will now exit when an unrecognized CLI param is passed.

Bug Fixes:

• Improved ALPN version detection when using Google Bigtable.
• Fix the DumpSeries class to support appended data point types.
• Fix queries where groupby is set to false on all filters.
• Fix a missing attribute in the Nagios check script (#728).
• Fix a major security bug where requesting a PNG with certain URI params could execute code
  on the host (#793, #781).
• Return a proper error code when dropping caches with the DELETE HTTP verb (#830).
• Fix backwards compatibility with HBase 0.94 when using explicit tags by removing the
  fuzzy filter (#837).
• Fix an RPM build issue when creating the GWT directory.

OpenTSDB 2.2.1

• Version 2.2.1 (2016-10-08)

Noteworthy Changes

• Generate an incrementing TSMeta request only if both enable_tsuid_incrementing and
  tsd.core.meta.enable_realtime_ts are enabled. Previously, increments would run
  regardless of whether or not the real time ts setting was enabled. If tsuid
  incrementing is disabled then a get and optional put is executed each time without
  modifying the meta counter field.
• Improve metadata storage performance by removing an extra getFromStorage() call.
• Add global Annotations to the gnuplot graphs (#773)
• Allow creation of a TSMeta object without a TSUID (#778)
• Move to AsyncHBase 1.7.2

Bug Fixes:

• Fix Python scripts to use the environment directory.
• Fix config name for "tsd.network.keep_alive" in included config files.
• Fix an issue with the filter metric and tag resolution chain during queries.
• Fix an issue with malformed, double dotted timestamps (#724).
• Fix an issue with tag filters where we need a copy before modifying the list.
• Fix comments in the config file around TCP no delay settings.
• Fix some query stats calculations around averaging and estimating the number
  of data points (#784).
• Clean out old .SWO files (#821)
• Fix a live-lock situation when performing regular expression or wildcard queries (#823).
• Change the static file path for the HTTP API to be relative (#857).
• Fix an issue where the GUI could flicker when two or more tag filters were set (#708).