fix(deps): update module github.com/twmb/franz-go to v1.18.0

[renovate]

This PR contains the following updates:

Package	Change	Age	Adoption	Passing	Confidence
github.com/twmb/franz-go	v1.3.1 -> v1.18.0

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

Release Notes

twmb/franz-go (github.com/twmb/franz-go)

v1.18.0

Compare Source

===

This release adds support for Kafka 3.7, adds a few community requested APIs,
some internal improvements, and fixes two bugs. One of the bugfixes is for a
deadlock; it is recommended to bump to this release to ensure you do not run
into the deadlock. The features in this release are relatively small.

This adds protocol support for KIP-890 and KIP-994, and
adds further protocol support for [KIP-848][KIP-848]. If you are using
transactions, you may see a new kerr.TransactionAbortable error, which
signals that your ongoing transaction should be aborted and will not be
successful if you try to commit it.

Lastly, there have been a few improvements to pkg/sr that are not mentioned
in these changelog notes.

Bug fixes

• If you canceled the context used while producing while your client was
  at the maximum buffered records or bytes, it was possible to experience
  deadlocks. This has been fixed. See #​832 for more details.

• Previously, if using GetConsumeTopics while regex consuming, the function
  would return all topics ever discovered. It now returns only the topics that
  are being consumed.

Improvements

• The client now internaly ignores OutOfOrderSequenceNumber errors that are
  encountered when consuming if possible. If a producer produces very infrequently,
  it is possible the broker forgets the producer by the next time the producer
  produces. In this case, the producer receives an OutOfOrderSequenceNumber error.
  The client now internally resets properly so that you do not see the error.

Features

• AllowRebalance and CloseAllowingRebalance have been added to GroupTransactSession.
• The FetchTopic type now has includes the topic's TopicID.
• The ErrGroupSession internal error field is now public, allowing you to test how you handle the internal error.
• You may now receive a kerr.TransactionAbortable error from many functions while using transactions.

Relevant commits

• 0fd1959d kgo: support Kafka 3.8's kip-890 modifications
• 68163c55 bugfix kgo: do not add all topics to internal tps map when regex consuming
• 3548d1f7 improvement kgo: ignore OOOSN where possible
• 6a759401 bugfix kgo: fix potential deadlock when reaching max buffered (records|bytes)
• 4bfb0c68 feature kgo: add TopicID to the FetchTopic type
• 06a9c47d feature kgo: export the wrapped error from ErrGroupSession
• 4affe8ef feature kgo: add AllowRebalance and CloseAllowingRebalance to GroupTransactSession

v1.17.1

Compare Source

===

This patch release fixes four bugs (two are fixed in one commit), contains two
internal improvements, and adds two other minor changes.

Bug fixes

• If you were using the MaxBufferedBytes option and ever hit the max, odds are
  likely that you would experience a deadlock eventually. That has been fixed.

• If you ever produced a record with no topic field and without using DefaultProduceTopic,
  or if you produced a transactional record while not in a transaction, AND if the client
  was at the maximum buffered records, odds are you would eventually deadlock.
  This has been fixed.

• It was previously not possible to set lz4 compression levels.

• There was a data race on a boolean field if a produce request was being
  written at the same time a metadata update happened, and if the metadata
  update has an error on the topic or partition that is actively being written.
  Note that the race was unlikely and if you experienced it, you would have noticed
  an OutOfOrderSequenceNumber error. See this comment
  for more details.

Improvements

• Canceling the context you pass to Produce now propagates in two more areas:
  the initial InitProducerID request that occurs the first time you produce,
  and if the client is internally backing off due to a produce request failure.
  Note that there is no guarantee on which context is used for cancelation if
  you produce many records, and the client does not allow canceling if it is
  currently unsafe to do so. However, this does mean that if your cluster is
  somewhat down such that InitProducerID is failing on your new client, you
  can now actually cause the Produce to quit. See this comment
  for what it means for a record to be "safe" to fail.

• The client now ignores aborted records while consuming only if you have
  configured FetchIsolationLevel(ReadCommitted()). Previously, the client relied
  entirely on the FetchResponse AbortedTransactions field, but it's possible
  that brokers could send aborted transactions even when not using read committed.
  Specifically, this was a behavior difference in Redpanda, and the KIP that introduced
  transactions and all relevant documents do not mention what the broker behavior
  actually should be here. Redpanda itself was also changed to not send aborted
  transactions when using read committed, but we may as well improve franz-go as well.

• Decompression now better reuses buffers under the hood, reducing allocations.

• Brokers that return preferred replicas to fetch from now causes an info level
  log in the client.

Relevant commits

• 305d8dc kgo: allow record ctx cancelation to propagate a bit more
• 24fbb0f bugfix kgo: fix deadlock in Produce when using MaxBufferedBytes
• 1827add bugfix kgo sink: fix read/write race for recBatch.canFailFromLoadErrs
• d7ea2c3 bugfix fix setting lz4 compression levels (thanks @​asg0451!)
• 5809dec optimise: use byteBuffer pool in decompression (thanks @​kalbhor!)
• cda897d kgo: add log for preferred replicas
• e62b402 improvement kgo sink: do not back off on certain edge case
• 9e32bf9 kgo: ignore aborted txns if using READ_UNCOMMITTED

v1.17.0

Compare Source

===

This long-coming release, four months after v1.16.0, adds support for Kafka 3.7
and adds a few community added or requested APIs. There will be a kadm release
shortly following this one, and maybe a plugin release.

This adds full support for KIP-951, as well as protocol support for
KIP-919 (which has no client facing features) and KIP-848
(protocol only, not the feature!). KIP-951 should make the client faster at
handling when the broker moves partition leadership to a different broker.

There are two fairly minor bug fixes in the kgo package in this release, both
described below. There is also one bugfix in the pkg/sr independent (and
currently) untagged module. Because pkg/sr is untagged, the bugfix was released
a long time ago, but the relevant commit is still mentioned below.

Bug fixes

• Previously, upgrading a consumer group from non-cooperative to cooperative
  while the group was running did not work. This is now fixed (by @​hamdanjaveed, thank you!).

• Previously, if a cooperative consumer group member rebalanced while fetching
  offsets for partitions, if those partitions were not lost in the rebalance,
  the member would call OnPartitionsAssigned with those partitions again.
  Now, those partitions are passed to OnPartitionsAssigned only once (the first time).

Improvements

• The client will now stop lingering if you hit max buffered records or bytes.
  Previously, if your linger was long enough, you could stall once you hit
  either of the Max options; that is no longer the case.

• If you are issuing admin APIs on the same client you are using for consuming
  or producing, you may see fewer metadata requests being issued.

There are a few other even more minor improvements in the commit list if you
wish to go spelunking :).

Features

• The Offset type now has a new method AtCommitted(), which causes the
  consumer to not fetch any partitions that do not have a previous commit.
  This mirrors Kafka's auto.offset.reset=none option.

• KIP-951, linked above and the commit linked below, improves latency around
  partition leader transfers on brokers.

• Client.GetConsumeTopics allows you to query what topics the client is
  currently consuming. This may be useful if you are consuming via regex.

• Client.MarkCommitOffsets allows you to mark offsets to be committed in
  bulk, mirroring the non-mark API CommitOffsets.

Relevant commits

franz-go

• a7caf20 feature kgo.Offset: add AtCommitted()
• 55dc7a0 bugfix kgo: re-add fetch-canceled partitions AFTER the user callback
• db24bbf improvement kgo: avoid / wakeup lingering if we hit max bytes or max records
• 993544c improvement kgo: Optimistically cache mapped metadata when cluster metadata is periodically refreshed (thanks @​pracucci!)
• 1ed02eb feature kgo: add support for KIP-951
• 2fbbda5 bugfix fix: clear lastAssigned when revoking eager consumer
• d9c1a41 pkg/kerr: add new errors
• 54d3032 pkg/kversion: add 3.7
• 892db71 pkg/sr bugfix sr SubjectVersions calls pathSubjectVersion
• ed26ed0 feature kgo: adds Client.GetConsumeTopics (thanks @​UnaffiliatedCode!)
• 929d564 feature kgo: adds Client.MarkCommitOffsets (thanks @​sudo-sturbia!)

kfake

kfake as well has a few improvements worth calling out:

• 18e2cc3 kfake: support committing to non-existing groups
• b05c3b9 kfake: support KIP-951, fix OffsetForLeaderEpoch
• 5d8aa1c kfake: fix handling ListOffsets with requested timestamp

v1.16.1

Compare Source

===

This patch release fixes one bug and un-deprecates SaramaHasher.

SaramaHasher, while not identical to Sarama's partitioner, actually is
identical to some other partitioners in the Kafka client ecosystem. So, the old
function is now un-deprecated, but the documentation correctly points you to
SaramaCompatHasher and mentions why you may still want to use SaramaHasher.

For the bug: if you tried using CommitOffsetsSync during a group rebalance, and
you canceled your context while the group was still rebalancing, then
CommitOffsetsSync would enter a deadlock and never return. That has been fixed.

• cd65d77 and 99d6dfb kgo: fix bug
• d40ac19 kgo: un-deprecate SaramaHasher and add docs explaining why

v1.16.0

Compare Source

===

This release contains a few minor APIs and internal improvements and fixes two
minor bugs.

One new API that is introduced also fixes a bug. API-wise, the SaramaHasher
was actually not a 1:1 compatible hasher. The logic was identical, but there
was a rounding error because Sarama uses int32 module arithmetic, whereas kgo
used int (which is likely int64) which caused a different hash result. A new
SaramaCompatHasher has been introduced and the old SaramaHasher has been
deprecated.

The other bugfix is that OptValue on the kgo.Logger option panicked if you
were not using a logger. That has been fixed.

The only other APIs that are introduced are in the kversions package; they
are minor, see the commit list below.

If you issue a sharded request and any of the responses has a retryable error
in the response, this is no-longer returned as a top-level shard error. The
shard error is now nil, and you can properly inspect the response fully.

Lastly (besides other internal minor improvements not worth mentioning),
metadata fetches can now inject fake fetches if the metadata response has topic
or partition load errors. This is unconditionally true for non-retryable
errors. If you use KeepRetryableFetchErrors, you can now also see when
metadata fetching is showing unknown topic errors or other retryable errors.

• a2340eb improvement pkg/kgo: inject fake fetches on metadata load errors
• d07efd9 feature kversion: add VersionStrings, FromString, V3_6_0
• 8d30de0 bugfix pkg/kgo: fix OptValue with no logger set
• 012cd7c improvement kgo: do not return response ErrorCode's as shard errors
• 1dc3d40 bugfix: actually have correct sarama compatible hasher (thanks @​C-Pro)

v1.15.4

Compare Source

===

This patch release fixes a difficult to encounter, but
fatal-for-group-consuming bug.

The sequence of events to trigger this bug:

• OffsetCommit is issued before Heartbeat
• The coordinator for the group needs to be loaded (so, likely, a previous NOT_COORDINATOR error was received)
• OffsetCommit triggers the load
• a second OffsetCommit happens while the first is still running, canceling the first OffsetCommit's context

In this sequence of events, FindCoordinator will fail with context.Canceled
and, importantly, also return that error to Heartbeat. In the guts of the
client, a context.Canceled error should only happen when a group is being
left, so this error is recognized as a group-is-leaving error and the group
management goroutine exits. Thus, the group is never rejoined.

This likely requires a system to be overloaded to begin with, because
FindCoordinator requests are usually very fast.

The fix is to use the client context when issuing FindCoordinator, rather than
the parent request. The parent request can still quit, but FindCoordinator
continues. No parent request can affect any other waiting request.

This patch also includes a dep bump for everything but klauspost/compress;
klauspost/compress changed go.mod to require go1.19, while this repo still
requires 1.18. v1.16 will change to require 1.19 and then this repo will bump
klauspost/compress.

There were multiple additions to the yet-unversioned kfake package, so that an
advanced "test" could be written to trigger the behavior for this patch and
then ensure it is fixed. To see the test, please check the comment on PR
650.

• 7d050fc kgo: do not cancel FindCoordinator if the parent context cancels

v1.15.3

Compare Source

===

This patch release fixes one minor bug, reduces allocations on gzip and lz4
decompression, and contains a behavior improvement when OffsetOutOfRange is
received while consuming.

For the bugfix: previously, if the client was using a fetch session (as is the
default when consuming), and all partitions for a topic transfer to a different
broker, the client would not properly unregister the topic from the prior
broker's fetch session. This could result in more data being consumed and
discarded than necessary (although, it's possible the broker just reset the
fetch session anyway, I'm not entirely positive).

• fdf371c use bytes buffer instead of ReadAll (thanks @​kalbhor!)
• e6ed69f consuming: reset to nearest if we receive OOOR while fetching
• 1b6a721 bugfix kgo source: use the proper topic-to-id map when forgetting topics

v1.15.2

Compare Source

===

This patch release fixes two bugs and changes Mark functions to be no-ops when
not using AutoCommitMarks to avoid confusion. This also includes a minor commit
further improving the sticky balancer. See the commits for more details.

• 72778cb behavior change kgo: no-op mark functions when not using AutoCommitMarks
• e209bb6 bugfix kgo: pin AddPartitionsToTxn to v3 when using one transaction
• 36b4437 sticky: further improvements
• af5bc1f bugfix kgo: be sure to use topics when other topics are paused

v1.15.1

Compare Source

===

This patch release contains a bunch of internal improvements to kgo and
includes a bugfix for a very hard to encounter logic race. Each improvement
is a bit focused on a specific use case, so I recommend reading any relevant-to-you
commit message below.

As well, the kversion package now detects Kafka 3.6, and the kgo package now
handles AddPartitionsToTxn v4 (however, you will probably not be issuing this
request).

Lastly, this release is paired with a minor kadm release, which adds the
ErrMessage field CreateTopicsResponse and DeleteTopicsResponse, and,
importantly, fixes a data race in the ApiVersions request.

franz-go

• 2a3b6bd improvement kversion: detect 3.6
• fe5a660 improvement kgo: add sharding for AddPartitionsToTxn for KIP-890
• b2ccc2f improvement kgo: reintroduce random broker iteration
• 54a7418 improvement kgo: allow PreTxnCommitFnContext to modify empty offsets
• c013050 bugfix kgo: avoid rare panic
• 0ecb52b improvement kgo: do not rotate the consumer session when pausing topics/partitions
• 1429d47 improvement sticky balancer: try for better topic distribution among members

kadm

• 1955938 bugfix kadm: do not reuse ApiVersions in many concurrent requests
• 66974e8 feature kadm: include ErrMessage in topic response

v1.15.0

Compare Source

===

This release comes 74 days (just over two months) since the last minor release.
This mostly contains new features, with one relatively minor but important bug
addressed (and one very minor bug fixed).

Bug fixes

• Fetch sessions now properly send forgotten topics if we forget the entire
  topic (not just individual partitions while other partitions are still
  consumed on the broker). For long-running clients where partitions move
  around the cluster a bunch over time, this ensures we are not sending requests
  with null topics / null topic IDs. See #​535
  for more details.
• If the client talks to an http endpoint, we now properly detect the bytes
  'HTTP' and send a more relevant error message. This previously existed, but
  used the wrong int32 internally so 'HTTP' was not properly detected (thanks
  @​alistairking!).

Features

• RecordReader now supports %v{json} to parse json inputs
• RecordReader now supports %v{}, an empty no-op formatting directive
• Adds PurgeTopicsFromProducing and PurgeTopicsFromConsuming to purge either
  the producing or consuming half of the client, if you are producing to and
  consuming from the same topics in the same client.
• The new ConsiderMissingTopicDeletedAfter option allows finer grained
  control for how long a regexp-discovered topic can be missing on the cluster
  before the topic is considered deleted (and then internally purged in the
  client)
• Adds NewErrFetch to create a single-error Fetches, for use in end-user
  tests / stubs.
• The new MaxBufferedBytes option can control how many bytes are buffered
  when producing, an option similar to MaxBufferedRecords.
• Adds BufferedFetchBytes and BufferedProduceBytes to return the total
  bytes in records buffered (note this counts only keys, values, and headers).
• Adds PreTxnCommitFnContext to allow custom Metadata annotation for
  transactional commits.
• Adds LeaveGroupContext to control how long leaving a group can take, as
  well as to return any leave group error.

Relevant commits

franz-go

• 4dcfb06 feature kgo: add LeaveGroupContext
• 910e91d and 60b601a feature kgo: add PreTxnCommitFnContext
• c80d6f4 feature kgo: add Buffered{Fetch,Produce}Bytes
• 304559f feature kgo: support MaxBufferedBytes
• 310a5da feature kgo: add NewErrFetch
• 504a9d7 feature kgo: expose ConsiderMissingTopicDeletedAfter
• 253e1a9 feature kgo: add PurgeTopicsFrom{Producing,Consuming}
• 055e2d8 improvement kgo record formatter: accept %v{} to be a no-op (plain read/format)
• 37edfb9 improvement kgo.RecordReader: support %v{json} to read json values
• 8a9a459 bugfix kgo: track topic IDs in the fetch session
• 9d25d3a bugfix kgo: fix typo in parseReadSize to properly detect and warn about talking to HTTP endpoints (thanks @​alistairking!)

kadm

This release comes with a corresponding kadm release that contains a few
behavior changes and improvements.

• bfd07b2 kadm: fix occasionally empty topic/partitions in Lag
• 00ac608 kadm: change FetchOffsetsForTopics to only return requested topics by default

v1.14.4

Compare Source

===

This small patch fixes kversion.VersionGuess to properly guess versions against
zookeeper broker versions v2.7 through 3.4. See the commit for details.

• 5978156 bugfix kversion: fix version detection for Kafka v2.7 through 3.4

v1.14.3

Compare Source

===

This patch fixes regex consuming a deleted topic causing an unending internal
loop of metadata reloading (trying to discover where the topic went).

• 627d39a bugfix kgo: fix / improve handling deleted topics while regex consuming

v1.14.2

Compare Source

===

This patch fixes an internal logic race that can be easily encountered when
specifying exact offsets to consume from. If you encountered this bug, your
consumer could just stop consuming for an indeterminite amount of time. This
bug has existed for a long time and relies on both the client being slow and
the broker being fast to hit.

• 1f696ca bugfix kgo: avoid a consumer logic race where the consumer stops consuming

v1.14.1

Compare Source

===

This patch release, quick on the heels of v1.14.0, fixes a race condition
introduced in v1.14 in the PauseFetchTopics and PauseFetchPartitions
functions, a second race condition that can occur when purging a topic, and
fully addresses [#​493][https://github.com/twmb/franz-go/issues/493](https://redirect.github.com/twmb/franz-go/issues/493)3] which was
not completely addressed in v1.14.0.

• 8c785fa bugfix kgo: fix race between client closing and purging
• dc5283e kgo: re-fix #​493, supporting other buggy clients, and add a test
• 32ac27f bugfix kgo: ensure assignPartitions is locked when pausing topics/partitions

v1.14.0

Compare Source

===

This release contains a few new APIs, one behavior change, and one minor bugfix.

Bug fixes

Previously, HookBrokerRead and HookBrokerE2E could not be used at the same
time. This has been fixed.

Behavior changes

PauseFetch{Topics,Partitions} now causes the client to drop all buffered
fetches and kill all in-flight fetch requests. Importantly, this also means
that once you pause, it is no longer possible for what you paused to be
returned while polling. Previously, the client made no attempt to clear
internal buffers / in flight requests, meaning you could receive paused data
for a while.

Seed brokers now show up in logs as seed_### rather than seed ### (an
underscore has been added).

Features

• kgo.Offset now has an EpochOffset getter function that allows access
  to the actual epoch and offset that are inside the opaque Offset type.
• AddConsumePartitions allows adding individual partitions to consume, and
  the new counterpart RemoveConsumePartitions allows removing individual
  partitions from being consumed. Removing is different from purging, please
  see the docs.
• KeepRetryableFetchErrors bubbles up retryable errors to the end user that
  are encountered while fetching. By default, these errors are stripped.
• kversion now supports Kafka 3.5
• kversion now supports version guessing against KRaft by default
• kgo.DialTLS now exists to even more easily opt into TLS.
• kgo.Client.Opts now exists to return the original options that were used
  to configure the client, making initializing new clients easier.
• kgo.NodeName returns a string form of a broker node name. Internally, seed
  brokers use math.MinInt32 for node IDs, which shows up as massively negative
  numbers in logs sometimes. NodeName can help convert that to seed_<#>.

Relevant commits

• c3b083b improvement kgo: do not returned paused topics/partitions after pausing
• e224e90 bugfix kgo: allow HookBrokerRead and HookBrokerE2E to both be called
• 875761a feature kgo Offset: add EpochOffset getter field
• c5d0fc5 kgo: add a debug log for stripping retryable errors from fetches
• b45d663 kgo: add more context to opportunistic metadata loads while fetching
• 9dae366 kgo: allow retries on dial timeouts
• 00e4e76 kgo: tolerate buggy v1 group member metadata
• 34c8b3d feature kgo: add AddConsumePartitions and RemoveConsumePartitions
• b5cafba sasl: validate non-empty user/pass/token
• 76d2e71 feature kgo: add KeepRetryableFetchErrors
• 0df3ec0 kgo: fix new niche CI problem against Kafka 3.5
• 8ff1d0d feature pkg/kversion: attempt to guess KRaft by default as well
• e92f5d9 feature pkg/kversion: detect v3.5
• f1b923e feature kgo: add DialTLS option
• 9667967 feature kgo.Client: add Opts to return the original opts, to allow seeding new clients
• 8e14928 feature kgo: add NodeName for easy user formatting of internal seed node IDs

v1.13.6

Compare Source

===

This minor patch release allows an injected fake fetch to be returned if that
is the only fetch ready to be returned. A fake fetch is a fetch that actually
does not contain any data, but instead contains an error (and may contain an
empty topic with a single 0 partition). Previously, a fake fetch could only be
returned if actual data was also ready to be returned. Now, a fake fetch can
always be returned. Fake fetches are most commonly experienced if you are
unable to join a group -- the client injects a fake fetch to notify you, the
client, that an error occurred. This patch actually allows you to consume that
error.

• 5c87ce0 consumer: return from Poll if the only error is an injected fake

v1.13.5

Compare Source

===

This tiny patch release relaxes SASL/PLAIN to ignore any server response. This
patch is only useful if you are using this client against Tencent (at least as
we known at this moment). A normal broker does not reply to a successful PLAIN
auth with any data, but Tencent apparently does. Sarama and the Kafka client
itself both seem to ignore extra data once auth is successful, and if a broker
actually rejected the auth then the broker would close the connection, so
ignoring this data seems fine.

• 3addecc sasl plain: ignore any challenge data

v1.13.4

Compare Source

===

This bugfix release fixes a race condition when calling client.Close.
This bug was introduced in commit e45cd72 in
release v1.13.0. If a metadata request discovers a new broker as the client is
closing, there is a tiny window where the metadata goroutine can update a map
that the Close function is also concurrently accessing. The commit in this
release fixes that by guarding the Close access with a mutex.

• ee3d7c1 bugfix kgo: fix race condition in close (thanks @​matino

v1.13.3

Compare Source

===

This minor patch release adds a few internal improvements and fixes no bugs.

Release 1.13.2 patched commits from occuring between join and sync; this patch
extends that to blocking LeaveGroup or Close if an active join and sync is
occurring. This is necessary because revoke callbacks are called while leaving
a group, and usually, you commit in a revoke callback.

A few extra logs are added, some areas of code are better, and a
context.Canceled error that could occasionally be returned in fetches when an
internal consumer session changed is no longer returned.

This release went through three+ straight days of looped integration testing on
two laptops to track down extremely rare test failures; those have been tracked
down.

Lastly, if you use a regex consumer and delete topics, the client now
internally purges and stops fetching those topics. Previously, any topic
discovered while regex consuming was permanently consumed until you manually
called PurgeTopicsFromClient.

• bb66f24 kgo: purge missing-from-meta topics while regex consuming
• f72fdaf kgo: always retry on NotLeader for sharded requests
• 682d1f8 kgo: add info log when the client is throttled
• 88fa883 kgo: avoid counting pinReq version failures against retries
• de53fda kgo: add a bit more context to sharded logs, avoid info log on Close
• 7338bcf kgo: avoiding context.Canceled fetch from List/Epoch, improve testing&logs
• 055b349 consumer: do not use the partition epoch when assigning offsets
• d833f61 group consuming: block LeaveGroup between join&sync

v1.13.2

Compare Source

===

This patch improves the behavior of committing offsets while a rebalance is
happening.

In Kafka, if a commit arrives on the broker between JoinGroup and SyncGroup,
the commit is rejected with REBALANCE_IN_PROGRESS. If a commit is started
before JoinGroup but arrives to the broker after SyncGroup, the commit is
rejected with INVALID_GENERATION.

This patch changes how committing offsets interacts with join&sync: a client
that is joining the group will block OffsetCommit (preventing the first error),
and issuing an OffsetCommit will block the client from entering JoinGroup
(preventing the second error).

Previously, you could occasionally encounter these errors with the
cooperative-sticky group balancer because the standard behavior for eager
balancers is to commit in OnPartitionsRevoked before all partitions are dropped
at the end of a group session. You now should no longer encounter these errors
during a commit at all with any balancer unless something is going quite wrong
(massive client timeouts or broker hangs).

This does mean that OffsetCommit may take longer if you happen to commit in
while a client is joining and the join&sync takes a long time.

• ee70930 kgo groups: block join&sync while a commit is inflight

v1.13.1

Compare Source

===

This patch release fixes a bug where a producer could enter a deadlock if a
topic is deleted and recreated very quickly while producing.

• 769e02f producer: avoid deadlock when when quickly recreating a topic

v1.13.0

Compare Source

===

This release contains a few new APIs, two rare bug fixes, updates to plugins,
and changes the library to now require 1.18.

Go version

This library has supported Go 1.15 since the beginning. There have been many
useful features that this library has not been able to use because of continued
backcompat for 1.15. There is really no reason to support such an old version
of Go, and Go itself does not support releases prior to 1.18 -- and 1.18 is
currently only supported for security backports. Switching to 1.18 allows this
library to remove a few 1.15 / 1.16 backcompat files, and allows switching this
library from interface{} to any.

Behavior changes

If group consuming fails with an error that looks non-retryable, the error is
now injected into polling as a fake errored fetch. Multiple people have ran
into problems where their group consumers were failing due to ACLs or due to
network issues, and it is hard to detect these failures: you either have to pay
close attention to logs, or you have to hook into HookGroupManageError. Now,
the error is injected into polling.

Bug fixes

This release contains two bug fixes, one of which is very rare to encounter, and
one of which is very easy to encounter but requires configuring the client in
a way that (nearly) nobody does.

Rare: If you were using EndAndBeginTransaction, there was an internal race that
could result in a deadlock.

Rare configuration: If you configured balancers manually, and you configured
CooperativeSticky with any other eager balancer, then the client would
internally sometimes think it was eager consuming, and sometimes think it
was cooperative consuming. This would result in stuck partitions while
consuming.

Features

• HookClientClosed: A new hook that allows a callback when the client is closed.
• HookProduceRecordPartitioned: A new hook that is called when a record's partition is chosen.
• Client.ConfigValue: Returns the value for any configuration option.
• Client.ConfigValues: Returns the values for any configuration option (for multiple value options, or for strings that are internally string pointers).
• kadm: a few minor API improvements.
• plugin/klogr: A new plugin that satisfies the go-logr/logr interfaces.
• pkg/kfake: A new experimental package that will be added to over time, this mocks brokers and can be used in unit testing (only basic producing & consuming supported so far).

Relevant commits

• 1b229ce kgo: bugfix transaction ending & beginning
• 461d2ef kgo: add HookClientClosed and HookProduceRecordPartitioned
• 3a7f35e kgo.Client: add UpdateFetchMaxBytes
• b0fa1a0 kgo.Client: add ConfigValue and ConfigValues
• b487a15 kgo: inject the group lost error into polling as *ErrGroupLost
• bc638b0 plugin/kotel.Tracer: add KeyFormatter, only accept utf8 keys
• a568b21 bugfix kgo: do not default to eager if there is any eager balancer
• bf20ac0 plugin/kzap: support LogLevelNone
• a9369be kadm: add state config altering using the older AlterConfigs request
• f5ddf71 kadm: add NumPartitions,ReplicationFactor,Configs to CreateTopicResponse
• e45cd72 consuming: close fetch sessions when closing the client
• d8230ca plugin/klogr: support for go-logr
• 0f42e43 check hooks and flatten slices of hooks

v1.12.1

Compare Source

===

This patch adds back the pre-v1.11.7 behavior of returning UNKNOWN_TOPIC_ID
when consuming. Patch v1.11.7 explicitly stripped this error because it was
found during testing that Kafka 3.4 now returns the error occasionally when
immediately consuming from a new topic (i.e. in tests), and the error is
explicitly marked as retryable within the Kafka source code. However, with
topic IDs, there is no way for a client to recover if the topic has been
deleted and recreated. By returning the error, end users can notified that
their client is in a fatal state and they should restart. In the future, I will
add an option to consume recreated topics.

• efe1cdb consumer: return UnknownTopicID even though it is retryable

v1.12.0

Compare Source

===

This release is tested against Kafka 3.4, and kversion can now properly version
guess 3.4. There are two external kgo features in this release and there is one
internal feature.

Externally,

• Client.CommitMarkedOffsets makes committing marked offsets a bit easier
• Client.UpdateSeedBrokers allows for updating seed brokers, which can be useful on extremely long lived clients.

Internally,

• KIP-792 improves fencing of zombie group consumers using cooperative rebalancing

• c5f86ea feature kgo.Client: add UpdateSeedBrokers(...string) error
• 3e45339 internal improvement group balancer: support KIP-792
• fe727f8 feature kversion: cut Kafka 3.4
• 6751589 feature pkg/kgo: add Client.CommitMarkedOffsets (thanks @​celrenheit!)

v1.11.7

Compare Source

===

The integration test introduced in v1.11.6 was frequently failing in CI. After
some investigation, the panic is caused from PurgeTopicsFromClient while a
load of offsets was happening (i.e. consuming was being initialized). That
race is specifically easily triggered in the integration test.

This patch also relaxes the error stripping while consuming. While consuming,
retryable errors are stripped. The errors that were stripped was a small
specific set that was originally taken from Sarama (nearly 4yr ago). Kafka 3.4
returns a new retryable error frequently if consuming against a topic
immediately after the topic is created. This patch changes the error set to
just strip all retryable errors, since the kerr package already indicates which
errors are retryable. This is more compatible against the broker returning
errors that are retryable.

This is the last commit in the v1.11 series -- immediately after tagging this,
I will be working on v1.12 which will officially test against Kafka 3.4.

• 17567b0..78a12c3 bugfix consumer: fix potential panic when calling PurgeTopicsFromClient
• 17567b0 improvement kgo: always strip retryable errors when consuming

v1.11.6

Compare Source

===

v1.11.5 introduced a regression when:

• Using a direct consumer (not a consumer group)
• Using ConsumePartitions
• Not specifying all partitions in the topic

With the above setup, 1.11.5 introduced a bug that would consume all partitions
in the topic, i.e. all partitions in a topic that were not specified in
ConsumePartitions.

This patch restores the old behavior to only consume partitions that are
requested. This patch also makes it possible to purge topics or partitions from
the direct consumer, and makes it possible to re-add a topic. Lastly, three
integration tests were added to ensure that the problems fixed in 1.11.5 and in
this patch do not crop up again.

• edd0985 bugfix kgo: patch ConsumePartitions regression from 1.11.5

v1.11.5

===

v1.11.4 is retracted and because it actually requires a v2 release. When a
breaking kmsg release must happen, v2 of franz-go will happen and I will make a
change thorough enough such that major kmsg changes will not require a major
franz-go change again.

When this repo was initially written, the Kafka documentation for the wire
serialization of a Kafka record was wrong and different from how things were
actually implemented in the Java source. I used the documentation for my own
implementation. The documentation was later fixed.

What this means is that in this repo, kmsg.Record.TimestampDelta should be
changed from an int32 to int64. Rather than release a major version of kmsg, I
have added a TimestampDelta64 field.

The old 32 bit timestamp delta meant that this package could only represent
records that had up to 24 days of a delta within a batch. Apparently, this may
happen in compacted topics.

More minor: previously, AddConsumeTopics did not work on direct consumers nor
on a client that consumed nothing to begin with. These shortcomings have been
addressed.

• 12e3c11 bugfix franz-go: support 64 bit timestamp deltas
• f613fb8 bugfix pkg/kgo: patch AddConsumeTopics

v1.11.3

Compare Source

===

This patch release fixes a panic that can occur when fetching offsets in the
following scenario:

• You are fetching offsets for a group that the client has not yet loaded
  internally
• The internal load's FindCoordinator request fails OR the group cannot be
  loaded

FindCoordinator usually does not fail outright because the request internally
retries. As well, the group load is usually successful. Group loading only
fails if you are unauthorized to describe the group or if the group coordinator
is not loaded.

The most common case to encounter this error is when you issue a group request
against a new cluster. The first time a group request is seen, the group
coordinator loads. While loading, group requests are failed with
COORDINATOR_LOAD_IN_PROGRESS or some other similar error.

• 5289ef6 bugfix kgo.Client: avoid panic in OffsetFetchRequest when coordinator is not loaded

v1.11.2

Compare Source

===

This patch release fixes HookFetchRecordUnbuffered never being called if a
hook also implemented HookFetchRecordBuffered. No existing plugin currently
implements these hooks (though one will soon), so this patch is only relevant
to you if you manually have added these hooks.

• 2a37df9 bugfix kgo: patch HookFetchRecordUnbuffered

v1.11.1

Compare Source

===

This patch release fixes a bug in ConsumePreferringLagFn. The code could
panic if you:

• Consumed from two+ topics
• Two of the topics have a different amount of partitions
• The single-partition topic has some lag, the topic with more partitions has
  one partition with no lag, and another partition with more lag than the
  single-partition topic

In this case, the code previously would create a non-existent partition to
consume from for the single-partition topic and this would immediately result
in a panic when the fetch request was built.

See the commit for more details.

• 38f2ec6 bugfix pkg/kgo: bugfix ConsumePreferringLagFn

v1.11.0

Compare Source

===

This is a small release containing two minor features and a few behavior
improvements. The MarkedOffsets function allows for manually committing
marked offsets if you override OnPartitionsRevoked, and
UnsafeAbortBufferedRecords allows for forcefully dropping anything being
produced (albeit with documented caveats and downsides)

The client now guards against a broker that advertises FetchRequest v13+ (which
only uses TopicIDs) but does not actually return / use TopicIDs. If you have
an old IBP configured, the broker will not use TopicIDs even if the broker
indicates it should. The client will now pin fetching to a max version of 12 if
a topic has no TopicID.

The client now sets a record's Topic field earlier to DefaultProduceTopic,
which allows the Topic field to be known present (or known non-present) in
the OnRecordBuffered hook.

Lastly, we now universally use Go 1.19 atomic types if compiled with 1.19+. Go
uses compiler intrinsics to ensure proper int64 / uint64 alignment within
structs for the atomic types; Go does not ensure plain int64 / uint64 are
properly aligned. A lot of work previously went into ensuring alignment and
having a GitHub workflow that ran go vet on qemu armv7 emulation, but
apparently that was not comprehensive enough. Now, if you use a 32 bit arch, it
is recommended to just compile with 1.19+.

• d1b6897 feature kgo: add UnsafeAbortBufferedRecords
• d0c42ad improvement kgo source: do not use fetch topic IDs if the broker returns no ID
• cc3a355 and a2c4bad improvement kgo: universally switch to 1.19's atomics if on Go 1.19+
• 66e626f producer: set Record.Topic earlier
• 3186e61 feature kgo: add MarkedOffsets function

v1.10.4

Compare Source

===

This patch release fixes two bugs introduced with v1.10.0. These bugs are not
encountered in when using the client to simply consume or produce. Only admin
usages of the client may encounter the bugs this patch is fixing.

v1.10.0 introduced support for batch offset fetching or coordinator finding.
These changes introduced a bug where empty coordinator keys (i.e., group names
or transactional IDs) would be stripped from requests, and then a field in a
nil pointer could be accessed and panic the program. These changes also
introduced a bug that did not properly mirror one field for batched
FindCoordinator requests.

• ca67da4 bugfix kgo: fix batch coordinator fetching
• c6f7f9a bugfix kgo: allow empty groups when finding coordinator / fetching offsets

v1.10.3

Compare Source

===

This small patch release is another attempted fix at #​239.
It is only possible to encounter this bug if a broker completely dies and never
comes back, and you do not replace the broker (i.e., broker 3 dies and it is
just gone forever).

Previously, kgo would cache the broker controller until NOT_CONTROLLER is
seen. We now clear it a bit more widely, but this is just extra defensive
behavior: the controller is updated on every metadata request.

Worse however, kgo previously cached group or transactional-id coordinators
until COORDINATOR_NOT_AVAILABLE, COORDINATOR_LOAD_IN_PROGRESS, or
`NOT_CO

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[renovate]

ℹ Artifact update notice

File name: go.mod

In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

• 3 additional dependencies were updated
• The go directive was updated for compatibility reasons

Details:

Package	Change
go	1.17 -> 1.23.3
github.com/twmb/franz-go/pkg/kmsg	v0.0.0-20220114004744-91b30863ac2f -> v1.9.0
github.com/klauspost/compress	v1.13.6 -> v1.17.8
github.com/pierrec/lz4/v4	v4.1.11 -> v4.1.21