For 1.x release notes, please see v1.x/CHANGELOG.md
- #1317 Add enablePriorityLeaseAssignment config
- #1320 Update lease taker to get unassigned leases
- #1318 Internally construct and use stream ARNs for all streams in multi-stream mode
- #1291 Update RetrievalFactory implementations to utilize the StreamIdentifier field of StreamConfig
- #1308 Move shutdownComplete call to ShardConsumer
- #1313 Add additional integration tests for multi-stream and cross account access
- #1273 Optimize currentStreamConfigMap by cleaning up lingering streams
- #1302 Fix gracefulShutdown behavior in Scheduler
- #1278 Upgrade awssdk.version from 2.25.3 to 2.25.11
- #1279 Upgrade org.apache.maven.plugins:maven-gpg-plugin from 3.1.0 to 3.2.1
- #1280 Upgrade org.apache.commons:commons-lang3 from 3.12.0 to 3.14.0
- #1282 Upgrade org.apache.maven.plugins:maven-javadoc-plugin from 3.5.0 to 3.6.3
- #1277 Reuse 'ShardSyncTaskManager' instance for existing stream to avoid duplicate enqueue of 'ShardSyncTask'
- #1275 Update PollingConfig maxRecords method to return PollingConfig
- #1236 Upgrade commons-io:commons-io from 2.11.0 to 2.15.1
- #1189 Upgrade org.apache.maven.plugins:maven-resources-plugin from 3.3.0 to 3.3.1
- #1139 Upgrade maven-surefire-plugin from 2.22.2 to 3.1.2
- #1138 Upgrade maven-failsafe-plugin from 2.22.2 to 3.1.2
- #1125 Upgrade maven-gpg-plugin from 3.0.1 to 3.1.0
- #1271 Adding snapshot for 2.5.6-SNAPSHOT
- #1268 Upgrade ch.qos.logback:logback-classic dependency from 1.3.12 to 1.3.14
- #1268 Upgrade awssdk.version from 2.20.43 to 2.25.3
- #1268 Upgrade aws-java-sdk.version from 1.12.405 to 1.12.668
- #1268 Upgrade gsr.version from 1.1.17 to 1.1.19
- #1257 Prevent improper error logging during worker shutdown
- #1260 Add Deletion protection config
- #1258 Fix issue in configuring metricsEnabledDimensions
- #1259 Add snapshot to version
- #1232 Upgrade ch.qos.logback:logback-classic dependency from 1.3.0 to 1.3.12 in /amazon-kinesis-client
- #1233 Upgrade ch.qos.logback:logback-classic dependency from 1.3.0 to 1.3.12 in /amazon-kinesis-client-multilang
- #1230 Bug fix which now allows MultiLangDaemon to configure idleTimeBetweenReadsInMillis
- #1229 Added link to
javadoc.io-hosted Javadoc in the README - #1218 Added doc for leases and the lease lifecycle to help explain lease lifecycle logic.
- #1226 Upgraded KCL from 2.5.3 to 2.5.4-SNAPSHOT
- #1219 Provided streamArn in getRecords request
- #1216 Updated AWS Glue Schema Registry from version 1.1.14 to 1.1.17.
- #1205 Updated the FAQ with impact of changing default checkpoint.
- #1203 Added links from README.md to FAQ and doc folder.
- #1202 Introduced a FAQ for Kinesis Client Library
- #1200 Added test case for StreamIdentifier serialization.
- #1184 [#367] Enhanced multi-lang
AWSCredentialsProvider=...decoder and c… - #1186 Provided documentation for multilang's new NestedPropertyKey enhancement.
- #1181 CVE-2023-2976: Upgrade Google Guava dependency version from
32.0.0-jreto32.1.1-jre - #1159 Bug fix in lease refresher integration test with occasional failures
- #1157 Fix NPE on graceful shutdown before DDB
LeaseCoordinatorstarts. - #1152 Adding resharding integration tests and changing ITs to not run by default
- #1162 Only deleting resource created by ITs
- #1158 Checkstyle: tightened
LineLengthrestriction from 170 to 150. - #1151 Modified
dependabot.ymlto set the correctv[1|2].xlabel. - #1164 Upgraded KCL Version from 2.5.1 to 2.5.2-SNAPSHOT
- #1143 Upgrade MultiLangDaemon to support StreamARN
- #1145 Introduced GitHub actions to trigger Maven builds during merge/pull requests
- #1136 Added testing architecture and KCL 2.x basic polling/streaming tests
- #1153 Checkstyle: added
UnusedImportscheck. - #1150 Enabled Checkstyle validation of test resources.
- #1149 Bound Checkstyle to
validategoal for automated enforcement. - #1148 Code cleanup to faciliate Checkstyle enforcement.
- #1142 Upgrade Google Guava dependency version from 31.1-jre to 32.0.0-jre
- #1115 Update KCL version from 2.5.0 to 2.5.1-SNAPSHOT
- #1109 Add support for stream ARNs
- #1065 Allow tags to be added when lease table is created
- #1094 Code cleanup to introduce better testing
- #1088 Minimize race in PSSM to optimize shard sync calls
- #1086 Add additional SingleStreamTracker constructor with stream position parameter
- #1084 More consistent testing behavior with restartAfterRequestTimerExpires
- #1066 More consistent testing behavior with HashRangesAreAlwaysComplete
- #1072 Upgrade nexus-staging-maven-plugin from 1.6.8 to 1.6.13
- #1073 Upgrade slf4j-api from 2.0.6 to 2.0.7
- #1090 Upgrade awssdk.version from 2.20.8 to 2.20.43
- #1071 Upgrade maven-compiler-plugin from 3.8.1 to 3.11.0
- #1080 Added metric in
ShutdownTaskfor scenario when parent leases are missing. - #1077 Reverted changes to pom property
- #1069 Fixed flaky InitializationWaitsWhenLeaseTableIsEmpty test
- NOTE: Due to an issue during the release process, the 2.4.7 published artifacts are incomplete and non-viable. Please use 2.4.8 or later.
- #1063 Allow leader to learn new leases upon re-election to avoid unnecessary shardSyncs
- #1060 Add new metric to be emitted on lease creation
- #1057 Added more logging in
Schedulerw.r.t.StreamConfigs. - #1059 DRY: simplification of
HierarchicalShardSyncerTest. - #1062 Fixed retry storm in
PrefetchRecordsPublisher. - #1061 Fixed NPE in
LeaseCleanupManager. - #1056 Clean up in-memory state of deleted kinesis stream in MultiStreamMode
- #1058 Documentation: added
<pre>tags so fixed-format diagrams aren't garbled. - #1053 Exposed convenience method of
ExtendedSequenceNumber#isSentinelCheckpoint() - #1043 Removed a
.swpfile, and updated.gitignore. - #1047 Upgrade awssdk.version from 2.19.31 to 2.20.8
- #1046 Upgrade maven-javadoc-plugin from 3.3.1 to 3.5.0
- #1038 Upgrade gsr.version from 1.1.13 to 1.1.14
- #1037 Upgrade aws-java-sdk.version from 1.12.370 to 1.12.405
- #1041 Minor optimizations (e.g., calculate-once, put instead of get+put)
- #1035 Release Note updates to avoid duplication and bitrot (e.g., 1.x release
- #935 Pass isAtShardEnd correctly to processRecords call
- #1040 Increased logging verbosity around lease management
- #1024 Added logging w.r.t. StreamConfig handling.
- #1034 Optimization: 9~15% improvement in KinesisDataFetcher wall-time
- #1045 Fixed duplication of project version in children pom.xml
- #956 Fixed warning message typos
- #795 Fixed log message spacing
- #740 Fixed typo in Comment
- #1028 Refactored MultiStreamTracker to provide and enhance OOP for both
- #1027 Removed CHECKSTYLE:OFF toggles which can invite/obscure sub-par code.
- #1032 Upgrade rxjava from 3.1.5 to 3.1.6
- #1030 Upgrade awssdk.version from 2.19.2 to 2.19.31
- #1029 Upgrade slf4j-api from 2.0.0 to 2.0.6
- #1015 Upgrade protobuf-java from 3.21.5 to 3.21.12
- #1014 Use AFTER_SEQUENCE_NUMBER iterator type for expired iterator request
- #1017 Upgrade aws sdk
- aws-java-sdk.version from 1.12.296 -> 1.12.370
- awssdk.version from 2.17.268 -> 2.19.2
- #1020 Correct the KCL version in the main pom
- #980 logback-classic: 1.2.9 -> 1.4.0
- #983
- protobuf-java: 3.19.2 -> 3.21.5
- slf4j.version: 1.7.32 -> 2.0.0
- schema-registry-serde: 1.1.9 -> 1.1.13
- #984 awssdk.version from 2.17.108 to 2.17.267
- #987 guava: 31.0.1-jre -> 31.1-jre
- #988 jcommander: 1.81 to 1.82
- #990 Upgrade dependencies
- aws-java-sdk.version: 1.12.130 -> 1.12.296
- lombok: 1.18.22 -> 1.18.24
- rxjava: 3.1.3 -> 3.1.5
- maven-resources-plugin: 2.6 -> 3.3.0
- logback-classic: 1.4.0 -> 1.3.0
- awssdk.version: 2.17.267 -> 2.17.268
- #972 Upgrade Lombok to version 1.18.24
- #916 Upgrade to rxjava3
- #894 Bump protobuf-java from 3.19.1 to 3.19.2
- #924 Support Protobuf Data format with Glue Schema Registry.
- #868 Adding a new metric: Application-level MillisBehindLatest
- #879 Keep dependencies up-to-date
- #886 Get latest counter before attempting a take to ensure take succeeds
- #888 Configure dependabot for v1.x branch
- #866 Update logback dependency.
- #860 Upgrade Glue schema registry from 1.1.4 to 1.1.5.
- #861 Revert PR#847 due to regression for leases without owners and added new tests.
- #842 Fixing typo is debug logs.
- #846 Fix DynamoDBLeaseTaker logging of available leases
- #847 Make use of Java 8 to simplify computeLeaseCounts()
- #853 Add configurable initial position for orphaned stream
- #854 Create DynamoDB tables on On-Demand billing mode by default.
- #855 Emit Glue Schema Registry usage metrics
- #857 Fix to shutdown PrefetchRecordsPublisher in gracefull manner
- #858 Upgrade AWS SDK version to 2.17.52.
- #836 Upgraded AWS SDK version to 2.16.98
- #835 Upgraded Glue Schema Registry version to 1.1.1
- #828 Modified wildcard imports to individual imports.
- #817 Updated the Worker shutdown logic to make sure that the
LeaseCleanupManageralso terminates all the threads that it has started. - #794 Silence warning when there are no stale streams to delete.
- #824 Upgraded dependencies
- logback-classic version to 1.2.3
- AWS Java SDK version to 1.12.3
- AWS SDK version to 2.16.81
- #815 Converted Future to CompletableFuture which helps in proper conversion to Scala using Scala Future Converters.
- #810 Bump commons-io from 2.6 to 2.7
- #804 Allowing user to specify an initial timestamp in which daemon will process records.
- #802 Upgraded guava from 26.0-jre to 29.0-jre
- #801 Fixing a bug that causes to block indefinitely when trying to unlock a lock that isn't locked.
- #762 Added support for web identity token in multilang
- #788 Fixing a bug that caused paginated
ListShardscalls with theShardFilterparameter to fail when the lease table was being initialized.
- Fixing bug in PrefetchRecordsPublisher which was causing retry storms if initial request fails.
- Fixing bug where idleTimeBetweenReadsInMillis property was ignored in PollingConfig.
- Adding support for Glue Schema Registry. Deserialize and read schemas associated with the records.
- Updating AWS SDK version to 2.15.31
- Introducing support for processing multiple kinesis data streams with the same KCL 2.x for java consumer application
-
To build a consumer application that can process multiple streams at the same time, you must implement a new interface called MultistreamTracker (https://github.com/awslabs/amazon-kinesis-client/blob/0c5042dadf794fe988438436252a5a8fe70b6b0b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/processor/MultiStreamTracker.java)
-
MultistreamTracker will also publish various metrics around the current active streams being processed, the number of streams which are deleted at this time period or are pending deletion.
-
-
Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.
-
When bootstrapping an empty lease table, KCL utilizes the
ListShardAPI's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by theShardFilterparameter. TheShardFilterparameter enables you to filter out the response of theListShardsAPI, using theTypeparameter. KCL uses theTypefilter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.- Currently, the following shard filters are supported:
AT_TRIM_HORIZON- the response includes all the shards that were open atTRIM_HORIZON.AT_LATEST- the response includes only the currently open shards of the data stream.AT_TIMESTAMP- the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
ShardFilteris used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified atRetrievalConfig#initialPositionInStreamExtended.- For more information about ShardFilter, see the official AWS documentation on ShardFilter.
- Currently, the following shard filters are supported:
-
Introducing support for the
ChildShardsresponse of theGetRecordsand theSubscribeToShardAPIs to perform lease/shard synchronization that happens atSHARD_ENDfor closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.- For shared throughout consumer applications, this uses the
ChildShardsresponse of theGetRecordsAPI. For dedicated throughput (enhanced fan-out) consumer applications, this uses theChildShardsresponse of theSubscribeToShardAPI. - For more information, see the official AWS Documentation on GetRecords, SubscribeToShard, and ChildShard.
- For shared throughout consumer applications, this uses the
-
KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required.
PeriodicShardSyncManageris the new component that is responsible for running periodic lease/shard scans.- New configuration options are available to configure
PeriodicShardSyncManagerinLeaseManagementConfig
Name Default Description leasesRecoveryAuditorExecutionFrequencyMillis 120000 (2 minutes) Frequency (in millis) of the auditor job to scan for partial leases in the lease table. If the auditor detects any hole in the leases for a stream, then it would trigger shard sync based on leasesRecoveryAuditorInconsistencyConfidenceThreshold. leasesRecoveryAuditorInconsistencyConfidenceThreshold 3 Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync - New CloudWatch metrics are also now emitted to monitor the health of
PeriodicShardSyncManager:
Name Description NumStreamsWithPartialLeases Number of streams that had holes in their hash ranges. NumStreamsToSync Number of streams which underwent a full shard sync. - New configuration options are available to configure
-
Introducing deferred lease cleanup. Leases will be deleted asynchronously by
LeaseCleanupManagerupon reachingSHARD_END, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.- New configuration options are available to configure
LeaseCleanupManager.
Name Default Description leaseCleanupIntervalMillis 1 minute Interval at which to run lease cleanup thread. completedLeaseCleanupIntervalMillis 5 minutes Interval at which to check if a lease is completed or not. garbageLeaseCleanupIntervalMillis 30 minutes Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not. - New configuration options are available to configure
-
Introducing experimental support for multistreaming, allowing a single KCL application to multiplex processing multiple streams.
- New configuration options are available to enable multistreaming in
RetrievalConfig#appStreamTracker.
- New configuration options are available to enable multistreaming in
-
Fixing a bug in
PrefetchRecordsPublisherrestarting while it was already running. -
Including an optimization to
HierarchicalShardSyncerto only create leases for one layer of shards. -
Adding support to prepare and commit lease checkpoints with arbitrary bytes.
- This allows checkpointing of an arbitrary byte buffer up to the maximum permitted DynamoDB item size (currently 400 KB as of release), and can be used for recovery by passing a serialized byte buffer to
RecordProcessorCheckpointer#prepareCheckpointandRecordProcessorCheckpointer#checkpoint.
- This allows checkpointing of an arbitrary byte buffer up to the maximum permitted DynamoDB item size (currently 400 KB as of release), and can be used for recovery by passing a serialized byte buffer to
-
Upgrading version of AWS SDK to 2.14.0.
-
#725 Allowing KCL to consider lease tables in
UPDATINGhealthy.
- Adjusting HTTP2 initial window size to 512 KB
- Updating protobuf-java to version 3.11.4
- Updating the AWS Java SDK to version 2.13.25
- Fixing a bug in DynamoDB billing mode support for special regions.
- Adding request id logging to ShardConsumerSubscriber.
- Updating the AWS SDK version to 2.10.66.
- Adding request id logging to SubscribeToShard response.
- Updating the AWS SDK version to 2.10.56.
- PR#679
- NOTE: SDK has a known connection teardown issue when multiple H2 streams are used within a connection. This might result in shard consumers sticking to a stale service host and not progressing. If your shard consumer gets stuck, use the following configuration as a workaround. This configuration might result in up to 5X increase in total connections.
KinesisAsyncClient kinesisClient = KinesisAsyncClient.builder() .region(region) .httpClientBuilder(NettyNioAsyncHttpClient.builder().maxConcurrency(Integer.MAX_VALUE).http2Configuration(Http2Configuration.builder().maxStreams(1).build()) .build() - Making ShardConsumerTest resilient to race conditions.
- Updating integration test naming.
- Updating the AWS SDK version to 2.10.25
- Adding a configurable DynamoDB billing mode
- PR#582
- NOTE: Billing mode is not available in all regions; if your lease table cannot be created, use the following configuration as a workaround:
LeaseManagementConfig leaseManagementConfig = builder.leaseManagementConfig().billingMode(null).build();
- Updating the SDK version to 2.9.25.
- Clearing the local cache on a subscription termination, to avoid noisy logs on new subscriptions.
- Updating the SDK version to 2.10.0 in order to fix the premature H2 stream close issue.
- PR#649
- NOTE: SDK has a known connection teardown issue when multiple H2 streams are used within a connection. This might result in shard consumers sticking to a stale service host and not progressing. If your shard consumer gets stuck, use the following configuration as a workaround. This configuration might result in up to 5X increase in total connections.
KinesisAsyncClient kinesisClient = KinesisAsyncClient.builder() .region(region) .httpClientBuilder(NettyNioAsyncHttpClient.builder().maxConcurrency(Integer.MAX_VALUE).maxHttp2Streams(1)) .build()
- Updating Sonatype to dedicated AWS endpoint.
- Introducing a validation step to verify if ShardEnd is reached, to prevent shard consumer stuck scenarios in the event of malformed response from service.
- Making FanoutRecordsPublisher test cases resilient to delayed thread operations
- Drain delivery queue in the FanoutRecordsPublisher to make slow consumers consume events at their pace
- Fix to prevent the onNext event going to stale subscription when restart happens in PrefetchRecordsPublisher
- Fix to prevent data loss and stuck shards in the event of failed records delivery in Polling readers
- Fix to prevent invalid ShardConsumer state transitions due to rejected executor service executions.
- Fixing a bug in which initial subscription failure caused a shard consumer to get stuck.
- Making CW publish failures visible by executing the async publish calls in a blocking manner and logging on exception.
- Update shard end checkpoint failure messaging.
- A fix for resiliency and durability issues that occur in the reduced thread mode - Nonblocking approach.
- Preventing duplicate delivery due to unacknowledged event, while completing the subscription.
- Add periodic logging for the state of the thread pool executor service. This service executes the async tasks submitted to and by the ShardConsumer.
- Add logging of failures from RxJava layer.
- Updated License to Apache License 2.0
- Introducing configuration for suppressing logs from ReadTimeoutExceptions caused while calling SubscribeToShard.
Suppression can be configured by settingLifecycleConfig#readTimeoutsToIgnoreBeforeWarning(Count).
- Added a message to recommend using
KinesisClientUtilwhen an acquire timeout occurs in theFanOutRecordsPublisher. - Added a sleep between retries while waiting for a newly created stream consumer to become active.
- Added timeouts on all futures returned from the DynamoDB and Kinesis clients.
The timeouts can be configured by settingLeaseManagementConfig#requestTimeout(Duration)for DynamoDB, andPollingConfig#kinesisRequestTimeout(Duration)for Kinesis. - Upgraded to SDK version 2.5.10.
- Artifacts for the Amazon Kinesis Client for Java are now signed by a new GPG key:
pub 4096R/86368934 2019-02-14 [expires: 2020-02-14] uid Amazon Kinesis Tools <[email protected]>
- Fixed handling of the progress detection in the
ShardConsumerto restart from the last accepted record, instead of the last queued record. - Fixed handling of exceptions when using polling so that it will no longer treat
SdkExceptions as an unexpected exception. - Fixed a case where lease loss would block the
Schedulerwhile waiting for a record processor'sprocessRecordsmethod to complete.
- Introducing
SHUT_DOWN_STARTEDstate for theWorkerStateChangeListener. - Fixed a bug with
AWSSessionCredentialsusingAWSSecretIDinstead ofAWSAccessIDand vice versa. - Upgrading SDK version to 2.4.0, which includes a fix for a possible deadlock when using Enhanced Fan-Out.
- Introducing MultiLangDaemon support for Enhanced Fan-Out.
- MultiLangDaemon now supports the following command line options.
--properties-file: Properties file that the KCL should use to set up the Scheduler.--log-configuration: logback.xml that the KCL should use for logging.
- Updated AWS SDK dependency to 2.2.0.
- MultiLangDaemon now uses logback for logging.
- Fixed a deadlock condition that could occur when using the polling model.
When using thePollingConfigand a slower record processor it was possible to hit a deadlock in the retrieval of records. - Adjusted
RetrievalConfig, andFanOutConfigto use accessors instead of direct member access.
- Added method to retrieve leases from the LeaseCoordinator and LeaseTaker.
- Fixed a race condition shutting down the Scheduler before it has completed initialization.
- Added
HierarchicalShardSyncerwhich replaces the staticShardSyncer.
HierarchicalShardSyncerremoves the contention between multiple instances of the Scheduler when running under a single JVM. - Added
TaskExecutionListenerwhich allows monitoring of tasks being executed by theShardConsumer.
The listener is invoked before and after a task is executed by theShardConsumer.
- Fixed an issue where the
KinesisAsyncClientcould be misconfigured to use HTTP 1.1.
Using HTTP 1.1 withSubscribeToShardis unsupported, and could cause misdelivery of records to the record processor. - Lower the severity of
ReadTimeoutexceptions.
ReadTimeoutexceptions can occur if the client is unable to request data from Kinesis for more than client timeout, which defaults to 30 seconds. This can occur if the record processor blocks for more than the timeout period.ReadTimeoutcould also occur as part of Issue #391. - Added a callback that allows applications to take actions after DynamoDB table creation.
Applications can now install a callback that is called after creating the DynamoDB table by implementingTableCreatorCallback. - Updated the guava dependency to 26.0-jre.
- Added some additional debug logging around the initialization of the
FanOutRecordsPublisher. - Upgraded AWS SDK version to 2.0.6
- Fixed an issue where the a warning would be logged every second if
logWarningForTaskAfterMilliswas set.
The logging for last time of data arrival now respects the value oflogWarningForTaskAfterMillis. - Moved creation of
WorkerStateChangedListenerandGracefulShutdownCoordinatorto theCoordinatorConfig. Originally theWorkerStateChangedListenerandGracefulShutdownCoordinatorwere created by methods on theSchedulerCoordinatorFactory, but they should have been configuration options.
The original methods have been deprecated, and may be removed at a later date. - Removed dependency on Apache Commons Lang 2.6.
The dependency on Apache Commons Lang 2.6 has removed, and all usages updated to use Apache Commons Lang 3.7. - Fixed a typo in the MutliLang Daemon shutdown hook.
- Added method
onAllInitializationAttemptsFailed(Throwable)toWorkerStateChangedListenerto report when all initialization attempts have failed.
This method is a default method, and it isn't require to implement the method. This method is only called after all attempts to initialize theSchedulerhave failed.
- Mark certain internal components with
@KinesisClientInternalApiattribute.
Components marked as internal may be deprecated at a faster rate than public components. - Fixed an issue where
ResourceNotFoundExceptionon subscription to a shard was not triggering end of shard handling.
If a lease table contains a shard that is no longer present in the stream attempt to subscribe to that shard will trigger aResourceNotFoundException. These exception are treated the same as reaching the end of a shard. - Fixed an issue where the KCL would not Use the configured DynamoDB IOPs when creating the lease table.
- Make the maximum number of Scheduler initialization attempts configurable.
The maximum number ofSchedulerinitialization attempts can be configured viaCoordinatorConfig#maxInitializationAttempts. - Fixed an issue where it was possible to get a duplicate record when resubscribing to a shard.
Subscribe to shard requires periodic resubscribing, and uses a new concept of a continuation sequence number. If the continuation sequence number was equal to the last record that record would be processed a second time. Resubscribing now usesAFTER_SEQUENCE_NUMBERto ensure that only later records are returned. - Upgraded to AWS SDK 2.0.1
- Fixed an issue where time based restart of the subscription wasn't resetting the
lastRequestTime.
If a subscription hasn't delivered any data for more than 30 seconds it will be canceled and restarted. This detection is based of thelastRequestTimewhich wasn't getting reset after the restart was triggered. - Fixed an issue where requesting on the subscription from the
FanOutRecordsPublishercould trigger an unexpected failure.
Due to a race condition the underlying flow in the subscription could be set to something else. The method is now synchronized, and verifies that the subscriber it was created with is still the subscriber in affect.
This issue generally would only appear when multiple errors were occurring while connecting to Kinesis. - Fixed an issue where the number of requested items could exceed the capacity of the RxJava queue.
There was an off by one issue when determining whether to make a request to the SDK subscription. This changes the calculation to represent the capacity as a queue.
-
The Maven
groupId, along with theversion, for the Amazon Kinesis Client has changed fromcom.amazonawstosoftware.amazon.kinesis.
To add a dependency on the new version of the Amazon Kinesis Client:<dependency> <groupId>software.amazon.kinesis</groupId> <artifactId>amazon-kinesis-client</artifactId> <version>2.0.0</version> </dependency>
-
Added support for Enhanced Fan Out.
Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.- Records are now delivered via streaming, reducing end-to-end latency.
- The Amazon Kinesis Client will automatically register a new consumer if required.
When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise. SubscribeToShardmaintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
TheKinesisClientUtilhas been added to assist configuring themaxConcurrencyof theKinesisAsyncClient.
WARNING: The Amazon Kinesis Client may see significantly increased latency, unless theKinesisAsyncClientis configured to have amaxConcurrencyhigh enough to allow all leases plus additional usages of theKinesisAsyncClient.- The Amazon Kinesis Client now uses 3 additional Kinesis API's:
WARNING: If using a restrictive Kinesis IAM policy you may need to add the following API methods to the policy. - New configuration options are available to configure Enhanced Fan Out.
Name Default Description consumerArn Unset The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer. streamName Unset The name of the stream that a consumer should be create for if necessary consumerName Unset The name of the consumer to create. If this is not set the applicationName will be used instead. applicationName Unset The name of the application. This is used as the name of the consumer unless consumerName is set. -
Modular Configuration of the Kinesis Client The Kinesis Client has migrated to a modular configuration system, and the
KinesisClientLibConfigurationclass has been removed.
Configuration has been split into 7 classes. Default versions of the configuration can be created from theConfigsBuilder.
Please [see the migration guide for more information][migration-guide].CheckpointConfigCoordinatorConfigLeaseManagementConfigLifecycleConfigMetricsConfigProcessorConfigRetrievalConfig
-
Upgraded to AWS Java SDK 2.0
The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed. All configurations will only accept 2.0 clients.- When configuring the
KinesisAsyncClienttheKinesisClientUtil#createKinesisAsyncClientcan be used to configure the Kinesis Client - If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.
When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded
[Please see the migration guide for more information][migration-guide]
- When configuring the
-
MultiLangDaemon is now a separate module
The MultiLangDaemon has been separated to its own Maven module and is no longer available inamazon-kinesis-client. To include the MultiLangDaemon, add a dependency onamazon-kinesis-client-multilang.