Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PipeConsensus: Support deletion operation for PipeConsensus #13016

Merged
merged 98 commits into from
Oct 18, 2024

Commits on Jul 24, 2024

  1. Configuration menu
    Copy the full SHA
    79596ec View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ac53ff6 View commit details
    Browse the repository at this point in the history
  3. fix ut fail

    Pengzna committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    1271ce5 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. minor fix WAL close time

    Pengzna committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    d876131 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b0aecb8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    35d893d View commit details
    Browse the repository at this point in the history
  4. remove useless

    Pengzna committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    17aaa74 View commit details
    Browse the repository at this point in the history
  5. fix

    Pengzna committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    464191b View commit details
    Browse the repository at this point in the history
  6. fix

    Pengzna committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    e66ed10 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    8d32e06 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2024

  1. add config of deletion dirs

    Pengzna committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    5b85c76 View commit details
    Browse the repository at this point in the history
  2. add base ut

    Pengzna committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    eff3507 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. fix deletion read

    Pengzna committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    47a335d View commit details
    Browse the repository at this point in the history
  2. fix deserialize logic

    Pengzna committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    56f5c55 View commit details
    Browse the repository at this point in the history
  3. add recover ut

    Pengzna committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    4c45c33 View commit details
    Browse the repository at this point in the history
  4. remove duplicate

    Pengzna committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    ee9c40a View commit details
    Browse the repository at this point in the history
  5. remove ut

    Pengzna committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    d115cdc View commit details
    Browse the repository at this point in the history
  6. Merge remote-tracking branch 'refs/remotes/base/master' into deletion

    # Conflicts:
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/realtime/PipeRealtimeEventFactory.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionExtractor.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/receiver/protocol/pipeconsensus/PipeConsensusReceiver.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java
    #	iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/DataRegionTest.java
    #	iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/inner/InnerSeqCompactionWithFastPerformerTest.java
    #	iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/inner/InnerSeqCompactionWithReadChunkPerformerTest.java
    #	iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/settle/SettleRequestHandlerTest.java
    #	iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/concurrent/ThreadName.java
    Pengzna committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    21252bd View commit details
    Browse the repository at this point in the history
  7. fix conflict

    Pengzna committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    bc14898 View commit details
    Browse the repository at this point in the history
  8. spotless

    Pengzna committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    13bbbc0 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. avoid Concurrent issue

    Pengzna committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    f57b8a9 View commit details
    Browse the repository at this point in the history
  2. fix concurrent issue

    Pengzna committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    464c136 View commit details
    Browse the repository at this point in the history
  3. fix

    Pengzna committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    3f629f1 View commit details
    Browse the repository at this point in the history
  4. fix

    Pengzna committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    1468eab View commit details
    Browse the repository at this point in the history
  5. deletion close

    Pengzna committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    8a47b93 View commit details
    Browse the repository at this point in the history
  6. add log

    Pengzna committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    848c82d View commit details
    Browse the repository at this point in the history
  7. format

    Pengzna committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    fa0780c View commit details
    Browse the repository at this point in the history
  8. spotless

    Pengzna committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    edca176 View commit details
    Browse the repository at this point in the history
  9. fix deserialize

    Pengzna committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    19a4471 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Configuration menu
    Copy the full SHA
    2262724 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. fix review

    Pengzna committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    e60bea6 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Configuration menu
    Copy the full SHA
    360637a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d8743fc View commit details
    Browse the repository at this point in the history
  3. historical data process

    Pengzna committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    19216ca View commit details
    Browse the repository at this point in the history
  4. add ut

    Pengzna committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    7835d8b View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f457fe7 View commit details
    Browse the repository at this point in the history
  6. merge master

    Pengzna committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    cadbaab View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. fix bug and ut

    Pengzna committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    baa71f8 View commit details
    Browse the repository at this point in the history
  2. remove metadata(deletionNum)

    Pengzna committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    e251c43 View commit details
    Browse the repository at this point in the history
  3. Merge remote-tracking branch 'refs/remotes/base/master' into deletion

    # Conflicts:
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/service/DataNodeShutdownHook.java
    Pengzna committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    afe925b View commit details
    Browse the repository at this point in the history
  4. merge

    Pengzna committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    c3d3460 View commit details
    Browse the repository at this point in the history
  5. handle corrupted file

    Pengzna committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    fb8a029 View commit details
    Browse the repository at this point in the history
  6. add ut

    Pengzna committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    30f7956 View commit details
    Browse the repository at this point in the history
  7. add license

    Pengzna committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    b514c97 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. retrigger ci

    Pengzna committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    56ff4bc View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. Merge remote-tracking branch 'refs/remotes/base/master' into deletion

    # Conflicts:
    #	iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/consensus/index/impl/SimpleProgressIndex.java
    Pengzna committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    dc30eee View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2024

  1. fix review

    Pengzna committed Sep 14, 2024
    Configuration menu
    Copy the full SHA
    3c7d48d View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. fix review

    Pengzna committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b4de6d8 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Configuration menu
    Copy the full SHA
    b7e9ca3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    434f103 View commit details
    Browse the repository at this point in the history
  3. fix review: deletion will only be assigned to corresponding dataRegio…

    …n's extractor & deletion will not be extracted when persist failed.
    Pengzna committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    7e94141 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. fix review

    Pengzna committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    e13b3e7 View commit details
    Browse the repository at this point in the history
  2. fix

    Pengzna committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    535150e View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. feat: timout sync

    Pengzna committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    0b39c8b View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. todo: priority queue sort

    Pengzna committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    4045de7 View commit details
    Browse the repository at this point in the history
  2. fix ut

    Pengzna committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    22a3ca6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8bb6cc9 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    1508215 View commit details
    Browse the repository at this point in the history
  5. Merge remote-tracking branch 'refs/remotes/base/master' into deletion

    # Conflicts:
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/connection/PipeEventCollector.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionExtractor.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/realtime/PipeRealtimeDataRegionExtractor.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/realtime/assigner/PipeDataRegionAssigner.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/realtime/matcher/CachedSchemaPatternMatcher.java
    #	iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/consensus/index/impl/SimpleProgressIndex.java
    Pengzna committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    f1d052c View commit details
    Browse the repository at this point in the history
  6. spotless

    Pengzna committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    872f135 View commit details
    Browse the repository at this point in the history
  7. rename pipeConsensus to iotV2

    Pengzna committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    587c7df View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Configuration menu
    Copy the full SHA
    85fef64 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7062c37 View commit details
    Browse the repository at this point in the history
  3. fix: ut

    Pengzna committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    127e02b View commit details
    Browse the repository at this point in the history
  4. add log

    Pengzna committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    486ec81 View commit details
    Browse the repository at this point in the history
  5. fix ut

    Pengzna committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    845c196 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. fix ut

    Pengzna committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    ec11267 View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2024

  1. fix review

    Pengzna committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    d6daaab View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e7b64d2 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f65fcde View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. fix ut

    Pengzna committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    4121175 View commit details
    Browse the repository at this point in the history
  2. refactor

    SteveYurongSu committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    80e9191 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3be53ab View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    004663e View commit details
    Browse the repository at this point in the history
  5. add comment

    Pengzna committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    1fb8853 View commit details
    Browse the repository at this point in the history
  6. Merge remote-tracking branch 'refs/remotes/origin/deletion' into dele…

    …tion
    
    # Conflicts:
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/realtime/assigner/PipeDataRegionAssigner.java
    Pengzna committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    0f6858c View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    26443e8 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    8d49608 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    fa2a329 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    f194928 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    bae8bfe View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    a9d13fe View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    fe255f9 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. fix review

    Pengzna committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    d93ec4c View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'refs/remotes/origin/deletion' into dele…

    …tion
    
    # Conflicts:
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionExtractor.java
    Pengzna committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    86d2de5 View commit details
    Browse the repository at this point in the history
  3. merge

    Pengzna committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    a8d51d2 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Configuration menu
    Copy the full SHA
    bf7982e View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'refs/remotes/base/master' into deletion

    # Conflicts:
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/connection/PipeEventCollector.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/realtime/PipeRealtimeEventFactory.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionExtractor.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/realtime/listener/PipeInsertionDataNodeListener.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/realtime/matcher/CachedSchemaPatternMatcher.java
    Pengzna committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    e835d84 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    01d3cb9 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. Configuration menu
    Copy the full SHA
    a3368c5 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Configuration menu
    Copy the full SHA
    6c3602d View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'refs/remotes/base/master' into deletion

    # Conflicts:
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/thrift/sync/IoTDBDataRegionSyncConnector.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionExtractor.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/receiver/protocol/pipeconsensus/PipeConsensusReceiver.java
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java
    Pengzna committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    e931369 View commit details
    Browse the repository at this point in the history
  3. merge remote

    Pengzna committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    97b699d View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. Merge remote-tracking branch 'refs/remotes/base/master' into deletion

    # Conflicts:
    #	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/airgap/IoTDBDataRegionAirGapConnector.java
    Pengzna committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    6d51a19 View commit details
    Browse the repository at this point in the history
  2. merge remote

    Pengzna committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    ff3c9fc View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    86faeca View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6fe76f9 View commit details
    Browse the repository at this point in the history
  5. resolve conflicts

    SteveYurongSu committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    47813af View commit details
    Browse the repository at this point in the history