Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Internal refactoring and modernization #720

Merged
merged 73 commits into from
Feb 15, 2024
Merged

Internal refactoring and modernization #720

merged 73 commits into from
Feb 15, 2024

Conversation

ricellis
Copy link
Member

Checklist

  • Added tests for code changes or test/build only changes
  • Updated the change log file (CHANGES.md) or test/build only changes
  • Completed the PR template below:

Description

Merge modernization branch to main.

Fixes i320

Approach

The work has previously been reviewed in: #643 #650 #652 #658 #659 #660 #663 #668 #669 #670 #671 #672 #673 #676 #677 #678 #679 #682 #691 #692 #695 #696 #698 #702 #706 #708 #711

The only additional changes in this PR are from rebasing on main and cleaning up package-lock.json

  • [NEW] Included time in restored API events and CLI output.
  • [NEW] Added metadata to backup files.
  • [FIXED] Double output of errors to stderr when using CLI.
  • [FIXED] Error for broken JSON in backup files.
  • [FIXED] Wrong error code used for incomplete changes item.
  • [FIXED] Error if the log file already exists when starting a new backup.
  • [REMOVED] Dependency on async module.
  • [REMOVED] Dependency on tmp module.
  • [REMOVED] Unused request handling code.
  • [IMPROVED] Increased tolerance to server and network errors when spooling changes (via cloudant-node-sdk changes follower).
  • [IMPROVED] Avoided double parsing of JSON batches when resuming a backup.
  • [IMPROVED] Resumed backups identification of incomplete backup lines during restore.
  • [IMPROVED] Added line numbers to errors from reading backup or log files.
  • [IMPROVED] Replace custom liner with Node built-in readline.
  • [IMPROVED] Added warning that --buffer-size has no effect with --resume.
  • [IMPROVED] Documentation about compatibility.
  • [IMPROVED] Resolved linter warnings.
  • [IMPROVED] Internal enhancements with promises and pipelines replacing callbacks and event emitters.
  • [IMPROVED] Added some new test cases and improved test stability.
  • [NOTE] Versions older than 2.10.0 cannot restore backups created with 2.10.0. See compatibility note.

Schema & API Changes

  • Included time in restored API events and CLI output.
  • Added metadata to backup files.

Security and Privacy

  • "No change"

Testing

  • New tests added and tests refactored for modernization changes as described in PRs listed above.

Monitoring and Logging

  • Logging enhancements as outlined in PRs listed.

ricellis and others added 30 commits February 14, 2024 15:37
@ricellis
refactor: replace liner with built-in readline
d711e4c
@ricellis
refactor: add pipeline transform duplex streams
4fe19de 
* Add a BatchingStream for batching elements into arrays
* Add a SplittingStream for flat-mapping a batch back to elements
* Add a MappingStream for mapping stream elements to something else
* Add tests for the above
@ricellis @emlaver
refactor: use follower utility in spoolchanges
412d1b0 
Co-authored-by: Esteban Laver <[email protected]>
@ricellis @emlaver
test: update spoolchanges tests
8d5367d 
Update to use new follower based spoolchanges.
Use httpServer instead of nock for long running tests with
lots of changes requests because nock appears to leak memory
in those situations.

Co-authored-by: Esteban Laver <[email protected]>
@ricellis
refactor: add backup mapping functions
a5230b2
@emlaver @ricellis
refactor: add passthrough that performs side effect
4bfd264 
Signed-off-by: Esteban Laver <[email protected]>
@ricellis
refactor: add line count to liner
4049cbb
@ricellis
refactor: restore mappings
b239d20
@ricellis
refactor: put backup mappers in classes
723299e
@ricellis
refactor: use liner format in backup mappings
5ab0346
@ricellis
refactor: add a filter stream
d1c89c3
@ricellis
test: add end to end backup mapping case
33e222a
@ricellis
refactor: reorder and rename transforms
07e4aee
@ricellis
refactor: add DelegateWritable and WritablePassThrough
4b7bb28
@ricellis
refactor: use DelegateWriter in spoolchanges
9ac0c95
@ricellis
refactor: add chunkMapFn and postWriteFn
f800cf2
@eiri @ricellis
refactor: DRY and simplify validateArgs
67996b4
@eiri @ricellis
refactor: promisify backup and restore preflight checks
88e896a
@ricellis
fix: defer destroy to next tick
2976ce5
@ricellis
refactor: restore pipeline
44a4da0
@ricellis
refactor: test rename writer to restore
cf882ba
@ricellis
refactor: resume log file handling
5b4ad51
@ricellis
refactor: spoolchanges for backup pipeline
e847463
@ricellis
fix: spool changes uses non-existent error name
c5a1402
@ricellis
refactor: full backup as pipeline
71c83ae
@ricellis
fix: restore totals
89a5f74
@ricellis
refactor: event emitters
279bb09
@ricellis
test: fix write error test
ce4773a
@ricellis
refactor: remove obsolete code from request
8e50582
@ricellis
refactor: rename db client
553a962
ricellis and others added 25 commits February 15, 2024 13:57
@ricellis
test: avoid creating log file for resume tests
56f9296
@eiri @ricellis
refactor: explicitly name running time var
8eb9e0d
@ricellis
feat: add time to restore output
d6859da
@ricellis
fix: handle back-pressure in Liner
2ff9483 
Also reduce the amount buffered in memory at once.
@ricellis
fix: revert to complete spooling before backup
4ef7f0b
@ricellis
fix: minimize buffering on map passthroughs
00e179b
@ricellis
fix: batching Duplex with back-pressure
22cdccd 
Use a Duplex implementation instead of a trasnform.
Allow batching and rebatching (eliminate SplittingStream).
@ricellis
test: match follower batch size in spoolchanges
5290c76
@ricellis
fix: high water marks in mapping streams
a0f6360 
Override the default temporarily when we create mapping streams.
@emlaver @ricellis
test: shared client for use in hooks/compare test util functions
bb3e6a3 
Signed-off-by: Esteban Laver <[email protected]>
@emlaver @ricellis
ci: update URL env vars for IAM in test stage
618bb21 
Signed-off-by: Esteban Laver <[email protected]>
@emlaver @ricellis
feat: write metadata on first line of backup file
c8cbe62 
Co-authored-by: Rich Ellis <[email protected]>
Signed-off-by: Esteban Laver <[email protected]>
@ricellis
fix: increment line number in wrapLine
e3bf3c8 
This ensures tests aren't always using line 0.

Signed-off-by: Esteban Laver <[email protected]>
@emlaver @ricellis
test: add backup file metadata to fixtures
8ded51e 
Co-authored-by: Rich Ellis <[email protected]>
Signed-off-by: Esteban Laver <[email protected]>
@ricellis
test: allow metadata in backup file comparison
6f5bcd5 
Co-authored-by: Rich Ellis <[email protected]>
Signed-off-by: Esteban Laver <[email protected]>
@emlaver @ricellis
feat: parse backup file metadata on restore
da46b67 
* use JSON object for restore marker

Co-authored-by: Rich Ellis <[email protected]>
Signed-off-by: Esteban Laver <[email protected]>
@ricellis
test: add a restore from compressed file test
cce0f9a
@ricellis
test: assert only backup/restore promise for errors
d9cb7ad 
In an error scenario we might get multiple promise rejections, but we only want to assert the backup or restore promise.
In success cases we should check all the promises to ensure we aren't masking errors.
@emlaver @ricellis
fix: remove error event handlers in backup and restore bin scripts
e1593d7 
Signed-off-by: Esteban Laver <[email protected]>
@ricellis
test: destroy unused stream
7a508dd
@emlaver @ricellis
ci: check console output for pipe error or filehandle depreciation wa…
6db6757 
…rning

Signed-off-by: Esteban Laver <[email protected]>
@ricellis
docs: compatibility notes
79ac05a
@ricellis
docs: 2.10.0 changes
a5db52a
@ricellis
test: do dbcompare batch wise to reduce overhead
029c20e
@ricellis
chore: package-lock updates/cleanup
33c3a6b
@ricellis ricellis merged commit 3e06382 into main Feb 15, 2024
4 checks passed
@ricellis ricellis deleted the modernization branch February 15, 2024 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@emlaver emlaver emlaver approved these changes

@eiri eiri eiri approved these changes

Assignees

@ricellis ricellis

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ricellis @eiri @emlaver