Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge vos2 feature branch #565

Merged
merged 255 commits into from
Apr 3, 2024
Merged
Changes from 13 commits
Commits
Show all changes
255 commits
Select commit Hold shift + click to select a range
d244f50
basic function of NodeDAO: put, get, update, delete
pdowler Apr 6, 2023
7156929
unit test code for postgresql array parsing
pdowler Apr 6, 2023
2cdd2de
incomplete node iterator
pdowler Apr 13, 2023
7905cc0
working node child iterator
pdowler Apr 14, 2023
283fb18
Merge branch 'master' of https://github.com/opencadc/storage-inventor…
pdowler Apr 19, 2023
d47e896
fix and test use of Node.parentID
pdowler Apr 19, 2023
59e86b1
vault: add init database, update README with additional config
pdowler Apr 21, 2023
34bfcd8
vault: implement more NodePersistenceImpl methods
pdowler Apr 21, 2023
8082aab
NodeDAO: remove assign parentID in put
pdowler Apr 21, 2023
87c68bb
cadc-inventory-db: nominally assign version as 0.15
pdowler Apr 21, 2023
41dc8dc
update TODO
pdowler Apr 21, 2023
0af5c9c
Merge branch 'master' of https://github.com/opencadc/storage-inventor…
pdowler Apr 27, 2023
89b0908
Merge pull request #502 from pdowler/vos2
pdowler Apr 27, 2023
2512aa3
vault: TODO updates
pdowler Apr 28, 2023
39f51ea
vault: update README
pdowler May 2, 2023
83381d9
add vault to repo README
pdowler May 2, 2023
c842e42
Merge pull request #503 from pdowler/vos2
pdowler May 2, 2023
c542400
update for Node API changes
pdowler May 31, 2023
4af6232
Merge pull request #511 from pdowler/vos2
pdowler May 31, 2023
c1e0240
vault: NodePersistenceImpl implements extracted interface
pdowler Jun 1, 2023
dc1e726
Checkpoint
Jun 16, 2023
805cea2
Merge branch 'master' of https://github.com/opencadc/storage-inventor…
pdowler Jun 29, 2023
6a8e354
vault: add cadc-registry link to README
pdowler Jun 29, 2023
8c892f7
Merge branch 'master' of https://github.com/opencadc/storage-inventor…
pdowler Jul 10, 2023
642c4b7
Merge branch 'master' into vos2
pdowler Jul 10, 2023
ebc861c
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
pdowler Jul 10, 2023
5403a9a
cadc-inventory-db: node iterator bug fix
pdowler Jul 10, 2023
5092f91
typo
pdowler Jul 10, 2023
e1f47b5
Intermidiate commit
Jul 11, 2023
09e351f
Merge branch 'vos2' of https://github.com/pdowler/storage-inventory i…
Jul 11, 2023
a576479
cadc-inventory-db: improve txn close in catch to avoid leak
pdowler Jul 11, 2023
0478212
Merge branch 'vos2' of https://github.com/pdowler/storage-inventory i…
Jul 12, 2023
fa011e0
Fixed small issues
Jul 12, 2023
3fb9201
cadc-inventory-db: add connection.close() calls in NodeIterator to re…
pdowler Jul 12, 2023
ed3fa61
debug->info in test code
pdowler Jul 12, 2023
bf26dde
raven: limit cadc-inventory-db version to avoid cadc-vos-2.0 until re…
pdowler Jul 12, 2023
8099661
Merge branch 'vos2' of https://github.com/pdowler/storage-inventory i…
Jul 12, 2023
ce0898f
Re-factored the library
Jul 12, 2023
d310158
Merge pull request #522 from pdowler/vos2
pdowler Jul 12, 2023
5b4e272
Small fixes
Jul 13, 2023
1c885d7
Small fixes
Jul 13, 2023
a2c63e8
Renamed cadc-vos-server dependency
Jul 13, 2023
d364005
Fixed bug
Jul 13, 2023
8eef18b
Merge pull request #523 from andamian/CADC-12179
pdowler Aug 4, 2023
e6ca6ec
vault: added NodesTest integration test using cadc-test-vos-2.0
pdowler Aug 4, 2023
2b45eaa
vault: comment out unimplemented endpoints from capabilities
pdowler Aug 4, 2023
3efead8
vault: tweak root and trash node settings in NodePersistenceImpl
pdowler Aug 4, 2023
60651cb
vault: update dependencies
pdowler Aug 4, 2023
0531edc
vault: cleanup in NodePersistenceImpl
pdowler Aug 4, 2023
a79673b
move assign of owner and parent details to NodePersistenceImpl
pdowler Aug 8, 2023
4f455a8
vault: formatting
pdowler Aug 9, 2023
d4eab62
Merge pull request #525 from pdowler/vos2
pdowler Aug 10, 2023
679b3ee
cadc-inventory-db: adapt to ContainerNode api change
pdowler Aug 11, 2023
62645da
vault: quick hack fix for delete(Node) to allow repeated create/delet…
pdowler Aug 11, 2023
fcaa3d1
Merge pull request #526 from pdowler/vos2
pdowler Aug 15, 2023
8962bde
vault: fix imports
pdowler Aug 15, 2023
e59dde2
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
pdowler Aug 15, 2023
f875645
Merge branch 'master' of https://github.com/opencadc/storage-inventor…
pdowler Aug 23, 2023
5544cc1
Merge pull request #530 from pdowler/vos2
pdowler Aug 23, 2023
0420a6b
vault: added indentity cache to child iterator
pdowler Aug 23, 2023
1c26961
vault: fix class cast issue with ident cache in iterator
pdowler Aug 23, 2023
e6ef466
Changed type of group sets in Node to GroupURI
Sep 6, 2023
c6dc837
Merge pull request #531 from andamian/CADC-12708
pdowler Sep 7, 2023
f68e859
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
pdowler Sep 7, 2023
ddde9d9
cadc-inventory-db: fix NodeDAO.lock for subclass support
pdowler Sep 7, 2023
b3f51aa
vault: reject deletion of non-empty containers, perform delete action…
pdowler Sep 7, 2023
dc55e9a
Merge pull request #532 from pdowler/vos2
pdowler Sep 8, 2023
2911e7b
vault-quota: README and design doc
pdowler Sep 8, 2023
39df738
Merge pull request #533 from pdowler/vos2
pdowler Sep 8, 2023
e15cbd0
Update Design.md
pdowler Sep 11, 2023
07d759f
Update Design.md
pdowler Sep 11, 2023
5336389
vault-quota: update design doc with some validate ideas
pdowler Sep 12, 2023
1f2a51b
Merge pull request #534 from pdowler/vos2
pdowler Sep 12, 2023
dec5ed8
Support for recursive deletes
Oct 6, 2023
ec7120a
Support for recursive deletes
Oct 6, 2023
e578637
Support for permissions testing
Oct 10, 2023
eed7742
Checkstyle fixes
Oct 10, 2023
71a5002
Moved recursive delete functionality in library
Oct 14, 2023
3c0c00e
Rename the vault auth group and cert file in int tests
Oct 16, 2023
8baeee3
Re-work after code review
Oct 17, 2023
f33274b
Merge pull request #539 from andamian/CADC-12729
pdowler Oct 26, 2023
e15cdb4
Merge branch 'vos2' into CADC-12558
pdowler Oct 26, 2023
f028e45
Merge pull request #538 from andamian/CADC-12558
pdowler Oct 26, 2023
9463d1a
merge and tweak
pdowler Oct 26, 2023
353ba22
vault: tweaks for upstream changes
pdowler Oct 26, 2023
a02ded8
Merge pull request #540 from pdowler/vos2
pdowler Oct 26, 2023
6e982e7
vault: add props to immutable props list
pdowler Oct 26, 2023
9b5c2f8
Merge pull request #541 from pdowler/vos2
pdowler Oct 26, 2023
17fd2aa
Support for recursive props
Nov 8, 2023
0dab0cb
Merge branch 'master' of https://github.com/opencadc/storage-inventor…
pdowler Nov 24, 2023
6b1d4d5
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
pdowler Nov 24, 2023
82de71b
vault: added code to restrict to local groups only, currently disabled
pdowler Nov 28, 2023
976d699
Merge branch 'master' of https://github.com/opencadc/storage-inventor…
pdowler Dec 8, 2023
f1d4a63
Small updates
Dec 11, 2023
0debf14
Updated dependencies
Dec 11, 2023
f858aca
Rework after code review
Dec 11, 2023
fc2c7ab
Merge pull request #543 from andamian/CADC-12562
pdowler Dec 11, 2023
4b3e284
move transfer negotiation code from raven to cadc-inventory-server
pdowler Dec 11, 2023
7cf6750
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
pdowler Dec 11, 2023
234c073
moved storage site availability check to library
pdowler Dec 11, 2023
ff50827
cadc-inventory-db: add KeyPairDAO
pdowler Dec 11, 2023
a66a21d
vault: uws subpackage
pdowler Dec 12, 2023
1db9f37
ProtocolsGenerator ctor change
pdowler Dec 12, 2023
fed6c9b
change ProtocolsGenerator API
pdowler Dec 12, 2023
0f194e8
cadc-inventory-db: rename KeyPair to PreauthKeyPair
pdowler Dec 13, 2023
ff3281f
rename some sql files
pdowler Dec 13, 2023
595fda8
checkstyle fix
pdowler Dec 13, 2023
c9a37cc
ProtocolsGenerator ctor change and public members to denote optional …
pdowler Dec 13, 2023
5bb8441
vault: lib changes, keygen, transfer progress
pdowler Dec 13, 2023
540ccdb
oops: add new files
pdowler Dec 13, 2023
34ea368
Merge branch 'master' of https://github.com/opencadc/storage-inventor…
pdowler Dec 14, 2023
a8bc98f
Intial version
Dec 14, 2023
7139143
Small update
Dec 14, 2023
2775f23
tweak ProtocolsGenerator to work in a storage site db (storageLocatio…
pdowler Dec 14, 2023
8328264
wteak vault transfer code for unstored data note
pdowler Dec 14, 2023
53bb955
minoc: re-org config and add initial trusted service key handling, st…
pdowler Dec 14, 2023
19f09f8
Re-work after code review
Dec 14, 2023
0d811ea
Fixed typo
Dec 14, 2023
a848fd3
Merge pull request #546 from andamian/CADC-12946
pdowler Dec 15, 2023
8eb72c7
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
pdowler Dec 15, 2023
b261935
move getKeyAction from vault to cadc-inventory-server
pdowler Dec 15, 2023
7e87b28
add reg logging to minoc
pdowler Dec 15, 2023
23a5a97
warn to debug in lib
pdowler Dec 16, 2023
54abbbb
integrate and test magic key pair code
pdowler Dec 16, 2023
8110acc
raven: add an admin pool that can init the inventory db and store pre…
pdowler Dec 18, 2023
6f7ba0c
vault: made preventNotFound configurable
pdowler Dec 18, 2023
b810036
vault-specific dbrc entry for test setup
pdowler Dec 18, 2023
2f40969
rework to correctly setup recoverableNamespace config
pdowler Dec 18, 2023
c61721d
Merge pull request #547 from pdowler/vos2
pdowler Dec 19, 2023
243ba7e
added NodeDAO(boolean origin) ctor
pdowler Dec 21, 2023
17587ca
vault: fix date related node properties
pdowler Jan 12, 2024
c890930
Merge pull request #548 from pdowler/vos2
pdowler Jan 15, 2024
6270d40
bug fix for immutable property handling
pdowler Jan 15, 2024
e0d980f
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
pdowler Jan 15, 2024
d6e9d12
vault :supress unchecked cast warning
pdowler Jan 16, 2024
240fff8
Merge pull request #549 from pdowler/vos2
pdowler Jan 16, 2024
8ffe5e1
vault: add support for separate nodes and inventory pools
pdowler Jan 18, 2024
84120f0
log.warn -> debug
pdowler Jan 18, 2024
5229b10
bug fix: add explicit config map for PreauthKeyPairDAO
pdowler Jan 18, 2024
2f4d36d
minor cleanup in VaultInitAction
pdowler Jan 19, 2024
179ff9a
raven: made preauth key usage optional
pdowler Jan 22, 2024
61146cc
fix for disabled preauth key use
pdowler Jan 22, 2024
c645b73
Merge branch 'vos2' of https://github.com/pdowler/storage-inventory i…
pdowler Jan 22, 2024
e6c9593
Merge pull request #550 from pdowler/vos2
pdowler Jan 23, 2024
07380e1
minoc: support for filename override in the URL path using colon sepa…
pdowler Jan 23, 2024
ca5b220
vault: fix pubkey stored in JNDI; provide filename override to url gen
pdowler Jan 24, 2024
e46663b
cadc-inventory-server allow filename override for pullFromVoSpace
pdowler Jan 24, 2024
9e1f725
minoc: made filename override only work in head/get
pdowler Jan 25, 2024
4c59a88
Merge branch 'vos2' of https://github.com/pdowler/storage-inventory i…
pdowler Jan 25, 2024
c3e515d
make ProtocolsGenerator filter sites and log consistently
pdowler Jan 26, 2024
cdada9a
minoc: use trust preauth config when determining readable and writable
pdowler Jan 26, 2024
afe0eec
vault: adapt to PathResolver changes; fix dao config usage
pdowler Jan 26, 2024
fe6c2c7
update vault README
pdowler Jan 26, 2024
4b17635
Merge pull request #551 from pdowler/vos2
pdowler Jan 26, 2024
ab0972f
ban colon (:) in artifact uri path components
pdowler Feb 2, 2024
1c8b94a
Merge branch 'vos2' of https://github.com/pdowler/storage-inventory i…
pdowler Feb 2, 2024
d13dfde
tweaks to vault-quota/Design
pdowler Feb 2, 2024
2fae9d0
Merge pull request #552 from pdowler/vos2
pdowler Feb 2, 2024
0a5345d
cadc-inventory-0.10.0
pdowler Feb 2, 2024
71664dc
Merge pull request #553 from pdowler/vos2
pdowler Feb 2, 2024
fa98d3c
Added the files end point
Feb 5, 2024
3fc90a6
Ready for code review
Feb 9, 2024
701e01f
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
Feb 9, 2024
cf1f6f4
Skipped incomplete test
Feb 9, 2024
268b55c
Changes after code review
Feb 12, 2024
62c5b19
fix cadc-test-vos dependency
pdowler Feb 12, 2024
c5a468a
Merge pull request #554 from andamian/CADC-12561
pdowler Feb 12, 2024
6795629
merge
pdowler Feb 12, 2024
adbc699
Required changes to make production vostools work
Feb 13, 2024
5495a66
Reworked
Feb 13, 2024
45a5f76
Merge pull request #555 from andamian/regression
pdowler Feb 13, 2024
ab6c940
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
pdowler Feb 13, 2024
fe4a210
adapt base Entity for library ctor changes
pdowler Feb 14, 2024
2f39d9e
cadc-inventory-db: add instanceID to HarvestState
pdowler Feb 16, 2024
ef87e13
luskan: use the DelegatingStorageManager
pdowler Feb 16, 2024
438284e
minoc: adapt to db refactoring
pdowler Feb 16, 2024
8227999
raven: fix content-dispisition header, adapt to db refactoring
pdowler Feb 16, 2024
460269e
vault: adapt to db refactoring
pdowler Feb 16, 2024
70ecf9a
made genSchema required because it is needed for correct init
pdowler Feb 16, 2024
e990724
fenwick, ratik: adapt to refactoring in cadc-inventory-db
pdowler Feb 16, 2024
fb484a6
cadc-inventory-server: remove unused InitDatabaseAction
pdowler Feb 16, 2024
1b692bc
critwall: adapt to cadc-inventory-db refactoring
pdowler Feb 16, 2024
f197591
tantar: update for cadc-inventory-db refactoring
pdowler Feb 16, 2024
bd7e186
ringhold: update for cadc-inventory-db refactoring
pdowler Feb 16, 2024
479b0b7
update vault-quota/Design
pdowler Feb 19, 2024
76d975d
add missing dao config for minoc int test
pdowler Feb 22, 2024
ddd87e0
raven: restore test db config
pdowler Feb 22, 2024
86c5079
vault: fix for test dao setup
pdowler Feb 23, 2024
ba76c2f
update vault-quota/Design details
pdowler Feb 23, 2024
93b0860
cadc-inventory-db: updates to support node size metadata and workflow
pdowler Feb 23, 2024
4bad11d
add vospace upgrade file
pdowler Feb 23, 2024
94d395e
add comment to inventory upgrade script
pdowler Feb 26, 2024
7ab48e3
remove unused changes in StorageSite
pdowler Feb 26, 2024
3081552
Merge pull request #556 from pdowler/vos2
pdowler Feb 26, 2024
0f65ab9
Merge branch 'vos2x' into vos2
pdowler Feb 27, 2024
b68b237
update dependencies
pdowler Feb 27, 2024
44a1c29
vault: transfer generator bug, API doc fix
pdowler Feb 27, 2024
3cd0dfd
updates to vault-quota design
pdowler Feb 28, 2024
5c2dcc0
vault bytesUsed rework
pdowler Mar 4, 2024
2e4b8d6
vault: add allocationParent support
pdowler Mar 5, 2024
2cec055
remove duplicate column from Node table sql
pdowler Mar 8, 2024
91868b3
Initial version
Mar 9, 2024
4bd306e
Merge branch 'vos2' of https://github.com/pdowler/storage-inventory i…
Mar 9, 2024
3fed596
Updated after pulling upstream updates
Mar 9, 2024
2387d3a
vault: ArtifactSync init
pdowler Mar 12, 2024
d1cade9
Merge branch 'vos2' of https://github.com/pdowler/storage-inventory i…
pdowler Mar 12, 2024
6c2ff85
always close() connections in iterator.close()
pdowler Mar 12, 2024
3c1602e
Merge branch 'vos2' of https://github.com/pdowler/storage-inventory i…
pdowler Mar 12, 2024
7548471
vault: fix availability check dao config for singlePool=false
pdowler Mar 12, 2024
353d8f4
vault-quota: update design docs
pdowler Mar 13, 2024
819d68f
rework
pdowler Mar 13, 2024
22718c8
rework: remove unused imports
pdowler Mar 13, 2024
347773c
Merge pull request #557 from pdowler/vos2
pdowler Mar 13, 2024
4273146
vault: pull doc improvements from PR-558
pdowler Mar 13, 2024
bc2a19d
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
pdowler Mar 13, 2024
e0ce694
Merge pull request #560 from pdowler/vos2
pdowler Mar 13, 2024
022ae46
Merge branch 'vos2' of https://github.com/opencadc/storage-inventory …
Mar 13, 2024
4dde678
Initial version
Mar 14, 2024
43f98bd
Rework after code review
Mar 14, 2024
742bc6b
Merge pull request #561 from andamian/CADC-13214
pdowler Mar 18, 2024
6a62b1c
moved ArtifactSyncWorker to vospace pkg
pdowler Mar 18, 2024
5f374cc
vospace.Node.sql: explicit text array to match code usage
pdowler Mar 18, 2024
1853e77
Merge pull request #562 from pdowler/vos2
pdowler Mar 18, 2024
6dd4bb1
vault: integrate ArtifactSyncWorker
pdowler Mar 18, 2024
5276e40
vault: update DataNode.bytesUsed from artifact in get
pdowler Mar 18, 2024
e284c6e
vault: enable availability modes to disable and enable background Art…
pdowler Mar 19, 2024
ea52a02
update VERSIONs for service images and add change log
pdowler Mar 19, 2024
ba3bd5c
update image versions
pdowler Mar 20, 2024
3ca9577
cadc-inventory and cadc-inventory-db versions 1.0.0
pdowler Mar 20, 2024
8a72987
update cadc-inventory lib dependencies for 1.0 milestone
pdowler Mar 20, 2024
d762437
rename ArtifactSync to DataNodeSizeSync for clarity
pdowler Mar 20, 2024
5bb6d74
fix open connection leak in HarvestStateDAO table maintenance code
pdowler Mar 25, 2024
209a95e
import cleanup
pdowler Mar 25, 2024
af92038
temporarily remove ringhold from CI
pdowler Mar 25, 2024
f737dee
vault code cleanup
pdowler Mar 25, 2024
c2c650b
DataNodeSizeSync logging in machine readable format
pdowler Mar 25, 2024
0c4c36e
vault: machine-readable logging in DataNodeSizeSync
pdowler Mar 25, 2024
7aa2a80
CADC-13234 update ringhold README for new config properties
jburke-cadc Mar 26, 2024
f6bb379
CADC-13234 update README after review
jburke-cadc Mar 26, 2024
419ef83
vault: cleanup and doc update
pdowler Mar 26, 2024
15b86dd
vault: improve server header info for files endpoint
pdowler Mar 28, 2024
b53037b
CADC-13234 update ringhold to query with configured namespace and uri…
jburke-cadc Mar 28, 2024
8c91699
rework after review
pdowler Apr 2, 2024
adbadf2
CADC-13234 checkstyle fix
jburke-cadc Apr 2, 2024
4780012
remaining SI image version to 1.0.0
pdowler Apr 2, 2024
bf7c302
Merge pull request #563 from pdowler/vos2
pdowler Apr 3, 2024
0748e9a
Merge branch 'vos2' into CADC-13234
pdowler Apr 3, 2024
593bb1e
CADC-13234 restore ringhold in github build
jburke-cadc Apr 3, 2024
a0f5f27
CADC-13234 fix github build formatting
jburke-cadc Apr 3, 2024
1cf7ea5
Merge pull request #564 from jburke-cadc/CADC-13234
pdowler Apr 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -137,6 +137,9 @@ public void setResourceID(URI resourceID) {

@Override
public String toString() {
if (instanceID != null) {
return HarvestState.class.getSimpleName() + "[" + instanceID + "," + name + "," + resourceID + "]";
}
return HarvestState.class.getSimpleName() + "[" + name + "," + resourceID + "]";
}
}
Original file line number Diff line number Diff line change
@@ -736,13 +736,14 @@ public void setSiteID(UUID siteID) {

@Override
public ResourceIterator<Artifact> query(DataSource ds) {
StringBuilder select = getSelectFromSQL(Artifact.class, false);
StringBuilder sb = new StringBuilder(" WHERE");
int whereLen = sb.length();

StringBuilder sb = new StringBuilder();
boolean where = false;
if (storageLocationRequired != null && storageLocationRequired) {
// ArtifactDAO.storedIterator
sb.append(" WHERE");
where = true;
if (storageBucket != null) {

sb.append(" storageLocation_storageBucket LIKE ? AND");
}
sb.append(" storageLocation_storageID IS NOT NULL");
@@ -755,33 +756,51 @@ public ResourceIterator<Artifact> query(DataSource ds) {
sb.append(" ORDER BY storageLocation_storageBucket, storageLocation_storageID");
}
} else if (storageLocationRequired != null && !storageLocationRequired) {
// ArtifactDAO.unstoredIterator
sb.append(" WHERE");
where = true;
sb.append(" storageLocation_storageID IS NULL");
}

// optional params:
// uriBucket
// siteID
// namespace
// minLastModified
if (uriBucket != null) {
// ArtifactDAO.iterator(null, ...)
if (sb.length() > whereLen) {
if (where) {
sb.append(" AND");
} else {
sb.append(" WHERE");
where = true;
}
sb.append(" uriBucket LIKE ?");
}
if (siteID != null) {
// ArtifactDAO.iterator(UUID, ...)
if (sb.length() > whereLen) {
if (where) {
sb.append(" AND");
} else {
sb.append(" WHERE");
where = true;
}
sb.append(" siteLocations @> ARRAY[?]");
}
if (namespace != null) {
// ArtifactDAO.iterator(Namespace, ...)
if (sb.length() > whereLen) {
if (where) {
sb.append(" AND");
} else {
sb.append(" WHERE");
where = true;
}
sb.append(" uri LIKE ?");
}
if (minLastModified != null) {
if (sb.length() > whereLen) {
if (where) {
sb.append(" AND");
} else {
sb.append(" WHERE");
where = true;
}
sb.append(" lastModified >= ?");
}
@@ -793,16 +812,11 @@ public ResourceIterator<Artifact> query(DataSource ds) {
}
}

if (sb.length() > whereLen) {
select.append(sb.toString());
}
StringBuilder select = getSelectFromSQL(Artifact.class, false);
select.append(sb.toString());
String sql = select.toString();
log.warn("sql: " + sql);
log.debug("sql: " + sql);

// params:
// storageBucket OR uriBucket
// siteID
// namespace
try {
Connection con = ds.getConnection();
log.debug("ArtifactIterator: setAutoCommit(false)");
@@ -1774,7 +1788,7 @@ private class ArtifactResultSetIterator implements ResourceIterator<Artifact> {
private final Connection con;
private final ResultSet rs;
boolean hasRow;
boolean closeWhenDone = false; // not a pooled connection
boolean closeWhenDone = true; // return to pool | assume close suppressed for static connections

ArtifactResultSetIterator(Connection con, ResultSet rs) throws SQLException {
this.con = con;
11 changes: 11 additions & 0 deletions vault-quota/Design.md
Original file line number Diff line number Diff line change
@@ -8,6 +8,17 @@ completed. In the case of a `vault` service co-located with a global SI, the new
Artifact is visible in the database once it is synced from the site of the PUT to
the global database by `fenwick` (or worst case: `ratik`).

## NOTE
This design was for supporting propagation of space used up the tree so that
allocation space used was dynamically updated as content was modified. While the
algorithm for doing that is nominally viable, the algorithm to validate and repair
incorrect container node sizes in a live system is excessively complex and probably
impossible to implement in practice (deadlocks, excessive database load and query
processing, etc).

**This design will not be completed and implemented** and is retained here for future
reference.

## TODO
The design below only takes into account incremental propagation of space used
by stored files. It is not complete/verified until we also come up with a validation
85 changes: 85 additions & 0 deletions vault-quota/NodeSize.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# vault quota design/algorithms

The definitive source of content-length (file size) of a DataNode comes from the
`inventory.Artifact` table and is not known until a PUT to storage is completed.
In the case of a `vault` service co-located with a single storage site (`minoc`),
the new Artifact is visible in the database as soon as the PUT to `minoc` is
completed. In the case of a `vault` service co-located with a global SI, the new
Artifact is visible in the database once it is synced from the site of the PUT to
the global database by `fenwick` (or worst case: `ratik`).

## incremental DataNode size algorithm
DataNode(s) require the `bytesUsed` be set so that sizes can be output from listing
container nodes without a join or query to the artifact table.

This is an event watcher that gets Artifact events (after a PUT) and intiates the
propagation of sizes (space used).
```
track progress using HarvestState (source, name: TBD)
incremental query for new artifacts in lastModified order
for each new Artifact:
query for DataNode (storageID = artifact.uri)
if Artifact.contentLength != Node.size:
start txn
lock datanode
recheck size diff
set dataNode.size
update HarvestState
commit txn
```

## validate DataNode vs Artifact discrepancies
These can be validated in parallel by multiple threads, subdivide work by bucket if we add
DataNode.storageBucket (== Artifact.uriBucket).

```
discrepancy: Artifact exists but DataNode does not
explanation: DataNode created, transfer negotiated, DataNode removed, transfer executed
evidence: DeletedNodeEvent exists
action: remove artifact, create DeletedArtifactEvent

discrepancy: Artifact exists but DataNode does not
explanation: DataNode created, Artifact put, DataNode deleted, Artifact delete failed
evidence: only possible with singlePool==false
action: remove artifact, create DeletedArtifactEvent

discrepancy: DataNode exists but Artifact does not
explanation: DataNode created, Artifact never (successfully) put (normal)
evidence: DataNode.nodeSize == 0 or null
action: none

discrepancy: DataNode exists but Artifact does not
explanation: deleted or lost Artifact
evidence: DataNode.nodeSize != 0 (deleted vs lost: DeletedArtifactEvent exists)
action: lock nodes, fix dataNode and propagate delta to parent

discrepancy: DataNode.nodeSize != Artifact.contentLength
explanation: artifact written (if DataNode.size > 0: replaced)
action: lock nodes, fix DataNode and propagate delta to parent
```
Required lock order: child-parent or parent-child OK.

The most generic implementation is a merge join of two iterators (see ratik, tantar):
```
Iterator<Artifact> aiter = artifactDAO.iterator(vaultNamespace, bucket); // uriBucket,uri order
Iterator<DataNode> niter = nodeDAO.iterator(bucket); // storageBucket,storageID order
```

## database changes required
note: all field and column names TBD
* add `transient Long bytesUsed` to ContainerNode and DataNode
* add `bytesUsed` to the `vospace.Node` table
* add `storageBucket` to DataNode?? TBD
* add `storageBucket` to `vospace.Node` table

## cadc-inventory-db API required immediately
* incremental sync query/iterator: ArtifactDAO.iterator(Namespace ns, String uriBucketPrefix, Date minLastModified, boolean ordered)
order by lastModified if set
* lookup DataNode by storageID: NodeDAO.getDataNode(URI storageID)
* indices to support new queries

## cadc-inventory-db API required later (tentative)
* validate-by-bucket: use ArtifactDAO.iterator(String uriBucketPrefix, boolean ordered, Namespace ns)
* validate-by-bucket: NodeDAO.dataNodeIterator(String storageBucketPrefix, boolean ordered)
* indices to support new queries

8 changes: 4 additions & 4 deletions vault/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Storage Inventory VOSpace-2.1 service (vault)

The `vault` servcie is an implementation of the <a href="https://www.ivoa.net/documents/VOSpace/">IVOA VOSpace</a>
specification designed to co-exist with other storage-inventory components. It provides a heirarchical data
organization laye on top of the storage management of storage-inventory.
The `vault` service is an implementation of the <a href="https://www.ivoa.net/documents/VOSpace/">IVOA VOSpace</a>
specification designed to co-exist with other storage-inventory components. It provides a hierarchical data
organization layer on top of the storage management of storage-inventory.

The simplest configuration would be to deploy `vault` with `minoc` with a single metadata database and single
back end storage system. Details: TBD.
@@ -106,7 +106,7 @@ _all known_ sites. It only makes sense to enable this when `vault` is running in
genuine not-found cases: transfer negotiation to GET the file that was never PUT.

The _allocationParent_ is a path to a container node (directory) which contains space allocations. An allocation
is owned by a user (uisually different from the _rootOwner_ admin user) who is responsible for the allocation
is owned by a user (usually different from the _rootOwner_ admin user) who is responsible for the allocation
and all conntent therein. The owner of an allocation is granted additional permissions within their
allocation (they can read/write/delete anything) so the owner cannot be blocked from access to any content
within their allocation. This probably only matters for multi-user projects. Multiple _allocationParent_(s) may
30 changes: 30 additions & 0 deletions vault/src/intTest/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Storage Inventory VOSpace-2.1 service (vault)

The simplest configuration that deploys `vault` with `minoc` with a single metadata database and single
back end storage system is sufficient to run the `vault` integration tests. The tests also relly on the
presence of the root owner X509 certificate in `build/classes/java/intTest/vault-test.pem`.
Some tests (primarily permission tests) will be skipped unless the certificate of a second user is present
in `build/classes/java/intTest/vault-auth-test.pem`. This user has to be member of the `ivo://cadc.nrc.ca/gms?opencadc-vospace-test`
group. The names of these certificates and groups are hardcoded in the `vault` int tests classes.

The int tests suite also relies on a specific configuration of the `vault` service:
### vault.properties
```
# service identity
org.opencadc.vault.resourceID = ivo://opencadc.org/vault

# (optional) identify which container nodes are allocations
org.opencadc.vault.allocationParent = /

# consistency settings
org.opencadc.vault.consistency.preventNotFound=true

# vault database settings
org.opencadc.vault.inventory.schema = inventory
org.opencadc.vault.vospace.schema = vault
org.opencadc.vault.singlePool = true

# root container nodes
org.opencadc.vault.root.owner = {owner of root node}

```
92 changes: 92 additions & 0 deletions vault/src/intTest/java/org/opencadc/vault/Constants.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
/*
************************************************************************
******************* CANADIAN ASTRONOMY DATA CENTRE *******************
************** CENTRE CANADIEN DE DONNÉES ASTRONOMIQUES **************
*
* (c) 2024. (c) 2024.
* Government of Canada Gouvernement du Canada
* National Research Council Conseil national de recherches
* Ottawa, Canada, K1A 0R6 Ottawa, Canada, K1A 0R6
* All rights reserved Tous droits réservés
*
* NRC disclaims any warranties, Le CNRC dénie toute garantie
* expressed, implied, or énoncée, implicite ou légale,
* statutory, of any kind with de quelque nature que ce
* respect to the software, soit, concernant le logiciel,
* including without limitation y compris sans restriction
* any warranty of merchantability toute garantie de valeur
* or fitness for a particular marchande ou de pertinence
* purpose. NRC shall not be pour un usage particulier.
* liable in any event for any Le CNRC ne pourra en aucun cas
* damages, whether direct or être tenu responsable de tout
* indirect, special or general, dommage, direct ou indirect,
* consequential or incidental, particulier ou général,
* arising from the use of the accessoire ou fortuit, résultant
* software. Neither the name de l'utilisation du logiciel. Ni
* of the National Research le nom du Conseil National de
* Council of Canada nor the Recherches du Canada ni les noms
* names of its contributors may de ses participants ne peuvent
* be used to endorse or promote être utilisés pour approuver ou
* products derived from this promouvoir les produits dérivés
* software without specific prior de ce logiciel sans autorisation
* written permission. préalable et particulière
* par écrit.
*
* This file is part of the Ce fichier fait partie du projet
* OpenCADC project. OpenCADC.
*
* OpenCADC is free software: OpenCADC est un logiciel libre ;
* you can redistribute it and/or vous pouvez le redistribuer ou le
* modify it under the terms of modifier suivant les termes de
* the GNU Affero General Public la “GNU Affero General Public
* License as published by the License” telle que publiée
* Free Software Foundation, par la Free Software Foundation
* either version 3 of the : soit la version 3 de cette
* License, or (at your option) licence, soit (à votre gré)
* any later version. toute version ultérieure.
*
* OpenCADC is distributed in the OpenCADC est distribué
* hope that it will be useful, dans l’espoir qu’il vous
* but WITHOUT ANY WARRANTY; sera utile, mais SANS AUCUNE
* without even the implied GARANTIE : sans même la garantie
* warranty of MERCHANTABILITY implicite de COMMERCIALISABILITÉ
* or FITNESS FOR A PARTICULAR ni d’ADÉQUATION À UN OBJECTIF
* PURPOSE. See the GNU Affero PARTICULIER. Consultez la Licence
* General Public License for Générale Publique GNU Affero
* more details. pour plus de détails.
*
* You should have received Vous devriez avoir reçu une
* a copy of the GNU Affero copie de la Licence Générale
* General Public License along Publique GNU Affero avec
* with OpenCADC. If not, see OpenCADC ; si ce n’est
* <http://www.gnu.org/licenses/>. pas le cas, consultez :
* <http://www.gnu.org/licenses/>.
*
************************************************************************
*/

package org.opencadc.vault;

import ca.nrc.cadc.util.FileUtil;
import java.io.File;
import java.net.URI;
import org.apache.log4j.Logger;
import org.opencadc.gms.GroupURI;

/**
*
* @author pdowler
*/
public class Constants {
private static final Logger log = Logger.getLogger(Constants.class);

static URI RESOURCE_ID = URI.create("ivo://opencadc.org/vault");

static File ADMIN_CERT = FileUtil.getFileFromResource("vault-test.pem", Constants.class);
static File ALT_CERT = FileUtil.getFileFromResource("vault-auth-test.pem", Constants.class);

static GroupURI ALT_GROUP = new GroupURI(URI.create("ivo://cadc.nrc.ca/gms?opencadc-vospace-test"));

private Constants() {
}
}
6 changes: 3 additions & 3 deletions vault/src/intTest/java/org/opencadc/vault/FilesTest.java
Original file line number Diff line number Diff line change
@@ -87,9 +87,9 @@ public class FilesTest extends org.opencadc.conformance.vos.FilesTest {
Log4jInit.setLevel("org.opencadc.vospace", Level.DEBUG);
}

private static File ADMIN_CERT = FileUtil.getFileFromResource("vault-test.pem", FilesTest.class);

public FilesTest() {
super(URI.create("ivo://opencadc.org/vault"), ADMIN_CERT);
super(Constants.RESOURCE_ID, Constants.ADMIN_CERT);

enableTestDataNodePermission(Constants.ALT_CERT);
}
}
7 changes: 2 additions & 5 deletions vault/src/intTest/java/org/opencadc/vault/NodesTest.java
Original file line number Diff line number Diff line change
@@ -88,13 +88,10 @@ public class NodesTest extends org.opencadc.conformance.vos.NodesTest {
Log4jInit.setLevel("org.opencadc.vospace", Level.DEBUG);
}

private static File ADMIN_CERT = FileUtil.getFileFromResource("vault-test.pem", NodesTest.class);

public NodesTest() {
super(URI.create("ivo://opencadc.org/vault"), ADMIN_CERT);
super(Constants.RESOURCE_ID, Constants.ADMIN_CERT);

File altCert = FileUtil.getFileFromResource("vault-auth-test.pem", NodesTest.class);
enablePermissionTests(new GroupURI(URI.create("ivo://cadc.nrc.ca/gms?opencadc-vospace-test")), altCert);
enablePermissionTests(Constants.ALT_GROUP, Constants.ALT_CERT);

// vault does not check the actual groups in the permission props tests, hence they can be made up.
enablePermissionPropsTest(new GroupURI(URI.create("ivo://myauth/gms?gr1")), new GroupURI(URI.create("ivo://myauth/gms?gr2")));
65 changes: 2 additions & 63 deletions vault/src/intTest/java/org/opencadc/vault/TransferTest.java
Original file line number Diff line number Diff line change
@@ -67,23 +67,9 @@

package org.opencadc.vault;

import ca.nrc.cadc.db.ConnectionConfig;
import ca.nrc.cadc.db.DBConfig;
import ca.nrc.cadc.db.DBUtil;
import ca.nrc.cadc.util.FileUtil;
import ca.nrc.cadc.util.Log4jInit;
import java.io.File;
import java.net.URI;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.junit.Before;
import org.junit.Ignore;
import org.opencadc.inventory.StorageSite;
import org.opencadc.inventory.db.SQLGenerator;
import org.opencadc.inventory.db.StorageSiteDAO;

/**
*
@@ -104,55 +90,8 @@ public class TransferTest extends org.opencadc.conformance.vos.TransferTest {
static String DATABASE = "cadctest";
static String SCHEMA = "inventory";

private static File ADMIN_CERT = FileUtil.getFileFromResource("vault-test.pem", NodesTest.class);

public TransferTest() {
super(URI.create("ivo://opencadc.org/vault"), ADMIN_CERT);
}

@Before
public void checkGlobal() throws Exception {
// make sure inventory.StorageSite has a readable/writable ivo://opencadc.org/minoc

try {
DBConfig dbrc = new DBConfig();
ConnectionConfig cc = dbrc.getConnectionConfig(SERVER, DATABASE);
DBUtil.createJNDIDataSource("jdbc/inventory", cc);

Map<String,Object> config = new TreeMap<String,Object>();
config.put(SQLGenerator.class.getName(), SQLGenerator.class);
config.put("jndiDataSourceName", "jdbc/inventory");
config.put("invSchema", SCHEMA);
config.put("genSchema", SCHEMA);

StorageSiteDAO dao = new StorageSiteDAO(false);
dao.setConfig(config);
Set<StorageSite> sites = dao.list();
if (sites.isEmpty()) {
StorageSite ss = new StorageSite(URI.create("ivo://opencadc.org/minoc"), "vault-test-minoc", true, true);
dao.put(ss);
log.info("created record in local db: " + ss);
}
sites = dao.list();
for (StorageSite ss : sites) {
log.info("storage sites in local db: " + ss);
}
} catch (Exception ex) {
log.error("setup failed", ex);
throw ex;
}
}

@Override
public void asyncMoveTest() {
super.asyncMoveTest();
}

@Ignore
@Override
public void syncPushPullTest() {
super.syncPushPullTest();
super(Constants.RESOURCE_ID, Constants.ADMIN_CERT);
enableTestDataNodePermission(Constants.ALT_GROUP, Constants.ALT_CERT);
}


}
Original file line number Diff line number Diff line change
@@ -304,7 +304,7 @@ public boolean isAllocation(ContainerNode cn) {
}
ContainerNode p = cn.parent;
for (ContainerNode ap : allocationParents) {
if (absoluteEquals(p.parent, ap)) {
if (p.getID().equals(ap.getID())) {
return true;
}
}
37 changes: 34 additions & 3 deletions vault/src/main/java/org/opencadc/vault/VaultInitAction.java
Original file line number Diff line number Diff line change
@@ -88,11 +88,13 @@
import org.apache.log4j.Logger;
import org.opencadc.inventory.Namespace;
import org.opencadc.inventory.PreauthKeyPair;
import org.opencadc.inventory.db.HarvestStateDAO;
import org.opencadc.inventory.db.PreauthKeyPairDAO;
import org.opencadc.inventory.db.SQLGenerator;
import org.opencadc.inventory.db.StorageSiteDAO;
import org.opencadc.inventory.db.version.InitDatabaseSI;
import org.opencadc.inventory.transfer.StorageSiteAvailabilityCheck;
import org.opencadc.vault.metadata.ArtifactSync;
import org.opencadc.vospace.db.InitDatabaseVOS;
import org.opencadc.vospace.server.NodePersistence;
import org.springframework.dao.DataIntegrityViolationException;
@@ -133,6 +135,7 @@ public class VaultInitAction extends InitAction {
private String jndiPreauthKeys;
private String jndiSiteAvailabilities;
private Thread availabilityCheck;
private Thread artifactSync;

public VaultInitAction() {
super();
@@ -146,6 +149,7 @@ public void doInit() {
initNodePersistence();
initKeyPair();
initAvailabilityCheck();
initBackgroundWorkers();
}

@Override
@@ -165,6 +169,7 @@ public void doShutdown() {
}

terminateAvailabilityCheck();
terminateBackgroundWorkers();
}

/**
@@ -412,9 +417,9 @@ private void initKeyPair() {
}
}

void initAvailabilityCheck() {
private void initAvailabilityCheck() {
StorageSiteDAO storageSiteDAO = new StorageSiteDAO();
storageSiteDAO.setConfig(getDaoConfig(props));
storageSiteDAO.setConfig(getInvConfig(props));

this.jndiSiteAvailabilities = appName + "-" + StorageSiteAvailabilityCheck.class.getName();
terminateAvailabilityCheck();
@@ -423,7 +428,7 @@ void initAvailabilityCheck() {
this.availabilityCheck.start();
}

private final void terminateAvailabilityCheck() {
private void terminateAvailabilityCheck() {
if (this.availabilityCheck != null) {
try {
log.info("terminating AvailabilityCheck Thread...");
@@ -436,6 +441,8 @@ private final void terminateAvailabilityCheck() {
this.availabilityCheck = null;
}
}

// ugh: bind() is inside StorageSiteAvailabilityCheck but unbind() is here
try {
InitialContext initialContext = new InitialContext();
initialContext.unbind(this.jndiSiteAvailabilities);
@@ -444,4 +451,28 @@ private final void terminateAvailabilityCheck() {
}
}

private void initBackgroundWorkers() {
HarvestStateDAO hsDAO = new HarvestStateDAO();
hsDAO.setConfig(getDaoConfig(props));

terminateBackgroundWorkers();
this.artifactSync = new Thread(new ArtifactSync(hsDAO));
artifactSync.setDaemon(true);
artifactSync.start();
}

private void terminateBackgroundWorkers() {
if (this.artifactSync != null) {
try {
log.info("terminating ArtifactSync Thread...");
this.artifactSync.interrupt();
this.artifactSync.join();
log.info("terminating ArtifactSync Thread... [OK]");
} catch (Throwable t) {
log.info("failed to terminate ArtifactSync thread", t);
} finally {
this.artifactSync = null;
}
}
}
}
Original file line number Diff line number Diff line change
@@ -70,7 +70,6 @@
import ca.nrc.cadc.auth.AuthenticationUtil;
import ca.nrc.cadc.auth.IdentityManager;
import ca.nrc.cadc.net.ResourceNotFoundException;
import ca.nrc.cadc.reg.Standards;
import ca.nrc.cadc.uws.Parameter;
import ca.nrc.cadc.vosi.Availability;
import java.io.IOException;
@@ -193,7 +192,6 @@ private List<Protocol> handleDataNode(DataNode node, String filename, Transfer t
log.debug("requested protocol: " + p);
if (!protoURIs.contains(p.getUri())) {
Protocol anonProto = new Protocol(p.getUri());
//anonProto.setSecurityMethod(Standards.SECURITY_METHOD_ANON);
artifactTrans.getProtocols().add(anonProto);
protoURIs.add(p.getUri());
log.debug("Added anon protocol for " + p.getUri());
10 changes: 10 additions & 0 deletions vault/src/main/java/org/opencadc/vault/files/GetAction.java
Original file line number Diff line number Diff line change
@@ -67,14 +67,18 @@

package org.opencadc.vault.files;

import ca.nrc.cadc.auth.AuthenticationUtil;
import ca.nrc.cadc.net.TransientException;
import java.net.HttpURLConnection;
import java.net.URI;
import java.util.List;
import javax.security.auth.Subject;
import org.apache.log4j.Logger;
import org.opencadc.vospace.DataNode;
import org.opencadc.vospace.NodeProperty;
import org.opencadc.vospace.VOS;
import org.opencadc.vospace.VOSURI;
import org.opencadc.vospace.server.NodeFault;
import org.opencadc.vospace.server.Utils;
import org.opencadc.vospace.server.transfers.TransferGenerator;
import org.opencadc.vospace.transfer.Direction;
@@ -96,6 +100,12 @@ public GetAction() {
public void doAction() throws Exception {
DataNode node = resolveAndSetMetadata();

Subject caller = AuthenticationUtil.getCurrentSubject();
if (!voSpaceAuthorizer.hasSingleNodeReadPermission(node, caller)) {
// TODO: should output requested vos URI here
throw NodeFault.PermissionDenied.getStatus(syncInput.getPath());
}

if (node.bytesUsed == null || node.bytesUsed == 0L) {
// empty file
syncOutput.setCode(HttpURLConnection.HTTP_NO_CONTENT);
Original file line number Diff line number Diff line change
@@ -162,7 +162,7 @@ DataNode resolveAndSetMetadata() throws Exception {
}
}
syncOutput.setCode(200);
return (DataNode)node;
return dn;
}

}
157 changes: 157 additions & 0 deletions vault/src/main/java/org/opencadc/vault/metadata/ArtifactSync.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
/*
************************************************************************
******************* CANADIAN ASTRONOMY DATA CENTRE *******************
************** CENTRE CANADIEN DE DONNÉES ASTRONOMIQUES **************
*
* (c) 2024. (c) 2024.
* Government of Canada Gouvernement du Canada
* National Research Council Conseil national de recherches
* Ottawa, Canada, K1A 0R6 Ottawa, Canada, K1A 0R6
* All rights reserved Tous droits réservés
*
* NRC disclaims any warranties, Le CNRC dénie toute garantie
* expressed, implied, or énoncée, implicite ou légale,
* statutory, of any kind with de quelque nature que ce
* respect to the software, soit, concernant le logiciel,
* including without limitation y compris sans restriction
* any warranty of merchantability toute garantie de valeur
* or fitness for a particular marchande ou de pertinence
* purpose. NRC shall not be pour un usage particulier.
* liable in any event for any Le CNRC ne pourra en aucun cas
* damages, whether direct or être tenu responsable de tout
* indirect, special or general, dommage, direct ou indirect,
* consequential or incidental, particulier ou général,
* arising from the use of the accessoire ou fortuit, résultant
* software. Neither the name de l'utilisation du logiciel. Ni
* of the National Research le nom du Conseil National de
* Council of Canada nor the Recherches du Canada ni les noms
* names of its contributors may de ses participants ne peuvent
* be used to endorse or promote être utilisés pour approuver ou
* products derived from this promouvoir les produits dérivés
* software without specific prior de ce logiciel sans autorisation
* written permission. préalable et particulière
* par écrit.
*
* This file is part of the Ce fichier fait partie du projet
* OpenCADC project. OpenCADC.
*
* OpenCADC is free software: OpenCADC est un logiciel libre ;
* you can redistribute it and/or vous pouvez le redistribuer ou le
* modify it under the terms of modifier suivant les termes de
* the GNU Affero General Public la “GNU Affero General Public
* License as published by the License” telle que publiée
* Free Software Foundation, par la Free Software Foundation
* either version 3 of the : soit la version 3 de cette
* License, or (at your option) licence, soit (à votre gré)
* any later version. toute version ultérieure.
*
* OpenCADC is distributed in the OpenCADC est distribué
* hope that it will be useful, dans l’espoir qu’il vous
* but WITHOUT ANY WARRANTY; sera utile, mais SANS AUCUNE
* without even the implied GARANTIE : sans même la garantie
* warranty of MERCHANTABILITY implicite de COMMERCIALISABILITÉ
* or FITNESS FOR A PARTICULAR ni d’ADÉQUATION À UN OBJECTIF
* PURPOSE. See the GNU Affero PARTICULIER. Consultez la Licence
* General Public License for Générale Publique GNU Affero
* more details. pour plus de détails.
*
* You should have received Vous devriez avoir reçu une
* a copy of the GNU Affero copie de la Licence Générale
* General Public License along Publique GNU Affero avec
* with OpenCADC. If not, see OpenCADC ; si ce n’est
* <http://www.gnu.org/licenses/>. pas le cas, consultez :
* <http://www.gnu.org/licenses/>.
*
************************************************************************
*/

package org.opencadc.vault.metadata;

import java.net.URI;
import java.util.Date;
import java.util.UUID;
import org.apache.log4j.Logger;
import org.opencadc.inventory.Artifact;
import org.opencadc.inventory.db.HarvestState;
import org.opencadc.inventory.db.HarvestStateDAO;

/**
* Main artifact-sync agent that enables incremental sync of Artifact
* metadata to Node.
*
* @author pdowler
*/
public class ArtifactSync implements Runnable {
private static final Logger log = Logger.getLogger(ArtifactSync.class);

private static final long SHORT_SLEEP = 12000L;
private static final long LONG_SLEEP = 2 * SHORT_SLEEP;
private static final long EVICT_AGE = 3 * LONG_SLEEP;

private final UUID instanceID = UUID.randomUUID();
private final HarvestStateDAO dao;
private String name = Artifact.class.getSimpleName();
private URI resourceID = URI.create("jdbc/inventory");

public ArtifactSync(HarvestStateDAO dao) {
this.dao = dao;

// fenwick setup for production workload:
//dao.setUpdateBufferCount(99); // buffer 99 updates, do every 100
//dao.setMaintCount(999); // buffer 999 so every 1000 real updates aka every 1e5 events

// here, we need timestamp updates to retain leader status, so
// dao.setMaintCount(9999); // every 1e4
}

@Override
public void run() {
try {
Thread.sleep(SHORT_SLEEP);

while (true) {
boolean leader = false;
log.debug("check leader " + instanceID);
HarvestState state = dao.get(name, resourceID);
log.debug("check leader " + instanceID + " found: " + state);
if (state.instanceID == null) {
state.instanceID = instanceID;
dao.put(state);
state = dao.get(state.getID());
log.debug("created: " + state);
}
if (instanceID.equals(state.instanceID)) {
log.debug("still the leader...");
dao.put(state, true);
leader = true;
} else {
// see if we should perform a coup...
Date now = new Date();
long age = now.getTime() - state.getLastModified().getTime();
if (age > EVICT_AGE) {

state.instanceID = instanceID;
dao.put(state);
state = dao.get(state.getID());
leader = true;
log.debug("EVICTED " + state.instanceID + " because age " + age + " > " + EVICT_AGE);
}
}

if (leader) {
log.debug("leader " + state.instanceID + " starting worker...");
// TODO
dao.flushBufferedState();
Thread.sleep(SHORT_SLEEP / 2L); // for testing
log.debug("idle leader " + state.instanceID + " sleep=" + SHORT_SLEEP);
Thread.sleep(SHORT_SLEEP);
} else {
log.debug("not leader: sleep=" + LONG_SLEEP);
Thread.sleep(LONG_SLEEP);
}
}
} catch (InterruptedException ex) {
log.debug("interrupted - assuming shutdown", ex);
}
}
}
13 changes: 13 additions & 0 deletions vault/src/main/webapp/META-INF/context.xml
Original file line number Diff line number Diff line change
@@ -27,6 +27,19 @@
removeAbandoned="false"
testOnBorrow="true" validationQuery="select 123" />

<!-- NOTE single connection is hard coded here!! -->
<Resource name="jdbc/iterator"
auth="Container"
type="javax.sql.DataSource"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" closeMethod="close"
minEvictableIdleTimeMillis="60000" timeBetweenEvictionRunsMillis="30000"
maxWait="20000"
initialSize="0" minIdle="0" maxIdle="1" maxActive="1"
username="${org.opencadc.vault.inventory.username}" password="${org.opencadc.vault.inventory.password}"
driverClassName="org.postgresql.Driver" url="${org.opencadc.vault.inventory.url}"
removeAbandoned="false"
testOnBorrow="true" validationQuery="select 123" />


<Resource name="jdbc/uws"
auth="Container"