Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search for usks - INCOMPATIBLE CHANGE REQUIRES MATCHING UPDATE IN plugin-Library #8

Open
wants to merge 42 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
513bf0d
Clarify parameter name
debbiedub Jun 12, 2021
1706991
Show info on pages
debbiedub Jun 12, 2021
0b951a9
Add new category "NEW" for never attempted pages
debbiedub Jun 12, 2021
79c9a8f
Set comment when an URI is FAILED or SUCCEEDED
debbiedub Jun 12, 2021
84b60c5
Add counting of USK handling
debbiedub Jun 17, 2021
f0d8ee9
Subscribe USKs for queried URIs
debbiedub Jun 17, 2021
94637a1
Subscribe to all old USKs
debbiedub Jun 17, 2021
e93e963
Improve handling in onFoundEdition
debbiedub Jun 17, 2021
2a9fbd9
Patch the already created database with new status
debbiedub Jun 17, 2021
e4cc3d2
Factor out the subscribing from within the lock
debbiedub Jun 17, 2021
0a95c66
Count subscribed USKs and their replaced URIs separately
debbiedub Jun 18, 2021
2d910f1
Change to trigger fetches regularly instead of on event
debbiedub Jun 18, 2021
fba4c9b
Abort a fetch after 10 hours with a failure.
debbiedub Aug 8, 2021
4991a42
Improve logging
debbiedub Jan 9, 2022
e3506d3
Move the changes of the page out of the lock in startSubscribeUSK
debbiedub Jan 9, 2022
ca0aece
Avoid searching for new jobs if there isn't any slots available
debbiedub Jan 18, 2022
8d34f0c
Cleaned code according to Eclipse suggestions
debbiedub Jan 18, 2022
74045cc
Attempt to move entries to the right list
debbiedub Jan 14, 2022
1b4a538
Factor out the list of variables to index it on Status
debbiedub Jan 14, 2022
0b3d20e
Partial work for restructuring of pages
debbiedub Jan 19, 2022
d4b6de1
Fix Eclipse warnings
debbiedub Jan 22, 2022
9197ce7
Fix so that the Status-generated page lists can be saved
debbiedub Jan 23, 2022
f70c387
Never put USKs with search-suggested-edition (-) in the database
debbiedub Jan 23, 2022
284c124
Add a new way of persisting pages allowing for USKs
debbiedub Jan 23, 2022
c174e56
Add debug logging on PageCallBack
debbiedub Jan 23, 2022
417ef97
Add searching for USKs
debbiedub Feb 13, 2022
2177785
Fill up the queue more often
debbiedub Mar 16, 2022
b525315
Remove printout to stderr
debbiedub Mar 20, 2022
4742f0e
Allow info from library to show up on spider page
debbiedub Mar 20, 2022
8f64caa
Improved the handling of USKs
debbiedub Apr 3, 2022
e8976d3
Avoid having redirected USKs and KSKs ending up in DONE
debbiedub May 7, 2022
131e661
Separate control of different kinds of fetches
debbiedub May 21, 2022
5293d4d
Process fetched USKs with the oldest first
debbiedub Jun 5, 2022
dfff40f
Count editionsFound in the right variable
debbiedub Jun 5, 2022
da69a1b
Delay queueing of new editions
debbiedub Jun 19, 2022
556df6b
Cleanup log message
debbiedub Jul 15, 2022
42df11d
Index runningFetches on FreenetURI instead of String
debbiedub Jul 15, 2022
c5da094
Avoid having onFailure called before the future is set up
debbiedub Jul 16, 2022
e30f577
Fix problem caused by changing runningFetches to FreenetURI
debbiedub Jul 18, 2022
1738a4a
Shorten the bulk size and increase the frequency
debbiedub Jul 18, 2022
983065e
Shortened the logging on UnsafeContentTypeException.
debbiedub Sep 30, 2022
76d012d
Clean out uris if the bad listed list is augmented and fetch fails
debbiedub Sep 30, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .classpath
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<classpathentry kind="src" path="test"/>
<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
<classpathentry combineaccessrules="false" kind="src" path="/fred"/>
<classpathentry kind="lib" path="/usr/share/java/junit.jar"/>
<classpathentry kind="lib" path="/fred/lib/freenet/freenet-ext.jar" sourcepath="/Contrib/db4o/src/db4oj"/>
<classpathentry kind="con" path="org.eclipse.jdt.junit.JUNIT_CONTAINER/4"/>
<classpathentry kind="output" path="build"/>
</classpath>
15 changes: 10 additions & 5 deletions src/plugins/Spider/LibraryBuffer.java
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ public synchronized int bufferUsageEstimate() {
public void start() {
// Do in a transaction so it gets committed separately.
spider.db.beginThreadTransaction(Storage.EXCLUSIVE_TRANSACTION);
spider.resetPages(Status.NOT_PUSHED, Status.QUEUED);
spider.resetPages(Status.NOT_PUSHED, Status.NEW);
spider.db.endThreadTransaction();
}

Expand Down Expand Up @@ -161,13 +161,12 @@ private void sendBuffer(int bufferUsageEstimated) {
long tStart = System.currentTimeMillis();
try {
Logger.normal(this, "Sending buffer of estimated size " + bufferUsageEstimated + " bytes to Library");
long totalPagesIndexed = spider.getRoot().getPageCount(Status.INDEXED);
Bucket bucket = pr.getNode().clientCore.tempBucketFactory.makeBucket(3000000);
writeToPush(totalPagesIndexed, bucket);
writeToPush(totalPagesIndexed(), bucket);
innerSend(bucket);
Logger.normal(this, "Buffer successfully sent to Library, size = "+bucket.size());
// Not a separate transaction, commit with the index updates.
spider.resetPages(Status.NOT_PUSHED, Status.INDEXED);
spider.donePages();
} catch (IOException ex) {
Logger.error(this, "Could not make bucket to transfer buffer", ex);
}
Expand All @@ -186,6 +185,12 @@ private void sendBuffer(int bufferUsageEstimated) {
System.out.println("Restored data from last time from "+SAVE_FILE);
}
}

private long totalPagesIndexed() {
return spider.getRoot().getPageCount(Status.DONE)
+ spider.getRoot().getPageCount(Status.PROCESSED_KSK)
+ spider.getRoot().getPageCount(Status.PROCESSED_USK);
}

private synchronized Bucket writeToPush(long totalPagesIndexed, Bucket bucket) throws IOException {
OutputStream os = bucket.getOutputStream();
Expand Down Expand Up @@ -271,7 +276,7 @@ public void terminate() {
FileBucket bucket = new FileBucket(SAVE_FILE, false, false, false, false);
long totalPagesIndexed;
try {
totalPagesIndexed = spider.getRoot().getPageCount(Status.INDEXED);
totalPagesIndexed = totalPagesIndexed();
} catch (Throwable t) {
totalPagesIndexed = -1;
// FIXME I don't understand why this (ClassNotFoundException) happens, we have not closed the class loader yet.
Expand Down
Loading