Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read data in batches from solr using cursorMark #364

Open
Ahmed-Salah6011 opened this issue Dec 31, 2024 · 0 comments
Open

Read data in batches from solr using cursorMark #364

Ahmed-Salah6011 opened this issue Dec 31, 2024 · 0 comments

Comments

@Ahmed-Salah6011
Copy link

When attempting to read documents in batches using cursorMark and rows parameters with the above code sample
val solrDF = spark.read .format("solr") .option("zkHost", zookeeperHosts) .option("collection", collectionName) .option("query", configuredSolrQuery) .option("rows", batchSize) .option("cursorMark", cursorMark) .option("wt" , "json") .option("sort","MSG_REF_UK_ID asc") .load()

the returned solrDF doens't contain the nextCursorMark returned by solr in the response which doesn't allow using this to load the data from solr in batches

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant