-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add limit and metadata_version parameters to the settings v2 endpoint #830
Comments
Based on the quick investigation:
Moreover, If the page number param is not provided, all the records will be returned in one go to ensure backward compatibility.
While settings metadata is created from the settings object and values are updated based on the settings key and UUID. It inserts new records in the metadata table if the records are not already there else it updates the records on the same entries. |
@rehammuzzamil yeah on update the Duplicates are occurring since on the i.e
@dubdabasoduba Was this the intention? |
A little clarification, the connector needs to be able to pull all settings from the api, and be able to pull any updates/edits after. The plan as of right now is
(this is with the assumption that any updates and new records are assigned a new settingMetadataId which is incremental) so generally I think we only need to add these 2 parameters to that endpoint. |
Hi @Wambere, I believe In the case of serverVersion, as @bennsimon mentioned above, all the rows of setting_metadata are using the same server_version. |
The reason for this is to maintain compatibility with the v1 of this endpoint. If you updated the |
@Wambere I think the edge case @rehammuzzamil is pointing out here is that the data warehouse might have a setting with metadata id e.g 5 not updated since the warehouse will assume the setting was already fetched. We need a way to tell the warehouse these settings were updated. The solution I have in mind doesn't help @Wambere & team much considering it will do something they already do and are trying to avoid. Nway my thinking is we have a separate table (on the ware house) that stores the latest @bennsimon do you think this would work if we moved the same suggestion to OpenSRP instead. Add a |
@dubdabasoduba Why not update the behaviour of the There is a |
It does get incremented. But it increments all the metadata related to the |
Thanks @dubdabasoduba for your clarification. I have ambiguity over here, we still have the same |
@bennsimon Please add the suggestion you explain to me here. I think we can go with that. |
@dubdabasoduba Am thinking we can add a new column say |
Based on the discussion above, the solution will be to add a new column named Does it sound okay? @Wambere @dubdabasoduba @bennsimon |
I think that's okay. Though we really need to test that against the v1 & v2 endpoints to make sure we maintain the original functionality. |
any update on this? |
@Wambere PRs are under review. The coverage part is remaining, hoping to get it merged by Friday. |
@Wambere Implementation is completed and is a part of https://github.com/opensrp/opensrp-server-web/releases/tag/v2.8.6-SNAPSHOT. For your reference:
|
@rehammuzzamil it looks like the new column |
Hi @Wambere , Here is the sample response of Settings V1:
Here is the sample response for settings v2:
You can get find a field named as |
@rehammuzzamil can you please share request url as well? |
http://localhost:8080/opensrp/rest/v2/settings/?limit=2&metadata_version=1 @Wambere
|
Re-opening this issue as the endpoint does not work as expected 1. The response needs to be monotonic. 2. Should all serverVersions be changing? |
Hi @Wambere, As per the initial investigation:
|
@rehammuzzamil I am okay with adding the pagination. Just add it as something configurable since not all instances require the settings order by Could also test that the |
@rehammuzzamil I'm creating and updating by POSTing to
As for ordering, if we could get order by both serverVersion and metadataVersion that would be great! |
I have tested the flow and figured out that whenever a single record is being updated using v1, the
I have observed that in the V1 response, we only get the last updated server_version in the payload, and serverVersion values are not being populated in the individual settings record (i.e. 0 in all the settings).
|
Hi @Wambere, |
@rehammuzzamil the ETL is using v2, and no, we do not get all settings at once. We pull in batches using the version and limit |
As per discussion with @bennsimon we will be exposing a new end-point |
Where is the scoping/issue for this endpoint? |
Let me create a new one. |
@rehammuzzamil please create it as detailed as possible. I don't think get all is used for ETL purposes alone. It's nice to do the base work done on this |
Issue for the new |
The requirement is to be able to
So in order to implement the above, the changes we'd like are
https://<opensrp-server>/rest/v2/settings/?serverVersion=0
endpoint returns all results upto 1000. We would need a way to access the records after the first 1000. Previously this has been done by using thelimit
parameter together with the serverVersions. So for example fetch the first 100 withserverVersion=0&limit=100
and the next 100 withserverVersion=100&limit=100
...etcsettingMetadataId
which seems to be a good candidate for this, can you confirm that it is always incremented when we have new records/edits/deletes? and if yes can we add it as a parameter and use it to pull records that way?The text was updated successfully, but these errors were encountered: