LUI-198: Optimize patient dashboard loading by implementing pagination for observations #209

ODORA0 · 2024-11-13T06:20:31Z

Add pagination for patient observations in dashboard

Description:

Added pagination to optimize patient observations loading in the dashboard
Implemented configurable page size through global properties (default: 50)
Added pagination metadata to support UI implementation

Previously, the dashboard would load all patient observations at once, causing
performance issues for patients with many observations. This change:

Loads observations in configurable page sizes
Allows navigation through URL parameters (?page=0&pageSize=50)
Improves dashboard loading performance

Testing:

Test with patients having large numbers of observations

Ticket: https://openmrs.atlassian.net/browse/LUI-198

…nation for observations

dkayiwa · 2024-11-25T09:19:30Z

Did you get a chance to take a look at this? https://openmrs.atlassian.net/wiki/spaces/docs/pages/25477199/Pull+Request+Tips

ODORA0 · 2024-11-26T13:34:13Z

@dkayiwa please review again

dkayiwa · 2024-11-26T19:39:43Z

Can we also include the ticket id in the commit message?

dkayiwa · 2024-11-26T19:47:07Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

+						Integer page = getPageParameter(request);
+						Integer pageSize = getPageSizeParameter(request, as);
+
+						// Get all observations


For each time that one moves through the pages, you are still fetching and loading all the patient's observations in memory. My impression was that you would load in memory only those observations for the current page.

dkayiwa · 2024-12-02T11:42:13Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

+						boolean includeVoided = false;
+
+						// Get observations for the current page
+						List<Obs> paginatedObs = Context.getObsService().getObservations(persons, encounters, questions,


Though the comment says get observations for the current page, i do not seen anything in the getObservations method call that does so. Was it just a typo?

Ooh yeah, my bad the getObservations() method doesn't have parameters for offset/limit pagination, let me remove that comment

So then, shall we then have any dashboard loading optimisation if the method loads all of them in memory?

Okay, I critically looked at the ObsService interface, didn't notice that it lacks proper pagination parameters (offset/limit). Maybe this should be done as a DAO level implementation so we can have a DB level pagination support, but I would think of doing it this way as well i.e. we can use an existing API features like mostRecentN to limit initial load like so

Suggested change

List<Obs> paginatedObs = Context.getObsService().getObservations(persons, encounters, questions,

List<Obs> paginatedObs = Context.getObsService().getObservations(

persons,

null,

null,

null,

null,

null,

Collections.singletonList("obsDatetime desc"),

pageSize, // mostRecentN - only fetch what we need

null,

null,

null,

false

);

or using date filtering

Suggested change

List<Obs> paginatedObs = Context.getObsService().getObservations(persons, encounters, questions,

Calendar cal = Calendar.getInstance();

cal.add(Calendar.DAY_OF_MONTH, -30);

Date thirtyDaysAgo = cal.getTime();

List<Obs> recentObs = Context.getObsService().getObservations(

persons,

null,

null,

null,

null,

null,

sort,

null,

null,

thirtyDaysAgo,

new Date(),

false

);

Your thoughts @dkayiwa

Limiting the number of loaded obs with mostRecentN makes lots of sense to me. For as long as your use case allows it. That is, as long as you are aware that it behaves differently from page size in the sense that the rest of the obs will never be fetched.

Okay!
@eudson Any thoughts on this, is it acceptable to only show the most recent observations?

eudson · 2024-12-04T06:53:51Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

-						model.put("patientEncounters", Context.getEncounterService().getEncountersByPatient(p));
+						model.put(
+						    "patientEncounters",
+						    Context.getEncounterService().getEncounters(p.getPatientIdentifier().getIdentifier(), 0, 100,


@ODORA0 why are you hardcoding the start(0) and length(100) ? My understanding of these two variables is that they can help paginate the results, the start being the page and the length the number of records per page.

My bad here, I might have forgotten to make the change but changed so that we using the proper pagination parameters using the configured page size in the maximumNumberToShow GP or default to only 50 when empty and allow users to paginate through encounters.

eudson · 2024-12-04T06:57:16Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

-						Integer totalCount = Context.getObsService().getObservationCount(persons, encounters, questions,
-						    answers, personTypes, locations, obsGroupId, fromDate, toDate, includeVoided);
+						List<Obs> paginatedObs = Context.getObsService().getObservations(persons, null, null, null, null,
+						    null, Collections.singletonList("obsDatetime desc"), pageSize,


Can we apply the same concept here?

So I am using the using proper ObsService method signature for observations and using mostRecentN for pagination with the pageSize parameter, using same pageSize from global property or default which is set to 50 thus showing most recent observations up to pageSize limit

…lt to 50

dkayiwa · 2024-12-05T10:04:16Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

+
+	private Integer getPageParameter(HttpServletRequest request) {
+		try {
+			return Integer.parseInt(request.getParameter("page"));


Can you take a look at our naming convention for the pagination parameters? https://openmrs.atlassian.net/wiki/spaces/docs/pages/25469830/REST+Web+Services+API+For+Clients#Limiting-the-number-of-results-and-paging

Basing on the user interface screen that this controller serves, could you remind me on how this parameter is changed to a different value?

So the method retrieves the page parameter from the HTTP request, and if it'ss missing or is invalid like non numeric it will default to 0. The page value calculates which subset of data (encounters or observations) to fetch patientDashboard?page=2

What i mean is, how is this parameter changed in the HTTP request, by the user interface?

With the dropdown selector

Can you share a screenshot of that entire page?

Awesome!!! 👍

Did you check the link about naming conventions?

Yes, I have changed that, please review

dkayiwa · 2024-12-05T10:05:23Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

+				return Integer.parseInt(pageSizeParam);
+			}
+
+			String globalPageSize = as.getGlobalProperty("dashboard.encounters.maximumNumberToShow");


Is this pull request about encounters or observations?

And just in case you find this helpful: https://openmrs.atlassian.net/browse/LUI-188

Yes, we thought we could align encounters as well to limit it to based max numbers to show or default to the fallback property 50. cc @eudson
Encounters fetched are now limited to the value of the GP

The global property that you are using here is specifically for encounters. You can take a look at the ticket link that i shared above to get some brief historical background about what i am driving at. 😊

Okay so that solution effectively decouples the configuration for the Form Entry tab and the Encounters tab which tackles both usability and configurability needs without disrupting existing any setups. Are you trying to say we employ the same thing but for the patientEncounters jsp, LOL, I am having a brain fart right now with looking at old tech am not as familiar with

One should be able to set different values for encounters and observations.

This involves creating a different GP for Obs as in maximumNumberObservationsToShow? Please advise

Yes you need to create a separate global property for it.

dkayiwa · 2024-12-06T12:22:18Z

There is lots of formatting changes which makes it hard to review the actual changes. Do you mind doing something about it?

ODORA0 · 2024-12-06T13:05:07Z

@dkayiwa Most of the changes are formatting is in the config xml that happens when running mvn clean. The change there only involves adding the extra GP for obs

dkayiwa · 2024-12-06T13:21:41Z

Can you just avoid running it?

ODORA0 · 2024-12-06T13:46:03Z

@dkayiwa Done

dkayiwa · 2024-12-06T13:57:01Z

omod/src/main/resources/config.xml

@@ -382,6 +382,14 @@
 			Allows one to limit the number of encounters shown on the form entry tab of the patient dashboard specifically
 		</description>
 	</globalProperty>
+		<globalProperty>
+		<property>dashboard.observations.maximumNumberToShow</property>
+		<defaultValue>false</defaultValue>


The above default value does not look valid.

dkayiwa · 2024-12-06T14:02:48Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

+						List<Person> persons = Collections.singletonList(person);
+
+						// Get most recent observations using limit parameter
+						List<Obs> paginatedObs = Context.getObsService().getObservations(persons, null, null, null, null,


According to the user interface you shared, what do you do with the startIndex?

Select the page size

Is the page size the same as startIndex?

Yes, based on the link you shared I renamed pageSize to that

Where are you passing the two parameters in the getObservations api call?

We are only passing the limit and not startIndex directly in that method call

Which means that if you are on the last page, you still load all obs from the database to memory.

@dkayiwa Yes that's right, added that

dkayiwa · 2024-12-08T09:47:20Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

+						List<Obs> paginatedObs = Context.getObsService().getObservations(persons, null, null, null, null, null, 
+						Collections.singletonList("obsDatetime desc"), limit, startIndex, null, null, false);					
+
+						model.put("limit", limit);


Can you remind me of why we are storing limit in the model?

We storing limit in the model to make it accessible to/for the JSP and also keeping the pagination logic synchronised between the backend and JSP frontend

Can you point me to the JSP that is currently using this limit that you have put in the model?

The patientEncounters jsp

Do you mind pointing me to the actual line that uses limit?

Looking at the dashboard JSP file, no references to a limit or pagination for observations, safe deleting it.

dkayiwa · 2024-12-08T10:13:29Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

+
+						model.put(
+						    "patientEncounters",
+						    Context.getEncounterService().getEncounters(p.getPatientIdentifier().getIdentifier(),


Why not use the more efficient? https://github.com/openmrs/openmrs-core/blob/2.0.0/api/src/main/java/org/openmrs/api/EncounterService.java#L432

dkayiwa · 2024-12-08T10:17:31Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

@@ -425,4 +465,30 @@ public ModelAndView handleRequest(HttpServletRequest request, HttpServletRespons
 	protected void populateModel(HttpServletRequest request, Map<String, Object> model) {
 	}

+	private Integer getStartIndexParameter(HttpServletRequest request) {
+		try {
+			return Integer.parseInt(request.getParameter("startIndex"));


Can we also do the StringUtils.isNotBlank check for cases where startIndex may not be supplied?

Spring's StringUtils.hasText instead of isNotBlank that's from the javadoc as using isNotBlank threw errors

dkayiwa · 2024-12-08T10:18:22Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

+	private Integer getLimitParameter(HttpServletRequest request, AdministrationService as, String globalPropertyKey) {
+		try {
+			String limitParam = request.getParameter("limit");
+			if (limitParam != null) {


Can we instead use StringUtils.isNotBlank such that we also filter out non null values but with empty spaces?

…er signature

dkayiwa · 2024-12-08T18:30:28Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

 							String heightString = as.getGlobalProperty("concept.height");
 							ConceptNumeric heightConcept = null;
 							if (StringUtils.hasLength(heightString)) {
 								heightConcept = cs.getConceptNumeric(GeneralUtils.getConcept(heightString).getConceptId());
 							}
-							for (Obs obs : patientObs) {


Did you intentionally remove this for loop? Is it related to the pagination?

Rather than loading all observations and finding the latest in memory, we would leverage the database to do this work more efficiently. Not directly related to pagination but aligns with the optimization goals since we no longer need to iterate over all patientObs to extract weight and height.

With performance optimisations, unless you have tested and confirmed, you can be very shocked to find a different outcome. The database itself being an expensive resource, and you have anyway already fetched these observations in the above call, you can find that actually iterating through the collection in memory may be faster than making another expensive database call. So i would remove that loop only after i have done some confirmations.

The queries for weight and height are correctly implemented, unless there are other dependencies on the for loop logic that exist that are not unaccounted for I think I will put this back to be on a safer side.

@dkayiwa The paginated list only contains a limited number of observations (default 50), therefore BMI calculation couldn't find the necessary values, resulting in "?" if at all we add back the for loop while paginating. Now for this to work I would have to do something like this

// Paginated observations for display List<Obs> paginatedObs = Context.getObsService().getObservations(..., limit, startIndex, ...); // Additional query to get ALL observations for BMI calculation List<Obs> allObs = Context.getObsService().getObservations(persons, null, null, null, null, null, null, null, null, null, null, false); // Finding weight/height in complete set of observations for (Obs obs : allObs) { // ... looking for weight/height }

In essence, we'd now be using:

Paginated query for display purposes (performance optimization)

Complete query for BMI calculation (functionality requirement)

Does this sound an approach to use?

What do you see as the best approach?

I think I will go with this option I have suggested, in the long run we have separation of concerns. Change made

dkayiwa · 2024-12-08T20:29:26Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

 							if (latestWeight != null && latestHeight != null) {
+								model.put("patientWeight", latestWeight);


What happens to the model when latestHeight is null but latestWeight is not null?

No BMI in the model, since both latestWeight and latestHeight are required for BMI calculation, the model will not have patientBmi or patientBmiAsString if either is null. he model will only include the observation that is not null (patientWeight in this case). But since I will revert this, I think any changes like setting a default value or log warning wouldn't be necessary

dkayiwa · 2024-12-09T11:42:47Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

+						    null, Collections.singletonList("obsDatetime desc"), limit, startIndex, null, null, false);
+
+						// Get all observations for BMI calculation
+						List<Obs> allObs = Context.getObsService().getObservations(persons, null, null, null, null, null,


If you are loading all the patient's observations from the database into memory, then what optimisation is pagination doing?

Enhanced so that:

Retrieves the latest observation for weight and height for the patient using the same observation retrieval method (getObservations).

Sorting by obsDatetime desc ensures that only the most recent data is considered.
This has been tested and actually is optimised, loads faster for a patient with > 61k observation_count

dkayiwa · 2024-12-13T10:26:56Z

omod/src/main/java/org/openmrs/web/controller/PortletController.java

 								}
 							}
-							if (latestWeight != null) {


According to the original code, latestHeight or latestWeight was put in the model as long as it was not null. Did you intentionally change it?

Did you see the above comment?

ODORA0 added 2 commits November 13, 2024 09:18

(Enhancement) Optimize patient dashboard loading by implementing pagi…

b03e0de

…nation for observations

Refactor to focus on pagination

a06309a

ODORA0 marked this pull request as ready for review November 21, 2024 10:11

ODORA0 requested review from dkayiwa and ebambo November 25, 2024 04:57

ODORA0 changed the title ~~(Enhancement) Optimize patient dashboard loading by implementing pagination for observations~~ LUI-198: Optimize patient dashboard loading by implementing pagination for observations Nov 26, 2024

dkayiwa reviewed Nov 26, 2024

View reviewed changes

DB-level pagination to only load in memory obs for current page

ba7a0fc

ODORA0 requested a review from dkayiwa December 2, 2024 05:34

dkayiwa reviewed Dec 2, 2024

View reviewed changes

Limit loaded obs to MostRecentN

fa01899

eudson requested changes Dec 4, 2024

View reviewed changes

Remove hard coding and keep values dynamic based on GP set with defau…

e1c2f60

…lt to 50

ODORA0 requested a review from dkayiwa December 5, 2024 06:36

dkayiwa reviewed Dec 5, 2024

View reviewed changes

ODORA0 added 2 commits December 5, 2024 15:14

Change naming conventions

9456377

Create seperate GP for observations

98301a5

ODORA0 requested a review from dkayiwa December 6, 2024 11:28

Remove extra formatting

639a616

Remove extra formatting

3426cb9

dkayiwa reviewed Dec 6, 2024

View reviewed changes

Remove false as default value

1bf5bea

dkayiwa reviewed Dec 6, 2024

View reviewed changes

Add StartIndex to getObservation api

6bcaeda

dkayiwa reviewed Dec 8, 2024

View reviewed changes

Add proper null checks using StringUtils.hasText(), update getEncount…

d7e5866

…er signature

ODORA0 requested a review from dkayiwa December 8, 2024 15:04

dkayiwa reviewed Dec 8, 2024

View reviewed changes

Keep obs set for BMI calculations, remove unused limit param from model

a6b840c

ODORA0 requested a review from dkayiwa December 9, 2024 11:36

dkayiwa reviewed Dec 9, 2024

View reviewed changes

ODORA0 added 2 commits December 13, 2024 08:09

Enhance obs to load latest weight and height

b763c92

Remove commented out code

568c1f9

ODORA0 requested a review from dkayiwa December 13, 2024 05:17

dkayiwa reviewed Dec 13, 2024

View reviewed changes

Add model for latestHeight and weight

9bf378f

ODORA0 requested a review from dkayiwa December 13, 2024 11:03

-						List<Obs> paginatedObs = Context.getObsService().getObservations(persons, encounters, questions,
+Calendar cal = Calendar.getInstance();
+cal.add(Calendar.DAY_OF_MONTH, -30);
+Date thirtyDaysAgo = cal.getTime();
+List<Obs> recentObs = Context.getObsService().getObservations(
+    persons,
+    null,
+    null,
+    null,
+    null,
+    null,
+    sort,
+    null,
+    null,
+    thirtyDaysAgo,
+    new Date(),
+    false
+);

		if (latestWeight != null && latestHeight != null) {
		model.put("patientWeight", latestWeight);

LUI-198: Optimize patient dashboard loading by implementing pagination for observations #209

Are you sure you want to change the base?

LUI-198: Optimize patient dashboard loading by implementing pagination for observations #209

Conversation

ODORA0 commented Nov 13, 2024 • edited Loading

dkayiwa commented Nov 25, 2024

ODORA0 commented Nov 26, 2024

dkayiwa commented Nov 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkayiwa Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkayiwa commented Dec 6, 2024

ODORA0 commented Dec 6, 2024

dkayiwa commented Dec 6, 2024

ODORA0 commented Dec 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ODORA0 Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkayiwa Dec 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkayiwa Dec 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ODORA0 Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ODORA0 commented Nov 13, 2024 •

edited

Loading

dkayiwa Dec 5, 2024 •

edited

Loading

ODORA0 Dec 9, 2024 •

edited

Loading

dkayiwa Dec 8, 2024 •

edited

Loading

dkayiwa Dec 8, 2024 •

edited

Loading

ODORA0 Dec 9, 2024 •

edited

Loading