RANGER-4225: Possible Jackson serialization issue due to not comply with Java bean standards #252

sercanCyberVision · 2023-05-05T01:41:48Z

What changes were proposed in this pull request?

@JsonProperty annotation has been added to model classes for mapping the properties with their corresponding getter/setter methods. This will not effect Ranger's functionality directly, but it will provide consistency in case there is Jackson jar conflict (or when/if Jackson is upgraded to version-2).
Please see the Jira for more detailed analysis of the issue https://issues.apache.org/jira/browse/RANGER-4225

Basically, some of the model classes do not comply with JavaBean naming conventions, therefore it is possible that Ranger may hit this issue https://stackoverflow.com/questions/30205006/why-does-jackson-2-not-recognize-the-first-capital-letter-if-the-leading-camel-c

As a reference of the issue, please see http://futuretask.blogspot.com/2005/01/java-tip-6-dont-capitalize-first-two.html

How was this patch tested?

Built the project and made sure that deserialized responses on UI side have correct property names even though Jackson-2 jars are in the classpath, hence the corresponding UI components work as expected.

sercanCyberVision · 2023-05-05T12:11:35Z

@kumaab could you please take a look at this?

sercanCyberVision · 2023-08-09T18:08:10Z

@mneethiraj could you please take a look at this? I hit this issue earlier, and the solution here can save some grief for the end-users for the future.

bhavikpatel9977 · 2024-06-14T06:13:52Z

LGTM.

Kindly resolve the conflict.

sercanCyberVision · 2024-06-14T14:56:24Z

@bhavikpatel9977, thank you for reviewing the PR. I have resolved the conflict and quickly tested it again.
So, two actions have been taken here:

org.codehaus.jackson has been replaced with com.fasterxml.jackson.
@JsonProperty has been added to view classes to help jackson recognize java bean properties.

bhavikpatel9977 · 2024-06-17T04:52:58Z

@mneethiraj Can you kindly review once

mneethiraj

@sercanCyberVision - thank you for taking up this critical update to switch to fasterxml.jackson. In addition to the couple of review comments, can you please verify that the final packaging does not include codehaus jackson library? It seems Ranger admin continues to use codehaus jackson library to serialize REST API responses. For example, the response includes fields having null and empty value.

mneethiraj · 2024-06-22T23:55:26Z

pom.xml

@@ -91,7 +91,7 @@
        <cglib.version>2.2.0-b23</cglib.version>
        <checkstyle.plugin.version>3.1.0</checkstyle.plugin.version>
        <checkstyle.version>8.29</checkstyle.version>
-        <codehaus.jackson.version>1.9.13</codehaus.jackson.version>
+        <com.fasterxml.jackson.version>2.17.0</com.fasterxml.jackson.version> <!--here-->


Consider using existing property fasterxml.jackson.version (line # 215) instead of introducing new one com.fasterxml.jackson.version. Also, update the version in #215 and #216 from 2.14.0 to 2.17.0.

Thank you @mneethiraj for your review.

I have implemented above requested change and pushed it as a separate commit for now as it is easier for you to see the difference, the commit is cccbabf.

If needed, when we reach the final state, I can squash the commits or change the names.

mneethiraj · 2024-06-23T00:01:31Z

agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java

+import com.fasterxml.jackson.annotation.JsonAutoDetect;
+import com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;


 @JsonAutoDetect(getterVisibility = Visibility.NONE, setterVisibility = Visibility.NONE, fieldVisibility = Visibility.ANY)


With the move to fasterxml.jackson 2.x, consider replacing deprecated include = JsonSerialize.Inclusion.NON_EMPTY, with @JsonInclude(JsonInclude.Include.NON_EMPTY)

As stated above, the changes have been implemented with cccbabf commit.

sercanCyberVision · 2024-06-25T21:13:12Z

@sercanCyberVision - thank you for taking up this critical update to switch to fasterxml.jackson. In addition to the couple of review comments, can you please verify that the final packaging does not include codehaus jackson library? It seems Ranger admin continues to use codehaus jackson library to serialize REST API responses. For example, the response includes fields having null and empty value.

Thank you @mneethiraj for detailed review. You are right; codehaus.jackson was present in the admin library. I have checked the dependency tree and found that they were transitive. I have fixed the issue with this separate commit: 7eb06a5. If needed, we can squash it once the review is completed. I kept it separate so you can see the changes easily. The latest state is as follows:

ranger@ranger:/opt/ranger/ranger-3.0.0-SNAPSHOT-admin$ find . -name "*jackson*"
./ews/lib/jackson-core-2.17.0.jar
./ews/webapp/WEB-INF/lib/jackson-dataformat-smile-2.10.4.jar
./ews/webapp/WEB-INF/lib/jackson-dataformat-cbor-2.10.4.jar
./ews/webapp/WEB-INF/lib/jackson-databind-2.17.0.jar
./ews/webapp/WEB-INF/lib/jackson-core-2.17.0.jar
./ews/webapp/WEB-INF/lib/jackson-module-jaxb-annotations-2.17.0.jar
./ews/webapp/WEB-INF/lib/jackson-dataformat-yaml-2.10.4.jar
./ews/webapp/WEB-INF/lib/jackson-annotations-2.17.0.jar
./ews/webapp/WEB-INF/classes/ranger-plugins/knox/jackson-databind-2.17.0.jar
./ews/webapp/WEB-INF/classes/ranger-plugins/knox/jackson-core-2.17.0.jar
ranger@ranger:/opt/ranger/ranger-3.0.0-SNAPSHOT-admin$

I have one concern: when I deploy the project locally with Docker, I see the following issue on some endpoints after removing transitive dependencies:

However, I'm not sure if this is a real issue because when I saw the issue previously, others did not see it with the same codebase. For example, when I mentioned the issue here, in the next comment, the same codebase worked on another machine. Could you please check on your end?

mneethiraj · 2024-07-11T07:27:08Z

@sercanCyberVision - the error is likely due to missing Json provider in Ranger admin server. A provider should be implemented in Ranger servers like admin/KMS. In addition, Ranger client libraries should be updated to replace calls to ClientResponse.getEntity(cls) with JsonUtilsV2.jsonToObj(jsonStr, cls). I added these fixes on top of the changes in this PR, and created a .patch file that applies on the latest master branch. I will attach this file in RANGER-4225 shortly. Can you please review/verify these updates and create a new PR? Thank you again for driving this critical update.

…ith Java bean standards

sercanCyberVision · 2024-07-11T15:32:20Z

@mneethiraj, thank you for providing the fix. I have modified this PR for the complete solution.

At the moment, I see that everything is working fine on my end. However, I still see some ClientResponse.getEntity(cls):
https://github.com/sercanCyberVision/ranger/blob/f84dd021c4c77666e1995535449f10b272ccad3b/intg/src/main/java/org/apache/ranger/RangerClient.java#L567
or
https://github.com/sercanCyberVision/ranger/blob/71809108fd106b664b6f9d53e0efd86d4c5cd039/plugin-yarn/src/main/java/org/apache/ranger/services/yarn/client/YarnClient.java#L148
or
https://github.com/sercanCyberVision/ranger/blob/f84dd021c4c77666e1995535449f10b272ccad3b/agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java#L736
There are some more. Are they left as they are on purpose? Or, should we replace them with JsonUtilsV2.jsonToObj(jsonStr, cls)

Checks on UI:

mneethiraj · 2024-07-12T01:25:57Z

@sercanCyberVision - good catch on use of readEntity() in RangerClient and RangerAdminRESTClient. I updated these classes to deserialize the response using JsonUtilV2 methods. YarnClient uses readEntity() to read the response as String; such usage doesn't require using Jackson Json conversion.

One more issue I noticed was that the patch replaced NOT_NULL to NOT_EMPTY in several classes; this caused errors in UI as the response from server didn't include empty collection values - [] or {}.

I updated the patch to address above issues and will attach in RANGER-4225.

mneethiraj · 2024-07-12T15:10:46Z

The final patch is in the review board at https://reviews.apache.org/r/75082/. The patch is merged in master and ranger-2.5 branches.

sercanCyberVision changed the title ~~RANGER-4225: Possible Jackson serialization due to not complain to Java bean standards~~ RANGER-4225: Possible Jackson serialization issue due to not comply to Java bean standards May 5, 2023

sercanCyberVision changed the title ~~RANGER-4225: Possible Jackson serialization issue due to not comply to Java bean standards~~ RANGER-4225: Possible Jackson serialization issue due to not comply with Java bean standards May 5, 2023

sercanCyberVision force-pushed the RANGER-4225 branch 2 times, most recently from 602a07d to 77fac4d Compare May 5, 2023 12:08

kumaab requested review from mneethiraj, kulkabhay, rameeshm and pradeepagrawal8184 May 5, 2023 19:27

sercanCyberVision mentioned this pull request May 3, 2024

RANGER-3409: remove org.codehaus.jackson ver 1.9 / add com.fasterxml.jackson ver 2.17 #312

Closed

sercanCyberVision force-pushed the RANGER-4225 branch 3 times, most recently from 05c4764 to e68a9a1 Compare May 21, 2024 14:24

sercanCyberVision force-pushed the RANGER-4225 branch from e68a9a1 to 0da66da Compare May 29, 2024 13:17

sercanCyberVision force-pushed the RANGER-4225 branch from 0da66da to 54239ad Compare June 14, 2024 14:49

mneethiraj requested changes Jun 23, 2024

View reviewed changes

sercanCyberVision force-pushed the RANGER-4225 branch from 54239ad to 7eb06a5 Compare June 25, 2024 20:56

sercanCyberVision requested a review from mneethiraj June 25, 2024 21:13

RANGER-4225: Possible Jackson serialization issue due to not comply w…

f84dd02

…ith Java bean standards

sercanCyberVision force-pushed the RANGER-4225 branch from 7eb06a5 to f84dd02 Compare July 11, 2024 15:27

mneethiraj approved these changes Jul 12, 2024

View reviewed changes

mneethiraj closed this Jul 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RANGER-4225: Possible Jackson serialization issue due to not comply with Java bean standards #252

RANGER-4225: Possible Jackson serialization issue due to not comply with Java bean standards #252

sercanCyberVision commented May 5, 2023 •

edited

Loading

sercanCyberVision commented May 5, 2023

sercanCyberVision commented Aug 9, 2023

bhavikpatel9977 commented Jun 14, 2024

sercanCyberVision commented Jun 14, 2024

bhavikpatel9977 commented Jun 17, 2024

mneethiraj left a comment

mneethiraj Jun 22, 2024

sercanCyberVision Jun 25, 2024

mneethiraj Jun 23, 2024

sercanCyberVision Jun 25, 2024

sercanCyberVision commented Jun 25, 2024

mneethiraj commented Jul 11, 2024

sercanCyberVision commented Jul 11, 2024 •

edited

Loading

mneethiraj commented Jul 12, 2024

mneethiraj commented Jul 12, 2024

RANGER-4225: Possible Jackson serialization issue due to not comply with Java bean standards #252

RANGER-4225: Possible Jackson serialization issue due to not comply with Java bean standards #252

Conversation

sercanCyberVision commented May 5, 2023 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

sercanCyberVision commented May 5, 2023

sercanCyberVision commented Aug 9, 2023

bhavikpatel9977 commented Jun 14, 2024

sercanCyberVision commented Jun 14, 2024

bhavikpatel9977 commented Jun 17, 2024

mneethiraj left a comment

Choose a reason for hiding this comment

mneethiraj Jun 22, 2024

Choose a reason for hiding this comment

sercanCyberVision Jun 25, 2024

Choose a reason for hiding this comment

mneethiraj Jun 23, 2024

Choose a reason for hiding this comment

sercanCyberVision Jun 25, 2024

Choose a reason for hiding this comment

sercanCyberVision commented Jun 25, 2024

mneethiraj commented Jul 11, 2024

sercanCyberVision commented Jul 11, 2024 • edited Loading

mneethiraj commented Jul 12, 2024

mneethiraj commented Jul 12, 2024

sercanCyberVision commented May 5, 2023 •

edited

Loading

sercanCyberVision commented Jul 11, 2024 •

edited

Loading