[WFCORE-6935] Improve systemd service units #6179

yersan · 2024-09-18T10:37:19Z

Jira issue: https://issues.redhat.com/browse/WFCORE-6935

jamezp

Overall I think this looks good. Just a minor suggestion and a couple questions.

core-feature-pack/common/src/main/resources/content/bin/systemd/README

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-domain.service

yersan

Thanks for the review @jamezp , added some comments related

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-domain.service

core-feature-pack/common/src/main/resources/content/bin/systemd/README

jamezp

This looks good to me. We can figure out the version thing later and really maybe it doesn't make sense to keep it up to date. After all that is the full version and this is core.

core-feature-pack/common/src/main/resources/content/bin/systemd/README

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-domain.service

rpelisse · 2024-10-15T12:11:52Z

core-feature-pack/common/src/main/resources/content/bin/systemd/README

+
+== Start and enable WildFly
+
+    # systemctl start wildfly-$MODE.service


To respect Bash's best practises:
# systemctl start "wildfly-${MODE}.service"

rpelisse · 2024-10-15T12:12:41Z

core-feature-pack/common/src/main/resources/content/bin/systemd/README

+
+== Configure systemd
+
+    # cd /opt/wildfly/bin/systemd


To avoid hard coding /opt/wildfly everywhere, I would suggest defining a WILDFLY_HOME=/opt/wildfly at the beginning.

rpelisse · 2024-10-15T12:13:40Z

core-feature-pack/common/src/main/resources/content/bin/systemd/README

+
+    # cd /opt/wildfly/bin/systemd
+    # cp wildfly-$MODE.conf /etc/sysconfig/
+    # cp wildfly-$MODE.service $(pkg-config systemd --variable=systemdsystemunitdir)


To follow bash's best practices:

# cp "wildfly-${MODE}.conf" /etc/sysconfig/ # cp "wildfly-${MODE}.service" $(pkg-config systemd --variable=systemdsystemunitdir)

rpelisse · 2024-10-15T12:14:23Z

core-feature-pack/common/src/main/resources/content/bin/systemd/README

+
+== Remove WildFly systemd service
+
+    # rm -f $(pkg-config systemd --variable=systemdsystemunitdir)/wildfly-$MODE.service


rm -f "$(pkg-config systemd --variable=systemdsystemunitdir)/wildfly-${MODE}.service"

core-feature-pack/common/src/main/resources/content/bin/systemd/README

rpelisse · 2024-10-15T12:15:42Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+JBOSS_HOME="$(realpath "$SCRIPT_DIR/../..")"
+JBOSS_SYSTEMD_MODE="${1}"
+
+if [ -z "$JBOSS_SYSTEMD_MODE" ]; then


if [ -z "${JBOSS_SYSTEMD_MODE}" ]; then

for consistency on top of best practice

rpelisse · 2024-10-15T12:16:09Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+  exit 0;
+fi
+
+SYSTEMD_FILE="$SCRIPT_DIR/wildfly-${JBOSS_SYSTEMD_MODE}.service"


SYSTEMD_FILE="${SCRIPT_DIR}/wildfly-${JBOSS_SYSTEMD_MODE}.service"

rpelisse · 2024-10-15T12:16:53Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+
+SYSTEMD_FILE="$SCRIPT_DIR/wildfly-${JBOSS_SYSTEMD_MODE}.service"
+JBOSS_SYSTEMD_USER="${2:-wildfly}"
+JBOSS_SYSTEMD_GROUP="${3:-wildfly}"


My preference is to add single quote around literals to make it clear to the reader that it's supposed to be fixed characters strings:
JBOSS_SYSTEMD_GROUP="${3:-'wildfly'}"

That would move the quotes to the Systemd unit file as well, is that still valid?

rpelisse · 2024-10-15T12:18:27Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+JBOSS_SYSTEMD_GROUP="${3:-wildfly}"
+
+echo "INFO: Systemd Unit File to generate: ${SYSTEMD_FILE}"
+echo "INFO: Using JBOSS_HOME: $JBOSS_HOME"


As we have JBOSS_HOME here, we should use it at the beginning, to not repeat /opt/wildfly all the time.

Also:
echo "INFO: Using JBOSS_HOME: ${JBOSS_HOME}"

As we have JBOSS_HOME here, we should use it at the beginning, to not repeat /opt/wildfly all the time.

JBOSS_HOME is initialized at the beginning of the script and there is no "/opt/wildfly" path hardcoded in this script, so I don't follow you here.

Also:
echo "INFO: Using JBOSS_HOME: ${JBOSS_HOME}"

Done

rpelisse · 2024-10-15T12:19:38Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+echo "INFO: User: ${JBOSS_SYSTEMD_USER}"
+echo "INFO: Group: ${JBOSS_SYSTEMD_GROUP}"
+
+while true; do


Interactive shell is a bad idea. I would suggest using opts so that the user can pass a parameters if he wants to generate the systemd file.
We should also aimed at making this script idempotent. (I'll do another review in this regard, after this one)

Interactive shell is a bad idea.

It is not expected this file be invoked by a third-party tool, this is just a utility that is expected to be used by a user to make his life easier when he needs to replace the JBOSS_HOME, user:group in the systemd unit files. Asking interactively for confirmation here allows the user to review what he is going to configure, so I think it is useful here.

Would that convince you to leave it as it is?

No :) I really think it's a bad idea.

Bear in mind that this will be used by Ops (or DevOps), they won't expect an interactive shell and will be quite annoyed with it. Also, they are most likely going to automate around this (I've done so in the past, at customers). They may want to automate the deployment of Wildfly based and this and have script (or a tool like Ansible) automatically generate the file on the target system....

And right now, it's impossible to because the script is interactive by default. So if you really want to keep it, at least you should provide a mecanism around it.

IMHO, the best approach, that would match what people do in this kind of script, would be to test if the var is defined and act according to it:

if [ -z ${SYSTEMD_FILE} ]; then echo "No systemd file provided, not generating one." exit 0 fi

(last point if somehow you really, really want to keep this interaction thing, I would move into a function, because it's big chunk of code)

@rpelisse Ok, you bring more experience to this from Ansible automation, so I'll remove the interactive question or add a confirm flag for skipping it

@rpelisse I've added a new argument c,--confirm to avoid the interactive mode.

rpelisse · 2024-10-15T12:20:16Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+  esac
+done
+
+sed -e "s|@@@JBOSS_SYSTEMD_SERVER_HOME@@@|$JBOSS_HOME|g" \


sed -e "s|@@@JBOSS_SYSTEMD_SERVER_HOME@@@|${JBOSS_HOME}|g" \ -e "s|@@@JBOSS_SYSTEMD_USER@@@|${JBOSS_SYSTEMD_USER}|g" \ -e "s|@@@JBOSS_SYSTEMD_GROUP@@@|${JBOSS_SYSTEMD_GROUP}|g" \ "${SYSTEMD_FILE}.template" > "${SYSTEMD_FILE}"

rpelisse · 2024-10-15T12:20:30Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+    -e "s|@@@JBOSS_SYSTEMD_GROUP@@@|$JBOSS_SYSTEMD_GROUP|g" \
+    "${SYSTEMD_FILE}.template" > "$SYSTEMD_FILE"
+
+systemd-analyze verify --recursive-errors=no "$SYSTEMD_FILE"


systemd-analyze verify --recursive-errors=no "${SYSTEMD_FILE}"

rpelisse · 2024-10-15T12:20:48Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+echo ""
+echo "INFO: Systemd Unit File generated."
+echo "INFO: The ${JBOSS_SYSTEMD_USER}:${JBOSS_SYSTEMD_GROUP} are the user:group configured to launch the server. You have to ensure this user and group exist and have the necessary permissions to read and launch the server."
+echo "INFO: Use $JBOSS_HOME/bin/systemd/wildfly-${JBOSS_SYSTEMD_MODE}.conf to configure the server environment."


echo "INFO: Use ${JBOSS_HOME}/bin/systemd/wildfly-${JBOSS_SYSTEMD_MODE}.conf to configure the server environment."

rpelisse · 2024-10-15T12:22:04Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+echo "sudo systemctl start $(basename $SYSTEMD_FILE)"
+echo ""
+echo "INFO: In case of issues, you can check the service logs with:"
+echo "sudo journalctl -u $(basename $SYSTEMD_FILE)"


Be mindful of where to use quote:
echo sudo journalctl -u $(basename "${SYSTEMD_FILE}")
This approach will yield a more comprehensible error message if, somehow, a space has been inserted in the variable.

core-feature-pack/common/src/main/resources/content/bin/systemd/launch.sh

yersan

Thanks for the review @rpelisse , I fixed most of the suggestion and added some minor comments

yersan · 2024-10-24T15:30:30Z

Hello @rpelisse could you review this again?

rpelisse

@yersan I'm super sorry to have not reviewed this PR (again) more quickly... I did find a couple of areas I think we should look into before we merge.

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-standalone.service

...ature-pack/common/src/main/resources/content/bin/systemd/wildfly-standalone.service.template

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-domain.service.template

rpelisse · 2024-11-05T07:47:49Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+  exit 1
+}
+
+SCRIPT_NAME=$(basename "$0")


SCRIPT_NAME=$(basename "$0") to SCRIPT_NAME=$(basename "${0}")

rpelisse · 2024-11-05T07:48:21Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+}
+
+SCRIPT_NAME=$(basename "$0")
+SCRIPT_DIR="$(dirname "$(realpath "$0")")"


SCRIPT_DIR="$(dirname "$(realpath "$0")")" to SCRIPT_DIR="$(dirname "$(realpath "${0}")")"

rpelisse · 2024-11-05T07:52:51Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+    esac
+done
+
+if [ "${JBOSS_SYSTEMD_MODE}" != "standalone" ] && [ "${JBOSS_SYSTEMD_MODE}" != "domain" ]; then


Getting into super nickpicking zone :)

Technically by using "" you are telling the script interpreter to look for cmd substitution or variable to be replaced by their content into the string. So it would be better to have 'standalone' and 'domain' instead of "standalone" and "domain".

I did say it was nickpicking, right? ;)

right :) ... but it is a trivial change and your argument is valid, so done

rpelisse · 2024-11-05T07:55:11Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+echo "INFO: Using JBOSS_HOME: ${JBOSS_HOME}"
+echo "INFO: User: ${JBOSS_SYSTEMD_USER}"
+echo "INFO: Group: ${JBOSS_SYSTEMD_GROUP}"
+echo "INFO: Confirm: ${JBOSS_CONFIRM}"


I would not echo this variable, as it will only be used for interactive mode. Having this value in logfile of automated install will be ... strange, and might confused ops ("Wait, why my JBOSS is not confirmed? What does it means???".
If you intend to echo the value for debugging purpose, remember that simply running the script with the -x option will echo all the variable values.

It makes sense to me, I removed it.

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

rpelisse · 2024-11-05T07:59:26Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+if [ 'y' != "${JBOSS_CONFIRM}" ] && [ 'Y' != "${JBOSS_CONFIRM}" ];then
+  while true; do
+    read -p "Do you want to generate ${SYSTEMD_FILE}? (y/n): " choice
+    case "$choice" in


case "$choice" in to case "${choice}" in

Also, as I suspect choice will not change value after the script starts, I would suggest to go uppercase to cleary display that (so rename it to CHOICE).

Ideally, I would prefix any vars declaration by readonly, but I'm not sure if this supported across all interpreter or just a bash feature. It's a good practise to use readonly, because it prevent some already defined variable in the shell to override a value in the script.

case "$choice" in to case "${choice}" in

Done

Also, as I suspect choice will not change value after the script starts

choice will get a new value on each loop inside the while if the user does not write any expected answer. So I leave it untoched

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

yersan

@rpelisse I'm done with this, if there is anything else critital, we would like to merge this as it is now, please take another look, thank you!

yersan · 2024-11-08T11:44:07Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+  exit 1
+}
+
+SCRIPT_NAME=$(basename "$0")


yersan · 2024-11-08T11:44:14Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+}
+
+SCRIPT_NAME=$(basename "$0")
+SCRIPT_DIR="$(dirname "$(realpath "$0")")"


yersan · 2024-11-08T12:00:59Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+echo "INFO: Using JBOSS_HOME: ${JBOSS_HOME}"
+echo "INFO: User: ${JBOSS_SYSTEMD_USER}"
+echo "INFO: Group: ${JBOSS_SYSTEMD_GROUP}"
+echo "INFO: Confirm: ${JBOSS_CONFIRM}"


It makes sense to me, I removed it.

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

...ature-pack/common/src/main/resources/content/bin/systemd/wildfly-standalone.service.template

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-standalone.service

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-domain.service.template

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

yersan · 2024-11-08T12:24:36Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+    esac
+done
+
+if [ "${JBOSS_SYSTEMD_MODE}" != "standalone" ] && [ "${JBOSS_SYSTEMD_MODE}" != "domain" ]; then


right :) ... but it is a trivial change and your argument is valid, so done

yersan · 2024-11-08T12:31:04Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+if [ 'y' != "${JBOSS_CONFIRM}" ] && [ 'Y' != "${JBOSS_CONFIRM}" ];then
+  while true; do
+    read -p "Do you want to generate ${SYSTEMD_FILE}? (y/n): " choice
+    case "$choice" in


case "$choice" in to case "${choice}" in

Done

Also, as I suspect choice will not change value after the script starts

choice will get a new value on each loop inside the while if the user does not write any expected answer. So I leave it untoched

rpelisse

I think I spot a potential bug, that's why I'm putting this in "Request Changes". I can run shellcheck right now, but I suspect if you do, it will confirm the issue I found. Apart from that (and a couple missing "${xxx}" around variables), I think we're good!

rpelisse · 2024-11-16T16:17:33Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+}
+
+SCRIPT_NAME=$(basename "${0}")
+SCRIPT_DIR="$(dirname "$(realpath "${0}")")"


I think this might be broken:

"$(dirname "$(realpath "${0}")")"

I count 2 opening " and 3 closing one. Also, with command substitution double quotes can be omitted (by spec the $() does the same job as ""). So this should be enough, as the most important "" are around the variable replacement, to ensure the result is treated as one arguments (so a whitespace in the file path does NOT break the script):

$(dirname $(realpath "${0}"))

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

core-feature-pack/common/src/main/resources/content/bin/systemd/README

rpelisse · 2024-11-16T16:19:15Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+JBOSS_SYSTEMD_GROUP="wildfly"
+JBOSS_CONFIRM="n"
+
+while [ "$#" -gt 0 ]


I did not saw it on the first pass, but all the variables in the args treatement are "$XXX" instead of "${XXX}". For consistency purpose and respecting best practices, it might be better to fix that.

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

rpelisse · 2024-11-16T16:34:29Z

core-feature-pack/common/src/main/resources/content/bin/systemd/launch.sh

@@ -0,0 +1,26 @@
+#!/bin/sh
+if [ "$WILDFLY_SYSTEMD_DEBUG" == "true" ]; then


Another missing "${" that I failed to spot the first time :(

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-domain.service.template

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-standalone.service

...ature-pack/common/src/main/resources/content/bin/systemd/wildfly-standalone.service.template

Jira issue: https://issues.redhat.com/browse/WFCORE-6935 [WFCORE-6935] Minor fix for README [WFCORE-6935] Correct naming for systemd, unit, file [WFCORE-6935] Review launch.sh, add the ability to confirm the changes and allow specify areguments in any order [WFCORE-6935] Apply changes after review

yersan

@rpelisse I've followed your latest review, Could you check again and approve if you are fine? thansk!

yersan · 2024-11-18T09:11:23Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+}
+
+SCRIPT_NAME=$(basename "${0}")
+SCRIPT_DIR="$(dirname "$(realpath "${0}")")"


yersan · 2024-11-18T09:15:54Z

core-feature-pack/common/src/main/resources/content/bin/systemd/generate_systemd_unit.sh

+JBOSS_SYSTEMD_GROUP="wildfly"
+JBOSS_CONFIRM="n"
+
+while [ "$#" -gt 0 ]


yersan · 2024-11-18T09:16:32Z

core-feature-pack/common/src/main/resources/content/bin/systemd/launch.sh

@@ -0,0 +1,26 @@
+#!/bin/sh
+if [ "$WILDFLY_SYSTEMD_DEBUG" == "true" ]; then


rpelisse

LGTM!

yersan · 2024-11-19T12:50:13Z

Thanks @rpelisse and @jamezp

yersan added Feature This PR adds a new feature to WildFly missing-reqs This PR is missing external requirements before it can be merged labels Sep 18, 2024

yersan force-pushed the WFCORE-6935 branch 3 times, most recently from 0af8db3 to 5e7a85b Compare September 18, 2024 15:08

This comment was marked as off-topic.

Sign in to view

jamezp reviewed Oct 4, 2024

View reviewed changes

yersan commented Oct 8, 2024

View reviewed changes

jamezp approved these changes Oct 9, 2024

View reviewed changes

core-feature-pack/common/src/main/resources/content/bin/systemd/README Show resolved Hide resolved

core-feature-pack/common/src/main/resources/content/bin/systemd/wildfly-domain.service Show resolved Hide resolved

rpelisse reviewed Oct 15, 2024

View reviewed changes

core-feature-pack/common/src/main/resources/content/bin/systemd/README Show resolved Hide resolved

rpelisse reviewed Oct 15, 2024

View reviewed changes

core-feature-pack/common/src/main/resources/content/bin/systemd/launch.sh Show resolved Hide resolved

yersan force-pushed the WFCORE-6935 branch from 5e7a85b to 96f6ca3 Compare October 15, 2024 15:34

yersan commented Oct 15, 2024

View reviewed changes

yersan requested a review from rpelisse October 15, 2024 15:37

This comment was marked as outdated.

Sign in to view

yersan force-pushed the WFCORE-6935 branch from bcf9528 to d13f263 Compare October 21, 2024 15:05

This comment was marked as outdated.

Sign in to view

rpelisse suggested changes Nov 5, 2024

View reviewed changes

yersan force-pushed the WFCORE-6935 branch from d13f263 to 9da7785 Compare November 8, 2024 15:59

yersan commented Nov 8, 2024

View reviewed changes

This comment was marked as off-topic.

Sign in to view

rpelisse suggested changes Nov 16, 2024

View reviewed changes

yersan force-pushed the WFCORE-6935 branch from 9da7785 to ff100a9 Compare November 18, 2024 09:44

yersan commented Nov 18, 2024

View reviewed changes

rpelisse approved these changes Nov 18, 2024

View reviewed changes

yersan added the ready-for-merge This PR is ready to be merged and fulfills all requirements label Nov 18, 2024

yersan merged commit ebdf73f into wildfly:main Nov 19, 2024
12 of 13 checks passed

yersan deleted the WFCORE-6935 branch November 19, 2024 12:50


		== Start and enable WildFly

		# systemctl start wildfly-$MODE.service


		== Remove WildFly systemd service

		# rm -f $(pkg-config systemd --variable=systemdsystemunitdir)/wildfly-$MODE.service

		@@ -0,0 +1,26 @@
		#!/bin/sh
		if [ "$WILDFLY_SYSTEMD_DEBUG" == "true" ]; then

[WFCORE-6935] Improve systemd service units #6179

[WFCORE-6935] Improve systemd service units #6179

Conversation

yersan commented Sep 18, 2024

This comment was marked as off-topic.

jamezp left a comment

Choose a reason for hiding this comment

yersan left a comment

Choose a reason for hiding this comment

jamezp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rm -f "$(pkg-config systemd --variable=systemdsystemunitdir)/wildfly-${MODE}.service"

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yersan left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

yersan commented Oct 24, 2024

rpelisse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yersan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as off-topic.

rpelisse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yersan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rpelisse left a comment