Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BIGTOP-3867: Add an option for installing and using newer java version on bigtop-deploy #1067

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

yoda-mon
Copy link
Contributor

@yoda-mon yoda-mon commented Dec 1, 2022

Description of PR

This PR add an option that installs and uses newer java version on bigtop-deploy for Puppet.
Bigtop's default Java version is 8, this option enables some components (on this PR, Kafka) to use newer version.

  1. If user does not specify this option, there is no change.
  2. If users set --[component]-java-version [java_version] for docker-hadoop.sh, docker-hadoop.sh will set bigtop::${component}_java_version: "${version}" to config/hearadata/site.xml.
  3. Some specific component's init.pp receive the value from site.xml. init.pp compares the value with 8 and if it is greater than 8, try to install that java version. (To improve reusability I write defined component to jdk.pp ). In addition, init.pp places /etc/profile.d/bigtop_${component}_javaversion.sh. This will set BIGTOP_KAFKA_JAVA_VERSION=${JAVA_VERSION} to environment variable.
  4. bigtop-utils/bigtop-detect-component-specific-javahome $component check the environment variable BIGTOP_${COMPONENT}_JAVA_VERSION , and try to overwrite JAVA_HOME to the specific version during the process. By lauching or accessing through this script, that component can choose appropricate java version.

If users want to adds this option to another component, the steps below does.

  1. Add some jdk.pp calls to init.pp.
  2. Decorate server and client script with bigtop-detect-component-specific-javahome

Limitation

  • The order of stack deployment will be sensitive because after installation, Java version is switch to newer version. I think this problem could be acceptable because Kafka and the other candidates(e.g. Flink) usually come later part.

How was this patch tested?

I tested on CentOS 7 and Ubuntu 20.04 docker container on

  • CentOS 7/ arm64
  • Ubuntu 20.04 / x86
    host machine.

Prerequirement

This PR includes bigtop-utils arrangement, so required to use local repo.

./gradlew -POS="centos-7"  \
  bigtop-utils-clean bigtop-utils-pkg-ind \
  zookeeper-clean zookeeper-pkg-ind \
  kafka-clean kafka-pkg-ind \
  repo-ind

Deploy and Smoke Test

./docker-hadoop.sh -d \
  -C config_centos-7.yaml \
  -c 3 \
  --stack zookeeper,kafka --smoke-tests zookeeper,kafka \
  --enable-local-repo --repo file:///bigtop-home/output \
  --kafka-java-version 11

Check newer java version

./docker-hadoop.sh --exec 1 bash
[root@bf23cebac91c /]# ps aux|grep java
zookeep+     744  1.3  0.4 6060380 77240 ?       Sl   09:13   0:00 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/bin/java ...
kafka        998 11.5  1.5 6890636 241836 ?      Sl   09:13   0:04 /usr/lib/jvm/java-11-openjdk-11.0.17.0.8-2.el7_9.x86_64/bin/java  ...

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
  • Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant