Hadoop Puppet Helper Library

Table of Contents

1. Module Description - What the module does and why it is useful
2. Setup - The basics of getting started with hadoop_lib
   • What hadoop_lib affects
3. Usage - Beginning, configuration options, and additional functionality
4. Reference - An under-the-hood peek at what the module is doing and how
   • Classes
   • hadoop_lib::env
   • Resource Types
   • hadoop_lib::jdbc
   • hadoop_lib::postinstall
   • Functions
   • array_search
5. Limitations - OS compatibility, etc.
6. Development - Guide for contributing to the module

Module Description

Common puppet library for Hadoop and its addons. It manages alternatives as used in BigTop-based Hadoop distributions and provides necessary environment variables.

Setup

What hadoop_lib affects

• Files created (env class):
• /etc/profile.d/hadoop.sh
• /etc/profile.d/hadoop.csh
• Links created (jdbc resource type)
• Alternatives (postinstall resource type):
• /etc/<NAME>/conf: it can be switched to the new alternative, so original configuration is kept intact

Usage

Example: alternatives

Package['hive']
->
::hadoop_lib::postinstall { 'hive':
  alternatives => 'cluster',
}

Example: environment

::hadoop_lib::env

Example: MySQL JDBC jar

class { 'mysql::bindings':
  java_enable => true,
}
->
::hadoop_lib::jdbc { '/usr/lib/hive/lib':
  db => 'mysql',
}

Example: array search function

$_myid = array_search($hostnames, $::fqdn)

Reference

###Classes

• hadoop_lib::env: Sets environment for Hadoop

hadoop_lib::env

Parameters

(none)

###Resources

• hadoop_lib::postinstall: Post-installation steps
• hadoop_lib::jdbc: Link JDBC jar file from standard location

hadoop_lib::postinstall

Parameters

#####alternatives

Switches the alternatives used for the configuration. Default: 'cluster' (Debian) or undef.

It can be used only when supported (for example with Cloudera distribution).

hadoop_lib::jdbc

Link JDBC jar file from standard location. Jar files must be installed.

Example of using puppetlabs modules for jar files installation:

• puppetlabs-mysql: mysql::bindings class with java_enable parameter set
• puppetlabs-postgresql: postgresql::lib::java class

Parameters

#####db

Database type. Default: undef.

Values:

• mysql, mariadb: MySQL/MariaDB,
• postgresql: PostgreSQL

###Functions

• array_search: Search a value in array

array_search

Searches a value in the array and returns the index starting from 1.

Parameters

#####array

#####value

##Development

• Repository: https://github.com/MetaCenterCloudPuppet/cesnet-hadoop_lib
• Tests:
• basic: see .travis.yml
• vagrant: https://github.com/MetaCenterCloudPuppet/hadoop-tests
• Email: František Dvořák <[email protected]>