kernel tuning settings priority

[jewelnuruddin]

NOTE
I mark this is an issue for an existing and complex puppet environment.

For various reason we need to tuning kernel value by sysctl, in our environment we found that some value are update after that related service up which is not effective until we manually restart that service again, manually restart is not convenient and often forget.

For example:
Looking at the startup order of puppet, rmem has been set since td-agent was started. 
This caused a problem.

Dec 13 17:11:58 XXXXXXXX puppet-agent[28661]: (/Stage[main]/Fluentd/Service[td-agent]) Unscheduling refresh on Service[td-agent]
...
Dec 13 17:16:07 XXXXXXXX puppet-agent[28661]: (/Stage[main]/Essentials/Sysctl[net.core.rmem_default]/File[/etc/sysctl.d/net.core.rmem_default.conf]/ensure) defined content as '{md5}0ac3ec38daa9ef3371a7b209f1f7b7b1'
Dec 13 17:16:07 XXXXXXXX puppet-agent[28661]: (/Stage[main]/Essentials/Sysctl[net.core.rmem_default]/Exec[sysctl-net.core.rmem_default]/returns) executed successfully

What we want to do
We want to make sure that our kernel tuning value must be setup first before related services is up.

By Puppet resource
Puppet already have some way to manage execution priority/order by "contain","require", "before", "notify"

• To work with this need to think about priority from the very beginning.
• Easy to manage in a simple environment, but I guess production is more complex class,templates definition
• So this is not guaranteed

By Puppet "Run Stages"
There are some possibility to resolve this issue by using Run Stages

But this is not recommended until you know exactly what you want to do,
and also have limitation
Possible solution
Puppet "Run Stages" settings can solve this problem though it have some limitation,
Because of limitation current sysctl module need some little changes, this change will not effect existing users,
it just a re-arrangement of declaration,

Here I attached an example of puppet graph

What if we use Run Stages with this current settings ?

Problem is dependency cycle as we use

include sysctl::base

inside sysctl/manifests/init.pp

Error is

Error: Could not apply complete catalog: Found 1 dependency cycle:
(Exec[sysctl-kernel.panic] => Sysctl[kernel.panic] => Class[Basehost] => Stage[first] => Stage[main] => Class[Sysctl::Base] => File[/etc/sysctl.d] => File[/etc/sysctl.d/kernel.panic.conf] => Sysctl[kernel.panic])

This is happened because of limitation

[jewelnuruddin]

Possible Solution 1

We can ignore "sysctl::base" because its create directory only,

  file { '/etc/sysctl.d':
    ensure => directory,
    owner  => 'root',
    group  => 'root',
    mode   => '0755',
    # Magic hidden here
    purge  => $purge,
  }

this directory is also ensure by "sysctl" class

  file { "/etc/sysctl.d/${sysctl_d_file}":
    ensure  => $ensure,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => "${title} = ${value}\n",
  }

[jewelnuruddin]

Possible Solution 2

No big change on current settings, just comment out

#include sysctl::base

and call this from site.pp or nodes.pp or some other templates file for your convenient
in my case I use like declare a basehost class

class basehost {
    include sysctl::base
    sysctl { "net.ipv4.tcp_max_syn_backlog" : value => '65535' }
    sysctl { "net.core.somaxconn"           : value => '65535' }

Its works for me and fulfill my purpose.
It ensure every kernel settings hit first before services are run.