Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Click cli #44

Open
wants to merge 102 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
c27b746
Refactor for plugins
Apr 9, 2015
7017e59
Add debug plugins
Apr 11, 2015
6f5a308
Add debug plugins
Apr 11, 2015
11b5a2c
Moar debug plugins
Apr 11, 2015
aeb3651
Merge remote-tracking branch 'origin/plugins' into plugins
Apr 11, 2015
0967c0a
Fix test
Apr 11, 2015
782870d
Moar debuggers
Apr 13, 2015
51686b8
Version bump to 1.0.0
Apr 13, 2015
51f537a
Merge branch 'master' into plugins
May 7, 2015
4e43d02
Finish porting custom repo
May 8, 2015
d5afe6e
Fix provision
viglesiasce May 11, 2015
6f10d9c
First pass at docs
viglesiasce May 12, 2015
f35146d
Added eucalyptus_sosreport.py; updated debuggerplugin.py and setup.py
hspencer77 May 12, 2015
388bb40
Adding repo validation
May 12, 2015
7e1ff92
Use set instantiation that is compatible with Python 2.6
viglesiasce May 12, 2015
5bf77e6
repos validation
May 12, 2015
4f45142
repos validation
May 12, 2015
0ec9fda
updating debuggerplugin.py to have get_command tasks
hspencer77 May 12, 2015
27db135
Repos validation update
May 12, 2015
1bdde76
repos and storage validation
May 13, 2015
e734101
storage validation modifications
May 13, 2015
82f1e82
Updated eucalyptus-sosreport.py to run jobs in parallel. Downloads to…
hspencer77 May 13, 2015
a4b34e9
Storage validation update
May 13, 2015
88f122d
Storage validation update
May 13, 2015
6107d58
storage validation modifications
May 14, 2015
b7e3317
Changed function names get_command_* to get_file_*; moved tasks assoc…
hspencer77 May 14, 2015
371bdf9
Storage validation update
May 14, 2015
04370ef
Updated eucalyptus_sosreport.py to download sosreports from nodes to …
hspencer77 May 14, 2015
f144017
Updated README.md
hspencer77 May 14, 2015
3d7d2f0
Update README.md
hspencer77 May 14, 2015
5f2d3f2
Updated README.md; corrected list display
hspencer77 May 14, 2015
8d49807
Updated README.md; corrected list display, version 2
hspencer77 May 14, 2015
bcf172b
Updated README.md; corrected list display, version 3
hspencer77 May 14, 2015
1eed6e2
Updated README.md; corrected list display, version 4
hspencer77 May 14, 2015
9f482f1
Updated README.md; corrected list display, version 5; corrected links
hspencer77 May 14, 2015
145f6cb
Storage validation update
May 15, 2015
bb2aa71
Storage validation update
May 15, 2015
d1fbf5f
Storage validation update
May 15, 2015
015bf4b
envrionment update
May 15, 2015
87452fb
Minimized error message output when get_file_on_host fails
hspencer77 May 18, 2015
ae63d4f
Fix single line comments in debuggerplugin.py
hspencer77 May 18, 2015
12a0084
Merge pull request #37 from hspencer77/deploy-sosreports-addition
viglesiasce May 18, 2015
3a7b123
Doc: add Quick Start and ToC
viglesiasce May 18, 2015
56aad52
Merge remote-tracking branch 'origin/plugins' into plugins
viglesiasce May 18, 2015
8846d78
prepare spec file for use in jenkins/rpmfab/mock builds
mbacchi May 19, 2015
e04e15c
move version figuring to setup.py
mbacchi May 19, 2015
ad14fb9
update source to use rpmfab macro
mbacchi May 19, 2015
fbb2650
Doc: Start plugin documentation
viglesiasce May 20, 2015
b48acbd
update setup macro in RPM spec file
mbacchi May 20, 2015
e8227a2
add BuildRequires entries
mbacchi May 20, 2015
4d16b5d
correct capitalization of PyYAML
mbacchi May 20, 2015
1da9fe9
additional RPM dependencies, add config.yml, use RPM files syntax
mbacchi May 21, 2015
eed5b2c
forgot to tar config.yml before trying to package it in the RPM
mbacchi May 21, 2015
270dba4
Adding yml structure validation and repos updates
May 21, 2015
87576f2
Setup update
May 21, 2015
99939d9
Adding structure validation and repos updates
May 21, 2015
fd64610
Updated error handling
May 21, 2015
7d7e8eb
Adding structure validation and repos updates
May 21, 2015
48e185e
Adding structure validation and repos updates
May 22, 2015
b416182
Removing .idea files
May 22, 2015
869c8bb
Changing environment file to default
May 22, 2015
2701327
ship and install examples directory
mbacchi May 22, 2015
05318bf
Changing structre options
May 22, 2015
4756326
Merge pull request #39 from ccassler/plugins
viglesiasce May 22, 2015
1fff9ec
correct placement of config.yml in /etc/euca-deploy
mbacchi May 22, 2015
6a88eb0
Merge remote branch mbacchi:'origin/master' into plugins
mbacchi May 22, 2015
f3def4d
Merge pull request #40 from mbacchi/plugins
viglesiasce May 22, 2015
53b2afa
First pass at rename to calyptos
viglesiasce May 26, 2015
37e3b7b
Merge pull request #1 from eucalyptus/rename
viglesiasce May 26, 2015
7b7b70d
Update README.md
viglesiasce May 26, 2015
46c382f
Update README.md
viglesiasce May 26, 2015
82c5bd3
Update README.md
viglesiasce May 26, 2015
7e4e4f0
Refactor componentdeployer->rolebuilder
viglesiasce May 26, 2015
56af824
Merge pull request #2 from eucalyptus/refactor-componentdeployer
viglesiasce May 26, 2015
480e2aa
Update README.md
viglesiasce May 26, 2015
01d8a1c
Update README.md
viglesiasce May 26, 2015
1b54354
Remove references to Chef
viglesiasce May 26, 2015
587fc77
Check for DNS on UFS for 4.2
viglesiasce May 26, 2015
10b9dfd
Merge remote-tracking branch 'origin/master'
viglesiasce May 26, 2015
ad4aed6
Set euca-4.2 as default cookbook branch
viglesiasce May 26, 2015
eef9c03
Add debugger docs
viglesiasce May 26, 2015
c107d4b
Update README.md
viglesiasce May 26, 2015
828fd64
Prepare FS before writing env file
viglesiasce May 27, 2015
ac6f854
Merge remote-tracking branch 'origin/master'
viglesiasce May 27, 2015
c28661c
Install chefdk in chef deployer
viglesiasce May 27, 2015
541af48
Move ChefDK install to ChefManager
viglesiasce May 27, 2015
5541aac
Move install chefDK to before preparing FS
viglesiasce May 27, 2015
fec4db6
Configure ChefManager before installing ChefDK
viglesiasce May 27, 2015
b7f70f0
Make ChefDK install a staticmethod called as first step of _prepare_fs
viglesiasce May 27, 2015
77841bd
Second part of last fix
viglesiasce May 27, 2015
7225ccf
Ensure our version of ChefDK is installed
viglesiasce May 27, 2015
b251390
ship config.yml to /usr/share and copy to user home ~/.calyptos
mbacchi Jun 1, 2015
c878e0e
iterate through possible config file locations
mbacchi Jun 3, 2015
b4ced19
added ceph deploy in calyptos. Fixes QA-361
shaon Jun 6, 2015
8145708
added riakcs cluster deploy. Fixes QA-362
shaon Jun 8, 2015
d0de2a1
Merge pull request #6 from shaon/cephdeploy
viglesiasce Jun 9, 2015
3e32d88
Merge pull request #7 from shaon/riakcsdeploy
viglesiasce Jun 9, 2015
6cdc3d5
Merge pull request #4 from mbacchi/pkg_config
viglesiasce Jun 9, 2015
9954169
Make environment a required flag, remove default
viglesiasce Jun 9, 2015
dedfe01
update ceph+euca deployment yaml
shaon Jun 10, 2015
e6a3b27
Merge pull request #8 from shaon/cepheucayml
viglesiasce Jun 10, 2015
ffab720
Use click for main CLI
viglesiasce Jun 15, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Refactor componentdeployer->rolebuilder
  • Loading branch information
viglesiasce committed May 26, 2015
commit 7e4e4f001001546c6a68b7075d20cee12ecc3b99
4 changes: 2 additions & 2 deletions bin/calyptos
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@
import argparse
from fabric.colors import yellow
from stevedore import driver, extension
from calyptos.componentdeployer import ComponentDeployer
from calyptos.rolebuilder import RoleBuilder

if __name__ == '__main__':
parser = argparse.ArgumentParser()
@@ -23,7 +23,7 @@ if __name__ == '__main__':
parser.add_argument('--debug', action='store_true', default=False)
args = parser.parse_args()

component_deployer = ComponentDeployer(args.environment)
component_deployer = RoleBuilder(args.environment)
if args.operation == 'validate':
mgr = extension.ExtensionManager(
namespace='calyptos.validator',
30 changes: 18 additions & 12 deletions calyptos/plugins/deployer/chef.py
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
from fabric.tasks import execute
from calyptos.chefmanager import ChefManager
import os
from calyptos.componentdeployer import ComponentDeployer
from calyptos.rolebuilder import RoleBuilder


class Chef(DeployerPlugin):
@@ -19,15 +19,14 @@ def __init__(self, password, environment_file='etc/environment.yml',
self.hidden_outputs = []
else:
self.hidden_outputs = ['running', 'stdout', 'stderr']
self.component_deployer = ComponentDeployer(environment_file)
self.roles = self.component_deployer.get_roles()
self.role_builder = RoleBuilder(environment_file)
self.roles = self.role_builder.get_roles()
self.all_hosts = self.roles['all']
self.environment_name = self._write_json_environment()
self._prepare_fs(repo, branch, debug)
self.chef_manager = ChefManager(password, self.environment_name,
self.roles['all'])
self.config_file = config_file
self.config = self.read_config()
self.config = self.get_chef_config(config_file)

def _prepare_fs(self, repo, branch, debug):
ChefManager.create_chef_repo()
@@ -43,11 +42,19 @@ def _prepare_fs(self, repo, branch, debug):
'/cookbooks'),
debug=debug)

def read_config(self):
return yaml.load(open(self.config_file).read())
@staticmethod
def get_chef_config(config_file):
full_config = yaml.load(open(config_file).read())
if 'deployer' in full_config:
if 'chef' in full_config['deployer']:
return full_config['deployer']['chef']
else:
raise IndexError('Unable to find chef config in deployer section of config file')
else:
raise IndexError('Unable to find deployer section of config file')

def _get_recipe_list(self, component):
for recipe_dict in self.config['recipes']:
for recipe_dict in self.config['roles']:
if component in recipe_dict:
return recipe_dict[component]
raise ValueError('No component found for: ' + component)
@@ -95,7 +102,7 @@ def bootstrap(self):
def provision(self):
# Install all other components and configure CLC
self.chef_manager.clear_run_list(self.all_hosts)
for role_dict in self.config['recipes']:
for role_dict in self.config['roles']:
component_name = role_dict.keys().pop()
self.chef_manager.add_to_run_list(self.roles[component_name],
self._get_recipe_list(
@@ -104,7 +111,7 @@ def provision(self):
clc = self.roles['clc']
self.chef_manager.add_to_run_list(clc, ['eucalyptus::configure'])
self._run_chef_on_hosts(clc)
if self.component_deployer.get_euca_attributes()['network']['mode'] == 'VPCMIDO':
if self.role_builder.get_euca_attributes()['network']['mode'] == 'VPCMIDO':
midonet_gw = self.roles['midonet-gw']
create_resources = 'midokura::create-first-resources'
self.chef_manager.add_to_run_list(midonet_gw,
@@ -113,7 +120,6 @@ def provision(self):

def uninstall(self):
self.chef_manager.clear_run_list(self.all_hosts)
self.chef_manager.add_to_run_list(self.all_hosts,
self._get_recipe_list('nuke'))
self.chef_manager.add_to_run_list(self.all_hosts, ['eucalyptus::nuke'])
self._run_chef_on_hosts(self.all_hosts)
self.chef_manager.clear_run_list(self.all_hosts)
37 changes: 27 additions & 10 deletions calyptos/componentdeployer.py → calyptos/rolebuilder.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
import yaml


class ComponentDeployer():
class RoleBuilder():

# Global list of roles
ROLE_LIST = ['clc', 'user-facing', 'midonet-gw',
'cluster-controller', 'storage-controller',
'node-controller', 'midolman', 'all']

def __init__(self, environment_file='environment.yml'):
self.environment_file = environment_file
self.roles = self.get_roles()
@@ -14,26 +20,35 @@ def get_euca_attributes(self):
env_dict = self.read_environment()
return env_dict['default_attributes']['eucalyptus']

def _initialize_roles(self):
roles = {}
for role in self.ROLE_LIST:
roles[role] = set()
return roles

def get_roles(self):
roles = self._initialize_roles()
euca_attributes = self.get_euca_attributes()
topology = euca_attributes['topology']
if 'clc-1' not in topology:
raise IndexError("Unable to find CLC in topology")
roles = {'clc': set([topology['clc-1']]),
'user-facing': set(topology['user-facing']),
'cluster-controller': set(), 'storage-controller': set(),
'node-controller': set(), 'vmware-broker': set(), 'nuke': set(),
'midolman': set(), 'midonet-gw': set(),
'all': set([topology['clc-1']])
}

# Add CLC
roles['clc'] = set([topology['clc-1']])
roles['all'] = set([topology['clc-1']])

# Add UFS
roles['user-facing'] = set(topology['user-facing'])
for ufs in topology['user-facing']:
roles['all'].add(ufs)

# Add Walrus
if 'walrus' in topology:
roles['walrus'] = set([topology['walrus']])
roles['all'].add(topology['walrus'])
else:
# No walrus defined assuming RiakCS
roles['walrus'] = set()

# Add cluster level components
for name in topology['clusters']:
roles['cluster'] = {}
if 'cc-1' in topology['clusters'][name]:
@@ -57,6 +72,8 @@ def get_roles(self):
roles['node-controller'].add(node)
roles['cluster'][name].add(node)
roles['all'].update(roles['cluster'][name])

# Add Midokura roles
if euca_attributes['network']['mode'] == 'VPCMIDO':
roles['midolman'] = roles['node-controller']
roles['midonet-gw'] = roles['clc']
44 changes: 21 additions & 23 deletions etc/config.yml
Original file line number Diff line number Diff line change
@@ -1,24 +1,22 @@
---
recipes:
- clc:
- eucalyptus::cloud-controller
- eucalyptus::register-components
- user-facing:
- eucalyptus::user-facing
- eucalyptus::user-console
- walrus:
- eucalyptus::walrus
- cluster-controller:
- eucalyptus::cluster-controller
- storage-controller:
- eucalyptus::storage-controller
- vmware-broker:
- eucalyptus::vmware-broker
- node-controller:
- eucalyptus::node-controller
- midolman:
- midokura::midolman
- midonet-gw:
- midokura
- nuke:
- eucalyptus::nuke
deployer:
chef:
roles:
- clc:
- eucalyptus::cloud-controller
- eucalyptus::register-components
- user-facing:
- eucalyptus::user-facing
- eucalyptus::user-console
- walrus:
- eucalyptus::walrus
- cluster-controller:
- eucalyptus::cluster-controller
- storage-controller:
- eucalyptus::storage-controller
- node-controller:
- eucalyptus::node-controller
- midolman:
- midokura::midolman
- midonet-gw:
- midokura
4 changes: 2 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
@@ -140,6 +140,6 @@ def make_release_tree(self, base_dir, files):
'build_py': build_py_with_git_version,
'sdist': sdist_with_git_version
},
data_files=[('/etc/calyptos/', ['etc/config.yml']),
('/usr/share/calyptos/examples/', example_items)],
#data_files=[('/etc/calyptos/', ['etc/config.yml']),
# ('/usr/share/calyptos/examples/', example_items)],
)
4 changes: 2 additions & 2 deletions tests/test_component_deployer.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
from calyptos.componentdeployer import ComponentDeployer
from calyptos.rolebuilder import RoleBuilder


def test_constructor():
component_deployer = ComponentDeployer('etc/environment.yml')
component_deployer = RoleBuilder('etc/environment.yml')