Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change serialization of alarms and sensors for objects #273

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Change serialization of alarms and sensors for objects #273

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0e2feac
Fix for badly behaving super-micro sensor
Z903 Jun 10, 2021
028c113
Change serialization of alarms and sensors for objects
Z903 Jun 12, 2021
89e9b32
Merge tag 'v0.16.1' into main
Z903 Jun 12, 2021
7225abd
Add vmware_*_*_alarm metrics
Z903 Jul 7, 2021
4993ee9
Add --allow-url-target and remove 'default' as a requirement in confi…
Z903 Jul 7, 2021
c34898c
Fix type of sensor_status from 'unknown' to 3
Z903 Jul 7, 2021
63a9d68
Add timeout to VmwareCollector
Z903 Jan 6, 2022
ab16bb8
Replace 500 with 400 as per https://github.com/pryorda/vmware_exporte…
Z903 Jan 10, 2022
8ec432d
Remove old commented out code
Z903 Feb 6, 2023
5c310f4
Remove target and vsphere_host url parameters.
Z903 Feb 11, 2023
e10e78a
Merge remote-tracking branch 'remotes/pryorda/main' into main
Z903 Feb 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 49 additions & 54 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ default:
vsphere_user: "user"
vsphere_password: "password"
ignore_ssl: False
timeout: 120
specs_size: 5000
fetch_custom_attributes: True
fetch_tags: True
Expand All @@ -78,11 +79,12 @@ default:
hosts: True
snapshots: True

esx:
vcenter01:
vsphere_host: vc.example2.com
vsphere_user: 'root'
vsphere_password: 'password'
ignore_ssl: True
timeout: 120
specs_size: 5000
fetch_custom_attributes: True
fetch_tags: True
Expand All @@ -94,11 +96,12 @@ esx:
hosts: True
snapshots: True

limited:
vcenter02:
vsphere_host: slowvc.example.com
vsphere_user: '[email protected]'
vsphere_password: 'password'
ignore_ssl: True
timeout: 120
specs_size: 5000
fetch_custom_attributes: True
fetch_tags: True
Expand All @@ -114,39 +117,41 @@ limited:
Switching sections can be done by adding ?section=limited to the URL.

#### Environment Variables
| Variable | Precedence | Defaults | Description |
| --------------------------------------| ---------------------- | -------- | --------------------------------------------------------------------------|
| `VSPHERE_HOST` | config, env, get_param | n/a | vsphere server to connect to |
| `VSPHERE_USER` | config, env | n/a | User for connecting to vsphere |
| `VSPHERE_PASSWORD` | config, env | n/a | Password for connecting to vsphere |
| `VSPHERE_SPECS_SIZE` | config, env | 5000 | Size of specs list for query stats function |
| `VSPHERE_IGNORE_SSL` | config, env | False | Ignore the ssl cert on the connection to vsphere host |
| `VSPHERE_FETCH_CUSTOM_ATTRIBUTES` | config, env | False | Set to true to collect objects custom attributes as metric labels |
| `VSPHERE_FETCH_TAGS` | config, env | False | Set to true to collect objects tags as metric labels |
| `VSPHERE_FETCH_ALARMS` | config, env | False | Fetch objects triggered alarms, and in case of hosts hdw alarms as well |
| `VSPHERE_COLLECT_HOSTS` | config, env | True | Set to false to disable collection of host metrics |
| `VSPHERE_COLLECT_DATASTORES` | config, env | True | Set to false to disable collection of datastore metrics |
| `VSPHERE_COLLECT_VMS` | config, env | True | Set to false to disable collection of virtual machine metrics |
| `VSPHERE_COLLECT_VMGUESTS` | config, env | True | Set to false to disable collection of virtual machine guest metrics |
| `VSPHERE_COLLECT_SNAPSHOTS` | config, env | True | Set to false to disable collection of snapshot metrics |
| Variable | Precedence | Defaults | Description |
| ---------------------------------------------| ------------| -------- | ---------------------------------------------------------------------------------|
| `VSPHERE_HOST` | config, env | n/a | vsphere server to connect to |
| `VSPHERE_USER` | config, env | n/a | User for connecting to vsphere |
| `VSPHERE_PASSWORD` | config, env | n/a | Password for connecting to vsphere |
| `VSPHERE_SPECS_SIZE` | config, env | 5000 | Size of specs list for query stats function |
| `VSPHERE_IGNORE_SSL` | config, env | False | Ignore the ssl cert on the connection to vsphere host |
| `VSPHERE_TIMEOUT` | config, env | 120 | Set how long to wait before failing to collect |
| `VSPHERE_FETCH_CUSTOM_ATTRIBUTES` | config, env | False | Set to true to collect objects custom attributes as metric labels |
| `VSPHERE_FETCH_TAGS` | config, env | False | Set to true to collect objects tags as metric labels |
| `VSPHERE_FETCH_ALARMS` | config, env | False | Fetch objects triggered alarms, and in case of hosts hdw alarms as well |
| `VSPHERE_COLLECT_HOSTS` | config, env | True | Set to false to disable collection of host metrics |
| `VSPHERE_COLLECT_DATASTORES` | config, env | True | Set to false to disable collection of datastore metrics |
| `VSPHERE_COLLECT_VMS` | config, env | True | Set to false to disable collection of virtual machine metrics |
| `VSPHERE_COLLECT_VMGUESTS` | config, env | True | Set to false to disable collection of virtual machine guest metrics |
| `VSPHERE_COLLECT_SNAPSHOTS` | config, env | True | Set to false to disable collection of snapshot metrics |

You can create new sections as well, with very similiar variables. For example, to create a `limited` section you can set:

| Variable | Precedence | Defaults | Description |
| ----------------------------------------------| ---------------------- | -------- | --------------------------------------------------------------------------|
| `VSPHERE_LIMITED_HOST` | config, env, get_param | n/a | vsphere server to connect to |
| `VSPHERE_LIMITED_USER` | config, env | n/a | User for connecting to vsphere |
| `VSPHERE_LIMITED_PASSWORD` | config, env | n/a | Password for connecting to vsphere |
| `VSPHERE_LIMITED_SPECS_SIZE` | config, env | 5000 | Size of specs list for query stats function |
| `VSPHERE_LIMITED_IGNORE_SSL` | config, env | False | Ignore the ssl cert on the connection to vsphere host |
| `VSPHERE_LIMITED_FETCH_CUSTOM_ATTRIBUTES` | config, env | False | Set to true to collect objects custom attributes as metric labels |
| `VSPHERE_LIMITED_FETCH_TAGS` | config, env | False | Set to true to collect objects tags as metric labels |
| `VSPHERE_LIMITED_FETCH_ALARMS` | config, env | False | Fetch objects triggered alarms, and in case of hosts hdw alarms as well |
| `VSPHERE_LIMITED_COLLECT_HOSTS` | config, env | True | Set to false to disable collection of host metrics |
| `VSPHERE_LIMITED_COLLECT_DATASTORES` | config, env | True | Set to false to disable collection of datastore metrics |
| `VSPHERE_LIMITED_COLLECT_VMS` | config, env | True | Set to false to disable collection of virtual machine metrics |
| `VSPHERE_LIMITED_COLLECT_VMGUESTS` | config, env | True | Set to false to disable collection of virtual machine guest metrics |
| `VSPHERE_LIMITED_COLLECT_SNAPSHOTS` | config, env | True | Set to false to disable collection of snapshot metrics |
| Variable | Precedence | Defaults | Description |
| ---------------------------------------------| ----------- | -------- | ---------------------------------------------------------------------------------|
| `VSPHERE_LIMITED_HOST` | config, env | n/a | vsphere server to connect to |
| `VSPHERE_LIMITED_USER` | config, env | n/a | User for connecting to vsphere |
| `VSPHERE_LIMITED_PASSWORD` | config, env | n/a | Password for connecting to vsphere |
| `VSPHERE_LIMITED_SPECS_SIZE` | config, env | 5000 | Size of specs list for query stats function |
| `VSPHERE_LIMITED_IGNORE_SSL` | config, env | False | Ignore the ssl cert on the connection to vsphere host |
| `VSPHERE_LIMITED_TIMEOUT` | config, env | 120 | Set how long to wait before failing to collect |
| `VSPHERE_LIMITED_FETCH_CUSTOM_ATTRIBUTES` | config, env | False | Set to true to collect objects custom attributes as metric labels |
| `VSPHERE_LIMITED_FETCH_TAGS` | config, env | False | Set to true to collect objects tags as metric labels |
| `VSPHERE_LIMITED_FETCH_ALARMS` | config, env | False | Fetch objects triggered alarms, and in case of hosts hdw alarms as well |
| `VSPHERE_LIMITED_COLLECT_HOSTS` | config, env | True | Set to false to disable collection of host metrics |
| `VSPHERE_LIMITED_COLLECT_DATASTORES` | config, env | True | Set to false to disable collection of datastore metrics |
| `VSPHERE_LIMITED_COLLECT_VMS` | config, env | True | Set to false to disable collection of virtual machine metrics |
| `VSPHERE_LIMITED_COLLECT_VMGUESTS` | config, env | True | Set to false to disable collection of virtual machine guest metrics |
| `VSPHERE_LIMITED_COLLECT_SNAPSHOTS` | config, env | True | Set to false to disable collection of snapshot metrics |

You need to set at least `VSPHERE_SECTIONNAME_USER` for the section to be detected.

Expand All @@ -155,18 +160,24 @@ You need to set at least `VSPHERE_SECTIONNAME_USER` for the section to be detect
You can use the following parameters in the Prometheus configuration file. The `params` section is used to manage multiple login/passwords.

```
# Example of Multiple vCenter usage per #23

- job_name: 'vmware_vcenter'
metrics_path: '/metrics'
static_configs:
- targets:
- 'vcenter.company.com'
- default
- vcenter01
- vcenter02
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: __param_section
- source_labels: [__param_section]
target_label: instance
- target_label: __address__
replacement: localhost:9272
replacement: exporter_ip:9272

# Example using file service discovery

- job_name: 'vmware_esx'
metrics_path: '/metrics'
Expand All @@ -177,28 +188,12 @@ You can use the following parameters in the Prometheus configuration file. The `
section: [esx]
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: __param_section
- source_labels: [__param_section]
target_label: instance
- target_label: __address__
replacement: localhost:9272

# Example of Multiple vCenter usage per #23

- job_name: vmware_export
metrics_path: /metrics
static_configs:
- targets:
- vcenter01
- vcenter02
- vcenter03
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: exporter_ip:9272
```

## Current Status
Expand Down
22 changes: 21 additions & 1 deletion tests/unit/test_helpers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

from pyVmomi import vim

from vmware_exporter.helpers import batch_fetch_properties, get_bool_env
from vmware_exporter.helpers import batch_fetch_properties, get_bool_env, serialize, deserialize


class FakeView(vim.ManagedObject):
Expand All @@ -16,6 +16,26 @@ def Destroy(self):
pass


def test_serialize():

# Test basic usage
assert serialize('abc', 'def', g='1', h='2') == 'abc:def:g=1:h=2'

# Test escaping
assert serialize('\\\n\r,:=') == '\\\\\\n\\r\\,\\:\\='


def test_deserialize():

tests = [
'abc:def:g=1:h=2',
'\\\\\\n\\r\\,\\:\\=:\\\\\\n\\r\\,\\:\\==\\\\\\n\\r\\,\\:\\='
]
for value in tests:
arg, kwarg = deserialize(value)
assert serialize(*arg, **kwarg) == value


def test_get_bool_env():
# Expected behaviour
assert get_bool_env('NON_EXISTENT_ENV', True)
Expand Down
2 changes: 1 addition & 1 deletion vmware_exporter/defer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ class BranchingDeferred(defer.Deferred):
'''

def __init__(self):
self.callbacks = []
super().__init__()
self.result = None

def callback(self, result):
Expand Down
131 changes: 93 additions & 38 deletions vmware_exporter/helpers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,65 @@
# autopep8'd
import os
from pyVmomi import vmodl
import re


def serialize(*arg, **kwarg):
"""
Serialize into the format
item1:item2:key1=value1:key2:value2
"""
escape_dict = {'\\': '\\\\', '\n': '\\n', '\r': '\\r', ',': '\\,', ':': '\\:', '=': '\\='}
pattern = re.compile("|".join([re.escape(k) for k in sorted(escape_dict, key=len, reverse=True)]), flags=re.DOTALL)
items = []
for item in arg:
items.append(pattern.sub(lambda x: escape_dict[x.group(0)], str(item)))
for k, v in kwarg.items():
items.append("{}={}".format(
pattern.sub(lambda x: escape_dict[x.group(0)], str(k)),
pattern.sub(lambda x: escape_dict[x.group(0)], str(v))
))
return ':'.join(items)


def deserialize(s):
"""
Deserialize the format into a list and dict
item1:item2:key1=value1:key2:value2
"""
escape_dict = {'\\r': '\r', '\\\\': '\\', '\\=': '=', '\\:': ':', '\\n': '\n', '\\,': ','}
pattern = re.compile("|".join([re.escape(k) for k in sorted(escape_dict, key=len, reverse=True)]), flags=re.DOTALL)
arg = []
kwarg = {}
for name, eq, value in re.findall(r'((?:\\.|[^=:\\]+)+)(?:(=)((?:\\.|[^:\\]+)+))?', s):
if eq:
kwarg[pattern.sub(lambda x: escape_dict[x.group(0)], name)] = pattern.sub(
lambda x: escape_dict[x.group(0)], value)
else:
arg.append(pattern.sub(lambda x: escape_dict[x.group(0)], name))
return (arg, kwarg)


class TriggeredAlarm(object):
def __init__(self, name, status):
self.name = name
self.sensorStatus = status

def __str__(self):
return serialize('triggeredAlarm', self.name, self.sensorStatus)


class NumericSensorInfo(object):
def __init__(self, name, status, type="n/a", value="n/a", unitModifier="n/a", unit="n/a"):
self.name = name
self.type = type
self.sensorStatus = status
self.value = value
self.unitModifier = unitModifier
self.unit = unit

def __str__(self):
return serialize('numericSensorInfo', name=self.name, type=self.type, sensorStatus=self.sensorStatus, value=self.value, unitModifier=self.unitModifier, unit=self.unit)


def get_bool_env(key: str, default: bool):
Expand All @@ -26,13 +85,9 @@ def batch_fetch_properties(content, obj_type, properties):

if content.customFieldsManager and content.customFieldsManager.field:
allCustomAttributesNames.update(
dict(
[
(f.key, f.name)
for f in content.customFieldsManager.field
if f.managedObjectType in (obj_type, None)
]
)
(f.key, f.name)
for f in content.customFieldsManager.field
if f.managedObjectType in (obj_type, None)
)

try:
Expand Down Expand Up @@ -95,32 +150,37 @@ def batch_fetch_properties(content, obj_type, properties):
"""
triggered alarms
"""

try:
alarms = list(
'triggeredAlarm:{}:{}'.format(item.alarm.info.systemName.split('.')[1], item.overallStatus)
for item in prop.val
)
properties[prop.name] = [
TriggeredAlarm(
name=item.alarm.info.systemName.split('.')[1],
status=item.overallStatus
) for item in prop.val
]
except Exception:
alarms = ['triggeredAlarm:AlarmsUnavailable:yellow']

properties[prop.name] = ','.join(alarms)
properties[prop.name] = [
TriggeredAlarm(
name='AlarmsUnavailable',
status='yellow'
)
]

elif 'runtime.healthSystemRuntime.systemHealthInfo.numericSensorInfo' == prop.name:
"""
handle numericSensorInfo
"""
sensors = list(
'numericSensorInfo:name={}:type={}:sensorStatus={}:value={}:unitModifier={}:unit={}'.format(
item.name,
item.sensorType,
item.healthState.key,
item.currentReading,
item.unitModifier,
item.baseUnits.lower()
)
for item in prop.val
)
properties[prop.name] = ','.join(sensors)

properties[prop.name] = [
NumericSensorInfo(
name=item.name,
type=item.sensorType,
status=item.healthState.key,
value=item.currentReading,
unitModifier=item.unitModifier,
unit=item.baseUnits.lower()
) for item in prop.val
]

elif prop.name in [
'runtime.healthSystemRuntime.hardwareStatusInfo.cpuStatusInfo',
Expand All @@ -129,18 +189,13 @@ def batch_fetch_properties(content, obj_type, properties):
"""
handle hardwareStatusInfo
"""
sensors = list(
'numericSensorInfo:name={}:type={}:sensorStatus={}:value={}:unitModifier={}:unit={}'.format(
item.name,
"n/a",
item.status.key,
"n/a",
"n/a",
"n/a",
)
for item in prop.val
)
properties[prop.name] = ','.join(sensors)

properties[prop.name] = [
NumericSensorInfo(
name=item.name,
status=item.status.key
) for item in prop.val
]

else:
properties[prop.name] = prop.val
Expand Down
Loading