Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add InfluxDB V2 support #195

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Conversation

gcrevell
Copy link

This addresses issue #159, adding support for InfluxDB 2 to Varken.

To use InfluxDB 2, start an instance it. I recommend this docker image. The config is pretty different from version 1, requiring an organization name and token for authentication.

I recommend creating the bucket in the database ahead of time, because then a token can be created that only allows writes to the single bucket. If creating a bucket is needed, varken will need an all-access token, instead of a read-write token.

I've tested this myself and it seems to work pretty well, but let me know if there's any questions or issues!

@Steve0320
Copy link

I've pulled this down and it seems to work well! The only issue I've noticed is that the bucket setting in the ini doesn't work - looks like the setter line in iniparser.py is missing.

@gcrevell
Copy link
Author

gcrevell commented Mar 1, 2021

@Steve0320 Thanks! Fixed that up so now its using the correct bucket

Copy link

@PrivatePuffin PrivatePuffin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty solid!

@dirtycajunrice
Copy link
Member

I will try to look this evening

@gjtorikian

This comment has been minimized.

@samwiseg0

This comment has been minimized.

@YouveGotMeowxy
Copy link

I cant seem to get this working. Did I do it wrong?

I'm using: boerderij/varken:develop with InfluxDB 2.1.1

My .ini:

[influxdb]
url = influxdb
port = 8086
ssl = false
verify_ssl = false
username = 1232132132323231
password = O_TGI_LjlX0tawrr<redacted>BfAFB6bYoz4Q==

where my username is my organization and pass is the token.

I have also tried using the text version of my organization wrapped in double quotes. No matter what I use I just keep getting an unauthorized error in the log.

today at 9:18:42 AM2022-03-19 09:18:42 : DEBUG : connectionpool : Starting new HTTP connection (1): ********luxdb:8086
today at 9:18:42 AM2022-03-19 09:18:42 : DEBUG : connectionpool : http://********luxdb:8086 "GET /ping HTTP/1.1" 204 0
today at 9:18:42 AM2022-03-19 09:18:42 : INFO : dbmanager : Influxdb version: 2.1.1, 2.1.1
today at 9:18:42 AM2022-03-19 09:18:42 : DEBUG : connectionpool : http://********luxdb:8086 "GET /query?q=SHOW+DATABASES&db=varken HTTP/1.1" 401 48
today at 9:18:42 AMTraceback (most recent call last):
today at 9:18:42 AM  File "/app/Varken.py", line 94, in <module>
today at 9:18:42 AM    DBMANAGER = DBManager(CONFIG.influx_server)
today at 9:18:42 AM  File "/app/varken/dbmanager.py", line 25, in __init__
today at 9:18:42 AM    databases = [db['name'] for db in self.influx.get_list_database()]
today at 9:18:42 AM  File "/usr/local/lib/python3.9/site-packages/influxdb/client.py", line 570, in get_list_database
today at 9:18:42 AM    return list(self.query("SHOW DATABASES").get_points())
today at 9:18:42 AM  File "/usr/local/lib/python3.9/site-packages/influxdb/client.py", line 411, in query
today at 9:18:42 AM    response = self.request(
today at 9:18:42 AM  File "/usr/local/lib/python3.9/site-packages/influxdb/client.py", line 286, in request
today at 9:18:42 AM    raise InfluxDBClientError(response.content, response.status_code)
today at 9:18:42 AMinfluxdb.exceptions.InfluxDBClientError: 401: {"code":"unauthorized","message":"Unauthorized"}
today at 9:18:44 AMContainer stopped

Using the same organization number and token to add the datasource in the InfluxDB to connect to the varken bucket connects fine.

What am I screwing up?

@samwiseg0
Copy link
Member

samwiseg0 commented Mar 19, 2022

@gcrevell Can you look at a few things?

  • Rebase from develop
  • Set this PR to draft
  • Change the merge branch to develop not master
  • I would like this to keep a single influx config section. Can you put in a check for /ping and use the v1 config/db vs v2? As well as keep a single config section? This will allow backward compatibility and keep the config clean and easy to use.

@n1nj4888
Copy link

Hi @gcrevell. Thanks for looking into this - InfluxDBv2 support would be great in Varken! Are you still planning on addressing the items samwiseg0 mentions above?

@d-mcknight
Copy link

I rebased this branch on develop but ran into trouble with the URL validator when I pass my influx URL (https://influx.domain.tld). I bypassed the validation and everything is working for me, but I'm not great with regex so I'm not sure how to fix that check.

If there's still interest I can address the other feedback here and open a PR

@YouveGotMeowxy
Copy link

YouveGotMeowxy commented Jun 23, 2023

but I'm not great with regex so I'm not sure how to fix that check.

Which line? This one?

If there's still interest I can address the other feedback here and open a PR

I'm 10000000% interested in a working influxV2 version! :D

@d-mcknight
Copy link

d-mcknight commented Jun 24, 2023

Which line? This one?

I believe it should be handled as a domain, but I just bypassed the check to get it running.

I'm 10000000% interested in a working influxV2 version! :D

I did add some automation and push my working branch to ghcr.io/d-mcknight/varken:dev to run for myself if you want to test out that image.

@YouveGotMeowxy
Copy link

YouveGotMeowxy commented Jun 25, 2023

@d-mcknight

but ran into trouble with the URL validator when I pass my influx URL (https://influx.domain.tld).

What if you remove the https:// in your config and only use influx.domain.tld (and remove your bypass).

@noraemsu
Copy link

Note I'm using the boerderij/varken:develop branch
If anyone still has issues with getting influxdb 2.0 working with varken then just follow the steps here
#159 (comment)
The only thing I did differently was using -org-id <org-id> instead of -o <your org>
And using an All Access API Token for the -t <admin token>

Like the following lines:
influx bucket list -org-id <org-id> -t <admin token>

influx v1 dbrp create --db varken --rp varken-rp --bucket-id <bucket-id> --default -org-id <org-id> -t <admin token>

influx v1 auth create --read-bucket <bucket-id> --write-bucket <bucket-id> --username varken-user -org-id <org-id> -t <admin token>

For some reason using -o <your org> arg caused some error for me, but using -org-id <org-id> fixed it.

Then in varken.ini
username is in the 3rd command above
password is prompted after the 3rd command

url = ip/url
port = 8086
ssl = false
verify_ssl = false
username = <username>
password = <password>

Hope this helps someone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants