oom when using native influxDB ssl

[pheetr]

Hi, firstly thanks a lot for this sketch. It's just what I've been looking for... or it will be once I get it working. :)

I initially tried setting things up for InfluxDB. Unfortunately I ran into a connection issue, probably due to my setup having a self-signed cert atm. My attempt to skip certificate validation via influxClient.setInsecure() also failed for some reason.

As an alternative I figured I'd set it up via MQTT, but would still like to get the data into InfluxDB as well.
Do you know what MQTT topic the messages get posted to? I would need the topic to be able to leverage the Telegraf mqtt_consumer plugin.

[ChaseCares]

Hey, sorry you're having issues. Just to confirm your using InfluxDB 2.0 and have you tried it with just an IP rather than a URL with HTTPS? If you want to attach any errors or logs I could take a look and see if I can figure out exactly where the issue is.

To answer your specific question about MQTT, the topic has this structure:
homeassistant/sensor/<Mac address>/<sensor type>/<config or state>

Examples

For the PM2_5 state
homeassistant/sensor/c45bbe60f5fb/PM2_5/state

Tor the temperature config
homeassistant/sensor/c45bbe60f5fb/Temperature/config

[pheetr]

Thanks, appreciate the quick response and the MQTT details. :)

I am indeed using InfluxDB 2.0.
When I try the IP:port without the preceding "https://" I get the error message InfluxDB write failed: Invalid URL scheme

In my InfluxDB attempt described earlier, the only modification to the sketch (other than config) is that I added influxClient.setInsecure(); to the setup part just before timeSync(TZ_INFO, NTP_SERVER);.
When I run this I see the following error message in my InfluxDB container logs: http: TLS handshake error from airgradient_ip:60986: EOF" log_id=0YRt_raG000 service=http (real IP substituted)
On the serial monitor I see InfluxDB write failed: connection failed

What is a bit weird is that timesync seems to be failing as well. At least, I'm guessing I'm not supposed to see a 1970 date as a result in the console: Synchronized time: Thu Jan 1 01:04:45 1970

I'm afraid I'm new to Arduino, so I'm not entirely sure how to get more logs than what I'm able to see in the serial monitor.

[ChaseCares]

You're welcome, I'm happy to help. I should have been more specific about the address to use, you still need http:// here's an example #define INFLUXDB_URL "http://ip:port".

Also the time issue is almost certainly why the authentication isn't succeeding, so I think we should focus on getting NTP working. First I would back up or delete the air quality monitors sketch you have now, and download a fresh copy from github, just so were both on the same page as far as what the program is doing. Are you using the default NTP server? If your device is having trouble connecting to that server you might try a different one https://timetoolsltd.com/information/public-ntp-server/

Try the influxdb URL I mentioned above, then try a different NTP pool, and let me know how it goes.

[pheetr]

Ah, ok, I was already using the "https://ip:port" format.

I ended up setting up a local ntp server via a Chrony docker, so the ntp sync is now successful. I figured the time aspect wouldn't come into play given that I was going for no cert validation, but at least that's one less thing to worry about.

I replaced the sketch with an unmodified copy while leaving the config file in place.
This results in the same error message as with the version I was using before, which had the setInsecure() additional code. Thus I'm guessing that it doesn't even get that far in the connection perhaps.
After enabling additional logging, the logs seem to point to some out of memory issue (sensitive parts like ip, port, org, bucket redacted):

Running task: WriteToDatabase
[HTTP-Client][begin] url: https://ip:port/api/v2/write?org=org&bucket=bucket
[HTTP-Client][begin] host: ip port: port url: /api/v2/write?org=org&bucket=bucket
[HTTP-Client][sendRequest] type: 'POST' redirCount: 0
[hostByName] Host: ip is a IP!
:close
:ur 1
:dsrcv 0
:del
:ref 1
BSSL:_connectSSL: start connection
:oom(16709)@abi.cpp:72
:oom(16709)@abi.cpp:72

_iobuf_in:       0x00000000
_iobuf_out:      0x3fff9c34
_iobuf_in_size:  16709
_iobuf_out_size: 597
BSSL:_connectSSL: OOM error
[HTTP-Client] failed connect to ip:port
[HTTP-Client][returnError] error(-1): connection failed
[HTTP-Client][end] tcp is closed
InfluxDB write failed: connection failed

When looking for BSSL:_connectSSL: OOM error I've only found a few references, notably here and here, both of which refer to the ability to decrease BearSSL buffer sizes.
Accordingly, I tried setting influxClient.setBufferSizes(512, 512); in the sketch, but that throws an error during compilation: 'class InfluxDBClient' has no member named 'setBufferSizes'; did you mean 'uint8_t InfluxDBClient::_writeBufferSize'? (not accessible from this context)
I suppose this option may not be implemented in the InfluxDB library or might need to be called differently. Buffer sizes seem to be set to 1024 in the library.

[ChaseCares]

Will you please try with http:// not https://. So replace this line #define INFLUXDB_URL "http://ip:port" and just replace your IP and port number. If this doesn't work show me the same log you posted. I think we can fix this, just having trouble reproducing it. Let me know how it goes.

[pheetr]

Sure. Then it's complaining about the request not being https:

Running task: WriteToDatabase
[HTTP-Client][begin] url: http://ip:port/api/v2/write?org=org&bucket=bucket
[HTTP-Client][begin] host: ip port: port url: /api/v2/write?org=org&bucket=bucket
[HTTP-Client][sendRequest] type: 'POST' redirCount: 0
[hostByName] Host: ip is a IP!
:ref 1
[HTTP-Client] connected to ip:port
[String] Reallocating large String(137 -> 157 bytes) 'Authorizat ... xt/plain
'
[HTTP-Client] sending request header
-----
POST /api/v2/write?org=org&bucket=bucket HTTP/1.1
Host: ip:port
User-Agent: influxdb-client-arduino/3.9.0 (ESP8266 3.0.2)
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0
Connection: close
Authorization: Token token
Content-Type: text/plain
Content-Length: 62

-----
:wr 361 0
:wrc 361 361 0
:wr 62 0
:wrc 62 62 0
:ack 361
:rn 76
:rcl pb=0x3fff6774 sz=76
[HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.0 400 Bad Request
'
[HTTP-Client][handleHeaderResponse] RX: '
'
[HTTP-Client][handleHeade:er -14 0x00000000
rResponse] code: 400
:c0 48, 76
[HTTP-Client][end] tcp is closed
[HTTP-Client][end] tcp is closed
InfluxDB write failed: Client sent an HTTP request to an HTTPS server.

[ChaseCares]

Are you using a proxy or did you enable SSL natively in InfluxDB? I tried my set up with a reverse proxy and it works with https. If you're using a reverse proxy could you point Arduino directly at InfluxDB so you can use a http connection. If not you can try and add influxClient.setInsecure(); on line 233 (just inside the void setup;), this compiles and connects to the database.

[pheetr]

SSL is enabled natively in InfluxDB. There is no reverse proxy in the mix.
Adding influxClient.setInsecure(); on line 233 unfortunately leads to the same OOM error as before.

[ChaseCares]

okay I'm able to re-create your issue, it's not an issue with self signed certificates. I will try to find a solution, give me a bit to tinker with it.

[pheetr]

Cool, thanks!

[ChaseCares]

So I don't have an satisfying answer for you. The problem is upstream, and I don't think I can provide a solution. As far as I can tell the root of the issue is that the influx http server does not respond to probeMaxFragmentLength request, thus allowing for the smaller buffer size. This is probably something that will be updated in the future.

To collaborate my findings I found this excerpt from InfluxDB-Client-for-Arduino which states: The standalone InfluxDB OSS server doesn't support MFLN, InflluxDB Cloud yes. To leverage MFLN for standalone OSS, a reverse proxy needs to be used.

Potential solutions, disable SSL in influxdb and just use HTTP requests or add a reverse proxy (haproxy or nginx), nginx being the recommended method by influx. These three methods are tested and working. let me know if you have any other issues

[pheetr]

Hmm, that's unfortunate. Thanks for looking into this. Is there a corresponding InfluxDB Issue you could share?
I've been planning on setting up a reverse proxy eventually (likely Traefik). I may have to accelerate those plans. :)

[ChaseCares]

You're welcome! I couldn't find any official issues. From what I have found their official stance is to use a reverse proxy. Once set up I think a proxy is a very good solution, good luck.

[pheetr]

As a last attempt I'll see if folks at the InfluxDB library repo have perhaps encountered this before and what their position is on it. There is meant to be support both for ESP8266 boards and secure connections with InfluxDB, so I'm guessing this must have come up, unless it's new or environmental somehow.
Update: never mind, I just saw your update to your earlier comment with the excerpt from the InfluxDB client library repo.