Configuring options for apache and INGRESS mode

[tobbensoft]

In the new release of HA there is an option to by UI configure a dashboard as a webpage.

To be able to get that to work with the addon I did need to add this to the security.conf in /etc/apache2 in the addon

Comment out
#Header always set X-Fram-Options "SAMEORIGIN"
#Header always set Referrer-Policy "same-origin"
Add
Header set Content-Security-Policy "frame-ancestors" 'self' https://<myhass>:8123"

Another thing is that I don't really know how the HA Companion app is accessing this so having some difficulties adding access for it.

Should this be configurable option or is there a better approach?

Thanks! :)

[alexandrecuer]

@tobbensoft : interesting. I could not manage to make emoncms work with content security policy enabled...will try your trick :-)
What do yu mean with "configure a dashboard as a webpage" ? using emoncms addon in ingress mode ?

[tobbensoft]

Last option... yes it seems to be called ingress mode :)

[tobbensoft]

I'm not that good with certificates, and I noticed you don't get a prompt for a non valid cert when browsing it in ingress mode. (Right now using unencrypted HA and emoncms ports)

[alexandrecuer]

@tobbensoft : A more flexible approach to customize apache2 should be nice, I agree

The current approach is to modify the conf files with a oneshot service at the container start, so the user can tune some things through env vars...

emoncms/emoncms_pre.sh

Line 54 in 01caed1

echo "CUSTOMIZING APACHE CONF FOR EMONCMS"

do you know addons where you inject configuration files into ?

I think a good solution would be to completely remove security.conf (I am not very happy with it, thanks for pointing the default :-) and leave the user inject it at container startup if he wants to activate security headers

With docker compose, it is quite easy, you just have to mount the file as /etc/apache2/conf.d/security.conf

Within HA, maybe it is this ? https://developers.home-assistant.io/docs/add-ons/configuration#add-on-advanced-options

for certificates management, you can use the nginx proxy manager addon, a lot of folks go for this tool

I use acme.sh because it is a very light and versatile approach

[alexandrecuer]

@tobbensoft : just made a new version, which should solve your problem

when you want custom conf, just activate it in the conf

then use the File Editor addon, with Enforce Basepath turned off

And customize security.conf which is in /addon_configs/xxxxxx_emoncms

working from my side, tell me if OK from yours

[tobbensoft]

It looks like it's working :)

[tobbensoft]

I'm wondering if it's a good idea to add ingress to this addon. It would certainly be nice if it's possible.

https://developers.home-assistant.io/docs/add-ons/presentation/#ingress

[alexandrecuer]

Nice if it is working :-)

For ingress i dont know, i tried to understand what it was about...is it related to ingress docker ?
Cf https://docs.docker.com/reference/cli/docker/network/create/#ingress

But it seems there is no extra authentification in ha ingress mode...While the first thing to do in emoncms is to create a new user... Right now, I'm not sure how to approach this, any ideas ?

[tobbensoft]

When I think a little bit more about it. I think what at least I am after is a smooth way to access the "app" for my heatpump from HA Web and Companion App.

Below describes how I do it today.

https://<hass_dns_name>:7443/app/view?name=TestName&readkey=9cf83a8be33ea5e314f10928b36d6056&mode=power&embed=1
or
http://<hass_dns_name>:8002/app/view?name=TestName&readkey=9cf83a8be33ea5e314f10928b36d6056&mode=power&embed=1

If I understand it correctly it does not need any authentication if you use the readkey from the "Share button"

Also I add "embed=1" to not get the menus.

The issue I have is that I can't get it to work through the Companion app (Hard to see what's happening there). With a webbrowser it was easy to see when there is a certificate issue or similar.

It would also be nice if the same approach works for "dashboards" in emoncms. (Does not work for me today, maybe a JS issue?)

[alexandrecuer]

Dont you want to use a NAT rule and just open the 7443 port on your box ?

[tobbensoft]

I prefer not to have any open ports at all..

[alexandrecuer]

@borpin : any ideas on this as you are using the app module more than me ?

[borpin]

Hi @alexandrecuer - no sorry. I don't use the App in this way.

re Ingress, perhaps reach out to Franck on Discord or X for some insight. He is the author/maintainer of lots of addons and it does seem to work seamlessly once setup :)

[alexandrecuer]

@tobbensoft @borpin : I've finally managed to make emoncms work in ingress mode.

it started with a lot of js errors :

The blue path was the clue...

So modified /var/www/emoncms/core.php in the running container , method get_application_path

Just changed the return $path
into

$subdir = "api/hassio_ingress/xlUON7jkXMRyPI4emhkK_4_DTMyDXT_oOHdzSKm1n0Q/";
return $path.$subdir;

after restarting the container in cli, it is better :-)

But dnow I must find how home-assistant is generating the xlUON7jkXMRyPI4emhkK_4_DTMyDXT_oOHdzSKm1n0Q key

[tobbensoft]

Cool, just say if I should do any testing

[alexandrecuer]

well implementation is not ready online and new image is not build :-)

There is a possibility to test through the development environment... do you use visual studio code ?

[tobbensoft]

Yes I do

[alexandrecuer]

Just created a dev branch, you can pull it, open in visual studio code, and start home assistant to create the dev container

If the build through homeassistant fails, you can rebuild the image manually, once connected to the dev container in CLI :

cd /workspaces/emoncms
docker build -t local/amd64-addon-emoncms:alpine3.19_emoncms11.4.11_1 --build-arg="BUILD_FROM=alpine:3.19" --build-arg="TARGETPLATFORM=linux/amd64" --build-arg="PHP_VER=82" --build-arg="PHP_CONF=/etc/php82/conf.d" .

I am not very happy with the python file modifying the core.php of emoncms ;-(
I find it a bit cumbersome to have to run a 40-lines script to change a single path in a single file

[tobbensoft]

@alexandrecuer I'm not proficient with everything around setting up development for HA (yet). I will need some time to look into it.

If there is something else in the meantime I can do, just ask :)

[alexandrecuer]

@tobbensoft : no problem, take your time. same for me, all this is very new, this addon is my first one :-)

until now, i was not using visual studio code, i've tried it this week while testing ingress :-)
I've followed this tutorial which is very simple and very clear :
https://developers.home-assistant.io/docs/add-ons/testing

[alexandrecuer]

I am not very happy with the python file modifying the core.php of emoncms ;-( I find it a bit cumbersome to have to run a 40-lines script to change a single path in a single file

A more elegant solution is to modify emoncms core.php as proposed here : emoncms/emoncms#1878

[alexandrecuer]

@tobbensoft : forget what i said with the dev branch and the devcontainer....
You can test the classic way and ingress should be working fine...
If you have datas, make a backup before with the emoncms backup tool...
Are you familiar with the backup tool of homeassistant ? I had trouble twice while using it to restore previous versions of the addon...permissions for mariadb are not restored correctly. So i used chown in the running container, it worked the first time but not the second....i think i have something to add when homeassistant launch the backup...

opening new issue for that : #15

[tobbensoft]

@alexandrecuer not sure what you mean by testing the classic way?

I previously had issues with mariadb not restoring correctly but I think it might be another problem.

[alexandrecuer]

@alexandrecuer not sure what you mean by testing the classic way?

Just update :-) as I finally build the new image which is alpine3.19_emoncms11.5.0

[tobbensoft]

Yes it died..
2024-04-16 12:07:36 0 [Warning] Can't create test file '/data/emoncms/mysql/60ed1c84-emoncms.lower-test' (Errcode: 13 "Permission denied")
�mysqld: Can't change dir to '/data/emoncms/mysql/' (Errcode: 13 "Permission denied")
2024-04-16 12:07:36 0 [ERROR] Aborting

Is there a manual fix? Backup didnt work either

[alexandrecuer]

Shit, yes there is a manual fix, I am sorry for that. I had the same bug...

Just connect to the container.

Then :

nano /etc/s6-overlay/s6-rc.d/mariadb/run

adjust the run file so it looks like that :

#!/command/execlineb -P
mysqld --user=root

An also modify the apache2 run :

nano /etc/s6-overlay/s6-rc.d/apache2/run

so that it looks like :

#!/command/execlineb -P
/command/foreground { rm -f /var/run/apache2/httpd.pid } /usr/sbin/httpd -D FOREGROUND

Then modify the emoncms_pre.sh file

nano /opt/openenergymonitor/emoncms_pre.sh

so that it gives appropriates right to the data folder :

if ! [ -d "$EMONCMS_DATADIR" ]; then
    echo "Creating timeseries folders"
    mkdir -p "$EMONCMS_DATADIR"
    mkdir -p "$EMONCMS_DATADIR/backup"
    mkdir -p "$EMONCMS_DATADIR/backup/uploads"
    for i in $TS; do mkdir -p "$EMONCMS_DATADIR/$i"; done
else
    echo "Using existing timeseries"
fi

chown -R "$DAEMON" "$EMONCMS_DATADIR"

then restart the container in cli, not with ha

the other option, if you made a backup with the emoncms backup tool, to uninstall the addon and to reinstall a new one and then to restore the archive....

[alexandrecuer]

I finally managed to upload a working image to the docker hub....
So you can connect to your home assistant installation, repull the image :

docker pull alexjunk/emoncms:alpine3.19_emoncms11.5.0

then restart the addon with ha
and I think ingress is not compatible with

Header always set X-Frame-Options "SAMEORIGIN"
Header always set Referrer-Policy "same-origin"

so I deactivated the CUSTOM_APACHE_CONF

and the app started working in ingress mode

and also through port 443

Again, again, my apologies for the disorder, and hope it will normalize with the new image

[tobbensoft]

I did manage to get the sql error to go away, but all my inputs are dead now.. not sure what is wrong...

How do I do a sql backup inside of the addon?

Getting this error in HA log but nothing in addon log

Error saving data '{sensor.eb101_ep14_bt12_condensor_out_44058:37.1,sensor.eb101_ep14_bt3_return_temp_44055:32.7,sensor.bt1_outdoor_temperature_40004:10.0,sensor.master_entrance_motion_sensor_temperature:21.6,sensor.calc_supply_s1_43009:36.4,sensor.bf1_ep14_flow_40072:21.4,sensor.eb101_ep14_compressor_state_44457:1.0,number.degree_minutes_16_bit_43005:-139.9,sensor.eb101_speed_charge_pump_44396:49.0,sensor.supply_pump_speed_ep14_43437:53.0,sensor.eb101_ep14_actual_cpr_frequency_outdoor_unit_44701:32.5}' to 'http://172.16.2.10:8002/input/post.json'

[alexandrecuer]

Normally the inputs should come back alone....

to backup through emoncms, you have to use the backup module >Export > create backup

[alexandrecuer]

Are you using the component integration to post data from home assistant to emoncms, that's it ?
https://github.com/home-assistant/core/tree/dev/homeassistant/components/emoncms

[tobbensoft]

Found the error,, I have still not fixed my certificates so my http post url did not work...

Pressed show disabled ports and added 8002 -> 80 as I had before...

[alexandrecuer]

To what I see, he expect something on 8002, but as it is ingress, there is nothing on 8002, but you can reopen the port I think through show disabled ports

[tobbensoft]

Trying to access in the ingress now,, I get the login prompt but cannot authenticate...

Works when I do it outside the ingress.

http://hass.xx.xxx.xx:8123/api/hassio_ingress/slSgfFuHJptC629w_t-yPvPyoKlEY4Qi2XJe2RBD7Oo/broken

[alexandrecuer]

do you have CUSTOM_APACHE_CONF activated with the content security policy as chown in your first post ?

[tobbensoft]

do you have CUSTOM_APACHE_CONF activated with the content security policy as chown in your first post ?

No,, I did a reinstall of the new version with a drop in of a backup (Btw, that gives an error on the restart)

Did not touch anything else yet.

[alexandrecuer]

you mean a drop in of a backup (targ.gz) created by the emoncms backup tool ?

I could only put online the new docker working image only on 3:30 pm Paris time, if you pulled before, maybe the image is still incorrectly running mariadb with the mysql user

which kind of error on restart ?

when I first tried to open through ingress, I had 303 for a while, then I showed the interface after a while, I am running ha on a raspberry PI3, not very powerfull

I have core 2024.2.1

and os 11.5

[alexandrecuer]

Trying to access in the ingress now,, I get the login prompt but cannot authenticate...

Works when I do it outside the ingress.

http://hass.xx.xxx.xx:8123/api/hassio_ingress/slSgfFuHJptC629w_t-yPvPyoKlEY4Qi2XJe2RBD7Oo/broken

Is http://hass.xx.xxx.xx:8123/api/hassio_ingress/slSgfFuHJptC629w_t-yPvPyoKlEY4Qi2XJe2RBD7Oo/php-info.php showing phpinfo as expected ?

I dont understand where the "broken" comes from....you have this while inpecting when accessing via ingress, or outside ingress ?

[alexandrecuer]

Maybe I forgot something, only allowing connections from 172.30.32.2 in apache virtual host on port 80

cf https://developers.home-assistant.io/docs/add-ons/presentation#ingress

it means changing Require all granted by Require ip 172.30.32.2 in emoncms_pre.sh, line 90

[tobbensoft]

http://hass.xx.xxx.xx:8123/api/hassio_ingress/slSgfFuHJptC629w_t-yPvPyoKlEY4Qi2XJe2RBD7Oo/php-info.php

No, gives a 404

Tried change in emoncms_pre.sh and local restart,, same result

But I found this,, it might be https related. (My certificate is note valid)

[alexandrecuer]

Maybe I am misunderstanding, but other addons dont have any authentication process, that's why I thought it was not possible to make emoncms work with ingress

if you run ha in https while the ingress runs http on 80, the browser takes it as a cross site request.

what happens if you change the emoncms rememberme_model.php from Strict to None on line 64

https://github.com/emoncms/emoncms/blob/b9d044efca9ca821535c150e5f1e1523ace884ac/Modules/user/rememberme_model.php#L64

also on user_model.php on line158

https://github.com/emoncms/emoncms/blob/ca1f5c3165fc51cfda4451c808c68ec877009523/Modules/user/user_model.php#L158

and restart the container manually ?

before you should probably forget about the site as explained here : https://www.ssl2buy.com/wiki/how-to-clear-hsts-settings-on-chrome-firefox-and-ie-browsers or something else which can remove the emoncms session cookie

Another solution can be not to use the port 80 but the port 443 as the ingress_port in config.yaml so it will not be considered as a cross site cookie ?

[tobbensoft]

I will try later, but one thing that would work that not require authentication is using the readkey in the url.. (At least for viewing graphs)

Do you think that approach will work?

[alexandrecuer]

For the key, I dont know, maybe
Just one thing : do you run HA only in https/ssl ? Do you use Nabu Casa ?
cf https://www.home-assistant.io/integrations/http#ssl_profile
or
cf https://scotthelme.co.uk/setting-up-https-for-home-assistant/
I want to reproduce the bug and as everything is running fine from my side, I need to know what i have to change....

[tobbensoft]

Locally only http and remote with nabucasa

I probably just need to sort out the certificates, but need to have some spare time for that

[tobbensoft]

Hi,, I have now setup my HA with valid certs and now it works from the ingress as supposed. (Nginx / Nginx Proxy Manager )

But if I want to access emoncms full screen directly (port 7443) with a valid cert,, how should I approach that?

[alexandrecuer]

I would say it is good to keep things simple and not to have complex configurations cause they are hard to maintain, so I think it is better to use NPM as a reverse proxy for HA and to access to emoncms via ingress (and not to use the certs within the addon)

Anywway if you want to do that, you have to map the certs created by NPM in the addon and to fill in the CNAME correctly

then you have to create a NAT rule on your router

Your emoncms will be accessible from outside through : https://cname:7443
this address https://cname:7443 may not work from within your lan if your router doesn't support NAT loopback, you need to add an entry to the hosts file of each machine you want to use for browsing...