[Bug]: source 'City of Karlsruhe' stopped working

[alu-ka]

I Have A Problem With:

A specific source

What's Your Problem

Release 1.44.0:
Due to changes on the website the source 'City of Karlsruhe' (name: karlsruhe_de) stopped working.
I start troubleshooting and add my findings here.

Source (if relevant)

karlsruhe_de

Logs

No response

Relevant Configuration

No response

Checklist Source Error

• Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
• Checked that the website of your service provider is still working
• Tested my attributes on the service provider website (if possible)
• I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

• Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

• I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
• I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.

[alu-ka]

Looks like the API URL https://web6.karlsruhe.de/service/abfall/akal/akal.php changed to https://web6.karlsruhe.de/service/abfall/akal_2023/akal.php.

Change line with open("test.html", "w") as f: to with open("test.html", "w", encoding="utf-8") as f: ??
Change line bin_type = row.find("div", class_="col_3-2") to bin_type = row.find("div", class_="col_6-2") ??
Change line pickup_col = row.find("div", class_="col_3-3") to pickup_col = row.find("div", class_="col_6-3") ??

[alu-ka]

This code (file: custom_components\waste_collection_schedule\waste_collection_schedule\source\karlsruhe_de.py) works for me:

import datetime
import re

import requests
from bs4 import BeautifulSoup
from waste_collection_schedule import Collection  # type: ignore[attr-defined]

TITLE = "City of Karlsruhe"
DESCRIPTION = "Source for City of Karlsruhe."
URL = "https://www.karlsruhe.de/"
TEST_CASES = {
    "Östliche Rheinbrückenstraße 1": {
        "street": "Östliche Rheinbrückenstraße",
        "hnr": 1
    },
    "Habichtweg 4": {
        "street": "Habichtweg", 
        "hnr": 4
    },
    "Machstraße 5": {
        "street": "Machstraße", 
        "hnr": 5
    },
    "Bernsteinstraße 10 ladeort 1": {
        "street": "Bernsteinstraße",
        "hnr": 10,
        "ladeort": 1
    },
    "Bernsteinstraße 10 ladeort 2": {
        "street": "Bernsteinstraße",
        "hnr": 10,
        "ladeort": 2
     }
}


ICON_MAP = {
    "Restmüll": "mdi:trash-can",
    "Bioabfall": "mdi:leaf",
    "Papier": "mdi:package-variant",
    "Wertstoff": "mdi:recycle",
    "Sperrmüllabholung": "mdi:wardrobe"
}


API_URL = "https://web6.karlsruhe.de/service/abfall/akal_2023/akal.php"


class Source:
    def __init__(self, street: str, hnr: str | int, ladeort: int | None = None):
        self._street: str = street
        self._hnr: str | int = hnr
        self._ladeort: int | None = ladeort

    def fetch(self):
        args = {
            "strasse_n": self._street,
            "hausnr": self._hnr,
            "ladeort": self._ladeort,
            "anzeigen": "anzeigen"
        }

        # get json file
        r = requests.post(API_URL, data=args, params={"hausnr=": ""})
        r.raise_for_status()

        with open("test.html", "w", encoding="utf-8") as f:
            f.write(r.text)

        soup = BeautifulSoup(r.text, "html.parser")
        rows = soup.find_all("div", class_="row")
        entries = []

        for row in rows:
            column = row.find("div", class_="col_6-2")
            
            if column is None or not column.contents:
                column = row.find("div", class_="col_7-3")
                if column is None or not column.contents:
                    continue

                for content in column.contents:
                    if content.text.startswith("Sperrmüllabholung"):
                        bin_type = column.contents[0].text.strip()
                        if bin_type.endswith(":"):
                            bin_type = bin_type[:-1].strip()
                        icon = ICON_MAP.get(bin_type)  # Collection icon

                    elif content.text.startswith("Straßensperrmüll"):
                        dates = re.findall(r"\d{2}\.\d{2}\.\d{4}", content.text)
                        date = datetime.datetime.strptime(dates[0], "%d.%m.%Y").date()

                entries.append(Collection(date=date, t=bin_type, icon=icon))

            else:
                bin_type = column.contents[0].text.split(",")[0].strip()
                icon = ICON_MAP.get(bin_type)  # Collection icon

                pickup_col = row.find("div", class_="col_6-3")
                if pickup_col is None or not pickup_col.contents:
                    continue

                for date in re.findall(r"\d{2}\.\d{2}\.\d{4}", pickup_col.text):
                    date = datetime.datetime.strptime(date, "%d.%m.%Y").date()
                    entries.append(Collection(date=date, t=bin_type, icon=icon))

        return entries

[Angelx242]

The code works for me, too. I use API_URL https://web6.karlsruhe.de/service/abfall/akal/akal_2024.php instead. Result is the same.

Thanks!

[neforod]

According to the main website https://tsk.karlsruhe.de/abfuhrkalender, the API URL is now:
https://web6.karlsruhe.de/service/abfall/akal/akal_2024.php

[5ila5]

fixed in #1657
This will be part of release 1.45.0 or is available now as master (git) version (3-dot menu -> redownload -> change version to master -> fully restart HA (this will increase update cycle by a lot)

[yutamago]

@5ila5

fixed in #1657 This will be part of release 1.45.0 or is available now as master (git) version (3-dot menu -> redownload -> change version to master -> fully restart HA (this will increase update cycle by a lot)

This fix did not work for me (or it broke again?).
The request always responds with 404.

I had to do following changes to finally make it work, additionally to the fix in release 1.45.0:

• Change the API_URL to https://web4.karlsruhe.de/... instead of https://web6.karlsruhe.de/...
• AND send the request (line 68) with an additional verify=False argument

[neforod]

Seems like they changed the Api Url again. Their website now also points to web4

[5ila5]

should be fixed again in #1694

This will be part of release 1.46.0 or is available now as master (git) version (3-dot menu -> redownload -> change version to master -> fully restart HA (this will increase update cycle by a lot)