Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 2024.12.0 #3238

Merged
merged 46 commits into from
Dec 4, 2024
Merged
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
eeced44
Small refactor in deepl_api_client
david-venhoff Oct 21, 2024
b1212a9
Add support for glossaries for deepl translations
david-venhoff Oct 21, 2024
0286faf
Zammad: remove unique URL & add webhook token
svenseeberg Nov 8, 2024
b30d670
Add related issues header for issue templates (#3198)
david-venhoff Nov 15, 2024
aadc214
Merge pull request #3143 from digitalfabrik/feature/deepl-glossaries
david-venhoff Nov 15, 2024
28f6ad4
Make the number of online accesses in statistics collapsible
lunars97 Nov 15, 2024
1880e14
Merge pull request #3175 from digitalfabrik/enhancement/make-number-o…
lunars97 Nov 15, 2024
1d088e3
Fix error at bulk action in broken link list
MizukiTemma Nov 14, 2024
86a1eb7
Merge pull request #3207 from digitalfabrik/bug/broken_link_bulk_action
MizukiTemma Nov 15, 2024
1dfdbd1
Fix display of start and end of archived events
MizukiTemma Nov 14, 2024
fbb8a3f
Merge pull request #3206 from digitalfabrik/bug/archived_event_start_end
MizukiTemma Nov 15, 2024
dff3299
Ical: Add support for recurrence rules (#3205)
david-venhoff Nov 18, 2024
dd1dd2e
Revive export box of statistics
MizukiTemma Nov 18, 2024
2126d07
Merge pull request #3193 from digitalfabrik/feature/zammad-webhook-token
svenseeberg Nov 19, 2024
2177b9d
revive the title
MizukiTemma Nov 19, 2024
092dd2f
Merge pull request #3221 from digitalfabrik/bug/statistics_export_box
MizukiTemma Nov 19, 2024
5b3a0d6
Fix false positiv error in statistics in dashboard
MizukiTemma Nov 18, 2024
f1b3fdb
Add organization bulk actions
jarlhengstmengel Sep 23, 2024
0f72f67
Merge pull request #3220 from digitalfabrik/bug/statistics_in_dashboard
MizukiTemma Nov 19, 2024
d26dd9a
Merge pull request #3210 from digitalfabrik/feature/add_bulk_action_t…
jarlhengstmengel Nov 19, 2024
1813c05
Add fax icon svg
ulliholtgrave Nov 23, 2024
d8d1cbd
Merge pull request #3232 from digitalfabrik/feature/fax-icon
ulliholtgrave Nov 23, 2024
43c6117
Update tinymce to v7 (#3083)
david-venhoff Nov 23, 2024
aa87c52
Fix error with spacing in events
JoeyStk Nov 18, 2024
85d92a0
Redirect to edit after creating a new organization
lunars97 Nov 26, 2024
405b677
Fix release note of fax icon
JoeyStk Nov 26, 2024
68a5b6d
Merge pull request #3243 from digitalfabrik/bug/fix-fax-icon-release-…
JoeyStk Nov 26, 2024
110d878
Merge pull request #3230 from digitalfabrik/bug/archived_event_start_end
JoeyStk Nov 26, 2024
93e3f47
Merge pull request #3241 from digitalfabrik/bugfix/redirect-after-new…
lunars97 Nov 26, 2024
d5dd193
Add additional language colors
lunars97 Nov 26, 2024
d64f9d8
Merge pull request #3240 from digitalfabrik/feature/add-additional-la…
lunars97 Nov 27, 2024
f9f7ce2
Document how to debug with PyCharm (#3235)
david-venhoff Nov 27, 2024
3c0e0a4
Add test for organization actions
MizukiTemma Nov 25, 2024
15d888d
Merge pull request #3242 from digitalfabrik/test/organization_actions
MizukiTemma Nov 28, 2024
3ca5bc9
Set up Celery for Integreat Chat
svenseeberg Nov 23, 2024
1328f87
Merge pull request #3233 from digitalfabrik/feature/celery
svenseeberg Nov 29, 2024
4b9b6fc
Fix cancel button color
lunars97 Nov 28, 2024
094f97e
Improve link checker GUI
MizukiTemma Nov 8, 2024
8490250
Merge pull request #3191 from digitalfabrik/feature/Improve_link_chec…
MizukiTemma Nov 30, 2024
bca251a
Show related contacts in the POI form
MizukiTemma Oct 31, 2024
3f6a955
Merge pull request #3162 from digitalfabrik/feature/related_contacts_…
MizukiTemma Nov 30, 2024
e0c5a87
Merge pull request #3248 from digitalfabrik/bug/fix-cancel-button-color
lunars97 Dec 2, 2024
cd7f9f3
Multiple fixes for development setup problems caused by celery
charludo Dec 3, 2024
8af46a2
Merge pull request #3256 from digitalfabrik/fix/development-env-for-c…
charludo Dec 3, 2024
9dd2880
Remove checkbox appearence for roles who do not have edit permission
lunars97 Nov 27, 2024
d6205b6
Merge pull request #3246 from digitalfabrik/bug/fix-checkbox-appearen…
lunars97 Dec 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -19,6 +19,7 @@ jobs:
else
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .[dev-pinned,pinned]
fi
- save_cache:
4 changes: 4 additions & 0 deletions .github/ISSUE_TEMPLATE/bug-report.md
Original file line number Diff line number Diff line change
@@ -34,3 +34,7 @@ labels: 'bug'
```

</details>

### Related Issues
<!-- A collection of all issues that are related to this issue and could be useful. -->
<!-- Please also include the related integreat-app issues, if possible. -->
5 changes: 5 additions & 0 deletions .github/ISSUE_TEMPLATE/feature-request.md
Original file line number Diff line number Diff line change
@@ -28,3 +28,8 @@ As a [type of user] I want [goals or objectives] so that [values or benefits]. -

### Design Requirements
<!-- If the customization includes input from our design team, the detailed requirements will be collected here. Note: These will exist mainly in German to simplify internal communication. -->

### Related Issues
<!-- A collection of all issues that are related to this issue and could be useful. -->
<!-- Please also include the related integreat-app issues, if possible. -->
<!-- If the app issue does not exist yet, ping Toni on Mattermost to create one and then link it here. -->
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -55,6 +55,7 @@ webpack-stats.json

# Location file of redis unix socket
.redis_socket_location
dump.rdb

# Media Library
integreat_cms/media/
@@ -68,3 +69,7 @@ integreat_cms/xliff/download

# Postgres folder
.postgres

# Celery
celerybeat-schedule.db
celerybeat-schedule
34 changes: 34 additions & 0 deletions docs/src/debugging.rst
Original file line number Diff line number Diff line change
@@ -45,6 +45,40 @@ VSCodium
^^^^^^^^
Coming soon(TM)?

PyCharm (Professional)
^^^^^^^^^^^^^^^^^^^^^^

PyCharm Professional has built-in support for django projects, but it needs some configuration to work with the integreat CMS.
This is unfortunately not the case for the free edition of PyCharm. Students can get a free license for the professional version, however.

Enable Django Support
---------------------

#. Go to settings → Languages & Frameworks → Django
#. Click the ``Enable Django Support`` Checkbox
#. Set the root ``integreat_cms`` directory as the project root
#. For ``Settings`` use ``integreat_cms/core/docker_settings.py``
#. Check ``Do not use Django test runner``
#. For ``Manage script``, use ``integreat_cms/integreat-cms-cli``. If PyCharm does not let you select this script, because it does not end in .py, you can manually specify it in ``.idea/integreat-cms.iml``.

Your configuration should now look similar to this:

.. image:: images/debugging/debug-pycharm-01-django-config.png
:alt: Django configuration

Create a Run Configuration
--------------------------

#. Create a new ``Django Server`` run configuration
#. Use a different port (For example 8001) to avoid conflicts with the non-debug server at port 8000
#. At `Environment Variables`, add these: ``PYTHONUNBUFFERED=1;DJANGO_SETTINGS_MODULE=integreat_cms.core.docker_settings;INTEGREAT_CMS_DEBUG=True``

Start Debugging
---------------

#. First execute the ``./tools/run.sh`` to make sure that the database is available and all assets are compiled.
#. Once the server has started, you can start debugging by launching the run configuration

Neovim
^^^^^^

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
12 changes: 12 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
@@ -50,6 +50,13 @@
createdb -h "$INTEGREAT_CMS_DB_HOST" -p "$INTEGREAT_CMS_DB_PORT" -U "$INTEGREAT_CMS_DB_USER" "$INTEGREAT_CMS_DB_NAME"
'';
};
redis = pkgs.writeShellApplication {
name = "redis";
runtimeInputs = [ pkgs.redis ];
text = ''
redis-server --daemonize yes --unixsocket "$INTEGREAT_CMS_REDIS_SOCKET_LOCATION" --port "$INTEGREAT_CMS_REDIS_PORT" > /dev/null
'';
};
in
{
devShells.default = pkgs.mkShell {
@@ -72,6 +79,7 @@
pg-start
pg-stop
pg-reset
redis
];

LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [ pkgs.stdenv.cc.cc.lib pkgs.file ];
@@ -105,6 +113,10 @@
set_if_unset INTEGREAT_CMS_SUMM_AI_API_KEY "dummy"
set_if_unset INTEGREAT_CMS_LINKCHECK_DISABLE_LISTENERS 1

# Start redis and make it discoverable to the CMS
set_if_unset INTEGREAT_CMS_REDIS_SOCKET_LOCATION "./.redis_socket_location"
set_if_unset INTEGREAT_CMS_REDIS_PORT 6379

# Setting LD_LIBRARY_PATH can cause issues on non-NixOS systems
if ! command -v nixos-version &> /dev/null; then
unset LD_LIBRARY_PATH
63 changes: 0 additions & 63 deletions integreat_cms/api/v3/chat/chat_bot.py

This file was deleted.

47 changes: 12 additions & 35 deletions integreat_cms/api/v3/chat/user_chat.py
Original file line number Diff line number Diff line change
@@ -7,7 +7,6 @@
import json
import logging
import random
import socket
from typing import TYPE_CHECKING

from django.http import HttpResponse, JsonResponse
@@ -17,8 +16,8 @@

from ....cms.models import ABTester, AttachmentMap, Language, Region, UserChat
from ...decorators import json_response
from .chat_bot import ChatBot
from .zammad_api import ZammadChatAPI
from .utils.chat_bot import process_answer, process_user_message
from .utils.zammad_api import ZammadChatAPI

if TYPE_CHECKING:
from django.http import HttpRequest
@@ -227,15 +226,13 @@ def zammad_webhook(request: HttpRequest) -> JsonResponse:
"""
Receive webhooks from Zammad to update the latest article translation
"""
zammad_url = (
f"https://{socket.getnameinfo((request.META.get('REMOTE_ADDR'), 0), 0)[0]}"
region = get_object_or_404(
Region, zammad_webhook_token=request.GET.get("token", None)
)
region = get_object_or_404(Region, zammad_url=zammad_url)
client = ZammadChatAPI(region)
if not region.integreat_chat_enabled:
return JsonResponse({"status": "Integreat Chat disabled"})
webhook_message = json.loads(request.body)
message_text = webhook_message["article"]["body"]
zammad_chat = UserChat.objects.get(zammad_id=webhook_message["ticket"]["id"])
chat_bot = ChatBot()

actions = []
if webhook_message["article"]["internal"]:
@@ -249,34 +246,14 @@ def zammad_webhook(request: HttpRequest) -> JsonResponse:
webhook_message["article"]["created_by"]["login"]
== "tech+integreat-cms@tuerantuer.org"
):
actions.append("question translation")
client.send_message(
zammad_chat.zammad_id,
chat_bot.automatic_translation(
message_text, zammad_chat.language.slug, region.default_language.slug
),
True,
True,
actions.append("question translation queued")
process_user_message.apply_async(
args=[message_text, region.slug, webhook_message["ticket"]["id"]]
)
if answer := chat_bot.automatic_answer(
message_text, region, zammad_chat.language.slug
):
actions.append("automatic answer")
client.send_message(
zammad_chat.zammad_id,
answer,
False,
True,
)
else:
actions.append("answer translation")
client.send_message(
zammad_chat.zammad_id,
chat_bot.automatic_translation(
message_text, region.default_language.slug, zammad_chat.language.slug
),
False,
True,
actions.append("answer translation queued")
process_answer.apply_async(
args=[message_text, region.slug, webhook_message["ticket"]["id"]]
)
return JsonResponse(
{
3 changes: 3 additions & 0 deletions integreat_cms/api/v3/chat/utils/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
"""
Utils for the Integreat Chat
"""
111 changes: 111 additions & 0 deletions integreat_cms/api/v3/chat/utils/chat_bot.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
"""
Wrapper for the Chat Bot / LLM API
"""

from __future__ import annotations

import requests
from celery import shared_task
from django.conf import settings

from integreat_cms.cms.models import Region, UserChat
from integreat_cms.cms.utils.content_translation_utils import (
get_public_translation_for_link,
)

from .zammad_api import ZammadChatAPI


def format_message(response: dict) -> str:
"""
Transform JSON into readable message
"""
if "answer" not in response or not response["answer"]:
raise ValueError("Could not format message, no answer attribute in response")
if "sources" not in response or not response["sources"]:
return response["answer"]
sources = "".join(
[
f"<li><a href='{settings.WEBAPP_URL}{path}'>{title}</a></li>"
for path in response["sources"]
if (title := get_public_translation_for_link(settings.WEBAPP_URL + path))
]
)
return f"{response['answer']}\n<ul>{sources}</ul>"


def automatic_answer(message: str, region: Region, language_slug: str) -> str | None:
"""
Get automatic answer to question
"""
url = (
f"https://{settings.INTEGREAT_CHAT_BACK_END_DOMAIN}/chatanswers/extract_answer/"
)
body = {"message": message, "language": language_slug, "region": region.slug}
r = requests.post(url, json=body, timeout=120)
return format_message(r.json())


def automatic_translation(
message: str, source_language_slug: str, target_language_slug: str
) -> str:
"""
Use LLM to translate message
"""
url = f"https://{settings.INTEGREAT_CHAT_BACK_END_DOMAIN}/chatanswers/translate_message/"
body = {
"message": message,
"source_language": source_language_slug,
"target_language": target_language_slug,
}
response = requests.post(url, json=body, timeout=120).json()
if "status" in response and response["status"] == "success":
return response["translation"]
raise ValueError("Did not receive success response for translation request.")


@shared_task
def process_user_message(
message_text: str, region_slug: str, zammad_ticket_id: int
) -> None:
"""
Process the message from an Integreat App user
"""
zammad_chat = UserChat.objects.get(zammad_id=zammad_ticket_id)
region = Region.objects.get(slug=region_slug)
client = ZammadChatAPI(region)
if translation := automatic_translation(
message_text, zammad_chat.language.slug, region.default_language.slug
):
client.send_message(
zammad_chat.zammad_id,
translation,
True,
True,
)
if answer := automatic_answer(message_text, region, zammad_chat.language.slug):
client.send_message(
zammad_chat.zammad_id,
answer,
False,
True,
)


@shared_task
def process_answer(message_text: str, region_slug: str, zammad_ticket_id: int) -> None:
"""
Process automatic or counselor answers
"""
zammad_chat = UserChat.objects.get(zammad_id=zammad_ticket_id)
region = Region.objects.get(slug=region_slug)
client = ZammadChatAPI(region)
if translation := automatic_translation(
message_text, region.default_language.slug, zammad_chat.language.slug
):
client.send_message(
zammad_chat.zammad_id,
translation,
False,
True,
)
Loading