Implement mail tracker system

* Implement mail tracking

Signed-off-by: Victor "multun" Collod <[email protected]>

* Implement task merging

* Add a mail tracker title format pattern

* Autocomplete task names

* Fix comment display

* Track notification answers

* Add a socket timeout for the mail worker

A mail worker is a long running application. And sometimes, the IMAP server
just hangs for hours for no apparent reason. imaplib doesn't enable setting
a timeout, and setting it globally seems fine.

* Only validate the merge form when submitted

* Redirect to the new form when merging

* Prettier task edit UI

* Make task merging optional

* Test mail tracking

* Update documentation for mail tracking

* Update dependencies

* Add the TODO_COMMENT_CLASSES setting

* Fix dependencies install order

* Remove debug leftovers, improve documentation

* Fail on missing from_address

.travis.yml

@@ -9,9 +9,9 @@ addons:
   postgresql: "9.6"
 
 install:
-  - "pip3 install -e . --upgrade"
-  - "pip3 install git+https://github.com/pypa/pipenv.git"
+  - "pip3 install pipenv"
   - "pipenv install --dev"
+  - "pip3 install -e . --upgrade"
 
 language: python
 python:

Pipfile

@@ -11,6 +11,8 @@ django-extensions = "*"
 factory-boy = "*"
 titlecase = "*"
 bleach = "*"
+django-autocomplete-light = "*"
+html2text = "*"
 
 [dev-packages]
 mypy = "*"

README.md

@@ -23,6 +23,7 @@ assignment application for Django, designed to be dropped into an existing site
 * jQuery (full version, not "slim", for drag/drop prioritization)
 * Bootstrap (to work with provided templates, though you can override them)
 * bleach (`pip install bleach`)
+* django-autocomplete-light (optional, required for task merging)
 
 ## Overview
 
@@ -53,7 +54,7 @@ If using your own site, be sure you have jQuery and Bootstrap wired up and worki
 django-todo pages that require it will insert additional CSS/JavaScript into page heads,
 so your project's base templates must include:
 
-```
+```jinja
 {% block extrahead %}{% endblock extrahead %}
 {% block extra_js %}{% endblock extra_js %}
 ```
@@ -107,7 +108,7 @@ If you wish to use the public ticket-filing system, first create the list into w
 
 Optional configuration options:
 
-```
+```python
 # Restrict access to ALL todo lists/views to `is_staff` users.
 # If False or unset, all users can see all views (but more granular permissions are still enforced
 # within views, such as requiring staff for adding and deleting lists).
@@ -126,13 +127,105 @@ TODO_DEFAULT_LIST_SLUG = 'tickets'
 # Defaults to "/"
 TODO_PUBLIC_SUBMIT_REDIRECT = 'dashboard'
 
+# additionnal classes the comment body should hold
+# adding "text-monospace" makes comment monospace
+TODO_COMMENT_CLASSES = []
 ```
 
 The current django-todo version number is available from the [todo package](https://github.com/shacker/django-todo/blob/master/todo/__init__.py):
 
     python -c "import todo; print(todo.__version__)"
 
 
+## Mail tracking
+
+What if you could turn django-todo into a shared mailbox?
+Django-todo includes an optional feature that allows emails sent to a
+dedicated mailbox to be pushed into todo as new tasks, and responses to
+be added as comments on that original tasks.
+
+This allows support teams to work with a fully unified email + bug
+tracking system to avoid confusion over who's seen or responded to what.
+
+To enable the feature, you need to:
+
+ - define an email backend for outgoing emails
+ - define an email backend for incoming emails
+ - start a worker, which will wait for new emails
+
+```python
+from todo.mail.producers import imap_producer
+from todo.mail.consumers import tracker_consumer
+from todo.mail.delivery import smtp_backend, console_backend
+
+# email notifications configuration
+# each task list can get its own delivery method
+TODO_MAIL_BACKENDS = {
+    # mail-queue is the name of the task list, not the worker name
+    "mail-queue": smtp_backend(
+        host="smtp.example.com",
+        port=465,
+        use_ssl=True,
+        username="[email protected]",
+        password="foobar",
+        # used as the From field when sending notifications.
+        # a username might be prepended later on
+        from_address="[email protected]",
+        # additionnal headers
+        headers={}
+    ),
+}
+
+# incoming mail worker configuration
+TODO_MAIL_TRACKERS = {
+    # configuration for worker "test_tracker"
+    "test_tracker": {
+        "producer": imap_producer(
+            host="imap.example.com",
+            username="[email protected]",
+            password="foobar",
+            # process_all=False, # by default, only unseen emails are processed
+            # preserve=False, # delete emails if False
+            # nap_duration=1, # duration of the pause between polling rounds
+            # input_folder="INBOX", # where to read emails from
+        ),
+        "consumer": tracker_consumer(
+            group="Mail Queuers",
+            task_list_slug="mail-queue",
+            priority=1,
+            task_title_format="[TEST_MAIL] {subject}",
+        )
+    }
+}
+```
+
+A mail worker can be started this way:
+
+```sh
+./manage.py mail_worker test_tracker
+```
+
+If you want to log mail events, make sure to properly configure django logging:
+
+```python
+LOGGING = {
+    'version': 1,
+    'disable_existing_loggers': False,
+    'handlers': {
+        'console': {
+            'class': 'logging.StreamHandler',
+        },
+    },
+    'loggers': {
+        '': {
+            'handlers': ['console'],
+            'level': 'DEBUG',
+            'propagate': True,
+        },
+    },
+}
+```
+
 ## Upgrade Notes
 
 django-todo 2.0 was rebuilt almost from the ground up, and included some radical changes, including model name changes. As a result, it is *not compatible* with data from django-todo 1.x. If you would like to upgrade an existing installation, try this:
@@ -229,5 +322,3 @@ ALL groups, not just the groups they "belong" to)
 **0.9.1** - Removed context_processors.py - leftover turdlet
 
 **0.9** - First release
-
-

test_settings.py

@@ -29,6 +29,8 @@
     "django.contrib.sites",
     "django.contrib.staticfiles",
     "todo",
+    "dal",
+    "dal_select2",
 )
 
 ROOT_URLCONF = "base_urls"
@@ -61,3 +63,30 @@
         },
     }
 ]
+
+LOGGING = {
+    'version': 1,
+    'disable_existing_loggers': False,
+    'handlers': {
+        'console': {
+            'class': 'logging.StreamHandler',
+        },
+    },
+    'loggers': {
+        '': {
+            'handlers': ['console'],
+            'level': 'DEBUG',
+            'propagate': True,
+        },
+        'django': {
+            'handlers': ['console'],
+            'level': 'DEBUG',
+            'propagate': True,
+        },
+        'django.request': {
+            'handlers': ['console'],
+            'level': 'DEBUG',
+            'propagate': True,
+        },
+    },
+}

todo/__init__.py

@@ -8,3 +8,5 @@
 
 __url__ = 'https://github.com/shacker/django-todo'
 __license__ = 'BSD License'
+
+from . import check

todo/check.py

@@ -0,0 +1,19 @@
+from django.core.checks import Error, register
+
+# the sole purpose of this warning is to prevent people who have
+# django-autocomplete-light installed but not configured to start the app
+@register()
+def dal_check(app_configs, **kwargs):
+    from django.conf import settings
+    from todo.features import HAS_AUTOCOMPLETE
+
+    if not HAS_AUTOCOMPLETE:
+        return
+
+    errors = []
+    missing_apps = {'dal', 'dal_select2'} - set(settings.INSTALLED_APPS)
+    for missing_app in missing_apps:
+        errors.append(
+            Error('{} needs to be in INSTALLED_APPS'.format(missing_app))
+        )
+    return errors

todo/features.py

@@ -0,0 +1,11 @@
+HAS_AUTOCOMPLETE = True
+try:
+    import dal
+except ImportError:
+    HAS_AUTOCOMPLETE = False
+
+HAS_TASK_MERGE = False
+if HAS_AUTOCOMPLETE:
+    import dal.autocomplete
+    if getattr(dal.autocomplete, 'Select2QuerySetView', None) is not None:
+        HAS_TASK_MERGE = True

todo/mail/consumers/__init__.py

@@ -0,0 +1,9 @@
+def tracker_consumer(**kwargs):
+    def tracker_factory(producer):
+        # the import needs to be delayed until call to enable
+        # using the wrapper in the django settings
+        from .tracker import tracker_consumer
+
+        return tracker_consumer(producer, **kwargs)
+
+    return tracker_factory