Add support for more generic notification

This is a start to have different kinds of notifications. It is
possible to be notified via e-mail, matrix or gotify. It also extends
the current "periodic query" for due items by allowing notification
over different channels. A "generic periodic query" variant is added
as well.

website/site/content/docs/features/_index.md

@@ -47,8 +47,9 @@ description = "A list of features and limitations."
 - [Share](@/docs/webapp/share.md) documents via cryptic public links
   (optionally protected by a password)
 - [Send documents via e-mail](@/docs/webapp/mailitem.md)
-- [E-Mail notification](@/docs/webapp/notifydueitems.md) for documents
-  with due dates
+- [Notifications](@/docs/webapp/notification.md) for documents with
+  due dates or events via E-Mail, [Matrix](https://matrix.org) or
+  [Gotify](https://gotify.net)
 - [Read your mailboxes](@/docs/webapp/scanmailbox.md) via IMAP to
   import mails into docspell
 - [Edit multiple items](@/docs/webapp/multiedit.md) at once

website/site/content/docs/jsonminiquery/_index.md

@@ -0,0 +1,258 @@
++++
+title = "JSON (mini) query"
+[extra]
+mktoc = true
+hidden = true
++++
+
+A "JSON mini query" is a simple expression that evaluates to `true` or
+`false` for any given JSON value.
+
+It is used in docspell to filter notification events.
+
+The examples shown here assume the JSON at [the end of this
+page](#sample-json).
+
+# Structure
+
+A json mini query is a sequence of "segments" where each one selects
+contents from the "current result". The current result is always a
+list of JSON values and starts out with a single element list
+containing the given root JSON document.
+
+When the expression is applied, it is read from left to right where
+each segment is evaluated against the JSON. All results are always
+aggregated into a single list which is handed to the next segment as
+input. Each segment of the expression is always applied to every
+element in the list.
+
+The expression evaluates to `true` if the final result is non-empty
+and to `false` otherwise. So the actual values selected at the and are
+not really of interest. It only matters if the last result is the
+empty list or not.
+
+There are the following possible segments:
+
+- selecting fields: `fieldname[,fieldname2,…]`
+- selecting specific array elements: `(0[,1,…])`
+- filter list of nodes against given values: `=some-value`,
+  `!some-value`
+- combining the above (sequence, `&` and `|`)
+
+
+## Field selection
+
+The simplest segment is just a field name. It looks up the field name
+in the current JSON result and replaces each element in the result
+with the value at that field.
+
+```
+query: a
+current result: [{"a":1,"b":2}, {"a":5, "b":2}]
+next result: [1, 5]
+```
+
+You can select multiple fields at once by separating names by comma.
+This simply combines all results into a list:
+
+```
+query: a,b
+current result: [{"a":1,"b":2}, {"a":5, "b":2}]
+next result: [1, 2, 5, 2]
+```
+
+You can use dot-notation combining several field selection segments to
+select elements deeper in the JSON:
+
+```
+query: a.b.x,y
+current result:
+  [{"a": {"b": {"x": 1, "y": 2}}, "v": 0},
+   {"a": {"b": {"y": 9, "b": 2}}, "z": 0}]
+next result: [1, 2, 9]
+```
+
+
+## Array selection
+
+When looking at an array, you can select specific elements by their
+indexes. To distinguish it from field selection, you must surround it
+by parens:
+
+```
+query: (0,2)
+current result: [[1,2,3,4]]
+next result: [1,3]
+```
+
+If you combine field selection and array selection, keep in mind that
+a previous field selection combines both arrays into one before array
+selection is used!
+
+```
+query: a(0,2)
+current result: [{"a": [10,9,8,7]}, {"a": [1,2,3,4]}]
+next result: [10,8]
+```
+
+
+## Matching Values
+
+You can filter elements of the current result based on their value.
+This only works for primitive elements.
+
+- equals (case insensitive): `=`
+- not equals (case insensitive): `!`
+
+Values can be given either as a simple string or, should it contain
+whitespace or brackets/parens, you need to enclose it either in single
+or double quotes. If you want to check for `null` use the special
+`*null*` value.
+
+The match will be applied to all elements in the current result and
+filters out those that don't match.
+
+```
+query: =blue
+current result: ["blue", "green", "red"]
+next result: ["blue"]
+```
+
+```
+query: color=blue
+current result:
+    [{"color": "blue", "count": 2},
+     {"color": "blue", "count": 1},
+     {"color": "red", "count": 3}]
+next result:["blue", "blue"]
+```
+
+## Combining
+
+The above expressions can be combined by writing one after the other,
+sequencing them. This has been shown in some examples above. The next
+segment will be applied to the result of the previous segment. When
+sequencing field names they must be separated by a dot.
+
+Another form is to combine several expressions using `&` or `|`. The
+results of all sub expressions will be concatenated into a single
+list. When using `&`, results are only concatenated if all lists are
+not empty; otherwise the result is the empty list.
+
+This example filters all `count` equal to `6` and all `name` equal to
+`max`. Since there are now `count`s with value `6`, the final result
+is empty.
+
+```
+query: [count=6 & name=max]
+current result:
+  [{"name":"max", "count":4},
+   {"name":"me", "count": 3},
+   {"name":"max", "count": 3}
+  ]
+next result: []
+```
+
+Using `|` for combining lets all the `max` values through:
+
+```
+query: [count=6 & name=max]
+current result:
+  [{"name":"max", "count":4},
+   {"name":"me", "count": 3},
+   {"name":"max", "count": 3}
+  ]
+next result: ["max", "max"]
+```
+
+## Example walkthrough
+
+Let's look at an example:
+
+```
+content.added,removed[name=Invoice | category=expense]
+```
+
+Starting out with the root JSON document, a list is created containing
+it as the only element:
+
+```
+( {"eventType": "TagsChanged", "content: {…}, …} )
+```
+
+Then the field `content` is selected. This changes the list to contain
+this sub-document:
+
+```
+( {"account": "demo", "added":[…], "removed":[…], …} )
+```
+
+Then two fields are selected. They both select arrays. Both results
+are combined into a single list and arrays are flattened. So the
+result after `content.added,removed` looks like this:
+
+```
+( {"id":"Fy4…",name="…",category="…"}, {"id":"7zae…",…}, {"id":"GbXg…",…} )
+```
+
+At last, the remaining elements are filtered. It resolve all `name`
+fields and keeps only `invoice` values. It also resolves `category`
+and keeps only `expense` values. Both lists are then concatenated into
+one. The final result is then `["Invoice", "expense"]`, which matches
+the sample json data below.
+
+
+# Sample JSON
+
+Some examples assume the following JSON:
+
+```json
+{
+  "eventType": "TagsChanged",
+  "account": {
+    "collective": "demo",
+    "user": "demo",
+    "login": "demo"
+  },
+  "content": {
+    "account": "demo",
+    "items": [
+      {
+        "id": "4PvMM4m7Fwj-FsPRGxYt9zZ-uUzi35S2rEX-usyDEVyheR8",
+        "name": "MapleSirupLtd_202331.pdf",
+        "dateMillis": 1633557740733,
+        "date": "2021-10-06",
+        "direction": "incoming",
+        "state": "confirmed",
+        "dueDateMillis": 1639173740733,
+        "dueDate": "2021-12-10",
+        "source": "webapp",
+        "overDue": false,
+        "dueIn": "in 3 days",
+        "corrOrg": "Acme AG",
+        "notes": null
+      }
+    ],
+    "added": [
+      {
+        "id": "Fy4VC6hQwcL-oynrHaJg47D-Q5RiQyB5PQP-N5cFJ368c4N",
+        "name": "Invoice",
+        "category": "doctype"
+      },
+      {
+        "id": "7zaeU6pqVym-6Je3Q36XNG2-ZdBTFSVwNjc-pJRXciTMP3B",
+        "name": "Grocery",
+        "category": "expense"
+      }
+    ],
+    "removed": [
+      {
+        "id": "GbXgszdjBt4-zrzuLHoUx7N-RMFatC8CyWt-5dsBCvxaEuW",
+        "name": "Receipt",
+        "category": "doctype"
+      }
+    ],
+    "itemUrl": "http://localhost:7880/app/item"
+  }
+}
+```

website/site/content/docs/webapp/notification.md

@@ -0,0 +1,205 @@
++++
+title = "Notifications"
+weight = 60
+[extra]
+mktoc = true
++++
+
+Docspell can notify on specific events and it can run queries
+periodically and notify about the result.
+
+Notification can be configured per user: go to *User profile →
+Notifications*. You can choose between webhooks and two periodic
+queries. Webhooks are HTTP requests that are immediatly executed when
+some event happens in Docspell, for example a tag was added to an
+item. Periodic queries can be used for running queries regularly and
+get the result sent as message.
+
+Both require to first select a channel, for how the message should be
+sent.
+
+{{ imgnormal(file="notification-01.png", width="250px") }}
+
+# Channels
+
+Channels are means to deliver a message. Currently, docspell supports
+these channels:
+
+- E-Mail; you need to define SMTP settings
+  [here](@/docs/webapp/emailsettings.md#smtp-settings)
+- HTTP Requests: another option is to execute a generic HTTP request
+  with all event data in a JSON body.
+- [Matrix](https://matrix.org)
+- [Gotify](https://gotify.net)
+
+## Matrix
+
+Matrix is an open network for secure and decentralized communication.
+It relies on open standards and can be self-hosted.
+
+To receive messages into your matrix room, you need to give the room
+id, your access key and the url of your home server, for example
+`https://matrix.org`.
+
+You can find the room id in your room settings under "Advanced" in
+Element. The access key is in your user settings under tab "Help &
+About" in Element.
+
+{{ figure(file="notification-02.png") }}
+
+## Gotify
+
+Gotify is a simple application for receiving messages to be notified
+on several clients via websockets. It is great for connecting
+applications to your devices.
+
+It requires only your gotify url and the application secret.
+
+{{ figure(file="notification-03.png") }}
+
+
+## E-Mail
+
+E-Mails are sent using one of your configured [SMTP
+connections](@/docs/webapp/emailsettings.md#smtp-settings).
+
+The part `docspell.joex.send-mail.list-id` in joex' configuration file
+can be used to add a `List-Id` mail header to every notification mail.
+
+## HTTP Request
+
+The most generic form is the channel *HTTP Request*. This just sends a
+POST request to a configured endpoint. The requests contains a JSON
+body with the event details.
+
+# Webhooks
+
+Webhooks are http requests that are generated on specific events in
+Docspell.
+
+## Events
+
+You need to choose which events you are interested in.
+
+{{ figure(file="notification-04.png") }}
+
+You can do so by selecting multiple event types or by clicking the
+*Notify on all events* checkbox.
+
+Each event type generates different event data. This data is prepared
+as a JSON structure which is either send directly or used to generate
+a message from it.
+
+Additionally, it is possible to filter the events using an expression
+that is applied to the event data JSON structure.
+
+## Testing
+
+The webhook form allows you to look at some sample events. These
+events are generated from random data and show how the message would
+look like (roughly, because it obviously depends on how the channel
+displays it).
+
+You can also click the *Test Delivery* button. This generates a sample
+event of the first of the selected event (or some chosen one, if
+*Notify on all events* is active) and sends it via the current
+channel.
+
+## JSON filter expression
+
+This filter allows to further constrain the events that trigger a
+notification. For example, it can be used to be notified only when
+tags of a specific category are changed.
+
+It works by selecting paths into the JSON structure of the event. Thus
+you need to know this structure, in order to define this expression. A
+good way is to look at the sample events for the *HTTP Request*
+channel. These show the exact JSON structure that this filter is
+applied to (that applies to every channel).
+
+{{ figure(file="notification-05.png") }}
+
+As an example: Choose the event *TagsChanged* and this filter
+expression: `content.added,removed.category=document_type` to be
+notified whenever a tag is added or removed whose category is
+`document_type`.
+
+Please see [this page](@/docs/jsonminiquery/_index.md) for details
+about it.
+
+{% infobubble(mode="info", title="⚠ Please note") %}
+
+The webhook feature is still experimental. It starts out with only a
+few events to choose from and the JSON structure of events might
+change in next versions.
+
+{% end %}
+
+# Periodic Queries
+
+These are [background tasks](@/docs/joex/_index.md) that execute a
+defined query. If the query yields a non-empty result, the result is
+converted into a message and sent to the specified target system.
+
+For example, this can be used to regularly inform about due items, all
+items tagged *Todo* etc.
+
+## Due Items Task
+
+
+The settings allow to customize the query for searching items. You can
+choose to only include items that have one or more tags (these are
+`and`-ed, so all tags must exist on the item). You can also provide
+tags that must *not* appear on an item (these tags are `or`-ed, so
+only one such tag is enough ot exclude an item). A common use-case
+would be to manually tag an item with *Done* once there is nothing
+more to do. Then these items can be excluded from the search. The
+somewhat inverse use-case is to always tag items with a *Todo* tag and
+remove it once completed.
+
+The *Remind Days* field species the number of days the due date may be
+in the future. Each time the task executes, it searches for items with
+a due date lower than `today + remindDays`.
+
+If you don't restrict the search using tags, then all items with a due
+date lower than this value are selected. Since items are (usually) not
+deleted, this only makes sense, if you remove the due date once you
+are done with an item.
+
+The last option is to check *cap overdue items*, which uses the value
+in *Remind Days* to further restrict the due date of an item: only
+those with a due date *greater than* `today - remindDays` are
+selected. In other words, only items with an overdue time of *at most*
+*Remind Days* are included.
+
+## Generic Query Task
+
+This is the generic version of the *Due Items Task*. Instead of
+selecting teh items via form elements, you can define a custom
+[query](@/docs/query/_index.md).
+
+## Schedule
+
+Both tasks have a *Schedule* field to specify the periodicity of the
+task. The syntax is similiar to a date-time string, like `2019-09-15
+12:32`, where each part is a pattern to also match multple values. The
+ui tries to help a little by displaying the next two date-times this
+task would execute. A more in depth help is available
+[here](https://github.com/eikek/calev#what-are-calendar-events). For
+example, to execute the task every monday at noon, you would write:
+`Mon *-*-* 12:00`. A date-time part can match all values (`*`), a list
+of values (e.g. `1,5,12,19`) or a range (e.g. `1..9`). Long lists may
+be written in a shorter way using a repetition value. It is written
+like this: `1/7` which is the same as a list with `1` and all
+multiples of `7` added to it. In other words, it matches `1`, `1+7`,
+`1+7+7`, `1+7+7+7` and so on.
+
+You can click on *Start Once* to run this task right now, without
+saving the form to the database ("right now" means it is picked up by
+a free job executor).
+
+If you click *Submit* these settings are saved and the task runs
+periodically.
+
+You can see the task executing at the [processing
+page](@/docs/webapp/processing.md).

website/site/content/docs/webapp/notifydueitems.md

@@ -1,6 +1,8 @@
 +++
 title = "Notify about due items"
 weight = 60
+draft = true
+render = false
 [extra]
 mktoc = true
 +++

website/site/templates/overview.html

@@ -36,6 +36,7 @@
                     {% for section in section.subsections %}
 
                     {% set sub = get_section(path=section) %}
+                    {% if not sub.extra.hidden %}
 
                     <div class="column is-one-third-widescreen is-half-tablet">
                         <div class="card full-height">
@@ -55,6 +56,8 @@
                             </a>
                         </div>
                     </div>
+                    {% endif %}
+
                     {% endfor %}
                 </div>
             </div>