Update docs and screenshots

website/site/content/docs/api/_index.md

@@ -3,91 +3,9 @@ title = "Api"
 description = "Contains documentation about the REST API."
 weight = 70
 insert_anchor_links = "right"
-[extra]
+template = "pages.html"
-mktoc = true
+sort_by = "weight"
+redirect_to = "docs/api/intro"
 +++
 
-Docspell is designed as a REST server that uses JSON to exchange
+No Content
-data. The REST api can be used to integrate docspell into your
-workflow.
-
-[Docspell REST Api Doc](/openapi/docspell-openapi.html)
-
-The "raw" `openapi.yml` specification file can be found
-[here](/openapi/docspell-openapi.yml).
-
-The routes can be divided into protected and unprotected routes. The
-unprotected, or open routes are at `/open/*` while the protected
-routes are at `/sec/*`. Open routes don't require authenticated access
-and can be used by any user. The protected routes require an
-authenticated user.
-
-## Authentication
-
-The unprotected route `/open/auth/login` can be used to login with
-account name and password. The response contains a token that can be
-used for accessing protected routes. The token is only valid for a
-restricted time which can be configured (default is 5 minutes).
-
-New tokens can be generated using an existing valid token and the
-protected route `/sec/auth/session`. This will return the same
-response as above, giving a new token.
-
-This token can be added to requests in two ways: as a cookie header or
-a "normal" http header. If a cookie header is used, the cookie name
-must be `docspell_auth` and a custom header must be named
-`X-Docspell-Auth`.
-
-## Live Api
-
-Besides the statically generated documentation at this site, the rest
-server provides a swagger generated api documenation, that allows
-playing around with the api. It requires a running docspell rest
-server. If it is deployed at `http://localhost:7880`, then check this
-url:
-
-```
-http://localhost:7880/api/doc
-```
-
-## Examples
-
-These examples use the great command line tool
-[curl](https://curl.haxx.se/).
-
-### Login
-
-``` bash
-$ curl -X POST -d '{"account": "smith", "password": "test"}' http://localhost:7880/api/v1/open/auth/login
-{"collective":"smith"
-,"user":"smith"
-,"success":true
-,"message":"Login successful"
-,"token":"1568142350115-ZWlrZS9laWtl-$2a$10$rGZUFDAVNIKh4Tj6u6tlI.-O2euwCvmBT0TlyDmIHR1ZsLQPAI="
-,"validMs":300000
-}
-```
-
-### Get new token
-
-``` bash
-$ curl -XPOST -H 'X-Docspell-Auth: 1568142350115-ZWlrZS9laWtl-$2a$10$rGZUFDAVNIKh4Tj6u6tlI.-O2euwCvmBT0TlyDmIHR1ZsLQPAI=' http://localhost:7880/api/v1/sec/auth/session
-{"collective":"smith"
-,"user":"smith"
-,"success":true
-,"message":"Login successful"
-,"token":"1568142446077-ZWlrZS9laWtl-$2a$10$3B0teJ9rMpsBJPzHfZZPoO-WeA1bkfEONBN8fyzWE8DeaAHtUc="
-,"validMs":300000
-}
-```
-
-### Get some insights
-
-``` bash
-$ curl -H 'X-Docspell-Auth: 1568142446077-ZWlrZS9laWtl-$2a$10$3B0teJ9rMpsBJPzHfZZPoO-WeA1bkfEONBN8fyzWE8DeaAHtUc=' http://localhost:7880/api/v1/sec/collective/insights
-{"incomingCount":3
-,"outgoingCount":1
-,"itemSize":207310
-,"tagCloud":{"items":[]}
-}
-```

website/site/content/docs/api/intro.md

@@ -0,0 +1,93 @@
++++
+title = "Api Introduction"
+description = "Api Basics"
+weight = 10
+insert_anchor_links = "right"
+[extra]
+mktoc = true
++++
+
+Docspell is designed as a REST server that uses JSON to exchange
+data. The REST api can be used to integrate docspell into your
+workflow.
+
+[Docspell REST Api Doc](/openapi/docspell-openapi.html)
+
+The "raw" `openapi.yml` specification file can be found
+[here](/openapi/docspell-openapi.yml).
+
+The routes can be divided into protected and unprotected routes. The
+unprotected, or open routes are at `/open/*` while the protected
+routes are at `/sec/*`. Open routes don't require authenticated access
+and can be used by any user. The protected routes require an
+authenticated user.
+
+## Authentication
+
+The unprotected route `/open/auth/login` can be used to login with
+account name and password. The response contains a token that can be
+used for accessing protected routes. The token is only valid for a
+restricted time which can be configured (default is 5 minutes).
+
+New tokens can be generated using an existing valid token and the
+protected route `/sec/auth/session`. This will return the same
+response as above, giving a new token.
+
+This token can be added to requests in two ways: as a cookie header or
+a "normal" http header. If a cookie header is used, the cookie name
+must be `docspell_auth` and a custom header must be named
+`X-Docspell-Auth`.
+
+## Live Api
+
+Besides the statically generated documentation at this site, the rest
+server provides a swagger generated api documenation, that allows
+playing around with the api. It requires a running docspell rest
+server. If it is deployed at `http://localhost:7880`, then check this
+url:
+
+```
+http://localhost:7880/api/doc
+```
+
+## Examples
+
+These examples use the great command line tool
+[curl](https://curl.haxx.se/).
+
+### Login
+
+``` bash
+$ curl -X POST -d '{"account": "smith", "password": "test"}' http://localhost:7880/api/v1/open/auth/login
+{"collective":"smith"
+,"user":"smith"
+,"success":true
+,"message":"Login successful"
+,"token":"1568142350115-ZWlrZS9laWtl-$2a$10$rGZUFDAVNIKh4Tj6u6tlI.-O2euwCvmBT0TlyDmIHR1ZsLQPAI="
+,"validMs":300000
+}
+```
+
+### Get new token
+
+``` bash
+$ curl -XPOST -H 'X-Docspell-Auth: 1568142350115-ZWlrZS9laWtl-$2a$10$rGZUFDAVNIKh4Tj6u6tlI.-O2euwCvmBT0TlyDmIHR1ZsLQPAI=' http://localhost:7880/api/v1/sec/auth/session
+{"collective":"smith"
+,"user":"smith"
+,"success":true
+,"message":"Login successful"
+,"token":"1568142446077-ZWlrZS9laWtl-$2a$10$3B0teJ9rMpsBJPzHfZZPoO-WeA1bkfEONBN8fyzWE8DeaAHtUc="
+,"validMs":300000
+}
+```
+
+### Get some insights
+
+``` bash
+$ curl -H 'X-Docspell-Auth: 1568142446077-ZWlrZS9laWtl-$2a$10$3B0teJ9rMpsBJPzHfZZPoO-WeA1bkfEONBN8fyzWE8DeaAHtUc=' http://localhost:7880/api/v1/sec/collective/insights
+{"incomingCount":3
+,"outgoingCount":1
+,"itemSize":207310
+,"tagCloud":{"items":[]}
+}
+```

website/site/content/docs/api/upload.md

@@ -0,0 +1,163 @@
++++
+title = "Upload Request"
+description = "Describes the upload request"
+weight = 20
+insert_anchor_links = "right"
+[extra]
+mktoc = true
++++
+
+Uploads of files to docspell are always processed the same way, no
+matter if coming from a source, the integration endpoint or from the
+webapp.
+
+The request must be a http `multipart/form-data` request, with two
+possible fields:
+
+- meta
+- file
+
+The `file` field can appear multiple times and is required at least
+once. It is the part containing the file to upload.
+
+The `meta` part is optional and can define additional meta data, that
+docspell applies to items created from the given files. It allows to
+transfer structured information together with the unstructured binary
+files.
+
+This kind of request is very common and most programming languages
+have support for this.
+
+For example, here is a curl command uploading two files with meta
+data. Since `multiple` is `false`, both files are added to one item:
+
+``` bash
+curl -XPOST -F meta='{"multiple":false, "direction": "outgoing", "tags": {"items":["Order"]}}' \
+            -F file=@letter-en.pdf \
+            -F file=@letter-de.pdf \
+            http://192.168.1.95:7880/api/v1/open/upload/item/3H7hvJcDJuk-NrAW4zxsdfj-K6TMPyb6BGP-xKptVxUdqWa
+```
+
+# Metadata
+
+Each upload request can specify a set of metadata that should be
+applied to the item(s) that are created from this upload. This is
+specified via a JSON structure in a part with name `meta`:
+
+``` elm
+{ multiple: Bool
+, direction: Maybe String
+, folder: Maybe String
+, skipDuplicates: Maybe Bool
+, tags: Maybe StringList
+, fileFilter: Maybe String
+}
+```
+
+The `multiple` property is by default `true`. It means that each file
+in the upload request corresponds to a single item. An upload with 5
+files will result in 5 items created. If it is `false`, then docspell
+will create just one item, that will then contain all files.
+
+Furthermore, the direction of the document (one of `incoming` or
+`outgoing`) can be given. It is optional, it can be left out or
+`null`.
+
+A `folder` id can be specified. Each item created by this request will
+be placed into this folder. Errors are logged (for example, the folder
+may have been deleted before the task is executed) and the item is
+then not put into any folder.
+
+The `skipDuplicates` is optional and set to `false` if not specified.
+It configures the processing task. If set to `true`, the processing
+task will skip all input files that are already present in docspell.
+
+The `tags` field can be used to provide tags that should be applied
+automatically. The `StringList` is a json object containing one field
+`items` that is a list of strings:
+
+``` elm
+{ items: (List String)
+}
+```
+
+Tags can be defined using their ids or names.
+
+Then a `fileFilter` field can be used to filter files from uploaded
+archives. Say you upload a zip file and want only to include certain
+files, you could give a file filter like `*.pdf` to only select pdf
+files or `*.html|*.pdf` for selecting html and pdf files. This only
+applies to archive files, like zip or e-mails (where this is applied
+to the attachments).
+
+
+# Endpoints
+
+Docspell needs to the collective that owns the files. There are the
+following ways for this.
+
+
+## Authenticated User
+
+```
+/api/v1/sec/upload/item
+```
+
+This endpoint exists for authenticated users. That is, you must
+provide a valid cookie or `X-Docspell-Auth` header with the request.
+You can obtain this from the `login` endpoint.
+
+## URL protected
+
+```
+/api/v1/open/upload/item/5JE…-…-…-…oHri
+```
+
+A more flexible way for uploading files is to create a
+[“Source”](@/docs/webapp/uploading.md#anonymous-upload) that creates a
+“hard-to-guess” url. A source can be created in the webapp (via http
+calls) and associates a random id to a collective. This id is then
+used in the url and docspell can use it to associate the collective
+when uploading.
+
+When defining sources, you can also add metadata to it. These will be
+used as a fallback when inspecting the requests meta data part. Each
+metadata not defined in the request will be filled with the one from
+the corresponding source. Tags are applied from both inputs.
+
+## Integration Endpoint
+
+Another option for uploading files is the special *integration
+endpoint*. This endpoint allows an admin to upload files to any
+collective, that is known by name.
+
+```
+/api/v1/open/integration/item/[collective-name]
+```
+
+The endpoint is behind `/api/v1/open`, so this route is not protected
+by an authentication token (see [REST Api](@/docs/api/_index.md) for
+more information). However, it can be protected via settings in the
+configuration file. The idea is that this endpoint is controlled by an
+administrator and not the user of the application. The admin can
+enable this endpoint and choose between some methods to protect it.
+Then the administrator can upload files to any collective. This might
+be useful to connect other trusted applications to docspell (that run
+on the same host or network).
+
+The endpoint is disabled by default, an admin must change the
+`docspell.server.integration-endpoint.enabled` flag to `true` in the
+[configuration file](@/docs/configure/_index.md#rest-server).
+
+If queried by a `GET` request, it returns whether it is enabled and
+the collective exists.
+
+It is also possible to check for existing files using their sha256
+checksum with:
+
+```
+/api/v1/open/integration/checkfile/[collective-name]/[sha256-checksum]
+```
+
+See the [SMTP gateway](@/docs/tools/smtpgateway.md) or the [consumedir
+script](@/docs/tools/consumedir.md) for examples to use this endpoint.

website/site/content/docs/features/_index.md

@@ -67,6 +67,10 @@ description = "A list of features and limitations."
     link and send the file to docspell
   - [SMTP Gateway](@/docs/tools/smtpgateway.md): Setup a SMTP server
     that delivers mails directly to docspell.
+  - [Paperless Import](@/docs/tools/paperless-import.md) for importing
+    your data from paperless
+  - [Directory Cleaner](@/docs/tools/consumedir-cleaner.md) clean
+    directories with files that have been uploaded to docspell
 - License: GPLv3
 
 

website/site/content/docs/install/rpi.md

@@ -8,6 +8,10 @@ weight = 40
 Both component can run next to each other on a raspberry pi or
 similiar device.
 
+There is a [project on
+github](https://github.com/docspell/rpi-scripts) that can help with
+setting up a raspberry pi with docspell.
+
 
 ## REST Server
 

website/site/content/docs/tools/consumedir.md

@@ -45,14 +45,14 @@ The URL can be any docspell url that accepts uploads without
 authentication. This is usually a [source
 url](@/docs/webapp/uploading.md#anonymous-upload). It is also possible
 to use the script with the [integration
-endpoint](@/docs/webapp/uploading.md#integration-endpoint).
+endpoint](@/docs/api/upload.md#integration-endpoint).
 
 
 ## Integration Endpoint
 
 When given the `-i` or `--integration` option, the script changes its
 behaviour slightly to work with the [integration
-endpoint](@/docs/webapp/uploading.md#integration-endpoint).
+endpoint](@/docs/api/upload.md#integration-endpoint).
 
 First, if `-i` is given, it implies `-r` – so the directories are
 watched or traversed recursively. The script then assumes that there
@@ -121,7 +121,7 @@ directory, `./docs` in current directory, for new files. If a new file
 is detected, it is pushed to docspell.
 
 This utilizes the [integration
-endpoint](@/docs/webapp/uploading.md#integration-endpoint), which is
+endpoint](@/docs/api/upload.md#integration-endpoint), which is
 enabled in the config file, to allow uploading documents for all
 collectives. A subfolder must be created for each registered
 collective. The docker containers are configured to use http-header

website/site/content/docs/tools/smtpgateway.md

@@ -5,8 +5,8 @@ weight = 50
 +++
 
 One possible use case for the [integration
-endpoint](@/docs/webapp/uploading.md#integration-endpoint) is a SMTP
+endpoint](@/docs/api/upload.md#integration-endpoint) is a SMTP server
-server that forwards all local mail to docspell. This way there is no
+that forwards all local mail to docspell. This way there is no
 periodic polling involved and documents (e-mails) get into docspell
 without delay.
 
@@ -22,8 +22,8 @@ You need to own a domain and add the appropriate MX records to point
 to your server. In this document, the domain `test.org` is used.
 
 You need to enable the [integration
-endpoint](@/docs/webapp/uploading.md#integration-endpoint) in the
+endpoint](@/docs/api/upload.md#integration-endpoint) in the docspell
-docspell configuration.
+configuration.
 
 # Exim
 

website/site/content/docs/webapp/scanmailbox.md

@@ -29,6 +29,8 @@ You can enable or disable this task. A disabled task will not run
 periodically. You can still choose to run it manually if you click the
 `Start Once` button.
 
+## E-Mail Settings
+
 Then you need to specify which [IMAP
 connection](@/docs/webapp/emailsettings.md#imap-settings) to use.
 
@@ -41,12 +43,6 @@ mails in your inbox. Any other folder is usually case-sensitive
 except the INBOX folder). Type in a folder name and click the add
 button on the right.
 
-Then the field *Received Since Hours* defines how many hours to go
-back and look for mails. Usually there are many mails in your inbox
-and importing them all at once is not feasible or desirable. It can
-work together with the *Schedule* field below. For example, you could
-run this task all 6 hours and read mails from 8 hours back.
-
 The next two settings tell docspell what to do once a mail has been
 submitted to docspell. It can be moved into another folder in your
 mail account. This moves it out of the way for the next run. You can
@@ -55,20 +51,52 @@ deleted and not moved to your trash folder*. If both options are off,
 nothing happens with that mail, it simply stays (and could be re-read
 on the next run).
 
-When docspell creates an item from a mail, it needs to set a direction
+## Filtering
-value (incoming or outgoing). If you know that all mails you want to
+
-import have a specific directon, then you can set it here. Otherwise,
+The following properties allow to filter mails that are imported.
-*automatic* means that docspell chooses a direction based on the
+
-`From` header of a mail. If the `From` header is an e-mail address
+Then the field *Received Since Hours* defines how many hours to go
-that belongs to a “concerning” person in your address book, then it is
+back and look for mails. Usually there are many mails in your inbox
-set to "outgoing". Otherwise it is set to "incoming". To support this,
+and importing them all at once is not feasible or desirable. It can
-you need to add your own e-mail address(es) to your address book.
+work together with the *Schedule* field below. For example, you could
+run this task all 6 hours and read mails from 8 hours back.
+
+The *File Filter* can be specified as a glob to only import mail
+attachments based on their file name. For example, a value of `*.pdf`
+will only import files that have a `pdf` extension. The mail body is
+named `mail.html` by convention and would be excluded when only
+specifying `*.pdf`. You can combine multiple globs using OR via the
+pipe `|` symbol. For example, to also include the mail body as well as
+all pdf attachments: `*.pdf|mail.html`.
+
+The *Subject Filter* is a glob filter that is applied to the mail
+subject. This can be used to only import mails whose subject have some
+pattern. For example, if your scanner mails to you with a certain
+subject like _"Scanned Document 214"_, you could include those via a
+`Scanned Document*` pattern.
+
+## Metadata
+
+The last properties allow to specify some metadata that are
+automatically attached to the items being created.
+
+Every item in docspell has a direction value (incoming or outgoing).
+If you know that all mails you want to import have a specific
+directon, then you can set it here. Otherwise, *automatic* means that
+docspell chooses a direction based on the `From` header of a mail. If
+the `From` header is an e-mail address that belongs to a “concerning”
+person in your address book, then it is set to "outgoing". Otherwise
+it is set to "incoming". To support this, you need to add your own
+e-mail address(es) to your address book.
 
 The *Item Folder* setting is used to put all items that are created
 from mails into the specified [folder](metadata#folders). If you
 define a folder here, where you are not a member, you won't find
 resulting items.
 
+The *Tags* setting can be used to associate a fixed number of tags to
+all items that are imported from this mail task.
+
 The last field is the *Schedule* which defines when and how often this
 task should run. The syntax is similiar to a date-time string, like
 `2019-09-15 12:32`, where each part is a pattern to also match multple

website/site/content/docs/webapp/uploading.md

@@ -92,95 +92,7 @@ docspell count the files uploaded through the web interface, just
 create a source (can be inactive) with that name (`webapp`).
 
 
-# Integration Endpoint
+# Other options
 
-Another option for uploading files is the special *integration
+More details about the actual http request and other upload options
-endpoint*. This endpoint allows an admin to upload files to any
+can be found [here](@/docs/api/upload.md).
-collective, that is known by name.
-
-```
-/api/v1/open/integration/item/[collective-name]
-```
-
-The endpoint is behind `/api/v1/open`, so this route is not protected
-by an authentication token (see [REST Api](@/docs/api/_index.md) for
-more information). However, it can be protected via settings in the
-configuration file. The idea is that this endpoint is controlled by an
-administrator and not the user of the application. The admin can
-enable this endpoint and choose between some methods to protect it.
-Then the administrator can upload files to any collective. This might
-be useful to connect other trusted applications to docspell (that run
-on the same host or network).
-
-The endpoint is disabled by default, an admin must change the
-`docspell.server.integration-endpoint.enabled` flag to `true` in the
-[configuration file](@/docs/configure/_index.md#rest-server).
-
-If queried by a `GET` request, it returns whether it is enabled and
-the collective exists.
-
-It is also possible to check for existing files using their sha256
-checksum with:
-
-```
-/api/v1/open/integration/checkfile/[collective-name]/[sha256-checksum]
-```
-
-See the [SMTP gateway](@/docs/tools/smtpgateway.md) or the [consumedir
-script](@/docs/tools/consumedir.md) for examples to use this endpoint.
-
-# The Request
-
-This gives more details about the request for uploads. It is a http
-`multipart/form-data` request, with two possible fields:
-
-- meta
-- file
-
-The `file` field can appear multiple times and is required at least
-once. It is the part containing the file to upload.
-
-The `meta` part is completely optional and can define additional meta
-data, that docspell uses to create items from the given files. It
-allows to transfer structured information together with the
-unstructured binary files.
-
-The `meta` content must be `application/json` containing this
-structure:
-
-``` elm
-{ multiple: Bool
-, direction: Maybe String
-, folder: Maybe String
-, skipDuplicates: Bool
-}
-```
-
-The `multiple` property is by default `true`. It means that each file
-in the upload request corresponds to a single item. An upload with 5
-files will result in 5 items created. If it is `false`, then docspell
-will create just one item, that will then contain all files.
-
-Furthermore, the direction of the document (one of `incoming` or
-`outgoing`) can be given. It is optional, it can be left out or
-`null`.
-
-A `folder` id can be specified. Each item created by this request will
-be placed into this folder. Errors are logged (for example, the folder
-may have been deleted before the task is executed) and the item is
-then not put into any folder.
-
-The `skipDuplicates` is optional and set to `false` if not specified.
-It configures the processing task. If set to `true`, the processing
-task will skip all input files that are already present in docspell.
-
-This kind of request is very common and most programming languages
-have support for this. For example, here is another curl command
-uploading two files with meta data:
-
-``` bash
-curl -XPOST -F meta='{"multiple":false, "direction": "outgoing"}' \
-            -F file=@letter-en-source.pdf \
-            -F file=@letter-de-source.pdf \
-            http://192.168.1.95:7880/api/v1/open/upload/item/3H7hvJcDJuk-NrAW4zxsdfj-K6TMPyb6BGP-xKptVxUdqWa
-```