TheAnachronism/docspell
1
0
Fork 0
mirror of https://github.com/TheAnachronism/docspell.git synced 2025-03-28 09:45:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update documentation

Browse Source
This commit is contained in:
Eike Kettner 2020-05-22 23:49:07 +02:00
parent 69c46f9f1f
commit 4227278bb0
12 changed files with 420 additions and 49 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
Changelog.md
View File

@ -1,5 +1,43 @@
# Changelog
## v0.6.0
*Unknown*
- New feature "Scan Mailboxes". Docspell can now read mailboxes
periodically to import your mails.
- Fix the `find-by-checksum` route that, given a sha256 checksum,
returns whether there is such a file in docspell. It falsely
returned `false` although documents existed.
- Amend the mail-to-pdf conversion to include the e-mail date.
- Fix webapp for mobile devices.
- The document list on the front-page has been rewritten. The table is
removed and documents are now presented in a “card view”.
- Fix the search menu to remember dates in fields. When going back
from an item detail to the front-page, the search menu remembers the
last state, but dates were cleared.
- More fixes regarding character encodings when reading e-mails.
### Configuration Changes
The joex component has new config sections:
- Add `docspell.joex.mail-debug` flag to enable debugging e-mail
related code. This is only useful if you encounter problems
connecting to mail servers.
- Add `docspell.joex.user-tasks` with a `scan-mailbox` section to
configure the new scan-mailbox user task.
- Add `docspell.joex.files` section that is the same as the
corresponding section in the rest server config.
### REST Api Changes
- Change `/sec/email/settings` to `/sec/email/settings/smtp`
- Add `/sec/email/settings/imap`
- Add `/sec/usertask/scanmailbox` routes to configure one or more
scan-mailbox tasks
## v0.5.0
*May 1st, 2020*

238
modules/microsite/docs/doc/emailsettings.md Normal file
View File

@ -0,0 +1,238 @@
---
layout: docs
title: E-Mail Settings
permalink: doc/emailsettings
---
# {{page.title}}
Docspell has a good integration for E-Mail. You can send e-mails
related to an item and you can import e-mails from your mailbox into
docspell.
This requires to define settings to use for sending and receiving
e-mails. E-Mails are commonly send via
[SMTP](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol)
and for receiving
[IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol)
is quite common. Docspell has support for SMTP and IMAP. These
settings are associated to a user, so that each user can specify its
own settings separately from others in the collective.
*Note: Passwords to your e-mail accounts are stored in plain-text in
docspell's database. This is necessary to have docspell connect to
your e-mail account to send mails on behalf of you and receive your
mails.*
## SMTP Settings
For sending mail, you need to provide information to connect to a SMTP
server. Every e-mail provider has this information somewhere
available.
Configure this in *User Settings -> E-Mail Settings (SMTP)*:
<div class="thumbnail">
<img src="../img/mail-settings-1.png">
</div>
First, you need to provide some name that is used to recognize this
account. This name is also used in URLs to docspell and so it must not
contain whitespace or any special characters. A good value is the
domain of your provider, for example `gmail.com`, or something like
that.
These information should be available from your e-mail provider. For
example, for google-mail it is:
- SMTP Host: `smtp.gmail.com`
- SMTP Port: `587` or `465`
- SMTP User: Your Gmail address (for example, example@gmail.com)
- SMTP Password: Your Gmail password
- SSL: use `SSL` for port `465` and `StartSSL` for port `587`
Then you need to define the e-mail address that is used for the `From`
field. This is in most cases the same address as used for the SMTP
User field.
The `Reply-To` field is optional and can be set to define a different
e-mail address that your recipients should use to answer a mail.
Once this is setup, you can start sending mails within docspell. It is
possible to set up these settings for multiple providers, so you can
choose from which account you want to send mails.
## IMAP Settings
For receiving e-mails, you need to provide information to connect to
an IMAP server. Your e-mail provider should have this information
somewhere available.
Configure this in *User Settings -> E-Mail Settings (IMAP)*:
<div class="thumbnail">
<img src="../img/mail-settings-2.png">
</div>
First you need to define a *Name* to recognize this connection inside
docspell. This name is also used in URLs to docspell and so it must
not contain whitespace or any special characters. A good value is the
domain of your provider, for example `gmail.com`, or something like
that.
You can provide imap connections to multiple mailboxes.
Here is an example for posteo.de:
- IMAP Server: `posteo.de`
- IMAP Port: 143
- IMAP User: Your posteo address
- IMAP Password: Your posteo password
- SSL: use `StartTLS`
## SSL / TLS / StartTLS
*Please Note: If `SSL` is set to `None`, then mails will be sent
unencrypted to your mail provider! If `Ignore certificate check` is
enabled, connections to your mail provider will succeed even if the
provider is wrongly configured for SSL/TLS. This flag should only be
enabled if you know why.*
## GMail
Authenticating with GMail may be not so simple. GMail implements an
authentication scheme called *XOAUTH2* (at least for Imap). It will
not work with your normal password. This is to avoid giving an
application full access to your gmail account.
The e-mail integration in docspell relies on the
[JavaMail](https://javaee.github.io/javamail) library which has
support for XOAUTH2. It also has documentation on what you need to do
on your gmail account: <https://javaee.github.io/javamail/OAuth2>.
First you need to go to the [Google Developers
Console](https://console.developers.google.com) and create an "App" to
get a Client-Id and a Client-Secret. This "App" will be your instance
of docspell. You tell google that this app may send and read your
mails and then you get an *access token* that should be used instead
of the password.
Once you setup an App in Google Developers Console, you get the
Client-Id and the Client-Secret, which look something like this:
- Client-Id: 106701....d8c.apps.googleusercontent.com
- Client-Secret: 5Z1...Kir_t
Google has a python tool to help with getting this access token.
Download the `oauth2.py` script from
[here](https://github.com/google/gmail-oauth2-tools) and first create
an *oauth2-token*:
``` shell
./oauth2.py --user=your.name@gmail.com \
--client_id=106701....d8c.apps.googleusercontent.com \
--client_secret=5Z1...Kir_t \
--generate_oauth2_token
```
This will "redirect you" to an URL where you have to authenticate with
google. Afterwards it lets you add permissions to the app for
accessing your mail account. The result is another code you need to
give to the script to proceed:
```
4/zwE....q0QBAb-99yD7lw
```
Then the scripts produces this:
```
Refresh Token: 1//09zH.........Lj6oc2SmFlZww
Access Token: ya29.a0........SECDQ
Access Token Expiration Seconds: 3599
```
The access token can be used to sign in via IMAP with google. The
Refresh Token doesn't expire and can be used to generate new access
tokens:
```
./oauth2.py --user=your.name@gmail.com \
--client_id=106701....d8c.apps.googleusercontent.com \
--client_secret=5Z1...Kir_t \
--refresh_token=1//09zH.........Lj6oc2SmFlZww
```
Output:
```
Access Token: ya29.a0....._q-lX3ypntk3ln0h9Yk
Access Token Expiration Seconds: 3599
```
The problem is that the access token expires. Docspell doesn't support
updating the access token. It could be worked around by setting up a
cron-job or similiar which uses the `oauth2.py` tool to generate new
access tokens and update your imap settings via a [REST](../api) call.
``` bash
#!/usr/bin/env bash
set -e
## Change this to your values:
DOCSPELL_USER="[docspell-user]"
DOCSPELL_PASSWORD="[docspell-password]"
DOCSPELL_URL="http://localhost:7880"
DOCSPELL_IMAP_NAME="gmail.com"
GMAIL_USER="your.name@gmail.com"
CLIENT_ID="106701....d8c.apps.googleusercontent.com"
CLIENT_SECRET="secret=5Z1...Kir_t"
REFRESH_TOKEN="1//09zH.........Lj6oc2SmFlZww"
# Path to the oauth2.py tool
OAUTH_TOOL="./oauth2.py"
##############################################################################
## Script
# Login to docspell and store the auth-token
AUTH_DATA=$(curl --silent -XPOST \
-H 'Content-Type: application/json' \
--data-binary "{\"account\":\"$DOCSPELL_USER\",\"password\":\"$DOCSPELL_PASSWORD\"}" \
$DOCSPELL_URL/api/v1/open/auth/login)
if [ $(echo $AUTH_DATA | jq .success) == "false" ]; then
echo "Auth failed"
echo $AUTH_DATA
fi
TOKEN="$(echo $AUTH_DATA | jq -r .token)"
# Get the imap settings
UPDATE_URL="$DOCSPELL_URL/api/v1/sec/email/settings/imap/$DOCSPELL_IMAP_NAME"
IMAP_DATA=$(curl -s -H "X-Docspell-Auth: $TOKEN" "$UPDATE_URL")
echo "Current Settings:"
echo $IMAP_DATA | jq
# Get the new access token
ACCESS_TOKEN=$($OAUTH_TOOL --user=$GMAIL_USER \
--client_id="$CLIENT_ID" \
--client_secret="$CLIENT_SECRET" \
--refresh_token="$REFRESH_TOKEN" | head -n1 | cut -d':' -f2 | xargs)
# Update settings
echo "Updating IMAP settings"
NEW_IMAP=$(echo $IMAP_DATA | jq ".imapPassword |= \"$ACCESS_TOKEN\"")
curl -s -XPUT -H "X-Docspell-Auth: $TOKEN" \
-H 'Content-Type: application/json' \
--data-binary "$NEW_IMAP" "$UPDATE_URL"
echo
echo "New Settings:"
curl -s -H "X-Docspell-Auth: $TOKEN" "$UPDATE_URL" | jq
```

43
modules/microsite/docs/doc/mailitem.md
View File

@ -13,48 +13,11 @@ people outside docspell.
All sent mails are stored attached to the item.
## E-Mail Settings
## E-Mail Settings (SMTP)
To send mails, there are SMTP settings required. Since an e-mail
account is connected to a user, you need to check the *User Settings*
page from the top-right menu.
To send mails, there are SMTP settings required. Please see the page
about [e-mail settings](emailsettings#smtp-settings).
<div class="thumbnail">
<img src="../img/mail-settings-1.jpg">
</div>
First, you need to provide some name that is used to recognize this
account. This name is also used in URLs to docspell and so it must not
contain whitespace or any special characters. A good value is the
domain of your provider, for example `gmail.com`, or something like
that.
These information should be available from your e-mail provider. For
example, for google-mail it is:
- SMTP Host: `smtp.gmail.com`
- SMTP Port: `587` or `465`
- SMTP User: Your Gmail address (for example, example@gmail.com)
- SMTP Password: Your Gmail password
- SSL: use `SSL` for port `465` and `StartSSL` for port `587`
Then you need to define the e-mail address that is used for the `From`
field. This is in most cases the same address as used for the SMTP
User field.
The `Reply-To` field is optional and can be set to define a different
e-mail address that your recipients should use to answer a mail.
Once this is setup, you can start sending mails within docspell. It is
possible to set up these settings for multiple providers, so you can
choose from which account you want to send mails.
*Please Note: If `SSL` is set to `None`, then mails will be sent
unencrypted to your mail provider! If `Ignore certificate check` is
enabled, connections to your mail provider will succeed even if the
provider is wrongly configured for SSL/TLS. This flag should only be
enabled if you know why.*
## Sending Mails

6
modules/microsite/docs/doc/notifydueitems.md
View File

@ -6,9 +6,9 @@ permalink: doc/notifydueitems
# {{page.title}}
Each user that provides valid email (smtp) settings, can be notified
by docspell about due items. You will then receive an e-mail
containing a list of items, sorted by their due date.
A user that provides valid email (smtp) settings, can be notified by
docspell about due items. You will then receive an e-mail containing a
list of items, sorted by their due date.
You need first define smtp settings, please see [this
page](mailitem#e-mail-settings).

122
modules/microsite/docs/doc/scanmailbox.md Normal file
View File

@ -0,0 +1,122 @@
---
layout: docs
title: Scan Mailboxes
permalink: doc/scanmailbox
---
# {{page.title}}
User that provide valid email (imap) settings, can import mails from
their mailbox into docspell periodically.
You need first define imap settings, please see [this
page](emailsettings#imap-settings).
Go to *User Settings -> Scan Mailbox Task*. You can define periodic
tasks that connects to your mailbox and import mails into docspell. It
is possible to define multiple tasks, for example, if you have
multiple e-mail accounts you want to import periodically.
<div class="thumbnail">
<img src="../img/scanmailbox-list.png">
</div>
## Details
Creating a task requires the following information:
<div class="thumbnail">
<img src="../img/scanmailbox-detail.png">
</div>
You can enable or disable this task. A disabled task will not run
periodically. You can still choose to run it manually if you click the
`Start Once` button.
Then you need to specify which [IMAP
connection](emailsettings#imap-settings) to use.
A list of folders is required. Docspell will only look into these
folders. You can specify multiple folders. The "Inbox" folder is a
special folder, which will usually appear translated in you web-mail
client. You can specify "INBOX" case insensitive, it will then read
mails in your inbox. Any other folder is usually case-sensitive
(depends on the imap server, but usually they are case sensitive
except the INBOX folder). Type in a folder name and click the add
button on the right.
Then the field *Received Since Hours* defines how many hours to go
back and look for mails. Usually there are many mails in your inbox
and importing them all at once is not feasible or desirable. It can
work together with the *Schedule* field below. For example, you could
run this task all 6 hours and read mails from 8 hours back.
The next two settings tell docspell what to do once a mail has been
submitted to docspell. It can be moved into another folder in your
mail account. This moves it out of the way for the next run. You can
also choose to delete the mail, but *note that it will really be
deleted and not moved to your trash folder*. If both options are off,
nothing happens with that mail, it simply stays (and could be re-read
on the next run).
When docspell creates an item from a mail, it needs to set a direction
value (incoming or outgoing). If you know that all mails you want to
import have a specific directon, then you can set it here. Otherwise,
*automatic* means that docspell chooses a direction based on the
`From` header of a mail. If the `From` header is an e-mail address
that belongs to a “concerning” person in your address book, then it is
set to "outgoing". Otherwise it is set to "incoming". To support this,
you need to add your own e-mail address(es) to your address book.
The last field is the *Schedule* which defines when and how often this
task should run. The syntax is similiar to a date-time string, like
`2019-09-15 12:32`, where each part is a pattern to also match multple
values. The ui tries to help a little by displaying the next two
date-times this task would execute. A more in depth help is available
[here](https://github.com/eikek/calev#what-are-calendar-events). For
example, to execute the task every monday at noon, you would write:
`Mon *-*-* 12:00`. A date-time part can match all values (`*`), a list
of values (e.g. `1,5,12,19`) or a range (e.g. `1..9`). Long lists may
be written in a shorter way using a repetition value. It is written
like this: `1/7` which is the same as a list with `1` and all
multiples of `7` added to it. In other words, it matches `1`, `1+7`,
`1+7+7`, `1+7+7+7` and so on.
## Reading Mails twice / Duplicates
Since users can move around mails in their mailboxes, it can happen
that docspell unintentionally reads a mail multiple times. If docspell
reads a mail, it will first check if an item already exists that
originated from this mail. It only proceeds to import it, if it cannot
find any. If you deleted an item in the meantime, docspell would
import the mail again.
This check uses the
[`Message-ID`](https://en.wikipedia.org/wiki/Message-ID) of an e-mail.
This is usually there and should identify a complete mail. But it
won't catch duplicate mails, that are sent multiple times - they might
have different `Message-ID`s. Also some mails have no such ids and are
then imported from docspell without any checks.
In later versions, docspell may use the checksum of the generated eml
file to look for duplicates, too.
## How it works
Docspell will go through all folders and download mails in “batches”.
This size can be set by the admin in the [configuration
file](configure#joex) and applies to all these tasks (same for all
users). This batch only contains the mail headers and not the complete
mail.
Then each mail is downloaded completely one by one and converted into
an [eml](https://en.wikipedia.org/wiki/Email#Filename_extensions) file
which is then submitted to docspell. Then the usual processing
machinery starts, just like uploading an eml file via the webapp.
The number of folders and the number of mails to import can be limited
by an admin via the config file. Note that this limit applies to one
task run only, it is meant to reduce resource allocation of one task.

16
modules/microsite/docs/features.md
View File

@ -15,12 +15,16 @@ permalink: features
- Non-destructive: all your uploaded files are never modified and can
always be downloaded untouched
- Text is analysed to find and attach meta data automatically
- Manage document processing (cancel jobs, set priorities)
- [Manage document processing](doc/processing): cancel jobs, set
priorities
- Everything available via a documented [REST Api](api)
- Web-UI included
- Create “share-urls” to upload files anonymously
- Send documents via e-mail
- E-Mail notification for documents with due dates
- mobile-friendly Web-UI
- [Create “share-urls”](doc/uploading#anonymous-upload) to upload files
anonymously
- [Send documents via e-mail](doc/mailitem)
- [E-Mail notification](doc/notifydueitems) for documents with due dates
- [Read your mailboxes](doc/scanmailbox) via IMAP to import mails into
docspell
- REST server and document processing are separate applications which
can be scaled-out independently
- Everything stored in a SQL database: PostgreSQL, MariaDB or H2
@ -37,7 +41,7 @@ permalink: features
- zip
- [eml](https://en.wikipedia.org/wiki/Email#Filename_extensions)
(e-mail files in plain text MIME)
- Tooling:
- [Tooling](doc/tools):
- Watch a folder: watch folders for changes and send files to docspell
- Firefox plugin: right click on a link and send the file to docspell
- Simple CLI for uploading files

6
modules/microsite/src/main/resources/microsite/data/menu.yml
View File

@ -39,12 +39,18 @@ options:
- title: Find and Review
url: doc/curate
- title: E-Mail Settings
url: doc/emailsettings
- title: Send via E-Mail
url: doc/mailitem
- title: Notify on due Items
url: doc/notifydueitems
- title: Scan Mailboxes
url: doc/scanmailbox
- title: Joex
url: doc/joex

BIN
modules/microsite/src/main/resources/microsite/img/mail-settings-1.jpg
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 155 KiB

BIN
modules/microsite/src/main/resources/microsite/img/mail-settings-1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 71 KiB

BIN
modules/microsite/src/main/resources/microsite/img/mail-settings-2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

BIN
modules/microsite/src/main/resources/microsite/img/scanmailbox-detail.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 146 KiB

BIN
modules/microsite/src/main/resources/microsite/img/scanmailbox-list.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 72 KiB