diff --git a/Changelog.md b/Changelog.md index bd5efb6c..7788e364 100644 --- a/Changelog.md +++ b/Changelog.md @@ -1,5 +1,43 @@ # Changelog +## v0.6.0 + +*Unknown* + +- New feature "Scan Mailboxes". Docspell can now read mailboxes + periodically to import your mails. +- Fix the `find-by-checksum` route that, given a sha256 checksum, + returns whether there is such a file in docspell. It falsely + returned `false` although documents existed. +- Amend the mail-to-pdf conversion to include the e-mail date. +- Fix webapp for mobile devices. +- The document list on the front-page has been rewritten. The table is + removed and documents are now presented in a “card view”. +- Fix the search menu to remember dates in fields. When going back + from an item detail to the front-page, the search menu remembers the + last state, but dates were cleared. +- More fixes regarding character encodings when reading e-mails. + +### Configuration Changes + +The joex component has new config sections: + +- Add `docspell.joex.mail-debug` flag to enable debugging e-mail + related code. This is only useful if you encounter problems + connecting to mail servers. +- Add `docspell.joex.user-tasks` with a `scan-mailbox` section to + configure the new scan-mailbox user task. +- Add `docspell.joex.files` section that is the same as the + corresponding section in the rest server config. + +### REST Api Changes + +- Change `/sec/email/settings` to `/sec/email/settings/smtp` +- Add `/sec/email/settings/imap` +- Add `/sec/usertask/scanmailbox` routes to configure one or more + scan-mailbox tasks + + ## v0.5.0 *May 1st, 2020* diff --git a/modules/microsite/docs/doc/emailsettings.md b/modules/microsite/docs/doc/emailsettings.md new file mode 100644 index 00000000..050f90ac --- /dev/null +++ b/modules/microsite/docs/doc/emailsettings.md @@ -0,0 +1,238 @@ +--- +layout: docs +title: E-Mail Settings +permalink: doc/emailsettings +--- + +# {{page.title}} + +Docspell has a good integration for E-Mail. You can send e-mails +related to an item and you can import e-mails from your mailbox into +docspell. + +This requires to define settings to use for sending and receiving +e-mails. E-Mails are commonly send via +[SMTP](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol) +and for receiving +[IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol) +is quite common. Docspell has support for SMTP and IMAP. These +settings are associated to a user, so that each user can specify its +own settings separately from others in the collective. + +*Note: Passwords to your e-mail accounts are stored in plain-text in +docspell's database. This is necessary to have docspell connect to +your e-mail account to send mails on behalf of you and receive your +mails.* + + +## SMTP Settings + +For sending mail, you need to provide information to connect to a SMTP +server. Every e-mail provider has this information somewhere +available. + +Configure this in *User Settings -> E-Mail Settings (SMTP)*: + +
+ +
+ +First, you need to provide some name that is used to recognize this +account. This name is also used in URLs to docspell and so it must not +contain whitespace or any special characters. A good value is the +domain of your provider, for example `gmail.com`, or something like +that. + +These information should be available from your e-mail provider. For +example, for google-mail it is: + +- SMTP Host: `smtp.gmail.com` +- SMTP Port: `587` or `465` +- SMTP User: Your Gmail address (for example, example@gmail.com) +- SMTP Password: Your Gmail password +- SSL: use `SSL` for port `465` and `StartSSL` for port `587` + +Then you need to define the e-mail address that is used for the `From` +field. This is in most cases the same address as used for the SMTP +User field. + +The `Reply-To` field is optional and can be set to define a different +e-mail address that your recipients should use to answer a mail. + +Once this is setup, you can start sending mails within docspell. It is +possible to set up these settings for multiple providers, so you can +choose from which account you want to send mails. + + +## IMAP Settings + +For receiving e-mails, you need to provide information to connect to +an IMAP server. Your e-mail provider should have this information +somewhere available. + +Configure this in *User Settings -> E-Mail Settings (IMAP)*: + +
+ +
+ +First you need to define a *Name* to recognize this connection inside +docspell. This name is also used in URLs to docspell and so it must +not contain whitespace or any special characters. A good value is the +domain of your provider, for example `gmail.com`, or something like +that. + +You can provide imap connections to multiple mailboxes. + +Here is an example for posteo.de: + +- IMAP Server: `posteo.de` +- IMAP Port: 143 +- IMAP User: Your posteo address +- IMAP Password: Your posteo password +- SSL: use `StartTLS` + + +## SSL / TLS / StartTLS + +*Please Note: If `SSL` is set to `None`, then mails will be sent +unencrypted to your mail provider! If `Ignore certificate check` is +enabled, connections to your mail provider will succeed even if the +provider is wrongly configured for SSL/TLS. This flag should only be +enabled if you know why.* + + +## GMail + +Authenticating with GMail may be not so simple. GMail implements an +authentication scheme called *XOAUTH2* (at least for Imap). It will +not work with your normal password. This is to avoid giving an +application full access to your gmail account. + +The e-mail integration in docspell relies on the +[JavaMail](https://javaee.github.io/javamail) library which has +support for XOAUTH2. It also has documentation on what you need to do +on your gmail account: . + +First you need to go to the [Google Developers +Console](https://console.developers.google.com) and create an "App" to +get a Client-Id and a Client-Secret. This "App" will be your instance +of docspell. You tell google that this app may send and read your +mails and then you get an *access token* that should be used instead +of the password. + +Once you setup an App in Google Developers Console, you get the +Client-Id and the Client-Secret, which look something like this: + +- Client-Id: 106701....d8c.apps.googleusercontent.com +- Client-Secret: 5Z1...Kir_t + +Google has a python tool to help with getting this access token. +Download the `oauth2.py` script from +[here](https://github.com/google/gmail-oauth2-tools) and first create +an *oauth2-token*: + +``` shell +./oauth2.py --user=your.name@gmail.com \ + --client_id=106701....d8c.apps.googleusercontent.com \ + --client_secret=5Z1...Kir_t \ + --generate_oauth2_token +``` + +This will "redirect you" to an URL where you have to authenticate with +google. Afterwards it lets you add permissions to the app for +accessing your mail account. The result is another code you need to +give to the script to proceed: + +``` +4/zwE....q0QBAb-99yD7lw +``` + +Then the scripts produces this: + +``` +Refresh Token: 1//09zH.........Lj6oc2SmFlZww +Access Token: ya29.a0........SECDQ +Access Token Expiration Seconds: 3599 +``` + +The access token can be used to sign in via IMAP with google. The +Refresh Token doesn't expire and can be used to generate new access +tokens: + +``` +./oauth2.py --user=your.name@gmail.com \ + --client_id=106701....d8c.apps.googleusercontent.com \ + --client_secret=5Z1...Kir_t \ + --refresh_token=1//09zH.........Lj6oc2SmFlZww +``` + +Output: +``` +Access Token: ya29.a0....._q-lX3ypntk3ln0h9Yk +Access Token Expiration Seconds: 3599 +``` + +The problem is that the access token expires. Docspell doesn't support +updating the access token. It could be worked around by setting up a +cron-job or similiar which uses the `oauth2.py` tool to generate new +access tokens and update your imap settings via a [REST](../api) call. + +``` bash +#!/usr/bin/env bash +set -e + +## Change this to your values: + +DOCSPELL_USER="[docspell-user]" +DOCSPELL_PASSWORD="[docspell-password]" +DOCSPELL_URL="http://localhost:7880" +DOCSPELL_IMAP_NAME="gmail.com" + +GMAIL_USER="your.name@gmail.com" +CLIENT_ID="106701....d8c.apps.googleusercontent.com" +CLIENT_SECRET="secret=5Z1...Kir_t" +REFRESH_TOKEN="1//09zH.........Lj6oc2SmFlZww" +# Path to the oauth2.py tool +OAUTH_TOOL="./oauth2.py" + +############################################################################## +## Script + + +# Login to docspell and store the auth-token +AUTH_DATA=$(curl --silent -XPOST \ + -H 'Content-Type: application/json' \ + --data-binary "{\"account\":\"$DOCSPELL_USER\",\"password\":\"$DOCSPELL_PASSWORD\"}" \ + $DOCSPELL_URL/api/v1/open/auth/login) +if [ $(echo $AUTH_DATA | jq .success) == "false" ]; then + echo "Auth failed" + echo $AUTH_DATA +fi +TOKEN="$(echo $AUTH_DATA | jq -r .token)" + + +# Get the imap settings +UPDATE_URL="$DOCSPELL_URL/api/v1/sec/email/settings/imap/$DOCSPELL_IMAP_NAME" +IMAP_DATA=$(curl -s -H "X-Docspell-Auth: $TOKEN" "$UPDATE_URL") + +echo "Current Settings:" +echo $IMAP_DATA | jq + + +# Get the new access token +ACCESS_TOKEN=$($OAUTH_TOOL --user=$GMAIL_USER \ + --client_id="$CLIENT_ID" \ + --client_secret="$CLIENT_SECRET" \ + --refresh_token="$REFRESH_TOKEN" | head -n1 | cut -d':' -f2 | xargs) + +# Update settings +echo "Updating IMAP settings" +NEW_IMAP=$(echo $IMAP_DATA | jq ".imapPassword |= \"$ACCESS_TOKEN\"") +curl -s -XPUT -H "X-Docspell-Auth: $TOKEN" \ + -H 'Content-Type: application/json' \ + --data-binary "$NEW_IMAP" "$UPDATE_URL" +echo +echo "New Settings:" +curl -s -H "X-Docspell-Auth: $TOKEN" "$UPDATE_URL" | jq +``` diff --git a/modules/microsite/docs/doc/mailitem.md b/modules/microsite/docs/doc/mailitem.md index ad8c527d..2d08fa1d 100644 --- a/modules/microsite/docs/doc/mailitem.md +++ b/modules/microsite/docs/doc/mailitem.md @@ -13,48 +13,11 @@ people outside docspell. All sent mails are stored attached to the item. -## E-Mail Settings +## E-Mail Settings (SMTP) -To send mails, there are SMTP settings required. Since an e-mail -account is connected to a user, you need to check the *User Settings* -page from the top-right menu. +To send mails, there are SMTP settings required. Please see the page +about [e-mail settings](emailsettings#smtp-settings). -
- -
- -First, you need to provide some name that is used to recognize this -account. This name is also used in URLs to docspell and so it must not -contain whitespace or any special characters. A good value is the -domain of your provider, for example `gmail.com`, or something like -that. - -These information should be available from your e-mail provider. For -example, for google-mail it is: - -- SMTP Host: `smtp.gmail.com` -- SMTP Port: `587` or `465` -- SMTP User: Your Gmail address (for example, example@gmail.com) -- SMTP Password: Your Gmail password -- SSL: use `SSL` for port `465` and `StartSSL` for port `587` - -Then you need to define the e-mail address that is used for the `From` -field. This is in most cases the same address as used for the SMTP -User field. - -The `Reply-To` field is optional and can be set to define a different -e-mail address that your recipients should use to answer a mail. - -Once this is setup, you can start sending mails within docspell. It is -possible to set up these settings for multiple providers, so you can -choose from which account you want to send mails. - - -*Please Note: If `SSL` is set to `None`, then mails will be sent -unencrypted to your mail provider! If `Ignore certificate check` is -enabled, connections to your mail provider will succeed even if the -provider is wrongly configured for SSL/TLS. This flag should only be -enabled if you know why.* ## Sending Mails diff --git a/modules/microsite/docs/doc/notifydueitems.md b/modules/microsite/docs/doc/notifydueitems.md index a816c8fb..5a53eea7 100644 --- a/modules/microsite/docs/doc/notifydueitems.md +++ b/modules/microsite/docs/doc/notifydueitems.md @@ -6,9 +6,9 @@ permalink: doc/notifydueitems # {{page.title}} -Each user that provides valid email (smtp) settings, can be notified -by docspell about due items. You will then receive an e-mail -containing a list of items, sorted by their due date. +A user that provides valid email (smtp) settings, can be notified by +docspell about due items. You will then receive an e-mail containing a +list of items, sorted by their due date. You need first define smtp settings, please see [this page](mailitem#e-mail-settings). diff --git a/modules/microsite/docs/doc/scanmailbox.md b/modules/microsite/docs/doc/scanmailbox.md new file mode 100644 index 00000000..d8d6b4e9 --- /dev/null +++ b/modules/microsite/docs/doc/scanmailbox.md @@ -0,0 +1,122 @@ +--- +layout: docs +title: Scan Mailboxes +permalink: doc/scanmailbox +--- + +# {{page.title}} + +User that provide valid email (imap) settings, can import mails from +their mailbox into docspell periodically. + +You need first define imap settings, please see [this +page](emailsettings#imap-settings). + +Go to *User Settings -> Scan Mailbox Task*. You can define periodic +tasks that connects to your mailbox and import mails into docspell. It +is possible to define multiple tasks, for example, if you have +multiple e-mail accounts you want to import periodically. + +
+ +
+ + +## Details + +Creating a task requires the following information: + +
+ +
+ +You can enable or disable this task. A disabled task will not run +periodically. You can still choose to run it manually if you click the +`Start Once` button. + +Then you need to specify which [IMAP +connection](emailsettings#imap-settings) to use. + +A list of folders is required. Docspell will only look into these +folders. You can specify multiple folders. The "Inbox" folder is a +special folder, which will usually appear translated in you web-mail +client. You can specify "INBOX" case insensitive, it will then read +mails in your inbox. Any other folder is usually case-sensitive +(depends on the imap server, but usually they are case sensitive +except the INBOX folder). Type in a folder name and click the add +button on the right. + +Then the field *Received Since Hours* defines how many hours to go +back and look for mails. Usually there are many mails in your inbox +and importing them all at once is not feasible or desirable. It can +work together with the *Schedule* field below. For example, you could +run this task all 6 hours and read mails from 8 hours back. + +The next two settings tell docspell what to do once a mail has been +submitted to docspell. It can be moved into another folder in your +mail account. This moves it out of the way for the next run. You can +also choose to delete the mail, but *note that it will really be +deleted and not moved to your trash folder*. If both options are off, +nothing happens with that mail, it simply stays (and could be re-read +on the next run). + +When docspell creates an item from a mail, it needs to set a direction +value (incoming or outgoing). If you know that all mails you want to +import have a specific directon, then you can set it here. Otherwise, +*automatic* means that docspell chooses a direction based on the +`From` header of a mail. If the `From` header is an e-mail address +that belongs to a “concerning” person in your address book, then it is +set to "outgoing". Otherwise it is set to "incoming". To support this, +you need to add your own e-mail address(es) to your address book. + +The last field is the *Schedule* which defines when and how often this +task should run. The syntax is similiar to a date-time string, like +`2019-09-15 12:32`, where each part is a pattern to also match multple +values. The ui tries to help a little by displaying the next two +date-times this task would execute. A more in depth help is available +[here](https://github.com/eikek/calev#what-are-calendar-events). For +example, to execute the task every monday at noon, you would write: +`Mon *-*-* 12:00`. A date-time part can match all values (`*`), a list +of values (e.g. `1,5,12,19`) or a range (e.g. `1..9`). Long lists may +be written in a shorter way using a repetition value. It is written +like this: `1/7` which is the same as a list with `1` and all +multiples of `7` added to it. In other words, it matches `1`, `1+7`, +`1+7+7`, `1+7+7+7` and so on. + + +## Reading Mails twice / Duplicates + +Since users can move around mails in their mailboxes, it can happen +that docspell unintentionally reads a mail multiple times. If docspell +reads a mail, it will first check if an item already exists that +originated from this mail. It only proceeds to import it, if it cannot +find any. If you deleted an item in the meantime, docspell would +import the mail again. + +This check uses the +[`Message-ID`](https://en.wikipedia.org/wiki/Message-ID) of an e-mail. +This is usually there and should identify a complete mail. But it +won't catch duplicate mails, that are sent multiple times - they might +have different `Message-ID`s. Also some mails have no such ids and are +then imported from docspell without any checks. + +In later versions, docspell may use the checksum of the generated eml +file to look for duplicates, too. + + +## How it works + +Docspell will go through all folders and download mails in “batches”. +This size can be set by the admin in the [configuration +file](configure#joex) and applies to all these tasks (same for all +users). This batch only contains the mail headers and not the complete +mail. + +Then each mail is downloaded completely one by one and converted into +an [eml](https://en.wikipedia.org/wiki/Email#Filename_extensions) file +which is then submitted to docspell. Then the usual processing +machinery starts, just like uploading an eml file via the webapp. + +The number of folders and the number of mails to import can be limited +by an admin via the config file. Note that this limit applies to one +task run only, it is meant to reduce resource allocation of one task. diff --git a/modules/microsite/docs/features.md b/modules/microsite/docs/features.md index 52b7120d..62fc76be 100644 --- a/modules/microsite/docs/features.md +++ b/modules/microsite/docs/features.md @@ -15,12 +15,16 @@ permalink: features - Non-destructive: all your uploaded files are never modified and can always be downloaded untouched - Text is analysed to find and attach meta data automatically -- Manage document processing (cancel jobs, set priorities) +- [Manage document processing](doc/processing): cancel jobs, set + priorities - Everything available via a documented [REST Api](api) -- Web-UI included -- Create “share-urls” to upload files anonymously -- Send documents via e-mail -- E-Mail notification for documents with due dates +- mobile-friendly Web-UI +- [Create “share-urls”](doc/uploading#anonymous-upload) to upload files + anonymously +- [Send documents via e-mail](doc/mailitem) +- [E-Mail notification](doc/notifydueitems) for documents with due dates +- [Read your mailboxes](doc/scanmailbox) via IMAP to import mails into + docspell - REST server and document processing are separate applications which can be scaled-out independently - Everything stored in a SQL database: PostgreSQL, MariaDB or H2 @@ -37,7 +41,7 @@ permalink: features - zip - [eml](https://en.wikipedia.org/wiki/Email#Filename_extensions) (e-mail files in plain text MIME) -- Tooling: +- [Tooling](doc/tools): - Watch a folder: watch folders for changes and send files to docspell - Firefox plugin: right click on a link and send the file to docspell - Simple CLI for uploading files diff --git a/modules/microsite/src/main/resources/microsite/data/menu.yml b/modules/microsite/src/main/resources/microsite/data/menu.yml index e94eca55..55cfe096 100644 --- a/modules/microsite/src/main/resources/microsite/data/menu.yml +++ b/modules/microsite/src/main/resources/microsite/data/menu.yml @@ -39,12 +39,18 @@ options: - title: Find and Review url: doc/curate + - title: E-Mail Settings + url: doc/emailsettings + - title: Send via E-Mail url: doc/mailitem - title: Notify on due Items url: doc/notifydueitems + - title: Scan Mailboxes + url: doc/scanmailbox + - title: Joex url: doc/joex diff --git a/modules/microsite/src/main/resources/microsite/img/mail-settings-1.jpg b/modules/microsite/src/main/resources/microsite/img/mail-settings-1.jpg deleted file mode 100644 index bef5f46f..00000000 Binary files a/modules/microsite/src/main/resources/microsite/img/mail-settings-1.jpg and /dev/null differ diff --git a/modules/microsite/src/main/resources/microsite/img/mail-settings-1.png b/modules/microsite/src/main/resources/microsite/img/mail-settings-1.png new file mode 100644 index 00000000..8b4b8da6 Binary files /dev/null and b/modules/microsite/src/main/resources/microsite/img/mail-settings-1.png differ diff --git a/modules/microsite/src/main/resources/microsite/img/mail-settings-2.png b/modules/microsite/src/main/resources/microsite/img/mail-settings-2.png new file mode 100644 index 00000000..8b4a9a32 Binary files /dev/null and b/modules/microsite/src/main/resources/microsite/img/mail-settings-2.png differ diff --git a/modules/microsite/src/main/resources/microsite/img/scanmailbox-detail.png b/modules/microsite/src/main/resources/microsite/img/scanmailbox-detail.png new file mode 100644 index 00000000..c1ddf69b Binary files /dev/null and b/modules/microsite/src/main/resources/microsite/img/scanmailbox-detail.png differ diff --git a/modules/microsite/src/main/resources/microsite/img/scanmailbox-list.png b/modules/microsite/src/main/resources/microsite/img/scanmailbox-list.png new file mode 100644 index 00000000..98d6af80 Binary files /dev/null and b/modules/microsite/src/main/resources/microsite/img/scanmailbox-list.png differ