TheAnachronism/docspell
1
0
Fork 0
mirror of https://github.com/TheAnachronism/docspell.git synced 2024-11-13 02:31:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
NaN Commits 2 Branches 1 Tag 89 MiB
c9f2ed7185
Commit Graph

1709 Commits

Author SHA1 Message Date
Eike Kettner
6a1297fc95 Add a limit for text analysis 2020-03-27 22:54:49 +01:00
Eike Kettner
14a25fe23e Fix serializing mediatype parameters 2020-03-27 21:50:06 +01:00
Eike Kettner
aed5dfaff6 Fix mimetype extractors 2020-03-27 21:49:55 +01:00
Eike Kettner
75405dbcba Update documentation 2020-03-27 20:16:18 +01:00
Eike Kettner
16edf84752 Setup new site 2020-03-27 00:35:15 +01:00
Eike Kettner
9656ba62f4 scalafmtAll 2020-03-26 18:26:00 +01:00
Eike Kettner
09ea724c13 Store message-id of eml files 2020-03-25 22:00:51 +01:00
Eike Kettner
43efb4e6ba Use doobie support from emil project 2020-03-24 23:40:29 +01:00
Eike Kettner
e305b46708 Extract tnef attachments and fix incomplete html 
The wkhtmltopdf requires the content encoding set correctly in the
document.
 2020-03-24 23:40:29 +01:00
Eike Kettner
0b80572664 Fix encodings for mails with non-utf8 html parts 2020-03-24 23:40:29 +01:00
Eike Kettner
cf7ccd572c Improve handling encodings 
Html and text files are not fixed to be UTF-8. The encoding is now
detected, which may not work for all files. Default/fallback will be
utf-8.

There is still a problem with mails that contain html parts not in
utf8 encoding. The mail text is always returned as a string and the
original encoding is lost. Then the html is stored using utf-8 bytes,
but wkhtmltopdf reads it using latin1. It seems that the `--encoding`
setting doesn't override encoding provided by the document.
 2020-03-23 22:51:28 +01:00
Eike Kettner
b265421a46 Allow to use the browser's pdf viewer 
The viewerjs library has some limitations. Sometimes PDFs are quite
blurry and some content is displayed scrambled. Switching to the
browsers build-in PDF viewer (for chromium and firefox) fixes this. So
while on mobile the viewerjs is the only working viewer, for desktop
use it might be desireable to use the browsers builtin viewer instead.
 2020-03-22 22:03:43 +01:00
Eike Kettner
75ead33652 Provide a download link to the original archive file 2020-03-22 21:48:49 +01:00
Eike Kettner
7e6eec9533 Include archive infos in item detail 2020-03-22 21:35:50 +01:00
Eike Kettner
cbc95b11e6 Add routes to retrive the archive of an attachment 2020-03-22 21:21:49 +01:00
Eike Kettner
9a99c852a8 Fix typo in search menu 2020-03-22 21:08:01 +01:00
Eike Kettner
3703dce9a6 Update fs2 to 2.3.0 2020-03-20 22:47:09 +01:00
Eike Kettner
cba466ed47 Set item due date candidate 
After processing, set the due date of an item to the first candidate.
The earliest due date is considered best match.
 2020-03-20 22:39:09 +01:00
Eike Kettner
74a6cf1dd1 Remove unused migration directory 2020-03-19 22:43:41 +01:00
Eike Kettner
b1a1a2b837 Add archives to collective insights 2020-03-19 22:43:18 +01:00
Eike Kettner
d78bd4142c Update documentation 2020-03-19 22:42:58 +01:00
Eike Kettner
439aaee27b Search archives when looking for files via checksum 2020-03-19 22:42:48 +01:00
Eike Kettner
6b1156182c Add support for eml (rfc822 email) files 2020-03-19 22:42:40 +01:00
Eike Kettner
4ed7a137f7 Add support for archive files 
Each attachment is now first extracted into potentially multiple ones,
if it is recognized as an archive. This is the first step in
processing. The original archive file is also stored and the resulting
attachments are associated to their original archive.

First support is implemented for zip files.
 2020-03-19 22:42:27 +01:00
Eike Kettner
10f3d5b7ed Fix bug to select other attachments 2020-03-17 22:37:43 +01:00
Eike Kettner
f0449dd2ce Properly initialize thread pools 2020-03-17 22:37:12 +01:00
Eike Kettner
00ca6b5697 Improve text analysis 
- Search for consecutive labels

- Sort list of candidates by a weight

- Search for organizations using person labels
 2020-03-17 22:34:50 +01:00
Eike Kettner
718e44a21c Add cleanup jobs task 2020-03-09 20:24:00 +01:00
Eike Kettner
854a596da3 Integrate periodic tasks 
The first use case for periodic task is the cleanup of expired
invitation keys. This is part of a house-keeping periodic task.
 2020-03-08 22:49:49 +01:00
Eike Kettner
616c333fa5 Implement storage routines for periodic scheduler 2020-03-08 13:56:23 +01:00
Eike Kettner
1e598bd902 Sketch a scheduler for running periodic tasks 
Periodic tasks are special in that they are usually kept around and
started based on a schedule. A new component checks periodic tasks and
submits them in the queue once they are due.

In order to avoid duplicate periodic jobs, the tracker of a job is
used to store the periodic job id. Each time a periodic task is due,
it is first checked if there is a job running (or queued) for this
task.
 2020-03-08 12:55:03 +01:00
Eike Kettner
9b28858d06 Create a simple client for joex in its api module 
This client can be used within the backend app and later in other
modules. The `OJoex` object is replaced with a better implementation
where the http client is initialized once on app start.
 2020-03-03 23:07:49 +01:00
Eike Kettner
42c59179b8 Fix search by checksum to include source files 2020-03-02 20:56:32 +01:00
Eike Kettner
867b59ac10 Fix link in doc menu 2020-03-01 14:08:21 +01:00
Eike Kettner
d8bbcb1409 Fix front-page links for microsite 
The links work while testing locally with jekyll. Must be checked at
the published site.
 2020-03-01 09:45:38 +01:00
Eike Kettner
b7f2c051f4 Set next version to 0.4.0-SNAPSHOT 2020-02-28 21:19:01 +01:00
Eike Kettner
aa3b9258c4 Set version to 0.3.0 2020-02-28 20:52:39 +01:00
Eike Kettner
3f53779ae4 Change documentation side menu and front 2020-02-28 20:52:39 +01:00
Eike Kettner
ad8d64eded Fix microsite and add changelog 2020-02-27 23:59:03 +01:00
Eike Kettner
1bb464b9ed Extend tools/ds.sh to check for file existence 2020-02-27 20:03:46 +01:00
Eike Kettner
902fd63125 Fix initializing concerned equipment 2020-02-26 20:43:16 +01:00
Eike Kettner
2f87065b2e sbt scalafmtAll 2020-02-25 20:55:00 +01:00
Eike Kettner
c8d090ae28 Remove small notes form field in favor for the new one 2020-02-24 22:34:32 +01:00
Eike Kettner
381de1e198 Show project version in the documentation 2020-02-24 20:59:15 +01:00
Eike Kettner
25c3f2b541 Add more explaining tooltips 2020-02-24 15:18:42 +01:00
Eike Kettner
478797e2a4 Add a help link to the main menu 2020-02-24 15:11:58 +01:00
Eike Kettner
36093c5d52 Add reverse proxy doc 2020-02-24 15:11:50 +01:00
Eike Kettner
cc16b0c024 Fix query to also work with mariadb 2020-02-24 13:34:54 +01:00
Eike Kettner
5f32eadaba Fix dropdown in source create view 2020-02-23 23:01:48 +01:00
Eike Kettner
661cc3e65f Fix deleting attachments (again) 2020-02-23 20:18:13 +01:00
Eike Kettner
d937e0501a Add source files to collective insights 2020-02-23 20:17:53 +01:00
Eike Kettner
be8eacdbe9 Display full attachment name in title tooltip 2020-02-23 14:36:19 +01:00
Eike Kettner
1f431c3222 Make link to original file enabled if files are different 2020-02-23 14:33:22 +01:00
Eike Kettner
957073fe62 Return info about original files in item detail 
This adds data to the current rest api.
 2020-02-23 14:25:32 +01:00
Eike Kettner
ec419c7bfd Adopt nix modules to new config 2020-02-22 12:40:56 +01:00
Eike Kettner
74a037887d Fix deleting items and attachments to also remove the binary files 2020-02-22 00:54:55 +01:00
Eike Kettner
8cfecfb3dd Update docs 2020-02-22 00:48:58 +01:00
Eike Kettner
98576a5fb5 Add link to original file 2020-02-20 22:40:27 +01:00
Eike Kettner
72fd3b1a25 Implement downloading original file 2020-02-20 22:33:57 +01:00
Eike Kettner
39809f9d05 Sketch route for retrieving original file 2020-02-20 22:12:27 +01:00
Eike Kettner
7fe8843893 Update documentation sites 2020-02-20 21:43:37 +01:00
Eike Kettner
3f316ab4d0 Update config file doc 2020-02-20 21:10:00 +01:00
Eike Kettner
fbe0c1aec5 Allow more chars for mimetype 2020-02-20 00:39:31 +01:00
Eike Kettner
97305d27ff Integrate support for more files into processing and upload 
The restriction that only pdf files can be uploaded is removed. All
files can now be uploaded. The processing may not process all. It is
still possible to restrict file uploads by types via a configuration.
 2020-02-19 23:27:00 +01:00
Eike Kettner
9b1349734e Convert some files to pdf 2020-02-19 02:03:10 +01:00
Eike Kettner
5869e2ee6e Streamline extern-conv stdin/infile 2020-02-18 12:43:47 +01:00
Eike Kettner
0dcc00836b Make logger configurable in system commands 2020-02-18 12:02:43 +01:00
Eike Kettner
bd605b8c94 Add first drafts for converting 2020-02-18 01:31:22 +01:00
Eike Kettner
c665c212a0 Early draft for running wkhtmltopdf 2020-02-17 14:02:23 +01:00
Eike Kettner
e0682464b5 Configure pdf extraction; move Logger and DataType to common 2020-02-17 14:01:36 +01:00
Eike Kettner
3d615181e0 Early draft for text extraction 2020-02-17 01:57:22 +01:00
Eike Kettner
8143a4edcc Adding extraction primitives 2020-02-16 21:37:26 +01:00
Eike Kettner
851ee7ef0f Reorganize processing code 
Use separate modules for

- text extraction
- conversion to pdf
- text analysis
 2020-02-15 21:25:25 +01:00
Eike Kettner
919381be1e More research on how to create pdfs from other files 2020-02-15 13:57:21 +01:00
Eike Kettner
3deba44282 Rename example files 2020-02-15 12:52:24 +01:00
Eike Kettner
1309c8b7fa Move mimetype detection to docspell-files 2020-02-14 22:06:18 +01:00
Eike Kettner
5c3d2b2e28 Rename example-files to files 2020-02-14 11:14:09 +01:00
Eike Kettner
bf9bf25502 Rename example files 2020-02-14 11:10:54 +01:00
Eike Kettner
569aae3038 Add example files into its own project 
The text and convert module can use them in their tests.
 2020-02-11 22:46:23 +01:00
Eike Kettner
2c0425433e Move File class to common module 2020-02-11 22:42:04 +01:00
Eike Kettner
3026f199f7 Some research on pdf conversion 2020-02-11 22:41:44 +01:00
Eike Kettner
ce22b727b1 Add new convert module and sketch its integration 2020-02-11 00:33:52 +01:00
Eike Kettner
3be90d64d5 Move SystemCommand to common module 2020-02-10 22:23:06 +01:00
Eike Kettner
ba3865ef5e Starting to support more file types 
First, files are be converted to PDF for archiving. It is also easier
to create a preview. This is done via the `ConvertPdf` processing
task (which is not yet implemented).

Text extraction then tries first with the original file. If that
fails, OCR is done on the (potentially) converted pdf file.

To not loose information of the original file, it is saved using the
table `attachment_source`. If the original file is already a pdf, or
the conversion did not succeed, the `attachment` and
`attachment_source` record point to the same file.
 2020-02-10 12:42:45 +01:00
Eike Kettner
5c37efeaba Apply scalafmt to all files 2020-02-09 01:54:26 +01:00
Eike Kettner
533396d386 Using the new preview route to show the attachment in webui 2020-02-08 18:02:31 +01:00
Eike Kettner
8908ad2561 Add attachment preview url based on ViewerJS 
The viewerJS library can display PDF files easily using pdfjs. Another
attachment route redirects to the viewerjs application to display the
current attachment.

The attachment responses have been improved in that now the response
headers are added to all responses. Additional a HEAD route has been
added to support the viewerJS application.
 2020-02-08 18:02:31 +01:00
Eike Kettner
e1826f39ac Disable revolver plugin on non-app projects 
This allows to type `reStart` in the root sbt project to start both
applications.
 2020-02-08 18:02:31 +01:00
Eike Kettner
9b66604b96 Include item notes in search 2020-02-08 13:39:06 +01:00
Eike Kettner
d2edddd238 Show attachment meta data in ui 
Allow to view the extracted text and results from text analysis of an
attachment.
 2020-02-08 12:23:59 +01:00
Eike Kettner
f8aa5c28ac Update http4s to 0.21.0-RC3, fs2 to 2.2.2 2020-02-04 22:14:18 +01:00
Eike Kettner
c9c8672234 Fix line-breaks in mail body 2020-02-02 12:25:15 +01:00
Eike Kettner
518d6911f0 Edit notes in a larger area 2020-01-29 21:57:02 +01:00
Eike Kettner
c504a3df42 Fix elm-analyse issues 2020-01-29 20:56:14 +01:00
Eike Kettner
1c8a143623 Add a complete example for nixos 2020-01-24 23:12:08 +01:00
Eike Kettner
61bbdab8b5 nix: add user doc and pkg fixes 
- Add user doc for how to use with nix/nixos

- fix potential collisions in packages if both are installed via
  `nix-env`
 2020-01-24 21:56:48 +01:00
Eike Kettner
23af8acff8 Add support for integrating into nix/nixos 2020-01-20 00:21:15 +01:00
Eike Kettner
2454f358b1 Add sbt task to create a zip for things in tools/ 2020-01-19 20:32:52 +01:00
Eike Kettner
8f7e8c7800 Add redirect for root (/) to gui (/app) 2020-01-18 17:48:45 +01:00
Eike Kettner
1c13537f47 Set version to 0.3.0-SNAPSHOT 2020-01-12 15:36:09 +01:00
Eike Kettner
ab045b0ce6 Set version to 0.2.0 2020-01-12 13:58:04 +01:00
Eike Kettner
d8da55fa23 Update readme + fix typo 2020-01-12 13:56:50 +01:00
Eike Kettner
e8122e5f35 Fix input width for multi dropdown and recipients 2020-01-12 13:37:56 +01:00
Eike Kettner
4b4d236ca4 Show loader while sending mail 2020-01-12 12:31:31 +01:00
Eike Kettner
d057d4de7c Shorten attachment filenames in tab view 2020-01-12 11:56:31 +01:00
Eike Kettner
8058bdd398 Fix textarea issue 2020-01-12 10:58:48 +01:00
Eike Kettner
6d0c140e8e Add mariadb database migration 2020-01-12 01:17:49 +01:00
Eike Kettner
c6f3920351 Streamline query parameters 2020-01-12 01:13:03 +01:00
Eike Kettner
d535130c9e Provide email proposals from address book 2020-01-12 01:04:42 +01:00
Eike Kettner
c84a69aa9c Update documentation 2020-01-11 22:12:51 +01:00
Eike Kettner
bb1f948f25 Fix search menu width 2020-01-11 20:22:51 +01:00
Eike Kettner
c15701a21a Use PasswordInput in change password form 2020-01-11 18:55:33 +01:00
Eike Kettner
6e56aad251 Sow sent mails in item detail 2020-01-11 18:15:08 +01:00
Eike Kettner
88efe13209 Fix item route responses 
Also avoid storing empty strings in a nullable field.
 2020-01-11 12:58:04 +01:00
Eike Kettner
2ecfb679d9 Add routes to retrieve sent mails 2020-01-11 12:58:04 +01:00
Eike Kettner
b795a22992 Send mails for items 2020-01-10 00:45:29 +01:00
Eike Kettner
2d69d39dd1 Connect multiple items to a mail 2020-01-09 18:20:59 +01:00
Eike Kettner
7a3289c41d Prepare sending mail 2020-01-08 22:44:34 +01:00
Eike Kettner
51ce48997c Start with send-mail widget 2020-01-08 20:51:57 +01:00
Eike Kettner
4490a444a9 Allow dots in identifiers 2020-01-07 00:20:41 +01:00
Eike Kettner
32050a9faf Finish mail settings 2020-01-07 00:20:28 +01:00
Eike Kettner
f235f3a030 Starting with mail functionality 2020-01-05 23:23:28 +01:00
Eike Kettner
2e3454c7a1 Starting with mail settings 2020-01-05 15:31:32 +01:00
Eike Kettner
9020d9aa3b Don't require a prefix when configuring byte arrays 2020-01-05 15:29:58 +01:00
Eike Kettner
993a3d05e2 Fix redirection after login 2020-01-03 20:42:38 +01:00
Eike Kettner
06d84550f5 Fix documentation for ds.sh 2020-01-03 18:17:48 +01:00
Eike Kettner
6eaac50e63 Save search query state 
For when coming back from detail view.
 2020-01-03 00:22:07 +01:00
Eike Kettner
9a300e213d Correct secondary menu width 2020-01-02 23:46:56 +01:00
Eike Kettner
d90afa30aa Add search field to equipment view 2020-01-02 21:16:33 +01:00
Eike Kettner
4e7315c2a9 Add search field to tag view 2020-01-02 20:48:23 +01:00
Eike Kettner
7ba413e49f Add search field to person view 2020-01-02 20:42:51 +01:00
Eike Kettner
906ee9034e Add search field to organization view 2020-01-02 20:21:54 +01:00
Eike Kettner
8814de3c38 Allow simple search when listing meta data 2020-01-02 20:21:49 +01:00
Eike Kettner
eb6c483ef0 Add route to check for files by their checksum 
Adopt scripts in `tools/` to check for existing files using these
routes.
 2019-12-31 23:45:02 +01:00
Eike Kettner
d05e919eb4 Update doobie, use legacy java.time conversions 2019-12-31 13:55:09 +01:00
Eike Kettner
65641d1347 Add max-height for features image 2019-12-31 01:51:42 +01:00
Eike Kettner
7d047b13c9 Add reset search form button 2019-12-31 01:39:55 +01:00
Eike Kettner
c73cdd82ab Improve item detail view 
- Separate page (permalink) for item details

- Use available space and hide search menu

- Disable item navigation links if there is nothing to go to

- Show notes more prominently and allow to hide them
 2019-12-31 01:22:40 +01:00
Eike Kettner
36a6fdd746 Change webapp url paths 
Don't use `#`, because many people find it awkward. Now "normal" urls
are used and elm takes care to not issue a server request when these
change.
 2019-12-30 22:28:02 +01:00
Eike Kettner
b15b9cc217 Streamline routing 
- put openapi doc behind `api/doc` instead of `app/doc`
- don't require `index.html` for the webapp
 2019-12-30 22:03:25 +01:00
Eike Kettner
fc3e22e399 Apply scalafmt to all files 2019-12-30 21:44:13 +01:00
Eike Kettner
57e274e2b0 Upgrade microsite 2019-12-30 02:33:46 +01:00
Eike Kettner
2001cca88b Using elm-format for all files 2019-12-29 21:55:12 +01:00
Eike Kettner
a9e70401de Update dependencies 2019-12-28 12:38:11 +01:00
Eike Kettner
07a23b9611 Fix percent encoding 
Must use utf8 bytes, of course…
 2019-12-11 21:56:31 +01:00
Eike Kettner
2b10652082 Allow to reuse invitation keys on error 2019-11-17 17:13:39 +01:00
Eike Kettner
852d292d9b Add elm-analyse file and fix a few warnings 2019-11-17 00:46:34 +01:00
Eike Kettner
f747c6146d Use same meta data for creating and deleting cookie 
The cookie might not be removed by browsers, if these infos don't
match.
 2019-11-17 00:44:42 +01:00
Eike Kettner
fd311b9688 Disable autocomplete on some input fields 2019-10-28 18:05:34 +01:00
Eike Kettner
2ad1586d00 Set stricter compile options and fix cookie data 2019-09-28 22:17:45 +02:00
Eike Kettner
a07626533d Set version to 0.2.0-SNAPSHOT 2019-09-21 22:36:22 +02:00
Eike Kettner
7b5edb5a89 Set version to 0.1.0 2019-09-21 22:16:31 +02:00
Eike Kettner
4562786206 Fix path to docspell assets 2019-09-21 22:14:02 +02:00
Eike Kettner
3eb9916f1a Add a firefox add-on 2019-09-21 22:02:37 +02:00
Eike Kettner
30b5c9d7f8 Add new a tool ds.sh 
A commandline script to upload given files to docspell. The URLs can
be configured using a configuration file.
 2019-09-21 22:02:37 +02:00
Eike Kettner
c76af5cbf4 New logo 2019-09-21 22:02:36 +02:00
Eike Kettner
6ce7096af0 Add consumedir.sh script 
Script for watching a directory for new files and uploading them to
docspell.
 2019-09-21 22:02:36 +02:00
Eike Kettner
831cd8b655 Initial version. 
Features:

- Upload PDF files let them analyze

- Manage meta data and items

- See processing in webapp
 2019-09-21 22:02:36 +02:00
Eike Kettner
6154e6a387 Initial application stub 2019-09-21 14:54:03 +02:00