TheAnachronism/docspell
1
0
Fork 0
mirror of https://github.com/TheAnachronism/docspell.git synced 2024-11-13 02:31:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
NaN Commits 2 Branches 1 Tag 89 MiB
d737da768e
Commit Graph

909 Commits

Author SHA1 Message Date
Eike Kettner
394aeeccb6 Introduce a sql literal and constants in query builder 
The h2 jdbc driver could not translate the union query in QCollective
when the `kind` was set via a constant value. Using literals works
here. Renamed the corresponding elements in the query builder.
 2021-01-25 00:18:24 +01:00
mergify[bot]
6cc9c159d6
Merge pull request #590 from eikek/scan-mailbox-filter 
Refactor scan mailbox form and add flag for post-processing
 2021-01-24 01:06:51 +00:00
Eike Kettner
96612e0e59 Refactor scan mailbox form and add flag for post-processing 
Mails are filtered once by using an imap search and then by some globs
to filter files and subjects. Imap can search by subject via a
string-contains, but not via globs or patterns (afaik). The subject
filter is applied to all downloaded mail headers. Now for post
processing (moving to some target folder or deleting), it can be
chosen to post-process all "seen" mails or only those that matched all
filters.
 2021-01-24 01:46:31 +01:00
Eike Kettner
1b66e2af5c Fix classifier_settings table 2021-01-23 21:30:26 +01:00
Eike Kettner
c7e850116f Make the text length limit optional 2021-01-22 23:06:50 +01:00
mergify[bot]
8dd1672c8c
Merge pull request #583 from eikek/fix-baseurl-setting 
Render baseurl without trailing slash
 2021-01-21 23:44:14 +00:00
mergify[bot]
38e0a50942
Merge pull request #582 from eikek/delete-org-fix 
Fix deleting organization
 2021-01-21 22:56:56 +00:00
Eike Kettner
0ec620fcf0 Render baseurl without trailing slash 
The webapp expects it like this currently, because the url is only a
string.
 2021-01-21 21:42:08 +01:00
Eike Kettner
f4a03e7c69 Fix deleting organization 
The foreign key in person must be resetted.
 2021-01-21 21:27:02 +01:00
Eike Kettner
4cba96f390 Always return classifier results as suggestion 
The classifier results are spliced into the suggestion list at second
place. When linking they are only used if nlp didn't find anything.
 2021-01-21 21:05:28 +01:00
Eike Kettner
9957c3267e Add constraints from config to classifier training 
For large and/or many documents, training the classifier can lead to
OOM errors. Some limits have been set by default.
 2021-01-21 17:46:39 +01:00
Eike Kettner
363cf5aef0 Quote names in sql changesets 2021-01-21 00:22:58 +01:00
Eike Kettner
38387e00a0 Fix mariadb migration 2021-01-21 00:22:53 +01:00
Eike Kettner
a6c31be22f Update documentation 2021-01-20 22:47:15 +01:00
Eike Kettner
85ddc61d9d Move date proposal setting to nlp config 2021-01-20 19:17:29 +01:00
Eike Kettner
5d366c3bd6 Make labels in classifier settings more clear 2021-01-20 01:05:59 +01:00
Eike Kettner
b12d965223 Improve logging 2021-01-20 00:40:58 +01:00
Eike Kettner
27c24c128d Store tags guessed with classifier in database 2021-01-20 00:30:40 +01:00
Eike Kettner
9d83cb7fe4 Store item based proposals in separate table 
Classifier don't work on each attachment, but on all. So the results
must not be stored at an attachment. This reverts some previous
changes to put the classifier results for item entities into its own
table.
 2021-01-19 23:48:09 +01:00
Eike Kettner
3ff9284a64 Return classifier results as suggestions 2021-01-19 23:13:51 +01:00
Eike Kettner
75573c905e Use classifier results as fallback when linking proposed metadata 2021-01-19 23:13:34 +01:00
Eike Kettner
8455d1badf Lookup results from classifier 
The model may be out of date, data may change. Then it should be
looked up to fetch the id to be compatible with next stages.
 2021-01-19 22:56:01 +01:00
Eike Kettner
1cd3441462 Run classifier for item entities (concerned, correspondent) 
Store the results separately from nlp results in attachment metadata.
 2021-01-19 22:08:29 +01:00
Eike Kettner
d124f0c1a9 Rename db changeset 
It's not just a fix, but adds new things
 2021-01-19 22:08:29 +01:00
Eike Kettner
5c487ef7a9 Refactor running classifier in text analysis 2021-01-19 21:30:02 +01:00
Eike Kettner
99dcaae66b Learn classifiers for item entities 
Learns classifiers for concerned and correspondent entities. This can
be used as an alternative to or after nlp.
 2021-01-19 20:54:47 +01:00
Eike Kettner
a6f29153c4 Control what tag categories to use for auto-tagging 2021-01-19 01:20:13 +01:00
Eike Kettner
cce8878898 Exclude tags w/o category from classifying; remove obsolete models 2021-01-18 21:51:49 +01:00
Eike Kettner
3e28ce1254 Add the sql concat function to query builder 2021-01-18 21:51:45 +01:00
Eike Kettner
249f9e6e2a Extend guessing tags to all tag categories 2021-01-18 21:51:45 +01:00
Eike Kettner
3f75af0807 Add 9 more lanugages to the list of document lanugages 2021-01-18 17:41:40 +01:00
Eike Kettner
94bb18c152 Refactor solr language fields 2021-01-18 17:41:40 +01:00
Eike Kettner
26dff18ae0 Add spanish as an example 
Adding a new language without nlp requires now only to fill out the
pieces:

- define a list of month names to support date recognition
- add it to joex' dockerfile to be available for tesseract
- update the solr migration/field definitions
- update the elm file so it shows up on the client
 2021-01-18 17:41:40 +01:00
Eike Kettner
360cad3304 Refactoring solr/fts migration 
When re-indexing everything, skip intermediate populating the index
and do this as the very last step.

Parameterize adding new fields by their language.
 2021-01-18 17:41:40 +01:00
Eike Kettner
ff121d462c Disable memory intensive tests on travis 2021-01-18 17:41:40 +01:00
Eike Kettner
f01646aeb5 Reorganize nlp pipeline and add nlp-unsupported language italian 
Improves and reorganizes how nlp pipelines are setup. Now users can
choose from many options, depending on their hardware and usage
scenario.

This is the base to use more languages without depending on what
stanford-nlp supports. Support then is involves to text extraction and
simple regex-ner processing.
 2021-01-18 17:41:40 +01:00
Eike Kettner
a70e9ab614 Store used language for processing on attachmentmeta 
Issue: #570
 2021-01-17 22:56:33 +01:00
Eike Kettner
6cf3f9be5a Fix joex version endpoint in spec 2021-01-17 22:56:33 +01:00
Eike Kettner
aa937797be Choose nlp mode in config file 2021-01-17 22:56:33 +01:00
Eike Kettner
54a09861c4 Use model cache with basic annotator 2021-01-17 22:56:33 +01:00
Eike Kettner
a77f67d73a Make pipeline cache generic to be used with BasicCRFAnnotator 2021-01-17 22:56:33 +01:00
Eike Kettner
4462ebae0f Resurrect the basic ner classifier 2021-01-17 22:56:33 +01:00
Eike Kettner
a699e87304 Separate ner from classification 2021-01-17 22:56:33 +01:00
Eike Kettner
f02f15e5bd Move blocker into constructor of text analyser 2021-01-17 22:56:33 +01:00
Eike Kettner
b2b8ad625a scalafmt 2021-01-17 20:11:58 +01:00
Eike Kettner
f0f0e6e0d4 Search for categories case-insensitive 
The string was already lowercased, but the comparison was not.

Fixes #568
 2021-01-17 20:10:24 +01:00
Eike Kettner
623a61dbb6 Introduce a lowerEq operator to the query builder 2021-01-17 20:10:00 +01:00
Eike Kettner
54bd75e99e Set version to 0.19.0-SNAPSHOT 2021-01-11 23:27:47 +01:00
Eike Kettner
0d1b55a205 Set version to 0.18.0 2021-01-11 22:39:40 +01:00
Eike Kettner
d77b5855e4 Set default pool-size to 1 2021-01-11 22:30:59 +01:00
Eike Kettner
38ae7a9027 Make source a quick link on card and detail 2021-01-11 21:37:36 +01:00
Eike Kettner
33458766fe Correcty reset search menu when clicking on custom-field quick link 2021-01-11 14:03:23 +01:00
Eike Kettner
7beda302b2 Fix and improve tag search menu 
Show also "empty tags", where the count is 0. Before only tags with a
count > 0 were displayed. When searching this is fine, but when using
drag&drop to attach tags to items, it is good to see all. They can be
hidden via a button.

The tags are now ordered by their count descending, but regarding to
the overall count – not the current view. Otherwise the tags are
reordered when clicking on them, which is confusing. Also it then
shows the "more important" (most used) tags first, even when the
result is a subset.

A fix was made related to updating the menu. When coming back from
the detail view where a tag with prior count=0 was associated, the
menu didn't show it, because it relied on a previous state, where this
tag were not included.
 2021-01-11 13:01:38 +01:00
Eike Kettner
3fccc3df39 Return all tags in search stats result 
Before only tags with a count > 0 were included. Now those that have
not attached to any item are returned as well.
 2021-01-11 12:13:13 +01:00
Eike Kettner
75986c461f Fix ner date label boundary reporting 2021-01-10 09:10:39 +01:00
Eike Kettner
fb05e997ab Provide multiple date suggestions for English 
Issue: #561
 2021-01-10 09:02:26 +01:00
Eike Kettner
bddafa7d28 Fix looping over already seen mails when they are skipped 
When skipping mails due to a filter, it must still enter the
post-handling step. Otherwise it will be seen again on next run.

Issue: #551
 2021-01-09 15:07:18 +01:00
Eike Kettner
d712f8303d Make glob matching case-insensitive by default 2021-01-09 13:23:15 +01:00
Eike Kettner
cbca4d234f Fix scrolling to card 
That was broken due to the independent scroll in commit #bcb1b8.
 2021-01-09 02:00:01 +01:00
Eike Kettner
cef1c38cc4 Restrict height of job output 2021-01-09 01:49:55 +01:00
Eike Kettner
0abd7dea10 Fix scrolling to top in detail view 2021-01-09 01:16:59 +01:00
Eike Kettner
c0d7aba5d5 Improve selecting attachments of an item 
Use a list of small thumbnails instead of just names.

Closes: #396
 2021-01-09 01:16:59 +01:00
Eike Kettner
9bc2084499 Allow to click on custom fields in detail view 
Closes: #514
 2021-01-09 01:16:59 +01:00
Eike Kettner
48d182667d Harmonize login and register page 2021-01-09 01:16:59 +01:00
Eike Kettner
752c8f9be2 Show new-invite as normal page 2021-01-09 01:16:59 +01:00
Eike Kettner
bcb1b87fc0 Enable independent scrolling of search menu and list 
Fixes some other minor css issues.

Closes: #541
 2021-01-09 01:16:59 +01:00
Eike Kettner
3c12e3678f Allow to search for * in custom date fields 
This requires to pass the raw input through to the caller.

Closes: #550
 2021-01-09 01:16:59 +01:00
Bo Jeanes
14dacaa837
Fix typo 2021-01-09 10:41:53 +11:00
Eike Kettner
716252721c Fix cache clearing 
It must be cancelled when obtaining a pipeline.
 2021-01-07 23:31:01 +01:00
Eike Kettner
a670bbb6c2 Make idle interval when clearing nlp cache configurable 2021-01-06 23:03:00 +01:00
Eike Kettner
73a9572835 Poc for clearing stanford pipeline after some idle time 2021-01-05 23:56:20 +01:00
Eike Kettner
b08e88cd69 Add (inofficial) routes to get system information 2021-01-05 20:54:53 +01:00
Eike Kettner
30df887934 Sort custom field options in dropdown 2021-01-05 18:04:54 +01:00
Eike Kettner
668abf2140 Add a reset-password admin route 2021-01-04 20:59:31 +01:00
Eike Kettner
2a172ce720 Remove fulltext recreate-key config value 
It's now in the admin routes, protected by the
`admin-endpoint.secret`.
 2021-01-04 15:18:02 +01:00
Eike Kettner
306f064ad9 Create a place for admin routes 
And move re-creation of fulltext index in this place.
 2021-01-04 15:18:02 +01:00
Eike Kettner
0cfd8974d3 Add a flag to imap settings to enable/disable oauth2 scheme 2021-01-04 11:03:04 +01:00
mergify[bot]
84d160aa38
Merge pull request #534 from eikek/duplicate-check 
Fixing find-by-checksum with exclusions
 2021-01-03 11:55:28 +00:00
Eike Kettner
95fd386c14 Fixing find-by-checksum with exclusions 
The NOT-IN query must check for null separately, as everything with
null evaluates to false in sql resulting in not finding existing
duplicates.
 2021-01-03 12:29:03 +01:00
Eike Kettner
40b093ae72 Fix selecting imap connection in scan mailbox settings 
Issue: #524
 2021-01-03 10:31:36 +01:00
Eike Kettner
611e480eb4 Use more prominent log line to indicate start of processing 
Issue: #530
 2021-01-02 21:47:54 +01:00
Eike Kettner
97dfcece97 Fix duplicate check on restarts 
Issue: #530
 2021-01-02 21:18:05 +01:00
Eike Kettner
2b8ef3c2f7 Allow to hide search stats in user settings 2021-01-02 10:20:18 +01:00
Eike Kettner
ae57881f2d Sort custom fields in views 2021-01-02 10:09:05 +01:00
Eike Kettner
70983f115c Fix custom field summary table for mobile view 2021-01-02 09:55:12 +01:00
Eike Kettner
a9ed0364d2 Fix linking guessed tags 
Since tag names are lower-cased the search must happen lower-cased, too.
 2021-01-02 01:30:31 +01:00
Eike Kettner
36858da624 Fix search condition for empty items set 2020-12-17 23:07:04 +01:00
Eike Kettner
4ec133b0b9 Remove unused imports 2020-12-17 23:06:58 +01:00
Eike Kettner
69f57d1eb1 Replace empty hrefs with a href # 2020-12-17 21:15:33 +01:00
Eike Kettner
c9d4e8ec46 Fix selecting items in multi-edit mode 
For some to me unknown reason, changing the dom slightly (removing
hidden elements), resulted in a different event dispatching. The cards
while being attached to an event would reload the page as if the event
is propagated. This happned by commit #8d7b3c7d in Home/View.elm.
Adding the hidden nodes back into the dom, "fixed" it.

This change now gives a better fix in assuring that every anchor has
either a sensible `href` or an event and a `href #`.
 2020-12-17 21:02:23 +01:00
Eike Kettner
6346bf6a34 Add summary for fulltext searches 2020-12-17 00:11:33 +01:00
Eike Kettner
8d7b3c7d74 Show custom field summary above results 2020-12-16 22:06:34 +01:00
Eike Kettner
b66738b4c3 Add folder count to search menu 2020-12-16 19:20:30 +01:00
Eike Kettner
8fba637ebe Add folder counts to search summary 2020-12-16 01:14:27 +01:00
Eike Kettner
a995ea8729 Update tag counts in search menu 2020-12-16 00:56:12 +01:00
Eike Kettner
80e23d1c84 Add a route to get search summary 2020-12-15 23:37:02 +01:00
Eike Kettner
77627534bc Improve on basic search summary 2020-12-15 23:37:02 +01:00
Eike Kettner
f3855628d5 Extend query builder with more functions 2020-12-15 23:34:12 +01:00
Eike Kettner
4ca6dfccae Get basic search summary 2020-12-15 23:10:13 +01:00
Eike Kettner
56d6d2e2ac Allow changing more parts of a select 2020-12-15 22:12:44 +01:00