Eike Kettner
6a63694a3e
Convert unit tests to munit
2021-03-10 19:48:56 +01:00
Eike Kettner
9013d9264e
Add more convenient date parsers and some basic macros
2021-03-01 00:51:01 +01:00
Eike Kettner
e9ed998e3a
Basic poc to search via custom query
2021-03-01 00:51:01 +01:00
Eike Kettner
186014a1c6
Refactor search to separate between a base query and user query
...
The `findBase` is adding only strictly required conditions. Everything
else comes from the user.
2021-03-01 00:51:01 +01:00
Eike Kettner
e6d9ce2c37
Remove obsolete type capabilities
...
These are now detected by the new scala compiler and lead to compile
errors.
2021-03-01 00:16:30 +01:00
Eike Kettner
d7bc963450
Cleanup nodes that are not reachable anymore
2021-02-18 00:37:18 +01:00
Eike Kettner
48eee00c0b
Allow person to be correspondent, concerning or both
2021-02-16 22:49:55 +01:00
Eike Kettner
d99ce76d89
Remove person suggestion if it doesn't match with organization
2021-02-16 00:29:54 +01:00
Eike Kettner
dd935454c9
First version of new ui based on tailwind
...
This drops fomantic-ui as css toolkit and introduces tailwindcss. With
tailwind there are no predefined components, but it's very easy to
create those. So customizing the look&feel is much simpler, most of
the time no additional css is needed.
This requires a complete rewrite of the markup + styles. Luckily all
logic can be kept as is. The now old ui is not removed, it is still
available by using a request header `Docspell-Ui` with a value of `1`
for the old ui and `2` for the new ui.
Another addition is "dev mode", where docspell serves assets with a
no-cache header, to disable browser caching. This makes developing a
lot easier.
2021-02-14 01:46:13 +01:00
Eike Kettner
96612e0e59
Refactor scan mailbox form and add flag for post-processing
...
Mails are filtered once by using an imap search and then by some globs
to filter files and subjects. Imap can search by subject via a
string-contains, but not via globs or patterns (afaik). The subject
filter is applied to all downloaded mail headers. Now for post
processing (moving to some target folder or deleting), it can be
chosen to post-process all "seen" mails or only those that matched all
filters.
2021-01-24 01:46:31 +01:00
Eike Kettner
c7e850116f
Make the text length limit optional
2021-01-22 23:06:50 +01:00
Eike Kettner
4cba96f390
Always return classifier results as suggestion
...
The classifier results are spliced into the suggestion list at second
place. When linking they are only used if nlp didn't find anything.
2021-01-21 21:05:28 +01:00
Eike Kettner
9957c3267e
Add constraints from config to classifier training
...
For large and/or many documents, training the classifier can lead to
OOM errors. Some limits have been set by default.
2021-01-21 17:46:39 +01:00
Eike Kettner
a6c31be22f
Update documentation
2021-01-20 22:47:15 +01:00
Eike Kettner
85ddc61d9d
Move date proposal setting to nlp config
2021-01-20 19:17:29 +01:00
Eike Kettner
b12d965223
Improve logging
2021-01-20 00:40:58 +01:00
Eike Kettner
27c24c128d
Store tags guessed with classifier in database
2021-01-20 00:30:40 +01:00
Eike Kettner
9d83cb7fe4
Store item based proposals in separate table
...
Classifier don't work on each attachment, but on all. So the results
must not be stored at an attachment. This reverts some previous
changes to put the classifier results for item entities into its own
table.
2021-01-19 23:48:09 +01:00
Eike Kettner
75573c905e
Use classifier results as fallback when linking proposed metadata
2021-01-19 23:13:34 +01:00
Eike Kettner
8455d1badf
Lookup results from classifier
...
The model may be out of date, data may change. Then it should be
looked up to fetch the id to be compatible with next stages.
2021-01-19 22:56:01 +01:00
Eike Kettner
1cd3441462
Run classifier for item entities (concerned, correspondent)
...
Store the results separately from nlp results in attachment metadata.
2021-01-19 22:08:29 +01:00
Eike Kettner
5c487ef7a9
Refactor running classifier in text analysis
2021-01-19 21:30:02 +01:00
Eike Kettner
99dcaae66b
Learn classifiers for item entities
...
Learns classifiers for concerned and correspondent entities. This can
be used as an alternative to or after nlp.
2021-01-19 20:54:47 +01:00
Eike Kettner
a6f29153c4
Control what tag categories to use for auto-tagging
2021-01-19 01:20:13 +01:00
Eike Kettner
cce8878898
Exclude tags w/o category from classifying; remove obsolete models
2021-01-18 21:51:49 +01:00
Eike Kettner
249f9e6e2a
Extend guessing tags to all tag categories
2021-01-18 21:51:45 +01:00
Eike Kettner
360cad3304
Refactoring solr/fts migration
...
When re-indexing everything, skip intermediate populating the index
and do this as the very last step.
Parameterize adding new fields by their language.
2021-01-18 17:41:40 +01:00
Eike Kettner
f01646aeb5
Reorganize nlp pipeline and add nlp-unsupported language italian
...
Improves and reorganizes how nlp pipelines are setup. Now users can
choose from many options, depending on their hardware and usage
scenario.
This is the base to use more languages without depending on what
stanford-nlp supports. Support then is involves to text extraction and
simple regex-ner processing.
2021-01-18 17:41:40 +01:00
Eike Kettner
a70e9ab614
Store used language for processing on attachmentmeta
...
Issue: #570
2021-01-17 22:56:33 +01:00
Eike Kettner
aa937797be
Choose nlp mode in config file
2021-01-17 22:56:33 +01:00
Eike Kettner
a699e87304
Separate ner from classification
2021-01-17 22:56:33 +01:00
Eike Kettner
f02f15e5bd
Move blocker into constructor of text analyser
2021-01-17 22:56:33 +01:00
Eike Kettner
d77b5855e4
Set default pool-size to 1
2021-01-11 22:30:59 +01:00
Eike Kettner
bddafa7d28
Fix looping over already seen mails when they are skipped
...
When skipping mails due to a filter, it must still enter the
post-handling step. Otherwise it will be seen again on next run.
Issue: #551
2021-01-09 15:07:18 +01:00
Eike Kettner
d712f8303d
Make glob matching case-insensitive by default
2021-01-09 13:23:15 +01:00
Eike Kettner
a670bbb6c2
Make idle interval when clearing nlp cache configurable
2021-01-06 23:03:00 +01:00
Eike Kettner
b08e88cd69
Add (inofficial) routes to get system information
2021-01-05 20:54:53 +01:00
Eike Kettner
611e480eb4
Use more prominent log line to indicate start of processing
...
Issue: #530
2021-01-02 21:47:54 +01:00
Eike Kettner
97dfcece97
Fix duplicate check on restarts
...
Issue: #530
2021-01-02 21:18:05 +01:00
Eike Kettner
2dff686fa0
Introduce unit condition
2020-12-15 21:03:47 +01:00
Eike Kettner
80406cabc2
Refactoring some code into separate files
2020-12-15 21:03:47 +01:00
Eike Kettner
5e2c5d2a50
Extends query builder
2020-12-15 21:03:46 +01:00
Eike Kettner
35c62049f5
Start converting QItem
2020-12-15 21:03:46 +01:00
Eike Kettner
613696539f
Minor refactorings
2020-12-15 21:03:46 +01:00
Eike Kettner
e3f6892abd
Convert job record
2020-12-15 21:03:46 +01:00
Eike Kettner
3cef932ccd
Convert more records
2020-12-15 21:03:46 +01:00
Eike Kettner
10b49fccf8
Converting user and userimap records
2020-12-15 21:03:46 +01:00
Eike Kettner
f5ae389eea
Cleanup remember-me tokens periodically
2020-12-04 17:59:25 +01:00
Eike Kettner
290989f67f
Reorder correspondent person suggestion based on org relationship
2020-12-01 23:39:45 +01:00
Eike Kettner
3fabe0a582
Update to Scala 2.13.4
2020-11-27 20:26:24 +01:00
