TheAnachronism/docspell
1
0
Fork 0
mirror of https://github.com/TheAnachronism/docspell.git synced 2024-11-13 02:31:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
NaN Commits 2 Branches 1 Tag 89 MiB
ec237a2eaa
Commit Graph

963 Commits

Author SHA1 Message Date
Eike Kettner
18992ee374 Deprecate search endpoints 2021-03-01 15:31:02 +01:00
Eike Kettner
dadab0d308 Implement search by query in endpoints 2021-03-01 15:31:02 +01:00
Eike Kettner
698ff58aa3 Provide a more convenient interface to search 2021-03-01 11:50:07 +01:00
Eike Kettner
e079ec1987 Provide custom error structure for parse failures 2021-03-01 00:51:01 +01:00
Eike Kettner
d737da768e Move to munit in query module 2021-03-01 00:51:01 +01:00
Eike Kettner
9013d9264e Add more convenient date parsers and some basic macros 2021-03-01 00:51:01 +01:00
Eike Kettner
af73b59ec2 Parser improvements 
- default expressions into a and node
- fix parsing string lists that end in whitespace
- fix package names of internal classes
 2021-03-01 00:51:01 +01:00
Eike Kettner
a80d73d5d2 Optimize imports 2021-03-01 00:51:01 +01:00
Eike Kettner
e9ed998e3a Basic poc to search via custom query 2021-03-01 00:51:01 +01:00
Eike Kettner
186014a1c6 Refactor search to separate between a base query and user query 
The `findBase` is adding only strictly required conditions. Everything
else comes from the user.
 2021-03-01 00:51:01 +01:00
Eike Kettner
c3cdec416c Sketching some basic tests 2021-03-01 00:50:52 +01:00
Eike Kettner
be5c7ffb88 First draft of ast and parser 2021-03-01 00:46:57 +01:00
Eike Kettner
74a79a79d9 Initial project setup 2021-03-01 00:46:57 +01:00
Eike Kettner
cfa36a5270 Fix preview png tests 
Outcome was checked manually.
 2021-03-01 00:33:57 +01:00
Eike Kettner
e6d9ce2c37 Remove obsolete type capabilities 
These are now detected by the new scala compiler and lead to compile
errors.
 2021-03-01 00:16:30 +01:00
Eike Kettner
c7d4c77e6d Allow more suggestions for date variants in English 2021-02-26 00:35:17 +01:00
Yumekui
017241bccb
Fix typo 2021-02-22 15:09:10 +00:00
Eike Kettner
cd2f623773 Set version to 0.21.0-SNAPSHOT 2021-02-19 23:32:10 +01:00
Eike Kettner
4d3a25a8c1 Set version to 0.20.0 2021-02-19 22:36:55 +01:00
Eike Kettner
07a061febc Make sure certain styles are not stripped by purge-css 2021-02-19 22:36:55 +01:00
Eike Kettner
211499a7e6 Add bottom border to metadata view 2021-02-19 11:17:35 +01:00
Eike Kettner
d819878974 Re-add the button to create new tags in detail view 2021-02-19 11:07:42 +01:00
mergify[bot]
9156e56d65
Merge pull request #651 from eikek/language-source 
Add language to a source
 2021-02-19 01:30:24 +00:00
Eike Kettner
7ef3185659 Add language to a source 
Allows to define upload urls for different languages.
 2021-02-18 23:34:42 +01:00
Eike Kettner
5c6f57cc6a Trim values of some input data 
Some values don't make sense when padded with whitespace, like
organization name, source ids etc. These are trimmed now when
received.
 2021-02-18 22:07:32 +01:00
Eike Kettner
70fa57587b Fix setting ui theme for anonymous 2021-02-18 22:02:56 +01:00
Eike Kettner
0e9d8f8794 Fix hover when folder is a drop-target 2021-02-18 00:43:15 +01:00
Eike Kettner
d7bc963450 Cleanup nodes that are not reachable anymore 2021-02-18 00:37:18 +01:00
mergify[bot]
b16166f1e9
Merge pull request #645 from eikek/equipment-description 
Equipment description
 2021-02-17 22:17:03 +00:00
Eike Kettner
5181283b1b Add a short-name to organizations 2021-02-17 22:55:35 +01:00
Eike Kettner
20ccdda609 Add a notes field to equipments 2021-02-17 22:39:07 +01:00
Eike Kettner
e07d732193 Show item position in detail view 2021-02-17 22:18:22 +01:00
Eike Kettner
a79fa6f74a Add version info back to frontend 2021-02-17 21:54:53 +01:00
Eike Kettner
e177336155 Better align tags in multi-select field 2021-02-17 00:14:06 +01:00
Eike Kettner
4ef1996137 Fix extracted content border 2021-02-17 00:14:06 +01:00
Eike Kettner
33cfd6df6f Hide Load-more button when there is no more 2021-02-17 00:14:06 +01:00
Eike Kettner
29d80adace Ui updates: place result message below buttons 2021-02-17 00:14:06 +01:00
Eike Kettner
48eee00c0b Allow person to be correspondent, concerning or both 2021-02-16 22:49:55 +01:00
Eike Kettner
d99ce76d89 Remove person suggestion if it doesn't match with organization 2021-02-16 00:29:54 +01:00
Eike Kettner
eb308cfa85 Remove duplicate candidates when creating list of suggestions 2021-02-15 23:56:09 +01:00
Eike Kettner
64bacb665a Show queue counts in side menu 2021-02-15 00:27:55 +01:00
Eike Kettner
07edaaaf5e Try fix attachment selection for firefox 
It works great in chromium, in firefox the border is a bit off.
 2021-02-15 00:11:37 +01:00
Eike Kettner
9c90d7c4c2 Fix min height of tag area in detail 
This avoids "nervous/noisy" ui flickering when changing between items
where one has no tags/fields to show.
 2021-02-15 00:10:38 +01:00
Eike Kettner
a10971607e Always show custom field dropdown in detail mode 
Even if there are no custom fields define, the control is shown now.
The user can create a new one in this view.
 2021-02-14 23:34:18 +01:00
Eike Kettner
dec6148df6 Hide field stats component if there is nothing to show 2021-02-14 23:29:05 +01:00
Eike Kettner
31cb8284a5 Fix formatting icon in top nav bar 2021-02-14 23:22:37 +01:00
Eike Kettner
1dcc3941ed Fix overflowing content in item cards 
When displaying long lines this leads to horizontal scrolling the
complete item list.
 2021-02-14 23:18:18 +01:00
Eike Kettner
f29353adb3 Use different worker cache name per ui 
Avoid to have users clear their browser caches
 2021-02-14 11:29:36 +01:00
Eike Kettner
5461321e59 Remove unused styles and prevent postcss from purging relevant ones 2021-02-14 02:20:01 +01:00
Eike Kettner
f6e5bda2d8 Change default ui to be the new ui 
The old ui is still available, using a header: `Docspell-Ui: 1`. Using
a browser addon a "ui switch" can be easily achieved.
 2021-02-14 01:46:13 +01:00
Eike Kettner
dd935454c9 First version of new ui based on tailwind 
This drops fomantic-ui as css toolkit and introduces tailwindcss. With
tailwind there are no predefined components, but it's very easy to
create those. So customizing the look&feel is much simpler, most of
the time no additional css is needed.

This requires a complete rewrite of the markup + styles. Luckily all
logic can be kept as is. The now old ui is not removed, it is still
available by using a request header `Docspell-Ui` with a value of `1`
for the old ui and `2` for the new ui.

Another addition is "dev mode", where docspell serves assets with a
no-cache header, to disable browser caching. This makes developing a
lot easier.
 2021-02-14 01:46:13 +01:00
Eike Kettner
442b76c5af Setup tailwind with docspell build 2021-02-03 21:35:15 +01:00
Eike Kettner
62f596be35 Set version to 0.20.0-SNAPSHOT 2021-01-25 10:19:00 +01:00
Eike Kettner
73b54addc5 Set version to 0.19.0 2021-01-25 09:21:16 +01:00
Eike Kettner
394aeeccb6 Introduce a sql literal and constants in query builder 
The h2 jdbc driver could not translate the union query in QCollective
when the `kind` was set via a constant value. Using literals works
here. Renamed the corresponding elements in the query builder.
 2021-01-25 00:18:24 +01:00
mergify[bot]
6cc9c159d6
Merge pull request #590 from eikek/scan-mailbox-filter 
Refactor scan mailbox form and add flag for post-processing
 2021-01-24 01:06:51 +00:00
Eike Kettner
96612e0e59 Refactor scan mailbox form and add flag for post-processing 
Mails are filtered once by using an imap search and then by some globs
to filter files and subjects. Imap can search by subject via a
string-contains, but not via globs or patterns (afaik). The subject
filter is applied to all downloaded mail headers. Now for post
processing (moving to some target folder or deleting), it can be
chosen to post-process all "seen" mails or only those that matched all
filters.
 2021-01-24 01:46:31 +01:00
Eike Kettner
1b66e2af5c Fix classifier_settings table 2021-01-23 21:30:26 +01:00
Eike Kettner
c7e850116f Make the text length limit optional 2021-01-22 23:06:50 +01:00
mergify[bot]
8dd1672c8c
Merge pull request #583 from eikek/fix-baseurl-setting 
Render baseurl without trailing slash
 2021-01-21 23:44:14 +00:00
mergify[bot]
38e0a50942
Merge pull request #582 from eikek/delete-org-fix 
Fix deleting organization
 2021-01-21 22:56:56 +00:00
Eike Kettner
0ec620fcf0 Render baseurl without trailing slash 
The webapp expects it like this currently, because the url is only a
string.
 2021-01-21 21:42:08 +01:00
Eike Kettner
f4a03e7c69 Fix deleting organization 
The foreign key in person must be resetted.
 2021-01-21 21:27:02 +01:00
Eike Kettner
4cba96f390 Always return classifier results as suggestion 
The classifier results are spliced into the suggestion list at second
place. When linking they are only used if nlp didn't find anything.
 2021-01-21 21:05:28 +01:00
Eike Kettner
9957c3267e Add constraints from config to classifier training 
For large and/or many documents, training the classifier can lead to
OOM errors. Some limits have been set by default.
 2021-01-21 17:46:39 +01:00
Eike Kettner
363cf5aef0 Quote names in sql changesets 2021-01-21 00:22:58 +01:00
Eike Kettner
38387e00a0 Fix mariadb migration 2021-01-21 00:22:53 +01:00
Eike Kettner
a6c31be22f Update documentation 2021-01-20 22:47:15 +01:00
Eike Kettner
85ddc61d9d Move date proposal setting to nlp config 2021-01-20 19:17:29 +01:00
Eike Kettner
5d366c3bd6 Make labels in classifier settings more clear 2021-01-20 01:05:59 +01:00
Eike Kettner
b12d965223 Improve logging 2021-01-20 00:40:58 +01:00
Eike Kettner
27c24c128d Store tags guessed with classifier in database 2021-01-20 00:30:40 +01:00
Eike Kettner
9d83cb7fe4 Store item based proposals in separate table 
Classifier don't work on each attachment, but on all. So the results
must not be stored at an attachment. This reverts some previous
changes to put the classifier results for item entities into its own
table.
 2021-01-19 23:48:09 +01:00
Eike Kettner
3ff9284a64 Return classifier results as suggestions 2021-01-19 23:13:51 +01:00
Eike Kettner
75573c905e Use classifier results as fallback when linking proposed metadata 2021-01-19 23:13:34 +01:00
Eike Kettner
8455d1badf Lookup results from classifier 
The model may be out of date, data may change. Then it should be
looked up to fetch the id to be compatible with next stages.
 2021-01-19 22:56:01 +01:00
Eike Kettner
1cd3441462 Run classifier for item entities (concerned, correspondent) 
Store the results separately from nlp results in attachment metadata.
 2021-01-19 22:08:29 +01:00
Eike Kettner
d124f0c1a9 Rename db changeset 
It's not just a fix, but adds new things
 2021-01-19 22:08:29 +01:00
Eike Kettner
5c487ef7a9 Refactor running classifier in text analysis 2021-01-19 21:30:02 +01:00
Eike Kettner
99dcaae66b Learn classifiers for item entities 
Learns classifiers for concerned and correspondent entities. This can
be used as an alternative to or after nlp.
 2021-01-19 20:54:47 +01:00
Eike Kettner
a6f29153c4 Control what tag categories to use for auto-tagging 2021-01-19 01:20:13 +01:00
Eike Kettner
cce8878898 Exclude tags w/o category from classifying; remove obsolete models 2021-01-18 21:51:49 +01:00
Eike Kettner
3e28ce1254 Add the sql concat function to query builder 2021-01-18 21:51:45 +01:00
Eike Kettner
249f9e6e2a Extend guessing tags to all tag categories 2021-01-18 21:51:45 +01:00
Eike Kettner
3f75af0807 Add 9 more lanugages to the list of document lanugages 2021-01-18 17:41:40 +01:00
Eike Kettner
94bb18c152 Refactor solr language fields 2021-01-18 17:41:40 +01:00
Eike Kettner
26dff18ae0 Add spanish as an example 
Adding a new language without nlp requires now only to fill out the
pieces:

- define a list of month names to support date recognition
- add it to joex' dockerfile to be available for tesseract
- update the solr migration/field definitions
- update the elm file so it shows up on the client
 2021-01-18 17:41:40 +01:00
Eike Kettner
360cad3304 Refactoring solr/fts migration 
When re-indexing everything, skip intermediate populating the index
and do this as the very last step.

Parameterize adding new fields by their language.
 2021-01-18 17:41:40 +01:00
Eike Kettner
ff121d462c Disable memory intensive tests on travis 2021-01-18 17:41:40 +01:00
Eike Kettner
f01646aeb5 Reorganize nlp pipeline and add nlp-unsupported language italian 
Improves and reorganizes how nlp pipelines are setup. Now users can
choose from many options, depending on their hardware and usage
scenario.

This is the base to use more languages without depending on what
stanford-nlp supports. Support then is involves to text extraction and
simple regex-ner processing.
 2021-01-18 17:41:40 +01:00
Eike Kettner
a70e9ab614 Store used language for processing on attachmentmeta 
Issue: #570
 2021-01-17 22:56:33 +01:00
Eike Kettner
6cf3f9be5a Fix joex version endpoint in spec 2021-01-17 22:56:33 +01:00
Eike Kettner
aa937797be Choose nlp mode in config file 2021-01-17 22:56:33 +01:00
Eike Kettner
54a09861c4 Use model cache with basic annotator 2021-01-17 22:56:33 +01:00
Eike Kettner
a77f67d73a Make pipeline cache generic to be used with BasicCRFAnnotator 2021-01-17 22:56:33 +01:00
Eike Kettner
4462ebae0f Resurrect the basic ner classifier 2021-01-17 22:56:33 +01:00
Eike Kettner
a699e87304 Separate ner from classification 2021-01-17 22:56:33 +01:00
Eike Kettner
f02f15e5bd Move blocker into constructor of text analyser 2021-01-17 22:56:33 +01:00
Eike Kettner
b2b8ad625a scalafmt 2021-01-17 20:11:58 +01:00
Eike Kettner
f0f0e6e0d4 Search for categories case-insensitive 
The string was already lowercased, but the comparison was not.

Fixes #568
 2021-01-17 20:10:24 +01:00