TheAnachronism/docspell
1
0
Fork 0
mirror of https://github.com/TheAnachronism/docspell.git synced 2024-11-13 02:31:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
NaN Commits 2 Branches 1 Tag 89 MiB
ec237a2eaa
Commit Graph

263 Commits

Author SHA1 Message Date
Eike Kettner
f01646aeb5 Reorganize nlp pipeline and add nlp-unsupported language italian 
Improves and reorganizes how nlp pipelines are setup. Now users can
choose from many options, depending on their hardware and usage
scenario.

This is the base to use more languages without depending on what
stanford-nlp supports. Support then is involves to text extraction and
simple regex-ner processing.
 2021-01-18 17:41:40 +01:00
Eike Kettner
a70e9ab614 Store used language for processing on attachmentmeta 
Issue: #570
 2021-01-17 22:56:33 +01:00
Eike Kettner
f0f0e6e0d4 Search for categories case-insensitive 
The string was already lowercased, but the comparison was not.

Fixes #568
 2021-01-17 20:10:24 +01:00
Eike Kettner
623a61dbb6 Introduce a lowerEq operator to the query builder 2021-01-17 20:10:00 +01:00
Eike Kettner
3fccc3df39 Return all tags in search stats result 
Before only tags with a count > 0 were included. Now those that have
not attached to any item are returned as well.
 2021-01-11 12:13:13 +01:00
Eike Kettner
0cfd8974d3 Add a flag to imap settings to enable/disable oauth2 scheme 2021-01-04 11:03:04 +01:00
Eike Kettner
95fd386c14 Fixing find-by-checksum with exclusions 
The NOT-IN query must check for null separately, as everything with
null evaluates to false in sql resulting in not finding existing
duplicates.
 2021-01-03 12:29:03 +01:00
Eike Kettner
97dfcece97 Fix duplicate check on restarts 
Issue: #530
 2021-01-02 21:18:05 +01:00
Eike Kettner
a9ed0364d2 Fix linking guessed tags 
Since tag names are lower-cased the search must happen lower-cased, too.
 2021-01-02 01:30:31 +01:00
Eike Kettner
36858da624 Fix search condition for empty items set 2020-12-17 23:07:04 +01:00
Eike Kettner
8fba637ebe Add folder counts to search summary 2020-12-16 01:14:27 +01:00
Eike Kettner
77627534bc Improve on basic search summary 2020-12-15 23:37:02 +01:00
Eike Kettner
f3855628d5 Extend query builder with more functions 2020-12-15 23:34:12 +01:00
Eike Kettner
4ca6dfccae Get basic search summary 2020-12-15 23:10:13 +01:00
Eike Kettner
56d6d2e2ac Allow changing more parts of a select 2020-12-15 22:12:44 +01:00
Eike Kettner
f1c4b4adb0 Extract find-item query condition 2020-12-15 21:03:47 +01:00
Eike Kettner
2dff686fa0 Introduce unit condition 2020-12-15 21:03:47 +01:00
Eike Kettner
80406cabc2 Refactoring some code into separate files 2020-12-15 21:03:47 +01:00
Eike Kettner
278b1c22c9 Remove old code 2020-12-15 21:03:46 +01:00
Eike Kettner
2cecd01837 Convert rest of QItem 2020-12-15 21:03:46 +01:00
Eike Kettner
d1606d6f16 Remove old commented code 2020-12-15 21:03:46 +01:00
Eike Kettner
266fec9eb5 Convert find items query 2020-12-15 21:03:46 +01:00
Eike Kettner
5e2c5d2a50 Extends query builder 2020-12-15 21:03:46 +01:00
Eike Kettner
35c62049f5 Start converting QItem 2020-12-15 21:03:46 +01:00
Eike Kettner
a355767fdb Convert all query libs besides QItem 2020-12-15 21:03:46 +01:00
Eike Kettner
fd6d09587d Convert more records 2020-12-15 21:03:46 +01:00
Eike Kettner
613696539f Minor refactorings 2020-12-15 21:03:46 +01:00
Eike Kettner
d6f28d3eca Convert folder 2020-12-15 21:03:46 +01:00
Eike Kettner
87eb8c7f55 Convert more records 2020-12-15 21:03:46 +01:00
Eike Kettner
e3f6892abd Convert job record 2020-12-15 21:03:46 +01:00
Eike Kettner
1aa1f4367e Convert periodic tasks 2020-12-15 21:03:46 +01:00
Eike Kettner
3cef932ccd Convert more records 2020-12-15 21:03:46 +01:00
Eike Kettner
fe4815c737 Convert RSentMail 2020-12-15 21:03:46 +01:00
Eike Kettner
5cbf0d5602 Convert more records 2020-12-15 21:03:46 +01:00
Eike Kettner
10b49fccf8 Converting user and userimap records 2020-12-15 21:03:46 +01:00
Eike Kettner
c5c7f7ed3b Convert equipment record 2020-12-15 21:03:46 +01:00
Eike Kettner
adee496b77 Convert source record 2020-12-15 21:03:46 +01:00
Eike Kettner
2dbb1db2fd Initial outline for a simple query builder 2020-12-15 21:03:46 +01:00
Eike Kettner
27d087b14c Fix foreign key constraints 2020-12-14 14:34:22 +01:00
Eike Kettner
a0642905db Use remember-me cookie if present 2020-12-04 17:59:25 +01:00
Eike Kettner
c10c1fad72 Prepare remember-me authentication variant 2020-12-04 17:59:25 +01:00
Eike Kettner
290989f67f Reorder correspondent person suggestion based on org relationship 2020-12-01 23:39:45 +01:00
Eike Kettner
fc2668feee Allow to connect a person to an organization 2020-12-01 23:39:45 +01:00
Eike Kettner
0ee8ff66d5 Allow to search by source name 2020-11-30 14:07:45 +01:00
Eike Kettner
3fabe0a582 Update to Scala 2.13.4 2020-11-27 20:26:24 +01:00
Eike Kettner
0919eec3c2 Improve field query and fix mariadb's pickiness with parens 
If no query is given, don't search with `like '%'`. MariaDB doesn't
want parens around columns in the GROUP BY clause.
 2020-11-25 21:08:49 +01:00
Eike Kettner
52c6659f9f Add missing schema migrations for custom fields 2020-11-25 21:08:49 +01:00
Eike Kettner
9bea0298ad Allow to query custom field values with wildcards 2020-11-23 10:59:13 +01:00
Eike Kettner
7b7f1e4d6d Return custom field values with search results 2020-11-23 10:23:25 +01:00
Eike Kettner
066c856981 Allow to search for custom field values 2020-11-22 21:41:09 +01:00
Eike Kettner
1aefff37aa Return custom field values with item details 2020-11-22 21:41:09 +01:00
Eike Kettner
af1cca7d83 Fix condition for deleting custom field value 2020-11-22 21:41:09 +01:00
Eike Kettner
8d35d100d6 Change custom fields for multiple items 2020-11-22 21:41:09 +01:00
Eike Kettner
93295d63a5 Change custom field values for a single item 2020-11-22 21:41:09 +01:00
Eike Kettner
62313ab03a Add and change custom fields 2020-11-22 21:41:09 +01:00
Eike Kettner
248ad04dd0 Prepare custom fields 2020-11-22 21:41:09 +01:00
Eike Kettner
04ba14f802 Amend source form with tags and file-filter 
Allow to define tags and a file filter per source.
 2020-11-12 22:37:28 +01:00
Eike Kettner
10305bc82d Minor improvements 2020-11-09 21:16:53 +01:00
Eike Kettner
29455d638c Add startup task to find page counts of existing files 2020-11-09 20:35:35 +01:00
Eike Kettner
8c08bf233d Amend search results with attachment info 
This uses again another query per item to retrieve some information
about each attachment already in the search results.
 2020-11-09 14:24:28 +01:00
Eike Kettner
a77f34b7ba Add a processing step to retrieve page counts 2020-11-09 11:08:24 +01:00
Eike Kettner
d4bbb936b6 Count preview image sizes in insight data 2020-11-09 09:00:03 +01:00
Eike Kettner
f4e50c5229 Provide endpoints to submit tasks to re-generate previews 
The scaling factor can be given in the config file. When this changes,
images can be regenerated via POSTing to certain endpoints. It is
possible to regenerate just one attachment preview or all within a
collective.
 2020-11-09 09:00:02 +01:00
Eike Kettner
709848244c Create tasks to generate all previews 
There is a task to generate preview images per attachment. It can
either add them (if not present yet) or overwrite them (e.g. some
config has changed).

There is a task that selects all attachments without previews and
submits a task to create it. This is submitted on start automatically
to generate previews for all existing attachments.
 2020-11-08 23:46:02 +01:00
Eike Kettner
eede194352 Fix deleting preview files 2020-11-08 21:27:55 +01:00
Eike Kettner
757ad31165 Add a route to get the item preview 
This is the first available preview of an attachment wrt position. If
all attachments have a preview image, the preview of the first
attachment is returned.
 2020-11-08 15:12:56 +01:00
Eike Kettner
0841a33ae3 Add a table to hold the preview files 2020-11-08 01:25:38 +01:00
Eike Kettner
0461cfefe7 Fix sql error for mariadb <10.4 
MariaDB below 10.4 doesn't support parentheses around selects for
`intersect` and `union`.

https://mariadb.com/kb/en/intersect/#parentheses

Fixes #404
 2020-10-28 22:54:51 +01:00
Eike Kettner
b59696a9d3 Make sure to only remove/retry items in premature states 2020-10-26 23:39:26 +01:00
Eike Kettner
26e89bf84e Edit org/person/equipment of multiple items 2020-10-26 13:35:47 +01:00
Eike Kettner
2e6026b817 Edit dates of multiple items 2020-10-26 13:16:03 +01:00
Eike Kettner
d4043634ac Edit direction of multiple items 2020-10-26 12:48:15 +01:00
Eike Kettner
7ad37c8d26 Editing tags for multiple items 2020-10-26 11:54:04 +01:00
Eike Kettner
3e2d272746 Add unique constraint for equipment names 
Fixes #370
 2020-10-21 22:42:19 +02:00
Eike Kettner
3771587e55 Find duplicate tags without category 2020-10-19 00:30:41 +02:00
Eike Kettner
6a3386ce66 Fix sql comparison with optional values 2020-10-19 00:29:41 +02:00
Eike Kettner
80ddca9aa3 Add counter to joblog for correct log order 
This is to distinguish log entries created at the same time.
 2020-10-02 22:14:30 +02:00
Eike Kettner
d4354b8b49 Skip pdf conversion if a converted file exists 
For images the conversion also returns the extracted text. If this
would have failed to be saved, it is extracted in the following
text-extraction step.
 2020-10-02 17:39:39 +02:00
Eike Kettner
b6f23b038a Fix finding attachments for retries 
The attachments to process again must be searched in sources and
archives, too.
 2020-10-02 17:39:34 +02:00
Eike Kettner
e26d7129e7 Add fix for mariadb text columns 
The `text` data type can only store up to 64kb data. The `mediumtext`
up to 16M and `longtext` up to 4G.

Issue: #297
 2020-10-02 16:50:51 +02:00
Eike Kettner
552cdac1d3 Apply flyway api changes 2020-09-28 15:12:10 +02:00
Eike Kettner
f6f63000be Prepend a duplicate check when uploading files 2020-09-23 23:37:00 +02:00
Eike Kettner
c658677032 Autoformat 2020-09-09 00:29:32 +02:00
Eike Kettner
eb11b33028 Fix mariadb changsets 2020-09-07 20:02:50 +02:00
Eike Kettner
76ccfb8a81 Only learn from confirmed items 
Text classification should only learn from confirmed items. Log if
classification is disabled when processing an item.
 2020-09-07 13:04:40 +02:00
Eike Kettner
cb1a9e0699 Use separate sql migration for h2 2020-09-07 13:04:29 +02:00
Eike Kettner
06879456a6 Change job priority on queue page 2020-09-05 18:50:58 +02:00
Eike Kettner
4309bd8dfd Some cleanup 2020-09-02 21:22:30 +02:00
Eike Kettner
316b490008 Implement learning a text classifier from collective data 2020-09-02 18:28:14 +02:00
Eike Kettner
68bb65572b Integrate learn-classifier task into the app 2020-09-02 18:28:14 +02:00
Eike Kettner
8c4f2e702b Add classifier settings 2020-09-02 18:28:14 +02:00
Eike Kettner
de5b33c40d Add updated column to some tables 2020-08-24 21:30:52 +02:00
Eike Kettner
96d2f948f2 Use collective's addressbook to configure regexner 2020-08-24 14:40:52 +02:00
Eike Kettner
3986487f11 Add api docs and cleanup 2020-08-13 21:22:54 +02:00
Eike Kettner
69674eb485 Improve job-queue query to make sure jobs across all states show up 2020-08-13 01:06:13 +02:00
Eike Kettner
41ea071555 Add a task to convert all pdfs that have not been converted 2020-08-13 01:06:13 +02:00
Eike Kettner
07e9a9767e Add a task to re-process files of an item 2020-08-12 22:29:56 +02:00
Eike Kettner
098e4cf868 Fix uploading to enabled/disabled source endpoints 2020-08-09 09:21:23 +02:00
Eike Kettner
06ad9ac46c Add routes to conveniently set/toggle tags 2020-08-08 15:08:04 +02:00
Eike Kettner
1c8b66194b Add a route to return used tags 
This is part of the `/insights` route without queries for file usage.
 2020-08-08 08:35:35 +02:00