Anomaly detection
When AIDA reads a document, it does more than extract the data: it also checks that the data is plausible. If a value does not look convincing, AIDA flags it as an anomaly: the property is shown in yellow, with the icon and a short sentence explaining why it was flagged.
A document with anomalies is not validated automatically, even when an automation is active: AIDA leaves it Ready for validation, so that you can check it before the data is sent to the destinations.
info
An anomaly is not an error: it means "check this value". Often the value is correct and you only need to confirm it.
What to do when you find an anomaly
- Click the yellow property: AIDA shows you where on the document it read the value.
- If the value is correct, press Validate this value.
- If the value is wrong, correct it by typing the right one or by clicking the correct spot on the document.
- Once you have checked all the anomalies, validate the document with the Validate button.
If there are many anomalies, you can confirm them all at once with the button at the top of the property list (Approves all properties with anomalies), or enable Quick Validation.
Correct, don't confirm
What you validate becomes the reference for future documents. If you confirm a wrong value, AIDA will consider it normal and may not flag it the next time.
Value outside of normal history
"Value outside of normal history for this property"
This check detects values that are unexpected for a given document. AIDA does not look for the exact same value in previous documents: it compares the value it read with the ones you have already validated for the same property on similar documents, and warns you when it is too far from what is usual.
As a practical example, if you always receive electricity bills of about €100, a €200 bill will be flagged as an anomaly. Values "around" the previous ones, for example €90 or €120, will not be flagged.
AIDA learns over time. Every validated document becomes part of the history: if the bill increases little by little (€100, €105, €110, €115...), AIDA adapts and flags nothing. Even when you confirm an unusual but correct value, AIDA remembers it, and similar values gradually become normal. For the first documents of a new type AIDA has only a few examples to compare with, so it is normal to see more anomalies at the beginning.
The check also applies to codes and dates. If a VAT number always has 11 digits and AIDA reads a value with a letter or with one digit missing, or if a date is very far from the usual ones, the value is flagged.
When it protects you. Imagine a Shipping costs property that is almost always €0 or €10. If one day AIDA reads €1,250 because it took the number from the box next to it, the anomaly lets you notice before the wrong value reaches your management software.
When it gets in the way. Some properties naturally swing a lot. For example, the taxable amount on a car repair shop's quotes can range from €50 for a bulb replacement to €20,000 for a new engine. In these cases normal learning is not enough, because every document can be very different from the previous one, and a correct but unusual amount will be flagged. For these properties you can set Anomaly handling to Don't block validation.
Low OCR confidence
"Low OCR confidence in reading information off the document"
This anomaly means that AIDA is not sure it read the characters correctly. You read a document by recognising words and numbers, helped by the context. AIDA instead "sees" the document as an image made of dots (pixels), and rebuilds every single character from those dots. This is why a document that looks perfectly readable to you may contain details that make the reading uncertain for AIDA.
| What you see | What AIDA sees |
|---|---|
| A stamp or a signature across an amount | Part of the digits is covered: an 8 may look like a 3 |
| A slightly blurred, skewed or low-resolution scan | Faded, uncertain character outlines |
| A photo taken with a smartphone | Shadows, glare and out-of-focus areas |
| A document printed with low toner or with a dot-matrix printer | Broken characters, or characters made of dots |
| A fold in the paper, or a table line touching the text | Extra marks that blend with the characters |
| Text on a coloured or patterned background | Little contrast between text and background |
| Similar characters, such as 0 and O, 1 and I, 5 and S, comma and period | Nearly identical shapes, hard to tell apart without context |
This anomaly cannot be "learned". The history helps AIDA understand whether a value is plausible, but it cannot make an image sharper: every document is a new image, with its own defects. By confirming the value you fix that document, but if the next one arrives with the same stamp across the total, AIDA will have the same doubt again.
What you can do:
- When possible, use the original PDF generated by the sender's software instead of a scan or a photo.
- If you scan paper documents, use a resolution of at least 300 dpi and lay the sheet flat.
- Avoid stamps, signatures and notes over the data you need.
- If a sender always sends poor-quality documents and the property is not critical, consider setting Anomaly handling to Don't block validation.
Settings
Anomaly detection level
It is in the General configuration and applies to all document types.
| Level | What AIDA checks |
|---|---|
| Off | Nothing: all values are accepted as they were read |
| OCR only | Only the quality of the character reading. The history check is disabled |
| Default | The reading quality and values far from the usual ones. This is the recommended level |
| High | Like Default, but also flags small deviations. Useful when values are very regular and you want maximum control, but it increases the documents to check |
danger
With OCR only and Off, AIDA can no longer notice values read from the wrong spot of the document, as in the shipping costs example. If a few properties generate too many anomalies, it is safer to set Anomaly handling to Don't block validation on those properties only, and leave the level on Default.
Anomaly handling on a property
It is in the property configuration, in the Advanced configuration section, and decides what the anomalies of that property do.
| Level | What happens |
|---|---|
| Default | The property follows the anomaly detection level: its anomalies are shown in yellow and block the automatic validation of the document |
| Don't block validation | The anomalies are still shown in yellow, so you always know that AIDA had a doubt, but they no longer block the automatic validation of the document |
| Off | AIDA stops reporting anomalies on this property, whatever the detection level. A misread value looks correct, and no trace of the anomaly remains on the document |
With Off, values flagged by formulas or by AIDA Boost can still be shown in yellow, but they do not block the automatic validation.
Use Don't block validation on properties that naturally change a lot from one document to the next, for example:
- amounts on quotes, orders or invoices with very different values (taxable amount, VAT, total);
- quantities and mileages;
- codes that always change, such as chassis or serial numbers, if you often receive poor-quality documents.
Choose Off only when the yellow itself is the problem, for example on a property nobody checks, whose anomalies only distract from the ones that matter. With Off a reading error reaches the destinations without any warning.
Leave Default on properties with stable values, such as a fixed cost, a tax rate or a supplier code: on these, a value different from the usual one almost always means a reading error, and the anomaly is exactly what you need.
note
These levels do not change properties that require manual validation: they still need to be validated by hand. They also do not replace required values: if a required property is missing, the document still needs to be validated.
Quick Validation
It is in the General configuration. When enabled, validating a document with anomalies confirms them all at once, after a final confirmation request, without having to approve them one by one. It is handy when you check documents at a glance, but remember that it also confirms any wrong values you did not correct.
Mandatory manual validation
This is the opposite of Don't block validation: it requires you to validate the value by hand even when AIDA is sure. You can enable it on a single property, with the Request manual validation option in Additional behaviours, or on the whole document type, with the Require manual approval option.
Frequently asked questions
I confirmed the value, why does AIDA keep flagging it?
It depends on the reason:
- if it is an OCR reading anomaly, the cause is the image quality, which will recur with other similar documents (see above);
- if it is a history anomaly on a very variable property, every new document may again be far from the previous ones. In this case set Anomaly handling to Don't block validation on that property.
The document was not validated automatically, but I see no anomalies
Anomalies are not the only reason a document is left for validation. Check that:
- all required properties were found;
- an automation exists for that document and is not paused. You can check this in the document's History tab.
Should I set Don't block validation or Off on all properties?
No. AIDA would then automatically validate badly read values too, and you would only notice them in your management software. Use them only on properties that change a lot from one document to the next.
Should I lower the detection level or change the anomaly handling of a property?
The detection level applies to all properties of all document types, while Anomaly handling only affects the property you choose. If the problem concerns a few properties, set Don't block validation on those and leave the level on Default.