Computing with Evidence
(Note to Dmitry -- I'm doing everything manually at this point. You
can add automated learning to the next draft!)
While the latest thread on this topic was titled *Computing with
Confidence*, I realize that what I'm really doing is computing with
*evidence*. The NLP++ confidence operator (%%) is perhaps more
properly thought of as an evidence operator.
So "my" theory is that one can ac***ulate an infinite amount of
evidence for and against something, but the total amount of evidence
of either kind never exceeds 100%. Given 80% evidence for something
and adding a new datum at 70% *evidence level* yields something like
80 %% 70 = 86%
So my total ac***ulated evidence is now at the 86% level.
On the web, I see various discussions of theories of evidence, e.g.,
Dempster-Shafer theory
http://en.wikipedia.org/wiki/Dempster-Shafer_theory
where they also warn against confusing evidence with confidence and
probability.
As for the prototyping, that has been going very well, though the
theoretical foundation for evidence/confidence may not be entirely on
firm ground. The *computing with evidence* methodology enables me to
produce self-scored output, with a "confidence" value for each slot of
the output. For most purposes, the self-scored output is close enough
to reality that I don't bother manually correcting it at this point,
and it will vastly help produce a proper answer key. Self-score on
extraction from OCRed do***ents I'm processing is currently about 85%,
let's say that at worst the real score is 75% or greater. (Self-
scoring downgrades itself sometimes by producing 0/1 when it can't
find something that isn't there, while being optimistic at 1/1 when it
produces an incorrect or spurious output, or when it erroneously
decides that 0/0 is right -- that is, the item is missing or garbled.)
I'm using a modified scoring system a la MUC (Message Understanding
Conference). How many of you were at MUCK-I, by cracky? ;-)
http://www.aclweb.org/anthology-new/M/M91/M91-1002.pdf
Since I deal with OCRed text, I count agreement on missing/garbled, or
their NONcommital item (0/0), in the correct answer total and possible
answer total.
As for the NLP++ confidence/evidence operator, I wrote up the math for
that in the earlier thread but it was truncated, so I'm adding it here
again for anyone interested:
I needed a function with the following end points (E = evidence, C =
%evidence or "confidence").
For 0 <= C <= 100
if C = 0, E = 0
if C = 100, E = infinity
So a simple function that fits this is
C * constant
E = ---------------
(100 - C)
As C approaches 100 asymptotically, E approaches infinity. So when I
add a new piece of evidence to the evidence so far
C = C1 %% C2 ;
I'm computing the E1 for C1 and the E2 for C2 (their "evidence
equivalents"), then adding (E1 + E2) and plugging in to get the
composed confidence back. Much like adding logarithms to do
multiplication, then going to the log table to get the answer after
you've done the addition.
That's all there is to the NLP++ "confidence operator". Just a
function that lets me add evidence by combining "confidence" values.
Amnon
[ comp.ai is moderated ... your article may take a while to appear. ]
|
