0. Retrieval token metric in AutoRAG

Currently, in AutoRAG, the Retrieval token metric is only used by the Passage Compressor Node. It measures performance by comparing the compressed passage to Answer_gt.

When comparing Passage and Answer gt, the comparison is made on a per token basis, which you can see by looking at the example

✅ Basic Example

answer gt = ['Do you want to buy some?']

result = ['Do you want to buy some?', 'I want to buy some', 'I want to buy some water']

First, let's break up gt and result into tokens

• GT is a total of 6 tokens ['do', 'you', 'want', 'to', 'buy', 'some']
• The number of tokens in the result is 6, 5, and 6, respectively ['do', 'you', 'want', 'to', 'buy', 'some'], ['I', 'want', 'to', 'buy', 'some'], ['I', 'want', 'to', 'buy', 'some', 'water']

Next, let's look at the number of overlapping tokens in gt and result

• The first is that all 6 tokens overlap with GT, so the number of overlapping tokens is 6.
• The second has 4 tokens overlapping except for the 'I'.
• The third has 4 tokens overlapping except for 'I' and 'water'.

1. Token Precision

📌 Definition

Number of overlapping tokens / token length in result

✅ Apply Basic Example