CSpell

Performance Tests on Edit Distance Similarity Score

I. Test Setup

Data: Training Set
Gold Standard: non-word only
Dictionary: CSpell (Lexicon-based)
Corpus: none
Ranking: Edit Distance and Orthographic ranking

II. Test Results

Tests on various weighting factors (WF) on costs of the edit distance (delete, insert, substitute, and transpose). The WF for orthographic is 1.0, 0.0, 0.0 (only edit distance is used to represent orthographic score).

ID	Delete	Insert	Substitute	Transpose	Precision	Recall	F1	Notes
1	1.00	1.00	1.00	1.00	0.7465	0.7494	0.7479	Same ratio of WF
2	0.95	0.95	0.95	0.95	0.7465	0.7494	0.7479	Same ratio of WF
3	1.00	0.95	0.95	0.95	0.7439	0.7468	0.7453	Increased 1 WF
4	0.95	1.00	0.95	0.95	0.7014	0.7041	0.7028
5	0.95	0.95	1.00	0.95	0.7490	0.7519	0.7505
6	0.95	0.95	0.95	1.00	0.7477	0.7506	0.7492
7	0.90	0.95	0.95	0.95	0.7130	0.7158	0.7144	Decreased 1 WF
8	0.95	0.90	0.95	0.95	0.7555	0.7584	0.7569
9	0.95	0.95	0.90	0.95	0.7040	0.7067	0.7054
10	0.95	0.95	0.95	0.90	0.7400	0.7429	0.7415
11	0.95	0.90	1.00	0.95	0.7593	0.7623	0.7608	Try and error to find the WF for best F1
12	0.93	0.90	1.00	0.97	0.7580	0.7610	0.7595
13	0.97	0.90	1.00	0.93	0.7606	0.7636	0.7621
99-1	0.90	0.90	1.00	0.95	0.7529	0.7558	0.7544
99-2	0.90	0.90	0.95	1.00	0.7490	0.7519	0.7505
99-3	0.95	0.95	1.00	0.90	0.7452	0.7481	0.7466

Tests on various weighting factors (WF) on costs of the edit distance (delete, insert, substitute, and transpose). The WF for orthographic is 1.0, 1.0, 1.0.

ID Delete Insert Substitute Transpose Precision Recall F1
99-4 0.97 0.90 1.00 0.93 0.7593 0.7623 0.7608
99-5 0.95 0.95 1.00 0.90 0.7606 0.7636 0.7621

ID	Delete	Insert	Substitute	Transpose	Precision	Recall	F1
99-4	0.97	0.90	1.00	0.93	0.7593	0.7623	0.7608
99-5	0.95	0.95	1.00	0.90	0.7606	0.7636	0.7621

III. Discussion

The result of tests 1-2 are the same. That is the same ratio of weighting factors leads to same results
From the results of test 3-6, we observed the higher the weighting factor of substitute cost, the better the F1 score.
From the results of test 7-10, we observed the lower the weighting factor of insert cost, the better the F1 score.
Find the best F1 by try and error from tests 11-99-3.
Use test 13 for the weighting factors for costs of delete, insert, substitute and transpose.