Performance Tests on Phonetic Similarity Score
I. Test Setup
- Data: Training Set
- Gold Standard: non-word only
- Dictionary: CSpell (Lexicon-based)
- Corpus: none
- Ranking: Orthographic ranking
II. Test Results
- Tests on various phonetic coding system within orthographic similarity score ranking.
ID | Phonetic | Precision | Recall | F1
|
---|
11 | Double Metaphone | 0.7490 | 0.7519 | 0.7505
|
12 | Refined Soundex | 0.7332 | 0.7370 | 0.7351
|
13 | Caverphone-2 | 0.7172 | 0.7209 | 0.7191
|
14 | Metaphone | 0.7487 | 0.7506 | 0.7497
|
15 | Metaphone-3 | 0.7452 | 0.7481 | 0.7466
|
- Tests on various weighting factors (WF) on costs of the edit distance (delete, insert, substitute, and transpose) with Metaphone 2 in the orthographic similarity score.
ID | Delete | Insert | Substitute | Transpose | Precision | Recall | F1 | Notes
|
---|
1 | 0.95 | 0.95 | 0.95 | 0.95 | 0.7490 | 0.7519 | 0.7505 | Same ratio of WF
|
2 | 1.00 | 0.95 | 0.95 | 0.95 | 0.7349 | 0.7377 | 0.7363 | Increasing 1 WF
|
3 | 0.95 | 1.00 | 0.95 | 0.95 | 0.7275 | 0.7313 | 0.7294
|
4 | 0.95 | 0.95 | 1.00 | 0.95 | 0.7413 | 0.7442 | 0.7427
|
5 | 0.95 | 0.95 | 0.95 | 1.00 | 0.7490 | 0.7519 | 0.7505
|
6 | 0.90 | 0.95 | 0.95 | 0.95 | 0.7275 | 0.7313 | 0.7294 | Decreasing 1 WF
|
7 | 0.95 | 0.90 | 0.95 | 0.95 | 0.7439 | 0.7468 | 0.7453
|
8 | 0.95 | 0.95 | 0.90 | 0.95 | 0.7172 | 0.7209 | 0.7191
|
9 | 0.95 | 0.95 | 0.95 | 0.90 | 0.7439 | 0.7468 | 0.7453
|
10 | 0.95 | 0.90 | 0.95 | 1.00 | 0.7439 | 0.7468 | 0.7453 | Try and error to find the WF of cost and phonetic
|
99-1 | 0.95 | 0.95 | 0.95 | 0.90 | 0.7375 | 0.7403 | 0.7389
|
- Tests on various weighting factors (WF) on costs of the edit distance (delete, insert, substitute, and transpose). The WF for orthographic is 1.0, 1.0, 1.0.
III. Discussion
- From the results of test 11-15, we chose Double Metaphone as the phonetic system in the orthographic similarity score.
- From the results of test 2-5, we observed the higher the weighting factor of transpose cost, the better the F1 score.
- From the results of test 6-9, we observed the lower the weighting factor of insert cost, the better the F1 score.
- Find the best F1 by try and error from tests 10-99-1, that is lower the cost of insert and raise the cost of transpose.
- Use test 13 for the weighting factors for costs of delete, insert, substitute and transpose.