SPECIALIST Lexicon

Distilled Process Log - Walk Through, 2019

This page describes the filter processes from the MEDLINE n-gram set to generazte the distilled MEDLINE n-gram and candidate multiword list:

ID/Program	In No.	Filtered No. (%)	Out No.	Pass Rate	Acc. Pass Rate	Filter example and notes
Generate the MEDLINE n-gram set
Generate MEDLINE n-gram set	5,567,578,926 N=1: 32,469,239 N=2: 290,801,313 N=3: 993,227,476 N=4: 1,849,662,679 N=5: 2,401,418,219	5,542,912,110 (99.56%) N=1: 31,394,012 (96.69%) N=2: 284,464,615 (97.82%) N=3: 984,148,940 (99.09%) N=4: 1,843,933,089 (99.69%) N=5: 2,398,971,454 (99.90%)	24,666,816 N=1: 1,075,227 N=2: 6,336,698 N=3: 9,078,536 N=4: 5,729,590 N=5: 2,446,765	0.4430%	N/A	From MEDLINE TI & AB to the MDELINE n-gram set filter out n-grams with length > 50 filter out n-grams with word count < 30 Calculated by Excel (manualy input In and Out No.) Used data from n-gram set log file (not distilled)
Basic operation: Sort nGrams by DC\|WC\|Terms
ID-01 NGramFilter: SortNGramByDcWcTerm Param: 1, 01 Run Time: 1 Min.	24,666,816	0	24,666,816	100.0000%	100.0000%	Create link: ./05.ApplyFilters/nGram.${YEAR}
Apply General Exclusive Filters
ID-10 Filter: Pipe (\|) InTerm: nGram Param: 1, 10 Run time: 1 Min	24,666,816	13	24,666,803	99.9999%	99.9999%	\| (\|r\| \|\| Ag\|AgCl \|D\| \|E\| lambda(\|\|)
ID-11 Filter: Punctuation and space InTerm: core-term.lc Param: 2, 11 Run time: 1 Min.	24,666,803	597	24,666,206	99.9976%	99.9975%	= < +/- > - --> (+) (%) "+" ((-/-)) ==> [...] } //
ID-12 Filter: Digit InTerm: core-term.lc Param: 2, 12 Run time: 2 Min (norm - strip punc and space)	24,666,206	150,806	24,515,400	99.3886%	99.3872%	2 1 3 10 4 95% 2, 2000 3-5 +/-0.5 (+/-0.05) $1,500 "3 + 1" 55834 192.168.1.1 [192, 168] (+15%),
ID-13 Filter: Number InTerm: core-term.lc Create link: ./inData/NRVAR Param: 2, 13 Run time: 2 Min	24,515,400	4,907	24,510,493	99.9800%	99.3663%	and two one first three first and second one third twenty-eight NINE zeroth and Four hundred and forty-seven zero-one 'half' One"
ID-14 Filter: Digit and Stopword InTerm: core-term.lc Create link: ./inData/stopWords.data Param: 2, 14 Run time: 2 Min	24,510,493	185,640	24,324,853	99.2426%	98.6137%	of the in the to the and the on the In the and/or 50% of 1, 2, and 2003 to 2003 to 2007 for >=50% the 8:2 -196 to -174 OR-462 AND-34 IN-1130 And-1
Apply Exclusive Filters - pattern
ID-20 Filter: Parenthetic Acronym (ACR) InTerm: nGram Param: 2, 20 Run time: 1 Min	24,324,853	281,106	24,043,747	98.8444%	97.4741%	tomography (CT) imaging (MRI) resonance imaging (MRI) oxide (NO) reaction (PCR) chain reaction (PCR) polymerase chain reaction (PCR) magnetic resonance imaging (MRI) computed tomography (CT) enzyme-linked immunosorbent assay (ELISA) single nucleotide polymorphisms (SNPs) magnetic resonance (MR) imaging "Standards, Options and Recommendations" (SOR) (CREB)-binding protein (CBP) kinase (ASK) proline-rich polypeptide (PRP) semi-permeable membrane devices (SPMDs)
ID-21 Filter: Pattern of indefnite article InTerm: core-term Param: 2, 21 Run time: 3 Min	24,043,747	437,164	23,606,583	98.1818%	95.7018%	a significant a single a high a novel a case a very a group a dose-dependent A series A and B a meta-analysis a SIF A alpha C A nonseminomatous a delivery rate per A beta 2m a beta ab
ID-22 Filter: Pattern of Colon - UPPERCASE: InTerm: nGram Param: 2, 22 Run time: 1 Min	23,606,583	156,895	23,449,688	99.3354%	95.0657%	RESULTS: METHODS: CONCLUSIONS: CONCLUSION: BACKGROUND: OBJECTIVE: OBJECTIVE: To MATERIALS AND METHODS: SETTING: PURPOSE: To INTRODUCTION: AIM: The L: -DOPA 95% PI: PHPT: months [95% CI: vs N: mode MIC: [95 % CI:
ID-23 Filter: Pattern of Disallowed punctuation InTerm: nGram Param: 2, 23 Run time: 1 Min	23,449,688	170,551	23,279,137	99.2727%	94.3743%	(n = (P < (P = (p < P < (P < 0.05) 95% CI = P<0.001), CI}, US$ VSL#3 N^N group (n=6) received CYP3A7*1C studies; average n.; Trichoteleia sp. n.; Trichoteleia
ID-24 Filter: Pattern of Measurement InTerm: core-term Create link: ./inData/unit.data Param: 2, 24 Run time: 10 Min	23,279,137	389,964	22,889,173	98.3248%	92.7934%	two groups 6 months 24 h (ABSTRACT TRUNCATED AT 250 WORDS) the two groups 5 years at 37 degrees 3 times 100 mg January 1, 10 mg/kg 12-year-old at -20 degrees C September 2006 65 years or older with 20 cigarettes per day 3 - 6 months 6 hours plus minutes) per day, 5 days MMR + V 3 mg/EE 317615 x
ID-25 Filter: Pattern of Incomplete InTerm: nGram Param: 2, 25 Run time: 1 Min	22,889,173	209,979	22,679,194	99.0826%	91.9421%	group (P significant (P years) with significantly (P years) and interval [95% see text] The lt; 0.05) lower CENTRAL) (The nM (SD pOGH (ANG cB72.3(gamma new species (type
Apply Exclusive Filters - Lead-End-Terms
ID-30 Filter: Absolute Invalid Lead-Term InTerm: core-term Create: ./03.LeadEndTerm/invalidLeadTerms.data.abs Param: 2, 30 Run time: 45 Min (normalized)	22,679,194	5,944,446	16,734,748	73.7890%	67.8432%	of a that the from the is a of this The results was observed this study was about 50% - but not "what is AND COURSE iT reg of FoxM1b or spinal or conduction or spinal or conduction block,
ID-31 Filter: Absolute Invalid End-Term InTerm: core-term Create: ./03.LeadEndTerm/invalidEndTerms.data.abs Param: 2, 31 Run time: 10 Min (normalized)	16,734,748	3,447,546	13,287,202	79.3989%	53.8667%	patients with associated with at the suggest that between the in patients with results suggest that MATERIALS AND cross-reacted with (ST 36) and Zusanli (ST 36) and determine whether this could primarily composed of the tilt-in-space and systems, assays and ppm Cu as epidural or spinal or
ID-32 Filter: Lead-End-Term Pattern InTerm: core-term Create: ./03.LeadEndTerm/invalidLeadEndTermCandidates.data Param: 2, 32 Run time: 29 Min	13,287,202	3,233	13,283,969	99.9757%	53.8536%	in a to be with a as a may be In a in A. For one on NO anti-NOR plus AT I/a AS-ON anti-OF
ID-33 Filter: Lead-Term Pattern - no SpVar InTerm: core-term Create: ./03.LeadEndTerm/validLeadTerms.data.pat Param: 2, 33 Run time: 3 Min	13,283,969	1,840,951	11,443,018	86.1416%	46.3903%	to determine In addition, to evaluate to assess to investigate in the presence AT 250 As a result, ON THE TREATMENT as a possible treatment for in details, - for example, within working memory for various chronic in 0.1% trifluoroacetic in threatened preterm labor with the MIC90S On PTD plus LHRH-A with the MIC90S of
ID-34 Filter: End-Term Pattern - no SpVar InTerm: core-term Run time: 1 Min	11,443,018	1,847,412	9,595,606	83.8556%	38.9009%	effects of number of use of presence of used to Comparison of low cost of HPV) in NUMBER OF zymography was used to loss of two or more 1 goes to active with the MIC90s of syn. nov. of microg/mmol of
The final results of above is used as the distilled MEDLINE n-gram set
Apply Exclusive Filters - Project domain
ID-40 Filter: Lexicon InTerm: nGram Run time: 1 Min	9,595,606	851,813	8,743,793	91.1229%	35.4476%	of the in to a The We "The linear, "normal" {systematic name: systematic name anterior intermeniscal ligament regional low-flow perfusion Neo. Cannon & Polycentropus Penneys & % (month

The SPECIALIST Lexicon