unigrams | n=1 | < 1.0 hr. (from ./bin/02.NGramGenAll/logData/${YEAR}/N-gram/*.log) |
| - param: 10,1, (150000000)
- 40 min.
from log.heap.1.50:
- Documents: 31,850,051
- Sentences: 209,685,517
- Tokens: 4,365,354,060
- split: 1, no split
- 1-grams (not unique, from log.heap.1.50): 36,150,616
(it is unique beacuse no split, use wc -l )
- Files:
- nGram.out.1.heap.50.s01.0001-1062 (590 MB, use
ls -alh )
- param:
- 2 min.
- Group Alphabetically
- 1-gram (unique): 36,150,616
- Files:
- ${NGram}.g01.NO-NO (590 MB|36MB, from 1, from ./logData/${YEAR}/11-1.log1-1.log)
| - param: 12, 1, 30
- 1 min
- 1-gram (WC >= 30): 1,185,627
- File:
- 1-gram.${YEAR}.30 (20 MB)
- param: 13, 1, 30
- 1 min.
- 1-gram (sorted): 1,185,627
- File:
- 1-gram.${YEAR}.30.dwt (20 MB)
bigrams | n=2 | 3.2 hr. |
| - param: 10,2, (150000000)
- 2.3 hr.
- split: 3
- 2-gram (not unique from log.heap.2.50): 412,168,042
- Files:
- s01.0001-0583 (3.1 GB)
- s02.0584-0890 (3.0 GB)
- s03.0891-1062 (2.3 GB)
| - param: see file names below
- 21 min.
- Group Alphabetically
- 2-gram (unique, use wc -l): 325,314,728
- Files:
- ${NGram}.g01.NO-c (3.1GB|158MB)
- ${NGram}.g02.c-NO (3.6GB|167MB)
| - param: 12, 2, 30
- 22 min.
- 2-gram (WC >= 30): 7,093,113
- File:
- 2-gram.${YEAR}.30 (152 MB)
- param: 13, 2, 30
- 8 min.
- 2-gram (sorted): 7,093,113
- File:
- 2-gram.${YEAR}.30.dwt (152 MB)
trigrams | n=3 | 16 hr. |
| - param: 10,3, (150000000)
- 5.8 hr.
- split: 11
- 3-gram (not unique - from log.heap.3.50): 1,643,134,783
- Files:
- s01.0001-0209 (3.8 GB)
- s02.0210-0324 (3.7 GB)
- s03.0325-0422 (3.8 GB)
- s04.0423-0559 (3.8 GB)
- s05.0560-0646 (3.7 GB)
- s06.0647-0728 (3.8 GB)
- s07.0729-0800 (3.7 GB)
- s08.0801-0870 (3.8 GB)
- s09.0871-0938 (3.8 GB)
- s10.0939-1005 (3.8 GB)
- s11.1006-1062 (3.4 GB)
| - param: see file names below
- 10.0 hr.
- Group Alphabetically
- 3-gram (unique): 1,117,801,654
- Files:
- g01.NO-M (4.7GB|197MB)
- g02.M-b (4.7GB|187MB)
- g03.b-fn (4.6GB|173MB)
- g04.fn-n (4.6GB|177MB)
- g05.n-sh (5.0GB|190MB)
- g06.sh-NO (4.9B|191MB)
| - param: 12, 3, 30
- 20 min.
- 3-gram (WC >= 30): 10,327,977
- File:
- 3-gram.${YEAR}.30 (265 MB)
- param: 13, 3, 30
- 2 min.
- 3-gram (sorted): 10,327,977
- File:
- 3-gram.${YEAR}.30.dwt (265 MB)
| fourgrams | n=4 | 25 hr. |
| - param: 10,4, (130000000)
- 5.4 hr.
- split: 21
- 4-gram (not unique - from log.heap.4.50): 2,752,252,637
- Files:
- s01.0001-0077 (4.0 GB)
- s02.0078-0204 (3.9 GB)
- s03.0205-0272 (3.9 GB)
- s04.0273-0319 (3.9 GB)
- s05.0320-0372 (4.0 GB)
- s06.0373-0419 (4.0 GB)
- s07.0420-0517 (4.0 GB)
- s08.0518-0559 (3.9 GB)
- s09.0560-0605 (3.9 GB)
- s10.0606-0649 (4.0 GB)
- s11.0650-0696 (4.0 GB)
- s12.0697-0735 (4.0 GB)
- s13.0736-0773 (4.0 GB)
- s14.0774-0811 (4.0 GB)
- s15.0812-0848 (4.0 GB)
- s16.0849-0885 (4.0 GB)
- s17.0886-0921 (4.0 GB)
- s18.0922-0959 (4.1 GB)
- s19.0960-0995 (4.1 GB)
- s20.0996-1030 (4.1 GB)
- s21.1031-1062 (3.7 GB)
| - param: see file names below
- 18.7 hr.
- Group Alphabetically
- 4-gram (unique): 2,093,669,210
- Files:
- g01.NO-C (4.9GB|171MB)
- g02.C-S (5.1GB|165MB)
- g03.S-and (4.9GB|160MB)
- g04.and-bcm (3.9GB|128MB)
- g05.bcm-cs (5.2GB|164MB)
- g06.cs-f (4.6GB|140MB)
- g07.f-i (4.3GB|138MB)
- g08.i-j (3.6GB|115MB)
- g09.j-o (4.9GB|154MB)
- g10.o-pe (4.6GB|157MB)
- g11.pe-ri (5.0GB|153MB)
- g12.ri-th (4.5GB|141MB)
- g13.th-tr (4.5GB|153MB)
- g14.tr-NO (4.5GB|148MB)
| - param: 12, 4, 30
- 52 min.
- 4-gram (WC >= 30): 6,620,884
- File:
- 4-gram.${YEAR}.30 (201 MB)
| - param: 13, 4, 30
- 2 min.
- 4-gram (sorted): 6,620,884
- File:
- 4-gram.${YEAR}.30.dwt (201 MB)
|
---|
fivegrams | n=5 | 36.7 hr. |
| - param: 10,5, (120000000)
- 6.0 hr.
- split: 25
- 5-gram (not unique): 3,158,816,663
Files:
- s01.0001-0064 (4.3 GB)
- s02.0065-0112 (4.3 GB)
- s03.0113-0233 (4.3 GB)
- s04.0234-0279 (4.3 GB)
- s05.0280-0316 (4.4 GB)
- s06.0317-0360 (4.4 GB)
- s07.0361-0398 (4.4 GB)
- s08.0399-0482 (4.4 GB)
- s09.0483-0524 (4.4 GB)
- s10.0525-0558 (4.4 GB)
- s11.0559-0597 (4.4 GB)
- s12.0598-0633 (4.4 GB)
- s13.0634-0671 (4.4 GB)
- s14.0672-0705 (4.4 GB)
- s15.0706-0736 (4.4 GB)
- s16.0737-0767 (4.4 GB)
- s17.0768-0798 (4.4 GB)
- s18.0799-0828 (4.4 GB)
- s19.0829-0858 (4.4 GB)
- s20.0859-0889 (4.5 GB)
- s21.0890-0918 (4.4 GB)
- s22.0919-0950 (4.5 GB)
- s23.0951-0980 (4.5 GB)
- s24.0981-1008 (4.4 GB)
- s25.1009-1036 (4.4 GB)
- s26.1037-1062 (4.1 GB)
| - param: see file names below
- 29.4 hr.
- Group Alphabetically
- 5-gram (unique): 2,729,911,636
- Files:
- g01.NO-A (4.9GB|148MB)
- g02.A-I (4.7GB|128MB)
- g03.I-T (4.7GB|128MB)
- g04.T-a (2.6GB|69MB)
- g05.a-am (3.8GB|104)
- g06.am-ao (4.6GB|128MB)
- g07.ao-c (5.1GB|142MB)
- g08.c-ct (5.4GB|140)
- g09.ct-em (4.7GB|123MB)
- g10.em-ft (5.2GB|140MB)
- g11.ft-in (4.3GB|112MB)
- g12.in-io (4.4GB|119MB)
- g13.io-mj (5.2GB|143MB)
- g14.mj-of (3.2GB|85MB)
- g15.of-og (4.2GB|124MB)
- g16.og-pm (4.3GB|115MB)
- g17.pm-rem (4.9GB|127MB)
- g18.rem-sp (4.6GB|121MB)
- g19.sp-the (4.4GB|118MB)
- g20.the-thf (4.5GB|128MB)
- g21.thf-v (4.8GB|133MB)
- g22.v-NO (5.1GB|144MB)
| - param: 12, 5, 30
- 1.2 hr.
- 5-gram (WC >= 30): 2,875,651
- File:
- 5-gram.${YEAR}.30 (102 MB)
| - param: 13, 5, 30
- 2 min.
- 5-gram (sorted): 2,875,651
- File:
- 5-gram.${YEAR}.30.dwt (102 MB)
|
---|
|
---|
|
|
---|
|
|
|
---|