unigrams | n=1 | < 1.0 hr. |
| - param: 10,1, (150000000)
- 40 min.
from log.heap.1.50:
- Documents: 30,420,660
- Sentences: 196,566,513
- Tokens: 4,080,670,967
- split: 1, no split
- 1-grams (not unique, from log.heap.1.50): 33,884,283
(it is unique beacuse no split, use wc -l )
- Files:
- nGram.out.1.heap.50.s01.0001-1015 (552 MB, use
ls -alh )
- param:
- 2 min.
- Group Alphabetically
- 1-gram (unique): 33,884,283
- Files:
- ${NGram}.g01.NO-NO (552MB|33MB)
| - param: 12, 1, 30
- 1 min
- 1-gram (WC >= 30): 1,126,766
- File:
- 1-gram.${YEAR}.30 (19 MB)
- param: 13, 1, 30
- 1 min.
- 1-gram (sorted): 1,126,766
- File:
- 1-gram.${YEAR}.30.dwt (19 MB)
bigrams | n=2 | 3.0 hr. |
| - param: 10,2, (150000000)
- 2.1 hr.
- split: 3
- 2-gram (not unique from log.heap.2.50): 386,803,783
- Files:
- s01.0001-0583 (3.1 GB)
- s02.0584-0890 (3.0 GB)
- s03.0891-1015 (1.8 GB)
| - param: see file names below
- 17 min.
- Group Alphabetically
- 2-gram (unique, use wc -l): 306,580,281
- Files:
- ${NGram}.g01.NO-c (2.9GB|148MB)
- ${NGram}.g02.c-NO (3.4GB|158MB)
| - param: 12, 2, 30
- 19 min.
- 2-gram (WC >= 30): 6,702,698
- File:
- 2-gram.${YEAR}.30 (143 MB)
- param: 13, 2, 30
- 8 min.
- 2-gram (sorted): 6,702,698
- File:
- 2-gram.${YEAR}.30.dwt (143 MB)
trigrams | n=3 | 11.5 hr. |
| - param: 10,3, (150000000)
- 4.0 hr.
- split: 11
- 3-gram (not unique - from log.heap.3.50): 1,533,796,162
- Files:
- s01.0001-0209 (3.8 GB)
- s02.0210-0324 (3.7 GB)
- s03.0325-0422 (3.8 GB)
- s04.0423-0559 (3.8 GB)
- s05.0560-0647 (3.8 GB)
- s06.0648-0729 (3.8 GB)
- s07.0730-0801 (3.7 GB)
- s08.0802-0871 (3.8 GB)
- s09.0872-0939 (3.8 GB)
- s10.0940-1006 (3.8 GB)
- s11.1007-1015 (0.6 GB)
| - param: see file names below
- 7.0 hr.
- Group Alphabetically
- 3-gram (unique): 1,052,247,625
- Files:
- g01.NO-M (4.3GB|183MB)
- g02.M-b (4.4GB|176MB)
- g03.b-fn (4.4GB|163MB)
- g04.fn-n (4.4GB|167MB)
- g05.n-sh (4.7GB|179MB)
- g06.sh-NO (4.6B|181MB)
| - param: 12, 3, 30
- 20 min.
- 3-gram (WC >= 30): 9,677,700
- File:
- 3-gram.${YEAR}.30 (248 MB)
- param: 13, 3, 30
- 2 min.
- 3-gram (sorted): 9,677,700
- File:
- 3-gram.${YEAR}.30.dwt (248 MB)
| fourgrams | n=4 | 21 hr. |
| - param: 10,4, (130000000)
- 5.2 hr.
- split: 20
- 4-gram (not unique - from log.heap.4.50): 2,572,515,490
- Files:
- s01.0001-0077 (4.0 GB)
- s02.0078-0204 (3.9 GB)
- s03.0205-0272 (3.9 GB)
- s04.0273-0319 (3.9 GB)
- s05.0320-0372 (4.0 GB)
- s06.0373-0419 (4.0 GB)
- s07.0420-0517 (4.0 GB)
- s08.0518-0559 (3.9 GB)
- s09.0560-0605 (3.9 GB)
- s10.0606-0649 (4.0 GB)
- s11.0650-0696 (4.0 GB)
- s12.0697-0735 (4.0 GB)
- s13.0736-0773 (4.0 GB)
- s14.0774-0811 (4.0 GB)
- s15.0812-0848 (4.0 GB)
- s16.0849-0885 (4.0 GB)
- s17.0886-0921 (4.0 GB)
- s18.0922-0959 (4.1 GB)
- s19.0960-0995 (4.1 GB)
- s20.0996-1015 (2.3 GB)
| - param: see file names below
- 15.0 hr.
- Group Alphabetically
- 4-gram (unique): 1,966,182,302
- Files:
- g01.NO-C (4.5GB|160MB)
- g02.C-S (4.7GB|153MB)
- g03.S-and (4.6GB|150MB)
- g04.and-bcm (3.6GB|120MB)
- g05.bcm-cs (4.9GB|155MB)
- g06.cs-f (4.3GB|132MB)
- g07.f-i (4.0GB|130MB)
- g08.i-j (3.4GB|108MB)
- g09.j-o (4.6GB|145MB)
- g10.o-pe (4.4GB|148MB)
- g11.pe-ri (4.7GB|144MB)
- g12.ri-th (4.2GB|132MB)
- g13.th-tr (4.3GB|144MB)
- g14.tr-NO (4.3GB|139MB)
| - param: 12, 4, 30
- 42 min.
- 4-gram (WC >= 30): 6,154,320
- File:
- 4-gram.${YEAR}.30 (187 MB)
| - param: 13, 4, 30
- 2 min.
- 4-gram (sorted): 6,154,320
- File:
- 4-gram.${YEAR}.30.dwt (187 MB)
|
---|
fivegrams | n=5 | 31.0 hr. |
| - param: 10,5, (120000000)
- 5.7 hr.
- split: 25
- 5-gram (not unique): 2,952,618,452
Files:
- s01.0001-0064 (4.3 GB)
- s02.0065-0112 (4.3 GB)
- s03.0113-0233 (4.3 GB)
- s04.0234-0279 (4.3 GB)
- s05.0280-0316 (4.4 GB)
- s06.0317-0360 (4.4 GB)
- s07.0361-0398 (4.4 GB)
- s08.0399-0482 (4.4 GB)
- s09.0483-0524 (4.4 GB)
- s10.0525-0558 (4.4 GB)
- s11.0559-0597 (4.4 GB)
- s12.0598-0633 (4.4 GB)
- s13.0634-0671 (4.4 GB)
- s14.0672-0705 (4.4 GB)
- s15.0706-0736 (4.4 GB)
- s16.0737-0767 (4.4 GB)
- s17.0768-0798 (4.4 GB)
- s18.0799-0828 (4.4 GB)
- s19.0829-0858 (4.4 GB)
- s20.0859-0889 (4.5 GB)
- s21.0890-0918 (4.4 GB)
- s22.0919-0950 (4.5 GB)
- s23.0951-0980 (4.5 GB)
- s24.0981-1008 (4.4 GB)
- s25.1009-1015 (1.0 GB)
| - param: see file names below
- 23.5 hr.
- Group Alphabetically
- 5-gram (unique): 2,558,650,539
- Files:
- g01.NO-A (4.6GB|138MB)
- g02.A-I (4.4GB|118MB)
- g03.I-T (4.4GB|118MB)
- g04.T-a (2.4GB|65MB)
- g05.a-am (3.6GB|98MB)
- g06.am-ao (4.3GB|120MB)
- g07.ao-c (4.8GB|133MB)
- g08.c-d (5.4GB|139MB)
- g09.d-ep (4.7GB|122MB)
- g10.ep-g (4.6GB|124MB)
- g11.g-in (3.7GB|99MB)
- g12.in-io (4.1GB|112MB)
- g13.io-mk (4.9GB|134MB)
- g14.mk-of (3.0GB|80MB)
- g15.of-og (4.0GB|116MB)
- g16.og-pm (4.0GB|108MB)
- g17.pm-rem (4.6GB|119MB)
- g18.rem-sp (4.3GB|114MB)
- g19.sp-the (4.2GB|111MB)
- g20.the-thf (4.3GB|121MB)
- g21.thf-v (4.5GB|125MB)
- g22.v-NO (4.8GB|135MB)
| - param: 12, 5, 30
- 1 hr.
- 5-gram (WC >= 30): 2,649,324
- File:
- 5-gram.${YEAR}.30 (94 MB)
| - param: 13, 5, 30
- 30 min.
- 5-gram (sorted): 2,649,324
- File:
- 5-gram.${YEAR}.30.dwt (94 MB)
|
---|
|
---|
|
|
---|
|
|
|
---|