Queue: PublicMetaMap
Subject: Utterances are returned that are not found in the document text
Status: resolved
Created: Tue Mar 19 15:01:42 2013
Resolved: Tue Jun 11 10:26:38 2013
LastUpdated: Tue Jun 11 10:26:38 2013
Subject: Utterances are returned that are not found in the document text
Status: resolved
Created: Tue Mar 19 15:01:42 2013
Resolved: Tue Jun 11 10:26:38 2013
LastUpdated: Tue Jun 11 10:26:38 2013
Subject:
Creator: 28
Created: 2013-03-19 19:01:42
Transaction: 1872
Parent: 0
MessageId: [email redacted]
Filename:
ContentType: text/plain
ContentEncoding: none
Content:
>Category: mmtxT
>Synopsis: Utterances are returned that are not found in the document text
>Confidential: no
>Severity: serious
>Priority: high
>Class: sw-bug
>Submitter-Id: mmtxWeb
>Release:
>Environment:
>Operating System - MetaMap:
>OS Version - Mac 10.6.8
>Java Version - Java 1.6
>Description:
Utterances returned about processing by MetaMap do not exist in the text. It appears that document text lines are sometimes joined together, giving a single line utterance that is composed of two lines.
This is a problem since the utterance can not be found in the document text and calls to the API to return the utterance will not be able to return the utterance.
A bugfix (hack) could be possible in the MetaMapAnnotator.java file by searching for a findable utterance string in the document text (backtracking by one) but this is a horrible and would be better to be fixed at a lower level.
I did attach the entire document tested, but it is from the SharClef2013 challenge. I also modified a few numbers since distribution of the original document is restricted.
>How-To-Repeat:
Actual Text to Reproduce Problem in Input Document (2 lines)
27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08 00:00:00.0 |||| |||| |||| ||||
Admission Date: [**2015-03-28**] Discharge Date: [**2014-05-08**]
Utterance Returned (one line)
27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08 00:00:00.0 |||| |||| |||| |||| Admission Date: [**2015-03-28**] Discharge Date: [**2014-05-08**]
>Unformatted:
Subject: Creator: 28 Created: 2013-03-20 19:35:44 Transaction: 1878 Parent: 0 MessageId: [email redacted] Filename: ContentType: text/html ContentEncoding: none Content: Each line will need to separated by a blank line if you want the annotator to treat them as separate lines.
the input:
27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08 00:00:00.0 |||| |||| |||| ||||
Admission Date: [**2015-03-28**] Discharge Date: [**2014-05-08**]
will result in:
Utterance:
Utterance text: 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08 00:00:00.0 |||| |||| |||| ||||
Utterance:
Utterance text: Admission Date: [**2015-03-28**] Discharge Date: [**2014-05-08**]
I not sure if this is what you're looking for...
Sincerely,
Willie Rogers
On Tue Mar 19 15:01:42 2013, wjrogers wrote:
>
> >Submitter: [email redacted]
> >Category: mmtxT
> >Synopsis: Utterances are returned that are not found in the document
> text
> >Confidential: no
> >Severity: serious
> >Priority: high
> >Class: sw-bug
> >Submitter-Id: mmtxWeb
> >Originator: John Osborne ([email redacted])
> >Release:
> >Environment:
> >Operating System - MetaMap:
> >OS Version - Mac 10.6.8
> >Java Version - Java 1.6
> >Description:
> Utterances returned about processing by MetaMap do not exist in the
> text. It appears that document text lines are sometimes joined
> together, giving a single line utterance that is composed of two
> lines.
>
> This is a problem since the utterance can not be found in the document
> text and calls to the API to return the utterance will not be able
> to return the utterance.
>
> A bugfix (hack) could be possible in the MetaMapAnnotator.java file by
> searching for a findable utterance string in the document text
> (backtracking by one) but this is a horrible and would be better to
> be fixed at a lower level.
>
> I did attach the entire document tested, but it is from the
> SharClef2013 challenge. I also modified a few numbers since
> distribution of the original document is restricted.
> >How-To-Repeat:
> Actual Text to Reproduce Problem in Input Document (2 lines)
> 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
> 00:00:00.0 |||| |||| |||| ||||
> Admission Date: [**2015-03-28**] Discharge Date:
> [**2014-05-08**]
>
> Utterance Returned (one line)
> 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
> 00:00:00.0 |||| |||| |||| |||| Admission Date: [**2015-03-
> 28**] Discharge Date: [**2014-05-08**]
> >Unformatted:
Subject: [PublicMetamap #40] Utterances are returned that are not found in the document text
Creator: 1
Created: 2013-03-20 19:35:44
Transaction: 1879
Parent: 0
MessageId: [email redacted]
Filename:
ContentType: text/plain
ContentEncoding: none
Content: Each line will need to separated by a blank line if you want the annotator to
treat them as separate lines.
the input:
27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08 00:00:00.0 ||||
|||| |||| ||||
Admission Date: [**2015-03-28**] Discharge Date: [**2014-05-08**]
will result in:
Utterance:
Utterance text: 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
00:00:00.0 |||| |||| |||| ||||
Utterance:
Utterance text: Admission Date: [**2015-03-28**] Discharge Date:
[**2014-05-08**]
I not sure if this is what you're looking for...
Sincerely,
Willie Rogers
On Tue Mar 19 15:01:42 2013, wjrogers wrote:
>
> >Category: mmtxT
> >Synopsis: Utterances are returned that are not found in the document
> text
> >Confidential: no
> >Severity: serious
> >Priority: high
> >Class: sw-bug
> >Submitter-Id: mmtxWeb
> >Release:
> >Environment:
> >Operating System - MetaMap:
> >OS Version - Mac 10.6.8
> >Java Version - Java 1.6
> >Description:
> Utterances returned about processing by MetaMap do not exist in the
> text. It appears that document text lines are sometimes joined
> together, giving a single line utterance that is composed of two
> lines.
>
> This is a problem since the utterance can not be found in the document
> text and calls to the API to return the utterance will not be able
> to return the utterance.
>
> A bugfix (hack) could be possible in the MetaMapAnnotator.java file by
> searching for a findable utterance string in the document text
> (backtracking by one) but this is a horrible and would be better to
> be fixed at a lower level.
>
> I did attach the entire document tested, but it is from the
> SharClef2013 challenge. I also modified a few numbers since
> distribution of the original document is restricted.
> >How-To-Repeat:
> Actual Text to Reproduce Problem in Input Document (2 lines)
> 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
> 00:00:00.0 |||| |||| |||| ||||
> Admission Date: [**2015-03-28**] Discharge Date:
> [**2014-05-08**]
>
> Utterance Returned (one line)
> 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
> 00:00:00.0 |||| |||| |||| |||| Admission Date: [**2015-03-
> 28**] Discharge Date: [**2014-05-08**]
> >Unformatted:
Subject: [PublicMetamap #40] Utterances are returned that are not found in the document text
Creator: 1
Created: 2013-03-20 19:35:44
Transaction: 1880
Parent: 0
MessageId: [email redacted]
Filename:
ContentType: text/plain
ContentEncoding: none
Content: Each line will need to separated by a blank line if you want the annotator to
treat them as separate lines.
the input:
27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08 00:00:00.0 ||||
|||| |||| ||||
Admission Date: [**2015-03-28**] Discharge Date: [**2014-05-08**]
will result in:
Utterance:
Utterance text: 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
00:00:00.0 |||| |||| |||| ||||
Utterance:
Utterance text: Admission Date: [**2015-03-28**] Discharge Date:
[**2014-05-08**]
I not sure if this is what you're looking for...
Sincerely,
Willie Rogers
On Tue Mar 19 15:01:42 2013, wjrogers wrote:
>
> >Category: mmtxT
> >Synopsis: Utterances are returned that are not found in the document
> text
> >Confidential: no
> >Severity: serious
> >Priority: high
> >Class: sw-bug
> >Submitter-Id: mmtxWeb
> >Release:
> >Environment:
> >Operating System - MetaMap:
> >OS Version - Mac 10.6.8
> >Java Version - Java 1.6
> >Description:
> Utterances returned about processing by MetaMap do not exist in the
> text. It appears that document text lines are sometimes joined
> together, giving a single line utterance that is composed of two
> lines.
>
> This is a problem since the utterance can not be found in the document
> text and calls to the API to return the utterance will not be able
> to return the utterance.
>
> A bugfix (hack) could be possible in the MetaMapAnnotator.java file by
> searching for a findable utterance string in the document text
> (backtracking by one) but this is a horrible and would be better to
> be fixed at a lower level.
>
> I did attach the entire document tested, but it is from the
> SharClef2013 challenge. I also modified a few numbers since
> distribution of the original document is restricted.
> >How-To-Repeat:
> Actual Text to Reproduce Problem in Input Document (2 lines)
> 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
> 00:00:00.0 |||| |||| |||| ||||
> Admission Date: [**2015-03-28**] Discharge Date:
> [**2014-05-08**]
>
> Utterance Returned (one line)
> 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
> 00:00:00.0 |||| |||| |||| |||| Admission Date: [**2015-03-
> 28**] Discharge Date: [**2014-05-08**]
> >Unformatted:
Subject: Creator: 28 Created: 2013-03-22 19:58:52 Transaction: 1895 Parent: 0 MessageId: [email redacted] Filename: ContentType: multipart/mixed ContentEncoding: Content:
Subject: Creator: 28 Created: 2013-03-22 19:58:52 Transaction: 1895 Parent: 165 MessageId: Filename: ContentType: text/plain ContentEncoding: Content:
Subject:
Creator: 28
Created: 2013-03-22 19:58:52
Transaction: 1895
Parent: 165
MessageId:
Filename:
ContentType: message/rfc822
ContentEncoding: none
Content: Received: from NIHHT02.nih.gov (156.40.71.21) by CASHTV16.nih.gov
(156.40.71.179) with Microsoft SMTP Server (TLS) id 14.2.328.9; Fri, 22 Mar
2013 11:28:29 -0400
Received: from mailfwd.nih.gov (128.231.90.76) by NIHHT02.nih.gov
(156.40.71.21) with Microsoft SMTP Server id 8.3.279.5; Fri, 22 Mar 2013
11:28:25 -0400
Received: from [130.14.111.27] (130.14.111.27) by mailfwd.nih.gov
(128.231.90.107) with Microsoft SMTP Server id 14.2.318.4; Fri, 22 Mar 2013
11:28:26 -0400
Message-ID: <[email redacted]>
Date: Fri, 22 Mar 2013 11:28:25 -0400
From: "Willie Rogers (NIH/NLM/LHC) [C]" <[email redacted]>
Organization: National Library of Medicine
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:17.0) Gecko/20130307 Thunderbird/17.0.4
To: "John David Osborne (Campus)" <[email redacted]>
Subject: Re: Ticket: #40: Utterances are returned that are not found in the
document text
References: <[email redacted]>
In-Reply-To: <[email redacted]>
Content-Type: multipart/mixed;
boundary="------------080107010100070101050205"
Return-Path: [email redacted]
X-MS-Exchange-Organization-AuthSource: NIHHT02.nih.gov
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 10
X-MS-Exchange-Organization-AVStamp-Mailbox: MSFTFF;1;0;0 0 0
MIME-Version: 1.0
--------------080107010100070101050205
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit
I've modified the adapter to fixed the issue of the adapter inserting -1
when it couldn't find the utterance or phrase text from
jcas.getDocumentText(). I've attached the source as well as an update
TypeSystem Descriptor (fixes a problem which occurs when using Java 6 or
7).
The updated distribution for UIMA is available at:
http://metamap.nlm.nih.gov/download/public_mm_uima_2012.tar.bz2
Sorry for the problems, we had only tested the annotator with Journal
Articles not chunked text such as clinical notes.
Sincerely,
Willie Rogers
On 03/21/13 17:29, John David Osborne (Campus) wrote:
> Hi Will,
>
> Try this file, it dies for me using the UIMA annotator on both linux and
> on mac. I'm using the 2012 MetaMap Annotator.
>
> -John
>
>
> On 3/21/13 1:47 PM, "Willie Rogers (NIH/NLM/LHC) [C]"
> <[email redacted]> wrote:
>
>> I wasn't able to reproduce the position values of '-1' using the section
>> of text you sent me. I was wondering which versions of MetaMap API,
>> MetaMap UIMA, and Apache UIMA you were using. Also, which operating
>> system your running the system on.
>>
>> Sincerely,
>> Willie Rogers
>>
>> On 03/21/13 11:50, John David Osborne (Campus) wrote:
>>> I don't think I am describing the problem very well or I am missing
>>> something. In the file MetaMapAnnotator.java
>>> Ina the processUtterance function the line:
>>>
>>> int utteranceStart = docText.indexOf(utterance.getString(),
>>> citationStart
>>> + utterance.getPosition().getX());
>>>
>>> Can yield an utteranceStart of -1 for input data in the generated
>>> org.metamap.uima.ts.Utterance object. This can also be true for
>>> org.metamap.uima.ts.Phrase.
>>>
>>> Would you expect this or would you say this is a bug? I would argue all
>>> NO
>>> utterances should ever have a -1 start.
>>>
>>> Perhaps the input I sent off is not generating the same error, as the
>>> copy/paste text/capture didn't generate the same error for you?
>>>
>>> -John
>>>
>>>
>>> On 3/20/13 2:38 PM, "Willie Rogers (NIH/NLM/LHC) [C]"
>>> <[email redacted]> wrote:
>>>
>>>> Each line will need to separated by a blank line if you want the
>>>> annotator to treat them as separate lines.
>>>>
>>>>
>>>> the input:
>>>>
>>>> 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
>>>> 00:00:00.0 |||| |||| |||| ||||
>>>>
>>>> Admission Date: [**2015-03-28**] Discharge Date:
>>>> [**2014-05-08**]
>>>>
>>>> will result in:
>>>>
>>>> Utterance:
>>>> Utterance text: 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY ||||
>>>> 2015-04-08 00:00:00.0 |||| |||| |||| ||||
>>>> Utterance:
>>>> Utterance text: Admission Date: [**2015-03-28**]
>>>> Discharge Date: [**2014-05-08**]
>>>>
>>>>
>>>> I not sure if this is what you're looking for...
>>>>
>>>> Sincerely,
>>>> Willie Rogers
>>>>
>>>>>> Submitter: [email redacted]
>>>>>> Category: mmtxT
>>>>>> Synopsis: Utterances are returned that are not found in the document
>>>>>> text
>>>>>> Confidential: no
>>>>>> Severity: serious
>>>>>> Priority: high
>>>>>> Class: sw-bug
>>>>>> Submitter-Id: mmtxWeb
>>>>>> Originator: John Osborne ([email redacted])
>>>>>> Release:
>>>>>> Environment:
>>>>>> Operating System - MetaMap:
>>>>>> OS Version - Mac 10.6.8
>>>>>> Java Version - Java 1.6
>>>>>> Description:
>>>>> Utterances returned about processing by MetaMap do not exist in the
>>>>> text. It appears that document text lines are sometimes joined
>>>>> together,
>>>>> giving a single line utterance that is composed of two lines.
>>>>>
>>>>> This is a problem since the utterance can not be found in the document
>>>>> text and calls to the API to return the utterance will not be able to
>>>>> return the utterance.
>>>>>
>>>>> A bugfix (hack) could be possible in the MetaMapAnnotator.java file by
>>>>> searching for a findable utterance string in the document text
>>>>> (backtracking by one) but this is a horrible and would be better to be
>>>>> fixed at a lower level.
>>>>>
>>>>> I did attach the entire document tested, but it is from the
>>>>> SharClef2013 challenge. I also modified a few numbers since
>>>>> distribution
>>>>> of the original document is restricted.
>>>>>> How-To-Repeat:
>>>>> Actual Text to Reproduce Problem in Input Document (2 lines)
>>>>> 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
>>>>> 00:00:00.0 |||| |||| |||| ||||
>>>>> Admission Date: [**2015-03-28**] Discharge Date:
>>>>> [**2014-05-08**]
>>>>>
>>>>> Utterance Returned (one line)
>>>>> 27559 |||| 211 |||| 5653 |||| DISCHARGE_SUMMARY |||| 2015-04-08
>>>>> 00:00:00.0 |||| |||| |||| |||| Admission Date: [**2015-03-28**]
>>>>> Discharge Date: [**2014-05-08**]
>>>>>> Unformatted:
>>>>
>>>>
>>>> --
>>>> Willie Rogers,
>>>> Contractor, Lister Hill Center, National Library of Medicine, NIH
>>>> email: [email redacted]
>>>
>>
>>
>> --
>> Willie Rogers,
>> Contractor, Lister Hill Center, National Library of Medicine, NIH
>> email: [email redacted]
>
--
Willie Rogers,
Contractor, Lister Hill Center, National Library of Medicine, NIH
email: [email redacted]
--------------080107010100070101050205
Content-Type: text/plain; charset="us-ascii"; name="MetaMapAnnotator.java"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="MetaMapAnnotator.java"
package gov.nih.nlm.nls.metamap.uima;
import java.util.List;
import java.util.ArrayList;
import java.io.*;
import gov.nih.nlm.nls.metamap.*;
import gov.nih.nlm.nls.metamap.uima.ts.AcronymAbbrev;
import gov.nih.nlm.nls.metamap.uima.ts.Document;
import org.apache.uima.util.Level;
import org.apache.uima.UimaContext;
import org.apache.uima.jcas.JCas;
import org.apache.uima.jcas.JCasRegistry;
import org.apache.uima.jcas.cas.FSArray;
import org.apache.uima.jcas.cas.StringArray;
import org.apache.uima.jcas.cas.IntegerArray;
import org.apache.uima.analysis_component.AnalysisComponent;
import org.apache.uima.analysis_component.JCasAnnotator_ImplBase;
import org.apache.uima.resource.ResourceInitializationException;
import org.metamap.uima.ts.CuiConcept;
import org.metamap.uima.ts.Span;
/**
* Describe class MetaMapAnnotator here.
*
*
* Created: Fri Dec 4 11:32:31 2009
*
* @author Willie Rogers
* @version 1.0
*/
public class MetaMapAnnotator extends JCasAnnotator_ImplBase implements
AnalysisComponent {
private static final String MESSAGE_DIGEST = "AnnotatorMessages";
public static final String MMSERVER_PATH_PARAMETER = "metamap_server_path";
public static final String TEMPDIR_PATH_PARAMETER = "tempdir_path";
public static final String SKRMEDPOSTCTL_PATH_PARAMETER = "skrmedpostctl_path";
public static final String WSDSERVERCTL_PATH_PARAMETER = "wsdserverctl_path";
public static final String METAMAP_OPTIONS_PARAMETER = "metamap_options";
public static final String METAMAP_TIMEOUT_PARAMETER = "0";
private String mmserverPath;
private String skrmedpostctlPath;
private String wsdserverctlPath;
private String metamapOptions;
private String apiTimeoutInterval;
MetaMapApi api;
/**
* @generated
* @ordered
*/
public final static int typeIndexID = JCasRegistry
.register(MetaMapAnnotator.class);
/**
* @generated
* @ordered
*/
public final static int type = typeIndexID;
/** @generated */
public int getTypeIndexID() {
return typeIndexID;
}
/**
* Creates a new MetaMapAnnotator instance.
*
*/
public MetaMapAnnotator() {
}
/**
* Write your own initialization here
*
* @generated modifiable
*/
//private void readObject() {
//}
/** Performs any startup tasks required by this component. */
public void initialize(UimaContext aContext) throws ResourceInitializationException {
super.initialize(aContext);
this.getParameters();
this.api = new MetaMapApiImpl(0);
this.startTaggerServer();
this.startDisambiguationServer();
this.startMetaMapServer();
this.setupMetaMapOptions();
}
/** Completes the processing of a batch of CASes. */
//public void batchProcessComplete() {
//}
/**
* Notifies this AnalysisComponent that processing of an entire collection
* has been completed.
*/
//public void collectionProcessComplete() {
//}
/** Frees all resources held by this AnalysisComponent. */
public void destroy() {
}
/**
* Returns the specific CAS interface that this AnalysisComponent requires
* the framework to pass to its process(AbstractCas) method.
*/
// public Class getRequiredCasInterface() { return null; }
/** Inputs a CAS to the AnalysisComponent. */
public void process(JCas jcas) {
String docText = jcas.getDocumentText();
List resultList = this.api.processCitationsFromString(docText.trim());
int resultStartIndex = 0;
for (Result result: resultList) {
if (result != null) {
try {
int citationStart = docText.indexOf(result.getInputText(), resultStartIndex);
if (citationStart == -1) { citationStart = 0; }
Document item =
new gov.nih.nlm.nls.metamap.uima.ts.Document(jcas,
citationStart,
citationStart + result.getInputText().length());
List acronymsAbbrevs = processAcronymsAbbrevs(jcas, result.getAcronymsAbbrevsList());
FSArray aaArray = new FSArray(jcas, acronymsAbbrevs.size());
aaArray.copyFromArray(acronymsAbbrevs.toArray(new gov.nih.nlm.nls.metamap.uima.ts.AcronymAbbrev[0]), 0, 0, aaArray.size());
item.setAcronymsAbbrevs(aaArray);
aaArray.addToIndexes(jcas);
List negations = processNegations(jcas, result.getNegationList(), citationStart);
FSArray negationArray = new FSArray(jcas, negations.size());
negationArray.copyFromArray(negations.toArray(new org.metamap.uima.ts.Negation[0]), 0, 0, negationArray.size());
item.setNegations(negationArray);
negationArray.addToIndexes(jcas);
List utterances = processUtterances(jcas,
result.getUtteranceList(),
citationStart);
FSArray utteranceArray = new FSArray(jcas, utterances.size());
utteranceArray.copyFromArray(utterances.toArray(new org.metamap.uima.ts.Utterance[0]), 0, 0, utteranceArray.size());
item.setUtterances(utteranceArray);
utteranceArray.addToIndexes(jcas);
item.addToIndexes(jcas);
} catch (Exception e) {
e.printStackTrace();
}
resultStartIndex = resultStartIndex + result.getInputText().length();
}
}
}
List processAcronymsAbbrevs(JCas jcas, List acronymsAbbrevsList)
throws Exception
{
List acronymAbbrevList =
new ArrayList
(acronymsAbbrevsList.size());
for (AcronymsAbbrevs aa: acronymsAbbrevsList) {
AcronymAbbrev currentAA = new AcronymAbbrev(jcas);
currentAA.setAcronym(aa.getAcronym());
currentAA.setExpansion(aa.getExpansion());
int i = 0;
int[] countVector = new int[aa.getCountList().size()];
for (Integer count : aa.getCountList()) {
countVector[i] = count.intValue();
i++;
}
IntegerArray countArray = new IntegerArray(jcas, countVector.length);
countArray.copyFromArray(countVector, 0, 0, countVector.length);
currentAA.setCountList(countArray);
countArray.addToIndexes(jcas);
StringArray cuiArray = new StringArray(jcas, aa.getCUIList().size());
cuiArray.copyFromArray(aa.getCUIList().toArray
(new String[0]), 0, 0, aa.getCUIList().size());
currentAA.setCuiList(cuiArray);
cuiArray.addToIndexes(jcas);
acronymAbbrevList.add(currentAA);
currentAA.addToIndexes(jcas);
}
return acronymAbbrevList;
}
List processNegations(JCas jcas, List aNegationList, int citationStart)
throws Exception
{
/** uima types from metamap uima package */
List negationList =
new ArrayList
(aNegationList.size());
CuiConcept currentCuiConcept = null;
List cuiConcepts = null;
// System.out.println("MetaMapAnnotator: result.getNegationList(): " + result.getNegationList());
for (Negation neg : aNegationList) {
org.metamap.uima.ts.Negation currentNegation =
new org.metamap.uima.ts.Negation(jcas);
cuiConcepts = new ArrayList();
currentNegation.setNegType(neg.getType());
currentNegation.setNegTrigger(neg.getTrigger());
FSArray fsArray = createSpanFSArray(jcas, neg.getTriggerPositionList(), citationStart);
currentNegation.setNtSpans(fsArray);
fsArray.addToIndexes();
for (ConceptPair conceptPair : neg.getConceptPairList()) {
currentCuiConcept = new CuiConcept(jcas);
currentCuiConcept.setNegExConcept(conceptPair.getPreferredName());
currentCuiConcept.setNegExCui(conceptPair.getConceptId());
cuiConcepts.add(currentCuiConcept);
currentCuiConcept.addToIndexes();
}
fsArray = createSpanFSArray(jcas, neg.getConceptPositionList(), citationStart);
currentNegation.setNcSpans(fsArray);
fsArray.addToIndexes();
fsArray = new FSArray(jcas, cuiConcepts.size());
fsArray.copyFromArray(cuiConcepts.toArray(new CuiConcept[0]), 0, 0, fsArray.size());
currentNegation.setCuiConcepts(fsArray);
fsArray.addToIndexes();
currentNegation.addToIndexes();
negationList.add(currentNegation);
}
return negationList;
}
List processUtterances(JCas jcas, List anUtteranceList, int citationStart)
throws Exception
{
String docText = jcas.getDocumentText().replace("\n"," ");
/** uima types from metamap uima package */
List utteranceList =
new ArrayList
(anUtteranceList.size());
// System.out.println("MetaMapAnnotator: result.getUtteranceList(): " + anUtteranceList);
for (Utterance utterance : anUtteranceList) {
int utteranceStart = docText.indexOf(utterance.getString(), citationStart + utterance.getPosition().getX());
/* if indexOf can't find text because of missing spaces or
* newlines replaced by spaces then trust values UIMA is
* providing. */
if (utteranceStart == -1) {
utteranceStart = citationStart + utterance.getPosition().getX();
}
org.metamap.uima.ts.Utterance currentUtterance =
new org.metamap.uima.ts.Utterance (jcas,
utteranceStart,
utteranceStart + utterance.getPosition().getY());
currentUtterance.setPmid(utterance.getId());
processPCMList(jcas, utterance, currentUtterance, citationStart);
currentUtterance.addToIndexes();
utteranceList.add(currentUtterance);
}
return utteranceList;
}
void processPCMList(JCas jcas,
gov.nih.nlm.nls.metamap.Utterance utterance,
org.metamap.uima.ts.Utterance currentUtterance,
int citationStart)
throws Exception
{
String docText = jcas.getDocumentText();
org.metamap.uima.ts.Phrase currentPhrase;
// org.metamap.uima.ts.CuiConcept currentCuiConcept = null;
List phrases = null;
List candidates = null;
List mappings = null;
phrases = new ArrayList();
mappings = new ArrayList();
// System.out.println("MetaMapAnnotator: utterance.getPCMList(): " + utterance.getPCMList());
for (PCM pcm : utterance.getPCMList()) {
// System.out.println("MetaMapAnnotator: pcm.getPhrase().getPhraseText(): " + pcm.getPhrase().getPhraseText());
int phraseStart = docText.indexOf(pcm.getPhrase().getPhraseText(),
citationStart + pcm.getPhrase().getPosition().getX());
/* if indexOf can't find text because of missing spaces or
* newlines replaced by spaces then trust values UIMA is
* providing. */
if (phraseStart == -1) {
phraseStart = citationStart + pcm.getPhrase().getPosition().getX();
}
currentPhrase = new org.metamap.uima.ts.Phrase
(jcas,
phraseStart,
phraseStart + pcm.getPhrase().getPosition().getY() );
phrases.add(currentPhrase);
candidates = processCandidates(jcas, pcm.getCandidateList(), citationStart);
FSArray fsArray = new FSArray(jcas, candidates.size());
fsArray.copyFromArray(candidates.toArray
(new org.metamap.uima.ts.Candidate[0]),
0, 0, fsArray.size());
currentPhrase.setCandidates(fsArray);
fsArray.addToIndexes();
mappings = processMappings(jcas, pcm, citationStart);
fsArray = new FSArray(jcas, mappings.size());
fsArray.copyFromArray(mappings.toArray(new org.metamap.uima.ts.Mapping[0]), 0, 0, fsArray.size());
currentPhrase.setMappings(fsArray);
fsArray.addToIndexes();
currentPhrase.addToIndexes();
}
int beginUtteranceEnd = utterance.getId().indexOf(".");
int endUtteranceEnd = utterance.getId().lastIndexOf(".");
currentUtterance.setLocation(utterance.getId().substring(beginUtteranceEnd,
endUtteranceEnd));
FSArray fsArray = new FSArray(jcas, phrases.size());
fsArray.copyFromArray(phrases.toArray(new org.metamap.uima.ts.Phrase[0]),
0, 0, fsArray.size());
currentUtterance.setPhrases(fsArray);
fsArray.addToIndexes();
}
List processCandidates(JCas jcas, List candidateList, int citationStart)
throws Exception
{
String docText = jcas.getDocumentText();
org.metamap.uima.ts.Candidate currentCandidate;
List candidates = new ArrayList();
for (Ev ev : candidateList) {
// System.out.println("MetaMapAnnotator: ev.getScore(): " + ev.getScore());
// System.out.println("MetaMapAnnotator: ev.getConceptId(): " + ev.getConceptId());
int begin = 9999;
int end = 0;
for (Position pos : ev.getPositionalInfo()) {
begin = Math.min(begin, (int)pos.getX());
end = Math.max(end, pos.getX() + pos.getY());
// System.out.println("span: [" + pos.getX() + "," + pos.getX() + pos.getY() + "]");
}
int start = citationStart + begin;
int finish = citationStart + end;
// System.out.println("maximal span: [" + begin + "," + end + "]");
currentCandidate = new org.metamap.uima.ts.Candidate(jcas, start, finish);
currentCandidate.setScore(ev.getScore());
currentCandidate.setCui(ev.getConceptId());
currentCandidate.setConcept(ev.getConceptName());
currentCandidate.setPreferred(ev.getPreferredName());
currentCandidate.setHead(ev.isHead());
currentCandidate.setOvermatch(ev.isOvermatch());
List matchMaps = new ArrayList();
gov.nih.nlm.nls.metamap.uima.ts.MatchMap currentMatchMap;
for (MatchMap matchMap: ev.getMatchMapList()) {
currentMatchMap = new gov.nih.nlm.nls.metamap.uima.ts.MatchMap(jcas);
currentMatchMap.setPhraseMatchStart(matchMap.getPhraseMatchStart());
currentMatchMap.setPhraseMatchEnd(matchMap.getPhraseMatchEnd());
currentMatchMap.setConceptMatchStart(matchMap.getConceptMatchStart());
currentMatchMap.setConceptMatchEnd(matchMap.getConceptMatchEnd());
currentMatchMap.setLexVariation(matchMap.getLexMatchVariation());
matchMaps.add(currentMatchMap);
}
FSArray matchMapArray = new FSArray(jcas,ev.getMatchMapList().size());
matchMapArray.copyFromArray(matchMaps.toArray(new gov.nih.nlm.nls.metamap.uima.ts.MatchMap[0]),
0, 0, matchMapArray.size());
currentCandidate.setMatchMap(matchMapArray);
matchMapArray.addToIndexes();
StringArray sourceArray = new StringArray(jcas, ev.getSources().size());
// ev.getSources().toArray()
sourceArray.copyFromArray(ev.getSources().toArray
(new String[0]), 0, 0, ev.getSources().size());
currentCandidate.setSources(sourceArray);
sourceArray.addToIndexes();
StringArray semTypeArray = new StringArray
(jcas, ev.getSemanticTypes().size());
// ev.getSemanticTypes().toArray()
semTypeArray.copyFromArray(ev.getSemanticTypes().toArray
(new String[0]), 0, 0, ev.getSemanticTypes().size());
currentCandidate.setSemanticTypes(semTypeArray);
semTypeArray.addToIndexes();
StringArray matchedWordArray = new StringArray(jcas, ev.getMatchedWords().size());
matchedWordArray.copyFromArray(ev.getMatchedWords().toArray
(new String[0]), 0, 0, ev.getMatchedWords().size());
currentCandidate.setMatchedwords(matchedWordArray);
matchedWordArray.addToIndexes();
FSArray fsArray = createSpanFSArray(jcas, ev.getPositionalInfo(), citationStart);
currentCandidate.setSpans(fsArray);
fsArray.addToIndexes();
candidates.add(currentCandidate);
currentCandidate.addToIndexes();
}
return candidates;
}
List processMappings(JCas jcas, PCM pcm, int citationStart)
throws Exception
{
org.metamap.uima.ts.Mapping currentMapping;
List candidates;
List mappings = new ArrayList();
for (gov.nih.nlm.nls.metamap.Mapping map : pcm.getMappingList()) {
// System.out.println("MetaMapAnnotator: map.getScore(): " + map.getScore());
currentMapping = new org.metamap.uima.ts.Mapping(jcas);
candidates = processCandidates(jcas, map.getEvList(), citationStart);
FSArray mappedCandidatesFSArray = new FSArray(jcas, candidates.size());
mappedCandidatesFSArray.copyFromArray(candidates.toArray
(new org.metamap.uima.ts.Candidate[0]),
0, 0, mappedCandidatesFSArray.size());
currentMapping.setCandidates(mappedCandidatesFSArray);
mappedCandidatesFSArray.addToIndexes();
mappings.add(currentMapping);
currentMapping.setScore(map.getScore());
currentMapping.addToIndexes();
}
return mappings;
}
public FSArray createSpanFSArray (JCas jcas, List positionList, int citationStart) {
List spans = createSpanList(jcas, positionList, citationStart);
FSArray fsArray = new FSArray(jcas, spans.size());
fsArray.copyFromArray(spans.toArray(new org.metamap.uima.ts.Span[0]),
0, 0, fsArray.size());
return fsArray;
}
public List createSpanList(JCas jcas, List positionList, int citationStart) {
List spans = new ArrayList();
for (Position pos : positionList) {
Span currentSpan = new Span(jcas); ;
if (pos != null) {
currentSpan.setBegin(citationStart + pos.getX());
currentSpan.setEnd(citationStart + pos.getX() + pos.getY()); // getY() actually is the Span length (http://skr.nlm.nih.gov/Help/MMO_08_Info.html)
} else {
System.err.println("Null positional information in pos: " + pos);
}
spans.add(currentSpan);
currentSpan.addToIndexes();
}
return spans;
}
/*
* Alerts this AnalysisComponent that the values of its configuration
* parameters or external resources have changed.
*/
//public void reconfigure() {
//}
/** Sets the ResultSpecification that this AnalysisComponent should use. */
//public void setResultSpecification(ResultSpecification aResultSpec) {
//}
/** add any extra options (besides machine output) to be used for processing, see parameter metamap_options */
private void setupMetaMapOptions() {
// Mandatory options.
if ((this.metamapOptions != null) && (! this.metamapOptions.isEmpty())) {
this.api.setOptions(metamapOptions);
}
}
/**
* Modified version of getParameters() from Kai Schlamp's BasicMetaMapAnnotator.
*/
private void getParameters() {
this.mmserverPath = (String) getContext().getConfigParameterValue(MMSERVER_PATH_PARAMETER);
this.skrmedpostctlPath = (String) getContext().getConfigParameterValue(SKRMEDPOSTCTL_PATH_PARAMETER);
this.wsdserverctlPath = (String) getContext().getConfigParameterValue(WSDSERVERCTL_PATH_PARAMETER);
// tempdirPath = (String) getContext().getConfigParameterValue(TEMPDIR_PATH_PARAMETER);
this.metamapOptions = (String) getContext().getConfigParameterValue(METAMAP_OPTIONS_PARAMETER);
this.apiTimeoutInterval = (String) getContext().getConfigParameterValue(METAMAP_TIMEOUT_PARAMETER);
}
/**
* Taken from Kai Schlamp's BasicMetaMapAnnotator.
*
*/
private void startTaggerServer() {
File skrmedpostctlFile = null;
if (this.skrmedpostctlPath != null && !this.skrmedpostctlPath.isEmpty()) {
skrmedpostctlFile = new File(this.skrmedpostctlPath);
if (!skrmedpostctlFile.isFile()) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "wrong_skrmedpostctl_path");
skrmedpostctlFile = null;
}
if (!skrmedpostctlFile.canExecute()) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "not_executable_skrmedpostctl");
skrmedpostctlFile = null;
}
}
boolean taggerServerRunning = false;
String os = System.getProperty("os.name");
if ((skrmedpostctlFile != null) && (os.contains("Linux") || os.contains("linux"))) {
try {
taggerServerRunning = checkProcess("taggerServer");
if (!taggerServerRunning && skrmedpostctlFile != null) {
ProcessBuilder taggerServerStartProcessBuilder = new ProcessBuilder(skrmedpostctlFile.getAbsolutePath(), "start");
try {
taggerServerStartProcessBuilder.start().waitFor();
Thread.sleep(2000);
} catch (InterruptedException e) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "failure_tagger_server_start", e);
}
}
}
catch (IOException e) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "failure_searching_server_processes", e);
}
}
}
/**
* Taken from Kai Schlamp's BasicMetaMapAnnotator.
*
*/
private void startDisambiguationServer() {
File wsdserverctlFile = null;
if (this.wsdserverctlPath != null && !this.wsdserverctlPath.isEmpty()) {
wsdserverctlFile = new File(this.wsdserverctlPath);
if (!wsdserverctlFile.isFile()) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "wrong_wsdserverctl_path");
wsdserverctlFile = null;
}
if (!wsdserverctlFile.canExecute()) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "not_executable_wsdserverctl");
wsdserverctlFile = null;
}
}
boolean disambiguatorServerRunning = false;
String os = System.getProperty("os.name");
if ((wsdserverctlFile != null) && (os.contains("Linux") || os.contains("linux"))) {
try {
disambiguatorServerRunning = checkProcess("wsd.server.DisambiguatorServer");
if (!disambiguatorServerRunning && wsdserverctlFile != null) {
ProcessBuilder disambiguatorStartProcessBuilder = new ProcessBuilder(wsdserverctlFile.getAbsolutePath(), "start");
try {
disambiguatorStartProcessBuilder.start().waitFor();
Thread.sleep(2000);
} catch (InterruptedException e) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "failure_disambiguator_server_start", e);
}
}
}
catch (IOException e) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "failure_searching_server_processes", e);
}
}
}
/**
* Taken from Kai Schlamp's BasicMetaMapAnnotator.
*
*/
private boolean checkProcess(String processName) throws IOException {
ProcessBuilder processBuilder = new ProcessBuilder("ps", "-ef");
Process psProcess = processBuilder.start();
BufferedReader input = new BufferedReader(new InputStreamReader(psProcess.getInputStream()));
String line;
while ((line = input.readLine()) != null) {
if (line.contains(processName)) {
return true;
}
}
input.close();
return false;
}
private void startMetaMapServer() {
File mmserverFile = null;
if (this.mmserverPath != null && !this.mmserverPath.isEmpty()) {
mmserverFile = new File(this.mmserverPath);
if (!mmserverFile.isFile()) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "wrong_mmserver_path");
mmserverFile = null;
}
if (mmserverFile != null && !mmserverFile.canExecute()) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "not_executable_mmserver");
mmserverFile = null;
}
}
boolean disambiguatorServerRunning = false;
String os = System.getProperty("os.name");
if ((mmserverFile != null) && (os.contains("Linux") || os.contains("linux"))) {
try {
disambiguatorServerRunning = checkProcess("mmserver");
if (!disambiguatorServerRunning && mmserverFile != null) {
ProcessBuilder disambiguatorStartProcessBuilder = new ProcessBuilder(mmserverFile.getAbsolutePath(), "start");
try {
disambiguatorStartProcessBuilder.start().waitFor();
Thread.sleep(2000);
} catch (InterruptedException e) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "failure_disambiguator_server_start", e);
}
}
}
catch (IOException e) {
getContext().getLogger().logrb(Level.WARNING, null, null, MESSAGE_DIGEST, "failure_searching_server_processes", e);
}
}
}
}
--------------080107010100070101050205
Content-Type: text/xml; name="MetaMapApiTypeSystem.xml"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="MetaMapApiTypeSystem.xml"
Document
Document Representation
1.0
org.metamap.uima.ts.Candidate
The meta elements found in the text. Those are then used as candidates and mappings.
uima.tcas.Annotation
score
The score of this meta element (0 <= x <= 1000).
uima.cas.Integer
spans
uima.cas.FSArray
org.metamap.uima.ts.Span
cui
uima.cas.String
concept
uima.cas.String
preferred
uima.cas.String
head
uima.cas.Boolean
overmatch
uima.cas.Boolean
sources
uima.cas.StringArray
semanticTypes
uima.cas.StringArray
matchedwords
uima.cas.StringArray
matchMap
uima.cas.FSArray
gov.nih.nlm.nls.metamap.uima.ts.MatchMap
org.metamap.uima.ts.Mapping
A meta mapping.
uima.cas.TOP
score
The score of this mapping.
uima.cas.Integer
candidates
An array of elements of this mapping.
uima.cas.FSArray
org.metamap.uima.ts.Candidate
org.metamap.uima.ts.Phrase
uima.tcas.Annotation
candidates
uima.cas.FSArray
org.metamap.uima.ts.Candidate
mappings
uima.cas.FSArray
org.metamap.uima.ts.Mapping
org.metamap.uima.ts.Span
uima.tcas.Annotation
org.metamap.uima.ts.Utterance
uima.tcas.Annotation
pmid
uima.cas.String
phrases
uima.cas.FSArray
org.metamap.uima.ts.Phrase
location
uima.cas.String
org.metamap.uima.ts.Negation
uima.tcas.Annotation
negType
uima.cas.String
negTrigger
uima.cas.String
ntSpans
uima.cas.FSArray
org.metamap.uima.ts.Span
cuiConcepts
uima.cas.FSArray
org.metamap.uima.ts.CuiConcept
ncSpans
uima.cas.FSArray
org.metamap.uima.ts.Span
org.metamap.uima.ts.CuiConcept
uima.cas.TOP
negExCui
uima.cas.String
negExConcept
uima.cas.String
gov.nih.nlm.nls.metamap.uima.ts.MatchMap
uima.tcas.Annotation
phraseMatchStart
uima.cas.Integer
phraseMatchEnd
uima.cas.Integer
conceptMatchStart
uima.cas.Integer
conceptMatchEnd
uima.cas.Integer
lexVariation
uima.cas.Integer
gov.nih.nlm.nls.metamap.uima.ts.AcronymAbbrev
Container for MetaMap Acronym or Abbreviation annotation.
uima.tcas.Annotation
acronym
Acronym or Abbreviation
uima.cas.String
expansion
expansion of the acronym or abbreviation
uima.cas.String
countList
count list
uima.cas.IntegerArray
cuiList
list of concept id for acronym or abbreviation
uima.cas.StringArray
gov.nih.nlm.nls.metamap.uima.ts.Document
A citation or item of free-text
uima.tcas.Annotation
acronymsAbbrevs
acronym and abbreviations
uima.cas.FSArray
gov.nih.nlm.nls.metamap.uima.ts.AcronymAbbrev
negations
uima.cas.FSArray
org.metamap.uima.ts.Negation
utterances
uima.cas.FSArray
org.metamap.uima.ts.Utterance
--------------080107010100070101050205--