You are here

C3PI: Computational Photography Project for Pill Identification

Project information
Researchers: 

The Computational Photography Project for Pill Identification (C3PI) is creating:

  • The RxIMAGE database of freely available high-quality digital images of prescription pills and associated data
  • The freely-accessible RxIMAGE API (Application Programming Interface) that software developers can use to create apps for text-based search and retrieval from the RxIMAGE database for public use
  • The SPLIMAGE database and SPLIMAGE portal to help the pharmaceutical industry include pill images in the information it provides to the public

By developing and using these resources, C3PI computer scientists conduct computer vision research in text- and image-based search and retrieval. They seek object identification metrics and methods that are invariant with respect to camera angle, lighting, and the color transfer functions often found in digital cameras.

RxIMAGE: DATABASE OF PILL IMAGES AND DATA FOR THE PUBLIC, API FOR SOFTWARE DEVELOPERS TO ACCESS THE DATABASE

  • The RxIMAGE database is the Nation’s only portfolio of curated, freely available, increasingly comprehensive, high-quality digital images of prescription pills and associated data. Examples of pills are capsules and tablets intended for oral use.
  • Photographs of pills for the RxIMAGE database were taken under laboratory lighting conditions, from a camera directly above the front and the back faces of the pill, at high resolution, and using specialized digital macro-photography techniques. Image segmentation algorithms were then applied to create the JPEG images in the database.
  • Software developers can use the freely accessible RxIMAGE API to create apps for text-based search and retrieval from the RxIMAGE database.
  • The general public can use those apps to search for and retrieve images and data from the RxIMAGE database.
  • Usage example: A health care provider uses a mobile device to call up an app that executes an RxIMAGE API request to search the RxIMAGE database for an unknown pill by appearance. That may be the only way to identify the pills an unconscious person or an evacuee has on his or her person during a natural disaster.

PILL IMAGE RECOGNITION CHALLENGE

The NLM Pill Image Recognition Challenge was announced on January 19, 2016, and the winners were announced on August 1, 2016. The Challenge Web site includes information about the Challenge and the winners.

The Challenge invited individuals and teams to submit algorithms and software for pill image recognition. The submissions were to rank images taken from NLM's RxIMAGE database of high-quality prescription pill images by similarity to consumer-quality images of the same pills. NLM plans to use the submissions in creating a future software system for future Application Programming Interfaces (APIs) that can be used in identifying an unknown prescription pill from a photo taken of that pill by a smart phone.

The Challenge was preceded by a Pill Image Recognition pilot. The Pill Image Recognition (PIR) Request For Information (RFI) Pilot and its Instructions for Responding were posted on February 2, 2015.

SPLIMAGE: PILL IMAGE DATABASE AND PORTAL FOR THE PHARMACEUTICAL INDUSTRY

  • The SPLIMAGE database and SPLIMAGE portal help the pharmaceutical industry include high-quality pill images in the prescription pill information they make available to health care providers and consumers.
  • SPL stands for Structured Product Labeling, “a document markup standard approved by Health Level 7 (HL7) and adopted by the FDA as a mechanism for exchanging product and facility information”[1]. FDA is the U.S. Food and Drug Administration.
  • “Participants” are pharmaceutical manufacturers, private-label distributors, re-packagers, wholesalers, and retail and institutional pharmacies that have completed their response to Federal Register Notice 79 FR 56381 entitled “Call for Participation for Computational Photography Project for Pill Identification (C3PI)”. Only participants can use the SPLIMAGE portal.
  • At no cost, participants can follow the instructions at the SPLIMAGE portal to send pills to NLM’s imaging facility and have those pills photographed.
  • Also at no cost, participants can then retrieve files of images of those photographed pills from the SPLIMAGE database.
  • The image files in the SPLIMAGE database meet the recommended SPL image guidelines described below.
  • Participants can include those images in their submissions to the FDA. Images approved through the FDA SPL program can appear on printed patient inserts that accompany prescription pills.
  • Usage example: A person picking up a prescription at a pharmacy checks that the pills he or she is given look like the pill image, which came from the SPLIMAGE database, on the printed patient insert provided with the prescription.

SPL IMAGE GUIDELINES

C3PI developed recommended SPL image guidelines in collaboration with the FDA. The guidelines address image size, color, file format, and file size; layout; scales; and image processing. The overall goal of the guidelines is that a person or a computing system can readily identify a pill’s distinguishing visual characteristics, except for thickness.

DIFFERENCES BETWEEN RxIMAGE IMAGES AND SPLIMAGE IMAGES

The “views” of the fronts and the backs of pills are the same in the RxIMAGE database and the SPLIMAGE database. However, in images in the RxIMAGE database, the background includes an NLM watermark, and other characteristics are different from images in the SPLIMAGE database. Consequently, images in the RxIMAGE database do not meet the guidelines.

RxIMAGE USAGE BY NLM

Images and associated data in the RxIMAGE database are used by NLM’s DailyMed and RxNav.

ACKNOWLEDGEMENT

NLM/LHNCBC/OHPCC appreciates the high quality RxIMAGE and SPLIMAGE imaging and database work being done under contract by Medicos Consultants, LLC.


[1] http://www.fda.gov/forindustry/datastandards/structuredproductlabeling/default.htm

 

Publications/Tools: 
Yoo TS. Dataset: SPLIMAGE database
Yoo TS. Dataset: RxIMAGE database
Caban JJ, Rosebrock A, Yoo TS. Automatic identification of prescription drugs using shape distribution models. In: 19th IEEE International Conference on Image Processing (ICIP); 2012 Sep 30-Oct 3; Lake Buena Vista, Florida. p. 1005-8. DOI: 10.1109/ICIP.2012.6467032