The primary function of the Entrez programmatic interface is to help users manipulate sets of UIDs, and fetch data records identified by those UIDs. Entrez, itself, must also format data for Web-based users, but the programmatic interface leaves data display to the client program.

The interface allows programs to:

• define a set of UIDs,

• display the contents of records identified by a set of UIDs,

• create a new UID set from an existing set by choosing members of the existing set whose data records satisfy specified criteria, and

• create a new set of UIDs representing records that are in some way related to members of the records identified by an existing set of UIDs.

These primitive capabilities can be combined into powerful sequences that can integrate data from among most of the Entrez data resources. In fact, they make it possible to (partially) mimic relational database operations such as selects and joins on data in separate data resources.

Note, however, that record content may be retrieved in a limited number of report formats, where a report type contains a fixed subset of elements taken from the raw data record. As a result, additional processing may be required to prune report data for subsequent display or use, and/or multiple requests may be required to retrieve data in multiple report formats to obtain all desired data fields.

The query result database can be used by (most of the) programs that implement the Entrez set manipulation functions listed above, and is so important for efficient use of Entrez that this presentation is almost entirely oriented around it. "Efficient" use of the query result database allows users to download large numbers of records without violating the access rate limits that NCBI imposes upon remote queries.

Each UID set in the Core database is identified by 3 pieces of information:

• a query identifier, known as the "query key",

• the name of the database used to generate the associated UID set, and

• an identifier for the state of the database at the time of the query, known as the "web environment".

Query keys are integers, but are often displayed as a pound sign (#) followed by an integer. The Entrez databases names are strings like "snp", "nuc", "nucest", "gene", etc. Web environment identifiers are long (around 60 character) strings.

Here is a schematic query result database entry:

The two most popular approaches are:

• the Web Common Gateway Interface (CGI), and

• Remote Procedure Calls (RPC) over SOAP, sometimes known as JAX-RPC or "Web Services".

NCBI supports both of these interfaces to the Entrez Core. In addition, NCBI provides an educational Perl module (NCBI_PowerScripting.pm) that defines a set of objects that call the CGI services behind the scenes.

The CGI and Web Services routines are known as the "eUtilities" or "eUtils", and may be categorized with respect to the UID manipulation functions listed above as:

This presentation will deal only with the CGI functions, but the Web Services provide identical functionality within the JAX-RPC framework. (Note that the Web Services are not currently, circa 2007, available via Perl.)

Here is an URL that uses the epost.fcgi script to insert (or "post") 2 UIDs (242 and 2885398) into the query result database:

   http://eutils.ncbi.nlm.nih.gov/entrez/eutils/epost.fcgi?db=snp&id=242,28853987

and the query result database will then include a new record containing the 2 UIDs specified by using the "id" parameter:

If you then specify the "db", "query_key", and "WebEnv" parameters in a URL like:

  http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=snp&query_key=1&\
    WebEnv=01yWrS_p1DdWzAUPU6eOwxX2...s@1FBE5D3B634BAF50_0012SID

#!/usr/bin/perl -w

use LWP::Simple;

$url = 
 "http://eutils.ncbi.nlm.nih.gov/entrez/eutils/epost.fcgi?db=snp&id=242,28853987";

@result_array = get ( "$url" );  # note that epost is returning an array lines of XML.

print @result_array;

Now here is a slight modification that will filter a result_array to print only the query key and Web environment values:

bash-2.05$ perl test-ncbi-2.pl
Query Key: 1
Web Environment: 03bGQckqzaWiGXQqZHvYpXvTh...oNb-J@1FBE58C06361F480_0005SID

Now the eSummary query can be added to the program to get the contents of records identified by their UIDs. Note, however, that eSummary returns a long string, rather than an array of strings (as did ePost). The long string can be split up into separate lines, each of which is placed into a separate array entry by using the Perl function "split" and the Perl representation for a line separator: the "newline" or '\n' character.

The resulting program looks like this:

bash-2.05$ perl test-ncbi-4.pl

SNP_ID = 242
GENE = 
FXN_CLASS = 
TAX_ID = 9606
SNP class = in-del
Chromosome:Position = 1:20742047

SNP_ID = 28853987
GENE = LOC653635
FXN_CLASS = locus-region
TAX_ID = 9606
SNP class = snp
Chromosome:Position = 1:800

Note that this program extracts significant information by using a set of print...if statements that constitute a "naive" XML parsing process that may be too inefficient for processing larger or more complex files. Such cases may require the use of a formal XML parser. See the Perl CPAN site for some options.

(Note also that this parsing step can occur "automatically" as part of the SOAP message processing performed by some of the Web Services versions of the eUtilties, but not by the CGI versions.)

• search an NCBI database and put the list of UIDs whose contents match the search conditions into a new query result database entry, and

• search an existing query result entry UID list and put UIDs whose content matches search conditions into a new query result database entry.

eSearch can be used to insert a new entry in the query result database by using a query like:

  http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=snp&\
     usehistory=y&term=1[CHR]+AND+9606[TAX_ID]

which requests the UIDs for all SNPs located on human chromosome 1.

Note the presence of the "usehistory" parameter. Without this parameter eSearch will return a UID list to the user, but will NOT insert the UID list into the query result database.

eSearch can also be used to create a new UID list from an existing list by using a query like:

  http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=snp&query_key=1&\
    WebEnv=01yWrS_p1DdWzAUPU6eOwxX2...s@1FBE5D3B634BAF50_0012SID&\
    term=snp[FXN_CLASS]&usehistory=y

In this query the existing list is identified by the db, query_key, and WebEnv parameters, and eSearch is directed to select only those SNPs whose records include a SNP function class tag, and store the resulting UID list in a new query database entry (by the "usehistory" parameter).

 http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?db=all\
     &id=242&dbfrom=snp&cmd=neighbor_history

The information "linked to" can be retrieved by specifying the query key and Web environment in eSummary or eFetch requests. For example the the information in the unists database that is linked to by the request above can be retrieved by using a request like:

   http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=unists&\
   query_key=4\&WebEnv=00O9qrUHud2D8LXbpvILvTB...1FBE6E626363B8F0_0105SID

The output from this program for the JAK3 UID (3718) is a list of selected information for every SNP linked to from UID 3718. The output is an extended version of the following:

Note that the program above essentially implements an SQL select command similar to:

This limitation may cause eUtility requests containing long lists of UIDs to fail, and to avoid this problem you must use the Perl module LWP in place of LWP::Simple. If you use LWP you must rewrite all your eUtil calls, since both HTTP GETs and POSTs are handled differently with LWP.

Here is an example ePost invocation using LWP. Note that the "email" and "tool" parameters are used to identify the requester and requesting program, and they can be used with LWP::Simple get requests as well. NCBI uses this information to help manage server access problems such as request overloads:

Having posted the UID list, the record content for each UID can be retrieved using either the LWP::Simple get method or the LWP post method. However, the number of UIDs in $db_list should be determined first, and eFetch content should be downloaded in "batches" of not more than 500 UIDs to conform to NCBI guidelines. In addition, request initiations should be spaced at least 15 seconds apart during peak-usage hours and 3 seconds apart during off-peak hours.

To facilitate this limitation, the eFetch and eSummary routines allow users to specify the first UID entry to be processed and the maximum number to be processed, beginning with the first, by using the "retstart" and "retmax" parameters, respectively.

Here is a continuation of the previous example that will print the eFetch content for a list of UIDs in the query result database entry. This code uses LWP rather than LWP::Simple to send eFetch requests, and waits at least 3 seconds between request initiations.

Also, if this code fragment is to operate on an extant query result database entry, you can find out how many items are in the list by using a search like:

Note that most eSummary or eFetch requests asking for batches of 500 records will take more than 3 seconds, so that the NCBI restrictions will not signficantly degrade performance during off-peak work. They will, however, assure that multiple simultaneous requests will get their fair shares of the system resources.

The method described earlier that uses eLink to get SNP lists for a single gene, may not generalize efficiently to getting separate SNP lists for a large number of genes. The problem is that there may be too many genes to process serially in a reasonable amount of time, and separate SNP lists for each gene might not result in large enough batch requests.