ERDDAP   Easier access to scientific data

Brought to you by NOAA NMFS SFSC ERD

ERDDAP is an all-open source, all-Java (servlet), web application that runs in a web application server (for example, Tomcat).
This web page is mostly for people ("ERDDAP administrators") who want to set up their own ERDDAP installation at their own web site.

Why use ERDDAP to distribute your data?

• If you already have a web service for distributing your data, you can set up ERDDAP to access your data via the existing service.
  Or, you can set up ERDDAP to access your data directly from local files.
• For each dataset, you only have to write a small chunk of XML to tell ERDDAP how to access the dataset.
• Once you have ERDDAP serving your data, end users can:
  • Request the data in various ways (DAP, WMS, and more in the future).
  • Get the response in various file formats. (That's probably the biggest reason!)
  • Make graphs and maps. (Everyone likes pretty pictures.)

How To Do the Initial Setup of ERDDAP on Your Server

1. ERDDAP should run on any server that supports Java and Tomcat (and perhaps other application servers).
   ERDDAP has been tested on Linux and Windows computers.
2. Type "java -version" from your server's command line to make sure you have Java version 1.5 update 11 (or higher) or version 1.6 update 1 (or higher) installed.
3. Set up Tomcat (or some other application server) on your server.
   Below, the Tomcat directory will be referred to as <tomcat> .
   Warning! If you already have a Tomcat running some other web application (especially THREDDS), we recommend that you install ERDDAP in a second Tomcat, because ERDDAP may need different settings and shouldn't have to contend with other applications for memory.
   • On Linux, add "export JAVA_HOME=YourJavaDirectory" to the top of <tomcat>/bin/startup.sh .
     On Windows, you probably don't need to set JAVA_HOME.
   • On Linux, add "export JAVA_OPTS='-Djava.awt.headless=true -Xmx1500M' " to the top of <tomcat>/bin/startup.sh .
     On Windows, right click on the Tomcat Manager icon in the system tray;
       select "Configure : Java";
        set "Maximum Memory Pool" to 1500 (or less if just for testing?) and
        (optional) add "-verbose:gc" to the list of Java Options.

     The Xmx memory setting is important because, ERDDAP can run out of memory.
     The more physical memory in the server the better: 4+ GB is really good
     (extra is used for swap space, which is useful), 2 GB is okay, less is not recommended.
     Even with abundant physical memory, Tomcat/Java won't run if you try to set -Xmx much above 1500M.
     If your server has less than 2GB of memory, reduce the -Xmx value (in 'M'egaBytes) to 1/2 of the physical memory.

     Optional: You can add "-verbose:gc" to the JAVA_OPTS. It tells Java to send Java garbage collection information to <tomcat>/logs/catalina.out
     (or some other Tomcat log file) which is useful if your ERDDAP has memory problems.

   • Optional fonts for images: Download BitstreamVeraSans.zip and unzip it into <JAVA_HOME>/jre/lib/fonts so Java can find the fonts.
     This isn't essential, but we prefer these free fonts to the standard Linux/Java fonts.
   • Troubles with the Tomcat installation?
     See the Tomcat web site or search the web for help,
     but please let us know the problems you had and the solutions you found.
4. If Tomcat is running in Apache, you need to modify the /etc/httpd/conf/httpd.conf file to allow HTTP traffic to/from ERDDAP:
   To the "VirtualHost" tag, add the lines:
      ProxyPass /erddap <YourServer'sURL>:8080/erddap
      ProxyPassReverse /erddap <YourServer'sURL>:8080/erddap
5. Download erddapContent.zip and unzip it into <tomcat>, creating <tomcat>/content/erddap .
   • Read the comments in <tomcat>/content/erddap/setup.xml and make the requested changes.
     setup.xml is the file with all of the settings which specify how your ERDDAP behaves.
   • Read the comments in Working with the datasets.xml File, then modify the XML in
     <tomcat>/content/erddap/datasets.xml to specify all of the datasets you want your ERDDAP to serve.
   After you edit the .xml files, it is a good idea to verify that the result is well-formed XML by pasting the XML text into an XML checker like RUWF.
6. Download erddap.war into <tomcat>/webapps . Restart Tomcat.
   Or, if you use the Tomcat Web Application Manager:
   * Download erddap.war into a temporary directory on your computer.
   * Use "Select WAR file to upload" to pick the erddap.war file.
   * Click on "Deploy".
   The .war file is big because it contains high resolution coastline, boundary, and elevation data needed to create maps.
7. Hopefully, you can now use a browser to view <YourServer'sURL>/erddap/ and see ERDDAP immediately.
   ERDDAP starts up immediately, but without any datasets loaded. Datasets are loaded in a background thread and so become available one-by-one.

How To Do an Update of ERDDAP on Your Server

1. Download erddapContent.zip and unzip it into a temporary directory.
   • If you made changes to your previous copy of messages.xml, make the same changes to the new messages.xml in the temporary directory.
   • Move messages.xml from the temporary directory into <tomcat>/content/erddap .
   • Keep using your current setup.xml and datasets.xml.
2. Make the changes recommended in List of Changes for all of the ERDDAP versions since the version you were using.
3. Download erddap.war into a temporary directory.
4. In Tomcat Manager, "Undeploy" ERDDAP.
5. In Tomcat Manager, "Deploy" the erddap.war file.

Things You Need To Know

• Status Page - You can view the status of your ERDDAP from any browser by going to <baseUrl>/erddap/status.html .
  This page is generated dynamically, so it always has up-to-the-moment statistics for your ERDDAP.
  It includes statistics regarding the number of requests, memory usage, thread stack traces, the taskThread, etc.
  Because the Status Page can be viewed by anyone, it doesn't include quite as much information as the Daily Report.
   
• log.txt - When you are first installing ERDDAP or any time something isn't working as expected, it is very useful to see a log file with diagnostic messages.
  ERDDAP always puts diagnostic messages in the log.txt file in <bigParentDirectory>/logs (as specified in setup.xml).
  You can control the level of messages by setting <logLevel> (info or all) in setup.xml.
  When the log.txt files gets to 5 MB, it is renamed log.txt.previous and a new log.txt file is created.
  So log files don't accumulate.
  Whenever you restart ERDDAP, it makes an archive copy of the log.txt and log.txt.previous files with a time stamp.
  If there was trouble before the restart, it may be useful to analyze these archived files
  for clues as to what the trouble was.
  You can delete the archive files if they are no longer needed.
   
• Tomcat Log Files - Tomcat errors are logged to <tomcat>/logs/catalina.(today).log.
  These are often particularly helpful if ERDDAP won't even start up.
• emailLogYEAR-MM-DD.txt - ERDDAP always writes the text of all out-going email messages in the current day's emailLogYEAR-MM-DD.txt file
  in <bigParentDirectory>/logs (as specified in setup.xml).
  If the server can't send out email messages, or if you have configured ERDDAP not to send out email messages,
  or if you are just curious, this file is a convenient way to see all of the email messages that have been sent out.
  You can delete previous days' email log files if they are no longer needed.
   
• Daily Report - The Daily Report has lots of useful information -- more than the Status Page.
  It is the most complete summary of your ERDDAP's status.
  Among other statistics, it includes a list of datasets that didn't load and the exceptions they generated.
  It is emailed to <emailDailyReportsTo> and <emailEverythingTo> (as specified in setup.xml) provided you have set up the email email system (in setup.xml).
  It is also written to the log file.
  It is generated when you start up ERDDAP, just after ERDDAP's first attempt to load all of the datasets.
  It is also generated soon after 7 am local time every morning.
   
• Adding/Changing Datasets - ERDDAP usually rereads datasets.xml every <loadDatasetsMinMinutes> (specified in setup.xml). So you can make changes to datasets.xml any time, even while ERDDAP is running. A new dataset will be detected soon, usually within <loadDatasetsMinMinutes>. A changed dataset will be reloaded when it is <reloadEveryNMinutes> old (as specified in datasets.xml).
   
• A Flag File Tells ERDDAP to Try to Reload a Dataset As Soon As Possible
  If a dataset is active in ERDDAP and you want to force ERDDAP to reload it as soon as possible (before the dataset's <reloadEveryNMinutes> would cause it to be reloaded), put a file in <bigParentDirectory>/flag (as specified in setup.xml) that has the same name as the datasetID (as specified in datasets.xml).
  The contents of the flag file are irrelevant.
  ERDDAP continuously looks for flag files.
  When ERDDAP finds a flag file, it deletes the file and tries to reload the dataset very soon (usually within seconds).
  Note that when a dataset is reloaded, all files in the bigParentDirectory/cache/datasetID directory are deleted.
  Note that if the dataset's xml includes active="false", a flag will cause the dataset to be made inactive (if it is active), and in any case, not reloaded.

  ERDDAP has a web service so that flags can be set via URLs. For example,
  http://coastwatch.pfeg.noaa.gov/erddap/setDatasetFlag.txt?datasetID=rPmelTao&flagKey=123456789
  (but with a different flagKey) will set a flag for the rPmelTao dataset.
  There is a different flagKey for each datasetID.
  Administrators can see a list of flag URLs for all datasets by looking at the bottom of their Daily Report email.
  Administrators should treat these URLs as confidential, since they give someone the right to reset a dataset at will.
  If you think the flagKeys have fallen into the hands of someone who is abusing them,
  you can change <flagKeyKey> in setup.xml and restart ERDDAP to force
  ERDDAP to generate and use a different set of flagKeys.
  If you change <flagKeyKey>, remember to send the new URLs to the people who you do want to have them.

  The flag system can serve as the basis for a more efficient mechanism for telling ERDDAP when to reload a dataset.
  For example, you could set a dataset's <reloadEveryNMinutes> to a large number (e.g., 10080 = 1 week).
  Then, when you know the dataset has changed (perhaps because you added a file to the dataset's data directory),
  set a flag so that the dataset is reloaded as soon as possible.
  This is much more responsive and much more efficient than setting <reloadEveryNMinutes> to a small number.

• Force Dataset Removal - If a dataset is active in ERDDAP, and you want to deactivate it as soon as possible,
  add active="false" to the dataset tag and set a flag.
  Once the dataset is not active (i.e., not visible in ERDDAP's list of datasets),
  you can remove the dataset's description from the datasets.xml file if you want to.
   
• Cached Responses - In general, ERDDAP doesn't cache (store) responses to user requests.
  The rationale was that most requests would be slightly different so the cache wouldn't be very effective.
  The biggest exceptions are requests for image files
  (which are cached since browsers and programs like Google Earth often re-request images)
  and requests for .nc files (because they can't be created on-the-fly).
  ERDDAP stores each dataset's cached files in a different directory: bigParentDirectory/cache/datasetID
  since a single cache directory might have a huge number of files which might become slow to access.
  Files are removed from the cache for one of three reasons:
  • All files in this cache are deleted when ERDDAP is restarted.
  • Periodically, any file more than <cacheMinutes> old (as specified in setup.xml) will be deleted.
    Removing files in the cache based on age (not Least-Recently-Used) ensures that files won't stay in the cache very long.
    Although it might seem like a given request should always return the same response, that isn't true.
    For example, a tabledap request which includes &time>someTime will change if new data arrives for the dataset.
    And a griddap request which includes [last] for the time dimension will change if new data arrives for the dataset.
  • Images showing error conditions are cached, but only for a few minutes (it's a difficult situation/choice).
  • Every time a dataset is reloaded, all files in that dataset's cache are deleted.
    Because requests may be for the "last" index in a gridded dataset,
    files in the cache may become invalid when a dataset is reloaded.
     
• Stored Dataset Information -
  For all types of datasets, ERDDAP gathers lots of information when a dataset is loaded and keeps that in memory.
  This allows ERDDAP to respond very quickly to searches, requests for lists of datasets, and requests for information about a dataset.

  For a few types of datasets (notably EDDGridCopy, EDDTableCopy, EDDGridFromXxxFiles, and EDDTableFromXxxFiles),
  ERDDAP stores on disk some information about the dataset that is reused when the dataset is reloaded.
  This greatly speeds the reloading process.
  The dataset information files are human-readable .json files and are stored in <bigParentDirectory>/datasetInfo .
  ERDDAP only deletes these files in unusual situations.
  It shouldn't ever be necessary for you to delete these files because the datasets verify and update the stored information when they reload.
  But if you ever do need to delete these files (why?), you can do it when ERDDAP is running. (Then set a flag.)
  If you want to encourage ERDDAP to update the stored dataset information
  (for example, if you just added, removed, or changed some files to the dataset's data directory),
  use the flag system.

• sitemap.xml - As the www.sitemaps.org web site says:

  Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.

  Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using the associated metadata. Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.

  Actually, since ERDDAP is RESTful, search engine spiders can easily crawl your ERDDAP.
  But they tend to do it more often (daily!) than necessary (monthly?).
  Given that each search engine may be crawling your entire ERDDAP every day,
  this can lead to a lot of unnecessary requests.
  So ERDDAP generates a sitemap.xml file for your ERDDAP which tells search engines
  that your ERDDAP only needs to be crawled every month.
  You can and should tell the various search engines about the sitemap.xml file by visiting these URLs
  (but change "www.yoursite.org" to your ERDDAP's URL):
  • http://search.yahoo.com/ping?sitemap=http%3A%2F%2Fwww.yoursite.org%2Ferddap%2Fsitemap.xml
  • http://www.altavista.com/ping?sitemap=http%3A%2F%2Fwww.yoursite.org%2Ferddap%2Fsitemap.xml
  • http://www.ask.com/ping?sitemap=http%3A%2F%2Fwww.yoursite.org%2Ferddap%2Fsitemap.xml
  • http://www.bing.com/ping?sitemap=http%3A%2F%2Fwww.yoursite.org%2Ferddap%2Fsitemap.xml
  • http://www.dogpile.com/ping?sitemap=http%3A%2F%2Fwww.yoursite.org%2Ferddap%2Fsitemap.xml
  • http://www.google.com/ping?sitemap=http%3A%2F%2Fwww.yoursite.org%2Ferddap%2Fsitemap.xml
  • http://www.live.com/ping?sitemap=http%3A%2F%2Fwww.yoursite.org%2Ferddap%2Fsitemap.xml
  Or, substitute other search engine URLs at the beginning of the URL.
  (I think) you just need to ping each search engine once. They will detect changes to sitemap.xml periodically.
   
• Email Notification of Updates - If you want to receive an email whenever a new version of ERDDAP is available, send an email to bob dot simons at noaa dot gov requesting this.
   
• Security - By default, ERDDAP runs as an entirely public server (using http) with no login system and no restrictions to data access.
  If you want to restrict access to some or all datasets to some users, you can use ERDDAP's built-in security system.
  When the security system is in use:
  • ERDDAP uses role-based access control.
    The ERDDAP administrator defines users (username, password, and roles information) with <user> tags in datasets.xml.
    And the ERDDAP administrator defines which roles have access to which datasets via an
    <accessibleTo> tag in datasets.xml for any dataset that shouldn't have public access.
  • The user's log in status (and a link to log in/out) will be shown at the top of every web page.
    (But a logged in user will appear to ERDDAP to be not logged in if he uses an http URL.)
    If a user tries and fails to log in 3 times, the user is blocked from trying to log in for 15 minutes.
  • If a user is not logged in, the user can use http URLs (the default) or https URLs (but why? it's less efficient).
    If a user is logged in, the user must use https URLs to continue to be treated as logged in.
    (This helps prevent session hijacking and sidejacking.)
  • Anyone who isn't logged in can access and use the public datasets.
    By default, private datasets don't appear in lists of datasets if a user isn't logged in.
    If the administrator has set setup.xml's <listPrivateDatasets> to true, they will appear.
    Attempts to request data from private datasets (if the user knows the URL) will be redirected to the login page.
  • Anyone who is logged in will be able to see and request data from any public dataset
    and any private dataset to which their role allows them access.
    By default, private datasets to which a user doesn't have access don't appear in lists of datasets.
    If the administrator has set setup.xml's <listPrivateDatasets> to true, they will appear.
    Attempts to request data from private datasets to which the user doesn't have access will be redirected to the login page.
  • The RSS information for all datasets is always available to anyone.
    (This is not ideal. But RSS readers won't ever log in, so they need access without loggin in.)
    But, since private datasets that aren't accessible aren't advertised, their RSS links are not advertised either.
  • Email subscriptions can only be set up when a user has access to a dataset.
    Once set up, they continue to function after the user has logged out.

  To set up the security system:

  • Do the standard ERDDAP initial setup.
  • In setup.xml,
    • Add/change the <authenticate> value from nothing to "custom" or "openid".
    • Add/change the <baseHttpsUrl> value.
    • Insert/uncomment &loginInfo; in <startBodyHtml> to display the user's log in/out info at the top of each web page.
  • Configure tomcat to support SSL (the basis for https connections)
    by creating a keystore with an self-signed site certificate.
    On Windows, Tomcat may have to move .keystore from "c:\Documents and Settings\you\.keystore" to
    "c:\Documents and Settings\Default User\.keystore" or "c:\.keystore"
    (see [tomcat]/logs/catalina.(today).log if the application doesn't load or users can't see the log in page).
    You can see when the .keystore certificate will expire by examining the certificate when you log in.
    For additional security, create a signed certificate from a trusted source. (Search the web for more information.)
  • If Tomcat is running in Apache, you need to modify the /etc/httpd/conf/httpd.conf file to allow HTTP traffic to/from ERDDAP:
    To the "VirtualHost" tag, add the lines:
       ProxyPass /erddap <YourServer'sHttpsURL>/erddap
       ProxyPassReverse /erddap <YourServer'sHttpsURL>/erddap
    (This is untested. If you do this and it works or doesn't work, let us know.)
  • In datasets.xml, create a <user> tag for each user with username, password, and roles information.
    This is the authorization part of ERDDAP's security system.
  • In datasets.xml, add an <accessibleTo> tag to each dataset that shouldn't have public access.
    <accessibleTo> lets you specify which roles have access to that dataset.
  • Restart Tomcat.
    Trouble? Check the Tomcat logs.
  • CHECK YOUR WORK! Any mistake could lead to a security flaw.
  • Check that the login page uses https (not http).
    Attempts to connect via http should be automatically redirected to https and port 8443.
    You may need to work with your network administrator to allow external web requests to access port 8443 on your server.
  • You can change the <user> and <accessibleTo> tags at any time.
    The changes will be applied at the next regular reload of any dataset,
    or immediately if you use a flag.
  • It worked for a few months, now users can't get to the log in page?
    Check the Tomcat logs. Your .keystore certificate may have expired and you may need to make a new one.
    You can see when the .keystore certificate will expire by examining the certificate when you log in.

  Authentication (logging in) - Currently, ERDDAP supports Custom and OpenID (recommended) authentication.
  We recommend OpenID because it frees you from storing and handling user's passwords.
  Remember that users often use the same password at different sites.
  So they may be using the same password for your ERDDAP as they do at their bank.
  That makes their password very valuable -- much more valuable to the user than the data they are requesting.
  So you need to do as much as you can to keep the passwords private. That is a big responsibility.
  OpenID takes care of passwords, so you don't have to gather, store, or work with them.
  So you are freed from that responsibility.

  • "Custom" is ERDDAP's custom system for letting users log in by entering their User Name and Password in a form on a web page.
    This is secure because the User Name and Password are transmitted via https (not http),
    but OpenID is better because it frees you from having to handle passwords.
    This approach requires you to collect User Names and Passwords (use your phone! email isn't secure!)
    and store them in datasets.xml in <user> tags.

    This approach uses a cookie on the user's computer, so the user's browser must be set to allow cookies.
    If a user is making ERDDAP requests from a computer program (not a browser), cookies are hard to work with. Sorry.

  • OpenID is an open standard that lets users log in with your password at one web site
    and then log in without your password at many other web sites, including ERDDAP.
    OpenID is very convenient for ERDDAP administrators -- you don't ever have to deal with passwords.
    All you need is a user's OpenID URL (which is public information)
    so that you can define the users and their roles in datasets.xml with <user> tags.

    OpenID uses a cookie on the user's computer, so the user's browser must be set to allow cookies.
    If a user is making ERDDAP requests from a computer program (not a browser), cookies are hard to work with. Sorry.

  ERDDAP doesn't support BASIC authentication because:
  • BASIC seems geared toward predefined web pages needing secure access or blanket on/off access to the whole site,
    but ERDDAP allows (restricted access) datasets to be added on-the-fly.
  • BASIC authentication doesn't offer a way for users to log out!

  Secure Data Sources - If a data set is to have restricted access to ERDDAP users,
  the data source (from where ERDDAP gets the data) should not be publicly accessible.
  So how can ERDDAP get the data for restricted access datasets? Some options are:

  • ERDDAP can serve data from local files (for example, via EDDTableFromFiles or EDDGridFromFiles).
     
  • ERDDAP can be in a DMZ and the data source (e.g., an OPeNDAP server or a database) can be behind a firewall,
    where it is accessible to ERDDAP but not to the public.
     
  • The data source can be on a public web site, but require a login to get the data.
    The one type of dataset that ERDDAP can log on to access is EDDTableFromDatabase.
    These datasets support (and should always use) user names, passwords, SSL connections, and other security measures.

    But in general, currently, ERDDAP can't deal these data sources because it has no provisions for logging on to the data source.
    This is the reason why access to EDDGridFromErddap and EDDTableFromErddap datasets can't be restricted.
    Currently, the local ERDDAP has no way to login and access the metadata information from the remote ERDDAP.
    And putting the remote ERDDAP behind your firewall and removing its dataset's
    accessibleTo restrictions doesn't solve the problem:
    since user requests for EDDXxxFromErddap data need to be redirected to the remote ERDDAP,
    the remote ERDDAP can't be behind a firewall.

  Questions? Suggestions? If you have any questions about ERDDAP's security system
  or have any questions, doubts, concerns, or suggestions about how it is set up,
  please email bob dot simons at noaa dot gov.

Things You Don't Need To Know

• Setting Up a Second ERDDAP for Testing/Development
  If you want to do this, there are two approaches:
  1. (Best) Install Tomcat and ERDDAP on a computer other than the computer that has your public ERDDAP.
     If you use your personal computer,
     • Do the installation one step at a time. Get Tomcat up and running first.
     • When Tomcat is running, the Tomcat Manager should be at http://127.0.0.1:8080/manager/html/
     • In ERDDAP's setup.xml, set baseUrl to http://127.0.0.1:8080
     • ERDDAP will be at http://127.0.0.1:8080/erddap
  2. (Second Best) Install another Tomcat on the same computer as your public ERDDAP.
     1. Set the new Tomcat's port to 8081 (see these directions, but use port 8081 instead of 8080 or 80).
     2. Install ERDDAP in the new Tomcat.
        In the new ERDDAP's setup.xml, set baseUrl to http://www.your.site.name:8081
        You can then access the new ERDDAP via http://www.your.site.name:8081/erddap .
         
• Heavy Loads / Constraints - With heavy use, a standalone ERDDAP may be constrained by various problems.
  For more information, see the list of constraints and solutions.
   
• Grids, Clusters, and Federations - Under very heavy use, a single standalone ERDDAP will run into
  one or more constraints and even the suggested solutions will be insufficient.
  For such situations, ERDDAP has features that make it easy to construct scalable grids
  (also called clusters or federations) of ERDDAPs which allow the system to handle very heavy use (e.g., for a large data center).
  For more information, see grids, clusters, and federations of ERDDAPs.
   
• Cloud Computing - Several companies are starting to offer cloud computing services (e.g., Amazon Web Services).
  Web hosting companies have offered a range of roughly similar services since the mid-1990's.
  You can use these services to set up a grid/cluster of ERDDAPs to handle very heavy use.
  For more information, see cloud computing with ERDDAP.
   
• WaitThenTryAgain Exception - In unusual circumstances, a user may get an error message like WaitThenTryAgainException:
  There was a (temporary?) problem. Wait a minute, then try again. (In a browser, click the Reload button.)
  Details: GridDataAccessor.increment: partialResults[0]="123542730" was expected to be "123532800".
  The explanation is:
  For each EDDGrid dataset, ERDDAP keeps the axis variable values in memory.
  They are used, for example, to convert requested axis values that use the "()" format into index numbers.
  For example, if the axis values are "10, 15, 20, 25", a request for (20)
  will be interpreted as a request for index #2 (0-based indices).
  When ERDDAP gets a request for data and gets the data from the source,
  it verifies that the axis values that it got from the source match the axis values in memory.
  Normally, they do.
  But sometimes the data source has changed in a significant way:
  for example, index values from the beginning of the axis variable may have
  been removed (e.g., "10, 15, 20, 25" may have become "20, 25, 30").
  If that happens, it is clear that ERDDAP's interpretation of the request (e.g., "(20)" is index #2) is now wrong.
  So ERDDAP throws an exception and marks the dataset as needing to be reloaded.
  ERDDAP will update the dataset soon (often in a few seconds, usually within a minute).
  Other, similar problems also throw the WaitThenTryAgain exception.

  Starting with ERDDAP version 1.14, it became much less likely that a user would actually see this error.
  Now, when the underlying error occurs, ERDDAP automatically internally tries to reload the dataset
  and resubmit the request to the reloaded dataset.
  Often this succeeds. When it does, the user will simply see that a given request took a little longer than usual.
  If it fails, the user should (as the message says) wait a minute, then try again.

• Responding Slowly -
  If ERDDAP is responding slowly, view your ERDDAP's Status Page from any browser: <baseUrl>/erddap/status.html .
  It has lots of useful information about memory usage, number of requests, thread usage, etc.
  Is recent memory usage really high? Are a large number of threads in use?
  And/or you can use the "Server Status" link in the Tomcat Manager
  to check on the status of all of the response threads.
  Is your network / internet connection running slowly?
  There may be other useful diagnostic information in the ERDDAP log or in <tomcat>/logs/catalina.out.
   
• Frozen, Hung, Locked Up - If ERDDAP freezes, hangs, or locks up, you need to restart Tomcat or perhaps reboot the server.
  Before you do, if the server uses Linux, from a command line,
  1) Type "ps -u <tomcatUser>" (but substitute the name of the user that ran Tomcat).
  2) In the ps output, find the processID for the "java" process.
  3) Type "kill -3 <processID>" to tell Java (which is running Tomcat) to do a thread dump to catalina.out.
  4) After you reboot, you can diagnose the problem by finding the thread dump information
     (and any other useful information above it) in <tomcat>/logs/catalina.out and
     also by reading relevant parts of the ERDDAP log archive.
  5) Please email that information to bob dot simons at noaa dot gov so we can see what went wrong.
   

Programmer's Guide

• Source Code - The source code for ERDDAP is in the erddap.war file.
  ERDDAP has a very liberal, open-source license, so you can use the code for any purpose.
   
• Use the Code for Other Projects
  While you are welcome to use parts of the ERDDAP code for other projects, be warned that the code can and will change.
  We don't promise to support other uses of our code.
  Our main goal is to make a web application that people can download and use, as is, to distribute data.

  For many situations where you might be tempted to use parts of ERDDAP in your project,
  we think you will find it much easier to install and use ERDDAP as is,
  and then write other services which use ERDDAP's services.
  You can set up your own ERDDAP installation crudely in an hour or two.
  You can set up your own ERDDAP installation in a polished way in a few days
  (depending on the number and complexity of your datasets).
  But hacking out parts of ERDDAP for your own project is likely to take weeks (and months to catch subtleties).
  We (obviously) think there are many benefits to using ERDDAP as is and making your ERDDAP installation publicly accessible.
  However, in some circumstances, you might not want to make your ERDDAP installation publicly accessible.
  Then, your service can access and use your private ERDDAP and your clients needn't know about ERDDAP.

  Half Way - Or, there is another approach which you may find useful
  which is half way between delving into ERDDAP's code and using ERDDAP as a stand-alone web service:
  In the EDD class, there is a static method which lets you make an instance of a dataset (based on the specification in datasets.xml): oneFromDatasetXml(String tDatasetID)
  It returns an instance of an EDDTable or EDDGrid dataset.
  Given that instance, you can call
  makeNewFileForDapQuery(String userDapQuery, String dir, String fileName, String fileTypeName)
  to tell the instance to make a data file, of a specific fileType, with the results from a user query.
  Thus, this is a simple way to use ERDDAP's methods to request data and get a file in response,
  just as a client would use the ERDDAP web application.
  But this approach works within your Java program and bypasses the need for an application server like Tomcat.
  We use this approach for many of the unit tests of EDDTable and EDDGrid subclasses,
  so you can see examples of this in the source code for all of those classes.

• Development Environment
  • Directory Structure
    • If you are installing ERDDAP in a Tomcat (whether or not you will actually use it that way),
      follow the instruction above.
    • If you aren't installing ERDDAP in a Tomcat, you still need to make a Tomcat-like directory structure,
      so that ERDDAP can find the setup files in [tomcat]/content/erddap .
      • Make a directory somewhere called "tomcat" (it can be something else, but this is easier to explain).
      • As indicated above, unzip erddapContent.zip into <tomcat>, creating <tomcat>/content/erddap .
        Follow the instructions above to modify setup.xml and datasets.xml.
      • Make a [tomcat]/webapps/erddap directory.
      • .war files are just .zip files that follow a few additional conventions.
        So you can use an unzip program to unzip erddap.war into [tomcat]/webapps/erddap .
        That has all of ERDDAP's .java classes and many other files.
  • Our development environment is just a programmer's editor (EditPlus -- not an endorsement since that isn't allowed, just a statement of fact).
  • We use a batch file which deletes all of the .class files in the source tree.
  • We currently use javac 1.5 to compile gov.noaa.pfel.coastwatch.TestAll and java 1.5 to run it.
    • When we run javac or java, the current directory is [tomcat]/webapps/erddap/WEB-INF .
      (Actually, for us, for historical reasons, it is "/cwexperimental/", instead of "/erddap/".)
    • Our javac and java classpath (including some unnecessary items) is currently
      ./classes;../../../lib/servlet-api.jar;lib/activation.jar;lib/axis.jar;lib/commons-discovery-0.2.jar;lib/itext-1.3.1.jar;lib/janino.jar;lib/jaxrpc.jar;lib/joda-time-1.4.jar;lib/joid.jar;lib/mail.jar;lib/netcdfAll-latest.jar;lib/postgresql-8.3-604.jdbc3.jar;lib/saaj.jar;lib/slf4j-jdk14.jar;lib/tsik.jar;lib/wsdl4j-1.5.1.jar
    • So your javac command line will be something like
      javac -cp (classpath above) classes/gov/noaa/pfel/coastwatch/TestAll.java
    • And your java command line will be something like
      java -cp (classpath above) -Xmx1500M classes/gov/noaa/pfel/coastwatch/TestAll
      Optional: you can add "-verbose:gc". It tells Java to print garbage collection statistics.
    • If TestAll compiles, everything ERDDAP needs has been compiled.
      Lots of classes are compiled that aren't needed for ERDDAP.
      If compiling TestAll succeeds but doesn't compile some class, that class isn't needed.
      (There are some unfinished/unused classes.)
  • We use some 3rd party source code instead of .jar files (notably for SSHTools and Dods)
    and have modified them slightly to avoid problems compiling with Java 1.5.
    We have often made other slight modifications (notably to Dods) for other reasons.
  • Most classes have test methods. We run lots of tests via TestAll.
    Unfortunately, many of the tests are specific to our set up. (Sorry.)
     
• Important Classes - If you want to look at the source code and try to figure out how ERDDAP works, please do.
  • The code has JavaDoc comments, but the JavaDocs haven't been generated. Feel free to generate them.
  • The most important classes (including the ones mentioned below) are within gov/noaa/pfel/erddap.
  • The Erddap class has the highest level methods. It extends HttpServlet.
  • Erddap passes requests to instances of subclasses of EDDGrid or EDDTable, which represent individual datasets.
  • EDDGrid and EDDTable subclasses parse the request, get data from subclass-specific methods, then format the data for the response.
  • EDDGrid subclasses push data into GridDataAccessor (the internal data container for gridded data).
  • EDDTable subclasses push data into TableWriter subclasses, which write data to a specific file type on-the-fly.
     
• Code Contributions - If you have written some code which would be a nice addition to ERDDAP,
  please contact bob dot simons at noaa dot gov. We'll work out the details.
  The two situations that are most likely are:
  1. You want to write another subclass of EDDGrid or EDDTable to handle another data source type.
     If so, we recommend that you find the closest existing subclass and use that code as a starting point.
  2. You want to write another saveAsFileType method.
     If so, we recommend that you find the closest existing saveAsFileType method in EDDGrid or EDDTable and use that code as a starting point.
  Both of these situations have the advantage that the code you write is self-contained.
  You won't need to know all the details of ERDDAP's internals.
  And it will be easy for us to incorporate your code in ERDDAP.
  Note that if you do submit code, the license will need to be as liberal as the ERDDAP license
  (although you can hold the copyright to your code and we'll list your contribution in the credits).
  Yes, we are considering using SourceForge or some similar system, but haven't set it up yet.
   

List of Changes

• Changes in version 1.22 (released 2009-07-05)
  • The SlideSorter bug introduced in 1.20 is fixed.
  • The OBIS bug introduced in 1.20 is fixed.
  • The references to Jason datasets on the images/gadgets/GoogleGadgets page were removed.
     
• Changes in version 1.20 (released 2009-07-02)
  • ERDDAP administrators, please add this to your setup.xml file:

    <!-- If you want to restrict access to some datasets, you need to specify
    the method used for logging on (authentication).
    See the info at http://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security
    Currently, the options are: "" (logins not supported, the default), "custom", "openid".
    Note that openid login doesn't work when testing with localhost (https://127.0.0.1:8443).
    -->
    <authentication></authentication>
    
    <!-- This specifies how you have stored passwords in the roles tags in datasets.xml.
    If you aren't storing any passwords this is irrelevant.
    The options (in order of increasing security) are: 
    "plaintext", "MD5", or "UEPMD5" (MD5(UserName:ERDDAP:Password), the default).
    You should only use "plaintext" or "MD5" if you need to match values 
    stored that way in an external password database.
    See the info at http://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security
    -->
    <passwordEncoding>UEPMD5</passwordEncoding>
    
    <!-- This determines whether datasets that the user doesn't currently have access to 
    (because he isn't logged in or because his roles don't allow access)
    should be shown on lists of data sets 
    (e.g., from full text search, categorize, view all datasets, ...).
    The options are: "true", or "false" (the default).
    If false, no information about the dataset (even its existence) is shown 
      to users who don't have access to it.
    If true, some information about the dataset (title, summary, etc) is shown
      to users who don't have access to it.  
      If the user clicks on a link to a dataset he doesn't have access to,
      he will get an error message and be prompted to log in.
    -->
    <listPrivateDatasets>false</listPrivateDatasets>
    
    <!-- If the number of requests between two runs of LoadDatasets exceeds 
    unusualActivity, an email is sent to emailEverythingTo.
    The default is 10000.
    -->
    <unusualActivity>10000</unusualActivity>
    

  • New dataset types EDDGridCopy and EDDTableCopy make and maintain a local copy
    of another EDDGrid or EDDTable dataset's data and serve data from the local copy.
    These are very easy to use and very effective
    solutions to some of the biggest problems with serving data from remote data sources:
    • Accessing data from a remote data source can be slow (for a variety of reasons).
    • The remote dataset is sometimes unavailable (again, for a variety of reasons).
    • Relying on one source for the data doesn't scale well (e.g., when many users and many ERDDAPs utilize it).
    Plus, the local copy is a backup of the original, which is useful in case something happens to the original.

    There is nothing new about making a local copy of a dataset. What is new here is that these classes
    make it *easy* to create and *maintain* a local copy of data from a *variety* of types
    of remote data sources and *add metadata* while copying the data.

    These dataset types are part of a complete set of features that simplify the creation of
    grids/clusters/federations of ERDDAPs to handle very heavy loads (e.g., in a data center).

  • New dataset type EDDTableFromDatabase gets data from a local or remote database table.
  • ERDDAP now has a security system that supports authentication (letting users log in)
    and authorization (granting them access to certain private datasets).
  • There are two, new, command-line tools to help ERDDAP administrators generate the XML for a new dataset in datasets.xml:
    • GenerateDatasetsXml can generate a rough draft of the dataset XML for almost any type of datasets.
    • DasDds helps you repeatedly test and refine the XML for a dataset.
    ERDDAP's GenerateDatasetsXml web pages have been removed. For security reasons, they only supported
    a few dataset types. The new command line tools are a better solution.
  • The new status page lets anyone (but especially administrators) view the status of an ERDDAP
    from any browser by going to <baseUrl>/erddap/status.html .
  • Tabledap now supports filters:
    • &distinct() removes duplicate rows from the response table,
    • &orderBy(...) lets you specify how the response table should be sorted,
    • &orderByMax(...) lets you specify how the response table should be sorted and removes all rows
      except for the rows with the maximum values in the last specified column.
      This can be used, for example, to get the last available data for each station.
  • Tabular datasets can now include additional dateTime variables which aren't named "time".
    These variables are recognized by their "units" metadata, which must contain " since " (for numeric dateTimes)
    or "yy" or "YY" for formatted String dateTimes.
    But please still use the destinationName "time" for the main dateTime variable.
  • ERDDAP now generates a sitemap.xml file, which tells search engines that your ERDDAP
    only needs to be crawled every month.
  • ERDDAP's error messages are now much shorter and geared to clients (not programmers).
    Thanks to Greg Williams.
  • <requestBlacklist> now also supports IP addresses where the last number has been replaced by *.
  • Requests for .json and .geoJson files may now include an optional jsonp request by adding "&.jsonp=functionName" to the end of the query.
    Basically, this just tells ERDDAP to add "functionName(" to the beginning of the response and ")" to the end of the response.
    If originally there was no query, leave off the "&" in your query.
    Thanks to Greg Williams.
  • Lots of new statistics were added to the Daily Report.
  • On web pages with lists of datasets, institution and id are now at the far right.
    This moves subscription and other more useful columns into view on narrow computer screens.
  • On all web pages, the page's title (based on the <title> in the <startHeadHtml>
    that you define in setup.xml) is modified to include a better description of the web page
    (for example, by including the current dataset's title and institution).
  • Xmx information is now included with the memory information printed in log.txt, the Daily Report,
    and on status.html. Thanks to Ellyn Montgomery.
  • Erddap has additional, general-purpose protection against all errors (e.g., OutOfMemoryError).
    Thanks to Charles Carleton.
  • Improvements to error handling if the response has already been committed.
  • Change: EDDTableFromFiles and EDDGridFromFiles now just allow <metadataFrom> "first" or "last".
    "penultimate" is no longer supported.
    And first and last are now based on the files' lastModifiedTime.
  • Bug fix: in EDDTableFromSOS, invalid info for one station threw an exception
    and caused the whole dataset to be rejected. Now, those stations are just ignored
    (and the error message is logged to log.txt). Thanks to Rick Blair.
     
• Changes in version 1.18 (released 2009-04-08)
  • Bug fix: Starting in 1.14, the EDDTable Data Access Form and Make A Graph web page didn't properly deal with quoted constraints.
  • Bug fix: Starting in 1.14, EDDTableFromDapSequence didn't handle time constraints correctly if the source time units weren't "seconds since 1970-01-01T00:00:00".
     
• Changes in version 1.16 (released 2009-03-26)
  • ERDDAP administrators:
    • This is an important release because it fixes a bug that left an ERDDAP thread running
      if you used Tomcat Manager to Stop/Start or Reload ERDDAP.
      So when you install 1.16, don't just use Tomcat manager to undeploy the old ERDDAP and deploy the new ERDDAP.
      Instead: undeploy the old ERDDAP, restart Tomcat (or the server), then deploy the new ERDDAP.
      It's always a good idea to do that when installing a new version.
    • Please add <requestBlacklist></requestBlacklist> to your datasets.xml.
      This can be used to specify a list of client IP addresses to be blocked
      (e.g., to fend off a Denial of Service attack or an overly zealous web robot).
  • There is now a <bigParentDirectory>/logs directory to hold the ERDDAP log files.
    When you start ERDDAP, it makes an archive copy of the log.txt and log.txt.previous files with a time stamp.
    If there was trouble before the restart, it may be useful to analyze these files.
  • ERD's ERDDAP now has the subscription system turned on.
  • ERDDAP once again allows (but still doesn't recommend) the "%26" encoding of "&" in request URLs
    (see the related v1.14 change).
  • Several new additions to the Tally section of the Daily Report.
  • Small bug fixes in generateDatasetsXml.
  • A few small bug fixes.
     
• Changes in version 1.14 (released 2009-03-17)
  • Changes for users:
    • In grid data requests, ERDDAP now supports: last-n where n is an integer number of indices and
      (last-d) where d is a numeric value (for time, it is in seconds).
    • In tabular data requests, String constraints now require double quotes around the value, for example, &id="NDBC40121"
      This is required by the DAP protocol.
    • In tabular data requests, ERDDAP now requires that all constraints be properly percent encoded.
      Browsers do this automatically, so this mostly affects computer programs/scripts that are accessing ERDDAP.
    • Previously, the embed a graph web page and the ERDDAP Google Gadget web page said to replace the "&" in the image's URL with "%26".
      From now on, you should replace the "&" in the image's URL with "&amp;".
      So you need to replace any "%26" in existing web pages and Google Gadgets with "&amp;". (Sorry)
  • ERDDAP administrators, please:
    • Add the following to your setup.xml file (and change the flagKeyKey value):

      <!-- ERDDAP has a service that lets remote users set a flag
      to notify ERDDAP to try to reload a dataset.
      These requests use a key which is generated based
      on baseUrl/warName, a datasetID, and flagKeyKey.
      *** Change this once, to any text (a favorite quote? random text? It doesn't matter).
      Normally, you won't ever change this again.
      But if you you think someone is abusing the flag system,
      change this text again, restart ERDDAP, and send
      all of the users of the flag system the relevant new flagKeys
      (see the list in the Daily Report). -->
      <flagKeyKey>A stitch in time saves nine. YOU SHOULD CHANGE THIS!!!</flagKeyKey>
      
      <!-- ERDDAP has an email/URL subscription system which sends a user an email
      or pings a url whenever a dataset of interest changes.
      (This is different from the RSS system, which is always active.)
      The system relies on the server being able to send out 
      emails to people to validate their subscription requests.
      The emails appear to come from the emailFromAddress below.
      So if your server can't send out emails, don't make this system active.
      You may choose (for whatever reason) to make this system active or not, 
      so valid values below are "true" (the default) and "false".
      Note that if you change this and restart ERDDAP, the list of subscriptions 
      (in [bigParentDirectory]/subscriptionsV1.txt) isn't affected.
      See also the subscriptionEmailBlacklist in datasets.xml.
      -->
      <subscriptionSystemActive>true</subscriptionSystemActive>  

    • On the line after <emailUserName> in in setup.xml, add
      <emailPassword>myPassword</emailPassword> <!-- optional; if absent, emails can't be sent to non-local addresses >
      and enter your real password.
    • You can change <wmsSampleBBox> in setup.xml to include longitude values up to 360, e.g.,

      <!-- The bounding box values are minLongitude,minLatitude,maxLongitude,maxLatitude.
         Longitude values within -180 to 180, or 0 to 360, are now okay. -->
      <wmsSampleBBox>0,-75,360,75</wmsSampleBBox>  

    • In your datasets.xml file, rename dataset type EDDTableFromNc4DFiles to EDDTableFromNcFiles (which now supports files with any number of dimensions).
      If you had an EDDTableFromNc4DFiles dataset:
      1) You MUST change to type="EDDTableFromNcFiles" in your datasets.XML file.
      2) You MUST add a <nDimensions>4</nDimensions> tag to the dataset's XML.
      3) You may add the new <sortFilesBySourceNames> tag to specify the internal order for the files, which determines the overall order of the data returned.
      For details, see EDDTableFromFiles.
    • In the past, for EDDTableFromDapSequence, for OPeNDAP DRDS servers, in datasets.xml, we used
      <sourceCanConstrainStringsRegex>~=</sourceCanConstrainStringRegex>.
      But we now see that the DRDS regex support is more limited than ERDDAP's, so we recommend
      <sourceCanConstrainStringsRegex></sourceCanConstrainStringRegex>
      so that regex constraints are not passed to the source, but are instead handled by ERDDAP.
    • Revamped handling of sourceCanConstrain... in datasets.xml by EDDTableFromDapSequence and (internally) all EDDTable dataset types.
      The new system is simpler and better reflects the variability of different data sources.
      You may need to modify the XML for your datasets in datasets.xml.
  • There are several new features which are useful by themselves,
    but when combined, also facilitate the creation of grids/clusters/federations of ERDDAPs.
    • New dataset types:
      • EDDGridFromErddap and EDDTableFromErddap
        which let one ERDDAP include a dataset from another ERDDAP in a very simple and very efficient way.
      • EDDGridFromFiles (and its subclass, EDDGridFromNcFiles which can read NetCDF .nc, GRIB .grb, and HDF .hdf files).
      • EDDTableFromNcFiles which can read NetCDF .nc which have a table-like structure.
    • RunLoadDatasets and LoadDatasets were revamped so that ERDDAP is very responsive to reloading
      datasets based on files in the flag directory (often <5 seconds if main loadDatasets is currently done).
    • New service to allow a URL to create a flag file for a given dataset, e.g.,
      http://coastwatch.pfeg.noaa.gov/erddap/setDatasetFlag.txt?datasetID=rPmelTao&flagKey=123456789
      creates a flag file in the flag directory for rPmelTao (although the flagKey here is wrong).
    • New subscription service so that any client can specify an action
      which will be done when a specific dataset is created (when ERDDAP is restarted) and
      whenever the dataset changes in any way.
      This system can be disabled via <subscriptionSystemActive> in setup.xml.
      The ERDDAP Daily Report now lists all of the subscriptions and includes the URL needed to cancel each one,
      in case you feel the system is being abused.
      In datasets.xml, there is a new, optional <subscriptionsEmailBlacklist> tag so that administrators can specify a comma-separated list of email address which are immediately blacklisted from the subscription system.
    • New <onChange> attribute in datasets.xml lets the ERDDAP administrator specify an action
      which will be done when a specific dataset is created (when ERDDAP is restarted) and
      whenever the dataset changes in any way.
    • Improvements to full text search: storing the search string for each dataset now uses 1/2 the memory.
      The search algorithm (Boyer-Moore-like) is now 3X faster.
    • Emails from ERDDAP now always prepend the subject and content with [erddapUrl],
      so that it will be clear which ERDDAP this came from (in case you administer multiple ERDDAPs).
    • More extensive statistics gathering for the Daily Report email.
    • New log file <bigParentDirectory>/emailLogYEAR-MM-DD.txt logs all emails sent by ERDDAP each day.
      This is especially useful if your server can't actually send emails -- you can at least read them in the log.
    • ERDDAP now makes a <bigParentDirectory>/cache/(datasetID)
      directory for each dataset since there may be lots of files cached.
  • New RSS 2.01 feed for each dataset (look for the orange RSS icons on lists of datasets, Data Access Forms, and Make A Graph web pages).
  • EDDGrid .kml responses now use tiled images ("superoverlays" - dynamically generated quadtree images).
    The initial image loads into GoogleEarth much faster than before.
    The resolution of the map increases as you zoom in, up to the full resolution of the dataset.
    Recommend: users should request .kml for one time point,
    but the dataset's entire longitude,latitude range.
    Unfortunately, support for time ranges was removed (I hope it will come back).
  • ERDDAP now adds Expires and Cache-Control max-age headers to all files requested from the /images directory.
    This greatly reduces the number of static file requests sent to ERDDAP
    and thus greatly speeds up most ERDDAP page loads.
    Also, many JavaScript file references moved to the bottom of their HTML pages,
    which also speeds up many ERDDAP page loads.
    Thanks to the book "High Performance Web Sites" by Steve Souders and the ySlow addition to the FireBug plugin in FireFox.
  • ERDDAP switched from netcdf-java 2.2.22 to netcdf-java 4.0.
    Among other things, this allows EDDGridFromNcFiles to read HDF .hdf, as well as GRIB .grb and NetCDF .nc files.
  • EDDGridFromDap and EDDGridFromNcFiles now also support DArray (as well as DGrid) dataVariables.
    If a dimension doesn't have a corresponding coordinate variables,
    ERDDAP creates an axis variable with the index values (e.g., 0, 1, 2, ..., 311, 312).
    So all other aspects of EDDGrid remain the same:
    * It still serves all datasets as Grids, with an axis variable for each dimension.
    * Queries can still request values from the axis variables.
    Thanks to Charles Carleton, Thomas Im, Dorian Raymer, and others.
  • The WMS OpenLayers pages now have a default longitude,latitude range that is a little larger than dataset's range
    (not the exact range, so the context of small datasets is more obvious).
    The default range may now also be 0 to 360, which allows the full range of many datasets to be shown now.
    Thanks to Todd Spindler.
  • New sliders on some Data Access Forms and Make A Graph web pages.
    They simplify (crude) specification of the desired data and offer good visual feedback.
  • A new option for the <dataset> tags in datasets.xml: active="false".
  • References to ERD's ERDDAP changed from coastwatch.pfel (still works via proxy) to coastwatch.pfeg (preferred).
  • New support for data_min and data_max variable metadata attributes.
  • A partial solution to the WaitThenTryAgain / Partial Results Exception:
    Now, some requests that previously failed when a data source change was detected
    will succeed because ERDDAP will reload the dataset and re-request the data automatically,
    all in the context of the original request.
  • Bug fix: generateDatasetsXml was disabled in ERDDAP version 1.12.
    Thanks to Ellyn Montgomery for pointing this out.
  • Small changes to error handling.
  • Many improvements to avoid/deal with possible race conditions (i.e., possible problems arising from the multi-threaded nature of ERDDAP) which caused small, infrequent problems.
  • Now, if an error message is written on an image, the image will only stay in the cache for ~5-10 minutes (not 60).
    Thanks to Cara Wilson.
  • The standard message when there is no data is now "Your query produced no matching results.", which is shorter, more accurate, and matches OPeNDAP servers.
  • EDDGrid no longer allows tied axis values.
  • Small changes to .ver and .help requests.
  • Many small changes and bug fixes.
     
• Changes in version 1.12 (released 2008-10-31)
  • EDDTableFromSOS once again works with NDBC SOS and works with the new NOS SOS.
  • EDDTableFromBMDE now requires ERDDAP admin to specify dataVariables.
  • EDDGrid no longer requires that lat and lon be evenly spaced for .transparentPng or .kml. Thanks to Todd Spindler.
  • A few small changes.
     
• Changes in version 1.10 (released 2008-10-14)
  • New "colorBar" metadata for data variables in datasets.xml defines the default color bar settings for graphs and maps. See more information. This is important because it greatly improves the appearance of the default graphs and maps produced by Make A Graph and because the default graphs and maps now have a consistent color bar even when the client changes the requested time or geographic range. Also, this was necessary for WMS.
  • ERDDAP now serves most grid data via a WMS service. This is important because it shows that, in addition to getting data from many types of data servers, ERDDAP can distribute data via different protocols (DAP, WMS, ... more in future). See the client documentation. Or the documentation for administrators. Or try it out.
  • New support for longitude values >180 in .kml files.
  • New cdm_data_type: Other .
  • ERDDAP now supports "boolean" source dataType. See more information This will become useful for the future EDDTableFromDatabase.
  • New EDDTableFromBMDE supports DiGIR/BMDE data sources.
  • EDVGridAxis now allows descending sorted values. The pmelOscar datasets needed this.
  • ERDDAP now returns HTTP errors (e.g., "404 for resource/page not found") in more situations, instead of HTML pages with error messages.
  • Lots of changes/additions to the ERDDAP documentation.
  • Lots of small changes.
  • Some bug fixes.
  • Things ERDDAP administrators should do to upgrade to this version:
    • In datasets.xml, for any EDDTableFromSOS datasets, change "observedProperty" metadata to "sourceObservedProperty".
    • The rules for an axisVariable or dataVariable's destinationName are now stricter.
      You need to check that your variable names are valid.
      Either check them by hand, or run ERDDAP and look at the error messages in the report that is emailed to the administrator.
    • In datasets.xml, if you want a grid data variable to be accessible via WMS, you need to add colorBar metadata. At least, for example,
        <att name="colorBarMinimum" type="double">0</att>
        <att name="colorBarMaximum" type="double">32</att>
      See more information.
    • Add the following to your setup.xml (but customize with your information):

      
      <!-- drawLand specifies the default Make A Graph setting for whether the 
      landmask should be drawn "over" (the default) or "under" surface data on maps. 
      "over" is recommended for primarily oceanographic data 
      (so that grid data over land is obscured by the landmask).
      "under" is recommended for all other data.
      -->
      <drawLand>over</drawLand>  
      
      <!-- Information about the ERDDAP administrator is used for the SOS and WMS servers. 
      You MUST CHANGE these to describe your installation. 
      -->
      <adminInstitution>NOAA Environmental Research Division</adminInstitution>
      <adminIndividualName>Your Name</adminIndividualName>
      <adminPosition>Webmaster</adminPosition>
      <adminPhone>your-phone-number</adminPhone>
      <adminAddress>1352 Lighthouse Ave.</adminAddress>
      <adminCity>Pacific Grove</adminCity>
      <adminStateOrProvince>CA</adminStateOrProvince>
      <adminPostalCode>93950</adminPostalCode>
      <adminCountry>USA</adminCountry>
      <adminEmail>yourName@yourSite</adminEmail>
      
      <!-- Information about the ERDDAP administrator is used for ERDDAP's SOS server. 
      You MUST CHANGE these to describe your installation. 
      -->
      <sosTitle>NOAA Environmental Research Division SOS</sosTitle>
      <sosAbstract>NOAA Environmental Research Division's ERDDAP makes data from multiple sources available via the SOS protocol.</sosAbstract>
      <sosKeywords>Weather, Ocean Currents, Temperature, Salinity</sosKeywords>
      <sosAccessConstraints>NONE</sosAccessConstraints>
      <sosFees>NONE</sosFees>
      
      <!-- Information about the ERDDAP administrator is used for ERDDAP's WMS server. 
      You MUST CHANGE these to describe your installation. 
      -->
      <wmsTitle>NOAA Environmental Research Division WMS</wmsTitle>
      <wmsAbstract>NOAA Environmental Research Division's ERDDAP makes data from multiple sources available via the WMS protocol.</wmsAbstract>
      <wmsKeywords>Weather, Ocean Currents, Temperature, Salinity</wmsKeywords> 
      <wmsAccessConstraints>NONE</wmsAccessConstraints>
      <wmsFees>NONE</wmsFees>
      <!-- For the wms examples, pick one of your grid datasets that has longitude and latitude axes.
      The sample variable must be a variable in the sample grid dataset.
      The bounding box values are minx,miny,maxx,maxy.
      -->
      <wmsSampleDatasetID>erdBAssta5day</wmsSampleDatasetID>
      <wmsSampleVariable>sst</wmsSampleVariable>
      <wmsSampleBBox>0,-75,180,75</wmsSampleBBox>
      
      

• Changes in version 1.08 (released 2008-07-13)
  • A new web service in ERDDAP, generateDatasetsXml, assists ERDDAP administrators by creating a rough draft of the XML needed to describe a dataset in datasets.xml
  • Some changes/bug fixes related to allowing griddap to be seen by netcdf-java as an opendap server, including: global metadata is now labeled "NC_GLOBAL" (instead of "GLOBAL").
  • The EDDGrid and EDDTable Data Access Forms now utilize query information in the URL. So, for example, if a user goes from a Make A Graph form to a Data Access Form, the constraints are now properly transferred.
  • tabledap's Make A Graph now allows constraints on String variables.
  • EDDTable's Make A Graph now allows NaN constraints. Thanks to Steve Hankin.
  • Bug fix: EDDTable saveAsImage didn't properly recognize the .colorbar min and max values. Thanks to Steve Hankin
  • Many improvements to setupDatasetsXml. Thanks to Ellyn Montgomery.
  • Griddap requests now allow ()-style requests slightly outside of the actual axis range. This is appropriate since ()-values are rounded to the nearest actual value. Thanks to Cindy Bessey
  • I made the FloatArray and DoubleArray test of isEvenlySpaced more sophisticated. It will always be imperfect (because the test would need to be customized for each dataset), but it should be better. Thanks to Ellyn Montgomery.
  • I moved setup.html and setupDatasetsXml.html erddap's /download directory and hard coded all links to them. Now, I can make changes and update the setup information immediately.
  • Many small changes. A few small bug fixes.
  • Things ERDDAP administrators should do to upgrade to this version:
    • Move <theShortDescriptionHtml> from your messages.xml to your setup.xml. It specifies the text that appears in the middle of the left side of the ERDDAP home page.
      Also, add <h1>ERDDAP</h1> (or some other headline) to the top of it.
      Or, copy <theShortDescriptionHtml> in the new setup.xml (from the new erddapContent.zip) into your setup.xml.
       
• Changes in version 1.06 (released 2008-06-20)
  • New support for IOOS DIF SOS data sources.
  • Many small changes. A few small bug fixes.
     
• Changes in version 1.04 (released 2008-06-10)
  • New Slide Sorter feature.
  • New Google Gadgets page and examples.
  • Bug fix in EDDGrid.saveAsNc for variable with scale and addOffset.
     
• Changes in version 1.02 (released 2008-05-26)
  • New EDDGridSideBySide allows for different axisVariables[0] sourceValues.
  • All of the currents and winds datasets were merged into EDDGridSideBySide datasets.
  • Images from image requests are now cached for 1 hour.
     
• Changes in version 1.00 (released 2008-05-06)
  • Make A Graph web pages and graphics commands in URLs.
  • Support for flag files to force reloading a dataset.
  • New dataset type: EDDTableFrom4DFiles (the first subclass of EDDTableFromFiles).
     

Credits

• ERDDAP is a Java Servlet program. At ERD, it runs inside of a Tomcat application server (license: Apache), with an Apache web server (license: Apache), running on a computer using the Red Hat Linux operating system (license: GPL).
• The data sets are from various sources. See the metadata (in particular the "institution") for each dataset.
• Data from OPeNDAP servers are read with Java DAP 1.1.7 (license: LGPL).
• NetCDF files (.nc) and GMT-style NetCDF files (.grd) are read and written with code in the NetCDF Java Library (license: LGPL) from Unidata.
• The NetCDF Java Library reads GRIB files via the Unidata GRIB decoder (grib-6.0.jar) (license: LGPL).
• The NetCDF Java Library reads BUFR files via the Unidata BUFR decoder (bufrTables-1.5.jar) (license: LGPL).
• The NetCDF Java Library requires a logger facade (we chose slf4j-jdk14.jar) from the Simple Logging Facade for Java project (license: MIT/X).
• The NetCDF Java Library uses code from several .jar files from the Apache project: commons-codec, commons-discovery, commons-httpclient, and commons-logging (license: Apache).
• The NetCDF Java Library uses XML processing code from JDOM (license: Apache).
• The graphs and maps are created on-the-fly with a modified version of NOAA's SGT version 3 (a Java-based Scientific Graphics Toolkit written by Donald Denbo at NOAA PMEL) (license: copyrighted open source).
• Big, HTML tooltips on ERDDAP's HTML pages are created with Walter Zorn's wz_tooltip.js (license: LGPL).
• Sliders and the drag and drop feature of the Slide Sorter are created with Walter Zorn's wz_dragdrop.js (license: LGPL).
• The .pdf files are created with iText version 1.3.1 (license: Mozilla), a free Java-PDF library by Bruno Lowagie and Paulo Soares.
• The shoreline data for the landmask is from GSHHS - A Global Self-consistent, Hierarchical, High-resolution Shoreline Database (license: public domain) from NOAA NGDC and created by Paul Wessel and Walter Smith.
• The political boundary data is from the pscoast program in GMT, which uses data from the CIA World Data Bank II (license: public domain).
• The bathymetry data used in the background of some maps is the ETOPO2v2g Global 2-Minute Gridded Elevation Data Set (license: public domain), which is distributed for free by NOAA NGDC.
• Emails are sent using code in activation.jar and mail.jar from Sun's JavaMail API (license: copyrighted open source).
• For OpenID authentication, ERDDAP uses the joid.jar and tsik.jar libraries from the joid project from Verisign (license: Apache 2.0).
• ERDDAP uses code from the CoastWatch Browser project from the NOAA CoastWatch West Coast Regional Node (license: copyrighted open source). That project was initiated and managed by Dave Foley, the Coordinator of the NOAA CoastWatch West Coast Regional Node. The CoastWatch Browser code was written by Bob Simons.
• The ERDDAP distribution includes code from the J2SSH project which is distributed by SSHTools (version 0.2.2, which is old (2003) but the most recent available) (license: Apache). It is based on j2ssh/examples/SftpConnect.java (License: LGPL) which is Copyright (C) 2002 Lee David Painter (lee@sshtools.com).
• The ERDDAP distribution includes the Apache Jakarta Commons Logging .jar file (version 1.0.4) (license: Apache).

License

ERDDAP, Copyright 2009, NOAA.

PERMISSION TO USE, COPY, MODIFY, AND DISTRIBUTE THIS SOFTWARE AND ITS DOCUMENTATION FOR ANY PURPOSE AND WITHOUT FEE IS HEREBY GRANTED, PROVIDED THAT THE ABOVE COPYRIGHT NOTICE APPEAR IN ALL COPIES, THAT BOTH THE COPYRIGHT NOTICE AND THIS PERMISSION NOTICE APPEAR IN SUPPORTING DOCUMENTATION, AND THAT REDISTRIBUTIONS OF MODIFIED FORMS OF THE SOURCE OR BINARY CODE CARRY PROMINENT NOTICES STATING THAT THE ORIGINAL CODE WAS CHANGED AND THE DATE OF THE CHANGE. THIS SOFTWARE IS PROVIDED "AS IS" WITHOUT EXPRESS OR IMPLIED WARRANTY.

Disclaimers

DISCLAIMER FOR EXTERNAL LINKS
The appearance of external links on this World Wide Web site does not constitute endorsement by the Department of Commerce/National Oceanic and Atmospheric Administration of external Web sites or the information, products or services contained therein. For other than authorized activities, the Department of Commerce/NOAA does not exercise any editorial control over the information you may find at these locations. These links are provided consistent with the stated purpose of this Department of Commerce/NOAA Web site.

DISCLAIMER OF LIABILITY
Neither the data providers, ERD, CoastWatch, NOAA, nor the United States Government, nor any of their employees or contractors, makes any warranty, express or implied, including warranties of merchantability and fitness for a particular purpose, or assumes any legal liability for the accuracy, completeness, or usefulness, of any information at this site.

Contact