Consortium for Management of Experimental Data in Structural Biology
Second imgCIF workshop (new series)
at BNL on Thursday 24 May 2007
in conjunction with the 2007 Joint NSLS and CFN Users' Meeting

Workshop Report

The pace of data collection and the volume of data collected at synchrotron beam lines is increasing. The ACA Data, Standards, and Computing Committee is spearheading an effort to improve the efficiency of the handling and storage of these data by encouraging the adoption of common data formats and standard software interfaces. The goal of this is firstly to have the data be self defining, therefore equally accessible to data-reduction and -visualization codes. The second goal, for the purposes of secure archiving, is to provide robust internal documentation of the source of the data.

The first imgCIF/CBF workshop took place at Brookhaven National Laboratory in 1997 and proposed a format combining support for an efficient binary representation of images with a fully CIF-compliant ASCII equivalent. An imgCIF/CBF dictionary and software to support the format were created, are available on the web, and are described in Volume G of the IUCr International Tables for Crystallography. Now the community should adopt a consensus standard for management of data at synchrotron beam lines, and to make it easier for users to process data taken from various beam lines. Also, as our science evolves, new concepts will be considered: possibilities include NeXus and XML.

The first workshop in the new series on "Management of Synchrotron Image Data: imgCIF File System and Beyond", was held on 22 July 2006 as part of the 2006 ACA meeting in Honolulu, Hawaii. That workshop concluded that that was "the right time for more widespread use of imgCIF ... [and that] SR sources should start writing imgCIF image files as soon as possible, employing the imgCIF dictionary already adopted by the IUCr Committee on the Maintenance of the CIF Standard (COMCIFS) and published on the web and in International Tables Volume G.] " [from the report of the workshop, see http://www.medsbio.org/meetings/ACA_2006_WK02_Report.html].

Subsequent to the Hawaii workshop intensive work was started in response to these recommendations. Both the imgCIF dictionary and the supporting software library were reviewed and, after meetings at SLS and ESRF, extended. See SLS_report.html and ESRF_report.html for more information on those meetings. The work continued in collaboration with members of the community (see the imgCIF mailing list http://www.iucr.org/iucr-top/cif/cbf/imgcif-l/.

In light of this activity a workshop on data formats for synchrotron image data was held after the NSLS/CFN meeting on 24 May 2007 at BNL in the Biology Department. Topics discussed included proposed extensions to imgCIF, the use of NeXus, progress on software and the status of imgCIF at Diamond and at SLS. The workshop was organized by Herbert J. Bernstein of Dowling College and Bob Sweet of Brookhaven National Laboratory. Because Bob Sweet could not attend, John Skinner of BNL filled in for Bob Sweet at the workshop itself.

The topics for the workshop were:

• Review of imgCIF and CBFlib
• Proposed extensions to the imgCIF dictionary
• Status of imgCIF adoption at SLS, Diamond, ...
• Future directions
• Discussion

Participants, left to right: G. Todorov, J. Skinner, C. Nielsen, J. Steinbrener, D. O'Brien, Q. Guo (standing), I. Awuah Asiamah (seated), J. Jemilawon, N. Darakev, [not visible: G. Darakev, J. Westboook], F. Bernstein, A. Howard, N. Sauter, [behind the camera: H. Bernstein]

The workshop was attended by 15 participants and began with a brief welcome and introduction by Herbert Bernstein. The particpants then introduced themselves. This was followed by talks by John Skinner of Brookhaven National Laboratory, Andy Howard of IIT, John Westbrook of Rutgers, Chris Nielsen of ADSC, Nick Sauter of Lawrence Berkeley National Laboratory, Jan Steinbrener of the State University of New York at Stony Brook, and by Herbert Bernstein of Dowling College. The remainder of the meeting was devoted to open discussion of the issues raised in the presentations and tutorials to clarify both the dictionary and the software.

The discussion was wide-ranging and constructive. The overall impression was of an acceptance of the need to be able to use imgCIF in the PX world, and to work out sensible interfaces to other approaches to synchrotron image data management (e.g. hdf5 and NeXus).

The major action items agreed to at the workshop were:

• More support for the handling of uncorrected images, including both sensitivity and spatial corrections. One important extension that will be needed is support for true bitmaps (i.e. supporting 1-bit Boolean data, 8 bits per byte, rather than the current, more inefficient 8/16/32 bit Booleans). This work will be a significant long-term effort that will be coordinated by H. Bernstein in consultation with all interested parties on the imgcif list (see http://www.iucr.org/iucr-top/cif/cbf/imgcif-l/). An important open issue is that handling of formulae-based corrections may benefit from the recent work on DDLm (see http://www.iucr.org/iucr-top/cif/ddlm/).

• Creation of a new utility to "tidy" CIFs by merging, splitting and copying imgCIF files to produce imgCIF files with data in a particular desired order (e.g. header -- data image -- crystal jpeg -- image thumbnails for efficient data processing, or header -- image thumbnails -- crystal jpeg -- raw data image -- correction file -- corrected data image for an experiment notebook). This will be pursued by C. Nielsen, A. Howard and H. Bernstein

• Interface with XML, NeXus and hdf5. A great deal of work has been done on NeXus and hdf5, but firm agreement is needed on the details of representation of the PX experiment in these tree-like frameworks. This will be a major focus of the third imgCIF workshop being organized by H. Bernstein and A. Ashton in conjunction with BSR 2007.

• Clarification of the distinctions among detector types in the context of imgCIF. The major remaining obstacle to completing the integration of imgCIF into such packages and mosflm and LABELIT is a clear and consistent approach toward dealing with the various choices of specification of the relationship between detector element pixel data arrays and the physical location of those pixels in the laboratory. Much of the time in the afternoon discussion was spent on a tutorial discussion of the imgCIF axis system and its relationship to other axis systems in current use. It was the strong feeling of those present at those workshop that pixel locations should be specified by their centers in millimeters, but it was recognized that other conventions would need to be supported, by, for example, allowing for a tag to specify the units in which a given beam-center has been stated (pixels, bins, millimeters, ...), and that cookbooks were needed to deal with pixel locations that are specified by the lower-left, upper-left, ... corners. C. Nielsen and H. Bernstein will pursue further work on this issue.

• Tags for robotics and remote access. There was considerable agreement that careful consideration of tags for robotic sample handling was needed, but that it had to be done in a way that would allow for meaningful searches and comparisons by use of controlled vocabularies. Andy Howard and John Westbrook will spearhead a small working group to propose appropriate tags to add to the imgCIF or mmCIF dictionaries. The discussion of tags for remote access began in a similar way, but John Skinner invited Annie Héroux and Howard Robinson to join the workshop for that discussion and they pointed out that the distinction between remote access and local access is disappearing, with local and remote users using exactly the same interfaces to experiments. In the end it appears that the real need is for a set of tags to identify the personnel who perform particular classes of actions in managing an experiment. Andy Howard will propose additional tags for this purpose.

• The need for more cookbooks, tutorials and validation suites. As more people who are new to imgCIF are faced with the need to work with imgCIF and to integrate related software into existing packages, it is essential to have more cookbooks and tutorials that help beamline managers and software developers to start from the beginning with no knowledge of imgCIF conventions and to build the templates and other infrastructure they need. Extensive validation suites are needed, especially ones that will help to distinguish errors that may cancel one-another and be difficult to distinguish with less precise tests. All interested developers have been invited to make use of the open source development environment for CBFlib at http://blondie.dowling.edu.

(The pre-workshop agenda and charge is at http://www.medsbio.org/meetings/BNL_May07_imgCIF_Workshop.html)

The agenda for the workshop as held was

8:30 am	Breakfast

9:00 am Welcome and introduction to the workshop    H. J. Bernstein, Dowling
                                                      [imgCIF dic,
                                                       CBFlib man]

9:10 am Participants introduce themselves:  

9:20 am "Data Management at the NSLS PXRR Beamlines"
                                                    J. Skinner, BNL
                                                      [abstract]

9:40 am "imgCIF, Robotics and Remote Access"          A. Howard, IIT
                                                      [handout]

10:00 am "Harvesting Data Collection Information for
         Protein Data Bank Depositions"             J. Westbrook, Rutgers
                                                      [handout]
        
10:20 am (how we log the information related to the
         experiment)                                C. Nielsen, ADSC
                                                      [handout]

10:40 am Break

11:00 am "Use of the CBF Library in the Context of 
         Real-Time Image Analysis"                  N. Sauter, LBNL   
                                                      [handout]

11:20 am "Theory of a unified field: standard file
        formats and subroutine libraries for
        diffraction imaging"                        J. Steinbrener, USB
                                                      [handout]
       
11:40 am "The current status of the imgCIF dictionary
        and related software -- a work in progress"
                                                    H. Bernstein, Dowling
                                                      [handout]
                                                   
12:00 pm Discussion and plans for the afternoon

12:15 pm Lunch.
         
      Note: those with a general interest in the subject
      are welcome to stay for the afternoon technical
      discussions.

1:00  pm Technical discussions

2:30  pm Break

3:00  Further discussion and conclusions

      At 4 pm, A. Howard, N. Sauter, H. Bernstein and F. Bernstein
      adjourned for continued informal discussions in a more
      comfortable setting.
            
      
      

• General issues
  • hdf5, NeXus and XML play important roles in management of image data.
  • Data rates, automation, numbers of experiments combining data from multiple sources are all increasing.
  • Modern experimental records now include many more images than the data itself, e.g. pictures of the sample, thumbnail images of the data.
  • Complete information needs to be kept with the data to be able to reconstruct the experiment.

• imgCIF progress
  • SLS is producing imgCIF data from their new Pilatus detector at very high data rates.
  • XDS is able to read imgCIF data from the new SLS Pilatus detector.
  • LABELIT is very close to being able to handle imgCIF.
  • CN is making progress in being able to produce imgCIF data from ADSC detectors for Diamond.
  • The latest mosflm is much improved in its handling of imgCIF data.
  • adxv can handle imgCIF data.
  • The CBFlib library has much improved over the past year, with better examples and documentation, new compression methods, handling of 3D data and reals, and better handling of beam centers.
  • The imgCIF dictionary has been extended over the past year for the new compression methods, handling of 3D data and reals, and better handling of beam centers.
  • Bindings for C++, Python and Fortran are being developed.

• imgCIF issues to be resolved
  • The interaction with hdf5, NeXus and XML needs to be resolved.
  • Multiple images related to an experiment. need to be handled, but data processing programs may want the actual data immediately after the header.
  • imgCIF tutorials and cookbooks are needed to help beamlines setup for imgCIF.
  • imgCIF tutorials and cookbooks are needed to help software developers set up for imgCIF.
  • Additional validation suites are needed.
  • There is a need for imgCIF utilities to merge images with multiple headers.
  • There is a need for imgCIF utilities to extract ordered subsets of information from arbitrarily ordered CIFs .
  • There is a need for dictionary definitions to record metadata about robotic sample handling.
  • There is a need for dictionary definitions to record metadata about how the experiment was controlled.
  • Some way is needed to handle imgCIF with HKL2000.

Participants

Name	Affiliation	Email address
Isaac Awuah Asiamah	Dowling College	catskill1331 at yahoo dot com
Frances C. Bernstein	Bernstein + Sons	fcb at bernstein-plus-sons dot com
Herbert J. Bernstein	Dowling College	yaya at bernstein-plus-sons dot com
Nikolay Darakev	Dowling College	darakevn at gmail dot com
Georgi Darakev	Dowling College	darakevg at gmail dot com
Quanzhong Guo	BNL	qguo at bnl dot gov
Andy Howard	IIT	howard at iit dot edu
John Jemilawon	Dowling College	damie225 at gmail dot com
Chris Nielsen	ADSC	cn at adsc-xray dot com
Daniel T. O'Brien	Dowling College	dto2 at dowling dot edu
Nicholas K. Sauter	LBL	nksauter at lbl dot gov
John Skinner	BNL	skinner at bnl dot gov
Jan F Steinbrener	USB	steinbre at xray1.physics.sunysb dot edu
Georgi Todorov	Dowling College	terahz at geodar dot com
John Westbrook	Rutgers	jwest at rcsb.rutgers dot edu

A. Howard, N. Sauter	J. Westbrook, F. Bernstein, A. Howard

J. Steinbrener, J. Lemilawon, N. Darakev, J. Westbrook, F. Bernstein, A. Howard

J. Skinner, G. Todorov, C. Nielsen, J. Steinbrener, D. O'Brien, A. Awuah Asiamah, J. Jemilawon, N. Darakev, J. Westbrook (partially visible), A. Howard	Q. Guo, J. Skinner

Travel arrangements

For the convenience of participants, forms to be used for travel reimbursement requests are here.

Acknowledgements