HDRMX Issues

The following is the consensus report from the AsCA 2018 HDRMX meeting.

1. There was unanimous agreement on the need to augment the metadata produced to facilitate the processing of data at facilities other than the one at which the data was produced. In particular, detailed descriptions of axis chains to be used to process the data are needed, both for sample goniometers and detector positioners.

   In general the requested aumentation of metadata was divided into two groups; first, metadata to be added via a templating mechanism in the Dectris software to be set-up before collection as static changes to the "master" files, and, second, metadata to be added after collection, possibly via H5copy. For simplicity we refer to the former as static and the later as dynamic.

   • Static:
     Some tags for static (i.e. Dectris template) additions are already available. imgCIF defines AXIS tags needed for specification of arbitrary and very general axis chains. NeXus defines the equivalent information in the NXtransformations base class. There was concern expressed about cluttering the templating mechanism with large numbers of tags used only in the most complex cases. To avoid such clutter the input to the template can be the path to either a CBF or a NeXus file with the appropriate axis information, along with the necessary software to automatically convert between CBF and NeXus axis conventions. One way or another all diffraction geometry and all detector geometry need to be described.

     Tags have been defined to carry metadata specifying the beamline and facilty.

     Note that the detector distance, wavelength and beam center are alredy specified and very necessary.

     For integrating detectors or other detectors that do not count single photons come into use in this performance range, detector gain will need to be specified.

     Tags are needed for the HDF5 software version, to declare the use of non-standard local format conventions, to list the files comprising a dataset, and to give the format of each particular file.

     As a partial example consider a beamline called XXX (ID1) at site SYNC with an omega axis, and pin_x, pin_y and pin_z translation axes stacked 5 millimetres apart, using hdf5_1.8.14 and NXmx 1.4. Then a portion of the necessary information presented as a CIF file might be:

     data_AMX_metadata
             loop_
             _axis.id
             _axis.type
             _axis.equipment
             _axis.depends_on
             _axis.vector[1] _axis.vector[2] _axis.vector[3]
             _axis.offset[1] _axis.offset[2] _axis.offset[3]
             source       .        source     .       0     0     1   . . .
             gravity      .        gravity    .       0    -1     0   . . .
             pin_x     translation goniometer .      -1     0     0   0 0 0
             omega     rotation    goniometer pin_x   1     0     0  -5 0 0
             pin_y     rotation    goniometer omega   0     1     0  -10 0 0
             pin_z     rotation    goniometer pin_y   0     0    -1  -15 0 0
     
             _array_intensities.gain           1.0 #counts/photon
     
             _diffrn_source.source   SYNCHROTRON
             _diffrn_source.type     'SYNC XXX (ID1)'
             _diffrn_source.pdbx_synchrotron  SYNC
             _diffrn_source.pdbx_synchrotron_beamline 'XXX (ID1)'
     
             _dataset_file_format.file_format  'hdf5_1.8.14 and NXmx 1.4'
     
             _diffrn_radiation.beam_width       7  #micrometres
             _diffrn_radiation.beam_height      5  #micrometres
             _diffrn_radiation.beam_flux        400000000000 #ph/s in the beam
     
     

   • Dynamic:
     Many tags for dynamic (non-Dectris-template) additions are also already available. For example, the monchromater, the beam_height, beam_width, beam_flux and sample sequence can all be placed by a beamline or user in a CIF or NeXus file for merging with H5copy into an existing master metadata file. The enisting imgcif and mmcif dictionaries provide possibilities, and more can be added. The attendees discussed:
     • sample provenance
     • sample physical characteristics
     • sample imagery
     • protein sequence
     • detector and sample environments, incl. temperature
     • sample delivery method
     • serial crystallography parameters (incl. pump probes)
     • spectroscopy
     • sample mount
     • detector ROI
     • beamline optics
     • source parameters, e.g. mode, current
     • collection strategy, scan type, scan mode
     • beam profile (Gaussian, tophat)
     • monochromator bandpass
     • beam divergences
     • beam collimation
     The plan is for H. Bernstein to gather simple examples and supporting software in January 2019 and expand to more complex examples through summer 2019 with an eye to presenting the results at some of the summer 2019 crystallographic meetings. The simplest of these examples, for a single-sweep single-axis continuous-rotation data collection, will be developed into a "Gold Standard" file. The Gold Standard will contain all metadata required to automatically process the data. This includes properly defined detector and sample geometry and the static HDF5 tags destined (by agreement within the community) for addition to the master file via the templating mechanism. The Gold Standard will give all beamlines a paradigm to follow. It will also be the basis of the output from the DECTRIS FileWriter2.
2. A seamless transition from HDF5 1.8 to 1.10 is needed. The current Filewriter and Stream interfaces support HDF5 1.8 in their current form and probably should continue to do so. The HDF5 format transition to 1.10 should be made with the new Eiger2 detectors and the Filewriter 2 and Stream 2 software
   1. We need a clear timeline from Dectris
   2. We need to convert eiger2cbf to support HDF5 1.10. HJB will provide such a version in January 2019
   3. We need 1.10 and 1.8 versions of dials
   4. We need 1.10 and 1.8 versions of neggia
   5. We need utilities to use H5check to switch automatically to the correct version
   6. We need albula in both 1.10 and 1.8 versions. Dectris plans to make Albula become open source
   7. We need to Talk to Andy Arvai about adxv
   8. We need to Talk to Kay Diederichs about XDS
   9. We need to examine all the detector code related to Eiger in CCP4 and PHENIX
   10. We need to talk to Wladek Minor about HKL
   11. We need to talk to Harry Powell about mosflm
   12. We need to talk to Graeme Winter about xia2
   13. We need to check on CrystFEL 1.8 data support versus 1.10 data support
   14. We need to get people primed to talk about this in the summer meetings
3. There have been reports of instances where use of corner and edge pixels has degraded quality. We need to investigate further, trying new mask files, and checking flat field corrections. See this email from Jie Nan and slides
4. As suggested by James Holton last year, there is a need to investigate beam stability versus data collection rates
5. Note that Albula will be on Python 3.
6. The question of upgrading existing DCUs to 40 Gb is still to be determined.. All Eiger 2 DCUs will be up to 100 Gb
7. There was unanimous agreement on the need for a seamless transition from HDF5 1.8 to HDF5 1.10.
8. There was discussion on the need for care in the transition to Python 3.
9. The participants expressed their appreciation to Dectris for sponsoring the meeting and to Chris Squire for local organization.