[IUCr Home Page] [MEDSBIO]

HDRMX

6 Dec 09:50 -- Validation of Eiger Image Metadata
Satellite to AsCA 2018/Crystal 32 on 6 - 7 December 2018


Herbert J. Bernstein
Rochester Institute of Technology
and
Ronin Institute for Independent Scholarship
Work Supported in Part by Dectris, Ltd, NIH and DOE

Our thanks to Dectris, Ltd for their sponsorship of this meeting

Validation of Eiger Image Metadata

The purpose of this session is to discuss the issue involved in validating metadata that accompanies Eiger images. Some of the metadata is or can be generated by Dectris. Some of the metadata is or can be generated by the beamline. Some of the metadata is the responsibility of the user and may not even be known until long after the data collection. Here we will focus on the process of validation. After the morning tea break we will discuss the specifics of what metadata is essential for MX images, and what is optional.

  • As noted in Toronto: "... heterogeneity and incompleteness of metadata in diffraction images from Eiger Detectors which can prevent processing of datasets at institutions other the one at which they are created."
  • Agreement is needed on what metadata is required, both for the community as a whole, and for specific experiments and specific beamlines. As noted above we will try to make some progress on that issue after the break. At the moment there are several different but related sets of metadata in use:

  • That Agreement should be expressed in machine-readable form accessible to the community as a whole as well as to specific experiments and specific beamlines. The CIF dictionaries and the NeXus base classes and application definitions are the proposed machine readable form.
  • Tools must be provided to allow for validation of NeXus/HDF5 data/metadata files against standards. There are are a variety of tools available for validation of data against NeXus application definitions and CIF dictionaries. Because of the high speeds of data collection involved here, we focus on the fastest of the NeXus/HDF5 validation programs: https://github.com/nexusformat/cnxvalidate by Mark Koennecke.
  • cnxvalidate can can rapidly validate an Eiger dataset against the NXmx application definition. For example, the command

    nxvalidate -a NXmx -l ~/definitions LysTest01_4706_data_master.h5

    against an Eiger9M lysozyme dataset at NSLS-II produced the following report:

    message="Missing required global file_name attribute" sev=error dataPath=/ dataFile=LysTest01_4706_data_master.h5 
    message="Missing required global file_time attribute" sev=error dataPath=/ dataFile=LysTest01_4706_data_master.h5 
    definition=NXmx.nxdl.xml message="Required field missing" nxdlPath=/NXentry/definition sev=error dataPath=/entry/definition dataFile=LysTest01_4706_data_master.h5 
    definition=NXmx.nxdl.xml message="Required field missing" nxdlPath=/NXentry/NXinstrument/NXdetector/depends_on sev=error dataPath=/entry/instrument/detector/depends_on dataFile=LysTest01_4706_data_master.h5 
    definition=NXmx.nxdl.xml message="Required field missing" nxdlPath=/NXentry/NXinstrument/NXdetector/data sev=error dataPath=/entry/instrument/detector/data dataFile=LysTest01_4706_data_master.h5 
    definition=NXmx.nxdl.xml message="Required group missing" nxdlPath=/NXentry/NXinstrument/NXdetector/NXdetector_module sev=error dataPath=/entry/instrument/detector dataFile=LysTest01_4706_data_master.h5 
    definition=NXmx.nxdl.xml message="Data type mismatch, expected NX_BOOLEAN, got H5T_STD_I32LE" nxdlPath=/NXentry/NXinstrument/NXdetector/pixel_mask_applied sev=error dataPath=/entry/instrument/detector/pixel_mask_applied dataFile=LysTest01_4706_data_master.h5 
    definition=NXmx.nxdl.xml message="Data type mismatch, expected NX_BOOLEAN, got H5T_STD_I32LE" nxdlPath=/NXentry/NXinstrument/NXdetector/countrate_correction_applied sev=error dataPath=/entry/instrument/detector/countrate_correction_applied dataFile=LysTest01_4706_data_master.h5 
    definition=NXmx.nxdl.xml message="Required group missing" nxdlPath=/NXentry/NXsample/NXbeam sev=error dataPath=/entry/sample dataFile=LysTest01_4706_data_master.h5 
    9 errors and 105 warnings found when validating LysTest01_4706_data_master.h5
    
  • This is very informative but a little hard to read, so we propose to add a little post-processing script which changes the output format to provide a bit more white space:
    9 errors and 105 warnings found when validating LysTest01_4706_data_master.h5
    
    message="Missing required global file_name attribute" 
    ... sev=error dataPath=/ dataFile=LysTest01_4706_data_master.h5 
    
    message="Missing required global file_time attribute" 
    ... sev=error dataPath=/ dataFile=LysTest01_4706_data_master.h5 
    
    definition=NXmx.nxdl.xml message="Required field missing" 
    ... nxdlPath=/NXentry/definition 
    ... sev=error dataPath=/entry/definition dataFile=LysTest01_4706_data_master.h5 
    
    definition=NXmx.nxdl.xml message="Required field missing" 
    ... nxdlPath=/NXentry/NXinstrument/NXdetector/depends_on 
    ... sev=error dataPath=/entry/instrument/detector/depends_on dataFile=LysTest01_4706_data_master.h5 
    
    definition=NXmx.nxdl.xml message="Required field missing" 
    ... nxdlPath=/NXentry/NXinstrument/NXdetector/data 
    ... sev=error dataPath=/entry/instrument/detector/data dataFile=LysTest01_4706_data_master.h5 
    
    definition=NXmx.nxdl.xml message="Required group missing" 
    ... nxdlPath=/NXentry/NXinstrument/NXdetector/NXdetector_module 
    ... sev=error dataPath=/entry/instrument/detector dataFile=LysTest01_4706_data_master.h5 
    
    definition=NXmx.nxdl.xml message="Data type mismatch, expected NX_BOOLEAN, got H5T_STD_I32LE" 
    ... nxdlPath=/NXentry/NXinstrument/NXdetector/pixel_mask_applied 
    ... sev=error dataPath=/entry/instrument/detector/pixel_mask_applied dataFile=LysTest01_4706_data_master.h5 
    
    definition=NXmx.nxdl.xml message="Data type mismatch, expected NX_BOOLEAN, got H5T_STD_I32LE" 
    ... nxdlPath=/NXentry/NXinstrument/NXdetector/countrate_correction_applied 
    ... sev=error dataPath=/entry/instrument/detector/countrate_correction_applied dataFile=LysTest01_4706_data_master.h5 
    
    definition=NXmx.nxdl.xml message="Required group missing" 
    ... nxdlPath=/NXentry/NXsample/NXbeam 
    ... sev=error dataPath=/entry/sample dataFile=LysTest01_4706_data_master.h5