SPIFpy application for conversion of raw binary data to single particle image format

Overview of the application

SPIFpy is a National Research Council of Canada (NRC) application that converts raw binary data generated by cloud imaging probes into single particle image format (SPIF). SPIF is a new, standardized imaging format that will help the cloud physics community more easily share data and findings.

Targeted users

• Atmospheric scientists, physicists, modellers
• Companies that develop cloud imaging probes and software
• Universities, research centres, governments and industries working with hydrometeor datasets, including radiative transfer, satellite validation with in-situ data, precipitation measurements, atmospheric charging, aircraft icing research, and other atmospheric applications

Benefits to users

• Allows the conversion from multiple proprietary binary data formats used by different manufacturers of cloud imaging probes to one single universal SPIF format
• Makes the processing and analysis of large microphysics datasets much simpler, allowing for significant time and effort savings
• Enables a significant increase in the collaborative capability of researchers from different groups in the cloud physics community
• Makes it easier to share data between groups, perform research collaboratively and standardize the comparison of observations

Description of the application

Shadow and grayscale imaging probes have been around for decades, capturing the size, shape, and concentration of hydrometeors, which are key variables in cloud microphysics research. However, due to diverse ownership of the underlying technology, IP restrictions, and lack of standards, the collected datasets have to be accessed and processed via an increasing number of software packages, implemented using a variety of incompatible standards. The benefits of standardized image formats and algorithms for simplified data analysis and for combined big-data analysis have been well recognized by the cloud physics community. As a result, a number of workshops were organized to address this need. This led to the creation of a working group tasked with producing a standard Optical Array Probe (OAP) image format and standardized processing algorithms. The result of this effort is the single particle image format (SPIF) proposal (first proposed by A. Bansemer, J. Crosier, M. Freer, G. Nott, P. Rosenberg).

Recently, the NRC has implemented codes for the conversion of raw binary data to the proposed SPIF format. The software is written in Python and can convert binary data files to a netCDF-based single particle image format. The resulting netCDF file stores the image data and any metadata or housekeeping parameters present in the original file. At the moment, the tool can convert data from the following cloud probes:

• 2-dimensional stereo (2DS, 10-1280 μm, SPEC Inc.) probe
• 2-dimensional cloud (2DC, 50-1600 μm, PMS) probe
• Precipitation imaging probe (PIP, 100-6400 μm, DMT)
• Cloud imaging probe (CIP, 25, 50 μm resolutions, mono and gray options , DMT)
• High volume precipitation spectrometer (HVPS-3, 50-19200 μm, SPEC Inc.) probe

System requirements

SPIFpy works on 64-bit Linux, OS X and Windows computers. SPIFpy is a Python based software. As such, it requires installation of a Python distribution (version 3.6 or higher) and the following packages:

• numpy
• netCDF4
• tdqm

Fees

SPIFpy is distributed as free software under the terms of the MIT Licence. Please review this licence before downloading the software. In practice, this licence imposes no restriction on using SPIFpy. However, if you want to further convey verbatim or modified versions of the code, or any work based on any SPIFpy component (including any such work operated remotely over a network), you must include acknowledgment of the original code.

How to access

The software source code can be downloaded from the NRC’s GitHub account or installed from PyPi using pip/conda.

Language

The documentation for the code is provided in English.