File formats and data conversion

The format in which you receive the data from the survey team might be dependent on the hardware used, the availability of software, or be determined by discipline-specific standards and customs.

The format choice is likely to be related to the specific software that you intend to use for data analysis. This might follow discipline-specific standards and customs, or be linked to the software skills or preference of the research team.

For many formats, there is a danger that they will become obsolete in the future, which would make the data impossible to read and interpret. Despite the backward compatibility of many software packages to import data created in previous software versions and the interoperability between competing popular software programmes, the safest option to guarantee long-term data access is to convert data to standard or open formats.

When selecting file formats for archiving and sharing, you should consider whether to use a proprietary file format, or a more ‘open’ and interoperable one. This may depend on the requirements of your collaborators, or the repository that will archive your data. If you plan to make data publicly available, you should choose an open, platform-independent format where possible.

Proprietary file formats are those owned by a company that claims intellectual property rights for the use of the software by granting licenses. Their specifications are generally not publicly available and there is a risk that future developments and software updates could impact the usability of files created in previous versions. While the most widely-used proprietary formats will have better long-term sustainability, obsolescence or backward incompatibility may still be an issue. Well-known standard proprietary formats include Microsoft Office software products, (e.g. MS Word and MS Excel), or the popular SPSS or Stata statistical packages. Proprietary formats are often not readable in other software packages unless import facilities exist, and changes may occur during conversion. For example, Stata format data files can be imported into SPSS, but the two software packages have different specifications for missing values, file and variable metadata and other elements, which may result in information loss.

International open standards have been developed over time and are standardised across organisations to ensure a degree of interoperability between platforms. The files produced to these standards are known as open format files. Examples of these include:

• The American Standard Code for Information Interchange character encoding standard (ASCII), to which CSV (comma-separated values), TSV (tab-delimited format) and fixed-with text data files usually conform.
• Tagged Image File Format (TIFF, or TIF) image files.
• PDF/A, an ISO-standardised version of the Portable Document Format (.pdf), for use in the archiving and long-term preservation of electronic documents.

The specifications for open formats are publicly available and are widely recognised across countries and organisations. Other open standards include OpenDocument Format (ODF) and XML. These formats may still require software that is capable of reading them, such as a PDF reader, but free, open-source packages for these formats are generally widely available.

Some file formats are created specifically to save space, but result in a reduction of information and data quality. When this occurs, we can describe file formats as ‘lossy’, as opposed to ‘lossless’ (where no information is lost). Lossy formats save space by removing detailed information assumed (rightly or wrongly) to be unimportant. For example, the lossy format JPEG removes fine detail in images, whilst the lossless format TIFF keeps everything. Also, repeatedly editing and saving files in lossy format results in a greater loss of information, so this should be taken into consideration.

There may be valid reasons why the data cannot be converted to other formats. Even when commonly used proprietary formats are used, it is good practice to create an open format file in addition to ensure long-term preservation, even though this may result in a less ‘user-friendly’ format. For example, the creation of a CSV version of each SPSS or Stata format file is a good policy, although such files will not contain label metadata, which will need to be saved separately.

As we mentioned earlier, the answers to these questions might result in different file formats being suitable for different stages during the research lifecycle. Different versions should of course be documented and included in the version control strategy.