System-to-System Data Collection in business surveys applied to an agricultural survey: a Proof of Concept

On past October 22nd, my former colleague Ger Snijkers presented "System-to-System Data Collection in business surveys applied to an agricultural survey: a Proof of Concept" at the "UNECE Expert Meeting on Statistical Data Collection 2022".

After leaving Statistics Netherlands, it's really nice to see my old data collection project moving forward. It's in very good hands.

Authors: Ger Snijkers, José Gómez Pérez, and Tim de Jong 

Abstract:

At the end of the 2018 edition of this workshop, I pitched the idea of automated data collection for business surveys. In 2019, I presented the results of a first exploratory study. Now, 3 years later we have a Proof of Concept.

In the 20th century, sample surveys have proven to be a cost-efficient method to produce accurate statistics, although they come with a high cost both for the National Statistical Institutes (NSIs) and businesses, who may experience high response burden. Nowadays in the information age, there are a lot of new digital data sources in smart industries, like in precision farming. In some cases, these data sources allow for data communication with other computer systems without human intervention via Application Programming Interfaces (APIs). Based on these software interfaces, we developed a system-to-system data collection methodology that reduces response burden by automating the business’s internal data retrieval process. Applied to the official Crop Yield Survey, a software prototype was developed based on this methodology.

At the workshop, we will present the IT architecture we developed, showing how data capture and processing can be automated. We will discuss the automated pre-filling of the electronic Crop Yield Survey questionnaire using an API provided by a smart farming machine manufacturer, John Deere: the MyJohnDeere API. In a first Proof of Concept, it has been applied to data from a virtual farm, showing that it works, and the farmer’s workload to complete a questionnaire can be limited to a minimum.

Our next step is to conduct a small-scale field test with a small number of farmers to study the method in practice. This field test is planned for the fall of 2022. Hopefully, we can present the first results of this field test at this Expert Meeting in October.

It is our belief that this system-to-system method can be applied to business surveys in general and in the future will replace the manual completion of business survey questionnaires including the manual retrieval and rekeying of data.

The slides can be found here: https://unece.org/sites/default/files/2022-10/DC2022_S2_Netherlands_Snijkers%20et%20al_P_2.pdf 

Exploring sensor data for agricultural statistics: The fruit is not hanging as low as we thought

This month, the first article I am a co-author of is published by "Statistical Journal of the IAOS":

Exploring sensor data for agricultural statistics: The fruit is not hanging as low as we thought

Authors: Snijkers, Ger; Punt, Tim; De Broe, Sofie; Gómez Pérez, José

Abstract: New business processes are increasingly data driven as sensors have become ubiquitous. Sensor data could be a valuable new data source for official statistics. To study this presumption Statistics Netherlands conducted a small-scale use case in the area of agricultural statistics in collaboration with an  innovative farmer. A selection of his sensor data was explored for overlap with current data demands in  surveys. The aim of the study was to obtain insights in the available agricultural data, their data structure and quality, and explore new methods of data collection for agricultural statistics. The conclusion is that  these data are valuable for replacing or pre-filling (parts of) certain agricultural surveys. However, many more challenges surfaced than we expected, to which the title of this paper refers. These challenges will be discussed in this paper.

Keywords: Business data collection, official statistics, smart industry, precision farming, agriculture, data challenges

A new Method for Automating Business Data Collection of Official Statistics through APIs

On September 15th, I will participate in The European Establishment Statistics Workshop 2021 (EESW21). During Session 4: Use of alternative data sources, I will present the work that I have been doing together with my colleagues Ger Snijkers  and Sofie De Broe: A new Method for Automating Business Data Collection of Official Statistics through APIs, System-to-System data communication applied to John Deere data.  

You can find the presentation abstract below:

A a new Method for Automating Business Data Collection of Official Statistics through APIs

José Gómez Pérez, Ger Snijkers, and Sofie De Broe

Statistics Netherlands, Methodology Department, Heerlen

Sample surveys are a primary data collection method. In the 20th century sample surveys have proven to be a cost-efficient method to produce accurate statistics, although they come with a high cost both for the National Statistical Institutes (NSIs) and businesses, who may experience high response burden. Nowadays in the information age, there are a lot of new digital data sources in smart industries, such as in precision farming. In some cases, these data sources allow for data communication with other computer systems without human intervention via Application Programming Interfaces (APIs). Based on these software interfaces, we developed a system-to-system (S2S) data collection methodology that reduces response burden by automating the business collection process. As far as we know, this method has not been used yet. Applied to the official crop yield survey, a pilot software prototype was developed based on this new methodology.

At the workshop, we will demonstrate how data capture and processing can be automated. Furthermore, the results offered by the prototype in a small-scale field test with real data from a few farmers will be presented. Data quality issues among others, will also be discussed. 

By design, this method is easily extendable. In the short term, we plan to extend it to other agricultural surveys like the crop protection survey. Our methodology can be applied to arable farming in other countries because the main machine manufacturers (for instance, John Deere, CNH Industrial and Claas) provide APIs and have international presence in markets all over the world. Other sectors different from agriculture (for instance, transportation and economic activity tracking are fields where we can find a lot of APIs) can benefit from this methodology too. It can be applied by other National Statistical Institutes because the design is based on standard technologies.

You can find additional information about the workshop here:

• https://statswiki.unece.org/display/ENBES/EESW21+Programme
• https://sites.google.com/enbes.org/eesw21/home (alternative link)

Update (02-Nov-21): paper and slides can be found here 

https://statswiki.unece.org/display/ENBES/EESW21+Programme 

A pilot with System-to-System data communication applied to John Deere data

On November 13th, I participated in the BigSurv20 conference. Ger Snijkers and I presented our project: "A pilot with System-to-System data communication applied to John Deere data". You can find our presentation video below.