All-in-One Multimodal Computer Vision System
BINA Pro
BINA Pro is a low-cost, portable, multi-sensor, multimodal imaging system leveraging a Raspberry Pi single-board computer, developed as a prototype for agricultural research. The concept originated in 2016, with early prototypes demonstrating significant field monitoring capabilities, as evidenced by our published research papers. These initial prototypes were built in 2017, laying the foundation for the core project undertaken between 2019 and 2021, which culminated in the development of BINA Pro.
BINA Pro is designed primarily for high-throughput phenotyping and precision agriculture, integrating crop models and algorithms, Internet of Things (IoT) technology, and onboard computer vision. It combines thermal and multispectral bands (thermal-VIS-NIR) and integrates data from a microclimate unit to calculate a variety of crucial plant parameters, including crop canopy cover and crop coefficient.
The system features a user-friendly graphical user interface developed using C/C++ programming language with libraries like OpenCV and the Qt framework. BINA Pro incorporates features such as automatic distance-based image registration, target detection (e.g., plant leaf) and background removal. It can display up to three image layers simultaneously, including plant surface temperature, NDVI, CWSI, actual transpiration, etc., in real-time. BINA Pro stores raw and processed geo-tagged, timestamped images and exports metadata and model outputs in CSV format for easy analysis. The system is versatile, capable of analyzing targets ranging from individual leaves to entire canopies.
Key Features & Specifications
-
Optimized for Agriculture: Specifically designed and optimized for agricultural applications, including high-throughput phenotyping and precision agriculture.
-
Raspberry Pi Powered: Utilizes a Raspberry Pi single-board computer for efficient and cost-effective operation.
-
Multispectral Imaging: Combines thermal and spectral bands (red and near-infrared) for comprehensive plant analysis.
-
Real-time Processing: 100% on-board, real-time, and automatic image processing with a maximum frame rate of 5 frames per second.
-
Intuitive Interface: Features an on-board Graphical User Interface for easy operation and data visualization.
-
Microclimate Integration: Integrates data from a microclimate unit for accurate environmental context.
-
Multi-layered Imaging: Displays multiple image layers simultaneously, including surface temperature, multispectral imagery, NDVI, ΔT, CWSI, stomatal conductance, transpiration, and more.
-
Automatic Alignment: Enables automatic and real-time alignment of images based on distance from the target.
-
Key Plant Metrics: Calculates crucial plant parameters such as canopy cover percentage, canopy height, reference ET, crop coefficient (Kc), and crop ET (ETc).
-
Automatic White Balance: Automatically adjusts for changes in light source conditions for accurate measurements.
-
Enhanced Thermal Analysis: Calculates surface temperature of the target by automatically detecting and removing background from thermal images.
-
Data Storage & Export: Stores raw and processed images and exports metadata and a wide range of calculated plant parameters to a CSV file for easy analysis.
-
High-Resolution Imaging:
-
Thermal: 160 x 120 pixel resolution (19,200 pixels).
-
Multispectral: Up to 8 MP image resolution for detailed analysis.
-
Multimodal: Thermal and Multispectral Imaging
Accurate thermal analysis in agricultural settings often requires isolating the target plant from its surroundings (e.g., soil, water, artificial objects). To achieve this, BINA Pro integrates thermal imaging with multispectral imaging, specifically utilizing near-infrared (NIR) bands. This multimodal approach enhances target detection and background removal capabilities.
By combining thermal data with NIR information, the onboard computer vision algorithms can effectively distinguish between healthy leaves, dead plants, and non-organic objects. The inclusion of NIR data significantly improves the system's accuracy in identifying and characterizing plant health.
Furthermore, users can leverage the captured raw multispectral images or derived Normalized Difference Vegetation Index (NDVI) data for valuable insights into nutrient status and disease detection, enabling more informed crop management decisions.
Three Image Layers & Automatic Image Fusion
The user-friendly Graphical User Interface (GUI) enables the automatic overlay and real-time display of up to three image layers. These layers can include:
-
Surface Temperature
-
Canopy and Air Temperature Difference (ΔT)
-
Raw Visible Bands
-
Near-Infrared (NIR) Band
-
Multispectral Images
-
Normalized Difference Vegetation Index (NDVI) (various indices)
-
Crop Water Stress Index (CWSI)
-
Stomatal Conductance (SC)
-
Actual Transpiration (Ea)
The system offers flexibility, allowing for the easy integration of additional image layers, such as photosynthesis, to meet specific research or monitoring needs.
Automatic & Manual Image Alignment
Precise image alignment is crucial for accurate data analysis. BINA Pro achieves this through a combination of:
-
Automatic Alignment: Utilizes a calibrated mathematical equation and distance data measured by an embedded ultrasonic range finder to ensure accurate and automatic alignment of thermal and multispectral images.
-
Manual Alignment: Provides a user-friendly interface for simple and efficient manual image alignment, offering flexibility for specific research needs or challenging scenarios.
Automatic Background Removal
Accurate plant analysis requires precise isolation of the target plant from its background (e.g., soil, water). BINA Pro employs a robust background removal process that leverages advanced image segmentation techniques.
-
Current Implementation: The current system utilizes an automatic image segmentation algorithm based on NDVI values. By setting upper and lower NDVI thresholds, the software effectively separates plant leaves and canopies from the background in multispectral images.
-
Ongoing Development: We are actively developing and evaluating machine learning and deep learning models, including convolutional neural networks (CNNs), for semantic and instance segmentation. These advanced techniques aim to significantly enhance the accuracy and robustness of background removal, enabling the system to address a wider range of applications with greater precision.
.
Communication with Microclimate Unit
BINA Pro integrates critical environmental data by pulling microclimate information (air temperature, relative humidity, solar radiation, and wind speed) from a commercially available unit, such as the CS ClimaVUE50.
-
Current Implementation: Currently, communication with the microclimate unit is established via wired connection using the SDI-12 protocol.
-
Future Considerations: Depending on availability and feasibility, future versions may incorporate an Application Programming Interface (API) to enable data acquisition from a broader range of commercial weather stations, potentially replacing the in-field microclimate unit with a more flexible data source.
Calculation of Crop Coefficient
-
Canopy Cover Estimation: The GUI automatically calculates the canopy cover percentage (Cc) within the multispectral image.
-
Crop Coefficient Determination: To estimate crop coefficient (Kc), the system utilizes polynomial equations that define Kc as a function of Cc (Kc = f(Cc)).
-
These equations are available for a variety of common crops and can be readily developed for other crops as needed.
-
Refer to this article (link) for detailed information on using imaging data to determine crop coefficients.
-
-
Crop ET Calculation: Once Kc is determined, the system proceeds to calculate crop evapotranspiration (ETc) using standard meteorological equations.
Sensor Output & Data Retrieval
-
Data Storage and Export:
-
The GUI stores both raw and processed images for later analysis.
-
Processed data, including calculated plant parameters and other relevant information, is exported in a user-friendly CSV format.
-
-
Data Access:
-
FTP Server: The unit functions as an FTP server, enabling users to connect remotely using its unique FTP address to easily access and download stored images and output data.
-
Remote Desktop Access: With a stable Wi-Fi connection, users can remotely access the system's desktop and GUI from any computer using a "Remote Desktop" application, providing convenient and flexible data access and control.
-
Compatibility with Third-Party Hardware
BINA Pro currently utilizes a thermal imaging module with a 160x120 pixel resolution and a multispectral sensor capable of capturing images up to 8 MP.
-
Flexibility for Future Upgrades: While the current configuration provides sufficient resolution for many applications, the BINA Pro software is designed with flexibility in mind. With minor modifications, it can be adapted to work with other commercially available camera hardware setups, enabling users to leverage higher resolution sensors or specialized imaging modalities as their research needs evolve.
Building on a Legacy of Innovation
The BINA Pro project represents a culmination of years of research and development in crop thermal sensing. Building upon our team's extensive research efforts dating back to 2012, and drawing inspiration from the pioneering work of others in this field, BINA Pro leverages the latest advancements in imaging technology and data analysis to provide a powerful tool for agricultural research and precision farming.
To delve deeper into the history and evolution of BINA Pro, we encourage you to explore our blog articles and view these presentations:
These resources offer valuable insights into the research journey that led to the development of this innovative technology.
Applications
Monitoring of crop transpiration, and growth
High throughput phenotyping
Monitoring temperature in fruit storage facilities
Precision livestock management
Detection and prediction of pests
Fruit and tuber vegetables size measuremet
Fruit loss management (e.g. apple sunburn prevention)
Bud, blossom and fruit count
Irrigation scheduling
Handheld imaging-based leaf porometer
Non-invasive detection of disease before visual symptoms are apparent
Monitoring of greenhouse environment (root zone, heating system, irrigation system, fans, …)
Ground-truthing data collected by drone or satellite
Detection of crop water stress
Produce quality assessment and sorting
Automation and robotics
Detection of apple bitter bit disorder
Detection of disorders (e.g. leaf tip burn due to calcium deficiency at high light)
Detection of nutrient deficiency in controlled environment agriculture