Jianchao Yes Ci

The 3D reconstruction of plants is challenging due to their complex shape causing many occlusions. Next-Best-View (NBV) methods address this by iteratively selecting new viewpoints to maximize information gain (IG). Deep-learning-based NBV (DL-NBV) methods demonstrate higher computational efficiency over classic voxel-based NBV approaches but current methods require extensive training using ground-truth plant models, making them impractical for real-world plants. These methods, moreover, rely on offline training with pre-collected data, limiting adaptability in changing agricultural environments. This paper proposes a self-supervised learning-based NBV method (SSL-NBV) that uses a deep neural network to predict the IG for candidate viewpoints. The method allows the robot to gather its own training data during task execution by comparing new 3D sensor data to the earlier gathered data and by employing weakly-supervised learning and experience replay for efficient online learning. Comprehensive evaluations were conducted in simulation and real-world environments using cross-validation. The results showed that SSL-NBV required fewer views for plant reconstruction than non-NBV methods. It achieved IG prediction in 0.0038 seconds, making it over 800 times faster than a voxel-based NBV, and an online learning iteration in 0.099 seconds. SSL-NBV reduced training annotations by over 90% compared to a baseline DL-NBV. Furthermore, SSL-NBV could adapt to novel scenarios through online fine-tuning. Also using real plants, the results showed that the proposed method can learn to effectively plan new viewpoints for 3D plant reconstruction. Most importantly, SSL-NBV automated the entire network training and uses continuous online learning, allowing it to operate in changing agricultural environments. Both SSL-Global-NBV and SSL-Local-NBV focus on reconstructing the entire scene. This is suitable for tasks such as plant monitoring, where comprehensive scene information is required. However, many agricultural operations require information only about specific plant parts, such as fruits for harvesting or nodes for leaf removal. Exploring irrelevant regions in such cases wastes time and energy. To overcome this limitation, Chapter 5 introduces SSL-Semantic-NBV, a target-aware NBV approach that prioritizes viewpoint selection for plant parts that are relevant to task execution. Target-aware NBV is considerably more challenging due to target sparsity, as most viewpoints provide little or no useful signal for supervising network training. To address this challenge, a novel IG metric and loss function are proposed to enable effective learning from sparse information signals. To accelerate real-world deployment, functional–structural plant models are used to generate synthetic data for network pretraining. In simulation, SSL-Semantic-NBV improves reconstruction efficiency by 42%–104% compared to non-NBV methods and by 9%–18% compared to existing NBV approaches. In real-world experiments, improvements of 8%–16% and 3%–7% are achieved, respectively. The semantic-aware mechanism substantially improves both training efficiency and reconstruction efficiency, particularly for sparsely distributed plant parts. In summary, this thesis presents a series of online self-supervised learning–based implicit NBV methods that progressively address key challenges in robotic active perception for agriculture. By integrating online learning, local viewpoint planning, and target-aware perception, the proposed methods improve the adaptability of implicit NBV to novel environments, its scalability across different plant sizes, and its efficiency in perceiving task-relevant plant parts. These methods therefore enhance robotic perception systems to robustly and efficiently collect information under occlusions. By advancing robotic perception capabilities, this work can facilitate the development and deployment of robots in agricultural applications, helping to alleviate labor shortages and contribute to reliable food production.