REMARO REMARO
Below, you can find links for the papers, that have been published up until now, from the ESRs inside the REMARO program!
A Collision Avoidance Method for Autonomous Underwater Vehicles Based on Long Short-Term Memories
Authors: László Antal, Martin Aubard, Erika Ábrahám, Ana Madureira, Luís Madureira, Maria Costa, José Pinto and Renato Campos
Date: March, 2023
Abstract
Over the past decades, underwater robotics has enjoyed growing popularity and relevance. While performing a mission, one crucial task for Autonomous Underwater Vehicles (AUVs) is bottom tracking, which should keep a constant distance from the seabed. Since static obstacles like walls, rocks, or shipwrecks can lie on the sea bottom, bottom tracking needs to be extended with obstacle avoidance. As AUVs face a wide range of uncertainties, implementing these essential operations is still challenging.
A simple rule-based control method has been proposed in [7] to realize obstacle avoidance. In this work, we propose an alternative AI-based control method using a Long Short-Term Memory network. We compare the performance of both methods using real-world data as well as via a simulator.
DOI: 10.1007/978-3-031-27499-2_42
Please see the link for the citation
Real-Time Automatic Wall Detection and Localization Based on Side Scan Sonar Images
Authors: Martin Aubard, Ana Madureira, Luis Madureira and José Pinto
Date: September, 2022
Abstract
Accurate identification of an uncertain underwater environment is one of the challenges of underwater robotics. Autonomous Underwater Vehicle (AUV) needs to understand its environment accurately to achieve autonomous tasks. The method proposed in this paper is a real-time automatic target recognition based on Side Scan Sonar images to detect and localize a harbor’s wall. This paper explains real-time Side Scan Sonar image generation and compares three object detection algorithms (YOLOv5, YOLOv5-TR, and YOLOX) using transfer learning. The YOLOv5-TR algorithm has the most accurate detection with 99% during training, whereas the YOLOX provides the best accuracy of 91.3% for a recorded survey detection. The YOLOX algorithm realizes the flow chart validation’s real-time detection and target localization.
DOI: 10.1109/AUV53081.2022.9965813
Please see the link for the citation
Belief-Based Fault Recovery for Marine Robotics
Authors: Jeremy Paul Coffelt, Mahya Mohammadi Kashani, Andrzej Wąsowski and Peter Kampmann
Date: August, 2022
Abstract
We propose a framework expanding the capabilities of underwater robots to autonomously recover from anomalous situations. The framework is built around a knowledge model developed in three stages. First, we create a deterministic knowledge base to describe the “health” of hardware, software, and environment components involved in a mission. Next, we describe the same components probabilistically, defining probabilities of failures, faults, and fixes. Finally, we combine the deterministic and probabilistic knowledge into a minimal ROS package designed to detect failures, isolate the underlying faults, propose fixes for the faults, and determine which is the most likely to help. We motivate the solution with a camera fault scenario and demonstrate it with a thruster failure on a real AUV and a simulated ROV.
Please see the link for the citation
A Formal Model of Metacontrol in Maude
Authors: Juliane Päßler, Esther Aguadol, Gustavo Rezende Silva, Carlos Hernández Corbato, Silvia Lizeth Tapia Tarifa, Einar Broch Johnsen
Date: August, 2022
Abstract
Nowadays smart applications appear in domains spanning from commodity household applications to advanced underwater robotics. These smart applications require adaptation to dynamic environments, changing requirements and internal system errors Metacontrol takes a systems of systems view on autonomous control systems and self-adaptation, by means of an additional layer of control that manipulates and combines the regular controllers. This paper develops a formal model of a Metacontrol architecture. We formalise this Metacontrol architecture in the context of an autonomous house heating application, enabling different controllers to be dynamically combined in order to meet user requirements to a better extent than the individual controllers in isolation. The formal model is developed in the Maude rewriting system, where we show results comparing different scenarios.
DOI: 10.1007/978-3-031-19849-6_32
Please see the link for the citation
A Deep Learning Framework for Semantic Segmentation of Underwater Environments
Authors: Amos Smith, Jeremy Coffelt and Kai Lingemann
Date: August, 2022
Abstract
Perception tasks such as object classification and segmentation are crucial to the operation of underwater robotics missions like bathymetric surveys and infrastructure inspections. Marine robots in these applications typically use a combination of laser scanner, camera, and sonar sensors to generate images and point clouds of the environment. Traditional perception approaches often struggle to overcome water turbidity, light attenuation, marine snow, and other harsh conditions of the underwater world. Deep learning-based perception techniques have proven capable of overcoming such difficulties, but are often limited by the availability of relevant training data. In this paper, we propose a framework that consists of procedural creation of randomized underwater pipeline environment scenes, the generation of corresponding point clouds with semantic labels, and the training of a 3D segmentation network using the synthetic data. The resulting segmentation network is analyzed on real underwater point cloud data and compared with a traditional baseline approach.
DOI: 10.1109/OCEANS47191.2022.9977212
Please see the link for the citation
Model-Based Testing for System-Level Safety of Autonomous Underwater Robots
Authors: Sergio Quijano, Mahsa Varshosaz
Date: June, 2022
Abstract
For the deployment of autonomous robotic systems in mission- and safety-critical underwater environments, aspects such as reasoning and planning need to be designed to operate in highly dynamic, uncertain environments while assuring a safe and reliable operation. Systems are often deployed without a prior safety assessment or developed with safety analysis as a separate engineering process. In this paper, to tackle these challenges, we propose an initial research vision and plan with the envisioned contributions towards designing an approach for system-wide modeling and Model-Based Testing to support safety assessments of autonomous underwater robots.
DOI: 10.1109/ICST53961.2022.00063
Please see the link for the citation
MIMIR-UW: A Multipurpose Dataset for Marine Inspection and Navigation in Underwater Robotics
Authors: Olaya Álvarez-Tuñón, Hemanth Kanner, Luiza Ribeiro Marnet, Huy Xuan Pham, Jonas Le Fevre, Yury Brodskiy, and Erdal Kayacan
Date: 2023
Abstract
This paper presents MIMIR-UW, a multipurpose underwater synthetic dataset for SLAM, depth estimation, and object segmentation to bridge the gap between theory and application in underwater environments. MIMIR-UW integrates three camera sensors, inertial measurements, and ground truth for robot pose, image depth, and object segmentation. The underwater robot is deployed within a pipe exploration scenario, carrying artificial lights that create uneven lighting, in addition to natural artefacts such as reflections from natural light and backscattering effects. Four environments totalling eleven tracks are provided, with various difficulties regarding light conditions or dynamic elements. Two metrics for dataset evaluation are proposed, allowing MIMIR-UW to be compared with other datasets. State-of-art methods on SLAM, segmentation and depth estimation are deployed and benchmarked on MIMIR-UW. Moreover, the dataset’s potential for sim-to-real transfer is demonstrated by leveraging the segmentation and depth estimation models trained on MIMIR-UW in a real pipeline inspection scenario. To the best of the authors’ knowledge, this is the first underwater dataset targeted for such a variety of methods. The dataset is publicly available online
DOI:10.1109/IROS55552.2023.10341436
An overview of monocular visual SLAM: reevolution from geometry-based to deep-learned pipelines
Authors: Olaya Álvarez-Tuñón, Yury Brodskiy, and Erdal Kayacan
Date: 2023
Abstract
With the rise of deep learning, there is a fundamental change in visual simultaneous localization and mapping (SLAM) algorithms toward developing different modules trained as end-to-end pipelines. However, regardless of the implementation domain, visual SLAM’s performance is subject to diverse environmental challenges, such as dynamic elements in outdoor environments, harsh imaging conditions in underwater environments, or blurriness in high-speed setups. These environmental challenges need to be identified to study the real-world viability of SLAM implementations. Motivated by the aforementioned challenges, this article surveys the current state of visual SLAM algorithms according to the two main frameworks: geometry-based and learning-based SLAM. First, we introduce a general formulation of the SLAM pipeline that includes most of the implementations in the literature. Second, those implementations are classified and surveyed for geometry and learning-based SLAM. After that, environment-specific challenges are formulated to enable experimental evaluation of the resilience of different visual SLAM classes to varying imaging conditions. We address two significant issues in surveying visual SLAM, providing a consistent classification of visual SLAM pipelines and a robust evaluation of their performance under different deployment conditions. Finally, we give our take on future opportunities for visual SLAM implementations.
Marine Snow Simulation and Elimination in Video
Authors: J Coffelt, N Nowald, and P Kampmann
Date: 2023
Abstract
Marine snow is formed by the aggregation of small particles of organic debris and inorganic sediments that settle through the water column and give the appearance of snow falling through the sky. Although critical to aquatic ecosystems, marine snow can potentially compromise underwater computer vision tasks. This works begins with a survey of techniques for overcoming such issues. Since many recent techniques rely on machine learning, which in turn relies on large amounts of training data, we also survey current approaches at generating such data. After discussing opportunities in existing approaches, we present novel solutions for simulation, isolation, and removal of marine snow in both images and video. Unlike other approaches, we focus on simulating marine snow in industrial scenarios with moving cameras and photorealistic discoloration, lens distortion, and motion blur typical in imagery collected by underwater robots. To remove the marine snow, we use only synthetic data, including a new dataset that we share online, to train a convolutional autoencoder extended to consider temporal information in consecutive video frames. Experiments are then conducted on two real datasets – one collected by an autonomous underwater vehicle in a shallow lake and the other by a remotely-operated vehicle in the deep sea.
DOI:10.1109/UT49729.2023.10103393
SUAVE: An Exemplar for Self-Adaptive Underwater Vehicles
Authors: Gustavo Rezende Silva, Juliane Päßler, Jeroen Zwanepol, Elvin Alberts, S. Lizeth Tapia Tarifa, Ilias Gerostathopoulos, Einar Broch Johnsen, Carlos Hernández Corbato
Date: 2023
Abstract
Once deployed in the real world, autonomous underwater vehicles (AUVs) are out of reach for human supervision yet need to take decisions to adapt to unstable and unpredictable environments. To facilitate research on self-adaptive AUVs, this paper presents SUAVE, an exemplar for two-layered system-level adaptation of AUVs, which clearly separates the application and self-adaptation concerns. The exemplar focuses on a mission for underwater pipeline inspection by a single AUV, implemented as a ROS2-based system. This mission must be completed while simultaneously accounting for uncertainties such as thruster failures and unfavorable environmental conditions. The paper discusses how SUAVE can be used with different self-adaptation frameworks, illustrated by an experiment using the Metacontrol framework to compare AUV behavior with and without self-adaptation. The experiment shows that the use of Metacontrol to adapt the AUV during its mission improves its performance when measured by the overall time taken to complete the mission or the length of the inspected pipeline..
DOI:10.1109/SEAMS59076.2023.00031
LSTS Toolchain Framework with Deep Learning Implementation into Autonomous Underwater Vehicle
Authors: Martin Aubard, Ana Madureira, Luis Madureira, Renato Campos, Maria Costa, José Pinto, João Sousa
Date: 2023
Abstract
The development of increasingly autonomous underwater vehicles has long been a research focus in underwater robotics. Recent advances in deep learning have shown promising results, offering the potential for fully autonomous behavior in underwater vehicles. However, its implementation requires improvements to the current vehicles. This paper proposes an onboard data processing framework for Deep Learning implementation. The proposed framework aims to increase the autonomy of the vehicles by allowing them to interact with their environment in real time, enabling real-time detection, control, and navigation.
DOI:10.1109/OCEANSLimerick52467.2023.10244721
MROS: A framework for robot self-adaptation
Authors: Gustavo Rezende Silva, Darko Bozhinoski, Mario Garzon Oviedo, Mariano Ramírez Montero, Nadia Hammoudeh Garcia, Harshavardhan Deshpande, Andrzej Wasowski, Carlos Hernandez Corbato
Date: 2023
Abstract
Self-adaptation can be used in robotics to increase system robustness and reliability. This work describes the Metacontrol method for self-adaptation in robotics. Particularly, it details how the MROS (Metacontrol for ROS Systems) framework implements and packages Metacontrol, and it demonstrate how MROS can be applied in a navigation scenario where a mobile robot navigates in a factory floor
DOI:10.1109/ICSE-Companion58688.2023.00044
Towards a Cognitive Architecture for Marine Robots
Authors: J Coffelt, P Kampmann, M Beetz
Date: 2023
Abstract
This paper presents a cognitive architecture for a camera robotic assistant aimed at providing the proper camera view of the operating area in an autonomous way. The robotic system is composed of a miniature camera robot and an external robotic arm. The camera robot is introduced into the abdominal cavity and handled by the external robot through magnetic interaction. The cognitive architecture is provided with a long-term memory, which stores surgical knowledge, behaviors of the camera and learning mechanisms, and a short-term memory that recognizes the actual state of the task and triggers the corresponding camera behavior. To provide the proper camera view, each state of the task is characterized by a Focus of Attention (FOA), defined by an object, a position of the object in the image, and a zoom factor. The architecture also includes a learning mechanism to take into account particular preferences of surgeons concerning the viewpoint of the scene. The architecture proposed is validated through a set of in-vitro experiments.
DOI:10.1109/BIOROB.2014.6913866
Planning With Ontology-Enhanced States Using Problem-Dependent Rewritings
Authors: Tobias John, Patrick Koopmann
Date: 2023
Abstract
We present a framework to integrate OWL ontologies into planning specifications. The resulting planning problems consider states that correspond to OWL knowledge bases, so that implicit information deduced using OWL reasoning may influence the decisions taken by the planner. While other approaches integrate ontology languages directly into the planning specification, our approach keeps planning specification and ontology separated, and only loosely couples them through an interface. This allows the ontology to be developed and maintained by ontology experts, and the planning specification by planing experts. We developed a practical method for planning in those ontology-mediated planning specifications, which, different to other ontology based approaches to planning, supports full OWL-DL. Specifically, we implemented a problem dependent rewriting approach that translates the ontology-mediated planning specification including the planning domain and the planning problem—into a PDDL planning specification that can be processed by a standard PDDL planner.
Towards Ontology-Mediated Planning with OWL DL Ontologies
Authors: Tobias John, Patrick Koopmann
Date: 2023
Abstract
While classical planning languages make the closed-domain and closed-world assumption, there have been various approaches to extend those with DL reasoning, which is then interpreted under the usual open-world semantics. Current approaches for planning with DL ontologies integrate the DL directly into the planning language, and practical approaches have been developed based on first-order rewritings or rewritings into datalog. We present here a new approach in which the planning specification and ontology are kept separate, and are linked together using an interface. This allows planning experts to work in a familiar formalism, while existing ontologies can be easily integrated and extended by ontology experts. Our approach for planning with those ontology-mediated planning problems is optimized for cases with comparatively small domains, and supports the whole OWL DL fragment. The idea is to rewrite the ontology-mediated planning problem into a classical planning problem to be processed by existing planning tools. Different to other approaches, our rewriting is data-dependent. A first experimental evaluation of our approach shows the potential and limitations of this approach.
Towards a Semantic Digital Twin for Marine Robotics
Authors: D. Odonkor, J Coffelt, J Syrbe, M. Beetz
Date: 2023
Abstract
Digital twins (DTs) offer high-fidelity, dataenhanced virtual representations of physical entities. Introduced by NASA, DTs have since found popularity in the automotive, healthcare, energy, and, in particular, robotic sectors. In this work, we discuss the potential of semantic DTs (SemDTs) in the harsh, dynamic, and unstructured environments common to marine robots. The proposed SemDTs offer several improvements over traditional DTs including: ontology-based knowledge representations of both the robot and its environment, symbolic reasoning mechanisms relating the knowledge generated and required during underwater missions, and semantic scene graphs to interpret and explain the current state of the robot-environment system. To guide the discussion of the proposed framework, we consider a particularly promising application of the SemDT—that of an autonomous underwater vehicle performing inspection and monitoring of a subsea station.
DOI:10.13140/RG.2.2.27995.13604
Extending Neural Network Verification to a Larger Family of Piece-wise Linear Activation Functions
Authors: László Antal, Hana Masara, Erika Ábrahám
Date: 2023
Abstract
In this paper, we extend an available neural network verification technique to support a wider class of piece-wise linear activation functions. Furthermore, we extend the algorithms, which provide in their original form exact respectively over-approximative results for bounded input sets represented as star sets, to allow also unbounded input sets. We implemented our algorithms and demonstrated their effectiveness in some case studies.
Formal Modelling and Analysis of a Self-Adaptive Robotic System
Authors: Juliane Päßler, Maurice H. ter Beek, Ferruccio Damiani, S. Lizeth Tapia Tarifa, Einar Broch Johnsen
Date: 2023
Abstract
Self-adaptation is a crucial feature of autonomous systems that must cope with uncertainties in, e.g., their environment and their internal state. Self-adaptive systems are often modelled as two-layered systems with a managed subsystem handling the domain concerns and a managing subsystem implementing the adaptation logic. We consider a case study of a self-adaptive robotic system; more concretely, an autonomous underwater vehicle (AUV) used for pipeline inspection. In this paper, we model and analyse it with the feature-aware probabilistic model checker ProFeat. The functionalities of the AUV are modelled in a feature model, capturing the AUV’s variability. This allows us to model the managed subsystem of the AUV as a family of systems, where each family member corresponds to a valid feature configuration of the AUV. The managing subsystem of the AUV is modelled as a control layer capable of dynamically switching between such valid feature configurations, depending both on environmental and internal conditions. We use this model to analyse probabilistic reward and safety properties for the AUV.
Runtime Architecture and Task Plan Co-Adaptation for Autonomous Robots with Metaplan
Authors: Jeroen M. Zwanepol, Gustavo Rezende Silva, Carlos Hernández Corbato
Date: 2024
Abstract
Autonomous robots need to be able to handle uncertainties when deployed in the real world. For the robot to be able to robustly work in such an environment, it needs to be able to adapt both its architecture as well as its task plan. Architecture adaptation and task plan adaptation are mutually dependent, and therefore require the system to apply runtime architecture and task plan co-adaptation. This work presents Metaplan, which makes use of models of the robot and its environment, together with a PDDL planner to apply runtime architecture and task plan co-adaptation. Metaplan is designed to be easily reusable across different domains. Metaplan is shown to successfully perform runtime architecture and task plan co-adaptation with a self-adaptive unmanned underwater vehicle exemplar, and its reusability is demonstrated by applying it to an unmanned ground vehicle.
Modeling and Safety Analysis of Autonomous Underwater Vehicles Behaviors
Authors: Sergio Quijano, Mahsa Varshosaz, Andrzej Wąsowski
Date: 2023
Reliable Plan Selection with Quantified Risk-Sensitivity
Authors:Tobias John, Mahya Mohammadi Kashani, Jeremy Paul Coffelt, Einar Broch Johnsen, and Andrzej Wąsowski
Date: 2023
Abstract
Robots in many domains need to plan and make decisions under uncertainty; for example, autonomous underwater vehicles (AUVs) gathering data in environments inaccessible to humans, need to perform automated task planning. Planning problems are typically solved by risk-neutral optimization maximizing a single objective, such as limited time or energy consumption. A typical probabilistic planner synthesizes a plan to reach the desired goals with a maximum expected reward, given the possible initial states and actions of the world. In this work, we additionally consider risk metrics for selecting solutions to such planning problems. Consider a marine robotics mission scenario where the task is to survey pipeline segments safely based on various risk measurements.
Linear Time-Varying Parameter Estimation: Maximum A Posteriori Approach via Semidefinite Programming
Authors: Sasan Vakili, Mohammad Khosravi, Peyman Mohajerin Esfahani and Manuel Mazo Jr.
Date: 2023
Abstract
We study the problem of identifying a linear time-varying output map from measurements and linear time-varying system states, which are perturbed with Gaussian observation noise and process uncertainty, respectively. Employing a stochastic model as prior knowledge for the parameters of the unknown output map, we reconstruct their estimates from input/output pairs via a Bayesian approach to optimize the posterior probability density of the output map parameters. The resulting problem is a non-convex optimization, for which we propose a tractable linear matrix inequalities approximation to warm-start a first-order subsequent method. The efficacy of our algorithm is shown experimentally against classical Expectation Maximization and Dual Kalman Smoother approaches.
DOI: 10.1109/LCSYS.2023.3347198
Loss it right: Euclidean and Riemannian Metrics in Learning-based Visual Odometry
Authors: Olaya Alvarez-Tunon; Yury Brodskiy; Erdal Kayacan
Date: 2023
Abstract
This paper overviews different pose representations and metric functions in visual odometry (VO) networks. The performance of VO networks heavily relies on how their architecture encodes the information. The choice of pose representation and loss function significantly impacts network convergence and generalization. We investigate these factors in the VO network DeepVO by implementing loss functions based on Euler, quaternion, and chordal distance and analyzing their influence on performance. The results of this study provide insights into how loss functions affect the designing of efficient and accurate VO networks for camera motion estimation. The experiments illustrate that a distance that complies with the mathematical requirements of a metric, such as the chordal distance, provides better generalization and faster convergence.
DOI: https://doi.org/10.48550/arXiv.2401.05396
UNav-Sim: A Visually Realistic Underwater Robotics Simulator and Synthetic Data-generation Framework
Authors: Abdelhakim Amer, Olaya Álvarez-Tuñón, Halil Ibrahim Ugurlu, Jonas le Fevre Sejersen, Yury Brodskiy, Erdal Kayacan
Date: 2023
Abstract
Underwater robotic surveys can be costly due to the complex working environment and the need for various sensor modalities. While underwater simulators are essential, many existing simulators lack sufficient rendering quality, restricting their ability to transfer algorithms from simulation to real-world applications. To address this limitation, we introduce UNav-Sim, which, to the best of our knowledge, is the first simulator to incorporate the efficient, high-detail rendering of Unreal Engine 5 (UE5). UNav-Sim is open-source https://github.com/open-airlab/UNav-Sim and includes an autonomous vision-based navigation stack. By supporting standard robotics tools like ROS, UNav-Sim enables researchers to develop and test algorithms for underwater environments efficiently.
DOI: 10.1109/ICAR58858.2023.10406819
SubPipe: A Submarine Pipeline Inspection Dataset for Segmentation and Visual-inertial Localization
Authors: Olaya Álvarez-Tuñón, Luiza Ribeiro Marnet, László Antal, Martin Aubard, Maria Costa and Yury Brodskiy
Date: 2024
Abstract
This paper presents SubPipe, an underwater dataset for SLAM, object detection, and image segmentation. SubPipe has been recorded using a \gls{LAUV}, operated by OceanScan MST, and carrying a sensor suite including two cameras, a side-scan sonar, and an inertial navigation system, among other sensors. The AUV has been deployed in a pipeline inspection environment with a submarine pipe partially covered by sand. The AUV’s pose ground truth is estimated from the navigation sensors. The side-scan sonar and RGB images include object detection and segmentation annotations, respectively. State-of-the-art segmentation, object detection, and SLAM methods are benchmarked on SubPipe to demonstrate the dataset’s challenges and opportunities for leveraging computer vision algorithms. To the authors’ knowledge, this is the first annotated underwater dataset providing a real pipeline inspection scenario
https://doi.org/10.48550/arXiv.2401.17907
Mission Planning and Safety Assessment for Pipeline Inspection Using Autonomous Underwater Vehicles: A Framework based on Behavior Trees
Authors: Martin Aubard , Sergio Quijano , Olaya Álvarez-Tuñón , László Antal , Maria Costa and Yury Brodskiy
Date: 2024
Abstract
The recent advance in autonomous underwater robotics facilitates autonomous inspection tasks of offshore infrastructure. However, current inspection missions rely on predefined plans created offline, hampering the flexibility and autonomy of the inspection vehicle and the mission’s success in case of unexpected events. In this work, we address these challenges by proposing a framework encompassing the modeling and verification of mission plans through Behavior Trees (BTs). This framework leverages the modularity of BTs to model onboard reactive behaviors, thus enabling autonomous plan executions, and uses BehaVerify to verify the mission’s safety. Moreover, as a use case of this framework, we present a novel AI-enabled algorithm that aims for efficient, autonomous pipeline camera data collection. In a simulated environment, we demonstrate the framework’s application to our proposed pipeline inspection algorithm. Our framework marks a significant step forward in the field of autonomous underwater robotics, promising to enhance the safety and success of underwater missions in practical, real-world applications.
DOI:10.1109/OCEANS51537.2024.10682385
Sonar2Depth: Acoustic-Based 3D Reconstruction Using cGANs
Authors: Nael Jaber, Bilal Wehbe, Frank Kirchner
Date: 2023
Abstract
This work proposes the use of conditional Generative Adversarial Networks (cGANs) for acoustic-based 3D reconstruction. Acoustics being the most reliable sensor modality in underwater domains is accompanied with the loss of elevation angle in its images. The challenge of recovering the missing dimension in acoustic images have pushed researchers to try various methods and approaches over the past years. cGANs being an image-to-image translation method makes it possible to learn a desired style, and transforms the data from one modality to another. This was applied here as a way of transforming an acoustic image into another form which contains the elevation characteristics, such as depth images. Depth images are hard to acquire underwater, thus data was generated synthetically and used for training and testing the deep learning model. As a way of performance enhancement, real data was collected for training a Cycle-GAN network in the aim of transferring the realistic style into the synthetically generated images. Simulation experiments were conducted to evaluate the system and find out the best experimental setup, which was then used to carry out the real experiment. The system performed dense 3D reconstruction of the scanned object and proved to be applicable in real environments
DOI:10.1109/IROS55552.2023.10342251
Knowledge Distillation in YOLOX-ViT for Side-Scan Sonar Object Detection
Authors: Martin Aubard, László Antal, Ana Madureira, Erika Ábrahám
Date: 2024
Abstract
In this paper we present YOLOX-ViT, a novel object detection model, and investigate the efficacy of knowledge distillation for model size reduction without sacrificing performance. Focused on underwater robotics, our research addresses key questions about the viability of smaller models and the impact of the visual transformer layer in YOLOX. Furthermore, we introduce a new side-scan sonar image dataset, and use it to evaluate our object detector’s performance. Results show that knowledge distillation effectively reduces false positives in wall detection. Additionally, the introduced visual transformer layer significantly improves object detection accuracy in the underwater environment.
https://doi.org/10.48550/arXiv.2403.09313
Uncertainty Driven Active Learning for Image Segmentation in Underwater Inspection
Authors: Luiza Ribeiro Marnet, Yury Brodskiy, Stella Grasshof & Andrzej Wąsowski
Date: 2024
Abstract
Active learning aims to select the minimum amount of data to train a model that performs similarly to a model trained with the entire dataset. We study the potential of active learning for image segmentation in underwater infrastructure inspection tasks, where large amounts of data are typically collected. The pipeline inspection images are usually semantically repetitive but with great variations in quality. We use mutual information as the acquisition function, calculated using Monte Carlo dropout. To assess the effectiveness of the framework, DenseNet and HyperSeg are trained with the CamVid dataset using active learning. In addition, HyperSeg is trained with a pipeline inspection dataset of over 50,000 images. For the pipeline dataset, HyperSeg with active learning achieved 67.5% meanIoU using 12.5% of the data, and 61.4% with the same amount of randomly selected images. This shows that using active learning for segmentation models in underwater inspection tasks can lower the cost significantly.
https://doi.org/10.48550/arXiv.2403.09313
Bridging the Sim-to-Real GAP for Underwater Image Segmentation
Authors: Luiza Ribeiro Marnet, Stella Grasshof, Yury Brodskiy & Andrzej Wąsowski
Date: 2024
Abstract
Labeling images for every new task or data pattern a model needs to learn is a significant time bottleneck in real-world applications. Moreover, acquiring the necessary data for training the models can be challenging. Ideally, one would train the models with simulated images and adapt them for the desired real tasks using the least possible amount of data. Active learning can be used to solve this problem with minimal effort. In this work, we train SegFormer for pipeline segmentation with synthetic images from an underwater simulated environment and fine-tune the model with real underwater pipeline images recorded in a marina. The evaluation shows that selecting real data with active learning for fine-tuning the model gives better results than randomly selecting the images. As part of the work, we release the dataset recorded in the marina, MarinaPipe, which will be publicly available.
DOI:10.1109/OCEANS51537.2024.10682212
Template Decision Diagrams for Meta Control and Explainability
Authors: Clemens Dubslaff, Verena Klös, Juliane Päßler
Date: 2024
Abstract
Decision tree classifiers (DTs) provide an effective machine-learning model, well-known for its intuitive interpretability. However, they still miss opportunities well-established in software engineering that could further improve their explainability: separation of concerns, encapsulation, and reuse of behaviors. To enable these concepts, we introduce templates in decision diagrams (DDs) as an extension of multi-valued DDs. Templates allow to encapsulate and reuse common decision-making patterns. By a case study from the autonomous underwater robotics domain we illustrate the benefits of template DDs for modeling and explaining meta controllers, i.e., hierarchical control structures with underspecified entities. Further, we implement a template-generating refactoring method for DTs. Our evaluation on standard controller benchmarks shows that template DDs can improve explainability of controller DTs by reducing their sizes by more than one order of magnitude.
https://doi.org/10.1007/978-3-031-63797-1_12
Segmentation of Multibeam Bathymetry and Backscatter
Authors: Jeremy Coffelt, Amos Smith, Nicolas Conen, Peter Kampmann
Date: 2024
Abstract
Multibeam echosounders (MBES) are the tool of choice for high-precision underwater surveys, especially when water conditions render optical imagery ineffective. We present and evaluate the following approaches for MBES segmentation: (1) real-time processing of single sounding profiles using traditional machine learning techniques, (2) batch processing of “waterfall” pseudo-images using a standard U-Net model, (3) the same model adapted to 2D projections of 3D point clouds, and (4) post-mission, survey-level processing using modern networks specifically designed for sparse point clouds. Strengths and weaknesses of the methods are discussed, including data preprocessing requirements, robustness, and ease of implementation/interpretation. Evaluation is performed on real data collected by an autonomous underwater vehicle (AUV) during a deep-sea industrial pipeline inspection.
DOI: 10.1109/OCEANS51537.2024.10706267
SAVOR: Sonar-Aided Visual Odometry and Reconstruction
Authors: Jeremy Coffelt, Peter Kampmann, Bilal Wehbe
Date: 2024
Abstract
Visual odometry (VO) relies on sequential camera images to estimate robot motion. For underwater robots, this is often complicated by turbidity, light attenuation, and environments containing scarce or repetitive features. Even ideal imagery suffers from the issue of scale ambiguity common to all monocular VO implementations. To address these issues, we supplement a camera with a multibeam echosounder. This acoustic, time-of-flight sensor comes with its own challenges, including relatively slow and sparse measurements that can be further degraded by backscatter from suspended particulate matter as well as interfering sounds from nearby marine traffic. We propose a method for fusing only data from these two inspection sensors into a hybrid VO solution that does not rely on IMU, DVL, or any other positioning sensor. We demonstrate this method on real data collected by an autonomous underwater vehicle performing end-to-end pipeline inspection in the open ocean, where multiple passes through the same scene (i.e., the “loop closure” common to SLAM algorithms) is often time and cost prohibitive. We also show how this approach can be extended for the creation of dense point clouds that provide a colored reconstruction of the surveyed scene.
DOI: 10.1109/IROS58592.2024.10801846
Risk-Averse Planning and Plan Assessment for Marine Robots
Authors: Mahya Mohammadi Kashani, Tobias John, Jeremy Coffelt, Einar Broch Johnsen, Andrzej Wasowski
Date: 2024
Abstract
Autonomous Underwater Vehicles (AUVs) need to operate for days without human intervention and thus must be able to do efficient and reliable task planning. Unfortunately, efficient task planning requires deliberately abstract domain models (for scalability reasons), which in practice leads to plans that might be unreliable or under performing in practice. An optimal abstract plan may turn out suboptimal or unreliable during physical execution. To overcome this, we introduce a method that first generates a selection of diverse high-level plans and then assesses them in a low-level simulation to select the optimal and most reliable candidate. We evaluate the method using a realistic underwater robot simulation, estimating the risk metrics for different scenarios, demonstrating feasibility and effectiveness of the approach.
Modeling and Safety Analysis of Autonomous Underwater Vehicles Behaviors
Authors: Sergio Quijano, Mahsa Varshosaz, Andrzej Wąsowski
Date: 2024
Abstract
Testing underwater vehicles in operational conditions is expensive. Formal models can lower the cost of operational testing and validation for these systems, as they allow detecting problems earlier. We propose modeling the underwater vehicle’s behavior as Timed Automata, and discuss examples of property patterns which can be handled by observer Timed Automata and Timed CTL properties. The goal is to build a set of specifications which can then be used for testing and validation of controllers in underwater vehicles.
DOI:10.1109/ICSTW60967.2024.00022
Planning with OWL-DL Ontologies
Authors: Tobias John, Patrick Koopmann
Date: 2024
Abstract
We introduce ontology-mediated planning, in which planning problems are combined with an ontology. Our formalism differs from existing ones in that we focus on a strong separation of the formalisms for describing planning problems and ontologies, which are only losely coupled by an interface. Moreover, we present a black-box algorithm that supports the full expressive power of OWL DL. This goes beyond what existing approaches combining automated planning with ontologies can do, which only support limited description logics such as DL-Lite and description logics that are Horn. Our main algorithm relies on rewritings of the ontology-mediated planning specifications into PDDL, so that existing planning systems can be used to solve them. The algorithm relies on justifications, which allows for a generic approach that is independent of the expressivity of the ontology language. However, dedicated optimizations for computing justifications need to be implemented to enable an efficient rewriting procedure. We evaluated our implementation on benchmark sets from several domains. The evaluation shows that our procedure works in practice and that tailoring the reasoning procedure has significant impact on the performance.
https://doi.org/10.48550/arXiv.2408.07544
Mutation-Based Integration Testing of Knowledge Graph Applications
Authors: Tobias John, Einar Broch Johnsen, Eduard Kamburjan
Date: 2024
Abstract
With the advent of AI-driven applications, testing faces new challenges when it comes to the integration of software with AI components. We present a novel testing approach to tackle the integration of software with symbolic AI in the form of knowledge graphs (KG). As the KG is expected to change during the run- and lifetime of the software, we must ensure the robustness of the system w.r.t. changes in the KG. Starting with a singular KG, we mutate its content and test the unchanged software with the original test oracle. To address the specific challenges of KGs, we introduce two additional concepts. First, as generic mutations on single triples are too fine-grained to reliably generate a KG describing a different, consistent KG, we employ domain-specific mutation operators, that manipulate subgraphs in a domain-adherent way. Second, we need to specify those parts of the knowledge that the software relies on for correctness. We introduce the notion of a robustness mask as shapes in the graph that the mutant must conform to. We evaluate our approach on two software applications from the robotic and simulation domain that tightly integrate with their respective KG.
DOI:10.1109/ISSRE62328.2024.00052
NeRF-To-Real Tester: Neural Radiance Fields as Test Image Generators for Vision of Autonomous Systems
Authors: Laura Weihl, Bilal Wehbe, Andrzej Wąsowski
Date: 2024
Abstract
Autonomous inspection of infrastructure on land and in water is a quickly growing market, with applications including surveying constructions, monitoring plants, and tracking environmental changes in on- and off-shore wind energy farms. For Autonomous Underwater Vehicles and Unmanned Aerial Vehicles overfitting of controllers to simulation conditions fundamentally leads to poor performance in the operation environment. There is a pressing need for more diverse and realistic test data that accurately represents the challenges faced by these systems. We address the challenge of generating perception test data for autonomous systems by leveraging Neural Radiance Fields to generate realistic and diverse test images, and integrating them into a metamorphic testing framework for vision components such as vSLAM and object detection. Our tool, N2R-Tester, allows training models of custom scenes and rendering test images from perturbed positions. An experimental evaluation of N2R-Tester on eight different vision components in AUVs and UAVs demonstrates the efficacy and versatility of the approach.
https://doi.org/10.48550/arXiv.2412.16141
SeagrassFinder: Deep Learning for Eelgrass Detection and Coverage Estimation in the Wild
Authors: Jannik Elsäßer, Laura Weihl, Veronika Cheplygina, Lisbeth Tangaa Nielsen
Date: 2025
Abstract
Seagrass meadows play a crucial role in marine ecosystems, providing benefits such as carbon sequestration, water quality improvement, and habitat provision. Monitoring the distribution and abundance of seagrass is essential for environmental impact assessments and conservation efforts. However, the current manual methods of analyzing underwater video data to assess seagrass coverage are time-consuming and subjective. This work explores the use of deep learning models to automate the process of seagrass detection and coverage estimation from underwater video data. We create a new dataset of over 8,300 annotated underwater images, and subsequently evaluate several deep learning architectures, including ResNet, InceptionNetV3, DenseNet, and Vision Transformer for the task of binary classification on the presence and absence of seagrass by transfer learning. The results demonstrate that deep learning models, particularly Vision Transformers, can achieve high performance in predicting eelgrass presence, with AUROC scores exceeding 0.95 on the final test dataset. The application of underwater image enhancement further improved the models’ prediction capabilities. Furthermore, we introduce a novel approach for estimating seagrass coverage from video data, showing promising preliminary results that align with expert manual labels, and indicating potential for consistent and scalable monitoring. The proposed methodology allows for the efficient processing of large volumes of video data, enabling the acquisition of much more detailed information on seagrass distributions in comparison to current manual methods. This information is crucial for environmental impact assessments and monitoring programs, as seagrasses are important indicators of coastal ecosystem health. This project demonstrates the value that deep learning can bring to the field of marine ecology and environmental monitoring.
https://doi.org/10.48550/arXiv.2412.16141
Enhancing AUV Dynmaic Models With CFD-Derived Hydrodynamic Coefficients
Authors: Mahmoud Ibrahim, Renato Mendes, João Borges de Sousa
Date: 2025
Abstract
Efficient navigation and manoeuvrability are crucial for Autonomous Underwater Vehicles (AUVs), widely used in underwater missions. This study proposes a numerical method to enhance the dynamic model of the Light Autonomous Underwater Vehicles (LAUVs) engineered by Laboratório de Sistemas e Tecnologia Subaquática (LSTS), which rely on dynamic models derived from the REMUS 100 AUV, a well-documented benchmarking model. The original dynamic model’s reliance on analytical, empirical, and semi-empirical approaches introduces challenging assumptions, potentially compromising the accuracy of controller design and vehicle navigation. To address these issues, a comprehensive six degrees of freedom (DOF) Computational Fluid Dynamics (CFD) of the REMUS 100 was developed using ANSYS FLUENT. The study aimed to extract key hydrodynamic coefficients and derivatives for the AUV and fins like drag, lift, and added mass through steady-state and transient simulations across various degrees of freedom. Mesh sensitivity analysis identified that a medium grid with approximately 500,000 cells provided the optimal balance between accuracy and computational efficiency. Turbulence sensitivity studies confirmed that the K-kl-omega model was most effective for capturing laminar-transitional flow regimes. Results show that CFD-derived coefficients offer improved precision compared to available analytical and experimental data, highlighting the limitations of traditional methods. This enhanced accuracy supports the development of more reliable control systems, thereby improving AUV navigation and maneuverability.
DOI:10.1109/AUV61864.2024.11030781
ROSAR: An Adversarial Re-Training Framework for Robust Side-Scan Sonar Object Detection
Authors: Martin Aubard, László Antal, Ana Madureira, Luis F. Teixeira, Erika Ábrahám
Date: 2025
Abstract
This paper introduces ROSAR, a novel framework enhancing the robustness of deep learning object detection models tailored for side-scan sonar (SSS) images, generated by autonomous underwater vehicles using sonar sensors. By extending our prior work on knowledge distillation (KD), this framework integrates KD with adversarial retraining to address the dual challenges of model efficiency and robustness against SSS noises. We introduce three novel, publicly available SSS datasets, capturing different sonar setups and noise conditions. We propose and formalize two SSS safety properties and utilize them to generate adversarial datasets for retraining. Through a comparative analysis of projected gradient descent (PGD) and patch-based adversarial attacks, ROSAR demonstrates significant improvements in model robustness and detection accuracy under SSS-specific conditions, enhancing the model’s robustness by up to 1.85%. ROSAR is available at this https URL.
MarineLLM-PDDL: Generation of Planning Domains for Marine Vessels Using Past Incident Response Plans
Authors: Mahya Mohammadi Kashani, Stefan Heinrich, Andrzej Wasowski
Date: 2025
Abstract
Testing the hardware and software of marine vessels in field trials is a necessity to avoid technical and environmental catastrophes. Conducting tests with large vessels is costly. Multiple realistic domain descriptions based on past missions could increase the value of simulation tests, reducing the need for expensive field tests. In this paper, we generate scenarios from unstructured Incident Response Plan (IRP) documents using Large Language Models (LLMs), converting them to standard structured planning programs. The two synthesized marine test-domain datasets contain approximately 90% parsable, 75% solvable, and 57% correct planning programs.
https://doi.org/10.1007/978-3-031-89471-8_47
A Configurable Software Model of a Self-Adaptive Robotic System
Authors: Juliane Päßler, Maurice H. ter Beek, Ferruccio Damiani, S. Lizeth Tapia Tarifa, Einar Broch Johnsen
Date: 2025
Abstract
Self-adaptation, meant to increase reliability, is a crucial feature of cyber-physical systems operating in uncertain physical environments. Ensuring safety properties of self-adaptive systems is of utter importance, especially when operating in remote environments where communication with a human operator is limited, like under water or in space. This paper presents a software model that allows the analysis of one such self-adaptive system, a configurable underwater robot used for pipeline inspection, by means of the probabilistic model checker ProFeat. Furthermore, it shows that the configurable software model is easily extensible to further, possibly more complex use cases and analyses.
https://doi.org/10.1016/j.scico.2024.103221
Feature-Oriented Modelling and Analysis of a Self-Adaptive Robotic System
Authors: Juliane Päßler, Maurice H. ter Beek, Ferruccio Damiani, Clemens Dubslaff, S. Lizeth Tapia Tarifa, Einar Broch Johnsen
Date: 2025
Abstract
Improved autonomy in robotic systems is needed for innovation in, e.g., the marine sector. Autonomous robots that are let loose in hazardous environments, such as underwater, need to handle uncertainties that stem from both their environment and internal state. While self-adaptation is crucial to cope with these uncertainties, bad decisions may cause the robot to get lost or even to cause severe environmental damage. Autonomous, self-adaptive robots that operate in uncontrolled environments full of uncertainties need to be reliable! Since these uncertainties are hard to replicate in test deployments, we need methods to formally analyse self-adaptive robots operating in uncontrolled environments. In this paper, we show how feature-oriented techniques can be used to formally model and analyse self-adaptive robotic systems in the presence of such uncertainties. Self-adaptive systems can be organised as two-layered systems with a managed subsystem handling the domain concerns and a managing subsystem implementing the adaptation logic. We consider a case study of an autonomous underwater vehicle (AUV) for pipeline inspection, in which the managed subsystem of the AUV is modelled as a family of systems, where each family member corresponds to a valid configuration of the AUV which can be seen as an operating mode of the AUV’s behaviour. The managing subsystem of the AUV is modelled as a control layer that is capable of dynamically switching between such valid configurations, depending on both environmental and internal uncertainties. These uncertainties are captured in a probabilistic and highly configurable model. Our modelling approach allows us to exploit powerful formal methods for feature-oriented systems, which we illustrate by analysing safety properties, energy consumption, and multi-objective properties, as well as performing parameter synthesis to analyse to what extent environmental conditions affect the AUV. The case study is realised in the probabilistic feature-oriented modelling language and verification tool ProFeat, and in particular exploits family-based probabilistic and parametric model checking.
https://doi.org/10.1016/j.scico.2024.103221
Analysing Self-Adaptive Systems as Software Product Lines
Authors: Juliane Päßler, Maurice H. ter Beek, Ferruccio Damiani, S. Lizeth Tapia Tarifa, Einar Broch Johnsen
Date: 2025
Abstract
Self-adaptation is a crucial feature of autonomous systems that must cope with uncertainties in, e.g., their environment and their internal state. Self-adaptive systems (SASs) can be realised as two-layered systems, introducing a separation of concerns between the domain-specific functionalities of the system (the managed subsystem) and the adaptation logic (the managing subsystem), i.e., introducing an external feedback loop for managing adaptation in the system. We present an approach to model SASs as dynamic software product lines (SPLs) and leverage existing approaches to SPL-based analysis for the analysis of SASs. To do so, the functionalities of the SAS are modelled in a feature model, capturing the SAS’s variability. This allows us to model the managed subsystem of the SAS as a family of systems, where each family member corresponds to a valid feature configuration of the SAS. Thus, the managed subsystem of an SAS is modelled as an SPL model; more precisely, a probabilistic featured transition system. The managing subsystem of an SAS is modelled as a control layer capable of dynamically switching between these valid configurations, depending on both environmental and internal conditions. We demonstrate the approach on a small-scale evaluation of a self-adaptive autonomous underwater vehicle used for pipeline inspection, which we model and analyse with the feature-aware probabilistic model checker ProFeat. The approach allows us to analyse probabilistic reward and safety properties for the SAS, as well as the correctness of its adaptation logic.
https://doi.org/10.1016/j.jss.2024.112324
Sonar-based Deep Learning in Underwater Robotics: Overview, Robustness and Challenges
Authors: Martin Aubard, Ana Madureira, Luís Teixeira, José Pinto
Date: 2025
Abstract
With the growing interest in underwater exploration and monitoring, Autonomous Underwater Vehicles (AUVs) have become essential. The recent interest in onboard Deep Learning (DL) has advanced real-time environmental interaction capabilities relying on efficient and accurate vision-based DL models. However, the predominant use of sonar in underwater environments, characterized by limited training data and inherent noise, poses challenges to model robustness. This autonomy improvement raises safety concerns for deploying such models during underwater operations, potentially leading to hazardous situations. This paper aims to provide the first comprehensive overview of sonar-based DL under the scope of robustness. It studies sonar-based DL perception task models, such as classification, object detection, segmentation, and SLAM. Furthermore, the paper systematizes sonar-based state-of-the-art datasets, simulators, and robustness methods such as neural network verification, out-of-distribution, and adversarial attacks. This paper highlights the lack of robustness in sonar-based DL research and suggests future research pathways, notably establishing a baseline sonar-based dataset and bridging the simulation-to-reality gap.
DOI: 10.1109/JOE.2025.3531933
Language-Based Testing for Knowledge Graphs
Authors: Tobias John, Einar Broch Johnsen, Eduard Kamburjan, Dominic Steinhöfel
Date: 2025
Abstract
Knowledge graphs rely on a vast ecosystem of software tools, such as parsers, APIs and reasoners. Yet, tool developers have little support to ensure tool reliability. Here, we demonstrate how recent advantages in test case generation for highly structured and constrained inputs can support software developers in the semantic web field. We develop input generators for RDF Turtle and the OWL EL profile, and report on numerous bugs we found in parsers, reasoners and APIs of widely used tools and libraries, as well as imprecisions in standards and documentation. We provide actionable insights on using automated testing to increase the reliability of software tools for knowledge graphs.