Our Work
Here shows our work related to implicit human-robot shared control.
01
Human-Robot Shared Control for Surgical Robot
Based on Context-Aware Sim-to-Real Adaptation
Dandan Zhang; Zicong Wu; Junhong Chen; Ruiqi Zhu; Adnan Munawar; Bo Xiao; Yuan Guan
Hang Su; Yao Guo; Gregory Fischer; Benny Lo; Guang-Zhong Yang
Human-robot shared control, which integrates the advantages of both humans and robots, is an effective approach to facilitate efficient surgical operations.
Learning from demonstration (LfD) techniques can be used to automate some of the surgical subtasks for the construction of the shared control mechanism. However, a sufficient amount of data is required for the robot to learn the manoeuvres. Using a surgical simulator to collect data is a less resource-demanding approach. With sim-to-real adaptation, the manoeuvres learned from a simulator can be transferred to a physical robot. To this end, we propose a sim-to-real adaptation method to construct a human-robot shared control framework for robotic surgery.
In this paper, a desired trajectory is generated from a simulator using LfD method, while dynamic motion primitives (DMP) is used to transfer the desired trajectory from the simulator to the physical robotic platform. Moreover, a role adaptation mechanism is developed such that the robot can adjust its role according to the surgical operation contexts predicted by a neural network model.
The effectiveness of the proposed framework is validated on the da Vinci Research Kit (dVRK). Results of the user studies indicated that with the adaptive human-robot shared control framework, the path length of the remote controller, the total clutching number and the task completion time can be reduced significantly. The proposed method outperformed the traditional manual control via teleoperation.
02
A Self-Adaptive Motion Scaling Framework
for Surgical Robot Remote Control
Dandan Zhang; Bo Xiao; Baoru Huang; Lin Zhang; Jindong Liu; Guang-Zhong Yang
Master–slave control is a common form of human–robot interaction for robotic surgery. To ensure seamless and intuitive control, a mechanism of self-adaptive motion scaling during teleoperation is proposed in this letter. The operator can retain precise control when conducting delicate or complex manipulation, while the movement to a remote target is accelerated via adaptive motion scaling. The proposed framework consists of three components: 1) situation awareness, 2) skill level awareness, and 3) task awareness. The self-adaptive motion scaling ratio allows the operators to perform surgical tasks with high efficiency, forgoing the need of frequent clutching and instrument repositioning. The proposed framework has been verified on a da Vinci Research Kit to assess its usability and robustness. An in-house database is constructed for offline model training and parameter estimation, including both the kinematic data obtained from the robot and visual cues captured through the endoscope. Detailed user studies indicate that a suitable motion-scaling ratio can be obtained and adjusted online. The overall performance of the operators in terms of control efficiency and task completion is significantly improved with the proposed framework.
Full paper link:
03
An Ergonomic Shared Workspace Analysis Framework for
the Optimal Placement of a Compact Master Control Console
Dandan Zhang; Anzhu Gao; Jindong Liu; Guang-Zhong Yang
Master-Slave control is commonly used for Robot-Assisted Minimally Invasive Surgery (RAMIS). The configuration, as well as the placement of the master manipulators, can influence the remote control performance. An ergonomic shared workspace analysis framework is proposed in this paper. Combined with the workspace of the master manipulators and the human arms, the human-robot interaction-workspace can be generated. The optimal master robot placement can be determined based on three criteria: 1) interaction workspace volume, 2) interaction workspace quality, and 3) intuitiveness for slave robot control. Experimental verification of the platform is conducted on a da Vinci Research Kit (dVRK). An in-house compact master manipulator (Hamlyn CRM) is used as the master robot and the da Vinci robot is used as the slave robot. Comparisons are made between with and without using design optimization to validate the effectiveness of ergonomic shared workspace analysis. Results indicate that the proposed ergonomic shared workspace analysis can improve the performance of teleoperation in terms of task completion time and the number of clutching required during operation.
Full paper link：
04
Supervised Semi-Autonomous Control for Surgical Robot
Based on Bayesian Optimization
Supervisory control functions should be included to ensure flexibility and safety during the autonomous control phase. This paper presents a haptic rendering interface to enable supervised semi-autonomous control for a surgical robot. Bayesian optimization is used to tune user-specific parameters during the surgical training process. User studies were conducted on a customized simulator for validation. Detailed comparisons are made between with and without the supervised semi-autonomous control mode in terms of the number of clutching events, task completion time, master robot end-effector trajectory and average control speed of the slave robot. The effectiveness of the Bayesian optimization is also evaluated, demonstrating that the optimized parameters can significantly improve users' performance. Results indicate that the proposed control method can reduce the operator's workload and enhance operation efficiency.
Full paper link：
Junhong Chen#, Dandan Zhang#; Benny Lo; Guang-Zhong Yang
05
Deep Reinforcement Learning Based
Semi-Autonomous Control for Robotic Surgery
In this paper, we propose a semi-autonomous control framework for robotic surgery and evaluate this framework in a simulated environment. We applied deep reinforcement learning methods to train an agent for autonomous control, which includes simple but repetitive manoeuvres. Compared to learning from demonstration, deep reinforcement learning can learn a new policy by altering the goal via modifying the reward function instead of collecting new dataset for a new goal. In addition to the autonomous control, we also created a handheld controller for manual precision control. The user can seamlessly switch to manual control at any time by moving the handheld controller. Finally, our method was evaluated in a customized simulated environment to demonstrate its efficiency compared to full manual control.
Full paper link：
Ruiqi Zhu; Dandan Zhang; Benny Lo
06
Design and Verification of A Portable Master Manipulator
Based on an Effective Workspace Analysis Framework
Master manipulators represent a key component of Robot-Assisted Minimally Invasive Surgery (RAMIS). In this paper, an Analytic Hierarchy Process (AHP) method is used to construct an effective workspace analysis framework, which can assist the configuration selection and design evaluation of a portable master manipulator for surgical robot control and training. The proposed framework is designed based on three criteria: 1) compactness, 2) workspace quality, and 3) mapping efficiency. A hardware prototype, called the Hamlyn Compact Robotic Master (Hamlyn CRM), is constructed following the proposed framework. Experimental verification of the platform is conducted on the da Vinci Research Kit (dVRK) with which a da Vinci robot is controlled as a slave. The proposed Hamlyn CRM is compared with Phantom Omni, a commercial portable master device, with results demonstrating the relative merits of the new platform in terms of task completion time, average control speed and number of clutching.
Full paper link：
07
WSRender: A Workspace Analysis and Visualization Toolbox for
Robotic Manipulator Design and Verification
Dandan Zhang#; Francesco Cursi#; Guang-Zhong Yang
Workspace analysis is essential for robotic manipulators, which helps researchers to study, evaluate and optimize their designs based on specific criteria with due consideration of ergonomics and usability. Although workspace analysis is a common research topic, current solutions provide design-specific evaluation and there is a lack of generic software tools for different hardware configurations. This paper presents WSRender, a versatile research-oriented framework for workspace analysis and visualization. It is based on the Orocos Kinematics and Dynamics Library (KDL) and the Matlab Robotic Toolbox. The software architecture is presented with four use cases for demonstrating its practical use in single robot, dual-arm manipulator performance evaluation, multi-robot interaction analysis and master-slave mapping. The source code of WSRender is made publicly available for the benefit of the research community for the design or evaluation of robotic manipulators.
Full paper link：
08
Hamlyn CRM: A Compact Master Manipulator for
Surgical Robot Remote Control
Compact master manipulators have inherent advantages since they can have practical deployment within the general surgical environments easily and bring benefits to surgical training. To assess the advantages of compact master manipulators for surgical skills training and the performance of general robot-assisted surgical tasks, Hamlyn Compact Robotic Master (Hamlyn CRM), is built up and evaluated in this paper.
A compact structure for the master manipulator is proposed. A novel sensing system is designed while stable real-time motion tracking can be realized by fusing the information from multiple sensors. User studies were conducted based on a ring transfer task and a needle passing task to explore a suitable mapping strategy for the compact master manipulator to control a surgical robot remotely. The overall usability of the Hamlyn CRM is verified based on the da Vinci Research Kit (dVRK). The master manipulators of the dVRK control console are used as the reference.
Motion tracking experiments verified that the proposed system can track the operators' hand motion precisely. As for the master-slave mapping strategy, user studies proved that the combination of the position relative mapping mode and the orientation absolute mapping mode is suitable for Robot-Assisted Minimally Invasive Surgery (RAMIS), while key parameters for mapping are selected. Results indicated that the Hamlyn CRM can serve as a compact master manipulator for surgical training, and has potential applications for RAMIS.
Full paper link:
https://link.springer.com/article/10.1007/s11548-019-02112-y
09
A Handheld Master Controller for Robot-Assisted Microsurgery
Dandan Zhang; Yao Guo; Junhong Chen; Jindong Liu; Guang-Zhong Yang
Accurate master-slave control is important for Robot-Assisted Microsurgery (RAMS). This paper presents a handheld master controller for the operation and training of RAMS. A 9-axis Inertial Measure Unit (IMU) and a micro camera are utilized to form the sensing system for the handheld controller. A new hybrid marker pattern is designed to achieve reliable visual tracking, which integrated QR codes, Aruco markers, and chessboard vertices. Real-time multi-sensor fusion is implemented to further improve the tracking accuracy. The proposed handheld controller has been verified on an in-house microsurgical robot to assess its usability and robustness. User studies were conducted based on a trajectory following task, which indicated that the proposed handheld controller had comparable performance with the Phantom Omni, demonstrating its potential applications in microsurgical robot control and training.
Full paper link:
Ongoing Projects
-
Machine Learning-Based Adaptive Human-Robot Shared Control
-
Haptic Guidance for Dexterous Manipulation
-
Reinforcement Learning for Human-in-the-Loop Control
