Here shows our work related to Intelligent Microrobotics.
The main target of my work is to incorporate accurate perception and dexterous manipulation to a conventional optical tweezer (OT) system to build up a versatile micro-robotic platform for biomedical applications.
Recent technological advances in micro-robotic systems have demonstrated much potential for biomedical applications. These include diagnostics at the cellular level, drug delivery and microsurgery. Similar to the manipulation in macro-scale robotics, dexterous manipulation is one of the most active areas to study for micro-robotics, where new approaches and technologies are explored to manipulate micro-objects or microrobots for different purposes.
Micro-robotic systems include tethered and untethered microrobots. Among untethered micromanipulation, optical manipulation is one of the most accurate approaches and has been demonstrated in applications such as cell analysis and manipulation. Optical tweezer (OT) is a versatile tool for optical manipulation, and is the main focus of this project. To avoid damages to living cells caused by illuminating laser directly on them, optical microrobots controlled by OT can be used for indirect manipulation of biological objects in micro-scale. However, there are some fundamental challenges when using optical microrobots for biomedical applications. For a conventional OT system, the feedback system is normally limited to one camera view, while the transparency of the optical microrobot and its unobservable motions parallel to the optical axis bring great challenges to the pose estimation in 3D. Moreover, the out-of-plane rotation of the optical microrobots manipulated via multi-spot planar OT is difficult to control, which requires the special design of the microrobots' structure as well as the control strategies to reach this target. Therefore, to develop a dexterous micro-robotic platform for biomedical research, perception and manipulation of untethered optical microrobots via OT are two key aspects to be studied.
I envision that the work developed in this project may have impact on a wide range of biomedical research. The proposed design of optical microrobots can be used as a tool at the tip of fibre to manipulate micro-objects or implement tissue biopsy using the optical traps generated by multiple optical fibres. Moreover, the optical microrobots can be used as dexterous tools for the lab-on-the-chip system to realize cell diagnosis and can assist cell manipulation that requires rotational control. For example, the optical microrobots can be used to control the biological object at a desired pose, actuated by OT. In this way, the micromanipulation tools can be ensured to operate at the correct side of the target object and access the target without damaging the key components inside. This can assist applications of in vitro fertilization (IVF), preimplantation genetic diagnosis (PGD), intracytoplasmic sperm injection (ICSI), and organelle extraction analysis and diagnosis, where the cells are required to be oriented properly.
To avoid damaging living cells, microrobots controlled by optical tweezers can be used for the manipulation of cells or living organisms at the microscopic scale.
A SEM image, created by Dandan Zhang et al, shows one such robot. This image has been selected for "This month in pictures" feature on Advanced Science News.
Distributed Force Control for Microrobot Manipulation via
Planar Multi-Spot Optical Tweezer
Dandan Zhang; Antoine Barbot; Benny Lo; Guang-Zhong Yang
Optical tweezers (OT) represent a versatile tool for micro-manipulation. To avoid damages to living cells caused by illuminating laser directly on them, microrobots controlled by OT can be used for manipulation of cells or living organisms in microscopic scale. Translation and planar rotation motion of microrobots can be realized by using a multi-spot planar OT. However, out-of-plane manipulation of microrobots is difficult to achieve with a planar OT. This paper presents a distributed manipulation scheme based on multiple laser spots, which can control the out-of-plane pose of a microrobot along multiple axes. Different microrobot designs have been investigated and fabricated for experimental validation. The main contributions of this paper include: i) development of a generic model for the structure design of microrobots which enables multi-dimensional (6D) control via conventional multi-spot OT; ii) introduction of the distributed force control for microrobot manipulation based on characteristic distance and power intensity distribution. Experiments are performed to demonstrate the effectiveness of the proposed method and its potential applications, which include indirect manipulation of micro-objects.
Full paper link：
Data-Driven Microscopic Pose and Depth Estimation for
Optical Microrobot Manipulation
Dandan Zhang; Frank P.-W. Lo; Jian-Qing Zheng; Wenjia Bai; Guang-Zhong Yang; Benny Lo
Optical microrobots have a wide range of applications in biomedical research for both in vitro and in vivo studies. In most microrobotic systems, the video captured by a monocular camera is the only way for visualizing the movements of microrobots, and only planar motion, in general, can be captured by a monocular camera system. Accurate depth estimation is essential for 3D reconstruction or autofocusing of microplatforms, while the pose and depth estimation are necessary to enhance the 3D perception of the microrobotic systems to enable dexterous micromanipulation and other tasks. In this paper, we propose a data-driven method for pose and depth estimation in an optically manipulated microrobotic system. Focus measurement is used to obtain features for Gaussian Process Regression (GPR), which enables precise depth estimation. For mobile microrobots with varying poses, a novel method is developed based on a deep residual neural network with the incorporation of prior domain knowledge about the optical microrobots encoded via GPR. The method can simultaneously track microrobots with complex shapes and estimate the pose and depth values of the optical microrobots. Cross-validation has been conducted to demonstrate the submicron accuracy of the proposed method and precise pose and depth perception for microrobots. We further demonstrate the generalizability of the method by adapting it to microrobots of different shapes using transfer learning with few-shot calibration. Intuitive visualization is provided to facilitate effective human-robot interaction during micromanipulation based on pose and depth estimation results.
The proposed method was evaluated based on the suturing task with data obtained from the public available JIGSAWS database. Comparative studies were conducted to verify the proposed framework. Results indicated that the testing accuracy for the suturing task based on our proposed method is 87.13%, which outperforms most of the state-of-the-art algorithms.
Full paper link：
Micro-object pose estimation with
sim-to-real transfer learning using small dataset
Three-dimensional (3D) pose estimation of micro/nano-objects is essential for the implementation of automatic manipulation in micro/nano-robotic systems. However, out-of-plane pose estimation of a micro/nano-object is challenging, since the images are typically obtained in 2D using a scanning electron microscope (SEM) or an optical microscope (OM). Traditional deep learning-based methods require the collection of a large amount of labeled data for model training to estimate the 3D pose of an object from a monocular image. Here we present a sim-to-real learning-to-match approach for 3D pose estimation of micro/nano-objects. Instead of collecting large training datasets, simulated data is generated to enlarge the limited experimental data obtained in practice, while the domain gap between the generated and experimental data is minimized via image translation based on a generative adversarial network (GAN) model. A learning-to-match approach is used to map the generated data and the experimental data to a low-dimensional space with the same data distribution for different pose labels, which ensures effective feature embedding. Combining the labeled data obtained from experiments and simulations, a new training dataset is constructed for robust pose estimation. The proposed method is validated with images from both SEM and OM, facilitating the development of closed-loop control of micro/nano-objects with complex shapes in micro/nano-robotic systems.
Full paper link：
Dandan Zhang, Antoine Barbot, Florent Seichepine, Frank Lo, Wenjia Bai,
Guang-Zhong Yang, Benny Lo
A microsurgical robot research platform for robot-assisted
microsurgery research and training
A microsurgical robot research platform (MRRP) is introduced in this paper. The hardware system includes a slave robot with bimanual manipulators, two master controllers and a vision system. It is flexible to support multiple microsurgical tools. The software architecture is developed based on the robot operating system, which is extensible at high-level control. The selection of master–slave mapping strategy was explored, while comparisons were made between different interfaces.
Experimental verification was conducted based on two microsurgical tasks for training evaluation, i.e. trajectory following and targeting. User study results indicated that the proposed hybrid interface is more effective than the traditional approach in terms of frequency of clutching, task completion time and ease of control. Results indicated that the MRRP can be utilized for microsurgical skills training, since motion kinematic data and vision data can provide objective means of verification and scoring. The proposed system can further be used for verifying high-level control algorithms and task automation for RAMS research.
Full paper link：