Duy-Nguyen Ta

Applied Scientist / Roboticist · Robotics & AI Institute

Email GitHub LinkedIn CV (PDF)

I am a roboticist at the Robotics & AI Institute (RAI, formerly the Boston Dynamics AI Institute), where I work on whole-body loco-manipulation for the Boston Dynamics Spot robot — combining sampling-based model-predictive control, reinforcement learning, and policy learning, together with the large-scale simulation, data, and deployment infrastructure that makes them work on real hardware. Along the way I have been lucky to work closely with Simon Le Cléac'h, and with Bernadette Bucher and Preston Culbertson before they became professors (Michigan Robotics and Cornell CS).

Before RAI, I spent five years at Toyota Research Institute working on perception for dexterous manipulation on the team led by Prof. Russ Tedrake, led perception at Outrider, and built the visual SLAM backend that shipped on the iRobot Roomba 980. I received my Ph.D. from Georgia Tech, where I worked with Prof. Frank Dellaert on factor-graph methods that unify perception and optimal control for autonomous flight. In an earlier life, I built augmented-reality systems at Nokia Research Center and mixed-reality art installations in Singapore.

If my Vietnamese name looks hard to pronounce, it is roughly “Zwee” (Duy) and “Nwin” (Nguyen) — and here is how not to say it.

News

Aug 2026Martin Schuck’s paper on learning loco-manipulation from SMPC demonstrations with sparse offline-to-online RL was accepted to CoRL 2026. [arXiv]
Jun 2026John Z. Zhang’s Sumo was presented at the RSS 2026 Workshop on Whole-Body Control and Manipulation. [project]
Jan 2026Hongyu Li’s NovaFlow was accepted to ICRA 2026. [project]
2025We open-sourced judo, a user-friendly framework for sampling-based MPC. [code]
Apr 2025Yun Chang’s ASHiTA, automatic scene-grounded hierarchical task analysis, appeared at CVPR 2025. [arXiv]
Oct 2023I joined the Robotics & AI Institute (then the Boston Dynamics AI Institute).

Selected Projects

Whole-body loco-manipulation and robot learning at the Robotics & AI Institute, plus favorite first-author work from earlier chapters.

Spot and G1 robots performing loco-manipulation tasks

CoRL 2026

Loco-Manipulation from SMPC Demonstrations

Sampling-based MPC acts as an automated expert generating massive offline datasets in simulation; sparse offline-to-online RL then distills robust loco-manipulation policies, deployed on an arm-equipped Spot and a Unitree G1 humanoid. Led by Martin Schuck (TUM/ETH) with our team at RAI.

Project arXiv

Spot uprighting a car tire with whole-body manipulation

RSS 2026 WCBM Workshop

Sumo: Dynamic Whole-Body Loco-Manipulation

Test-time steering of a pre-trained whole-body RL policy with a sampling-based planner lets Spot upright, roll, drag, and stack 15 kg tires — and adapt to new objects and tasks at deployment without retraining. I support this work with GPU-parallel SMPC data generation (MuJoCo Warp), training infrastructure, and on-robot deployment tooling.

Project arXiv Code Video RAI Blog

judo interactive GUI running a cartpole task with sampled trajectories

Open Source

judo: Sampling-Based MPC Made Easy

A user-friendly Python framework for prototyping, benchmarking, and deploying sampling-based MPC controllers: MuJoCo physics, asynchronous execution for sim-to-hardware transfer, and an interactive tuning GUI. I co-maintain the package and its MuJoCo Warp integration for massive GPU-parallel rollouts.

Code RAI Page

Conditional probability density learned by an energy-based model

RSS 2022 Workshop

Conditional EBMs for Implicit Policies

Why are implicit, energy-based behavior-cloning policies so hard to train? My last project at TRI, with Russ Tedrake’s team — Siyuan Feng, Eric Cousineau, since spun out as Walden Robotics — documenting the gap between EBM theory and practice through many failed experiments: the very training pathologies that Cheng Chi’s Diffusion Policy, born at TRI that same summer, sidesteps with denoising diffusion. The team’s later results — robots learning dozens of dexterous kitchen skills from demonstration — are phenomenal.

arXiv

TRI robot loading dishes from a kitchen sink

TRI Demo · RSS 2019

The TRI Dish-Loading Demo

TRI’s flagship manipulation demo: a robot perceiving and loading dishes in a real kitchen sink, used as a testbed for the hard problems of reliable manipulation. I led the perception team — deep-learned probabilistic object pose estimation in clutter, multicamera and hand-eye calibration, and depth-based object tracking.

Project Video

ICRA 2018

Pose-Graph Sparsification for Lifelong SLAM

Lifelong mapping on a consumer robot means the pose graph grows without bound, and marginalizing old nodes creates dense, expensive cliques. This fast, near-optimal nonlinear approximation of node marginalization and edge sparsification keeps long-term graph SLAM tractable on tiny embedded processors like the Roomba’s.

Paper Video

A nonlinear constrained factor graph with its SQP primal and dual linear graphs

ICUAS 2014

SQP on Factor Graphs: Estimation Meets Control

Extending factor graphs from estimation to constrained optimal control: system dynamics, discretized by integration on Lie-group manifolds, enter the graph as hard constrained factors, and an SQP formulation solves the result — revealing an elegant duality between primal and dual factor graphs.

Paper

Earlier Work

Autonomous Flight — Georgia Tech (2011–2014)

Perception and optimal control for autonomous navigation on micro aerial vehicles, with Prof. Frank Dellaert.

Factor graphs for estimation and model-predictive control. We extended factor graphs from estimation to deterministic optimal control: dynamics constraints are discretized via integration on Lie-group manifolds and imposed as constrained factors, solved with an SQP formulation that reveals an elegant connection between primal and dual factor graphs (ICUAS 2014, with Prof. Marin Kobilarov). A companion line of work derived differential dynamic programming for optimal estimation (ICRA 2015).

Linear-time SLAM with incremental tree filtering. My thesis work explored object-centric approaches to fast estimation, leading to linear-time approximation algorithms for information-filtering SLAM — a problem that is traditionally cubic in the number of features (IROS 2014).

Vistas: far-away features for rotation aiding and autonomous flight. Distant features (“vistas”) barely translate in the image, making them ideal for estimating rotation and for steering straight flight. Our ARDrone flew autonomously down a hallway and through a narrow door using vistas alone (FUSION 2012). Videos: first autonomous flight · vista detection

Wall-floor features for lateral collision avoidance. With Kyel Ok, I used intersections between vertical edges and wall-floor boundary lines — detected with steerable filters — to infer wall structure that sparse point features cannot provide, enabling the drone’s first fully autonomous turns indoors (IROS workshop 2012; RAS journal 2014). Videos: feature detection

Monocular parallel tracking and mapping with odometry fusion. PTAM-style systems fail on forward-moving robots with a single frontal camera: too few corner features, and too little motion parallax to triangulate a map. Fusing odometry into a parallel tracking-and-mapping framework kept localization real-time while a slower thread inferred wall structure (IROS workshop 2013; RAS 2014). Videos: first turn · clean turn

MonoSLAM in feature-poor corridors. Corner-feature SLAM breaks down in bare hallways, so I built a monocular SLAM system on vertical edges, with EM for data association and inverse-depth parameterization (Gamma priors turn out to be delicate to tune). Videos: edge-based SLAM · with height estimation

Flying through a forest. A bio-inspired two-part vision system for an ARDrone in the wild: a peripheral system picks out motion-salient nearby objects via optical-flow subspace methods (by Richard Roberts), and a foveal system tracks the detected tree trunks with model-based edge tracking (SPIE 2012). Video: tree detection and tracking

Mobile Augmented Reality — Georgia Tech & Nokia Research (2007–2010)

SURFTrac: continuous tracking and recognition for outdoor AR. A lightweight method that matches SURF features against a database of geo-tagged, labeled images once, then tracks them locally frame-to-frame — fast enough to label buildings live on a Nokia N95 at 8–12 fps (CVPR 2009 oral, 5% acceptance; demos at ISMAR’08 and CVPR’09; US patent). Video: SURFTrac

SURFTrac3D: full 6-DoF outdoor tracking. Extending SURFTrac to 3D: a feature database reconstructed with structure-from-motion provides the map, RANSAC initializes the camera pose, and SURFTrac keeps the pose updated across frames — stable enough to pin virtual 3D content on real buildings. Video: SURFTrac3D

Template-based tracking for handheld AR. A PTAM-inspired template tracker for the Nokia N900, with performance comparable to the tracker that later grew into Qualcomm’s Vuforia. Video: patch tracking

Art of Defense: a collaborative handheld AR board game. A tabletop tower-defense game where players cooperate through camera phones, blending physical game pieces with virtual gameplay (ACM SIGGRAPH Symposium on Video Games 2009). Videos: SIGGRAPH 2009 · version 1 · ISMAR’08 demo

Lego edge tracking. With Yong-Dian Jian: real-time model-based edge tracking of textureless Lego structures, where point features fail. After experiments with particle filters, the stable solution was classic Gauss-Newton least squares — plus many small tricks. Video: edge tracking

Mixed-Reality Art & Entertainment — Singapore (2003–2007)

Magic Land. My Master’s thesis at NUS: a live 3D human-capture system using 12 cameras in a green room and a fast shape-from-silhouette algorithm, letting players see miniature 3D copies of themselves interact with virtual characters on an AR tabletop (TVCG 2005; exhibited at Singapore Science Center, SIGCHI’05, and WIRED NextFest’05). Video: Magic Land

Periphery. An interactive art installation using two infrared cameras to track visitors and their hands: approaching the board dissolves a live-captured modern scene into the forest it once was; pointing reveals the artist-drawn past along the visitor’s fingertip (2nd International Art and Science Exhibition, Tsinghua University, 2006).

Ultimate Commodity. One of the few traditional theater performances to use AR on stage: in a dystopian Singaporean story where everyone’s face becomes identical, we replaced actors’ heads with a virtual model of the mother’s head, live during the performance (Esplanade Theater, Singapore 2006; Fringe Festival, Toronto 2007).

Free Network Visible Network. With artists Clara Boj and Diego Diaz: wireless access points become AR markers, and the invisible network traffic between them is visualized as translucent boxes and donuts flying through the air (ISEA06 ZERO ONE, San Jose).

And more. The vision-tracking component of Poultry Internet, demonstrated live on Good Morning America; Age Invaders (SIGCHI’06); and Fragile!, an interactive media-art installation.

Publications

Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL
M. Schuck, M. Sorokin, S. Manni, D. Ta, A.P. Schoellig, M. Hutter, S. Le Cléac’h, J. Brüdigam. CoRL, 2026. arXiv
Sumo: Dynamic and Generalizable Whole-Body Loco-Manipulation
J.Z. Zhang, M. Sorokin, J. Brüdigam, B. Hung, S. Phillips, D. Yershov, F. Niroui, T. Zhao, L. Fermoselle, X. Zhu, C. Cao, D. Ta, T. Pang, J. Wang, P. Culbertson, Z. Manchester, S. Le Cléac’h. Presented at the RSS 2026 Workshop on Whole-Body Control and Manipulation. arXiv · project
NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos
H. Li, L. Sun, Y. Hu, D. Ta, J. Barry, G. Konidaris, J. Fu. ICRA, 2026. arXiv · project
ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
Y. Chang, L. Fermoselle, D. Ta, B. Bucher, L. Carlone, J. Wang. CVPR, 2025. arXiv
Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation
S. Raychaudhuri, D. Ta, K. Ashton, A.X. Chang, J. Wang, B. Bucher. arXiv preprint, 2024. arXiv
Conditional Energy-Based Models for Implicit Policies: The Gap Between Theory and Practice
D.N. Ta, E. Cousineau, H. Zhao, S. Feng. RSS 2022 Workshop on Implicit Representations for Robotic Manipulation. arXiv
Fast Nonlinear Approximation of Pose Graph Node Marginalization
D.N. Ta, N. Banerjee, S. Eick, E. Pittore, S. Lenser. ICRA, 2018. DOI
Differential Dynamic Programming for Optimal Estimation
M. Kobilarov, D.N. Ta, F. Dellaert. ICRA, 2015. PDF
Linear-Time Estimation with Tree Assumed Density Filtering and Low-Rank Approximation
D.N. Ta, F. Dellaert. IROS, 2014. PDF
A Factor Graph Approach to Estimation and Model Predictive Control on Unmanned Aerial Vehicles
D.N. Ta, M. Kobilarov, F. Dellaert. ICUAS, 2014. PDF
Vistas and Parallel Tracking and Mapping with Wall-Floor Features: Enabling Autonomous Flight in Man-made Environments
D.N. Ta, K. Ok, F. Dellaert. Robotics and Autonomous Systems, 2014. PDF
Monocular Parallel Tracking and Mapping with Odometry Fusion for MAV Navigation in Feature-Lacking Environments
D.N. Ta, K. Ok, F. Dellaert. IROS’13 Workshop on Vision-based Closed-Loop Control and Navigation of Micro Helicopters in GPS-denied Environments, 2013.
Saliency Detection and Model-based Tracking: A Two-Part Vision System for Small Robot Navigation in Forested Environments
R. Roberts, D.N. Ta, J. Straub, F. Dellaert. SPIE Defense, Security, and Sensing, 2012. PDF
Attitude Heading Reference System with Rotation-Aiding Visual Landmarks
C. Beall, D.N. Ta, K. Ok, F. Dellaert. FUSION, 2012. PDF
Vistas and Wall-Floor Intersection Features: Enabling Autonomous Flight in Man-made Environments
K. Ok, D.N. Ta, F. Dellaert. IROS Workshop on Visual Control of Mobile Robots, 2012. PDF
SURFTrac: Efficient Tracking and Continuous Object Recognition Using Local Feature Descriptors
D.N. Ta, W.C. Chen, N. Gelfand, K. Pulli. CVPR, 2009. (Oral, 5% acceptance rate) DOI
Art of Defense: A Collaborative Handheld Augmented Reality Board Game
D.N. Ta, K. Raveendran, Y. Xu, K. Spreen, B. MacIntyre. ACM SIGGRAPH Symposium on Video Games, 2009. DOI
Free Network Visible Network
C. Boj, D. Diaz, D.N. Ta, W. Liu, A.D. Cheok. ISEA06, 2006.
Age Invaders: Social and Physical Inter-Generational Family Entertainment
E.T. Khoo, S.P. Lee, A.D. Cheok, D.N. Ta. Springer Virtual Reality, VR-based Edutainment, 2006.
Social and Physical Interactive Paradigms for Mixed-Reality Entertainment
A.D. Cheok, K.S. Teh, D.N. Ta, Q.T.C. Tran, S.P. Lee, W. Liu, C.C. Li, D. Diaz, C. Boj. ACM Computers in Entertainment, 2006.
Real Time 3D Human Capture System for Mixed-Reality Art and Entertainment
D.N. Ta, Q.T.C. Tran, K. Xu, A.D. Cheok, et al. IEEE Transactions on Visualization and Computer Graphics, 2005. DOI
Magic Land: A 3D Human Capture Mixed Reality System for Museum Experiences
Q.T.C. Tran, D.N. Ta, A.D. Cheok, et al. Int’l Workshop: Re-Thinking Technology in Museums, 2005.
Magic Land: Live 3D Human Capture Mixed Reality Interactive System
Q.T.C. Tran, D.N. Ta, A.D. Cheok, et al. CHI’05 Extended Abstracts, 2005.
Searching the Web: A Semantics-Based Approach
T.H. Cao, D.N. Ta, Q.T.C. Tran. HPSC, 2003.

Patents

Systems, devices, and methods for generating a pose estimate of an object
with K. Hashimoto, E.A. Cousineau, R.L. Tedrake. US Patent 11308314, 2022.
Systems and methods for calibrating a depth-IR image offset
with A. Alspach, N.S. Kuppuswamy. US Patent 10628968, 2020.
Method, apparatus, and computer program product for object tracking
with W.-C. Chen, N. Gelfand, K. Pulli. US Patents US20100232643 A1 (2009), US8818024 B2 (2014).