[ICRA2024] Pedestrian Trajectory Prediction with Pose Estimation and Monte Carlo Dropout

Pedestrian Trajectory Prediction with Pose Estimation and Monte Carlo Dropout Shunya TADANO1, Yusuke TAMURA1, and Yasuhisa HIRATA1 1 Tohoku University 1. Introduction 4. Experiment Pedestrian Trajectory Prediction Data Collection and Model Evaluation Pedestrian movement prediction is essential for autonomous mobile robots (AMR) and social robots that share the same environment with humans. So far, many methods have been proposed such as Social Force Model[1], which expresses the effects surrounding pedestrians and the environment using forces of attraction and repulsion, and prediction models using machine learning[2]. The performance of the prediction model was evaluated using actual measured pedestrian data with RGBD camera. We created 297 training data sets and 127 test data sets. Prediction errors were evaluated using Average Displacement Error (ADE) and Final Displacement Error (FDE). Challenges Unable to deal with prediction failure • • Complex intersecting paths Sudden direction change of pedestrians Prediction from top-down view • • Evaluation Metrics Hard to get gesture information (human pose, gaze direction) Cannot obtain from robots Multiple Prediction Output • Using Monte Carlo Dropout • Considering the uncertainty Pedestrian pose provides useful cues in trajectory prediction 従来 Considering Uncertainty 2. Pose and Position Estimation Using the pixel values and depth information from the RGB-D camera, we estimated and predicted the position of a pedestrian. To obtain the pedestrian's position, we transformed the coordinates from image pixel coordinates to spatial coordinates. Egocentric View Capturing Coordinate transformation Pose 𝒑𝒕 skeleton points by OpenPose[3] (18 points) 20 steps Position + Pose 30 steps 10 steps 20 steps 30 steps 𝑥𝑤 𝑥𝑐 𝑦𝑤 = 𝑹𝑇 𝑦𝑐 − 𝒕 𝑧𝑤 𝑧𝑐 𝑥𝑐 , 𝑦𝑐 , 𝑧𝑐 : camera coordinates 𝑐𝑥 , 𝑐𝑦 : optical center 𝑓𝑥 , 𝑓𝑦 : focal length 3. Pedestrian Prediction Model Prediction Model Model Parameters generates latent variables from the pedestrian's movement trajectory. Input 15 steps Output 30 steps Feature 39 Optimization Adam Loss Function MSE randomly adjusts parameters in the model to improve the generalization performance. 従来 Position + Pose + MCD 𝑥𝑤 , 𝑦𝑤 , 𝑧𝑤 : camera coordinates 𝑹𝑇 : rotation matrix 𝒕: translation vector [3] Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. 2017. predict future trajectories based on the latent variables obtained from the LSTM layer. Highly Reliable Prediction by Monte Carlo Dropout 𝑥0,1 10 steps 従来 𝑢 − 𝑐𝑥 ⋅ 𝑑 𝑥𝑐 = , 𝑓𝑥 𝑣 − 𝑐𝑦 ⋅ 𝑑 𝑦𝑐 = , 𝑓𝑦 𝑧𝑐 = 𝑑 𝑥2,0 𝐹𝐷𝐸 = Pose Estimation [1] Alahi, Alexandre, et al. “Social lstm: Human trajectory prediction in crowded spaces.” CVPR. 2016 [2] Anvari, Bani, and Helge A. Wurdemann. "Modelling social interaction between humans and service robots in large public spaces." 2020 IROS.. 𝑥1,0 • Final Displacement Error (FDE) ADE [m] FDE [m] Position 1.46 1.70 Position + Pose 1.19 1.61 Position + Pose +MCD 0.02 0.04 Pose Estimation by Egocentric-View • Captured from RGBD camera • Considering human behavior Goal：Decrease Potential Prediction Failure 𝑥0,0 𝐴𝐷𝐸 = 𝑥𝑡 ′ − 𝑥ො𝑡 2 + 𝑦𝑡 ′ − 𝑦ො𝑡 ′ 2 , 𝑇 𝑥𝑇 − 𝑥ො𝑇 2 + 𝑦𝑇 − 𝑦ො𝑇 2 Model Contributions 𝑢, 𝑣 : pixel coordinates 𝑑: depth • Average Displacement Error (ADE) σ𝑇𝑡 ′ =1 • Iteration: 50 MCD can compensate for uncertainty in pedestrian prediction [5] Waghmare, Sagar M., et al. "SANPO: A Scene Understanding, Accessibility, Navigation, Pathfinding, Obstacle Avoidance Dataset." arXiv preprint arXiv:2309.12172 (2023). 5. Conclusion We proposed integration of pose estimation into pedestrian trajectory prediction considering uncertainty. Through experiments, • we confirmed that pose information improves the accuracy of pedestrian trajectory prediction. • by using Monte Carlo Dropout, we were able to output trajectories that are very close to the true value and compensate for the uncertainty of the pedestrian trajectory. Future Work • • Improving prediction model accuracy Data collection in complex environments 𝑥1,1 𝑥2,1 𝑥0,2 Acknowledgement 0 Randomly set output to 0 Multiple different outputs [3] Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. 2017. This work was supported by the Research Grant for Proposed Subject Research Projects 2022 from the Suzuki Foundation.