Compact Visuotactile World Models for Lifting: Prediction, Reward Alignment, and Force Constraints

Abstract:Accurate contact prediction is useful for robotic manipulation only if it supports effective decisions. We investigate this connection using a compact, randomly initialized visuotactile world model, trajectory-level uncertainty calibration, and behavior-initialized actor-critic learning in imagination. On 160 MuJoCo Lift episodes, adding touch reduces endpoint-force prediction error from 1.058 to 0.228 N and interval-peak error from 2.724 to 0.523 N across three training seeds. However, tactile persistence achieves lower errors of 0.095 and 0.498 N, respectively. Two exploratory control rounds comprise 680 executions on 40 independent test initial conditions. A matched reward revision on fresh test environments increases in-distribution 10 cm lifting success from 20.0% to 93.3%, while success within an 8 N per-finger budget reaches only 33.3%, compared with 70.0% for force feedback. Calibration margins reduce force violations at the cost of task completion. In a separate study of public GelSight recordings, a force regressor achieves 0.04234 N error, but frame-level calibration covers only 15.80% of complete trajectories; trajectory-level calibration raises this to 87.36% at nominal 90% coverage. Together, these findings distinguish improvements in sensing and task reward from improvements in force-constrained control. The evidence is limited to public sensing records and simulator execution, without a demonstrated transfer between them.

8 pages, 2 figures. Code and tabulated results included as ancillary material

Robotics (cs.RO); Artificial Intelligence (cs.AI)

arXiv:2609.09597 [cs.RO]

(or arXiv:2609.09597v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2609.09597 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

ENVIRONMENT.json

FILES_SHA256.json

PACKAGING_CHANGES.md

SOURCE_MANIFEST.json

THIRD_PARTY_NOTICES.md

outputs/research/analyze_force_envelopes.py

outputs/research/data_download_manifest.json

outputs/research/download_public_data.py

outputs/research/prepare_public_force_data.py

outputs/research/train_force_envelopes.py

outputs/simulation/audit_control_pairing.py

outputs/simulation/audit_dataset.py

outputs/simulation/evaluate_controller.py

outputs/simulation/generate_dataset.py

outputs/simulation/run_goal_aligned_matrix.py

outputs/simulation/run_policy_matrix.py

outputs/simulation/sim_adapter.py

outputs/world_model/LICENSE_TDMPC2.txt

outputs/world_model/analyze_results.py

outputs/world_model/controller_server.py

outputs/world_model/data.py

outputs/world_model/evaluate_executed_forecasts.py

outputs/world_model/model.py

outputs/world_model/run_goal_aligned_matrix.py

outputs/world_model/run_pilot_matrix.py

outputs/world_model/tdmpc2_layers.py

outputs/world_model/train.py

outputs/world_model/train_imagination_rl.py

recorded/outputs/research/data_download_manifest.json

recorded/outputs/research/real_force_alignment_sensitivity.csv

recorded/outputs/research/real_force_analysis_audit.json

recorded/outputs/research/real_force_bootstrap_intervals.csv

recorded/outputs/research/real_force_calibration.csv

recorded/outputs/research/real_force_data_audit.json

recorded/outputs/research/real_force_envelope_metrics.csv

recorded/outputs/research/real_force_predictor_metrics.csv

recorded/outputs/research/real_force_run_metadata.json

recorded/outputs/research/real_force_selective_metrics.csv

recorded/outputs/research/real_force_splits.json

recorded/outputs/research/real_force_summary_alpha10.csv

recorded/outputs/research/real_force_training_log.csv

recorded/outputs/simulation/dataset_audit.json

recorded/outputs/simulation/evaluation_goal_aligned/force_feedback/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/force_feedback/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/goal_aligned_evaluation_plan.json

recorded/outputs/simulation/evaluation_goal_aligned/pairing_audit.json

recorded/outputs/simulation/evaluation_goal_aligned/script/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/script/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_height_seed0_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_height_seed0_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_height_seed1_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_height_seed1_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_height_seed2_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_height_seed2_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_height_seed0_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_height_seed0_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_height_seed1_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_height_seed1_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_height_seed2_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_height_seed2_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_native_seed0_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_native_seed0_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_native_seed1_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_native_seed1_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_native_seed2_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_margin_native_seed2_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_native_seed0_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_native_seed0_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_native_seed1_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_native_seed1_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_native_seed2_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_native_seed2_rl/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_seed0_bc/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_seed0_bc/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_seed1_bc/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_seed1_bc/summary.json

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_seed2_bc/episodes.jsonl

recorded/outputs/simulation/evaluation_goal_aligned/visuotactile_seed2_bc/summary.json

recorded/outputs/simulation/evaluation_main/force_feedback/episodes.jsonl

recorded/outputs/simulation/evaluation_main/force_feedback/summary.json

recorded/outputs/simulation/evaluation_main/pairing_audit.json

recorded/outputs/simulation/evaluation_main/policy_evaluation_plan.json

recorded/outputs/simulation/evaluation_main/script/episodes.jsonl

recorded/outputs/simulation/evaluation_main/script/summary.json

recorded/outputs/simulation/evaluation_main/vision_seed0_bc/episodes.jsonl

recorded/outputs/simulation/evaluation_main/vision_seed0_bc/summary.json

recorded/outputs/simulation/evaluation_main/vision_seed0_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_main/vision_seed0_rl/summary.json

recorded/outputs/simulation/evaluation_main/vision_seed1_bc/episodes.jsonl

recorded/outputs/simulation/evaluation_main/vision_seed1_bc/summary.json

recorded/outputs/simulation/evaluation_main/vision_seed1_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_main/vision_seed1_rl/summary.json

recorded/outputs/simulation/evaluation_main/vision_seed2_bc/episodes.jsonl

recorded/outputs/simulation/evaluation_main/vision_seed2_bc/summary.json

recorded/outputs/simulation/evaluation_main/vision_seed2_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_main/vision_seed2_rl/summary.json

recorded/outputs/simulation/evaluation_main/visuotactile_margin_seed0_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_main/visuotactile_margin_seed0_rl/summary.json

recorded/outputs/simulation/evaluation_main/visuotactile_margin_seed1_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_main/visuotactile_margin_seed1_rl/summary.json

recorded/outputs/simulation/evaluation_main/visuotactile_margin_seed2_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_main/visuotactile_margin_seed2_rl/summary.json

recorded/outputs/simulation/evaluation_main/visuotactile_seed0_bc/episodes.jsonl

recorded/outputs/simulation/evaluation_main/visuotactile_seed0_bc/summary.json

recorded/outputs/simulation/evaluation_main/visuotactile_seed0_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_main/visuotactile_seed0_rl/summary.json

recorded/outputs/simulation/evaluation_main/visuotactile_seed1_bc/episodes.jsonl

recorded/outputs/simulation/evaluation_main/visuotactile_seed1_bc/summary.json

recorded/outputs/simulation/evaluation_main/visuotactile_seed1_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_main/visuotactile_seed1_rl/summary.json

recorded/outputs/simulation/evaluation_main/visuotactile_seed2_bc/episodes.jsonl

recorded/outputs/simulation/evaluation_main/visuotactile_seed2_bc/summary.json

recorded/outputs/simulation/evaluation_main/visuotactile_seed2_rl/episodes.jsonl

recorded/outputs/simulation/evaluation_main/visuotactile_seed2_rl/summary.json

recorded/outputs/simulation/generation_log.jsonl

recorded/outputs/simulation/manifest.json

recorded/outputs/simulation/reproducibility_check.json

recorded/outputs/world_model/analysis/analysis_metadata.json

recorded/outputs/world_model/analysis/control_analysis.json

recorded/outputs/world_model/analysis/controller_method_metrics.csv

recorded/outputs/world_model/analysis/controller_method_paired_differences.csv

recorded/outputs/world_model/analysis/controller_metrics.csv

recorded/outputs/world_model/analysis/controller_paired_differences.csv

recorded/outputs/world_model/analysis/executed_forecast_analysis.json

recorded/outputs/world_model/analysis/world_model_analysis.json

recorded/outputs/world_model/analysis/world_model_calibration.csv

recorded/outputs/world_model/analysis/world_model_paired_modality.csv

recorded/outputs/world_model/analysis/world_model_run_metrics.csv

recorded/outputs/world_model/analysis_goal_aligned/analysis_metadata.json

recorded/outputs/world_model/analysis_goal_aligned/control_analysis.json

recorded/outputs/world_model/analysis_goal_aligned/controller_method_metrics.csv

recorded/outputs/world_model/analysis_goal_aligned/controller_method_paired_differences.csv

recorded/outputs/world_model/analysis_goal_aligned/controller_metrics.csv

recorded/outputs/world_model/analysis_goal_aligned/controller_paired_differences.csv

recorded/outputs/world_model/analysis_goal_aligned/executed_forecast_analysis.json

recorded/outputs/world_model/analysis_goal_aligned/world_model_analysis.json

recorded/outputs/world_model/analysis_goal_aligned/world_model_calibration.csv

recorded/outputs/world_model/analysis_goal_aligned/world_model_paired_modality.csv

recorded/outputs/world_model/analysis_goal_aligned/world_model_run_metrics.csv

recorded/outputs/world_model/executed_forecasts/metrics.csv

recorded/outputs/world_model/executed_forecasts/provenance.json

recorded/outputs/world_model/executed_forecasts_goal_aligned/metrics.csv

recorded/outputs/world_model/executed_forecasts_goal_aligned/provenance.json

recorded/outputs/world_model/rl/batch_complete.json

recorded/outputs/world_model/rl/vision_seed0/run_metadata.json

recorded/outputs/world_model/rl/vision_seed0/training_log.csv

recorded/outputs/world_model/rl/vision_seed1/run_metadata.json

recorded/outputs/world_model/rl/vision_seed1/training_log.csv

recorded/outputs/world_model/rl/vision_seed2/run_metadata.json

recorded/outputs/world_model/rl/vision_seed2/training_log.csv

recorded/outputs/world_model/rl/visuotactile_margin_seed0/run_metadata.json

recorded/outputs/world_model/rl/visuotactile_margin_seed0/training_log.csv

recorded/outputs/world_model/rl/visuotactile_margin_seed1/run_metadata.json

recorded/outputs/world_model/rl/visuotactile_margin_seed1/training_log.csv

recorded/outputs/world_model/rl/visuotactile_margin_seed2/run_metadata.json

recorded/outputs/world_model/rl/visuotactile_margin_seed2/training_log.csv

recorded/outputs/world_model/rl/visuotactile_seed0/run_metadata.json

recorded/outputs/world_model/rl/visuotactile_seed0/training_log.csv

recorded/outputs/world_model/rl/visuotactile_seed1/run_metadata.json

recorded/outputs/world_model/rl/visuotactile_seed1/training_log.csv

recorded/outputs/world_model/rl/visuotactile_seed2/run_metadata.json

recorded/outputs/world_model/rl/visuotactile_seed2/training_log.csv

recorded/outputs/world_model/rl_goal_aligned/batch_complete.json

recorded/outputs/world_model/rl_goal_aligned/fixed_plan.json

recorded/outputs/world_model/rl_goal_aligned/initialization_audit.json

recorded/outputs/world_model/rl_goal_aligned/visuotactile_margin_seed0/run_metadata.json

recorded/outputs/world_model/rl_goal_aligned/visuotactile_margin_seed0/training_log.csv

recorded/outputs/world_model/rl_goal_aligned/visuotactile_margin_seed1/run_metadata.json

recorded/outputs/world_model/rl_goal_aligned/visuotactile_margin_seed1/training_log.csv

recorded/outputs/world_model/rl_goal_aligned/visuotactile_margin_seed2/run_metadata.json

recorded/outputs/world_model/rl_goal_aligned/visuotactile_margin_seed2/training_log.csv

recorded/outputs/world_model/rl_goal_aligned/visuotactile_seed0/run_metadata.json

recorded/outputs/world_model/rl_goal_aligned/visuotactile_seed0/training_log.csv

recorded/outputs/world_model/rl_goal_aligned/visuotactile_seed1/run_metadata.json

recorded/outputs/world_model/rl_goal_aligned/visuotactile_seed1/training_log.csv

recorded/outputs/world_model/rl_goal_aligned/visuotactile_seed2/run_metadata.json

recorded/outputs/world_model/rl_goal_aligned/visuotactile_seed2/training_log.csv

recorded/outputs/world_model/runs/all_runs.json

recorded/outputs/world_model/runs/data_manifest.json

recorded/outputs/world_model/runs/normalization.json

recorded/outputs/world_model/runs/vision_seed0/calibration.json

recorded/outputs/world_model/runs/vision_seed0/evaluation.json

recorded/outputs/world_model/runs/vision_seed0/summary.json

recorded/outputs/world_model/runs/vision_seed0/training.jsonl

recorded/outputs/world_model/runs/vision_seed1/calibration.json

recorded/outputs/world_model/runs/vision_seed1/evaluation.json

recorded/outputs/world_model/runs/vision_seed1/summary.json

recorded/outputs/world_model/runs/vision_seed1/training.jsonl

recorded/outputs/world_model/runs/vision_seed2/calibration.json

recorded/outputs/world_model/runs/vision_seed2/evaluation.json

recorded/outputs/world_model/runs/vision_seed2/summary.json

recorded/outputs/world_model/runs/vision_seed2/training.jsonl

recorded/outputs/world_model/runs/visuotactile_seed0/calibration.json

recorded/outputs/world_model/runs/visuotactile_seed0/evaluation.json

recorded/outputs/world_model/runs/visuotactile_seed0/summary.json

recorded/outputs/world_model/runs/visuotactile_seed0/training.jsonl

recorded/outputs/world_model/runs/visuotactile_seed1/calibration.json

recorded/outputs/world_model/runs/visuotactile_seed1/evaluation.json

recorded/outputs/world_model/runs/visuotactile_seed1/summary.json

recorded/outputs/world_model/runs/visuotactile_seed1/training.jsonl

recorded/outputs/world_model/runs/visuotactile_seed2/calibration.json

recorded/outputs/world_model/runs/visuotactile_seed2/evaluation.json

recorded/outputs/world_model/runs/visuotactile_seed2/summary.json

recorded/outputs/world_model/runs/visuotactile_seed2/training.jsonl

verify_package.py

(201 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.