AI agents just gained camera, lidar and radar as callable tools inside creative and simulation apps.
NVIDIA added Omniverse libraries to its Agent Toolkit. The libraries expose RTX sensor simulation, GPU-accelerated physics and asset validation as tools agents can call. They are available on GitHub and let agents inspect scenes, run validation workflows and prepare 3D content for simulation.
At SIGGRAPH NVIDIA described a local "super agent" configuration running on a DGX Station. In that setup Nemotron 3 Ultra, NemoClaw agent blueprints and the Omniverse libraries run together in a secure runtime with no internet required. NVIDIA frames the stack as a way for designers and engineers to build and run domain-specific agents that connect directly into creative apps via Model Context Protocol, which opens application context such as scenes and timelines to agents.
This is a capability and infrastructure move at once. Functionally, agents can now reason about three-dimensional worlds rather than only text or 2D assets. That matters for robotics, autonomous systems and digital twins, where sensor-level realism and physics matter for training and validation. Practically, putting these libraries on GitHub lowers the friction for teams that want agents to prepare simulation-ready worlds: automatic asset checks, CAD-to-OpenUSD conversions and simulated camera or lidar outputs become callable skills.
Equally important is the DGX Station demonstration. A secure, offline runtime addresses cases where data cannot leave a facility, such as regulated industrial design or pre-deployment testing for robots and vehicles. Running Nemotron 3 Ultra and agent blueprints locally shows heavier models can be paired with tooling to create closed-loop agent workflows on-premises.
There are limits. High-end compute like a DGX Station remains costly. Application vendors must adopt the Model Context Protocol for agents to operate inside the most widely used creative tools. Integrating these pieces into an engineering workflow will take work from toolmakers and teams that build domain-specific agents.
If those pieces align, the immediate effect will be faster, more automated paths from design to validated simulation. For businesses that need repeatable, offline simulation or realistic sensor data, this is the most concrete step yet toward operational physical AI.
Expect more demonstrations of local agent stacks and more creative apps exposing context to agents. That will determine whether this becomes a niche capability or a standard part of simulation and design pipelines.