Access to depth maps, custom camera models

Post Reply
Posts: 1
Joined: 16 Feb 2017 11:30

Access to depth maps, custom camera models

#1 Post by rsp1984 » 16 Feb 2017 12:21

Hello SCS Software,

there's a large community of robotics and self-driving car researchers, both academic as well as commercial, who would absolutely LOVE your software if you provided API access to some basic internal state. The way this community would use your software is to simulate the real world so that algorithms can then run on the data and perform inference and/or control.

Example 1: Robotics researchers could perform a run in which the game is played by a human operator and all the data gets recorded on disk. Then the recorded data could be fed into an inference algorithm (e.g. positional tracking) and compared with ground truth to train/tune the algorithm.

Example 2: The game is run live and its output is fed to a self-driving-car algorithm in real-time which then controls the game. The whole run could be recorded and later the control decisions of the algorithm could be evaluated against the ground truth state to improve the algorithm.

What would be needed to enable this is:
- Access to metric depth maps for each frame
- Ability to set and configure more virtual cameras (e.g. a stereo pair looking fwd and one looking bwd) and obtain their output for each frame
- Ability to query ground truth state, e.g. relative vehicle position, speed, etc. as seen from each camera. Positions of other vehicles in the scene.
- A timer mode that allows for fixed simulated time increments, independently of how fast the game renders (therefore making the game non-realtime).
- Ability to record all the data to disk.

Also it would be extremely nice to have the ability to simulate the output of typical sensors other than cameras, e.g. (spinning) LiDAR, Radar, depth cameras, IMUs. On that: it would be perfectly acceptable to just provide an API to query internal geometry so that people can write their own sensor simulators.

Please let me re-emphasize what a huge deal this would be for the robotics community. Because no good tool like this exists today people are using decades old software or spending time and resources to write their own basic stuff, just to have something:

Microsoft recently released their own internal tool ( but it's very basic and would be totally eclipsed by where you could take your simulator with little effort.

Udacity have their own little sim for their course work ( but it's also very basic and lacks even the most fundamental features.

Even Gazebo ( is still used by many although the graphics are way too basic to train an algorithm on it and there's basically no data available.

Here's an article by MIT Technology Review that describes the situation in some more detail: ... deo-games/

All the roads and cities that you have modeled for the Truck Simulator are a real treasure and it'd be a shame to only use them for the game itself.

Please feel free to reach out via email if this sounds interesting. I have a personal interest in the existence of such a tool and would love to contribute if you decide it's worth pursuing.

Post Reply

Return to “Public research”

Who is online

Users browsing this forum: No registered users and 1 guest