Game loading and saved state restoring takes about 4 minutes

[golcan]

Hi mates,

Very interesting topic.

The question then is - why does the caching utilize only about 25% of my CPU cores/threads? In comparison, when I run some test benchmark tools with full multicore support, I see all my CPU cores and threads working at 100%.

... So, clearly ETS2 caching is doing something not efficient enough. Maybe it spawns threads from time to time and there are no 4 threads running during all the caching process. Or maybe it suffers from the infamous "false sharing" issue and needs some paddings for its datastructures. I'm just guessing here, multithreading is tricky and sometimes it can go wrong.

... jut note that running four independent tasks with local data only is bit different than run four tasks, that share and access some common data ..

Exactly, how cpu is consumed by threads depends on what threads do and other factors (including OS resource management subsystems).
The fact that a given thread is not consuming cpu all the time doesn't mean that a problem is present in the code, unless one find clear information that proves the problem.

... the 2 minute task you found is preparation of trade routes based on map topology.

Please Max, if possible, what type of graphs and what variants of Dijkstra's algorithm are you using for trade routes calculations?
AFAIK, early versions of Dijkstra's algorithm spend a lot of time sorting nodes. Time is exponential when lists are used, but tends to be more linear for other modified versions (for example, when binary heap or Fibonacci heap are used).
Regarding the Thorup's Linear Time Algorithm, do you think that some variant of it could be used as replacement for Dijkstra's-based algorithms in the future? (More info here, including interesting performance comparisons - slides 122 onward).
Thank you very much in advance.

Kind regards.

[midix]

Thanks, just tried 1.3 - wow, it was blazing fast, only about 15 seconds from desktop to driving.
Of course, there's not much sense to compare caching algorithms without mods and large DLCs.

So, now I have a tough choice - keep v1.3 and enjoy fast loading without any mod goodness until my PC upgrade (or maybe SCS will invent some new trick in next updates) or return to 1.26 and train my patience and learn how to drive more carefully to avoid 4 minutes restoring times from saved state.

[Max]

we made some minor experiments here.
we cannot use hyperthreading (it is useful if you have two threads that wait to another one occasionally, ours are performing heavy computing non-stop), but we maybe could bit better utilize cores as main caching thread spend some time waiting on one core to collect data from all searching threads. instead of waiting it could do work for one of them and as a whole thus be a bit faster. its not much difference on many cores (few percents), but if you have just two cores it can help you in measurable way (we made some simulations).

it wont be early as current update is in finishing state and some other changes on the way, but eventually ...

[Komat]

what variants of Dijkstra's algorithm are you using for trade routes calculations?

We are using binary heap for sorting.

Regarding the Thorup's Linear Time Algorithm, do you think that some variant of it could be used as replacement for Dijkstra's-based algorithms in the future?

That algorithm assumes undirected graph while we are using a directed one.

[Reinhard]

So, now I have a tough choice - keep v1.3 and enjoy fast loading without any mod goodness until my PC upgrade (or maybe SCS will invent some new trick in next updates) or return to 1.26 and train my patience and learn how to drive more carefully to avoid 4 minutes restoring times from saved state.

I think the choice is easy. There were so many improvements since 1.3. Twenty-three major patches are a lot. You would have a tough time to find mods for this old version. Especially map mods. There was nothing comparable then. Too early.

If you are willing to sacrifice the map-mods anyway, the alternative, an up-to-date game without map mods, is far more attractive.

So the real question is: patience or massive map-mods.

Dijkstra? Gosh am I grateful that I can call the legacy of such dead guys "an other's people problem", now. No more headaches for me.

[golcan]

Hi Komat,

Thank you very much for your quick answer.
A version of Thorup's algorithm for planar digraphs is also available, but I guess you are not using a planar digraph for trade routes calculation.

Kind regards.

[Cadde]

Please correct me if i am wrong, i might not have caught on.

But can't you run that cache once and save it in binary format so it just loads the result of the last run?
And as the map checksum (with mods) changes, you recalculate it and save that?

Wouldn't loading times improve immensely the next time the game is started then?

[Komat]

But can't you run that cache once and save it in binary format so it just loads the result of the last run?
And as the map checksum (with mods) changes, you recalculate it and save that?

We are planing to eventually add that. I just was not a high priority as the operation was reasonably fast with the official content in relation to how frequently it should happen (my work PC with 5 year old CPU generates navigation for entire ETS2 map under 6 seconds).