eveofdarkness.org

Your mobs are sinking into hillsides: OpenDAoC navmeshes, and building them without Windows

If you run an OpenDAoC server and your mobs sink into hillsides, float above them, climb into the sky on a slope, or walk straight through buildings, your server probably has no navmeshes. Ours didn't, and nothing told us.

This post covers four things:

  1. how we noticed, and why your server won't warn you;
  2. what the Windows-only tool that builds navmeshes actually runs;
  3. building the same navmeshes on Linux and macOS, and what that turned up;
  4. the result on our server, and how to do it on yours.

If you only want to know whether your server is affected, the check in part 1 takes a minute.

1. Mobs in the hillsides

It started with roaming mobs on our test server. On hills, mobs sank into the slope or floated above it. Some climbed into the air. A mob would also leave its spawn, walk in a straight line through a house, and come out the other side.

The cause is in how OpenDAoC moves NPCs. When a mob roams, the server picks a random point near its spawn and walks it there. With a navmesh, that point comes from the mesh: somewhere a mob can actually stand, and reachable from where it is. Without one, the server picks a random point in a circle at the spawn's height and walks straight to it. There is no ground data at all. Walls, hills and floors don't exist as far as the server is concerned.

A navmesh is a map of where things can walk, built from the game's own zone geometry. OpenDAoC uses Recast and Detour, the same library many commercial games use, and loads one file per zone: zoneNNN.nav.

Navmeshes are optional. OpenDAoC documents them, but a server starts and runs fine without them, and that is the trap:

  • No native library. The server's C# code talks to a native Detour library (libDetour.so on Linux, Detour.dll on Windows). If it's missing, you get one DllNotFoundException among the startup noise, and pathing is off.
  • No .nav file for a zone. This is logged at Debug level only. At the default log level you see nothing.
  • An old .nav file. Files built before September 2025 use 32-bit tile references. Current servers use 64-bit ones. The server logs "Loading NavMesh successful" and then loads no tiles at all. We tested an older file: 0 of 145 tiles loaded, and a "successful" in the log.

So a server can run for years with no pathing, and a server whose log says the meshes loaded can still have empty ones.

Check your server

From the server's folder (the one with CoreServer.dll):

ls lib/ | grep -i detour      # you need libDetour.so (Linux) or Detour.dll (Windows), not only Detour.Managed.dll
ls navmesh/ pathing/ 2>/dev/null | head   # one zoneNNN.nav per zone
grep "Loading NavMesh" <your server log> | head

If you have .nav files, check them with navcheck.py from our repository. It needs only Python 3:

python3 navcheck.py pathing/*.nav

It reports the tile count, the agent settings, and 32-bit tile refs for the old files that load nothing.

2. What's inside BuildNav's RecastDemo.exe

The navmeshes are built with OpenDAoC-BuildNav, which grew out of Uthgard's buildnav. It reads the game client's zone files and works in two stages:

  1. C# code reads terrain, models, cities, dungeons, doors and ladders, and writes each zone's geometry to a file.
  2. A bundled Windows program, RecastDemo.exe, turns that geometry into the .nav file.

Stage 1 is source code. Stage 2 is a binary. The README says it's "slightly tweaked" and that the build parameters are "currently hardcoded and not configurable". The tweaked source isn't published. That one binary is why building navmeshes needs Windows, and why nobody else can reproduce or check them.

Our server runs on Linux. We wanted to build the meshes there, from the same retail client our players use (1.127/1.130), without running an unexplained binary. So we looked inside it, without running it, using a disassembler.

What it is. RecastDemo.exe is the demo application that ships with recastnavigation, compiled for Windows. There's no BuildNav code in it. Its debug-symbol path points at a local recastnavigation checkout.

The settings. Recast's demo keeps its build settings in one function that stores them into consecutive fields. In the binary, that's a run of constants, in the same order as the upstream source:

Setting Upstream default BuildNav's exe
cell size / height 0.3 / 0.2 0.1 / 0.1
agent height / radius 2.0 / 0.6 1.8 / 0.3
max slope 45° 80°
region merge size 20 96
detail sample distance 6 4
tile size 32 96
max off-mesh links 256 4096

Then come the "large world" changes, which let a zone have far more tiles than upstream allows, and one code change: polygons in BuildNav's "jump" area (the pads at the ends of ladders) get the JUMP flag. Upstream leaves it unset.

We got one thing wrong first. Our first reading took the tile size from the wrong constructor. The demo has several sample types, and one of them uses 48. With 48, outdoor zones would hit upstream's tile limit and silently lose most of their rows. That would have been a big finding, and it was wrong. We asked for one test run of the real exe on Windows, on a large flat test floor, and its output showed tile size 96 and no missing tiles. That run is why everything below was checked against Windows output, not reasoned from the disassembly.

Which version. We rebuilt the demo from recastnavigation's last release (v1.6.0) with these settings and compared it with the exe on a real zone, Camelot Hills. 935 of 45,796 tiles were different. We traced each difference back into the disassembly:

  • Most of them (871 tiles) came from the library version. The exe isn't built from a release but from upstream's development branch of spring 2025. We dated it by matching the strings in the binary against every revision of the demo's source: they match only commits from late April to early May 2025. That version includes a later fix to how Recast marks the edges of its detail meshes.
  • 46 tiles: the JUMP flag above.
  • 18 tiles: the exe was compiled with Microsoft's /fp:fast, upstream's own setting for the Windows demo. It lets the compiler rewrite floating-point arithmetic. In three places in Recast's detail-mesh code, it turned a / b into a * (1 / b) or the reverse. The results differ in the last bit, and at tile borders that decides whether there's one more sample point or one fewer. We wrote those three expressions the way the compiler did.

There was one more difference, found earlier on small test scenes: sort order. Recast sorts with the C library's qsort, and some of its comparisons have ties. Every C library breaks ties differently, so a Mac build put a few equal entries of each tile's search index in a different order than Windows. We added a qsort that orders ties the way Microsoft's runtime does.

With those changes, our build of upstream recastnavigation gives byte-for-byte the same tiles as BuildNav's RecastDemo.exe. The 33 MB Camelot Hills file is identical apart from 4 padding bytes per tile, which the exe leaves uninitialised and the server never reads. We then compared five more zones covering every kind BuildNav exports: an outdoor zone in Midgard, Jordheim, a sky city, and a large and a small dungeon. All five were identical with no further changes.

The full record, with addresses and the instructions behind each setting, is in recastdemo-settings.md.

3. Building them on Linux and macOS

The replacement for RecastDemo.exe is a small command-line program, gset2nav. It's upstream recastnavigation, pinned to the exe's version, with the changes above. BuildNav's C# stage needed less: it was tied to Windows by a few Windows Forms calls, a Windows-only image library used for doors, and a copied OpenTK.dll. We moved it to plain .NET 10, ported the one image function it used, and took OpenTK from NuGet.

Moving a Windows tool to other systems finds the assumptions it was quietly making. These are the ones that changed the meshes:

  • Upper and lower case. Windows doesn't care about case in file names; Linux does. BuildNav lower-cases paths, and the client's folders are named Nifs, Dnifs and TerrainTex. So the port resolves each path against real directory listings. Our first full build read the client from a Windows share on a Mac, whose answers about wrong-case names turned out to be inconsistent: the same folder was found under one spelling and not under another. That run lost geometry in 125 zones without stopping: missing dungeon rooms, missing buildings. The build completed, every file loaded, and the meshes were wrong. We now resolve every path from directory listings only.
  • Directory order. Windows lists files in sorted order. Linux file systems list them in whatever order they store them. BuildNav adds a city's buildings in listing order, and Recast's output depends on the order of its triangles. So three capital cities came out slightly differently on Linux. The port now sorts listings the way Windows does.
  • Fused multiply-add. Apple Silicon compilers fuse a * b + c into one instruction by default, which rounds differently. That changed one ladder endpoint by a tenth of a unit in one zone. Both tools are now built with fusing turned off.

And two that didn't change the output but broke the build or the run: a compile error with GCC's standard library, and a crash when BuildNav is started with nice. It tries to lower its own priority, and under nice that is a raise, which Linux refuses.

The checks. On the six real zones and two test scenes, gset2nav gives the same tiles as the Windows exe on both macOS and Linux. Then we built every zone of the retail client on each:

  • macOS (Apple Silicon, 8 cores, client on a network share): about 65 minutes;
  • Linux (x86-64, 32 cores, client on local disk): 28 minutes.

Both give 448 navmeshes, and after the fixes above the two sets are byte-identical. Every file loads every tile with the server's own loading code.

What we haven't checked: the C# stage against stock BuildNav on Windows. Every comparison with the exe used geometry exported by our port. Exporting one zone with stock BuildNav and comparing it with ours would close that gap.

4. On our server

We installed the native library and all 448 meshes, and restarted.

  • 437 zones loaded a navmesh. Before, none did.
  • Door errors dropped from 220 to 54. OpenDAoC registers doors in the navmesh so mobs path around closed ones, and a door in a zone without a mesh fails to register. Of the 54 left, 6 are in a zone missing from the client's zone list, so it has no mesh. All 54 are keep doors and postern doors. Keeps are placed from the server's database rather than the client's files, so our guess is that there's no mesh under those doors; we haven't looked yet.
  • Trials of Atlantis needed one more step. BuildNav skips 36 Atlantis zones (30–47 and 130–147). The client marks them as "proxies" of zones 73–90: the same areas again, in other regions. The server still looks for their own files. Meshes are stored in region coordinates, and each proxy zone sits at exactly the same position in its region as its source, both in the client's zone list and in the server's zone table. So a copy of the source's mesh is the right mesh. With those, 473 zones load a navmesh.

We then watched mobs in game on the retail 1.130 client. In the Vale of Mularn, hobgoblins walked up a hillside on its surface, neither sinking into the slope nor floating above it. In Jordheim, an ambient rat ran to a rock, climbed up and over it, and went on. Because BuildNav allows slopes up to 80°, small mobs climb boulders instead of walking round them.

Do it on your server

The tools and the full instructions are in the opendaoc-tools repository, under navmesh/. In short:

  1. Build OpenDAoC-Core's native Detour library and put it in the server's lib/.
  2. Build gset2nav and the BuildNav port (CMake, a C++ compiler, the .NET 10 SDK).
  3. Point BuildNav at your game client and run --all. Use the client version your players use.
  4. Check the files with navcheck.py, copy them to navmesh/ or pathing/, and copy the Atlantis proxies.
  5. Restart and look for Loading NavMesh successful for zone N.
  6. Check your spawns with /fixnpcspawn zone, which by default only reports and changes nothing. With a mesh, a roaming mob picks its next point from the mesh near its spawn, searching only 64 units up and down. A spawn further off the mesh than that finds no point, and the mob never roams. Nothing is logged.

OpenDAoC's docs say a server holding every mesh needs at least 5 GB of RAM.

Everything here is GPL-3.0 (the BuildNav port and our tools) or zlib (gset2nav, from recastnavigation). No client files or extracted geometry are in the repository: you build the meshes from your own client.

Thanks to the OpenDAoC team for BuildNav and the pathing code, to the Uthgard team for open-sourcing the original buildnav, and to the recastnavigation authors.