Windows: the game starts from a folder named in any language (UTF-8 as its code page) #1191

Merged

hckrman101 merged 1 commit from windows-utf8-paths into main

2026-11-09 16:58:09 +00:00

hckrman101 commented

2026-11-09 16:55:36 +00:00

(Migrated from github.com)

Cause: host/ and gpu/vk keep paths as char strings and call the A functions with them (GetModuleFileNameA, CreateFileA, GetFileAttributesA, MoveFileExA, CreateDirectoryA, SetCurrentDirectoryA, fopen/freopen, about 180 call sites). Without a manifest those use the system's ANSI code page (1252 on a Western Windows), so a name that code page cannot spell comes back from GetModuleFileNameA with ? in it. The game then cannot find image.bin, cannot open its log beside itself, and quits with exit code 2 and nothing on screen. The code already treats its char strings as UTF-8 everywhere it converts them (hos_update.c, thread names, every MultiByteToWideChar/WideCharToMultiByte is CP_UTF8), so the A calls were the one part that didn't match.

A Windows tester confirmed exactly this: from a folder named Hookshot-DLSS ゼルダ on English Windows, the game read its own path as Hookshot-DLSS ???\hookshot.exe, printed its usage line and exited with code 2. In the same tester's hookshot_log.txt, the fs: mounted ... lines (host paths from GetModuleFileNameA) showed é as � (ANSI 1252 bytes), while the other lines were UTF-8.

The fix: hookshot.exe now has an application manifest with <activeCodePage>UTF-8</activeCodePage> (nx2/win_manifest.py writes it as a .res in pure Python, and nx2/build.py links it in beside the icon's .res, so no rc/windres/mt is needed with either llvm-mingw or the MSVC target). From Windows 10 1903 on, this makes the process's code page UTF-8: every A function, the UCRT's fopen/_open/stat/getenv and argv take and give UTF-8, in hookshot.exe and in every DLL it loads (hookshot_vk.dll, SDL, FFmpeg). One linked resource covers all ~180 call sites, the C runtime and the third-party code, and new code written the same way keeps working. The manifest also says asInvoker (no change: a 64-bit process gets no UAC virtualization anyway) and nothing else, so DPI awareness, comctl version and OS compatibility stay as they were.

Why not convert every call to the W functions: that is ~180 call sites in host/ and gpu/vk, plus every fopen, stat, opendir and getenv on a path, plus wrappers for CRT functions that have no W twin, and any new narrow call would bring the bug back. The manifest is the approach Microsoft recommends for code that keeps UTF-8 char strings, and the codebase already did. What it does not cover is Windows older than 10 1903 (ignored there). The readme already asks for Windows 10 or 11, and 1903 (May 2019) and all older Windows 10 releases are out of support, so for those this PR adds a clear message instead of a silent exit.

Start-up check (host/hos_main.c, narrow_paths_ok): before anything opens a file, the game checks that GetModuleFileNameA's path names the exe. If it doesn't (an older Windows with such a folder name, or a path longer than MAX_PATH bytes), it writes a line to stderr and shows a message box naming the folder and what to do (move it to a plainer, shorter path, or update Windows), then exits. The message box is skipped for --prepare-shaders (the builder's last step, nothing to click) and with HOS_HEADLESS. On a working setup the check costs one GetModuleFileNameA and one GetFileAttributesA.

Command line and logs: under the manifest, GetCommandLineA, and so the C runtime's argv, are UTF-8 too, so main(argc, argv) gets the same text wmain would, without changing how arguments are parsed (--romfs/--user paths and image.bin's path included). hookshot_log.txt is opened with freopen on the UTF-8 path, and every line is UTF-8 now: the fs: lines' host paths included, which were ANSI bytes before (é showed as �), or ?.

Audit notes: the launcher and setup (builder/setup/setup.c) already use only W functions and -municode, so they needed nothing. The Hookshot Builder (.NET) and the build (Python, clang, lld) are Unicode already (and #200 covers the builder children's output). The Vulkan loader is loaded by name. Nothing in host/ or gpu/vk converts with CP_ACP, CP_OEMCP or CP_THREAD_ACP, and nothing calls SetFileApisToOEM. hos_update.c and DLSS's own path code use W functions, which this doesn't change. Linux doesn't change: the manifest is only linked into hookshot.exe, and the check is _WIN32 only.

Known limit: a path is limited to MAX_PATH bytes in UTF-8, so about 85 characters of Japanese (3 bytes each) instead of about 130 with Japanese Windows's CP932 (2 bytes each). The start-up check reports such a path clearly instead of failing silently.

Tests: tests/test_utf8_paths.py (in run_all.py) checks that the .res holds one RT_MANIFEST, resource 1, with activeCodePage UTF-8 and asInvoker; that nx2/build.py links it into hookshot.exe; that main runs narrow_paths_ok before anything else; and that host/ and gpu/vk (not host/posix or the vendored code) convert no text with an ANSI code page. It also builds tests/utf8_paths_test.c (with the real narrow_paths_ok pasted in) for Windows, once with the manifest and once without. On Windows (CI's windows-2022 job, when clang is on PATH) it runs both from a folder named ゲーム Zélda 🎮. With the manifest: GetACP() is 65001, the start-up check passes, a UTF-8/UTF-16 round trip of C:\ゲーム\Zélda 🎮\ (surrogate pair included) is exact, GetModuleFileNameA equals GetModuleFileNameW in UTF-8, argv has Hookshot-DLSS ゼルダ Zélda in UTF-8, a line printed into a freopen'd stderr (as hookshot_log.txt is) with the exe's path holds its UTF-8 bytes, a file made with fopen on the UTF-8 path is the file GetFileAttributesW/DeleteFileW find, and getenv gives UTF-8. Without it (the negative control, skipped on a PC set to "Use Unicode UTF-8 for worldwide language support"): the start-up check refuses the folder, as the game used to fail. On Linux, with HOOKSHOT_MINGW set, the probe is cross-built and llvm-readobj confirms the manifest resource is in it and not in the control.

Verified: the Linux build, and python -I -B tests/run_all.py on Linux (140 passed, 0 failed, 7 skipped; with HOOKSHOT_MINGW set, the probe is cross-built and its manifest checked). A full Windows cross-build of the game with llvm-mingw (UCRT): hookshot.exe and hookshot_vk.dll link, llvm-readobj --coff-resources hookshot.exe shows the RT_MANIFEST (resource 1) with activeCodePage UTF-8, and the start-up message is in the exe. I haven't run it on Windows itself (no Windows or Wine here), so there is a checklist for a tester below.

For a Windows tester (please check on Windows 10 1903+ or 11, on a Western-language Windows without the "Beta: Use Unicode UTF-8" option turned on):

Rename the built game folder (the one with hookshot.exe and image.bin) to something like C:\Games\ゼルダ Zélda 🎮 and start hookshot.exe from it: the game starts and plays as usual.
The case already reported: a folder named Hookshot-DLSS ゼルダ on English Windows. Before, the game read its path as Hookshot-DLSS ???\hookshot.exe, printed its usage line and exited with code 2. Now it starts.
hookshot_log.txt appears in that folder. Opened in Notepad (as UTF-8), every line is readable: the fs: mounted ... -> <path> lines show ゼルダ, é and emoji correctly, with no ? or �.
Saves and settings (user, hookshot_options.ini) are written and read back in that folder: change an option in the Insert menu, quit, start again, and it is kept. Save the game and load it.
Shader/pipeline caches are written in that folder (hookshot_pipelines.bin etc.), and the second start is as fast as from a plain folder.
The update offer (if a newer release is out) and the Builder's rebuild from that folder still work.
Building into such a folder with Hookshot Builder works too (the build's last step runs hookshot.exe --prepare-shaders there).
Optional: with a very long folder path, a message box explains it instead of nothing happening.

On Windows the game did not start at all from a folder whose name has characters outside the system's ANSI code page: Japanese (or Cyrillic, Greek, Chinese...) on a Western-language Windows, or an emoji on any Windows. No window, and no hookshot_log.txt either. Cause: host/ and gpu/vk keep paths as `char` strings and call the A functions with them (GetModuleFileNameA, CreateFileA, GetFileAttributesA, MoveFileExA, CreateDirectoryA, SetCurrentDirectoryA, fopen/freopen, about 180 call sites). Without a manifest those use the system's ANSI code page (1252 on a Western Windows), so a name that code page cannot spell comes back from GetModuleFileNameA with `?` in it. The game then cannot find image.bin, cannot open its log beside itself, and quits with exit code 2 and nothing on screen. The code already treats its `char` strings as UTF-8 everywhere it converts them (hos_update.c, thread names, every MultiByteToWideChar/WideCharToMultiByte is CP_UTF8), so the A calls were the one part that didn't match. A Windows tester confirmed exactly this: from a folder named `Hookshot-DLSS ゼルダ` on English Windows, the game read its own path as `Hookshot-DLSS ???\hookshot.exe`, printed its usage line and exited with code 2. In the same tester's hookshot_log.txt, the `fs: mounted ...` lines (host paths from GetModuleFileNameA) showed `é` as `�` (ANSI 1252 bytes), while the other lines were UTF-8. The fix: hookshot.exe now has an application manifest with `<activeCodePage>UTF-8</activeCodePage>` (nx2/win_manifest.py writes it as a .res in pure Python, and nx2/build.py links it in beside the icon's .res, so no rc/windres/mt is needed with either llvm-mingw or the MSVC target). From Windows 10 1903 on, this makes the process's code page UTF-8: every A function, the UCRT's fopen/_open/stat/getenv and argv take and give UTF-8, in hookshot.exe and in every DLL it loads (hookshot_vk.dll, SDL, FFmpeg). One linked resource covers all ~180 call sites, the C runtime and the third-party code, and new code written the same way keeps working. The manifest also says `asInvoker` (no change: a 64-bit process gets no UAC virtualization anyway) and nothing else, so DPI awareness, comctl version and OS compatibility stay as they were. Why not convert every call to the W functions: that is ~180 call sites in host/ and gpu/vk, plus every fopen, stat, opendir and getenv on a path, plus wrappers for CRT functions that have no W twin, and any new narrow call would bring the bug back. The manifest is the approach Microsoft recommends for code that keeps UTF-8 `char` strings, and the codebase already did. What it does not cover is Windows older than 10 1903 (ignored there). The readme already asks for Windows 10 or 11, and 1903 (May 2019) and all older Windows 10 releases are out of support, so for those this PR adds a clear message instead of a silent exit. Start-up check (host/hos_main.c, narrow_paths_ok): before anything opens a file, the game checks that GetModuleFileNameA's path names the exe. If it doesn't (an older Windows with such a folder name, or a path longer than MAX_PATH bytes), it writes a line to stderr and shows a message box naming the folder and what to do (move it to a plainer, shorter path, or update Windows), then exits. The message box is skipped for `--prepare-shaders` (the builder's last step, nothing to click) and with HOS_HEADLESS. On a working setup the check costs one GetModuleFileNameA and one GetFileAttributesA. Command line and logs: under the manifest, GetCommandLineA, and so the C runtime's `argv`, are UTF-8 too, so `main(argc, argv)` gets the same text `wmain` would, without changing how arguments are parsed (`--romfs`/`--user` paths and image.bin's path included). hookshot_log.txt is opened with freopen on the UTF-8 path, and every line is UTF-8 now: the `fs:` lines' host paths included, which were ANSI bytes before (`é` showed as `�`), or `?`. Audit notes: the launcher and setup (builder/setup/setup.c) already use only W functions and -municode, so they needed nothing. The Hookshot Builder (.NET) and the build (Python, clang, lld) are Unicode already (and #200 covers the builder children's output). The Vulkan loader is loaded by name. Nothing in host/ or gpu/vk converts with CP_ACP, CP_OEMCP or CP_THREAD_ACP, and nothing calls SetFileApisToOEM. hos_update.c and DLSS's own path code use W functions, which this doesn't change. Linux doesn't change: the manifest is only linked into hookshot.exe, and the check is `_WIN32` only. Known limit: a path is limited to MAX_PATH bytes in UTF-8, so about 85 characters of Japanese (3 bytes each) instead of about 130 with Japanese Windows's CP932 (2 bytes each). The start-up check reports such a path clearly instead of failing silently. Tests: tests/test_utf8_paths.py (in run_all.py) checks that the .res holds one RT_MANIFEST, resource 1, with activeCodePage UTF-8 and asInvoker; that nx2/build.py links it into hookshot.exe; that main runs narrow_paths_ok before anything else; and that host/ and gpu/vk (not host/posix or the vendored code) convert no text with an ANSI code page. It also builds tests/utf8_paths_test.c (with the real narrow_paths_ok pasted in) for Windows, once with the manifest and once without. On Windows (CI's windows-2022 job, when clang is on PATH) it runs both from a folder named `ゲーム Zélda 🎮`. With the manifest: GetACP() is 65001, the start-up check passes, a UTF-8/UTF-16 round trip of `C:\ゲーム\Zélda 🎮\` (surrogate pair included) is exact, GetModuleFileNameA equals GetModuleFileNameW in UTF-8, `argv` has `Hookshot-DLSS ゼルダ Zélda` in UTF-8, a line printed into a freopen'd stderr (as hookshot_log.txt is) with the exe's path holds its UTF-8 bytes, a file made with fopen on the UTF-8 path is the file GetFileAttributesW/DeleteFileW find, and getenv gives UTF-8. Without it (the negative control, skipped on a PC set to "Use Unicode UTF-8 for worldwide language support"): the start-up check refuses the folder, as the game used to fail. On Linux, with HOOKSHOT_MINGW set, the probe is cross-built and llvm-readobj confirms the manifest resource is in it and not in the control. Verified: the Linux build, and `python -I -B tests/run_all.py` on Linux (140 passed, 0 failed, 7 skipped; with HOOKSHOT_MINGW set, the probe is cross-built and its manifest checked). A full Windows cross-build of the game with llvm-mingw (UCRT): hookshot.exe and hookshot_vk.dll link, `llvm-readobj --coff-resources hookshot.exe` shows the RT_MANIFEST (resource 1) with activeCodePage UTF-8, and the start-up message is in the exe. I haven't run it on Windows itself (no Windows or Wine here), so there is a checklist for a tester below. For a Windows tester (please check on Windows 10 1903+ or 11, on a Western-language Windows without the "Beta: Use Unicode UTF-8" option turned on): - [ ] Rename the built game folder (the one with hookshot.exe and image.bin) to something like `C:\Games\ゼルダ Zélda 🎮` and start hookshot.exe from it: the game starts and plays as usual. - [ ] The case already reported: a folder named `Hookshot-DLSS ゼルダ` on English Windows. Before, the game read its path as `Hookshot-DLSS ???\hookshot.exe`, printed its usage line and exited with code 2. Now it starts. - [ ] hookshot_log.txt appears in that folder. Opened in Notepad (as UTF-8), every line is readable: the `fs: mounted ... -> <path>` lines show `ゼルダ`, `é` and emoji correctly, with no `?` or `�`. - [ ] Saves and settings (user\, hookshot_options.ini) are written and read back in that folder: change an option in the Insert menu, quit, start again, and it is kept. Save the game and load it. - [ ] Shader/pipeline caches are written in that folder (hookshot_pipelines.bin etc.), and the second start is as fast as from a plain folder. - [ ] The update offer (if a newer release is out) and the Builder's rebuild from that folder still work. - [ ] Building into such a folder with Hookshot Builder works too (the build's last step runs `hookshot.exe --prepare-shaders` there). - [ ] Optional: with a very long folder path, a message box explains it instead of nothing happening.

Commenting is not possible because the repository is archived.

No reviewers