fix(svc): svcQueryMemory answers a free range as the whole gap, Inaccessible past the address space #1683
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1683
Loading…
Reference in a new issue
No description provided.
Delete branch "rainy/Longshot:fix/svc-query-memory-free-gap"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Linux (Omarchy, kernel 7.2.15; Intel Core Ultra 7 270K Plus; RTX 4090, driver 615.78.08) · Needs: a run on Windows, any GPU
TL;DR
When a program asked the console which memory sits at a free address, Longshot answered with one page. The real console answers with the whole free range around it, and past the end of the address space it answers with one inaccessible block. This PR answers the way the console does. A new game-free test fails on main and passes here. The full check suite on this branch has the same 40 failures as main, and one more pass. It also ran in the game on Linux.
Breakdown
python -I -B tests/run_all.py --jobs 8, which has the same 40 failures as main and one more pass.Why
A code mod, sloperinaGZ 0.0.8, on snaildev99's mods/module-gaps branch is what showed this. The kernel answers a memory query with the memory block that holds the address. The kernel merges adjacent free blocks, so a free address gets the whole free range. Past the address space it answers one Inaccessible block from the space's end to 2^64. Atmosphère's kernel reimplementation does this in
KPageTableBase::QueryInfoandQueryInfoImplin mesosphere'skern_k_page_table_base.cpp. Longshot answered a free address with its one page. Code that lists the mappings walks from address 0, stepping to base + size each time. With one page per step it needs about 16,000 steps per 64 MiB of free space. A walk with a step limit then stops in the free space after the system region and never reaches the image or the heap. The game itself calls this from 2 places. The tracker has no issue or PR for QueryMemory.How to reproduce
Seen on: Linux (Omarchy, kernel 7.2.15; Intel Core Ultra 7 270K Plus; RTX 4090, driver 615.78.08). The answer itself does not depend on the GPU or driver.
Settings and switches: defaults
Steps: call svcQueryMemory on a free address, for example 0x07000000 just above the system region, from mod code or from the game-free test below run against main.
Expected: Free, base 0x07000000, size up to the start of the next mapping, the image.
Actual: Free, base 0x07000000, size 0x1000.
What changed
query_memoryanswers free memory with the whole gap and answers addresses past the address space as Inaccessible. Mapped regions answer as before.query_memoryinhost/hos_svc.c. An address at or aboveHOS_GUEST_SPACE, which is 1 << 39, gets baseHOS_GUEST_SPACE, size0 - HOS_GUEST_SPACE, state 0x10. A free address gets the range from the end of the nearest mapping below it to the start of the nearest mapping above it. The mappings are the system region, the image, the heap and the calling thread's stack.tests/test_query_memory_free.pycutsquery_memoryout of the source withtests/srccut.pyand compiles it with a stub layout. It checks each gap from several addresses, the Inaccessible end, the four mapped regions, a walk from 0 that has to end in a few steps, a stack inside the heap and an empty heap.Measurements
Nothing to measure. The change replaces one answer with another in a call the game makes from 2 places, and a timing would not show it.
Flags and settings
None
Testing
Tested:
python -I -B tests/run_all.py --jobs 8on Linux (Omarchy, kernel 7.2.15; Intel Core Ultra 7 270K Plus; RTX 4090, driver 615.78.08). main @2f1c0c9c: 685 passed, 40 failed, 97 skipped. this branch @2390b608: 686 passed, 40 failed, 97 skipped, 0 new failures compared by test name. The 40 failures also fail on main. They are:Tests that also fail on main @
2f1c0c9cin this environmenttests/test_bug_report_defaults.pytests/test_bug_report.pytests/test_capture_read_once.pytests/test_crash_groups.pytests/test_crash_record.pytests/test_crt_options.pytests/test_damage_debug_menu.pytests/test_dev_packs.pytests/test_display_startup_failure.pytests/test_dlss_feature_size.pytests/test_failed_compile_keeps_objects.pytests/test_guest_imported.pytests/test_hardware_report.pytests/test_high_fps_fixes.pytests/test_host_guest_writes.pytests/test_hud_layout_row.pytests/test_invalidate_quick.pytests/test_libc_hooks.pytests/test_link_path_comma.pytests/test_linux_preflight.pytests/test_linux_pressure.pytests/test_linux_update.pytests/test_log_open_failure.pytests/test_missing_nup.pytests/test_mods_size_table.pytests/test_pending_guest_import.pytests/test_pipeline_play_cache.pytests/test_prepare_cache_save.pytests/test_present_thread.pytests/test_profiling_mode.pytests/test_psychov35_fsr3_fg.pytests/test_readback_timeout_source.pytests/test_run_all_changed.pytests/test_slow_frames.pytests/test_song_note_cue.pytests/test_switch_table.pytests/test_upscaler_rows.pytests/test_vcore_escape.pytests/test_walk_key.pytests/test_vbytes.py(timed out at 600 s)With the fix's source file reverted to main,
python -I -B tests/test_query_memory_free.pyfails (FAIL:54 qb == 0 && qs == 0x01000000 ...). It passes on this branch. In the game, built from this branch on Linux with an RTX 4090: the game boots and plays as on main.Not testable by hand: the free-gap walk that showed the bug can only be reached by a code mod. The game's own 2 call sites run in a normal play session, so the in-game run covers them. The game-free test stands in for the free-gap case.
Not tested: Windows (no build or run there) and other GPUs. The same change also ran in the game on snaildev99's mods/module-gaps branch, where a mod's memory walk reached the heap.
Must be tested:
svclines inlongshot_log.txt(by hand: the author, before this PR leaves WIP)Risks
What could break: the game's 2 own call sites now see a large free range where they saw one page. A console answers them that way, so a problem would point to something else Longshot does differently.
Turning it off: revert the commit
Compatibility: Nothing changes
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.Tested in game: Linux (Omarchy, kernel 7.2.15), Intel Core Ultra 7 270K Plus, RTX 4090 driver 615.78.08, a build of this branch on v3.0.0-rc.8: the game boots and plays as on main.
From the code and the evidence this looks ready to merge. The new answers match the kernel: mesosphere's
KPageTableBase::QueryInfogives one Inaccessible block from the end of the address space to 2^64, and a merged free block for a free address. The loop inquery_memorygets each gap right. Regions with nothing in them are skipped, nested regions are handled (a stack inside the system region or the heap), and the size is never 0.What the game sees. I read the game's own callers of svcQueryMemory, without running anything. The svc has two wrappers. The one in rtld never runs, because Longshot does rtld's work itself. The one in the SDK is called from about 25 places. These are the ones a normal session reaches:
nnosInitialize, the program-argument check. It queries x1, whichhos_main.cpasses as 0, and reads only the state ("is it Code?"). It got Free before and gets Free now, so nothing changes.ThreadManagerImplByHorizonconstructor for the main thread. It queries its own frame, which falls in the calling thread's stack branch. That branch is unchanged.nn::ro::FindModuleHeader,ProtectFullRelroForAllModulesandnn::diag's backtrace query addresses in the image or on a stack. Those branches are unchanged.SetMemoryPermissionImpl,SetMemoryAttributeImplandIsMemoryLocked(throughExamineWholeAddressSpaceHasNoAttribute) step block by block. Over free memory they now take one step where they took one per page. svc 2 and 3 are no-ops here and the attribute is always 0, so the result is the same.UserExceptionHandlerandro::GetExceptionInforun only after a guest exception.The one caller whose result really changes is
AddressSpaceAllocator::CheckFreeSpace, which asks whether[addr, addr+size)lies inside one Free block. On main it said no to anything larger than a page. It now says yes. Its users areStackGuardManager::MapAtRandomAddress(alias stacks), the shared and transfer memoryMapandRemoteRegionManager. They are not reached in play: theCreateThreadHLE avoidsMapAliasStack, and every one of them would next call svcMapMemory, svcMapSharedMemory or svcMapTransferMemory. Longshot implements none of those, anda64_svc's default case stops the run. On main they fail with no address space instead. So nothing in play changes, which fits the in-game run in the comments.Suggestions (not blocking)
host/hos_svc.c:341: the gap only knows about the four regions. Memory Longshot hands out elsewhere still answers Free, and now as part of one large Free block: the address regions fromhos_region_alloc(AllocateAddressRegion/AllocateMemoryPages, which grow down from 512 GB) and the VAMM pieces mapped with MapPhysicalMemory. On a console these are Normal. A mod walking the map, the case this PR is for, will skip the game's page-allocated memory. ACheckFreeSpace-style caller would also be told a live range is free. This was wrong before too (one Free page), and nothing in play hits it today. Two ways to settle it: say so in the comment, or answerhos_region_alloc's live ranges as Normal in a follow-up.area/rendererlabel doesn't fit. This is host code.Measurements: none, and none are needed. The change replaces the value of one answer and costs nothing per frame. The fix is shown by the new game-free test: the author reports it fails on main (
FAIL:54) and passes here. Nothing was run for this review. The author ran the game on Linux with an RTX 4090. The answer doesn't depend on the GPU or the OS, and the code is plain portable C. The Windows run in Must be tested is a formality, not a gap.Overlaps: PR 1542 also edits
host/hos_svc.c, but at the start ofa64_svcrather than inquery_memory. They don't conflict, and either can merge first.@ -341,0 +341,4 @@else { /* Free: the whole gap, from the end of the region before addr to the start of the one after it, as the kernelanswers (free blocks are merged). A walk over the address space region by region, base + size each step,reaches every mapping in a few steps; at one page a step it never got past the system region */const uint64_t lo[] = {HOS_SYSTEM_BASE, hos_image_base, HOS_HEAP_BASE, stack};Nit: these four are all
query_memoryknows about. The ranges fromhos_region_alloc(AllocateAddressRegion/AllocateMemoryPages, growing down from 512 GB) and the VAMM pieces from MapPhysicalMemory fall inside the gap after the heap, so they answer as part of one Free block. On a console they are Normal. Nothing in play depends on this today, but it's worth a word in the comment: a mod's map walk will skip that memory, andAddressSpaceAllocator::CheckFreeSpacewould call a live range free.