WIP: feat(vk): PsychoV35's wide gamut, and the game's AA off as RenoDX's zelda-oot-oz advises #1394
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1394
Loading…
Reference in a new issue
No description provided.
Delete branch "LRF17/Longshot:psychov35-wide-aa"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: Windows 11, RTX 4090 (driver 610.47) · Needs: AMD or Intel (the game's network and the TAAU stand-in); Linux
Why
PsychoV35 fits colours into BT.2020, but the scene is clipped at 0 in BT.709 because the game's AA takes square roots:
every colour outside BT.709 (fire, magic, saturated light) loses its hue, and the most saturated oranges turn yellow.
And the game's AA softens the scene before DLSS or FSR, which anti-alias it again. RenoDX's zelda-oot-oz works its AA
in BT.2020, writes the reconstruction back in BT.709, and has an AA Strength its author advises at 0. Stacked on
#1393 (its commit shows here until that one merges).
What changed
The scene stays signed; the game's AA and its edge analysis read it in BT.2020; whatever upscales the picture writes
it back in BT.709. And with PsychoV35 the game's AA is off, as the mod's author advises: the upscaler anti-aliases the
scene alone.
ir_opt/working_gamut_pass.h: on the AA's two programs (26fe0c70ab9fc547 the edge analysis,5363e57b5646599d the AA, found in the IR as the mod's aa_prepare and aa) the scene texture's reads to BT.2020, at most
0; on the gameplay networks each image write back to BT.709. Both read
kSettingWorkingGamut(1 with PsychoV35):these compute programs are not keyed by the tone mapper, and with Vanilla they are the game's.
vk_shader.cpp: the passes run on those programs after translation (WorkingGamutRoleOf,AaStrengthProgram).shaders/dlss_out.compandshaders/taau_upscale.comp, their RGBA16F (PsychoV35-only) builds, writeBT.709; SPIR-V headers regenerated, the R11G11B10F builds unchanged.
psychov35.h: the scene signed (Exposure 0 still black).ir_opt/aa_strength_pass.hon the AA, after the working gamut: two pixels a thread, each channelthe bits of a square root plus a constant; the root's argument becomes lerp(scene, AA, strength), the scene from the
pixel's own fetch (x and x + 8, as the writes); 1 selects the AA's value untouched, 0 writes the scene as it is.
kSettingAaStrengthis 0 with PsychoV35 (the mod's AA Strength at 0, its author's advice; the mod's code defaults to100) and 1 without it; no menu row.
Measurements
Verdict: no measurable cost: the GPU time a frame is the same as main's within the log's 0.1 ms, PsychoV35 on and off, and the fps too.
Hardware: Windows 11 Pro, RTX 4090 (driver 610.47), 64 GB RAM, 16 threads; the picture streamed at 120 Hz while measured
Scene: Hyrule Castle Town at night, child Link standing still in front of the stables; 3840x2160, Native HDR, DLSS Quality (2560x1440, preset K), frame generation off, frame rate uncapped, V-Sync off
Method: the GPU time a frame from
longshot_log.txt(vk: GPU time ... ms a frame, every 120 presents): the median of the last ~60 lines of 2 minutes standing still; runs interleaved (main, this PR, main, this PR) after a warm-up run of each, the same build toolchain (v3.0.0-rc.7) for both sides and the options pinned withHOS_OPTIONS_FILE; fps alongside, bound by the stream's presentation (~91)How to reproduce: a save in Castle Town at night; the settings above; each run 2 minutes standing still once in place
544229a26ee92f93Flags and settings
None (no switch, no row: with PsychoV35 the game's AA is off, as the mod's author advises; with Vanilla it is the game's).
Testing
Tested:
python -I -B tests/run_all.pyon Windows 11: the same 68 failures as onmain @ 544229a2(listed below),nothing new;
tests/working_gamut_ir.cppchecks both passes againstRenoDX's matrices, on and off;
tests/aa_strength_ir.cppchecks the AA strength against lerp in both bases; thereference in
tests/test_psychov35.pyis signed. In the game (RTX 4090, Windows 11), Native HDR, PsychoV35, DLSS: thelog lines above; the AA's IR with exactly the scene's six reads converted and the direction texture's two left, and
its six strength reads with the fetch/write order checked; SDR (Vanilla): nothing converted, the picture unchanged.
Side by side with #1393 alone, PsychoV35 saturation at 100, Hyrule Castle Town at night: the torches' and
lanterns' orange stays orange here where it turns yellow there, as the mod's. The AA at the mod's 0 and 100 (a test
row, since removed): with DLSS hardly any difference, 0 the sharper picture (DLSS anti-aliases again); with the Upscaler
Off the AA's smoothing shows at 100 and is gone at 0.
Every upscaler this RTX 4090 offers, switched live at the same spot against DLSS: Game's own, NVIDIA DLAA, AMD FSR 3.1,
Intel XeSS and Off: the same colours, no cast over the picture.
The 68 checks that fail on main @
544229a2and here alike (this PC's environment: a compiler name it lacks, a CRLF checkout, tools not installed)nx2/test_mesh_first_types.py,tests/test_alt_enter.py,tests/test_audio_deadline.py,tests/test_audio_output_errors.py,tests/test_audio_wave_buffer_release.py,tests/test_bc_levels.py,tests/test_bind_memo.py,tests/test_capture_copy_helpers.py,tests/test_collect_wide.py,tests/test_cpu_report.py,tests/test_dlss_actor.py,tests/test_dlss_checks.py,tests/test_dlss_math.py,tests/test_draw_binds.py,tests/test_env_file.py,tests/test_failed_compile_keeps_objects.py,tests/test_finish_output.py,tests/test_fps_overlay_order.py,tests/test_fps_rate.py,tests/test_fps_stats.py,tests/test_framegen_auto.py,tests/test_framegen_gate.py,tests/test_fsr3_math.py,tests/test_gpu_choice.py,tests/test_hd_rumble.py,tests/test_import_span.py,tests/test_job_queue.py,tests/test_jobs_wait_runs_queued.py,tests/test_menu_mouse_scroll.py,tests/test_mirror_inherit.py,tests/test_missing_nup.py,tests/test_movie_upscale_policy.py,tests/test_nvn_texture_handle.py,tests/test_oot_jitter.py,tests/test_pass_hazard.py,tests/test_pool_span.py,tests/test_present_ahead.py,tests/test_present_thread.py,tests/test_prof_clock.py,tests/test_profiling_mode.py,tests/test_prompt_auto_face.py,tests/test_push_state.py,tests/test_quick_pipelines.py,tests/test_report_words.py,tests/test_rt_pool_minimum.py,tests/test_rumble_mapping.py,tests/test_rumble_menu.py,tests/test_rumble_refresh.py,tests/test_run_all_changed.py,tests/test_same_bind.py,tests/test_same_draw.py,tests/test_same_dyn.py,tests/test_seed_reader.py,tests/test_skip_code.py,tests/test_slow_frames.py,tests/test_status_line.py,tests/test_submitted_page.py,tests/test_table_track.py,tests/test_target_image_pool.py,tests/test_tested_configurations.py,tests/test_tex_spans_once_a_present.py,tests/test_texture_layout_sizes.py,tests/test_ub_gap.py,tests/test_upscale_independent.py,tests/test_upscale_label.py,tests/test_vertex_bind.py,tests/test_vk_swapchain_compat.py,tests/test_wb_pages.pyNot tested: AMD FSR 4 and its native AA (AMD RDNA 4 only), the game's network on AMD and Intel GPUs, Linux.
Must be tested:
Off, expecting the same colours on each and no cast over the picture (Test Farm: @Beanstalk amd, PsychoV35 colours with
each upscaler)
Risks
What could break: a cast over the whole picture on an upscale path that writes the network's output without going
back to BT.709 (the passes and stand-ins above cover the ones known); the AA's output with PsychoV35 if the AA program
changed shape (the passes check it and log why they leave it).
With #620 (DLSS 5 Neural Rendering over Native HDR, WIP): its post-upscaler pass reads the network's output before
dlss_out.comp's RGBA16F build takes it back to BT.709, so with PsychoV35 it would get BT.2020 light where it expectsBT.709 (its proxy and BT.709 luminance): the model would see paler colours and its edits would land slightly off, the
highlights kept. Whichever of the two lands second converts around that pass (BT.2020 to BT.709 before, back after).
Found by reading both, not run: no NR runtime here.
Turning it off: the HDR tone mapper row on Vanilla; else revert the commit.
Compatibility: the AA, network and scene programs are translated again once; with PsychoV35 the game's AA is off
(the upscaler anti-aliases alone; without an upscaler, Upscaler Off, edges are not smoothed).
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks. Not ticked yet: Risks says the interaction with #620 would shift colours, found by reading and not run.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.Made and published with the help of Claude Opus 5.5
Tested in game: RTX 4090, driver 610.47, Windows 11, Native HDR, PsychoV35: saturated oranges stay orange (yellow without it); the same colours with Game's own, DLSS, DLAA, FSR 3.1, XeSS and Off; no GPU cost
Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.