K 视图输入 COLMAP 初始化
3DGS Surfels 表示场景
3DGS surfels represent scene
研究动机：为什么高光照场景特别难？
Motivation: Why Are High-Illumination Scenes Hard?
逆渲染在高光照下面临三大互相纠缠的技术挑战
Three intertwined technical challenges for inverse rendering under high illumination
挑战一
Challenge 1
渲染-几何断裂Render-Geometry Break
无显式 surface → 阴影计算不可靠
Visibility 估计偏差大
无显式 surface → 阴影计算不可靠
Visibility 估计偏差大
➕
挑战二
Challenge 2
光照-材质耦合Light-Material Coupling
高光反射下材质和光混在一起
Ill-posed 反问题
高光反射下材质和光混在一起
Ill-posed 反问题
➕
挑战三
Challenge 3
辐射-优化冲突Radiance-Optimization Conflict
极亮高光"绑架"梯度
暗处细节被忽略
极亮高光"绑架"梯度
暗处细节被忽略
→
IR-HGP 方案
IR-HGP Solution
HVD + GIFP + PARC
三模块协同解耦
PSNR 33.61 SOTA
三模块协同解耦
PSNR 33.61 SOTA
🔥 高光照场景的三大技术瓶颈
🔥 Three Technical Bottlenecks Under High Illumination
瓶颈一：渲染-几何断裂
Bottleneck 1: Render-Geometry Break
- 3DGS 用高斯椭球表示场景，没有显式表面（surface）
- 3DGS represents scenes with Gaussians, no explicit surface
- 导致 visibility 计算和阴影投射不可靠
- Causing unreliable visibility and shadow computation
瓶颈二：光照-材质耦合
Bottleneck 2: Light-Material Coupling
- 强光下的镜面反射使材质颜色和光照亮度不可分
- Specular highlights make albedo and illumination inseparable
- 从稀疏视图同时估计两者是严重 ill-posed 问题
- Simultaneous estimation from sparse views is severely ill-posed
瓶颈三：辐射-优化冲突
Bottleneck 3: Radiance-Optimization Conflict
- 极亮高光区域主导梯度，暗部细节被忽略
- Extremely bright highlights dominate gradients, dark details ignored
- heuristic 正则化破坏物理一致性，产生 baked-in shadow
- Heuristic regularization breaks physics, causes baked-in shadows
❓ 为什么现有方案不够？
❓ Why Existing Solutions Fall Short?
| 方向Direction | 代表工作Works | 局限Limitation | |
| NeRF-based | TensoIR | 渲染速度慢 (<1 FPS) | Slow rendering (<1 FPS) |
| Gaussian-IR | GS-IR | 无 visibility 处理 | No visibility handling |
| SDF-based | R3DG, DiscretizedSDF | 高光场景 PSNR 不足 | Insufficient highlight PSNR |
| Ours | IR-HGP | 三模块协同，物理一致 | 3-module, physics-consistent |
💡 我们的核心洞察：
💡 Our Key Insight:
现有方法将三个问题分开处理或完全忽略。我们提出统一框架，
用 HVD 解决几何断裂、用 GIFP 分离光照材质、用 PARC 稳定优化——三者协同实现物理一致的逆渲染。
Existing methods address or ignore these issues separately. We propose a unified framework where
HVD fixes geometry break, GIFP separates light/material,
and PARC stabilizes optimization—working together for physics-consistent inverse rendering.
方法概览：IR-HGP 三模块框架
Method Overview: IR-HGP Three-Module Framework
从多视角照片到可重打光 3D 资产的完整流程
Complete pipeline from multi-view photos to relightable 3D assets
1
混合可见性分解Hybrid Visibility Decomp.
HVD — 渲染-几何断裂解决方案 HVD — Solving Render-Geometry Break
HVD — 渲染-几何断裂解决方案 HVD — Solving Render-Geometry Break
📍 输入：稀疏多视角图像Input: Sparse Multi-view Images
↓
🔄 双路径渲染架构Dual-Path Rendering
- 2D Gaussian Surfels
- 2D Gaussian surfels
- 实时渲染
- 提取三角网格 Mesh
- Extract triangle mesh
- 光线追踪
↓
📐 辐射分解公式Radiance Decomposition
$L_o(p,\omega_o) = L_{dir} \cdot V(p) + L_{ind}$
直接光 × Visibility
Direct × Visibility
+ 间接光 (SH)
+ Indirect (SH)
↓
✅ 精确阴影 + 实时速度Accurate Shadows + Real-time
Visibility 准确Accurate Visibility
92 FPS 保持92 FPS Maintained
解决渲染-几何断裂
SOLVES RENDER-GEOMETRY BREAK
2
生成式光照先验Generative Illum. Prior
GIFP — 光照-材质耦合解决方案 GIFP — Solving Light-Material Coupling
GIFP — 光照-材质耦合解决方案 GIFP — Solving Light-Material Coupling
🖼️ 粗略光照特征 $L_{Coarse}$Coarse Lighting Feature $L_{Coarse}$
从多视角输入提取环境光初步估计
Initial env-map estimation from multi-view inputs
HDR Environment Map
HDR Environment Map
↓
🎨 条件扩散模型先验Conditional Diffusion Prior
- 预训练于真实 HDR 环境图数据集
- Pre-trained on real HDR env-map dataset
- Condition: $L_{Coarse}$ 光照特征
- Condition: $L_{Coarse}$ lighting features
↓
📊 SDS 损失引导SDS Loss Guidance
$\mathcal{L}_{GIFP} = \mathbb{E}_{\epsilon,t}\left[\|\epsilon - \epsilon_\theta(E, t)\|^2\right]$
Score Distillation Sampling 拉住优化器
SDS pulls optimizer toward realistic distribution
↓
✅ 物理合理的光照估计Physically-Plausible Lighting
防止模糊/偏色No blur/color shift
材质干净分离Clean albedo separation
解决光照-材质耦合
SOLVES LIGHT-MATERIAL COUPLING
3
物理感知辐射校正Physically-Aware Rad. Correction
PARC — 辐射-优化冲突解决方案 PARC — Solving Radiance-Opt Conflict
PARC — 辐射-优化冲突解决方案 PARC — Solving Radiance-Opt Conflict
📷 HDR 渲染图 + 目标图HDR Rendered + Target Image
极端 luminance 差异 高光主导梯度
↓
🎛️ ACES 色调映射校正ACES Tone Mapping Correction
$\hat{I} = T_{ACES}(I;\,\beta)$ $\beta$: 可学习全局曝光参数
基于 ACES 曲线的自适应曝光映射ACES curve-based adaptive exposure mapping
仅 +1 自由度Only +1 DOF
仅 +1 自由度Only +1 DOF
↓
⚖️ 梯度友好空间 LossGradient-Friendly Loss Space
$\mathcal{L}_{total} = \|\hat{I}_{render} - \hat{I}_{target}\|_1$
所有区域获得均衡梯度Balanced gradients across all regions
↓
🏆 干净的材质贴图Clean Albedo Maps
消除 Baked-in Shadow ✨Eliminate Baked-in Shadow ✨
+1.49 dB PSNR gain
+1.49 dB PSNR gain
解决辐射-优化冲突
SOLVES RADIANCE-OPTIMIZATION CONFLICT
Fig. IR-HGP 三模块流水线：HVD 通过双路径（Surfels 快速渲染 + Mesh 精确光线追踪）分解直接/间接光并准确计算可见性；
GIFP 利用预训练条件扩散模型的 SDS 损失约束光照估计不偏离真实分布；
PARC 基于 ACES 色调映射引入可学习曝光参数 β，将 HDR 值映射到梯度友好空间，彻底消除 baked-in shadow。
Fig. IR-HGP 3-Module Pipeline: HVD decomposes direct/indirect light via dual-path (fast surfels + accurate mesh ray-tracing);
GIFP uses SDS loss from pre-trained conditional diffusion model to constrain lighting estimation;
PARC introduces learnable exposure β via ACES tone-mapping, mapping HDR to gradient-friendly space, eliminating baked-in shadows.
实验结果：全面领先的 SOTA 性能
Experiments: Full-Metric SOTA Performance
在所有基准测试上超越现有方法的量化对比
Quantitative comparison against state-of-the-art on all benchmarks
| 方法 / Method | Method | 类型 | Type | Mean PSNR ↑ | Mean SSIM ↑ | Mean LPIPS ↓ | 训练时间 | Time | FPS |
|---|---|---|---|---|---|---|---|---|---|
| TensoIR | NeRF | 28.22 | 0.9353 | 0.0840 | 5.4h | 5.4h | <1 | ||
| GS-IR | Gaussian | 29.25 | 0.9278 | 0.0880 | 0.6h | 0.6h | 208 | ||
| R3DG | Gaussian | 29.81 | 0.9645 | 0.0493 | 1.1h | 1.1h | 51 | ||
| DiscretizedSDF | Gaussian | 32.12 | 0.9700 | 0.0453 | 1.2h | 1.2h | 139 | ||
| ★ IR-HGP (Ours) | Gaussian | 33.61 | 0.9761 | 0.0369 | 1.5h | 1.5h | 92 |
消融实验：每个模块都有贡献
Ablation Study: Every Module Contributes
逐一移除各模块验证其必要性
Removing each module verifies its necessity
A1
移除 HVD 可见性计算
Remove HVD Visibility
去掉 mesh 光线追踪的可见性约束后：
After removing mesh ray-tracing visibility constraints:
- 阴影投射不准确，出现漏阴影或多重阴影伪影
- Inaccurate shadow casting, leaking/multiple shadow artifacts
- 直接光照项的 visibility 调制失效
- Direct light visibility modulation fails
PSNR ↓ 明显下降 | 阴影质量差
A2
移除 GIFP 扩散先验
Remove GIFP Diffusion Prior
去掉条件扩散模型的 SDS 先验约束后：
After removing conditional diffusion model's SDS prior:
- 光照估计退化为模糊、偏色的不合理结果
- Lighting degrades to unrealistic blurry/biased results
- 材质和光照重新耦合，albedo 不纯
- Material and lighting re-couple, impure albedo
PSNR ↓ ~2dB | 光照模糊偏色
A3
移除 PARC 辐射校正
Remove PARC Radiometric Corr.
去掉 ACES 色调映射的辐射校正后：
After removing ACES tone-mapping correction:
- 出现 baked-in shadow——阴影被错误烘焙进 albedo
- Baked-in shadow appears — shadows incorrectly baked into albedo
- 高光区域产生过曝伪影
- Overexposure artifacts in highlight regions
- 换光照后阴影依然存在（最致命）
- Shadows persist under relighting (most critical)
PSNR ↓ | Baked-in Shadow 复现
核心优势：为什么 IR-HGP 更强？
Key Advantages: Why IR-HGP Wins?
从设计哲学到实验效果的全方位优势总结
Comprehensive advantages from design philosophy to experimental results
01
双路径渲染：又快又准
Dual-Path Rendering: Fast & Accurate
传统方法要么快但不准（纯 3DGS），要么准但不快（NeRF）。我们的 HVD 模块采用"粗活细干"策略：
日常渲染走 2D Gaussian Surfels 路径（保持实时），定期切换到 Mesh 光线追踪计算精确 visibility 和阴影。
两者互补而非替代，实现了精度与速度的最佳平衡。
Traditional methods are either fast but inaccurate (pure 3DGS) or accurate but slow (NeRF).
Our HVD uses "coarse-fast + precise-slow": daily rendering via 2D Gaussian surfels (real-time),
periodically switching to mesh ray-tracing for exact visibility and shadows.
Complementary rather than replacement — optimal speed-accuracy balance.
92 FPS + 精确阴影 ★
02
扩散先验做光照"守门员"
Diffusion Prior as "Lighting Gatekeeper"
GIFP 是第一个将预训练扩散模型引入 3DGS 逆渲染的工作。
扩散模型就像一位"光照专家"，它见过海量真实 HDR 环境图，知道什么光照是合理的。
当优化器试图把光估成不合理的样子时，SDS 损失会把它拉回来。
结果：光照估计不再模糊偏色，材质贴图干净分离。
GIFP is the first to introduce pre-trained diffusion models into 3DGS inverse rendering.
The diffusion model acts as a "lighting expert" that has seen thousands of real HDR env-maps,
knowing what plausible lighting looks like. When the optimizer tries implausible estimates,
SDS loss pulls it back. Result: no more blurry/biased lighting, clean albedo separation.
~2dB PSNR from GIFP alone
03
1 个参数根治 Baked-in Shadow
1 Parameter Cures Baked-in Shadow
PARC 只引入了 1 个标量参数 β（可学习全局曝光），却从根本上解决了困扰领域多年的 baked-in shadow 问题。
基于 ACES 电影级色调映射曲线，将 HDR 值映射到梯度友好空间，
让优化器能同时看清亮部和暗部的细节。不是 heuristic hack，而是有物理依据的校正。
PARC introduces only 1 scalar parameter β (learnable global exposure),
yet fundamentally cures the long-standing baked-in shadow problem.
Based on cinematic ACES tone-mapping, it maps HDR values to gradient-friendly space,
letting the optimizer see details in both bright and dark regions simultaneously.
Not a heuristic hack — physically-grounded correction.
+1.49 dB | 仅 +1 DOF | 物理一致 ✦
04
高反射物体上的碾压级优势
Dominant Lead on Reflective Objects
IR-HGP 在高反射/高光照物体上的优势尤为惊人。
以 Shiny Blender 数据集的 Helmet 物体为例：
我们的 PSNR 达到 35.00，而第二名 DiscretizedSDF 只有 30.29——
差距近 5dB！这正是我们针对高光照场景设计的威力所在。
IR-HGP's advantage is especially dramatic on high-reflective/high-illumination objects.
For Shiny Blender's Helmet object: our PSNR reaches 35.00 vs.
runner-up DiscretizedSDF's 30.29 — a nearly 5dB gap!
This is exactly where our high-illumination-focused design shines.
+4.71 dB on Shiny Helmet ⭐
05
三模块协同 > 简单叠加
Synergy > Simple Stacking
HVD、GIFP、PARC 不是三个独立 patch，而是紧密协作的有机整体：
HVD 提供准确的 visibility 让 GIFP 的光照分解有可靠基础；
GIFP 提供合理的光照让 PARC 的校正有正确的目标；
PARC 保证整体优化的稳定性让 HVD/GIFP 的效果不被破坏。
1+1+1 > 3 的系统级设计。
HVD, GIFP, PARC are not independent patches but a tightly-integrated organic system:
HVD provides accurate visibility as foundation for GIFP's light decomposition;
GIFP supplies plausible lighting as correct target for PARC's correction;
PARC ensures overall optimization stability protecting HVD/GIFP effects.
A synergistic 1+1+1 > 3 system-level design.
System-level co-design ✦
06
面向落地：离线生产 + 在线渲染
Production-Ready: Offline Gen + Online Render
IR-HGP 的 pipeline 天然契合工业界需求：
离线阶段（1.5h/scene）从一组照片生成高质量 3D 资产（形状+材质+光照）；
在线阶段（92 FPS）支持任意重打光和自由视角浏览。
这正是游戏美术、电商 3D 展示、XR 内容生产的理想 workflow——
拍照即建模，建模即可用。
IR-HGP naturally fits industry workflows:
Offline phase (1.5h/scene): generate high-quality 3D assets (shape+material+lighting) from photos;
Online phase (92 FPS): arbitrary relighting and free-viewpoint browsing.
The ideal workflow for game art, e-commerce 3D showcases, XR content production —
shoot-to-model, model-to-use.
Game/XR/E-commerce ready 🎯