Modeling the spatial radiance distribution of light rays in a scene has been extensively explored for applications, including view synthesis. Spectrum and polarization, the wave properties of light, are often neglected due to their integration into three RGB spectral bands and their non-perceptibility to human vision. However, these properties are known to encompass substantial material and geometric information about a scene. Here, we propose to model spectro-polarimetric fields, the spatial Stokes-vector distribution of any light ray at an arbitrary wavelength.
We present Neural Spectro-polarimetric Fields (NeSpoF), a neural representation that models the physically-valid Stokes vector at given continuous variables of position, direction, and wavelength. NeSpoF manages inherently noisy raw measurements, showcases memory efficiency, and preserves physically vital signals - factors that are crucial for representing the high-dimensional signal of a spectro-polarimetric field. To validate NeSpoF, we introduce the first multi-view hyperspectral-polarimetric image dataset, comprised of both synthetic and real-world scenes. These were captured using our compact hyperspectral-polarimetric imaging system, which has been calibrated for robustness against system imperfections. We demonstrate the capabilities of NeSpoF on diverse scenes.
We propose modeling the spectro-polarimetric field $f(\cdot)$ using NeSpoF, a neural representation that describes the Stokes vector $\mathbf{s}$ and volumetric density $\sigma$ for the input variables of position, direction, and wavelength.
$\mathbf{s}, \sigma = F_\Theta(x,y,z,\theta, \phi, \lambda)$
We design NeSpoF with a positional MLP, $F_\Theta^p$, and a spectro-directional MLP, $F_\Theta^d$ and turn to model a physically-valid Stokes vector ($s_0^2 \geq s_1^2+s_2^2+s_3^2$) by using intermediate polarimetric properties as outputs ($s_0$, $\rho$, $\chi$, and $\psi$). Additionally, we perform coordinate conversions of Stokes vectors since the polarization state of a light ray can be described differently with Stokes vectors at different coordinates.
We built a portable hyperspectral polarimetric imaging system, consisting of a monochromatic camera, LCTF that controls the transmitted wavelength, and a quarter wave plate mounted on a motorized rotation stage.
For synthetic scenes, we apply polarimetric BRDF to mesh by fitting a fourth order-polynomial function at each pixel along the spectrum since only five spectral channels are available in existing dataset [Baek et al. 2020] and render images using spectro-polarimetric renderer Mitsuba 3.
For real-world scenes, we capture a scene for 21 wavelengths and 4 quarter-wave plate angles and reconstruct per-pixel Stokes vector by solving a least-squares problem with spatially-varying spectro-polarimetric calibration using these captured images.
NeSpoF enables novel-view rendering of intensity, which is the first Stokes-vector element $s_0$, as well as polarimetric properties as visualized in AoLP (angle of linear polarization), DoP (degree of polarization), and ToP (type of polarization).
Start Frame
End Frame
Compared to ground truth, visualizing intensity $s_0$ and polarization states with AoLP, DoP, and ToP, we found that NeSpoF enables accurate polarimetric reconstruction.
This scene includes a glass box surrounding certain objects and a specular crystal outside of the glass box. For this area, NeSpoF demonstrates modeling intricate variations of AoLP on the crystal surfaces. The polarization of this region also significantly changes according to viewpoints and wavelength.
This is another scene containing linear polarization film laptop and smartphone covered with protective film. When we visualize AoLP for this region, we observed that the linear polarization film and laptop display emits uniformly-distributed linearly polarized light, while the light from smartphone display interacts with protective film, resulting in interesting polarimetric variation. NeSpoF successfully models such variations, closely matching the ground truth even for different wavelengths.
NeSpoF enables decomposing light transport into unpolarized and polarized hyperspectral components. As a result, we can separate not only diffuse and specular components (red arrows) but also more complex light transport, such as polarized inter-reflection between the walls (blue arrows). Additionally, we can extract hyperspectral intensity for diffuse and specular components at a novel-view.
We identify elliptically-polarized sky light and extract illumination spectrum of the sky. NeSpoF enables rendering of the hyperspectral intensity at a novel view and relighting by replacing the original illumination with the target illumination, CIE D65 illuminant plotted as blue.
@inproceedings{kim2023neural,
title={Neural spectro-polarimetric fields},
author={Kim, Youngchan and Jin, Wonjoon and Cho, Sunghyun and Baek, Seung-Hwan},
booktitle={SIGGRAPH Asia 2023 Conference Papers},
pages={1--11},
year={2023}
}