In my art practice, I am always looking to explore the intersection where the human body meets the frontier of digital representation. Particularly I like using novel rendering techniques that have just barely escaped the research institutions they were created in. This interest was reignited when I encountered the groundbreaking 2023 SIGGRAPH paper on 3D Gaussian Splatting for Real-Time Radiance Field Rendering. Intrigued by its innovative use of point cloud reconstruction with Gaussian splats and neural radiance fields to craft view-dependent 3D scenes, I knew I had to dive in and experiment with it myself.

I assisted Zach Lieberman to create OfxSplat, an OpenFrameworks addon implementation of a NERF gaussian splatting renderer, based on Kevin Kwok's WebGL version. 

The process of obtaining the radiance field splat is phenomenal. I really think its optimal for getting 3D models from artists remotely, really easy to explain. I asked an artist I frequently work with, Sarah Davis AKA DJ_DAVE, to ask someone to record her, using any smartphone which has capability to record in slow motion standing still, circling her 3 times at different heights in slow motion (to reduce blurry frames). 

I got back a video like this! I cannot emphasize how great it was to get content I could use from an artist without having them go into a scan studio, or have me have to go to them to do it, or use some fancy hard-to-obtain hardware.

I input 700 of the frames of the video into COLMAP to extract the camera poses, you can see the sparse point cloud the software was able to obtain from older photogrammetry methods:

Then I ran it through the network I downloaded from the Github linked to the original paper to train for a new gaussian splat point cloud ply file. I got the whole splat back after 4 days of 30,000 iterations! The setup process was pretty smooth for an ML project! I only ran into a few hiccups. The biggest being that I had to use a specific version of COLMAP.

This is what the splat looked like in the aforementioned WebGL based renderer:

I cleaned up the model and removed the background using a Unity splat tool, and then started playing around with the model in OpenFrameworks. Mostly using the shaders to modify the space around it.

I asked Sarah to send me another video, but this time where she is posing.

I played around with duplicating the model:

I used splat artifacts to create the sparkles around her:

I started playing with the spherical harmonics of the model.

As always, I must have a special place to appreciate the glitches, or the mistakes that I run into:

To speak to the technical implementation that I worked on, here is the context: This method produces PLY files, where it has a data entry for every point, or splat, the coefficients and variables to calculate the direction and size of each splat, as well as 43 coefficients for calculating the spherical harmonic of the view dependent coloring. These special point clouds are called a splat, a synecdoche, since its a smaller part of the whole thing to describe the larger object. Sometimes its called NERF splat where NERF is a acronym for neural radiance field. This complicated naming scheme all an attempt to honor the lineage of development that helped the final technique get to where it is today. Gaussian splats were invented in the 90s, but were only recently combined w NERF to create this rendering technique. You can read more in this really lovely (& long) technical writeup really ties together the history as well. 

Now, why do we need these 43 coefficients per point? It takes up most of the space for this particular feature so it might be worth delving a little deeper on. You know how shiny or semi transparent things are notoriously hard to scan and render in 3D? This method works very well for that type of object, on top of being the best rendering method for reconstruction I have seen for even non-reflective objects. 

This is because of the use of spherical harmonics coefficients.

In short spherical harmonics are functions to describe colors on a sphere, and so you can use this instead to think about it as describing the colors of all directions given a point in space. The inputs to these functions are the 43 coefficients, the splat location, and the camera location.

This is well worth the effort to implement in my opinion because its what sets the rendering technique obviously apart from others. 

I was inspired previously by the work of Alex Carlier. He created some examples of what gaussian splatting representation can do in some threads he posted. Here you see you can create different images viewed from different angles. Here you can see the splat’s structure revealing itself.

Going forward, I am interested in how to interestingly break this kind of novel representation. So usually the scan subject is meant to stay still but if you move during the process it preserves the interpolated movement within the capture.

I would like to take more scans of reflective objects to really showcase this. Perhaps though clothing or chrome accessories in the models I work with.

Please feel free to email me with questions or comments at Ch4@mit.edu