3dgs.md

Method 1 - Gaussian Splats

Software requirements

• ffmpeg
• colmap
• brush

Installation on an Apple Silicon mac

• Install homebrew
• In a terminal, run brew install colmap ffmpeg to install colmap and ffmpeg
• From this page, download brush for with the link for "brush-app-aarch64-apple-darwin.tar.xz"
  • Double click it to extract it
  • You should see a binary named brush in the extracted folder

How to?

Step 0 - Camera Settings

We are going for "clean models" and not cinematic quality. We want as little motion blur as possible.

Depending on your camera's sensor, keep it on 60fps (if it has a big sensor and the image is not noisy with it), or 30fps. Be sure to keep a tiny shutter speed, preferrably 1/250s or even lower. Play with the settings a bit. We don't want a noisy image (some noise is fine).

Shoot with H.265 (or H.264 if your camera doesn't support H.265 encoding). Drop to 8-bit color. Try not to use 10-bit color or "actual HDR" (multiple exposure HDR is fine). Capture in 1080p.

Lock your white balance, ideally with a grey card. White balance adjustment mid-footage will throw the SFM algorithm off.

Whatever settings Arham is using on his camera seem to work well. Try to replicate those.

We want to emulate a drone. Keep a stable height and go slow. (TBD)

Step 1 - Frame extraction

We want to strike a balance between

• Enough frames for our "structure from motion" software to correlate objects.
• Not "too many" frames.
• Total number of frames based on the processing power and memory of your computer. We scale in a O(n^2), so double number of frames will require approximate quadruple the time to process.

We will tune this as we go. We want to start with 2 frames per second.

Create a folder to keep your work. Put your video in it with the name video.mp4 (or video.mov if your camera uses a mov container).

In a terminal in the project folder, run

$ rm -rf images                  # delete any already existing images
$ mkdir -p images                # create an images directory
$ ffmpeg -i video.mp4 -vf "fps=2" -q:v 2 images/frame_%04d.jpg # run ffmpeg to extract frames

After this, you should have frames in the images directory.

Step 2 - Feature extraction and mapping

This step will require some tuning based on the input.

In a terminal in the project directory, run

$ colmap gui

This will open colmap in a graphical window.

Step 2.1 - Creating a new colmap project

1. Click File > New Project
2. In the database field, press new. It should open a file picker.
   • In the file picker, navigate to your project folder.
   • Put database.db as the name
3. In the images field, click select.
   • Again, navigate to the project folder.
   • Select the images folder with all your frames.
4. Click Save.
5. Click File > Save Project and save it in your project folder as project.ini.

Step 2.2 - Feature extraction

This step will extract unique features from frames.

1. Click Processing > Feature extraction.
2. In the camera model, use OPENCV. For now, keep other parameters as-is. We will play with them later if the model turns out to be bad, or we keep looping while trying to extract / match the features.
3. Click Extract and let it run.
4. After this is done, close the feature extraction dialog.
5. File > Save Project.

Step 2.3 - Feature matching

This will match features from other frames. This is one of the most time consuming things.

1. Processing > Feature matching. In this dialog, go to the sequential tab.
2. Enable loop detection, and for now, keep other parameters as is. Keep an eye on the output log. If we end up looping, cancel it and playing with parameters here.
3. Press run.
4. After this is done, close the dialog.
5. File > Save Project.

Step 3.4 - Sparse reconstruction

Here, we'll try to construct a point cloud and triangulate the camera movement.

This will take a lot of time if we have bad footage.

1. Reconstruction > Start reconstruction.
2. Reconstruction > Reconstruction options

   Init	init_min_tri_angle	0.1
   Init	init_min_num_inliers	10
   Triangulation	tri_min_angle	0.01
   Triangulation	tri_ignore_two_view_tracks	☑️ checked
   Filter	filter_min_tri_angle	0.01
   Registration	abs_pose_min_num_inliers	5
   Registration	abs_pose_min_inlier_ratio	0.01
   General	multiple_models	0
   Bundle	refine_focal_length	☐ unchecked
   Bundle	refine_extra_params	☐ unchecked
   

3. Let it run. If the point cloud and camera path doesn't look good, we go play with reconstruction options.
4. File > Save Project
5. File > Export model as text. Save it in a new folder named sparse in your project folder.

Step 4 - Training a Gaussian Splat

1. Open brush
2. In brush, click directory and open your project folder.
3. Brush will show a popup. Keep everything as-is. Default "iterations" are 30k which should be "enough" for most uses.
4. Let brush run. You can see what's being created in its live preview window. You can interact with it.
5. Now when you're happy with the result, expand the controls, click pause and then click export model. Save it in your project folder. Be sure to let it run for atleast 8000 iterations (see the stats window).

Viewing the splat.

Open the SuperSplat Editor and open your model you exported from brush in it.