Skip to content

Latest commit

 

History

History
125 lines (94 loc) · 4.69 KB

readme.md

File metadata and controls

125 lines (94 loc) · 4.69 KB
Raw

logo
Threefiner

An interface for text-guided mesh refinement.

demo.mp4

Features

  • Mesh in, mesh out: we support ply with vertex colors, obj, and single object glb/gltf with textures!
  • Easy to use: both a CLI and a GUI is available.
  • Performant: Refine your texture in 1 minute with Deepfloyd-IF-II.

Install

We rely on torch and several CUDA extensions, please make sure you install them correctly first!

# tiny-cuda-nn
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

# nvdiffrast
pip install git+https://github.com/NVlabs/nvdiffrast

# [optional, will use pysdf if unavailable] cubvh:
pip install git+https://github.com/ashawkey/cubvh

To use Deepfloyd-IF models, please log in to your huggingface and accept the license.

To install this package:

# install from pypi
pip install threefiner

# install from github
pip install git+https://github.com/3DTopia/threefiner

# local install
git clone https://github.com/3DTopia/threefiner
cd threefiner
pip install .

Usage

### command line interface
threefiner --help
# this is short for
python -m threefiner.cli --help

### refine a coarse mesh ('input.obj') using Stable-diffusion and save to 'logs/hamburger.glb'
threefiner sd --mesh input.obj --prompt 'a hamburger' --outdir logs --save hamburger.glb

### if the initial texture is good, we recommend using IF2 for refinement.
# by default, it will save to './name_fine.glb'
threefiner if2 --mesh name.glb --prompt 'description'

### if the initial texture is not good, we recommend using SD or IF first.
threefiner sd --mesh name.glb --prompt 'description'
threefiner if --mesh name.glb --prompt 'description'

### if the initial geometry is good, you can fix the geometry.
threefiner sd_fixgeo --mesh name.glb --prompt 'description'
threefiner if_fixgeo --mesh name.glb --prompt 'description'
threefiner if2_fixgeo --mesh name.glb --prompt 'description'

### advanced
# directional text prompt (append front/side/back view in text prompt)
# you need to know the mesh's front facing direction and specify it by '--front_dir'
# we use the OpenGL coordinate system, i.e., +x is right, +y is up, +z is front (more details: https://kit.kiui.moe/camera/)
# clock-wise rotation can be specified per 90 degree, e.g., +z1, -y2
threefiner if2 --mesh input.glb --prompt 'description' --text_dir --front_dir='+z'

# adjust training iterations
threefiner if2 --mesh input.glb --prompt 'description' --iters 1000

# explicitly fix the geometry and only refine texture
threefiner if2 --fix-geo --geom_mode mesh --mesh input.glb --prompt 'description' # equals if2_fixgeo

# open a GUI to visualize the training progress (needs a desktop)
threefiner if2 --mesh input.glb --prompt 'description' --gui

Gradio demo:

# requires gradio 4
python gradio_app.py if2

For more examples, please see scripts.

Q&A

  • How to make sure --front_dir for your model?

    You may first visualize it in a 3D viewer that follows OpenGL coordinate system:

    example_front_dir

    The chair is facing down the Y axis (Green), so we can use `--front_dir="-y"` to rectify it to face +Z axis (Blue).

  • fatal error: EGL/egl.h: No such file or directory

    By default, we use the OpenGL rasterizer. This error means there is no OpenGL installation, which is often the case for headless servers. It's recommended to install OpenGL (along with NVIDIA driver) as it brings better performance. Otherwise, you can append --force_cuda_rast to use the CUDA rasterizer instead.

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!