UNIST’s DogRecon AI Turns a Single Photo into a Lifelike Moving 3D Dog

If you've ever had a pet, you’ve probably sighed at least once: “Why are their lives so short?”Because their time with us is fleeting, many pet owners fill their phones with thousands of photos, capturing every precious moment with their furry companions.

Apple's "Spatial Photos" feature was designed to address this desire to "capture" memories—not just as flat images, but in a 3D format that lets you step into the scene. But what if we want more than just stepping into a scene? What if we want our pets to "step out of the photo" and become animated companions?

This is where the AI technology "DogRecon" from the Ulsan National Institute of Science and Technology (UNIST) in Korea comes into play. You don’t need to circle your pet with a camera or collect piles of footage; with just one photo, you can create a 3D model of your dog that moves. So, for those of you who can't resist snapping photos of every cat and dog you see, DogRecon lets you own these furry friends in a 360-degree, lifelike form!

(Source: vision3d lab)

The name "DogRecon" combines "Dog" and "Reconstruction," and it's an AI technology specifically for creating 3D dog models. Developed by Professor Kyungsoo Cho's team at UNIST's Graduate School of Artificial Intelligence, this tool tackles the tricky problem of 3D dog modeling. UNIST is one of the four leading national universities in Korea specializing in technology research.

Creating a 3D model of a dog isn't as easy as it sounds. Dogs come in many breeds and appearances, with varying sizes, fur, and gaits. Their four-legged structure often means joints are hidden by fur in photos, leading to AI misjudgments and resulting in models that look like a blurry mess.

The DogRecon process starts with a single Shiba Inu photo and uses a canine knowledge model along with AI image synthesis to construct a basic 3D dog structure. It then renders this structure into a lifelike 3D Shiba model using 3D Gaussian Splatting. (Source: Kyungsoo Cho)

To solve these issues, DogRecon combines two core technologies: the Canine Prior Knowledge Model and Canine-centric Novel View Synthesis (NVS).

In simple terms, the Canine Prior Knowledge Model is like an encyclopedia of dogs, containing a vast amount of trained dog image data. It helps determine your pet's breed, body proportions, and skeletal structure, laying the groundwork for 3D modeling. The Canine-centric NVS allows DogRecon to simulate the dog from various angles, whether it's the side, tail, or those little paws hidden in fur, all generated by AI, eliminating the need for traditional surround photography or hundreds of photos or videos.

These simulated multi-angle images are integrated into a 3D point cloud model, which is then rendered using 3D Gaussian Splatting, making the final model look both natural and fluffy.

DogRecon can create a 3D model from a single Husky photo, whereas other models require videos or hundreds of images to achieve the same result. (Source: Kyungsoo Cho)

The research team compared DogRecon with existing pet modeling technologies. For instance, BANMo requires over 170 frames of video but does not support output animations; Fewshot-GART needs only three images, but the results aren't as realistic as DogRecon's.

DogRecon not only accurately recreates a dog's appearance while maintaining aesthetic appeal, but it also offers convenience: with just one photo, you can generate a 3D dog model. It even supports "text-generated animation," allowing your dog to walk, run, or wag its tail on screen with simple commands, as if your pet is leisurely strolling towards you.

Not only can the dog model be turned into a 3D figurine for display, but with text-based action input, it can also be animated to perform movements autonomously. (Source: Kyungsoo Cho)

Professor Kyungsoo Cho from UNIST stated, "With over a quarter of households owning pets in Korea, we aim to extend human-centric 3D reconstruction technology to pets. DogRecon will become a tool accessible to everyone, allowing pet owners to recreate their beloved dogs in digital spaces and even animate them."

The potential applications of this technology are exciting. Once you can easily 3D model your pets and pets you meet, new scenarios will emerge: from remembering beloved pets that are no longer with us to creating virtual pet interactive characters. You can bring them into VR, AR, or VIVERSE virtual spaces, making them your virtual avatar's adorable companions and continuing your bond online.

Unfortunately, DogRecon is not yet available for public testing. However, if a trial version or open-source release becomes available, we'll update you right away!