Machine Learning
We apply machine learning to user experience, content creation and digital twins, primarily in image and motion domains
Synthetic Data for Machine Learning
For a Big Tech industry partner, DigitalFish built a cloud pipeline to generate a large library of 3D virtual environments, automating the modularization of 6+ million 3D assets and over 10 thousand unique Unreal Engine 5 scenes. We used this library to create an even larger-scale synthetic dataset of high-fidelity visual walkthroughs and associated ground-truth data: image segmentation, object pose, and auto-generated labeling. This dataset is used in training advanced machine-learning models for scene understanding.
Training on very large synthetic datasets dramatically increases model accuracy and enables the training to include rare and unsafe features that don’t occur readily or would be unfeasible to create in real-world data. Synthetic data is also far faster and cheaper to generate than collecting and labeling real-world data.
The result is safer, more-dependable models.
Machine Learning for Content Creation
DigitalFish delivers solutions for digital storytelling, and within that we are applying machine learning (ML) to challenges within 3D character animation, digital production and augmented reality. Our focus areas include ML applied both in the image (including video) and 3D-motion domains. Our recent work includes:
Image Domain
- Face tracking
- 2D and 3D pose tracking
- Segmentation and classification
Motion Domain
- Denoising for motion-capture cleanup
- Retargeting from motion capture and pose tracking to rigged characters
- Motion style transfer
We apply varied deep neural network (DNN) architectures to these problems, both in offline and real-time contexts, using multiple deep-learning frameworks including TensorFlow, Caffe and PyTorch.
For training data, we use public databases, we synthesize data, and we have access through partnerships to proprietary datasets.
DigitalFish never uses artist-created works for ML training without explicit permission by the artists or content owners.
CartoonGAN: Chen, Lai and Lui; CVPR 2018
“We anticipate solutions enabled via motion ML will represent a large market opportunity in 2025 and beyond, with applications in games, narrative media and non-entertainment fields. Within these application areas, motion ML will both create new efficiencies for existing creators and will make 3D content creation accessible to a much larger audience of new creators for whom existing tools and workflows are too complex.”
—Dan Herman, DigitalFish CEO
AI Research & Development
We conduct applied research in AI, creating new models and exploring novel applications, in partnership with Silicon Valley’s largest tech giants.
- Computer vision and machine learning
- Image segmentation and object classification
- Object, body and face tracking and pose estimation
- Motion-capture denoising, retarget and style transfer
- Synthetic augmentation of real-world training data
- Training-data synthetic generation and auto-labeling
- Cloud-scale pipelines for data prep and training
- Foundation models for vision and generative AI
Relevant Recent Projects
Facebook Story Time
AR storytelling for families
LEGO Digital Play
Exploring the future of play
Smart Glasses
Mobile software for smart glasses
