SRA
SRA
SRA is a real-time digital human workforce solution designed to support businesses and customers anytime, anywhere. We iterated through multiple concepts before we achieved our launched product.
- Reported directly to Sr. VP of Research who lead the Generative AI Digital Human Platform
- Building Design Team from ground up as Hiring Manager to include Directors, CG Artists,
Product Designers, Concept Artists, Technical Artist, SWE, and Digital Asset Manager
- Building Design Team from ground up as Hiring Manager to include Directors, CG Artists,
- Building the automated Quality Control and Media Asset Management pipeline using
Python and FFMPEG for training realtime digital humans via generative AI, to assess
crops, transcodes, sound, frame anomalies, rig, key points, sync, color, proxies
- System runnable by a digtial asset manager with CLI experience
- editable Json, Yaml file, runnable on Linux and Mac
- FFMPEG, MediaArena QCtools, Mediapipe, SoX (sound),
- code that determines Loudness Units (Lufs) and correct codecs
- Analyze data as graphs. for instance, YUV diff graph to get bad frames by getting a spike in the graph. Since we know the data is recorded in one long shot.
- System runnable by a digtial asset manager with CLI experience
- Converting greenscreen stage from 12k resolution video capture to multicam broadcast
capture and mocap, as well as building the end-to-end hardware/software solution for
audio capture and DSLR facial photogrammetry within a reasonable budget.
- Problem: Too much future proofing by capturing too much data with unclear requirements.
- Solution: work with r and d on narrowing down requirements and building a capture system for production.
- Maintain flexibility for r and d, but keep the rigid requirements of software team.
- Increase speed of capture tenfold
- Problem: How to capture with a midsize budget
- Understand project requirements
- Ensure studio is color correct, but when deploying to various devices, its lesser priority
- Problem: How to sync cameras
- Dont trust hardware specs, always verify
- Our recorder claims to be able to shoot synced cameras, but found out via timecode it is off a few frames heads and tails
- wrote code to take videos with timecode and cuts heads and tails to insure sync
- Experimented with AI solutions first to see if there results are good
- Experimented with 3d Cameras but frames drop for long recording sessions
- Need timecode and genlock because will triangulate data later for inhouse mocap solution
- Our recorder claims to be able to shoot synced cameras, but found out via timecode it is off a few frames heads and tails
- Dont trust hardware specs, always verify
- On-set Technical Director for Talent data capture, and Pipeline Lead for the Design Team
- Previous experience as a filmmaker/improv and onset director helped with coaching actresses to perform better
- Actors strike affected getting good quality data because we couldnt hire SAG
- Solved all Technical issues onset
- Collaborating with Business Strategy and Product Designers on pitch and video decks in
order to prototype Minimal Viable Products using Maya, Blender, Resolve, Unreal, Unity- Presented to C-level people
- Working with R&D scientists on 2D and 3D digital human methods found in latest papers
- Focused on dataops and capture related portion
- determine how to emulate public datasets or buy them, since for product licensing prevents open use
- Renderpeople vs capture inhouse. Use as ground truth
- Driving an avatar via unreal engine, metahumans
- dependent on unreal engine platform
- works with iphone, but not so much about google phones
- arkit vs mediapipe
- at the time mediapipe did not do well with anything 3d points
- arkit was trained on blendshapes
- arkit vs mediapipe
- Asset Management
- Lead effort to storing data in Resource Asset Management for more security and less error prone file management
- Only deploy "published" data
- proper versioning across entire pipeline
- Project Asset Management solutions like Kitsu, Ftrack, Shotgrid
- Lead effort to storing data in Resource Asset Management for more security and less error prone file management
ML
- Building the 3D digital human publish/review pipeline, which processed hundreds of
scans for machine learning models using Python, Maya, Nuke, Shotgrid, Deadline, Unreal- Created a synthetic data set consisting of 1000 3d heads from scans
- Instead of vfx scanning, opted for third party to scan with handheld scanner
- cleaned up by character artists
- created a publish tool where the model is sent to dailies via shotgrid in a turntable with 3 point lighting
- Created mocap review from external vendor to assure the data was good
- Created a synthetic data set consisting of 1000 3d heads from scans
- Extending capabilities of the DSLR body and facial photogrammetry rigs using Python
with MetaShape, PYQT, Maya, OpenCV, Wrap, Deadline, and Flask web framework- worked with Scan Technician to improve the system
- ex. adding external gpus
- increasing boundary boxes
- working with scale markers
- fixing broken cameras, triggers, and grip rigging
- worked with Scan Technician to improve the system
- Creating CI/CD using Jenkins and Ansible, and hybrid renderfarm with Deadline and Google Cloud Platform in order to scale up rendering for volumetric video
- Worked with DevOps and IT to use our SW Deployment system to reimage inhouse
- Applied the CD to work with cloud render nodes via Google Cloud Platform.
- Refactoring cg artist tools to use latest APIs, be multiplatform, and utilize open source through Academy Software Foundation and VFX Reference Platform
- Looking at ways to scale up our platform, which is mostly license bound
- looked into open source solutions such as OpenCue, Blender, FFMPEG, Shotgrid, instead of Deadline, Maya, Kitsu, and Nuke
- refactored legacy code to follow more modular code
- ex. remove version dependencies on api calls.
- looked into open source solutions such as OpenCue, Blender, FFMPEG, Shotgrid, instead of Deadline, Maya, Kitsu, and Nuke
- Troubleshoot headset
- With immersive tech, there is no chatgpt or youtube videos to help you, need to work with engineers that created it directly.
- digital human group was a factor in increasing FOV vertically instead of horizontally
L
- Developing tools to monitor hundreds of hybrid render nodes using Python, Deadline,
Excel, Slack, and GCP, in order to democratize render wrangling across all teams- Joined, then three weeks later google acquisition announced.
- Render Wranglers contract was all canceled, in middle of production that still needs to be finished
- Justin Timberlake video was deployed
- parts of the lightfield version was finished, but never released
SIGG
Lead AR App Engineer | Video Production Director and Correspondant | 2019 Tech Lead for AR/VR/MR
PS
Providing technical and render support to CG artists, as well as automating third-party
3D asset conversion from games to VFX commercials, rendered offline with VRAY
- Hard to build a consistent pipeline because third party assets are all different
- wished there was a "USD" back then but it does require building alot of of short term pipeline really fast
- Experimented with game commercials rendered out of PS4, but post production rendering always wins because can tweak environments better and use better rendering (end goal is to sell, at whatever means necessary). Also less complicated pipeline
- Challenges, we are told not to "bother" the studio too much so we typically have to get internal tools running ourselves. For instance, for Uncharted, Had to pinpoint a siggraph paper and talk in order to get their "Surf" shaders to work
- For spiderman, i had to redo the toon shaders to match for rendering engine
- Experimented with game commercials rendered out of PS4, but post production rendering always wins because can tweak environments better and use better rendering (end goal is to sell, at whatever means necessary). Also less complicated pipeline
● Researching VFX pipelines and supporting next gen game console transition to VR 360,
HDR, and 4k resolution workflows for Cinema4D, Maya, Nuke, VRAY, and After Effects.
- Went to conventions like DevCon and was the liason to R&D to ensure the smooth transition from 1080p PS4 era to PSVR and and PS4Pro (4k and HD
● Full stack web development of internal wiki and technical documentation using Drupal, Capistrano, Git, and programmed in Javascript, PHP, Ruby, HTML, and CSS
● Managing and upgrading software, hardware, and network consistency with Sony IT across the renderfarm, servers, and workstations using Deadline Monitor and CLI
● Performing photography, studio lighting, and photo editing for Sony Products to be used for brand packaging, instruction manuals, and promotional marketing
● Volunteering with R&D for hackathons to create PlayStationVR demos with PS4 devkit
Pros and cons working in tech
- tech has surprisingly limited resource s
- alot of everything needs to be proprietary since launching a product with a bunch of third party dependencies cost money
- opensource over using DCC
- opensource over using DCC
- alot of build up and tear down pipelines when proving something doesnt work
- generalist over being a specialist, which has pros and cons
other light fiend companies
talk about SPIEVR.AR
-leia inc , looking glass, creal, magic leap internal demos