So there's this really interesting method for making videos with Stable Diffusion and EbSynth. It should increase the persistence, and lower flickering if I'm doing it right.
Looks like it works really well for rendering real people. But I'm thinking if I adjusted the technique a little bit, by adding a game environment and a talker avatar, I think I could use it for pretty much any animation style.
The other thing I think I need is more EbSynth keyframes. Four is going to be enough for about ten seconds, but that feels a little low to me. Especially for a longer test. It's no fun if we're not pushing it to the limit. So I'm thinking... 8-12 keyframes maybe? Will have to play with it.
The character and capture pipeline
My plan of attack is to compile a talker creating a new Tilde Dawn export, and convert her into a vrm. Do the HDRP thing, hopefully get the colliders working, do some cloth physics for that bitchin outfit, and then render a one minute video against the character, with my minimal mocap capabilities. So face neck and shoulders, and maybe hands if I'm feeling like I really hate myself.
I'll route the whole thing with vseeface, pumping in OSC data into Unity. We know I can do everything up until this point, because you've seen me do every step of this process before.
So, what do we know about rendering this particular character with AI? Well, we know that Tilde renders beautifully in Stable Diffusion against my bod_positivity_v3 model for sd 2.x. She was after all part of the training data. I'll have to pull it out of long term storage, but I do still have it.
Choosing the keyframes
Next thing I do is pull 8-12 frames out of the video. This has to include the character doing some of the essential blendshapes like pucker and JawOpen. The rest get reserved for context changes like the head moving or the character looking up down or side to side. I'll try to keep my head still while filming, but we'll see.
And then I pull out the frames by exporting them from Blender, or DaVinci Resolve. I know Blender too well at this point, and I don't know enough about davinci, so I'll probably do it that way.
The experimental rendering pass
This next part is unexplored territory.
Once I have the frames picked out, I'll drop them in my photo editor and make a sprite sheet with our frames on them.
Then I'll use them as inits, and pass them all into Stable Diffusion against a normal map with Controlnet.
The guys I saw using this technique were using Lineart instead of normal map, but I think normal map is going to yield more reliable facial expressions, except for eye blinks which could get weird. I think we can fix that with some light compositing if we have to, provided that we're very careful about how it's done.
Since we're passing this all to a procedural tool that works on a very narrow use case, we need these to be as precise as possible.
Anyway, then we just chop up the images, name the fuckers, and do the EbSynth thing we've done a hundred times, but with new flicker settings, woo hoo!
So I've got the very very first step of the process done. Export the fbx from cc4, and open her up in Blender to make sure we're not exporting any broken shapekeys. But just now I realized that it's 3am, and I have to do job hunting tomorrow.
I'll definitely work on this one this week.
Good night my children of chaos, my explorers of the unknown.
I love you all.