Will generative AI be used in VFX? We ask artists working in the industry

June 14, 2024

23 8 minutes read

Untitled design6 — iZUc7oK3gevzGm52AbDiqi 1200 80.jpg

The spectre of AI in the VFX industry is growing, but is it really viable? CG tools and software, like Cinema 4D and Houdini, have always aimed to enable artists to do more in less time. With the maker of 3ds Max researching its own Autodesk AI tools it feels unavoidable that generative AI will be used in the VFX industry.

But how, when and why will AI be used for creating visual effects and 3D animation? As part of our AI Week I’ve put together a snapshot of what the industry thinks, by asking those working in it to for their views. To be clear, the experts and artists I’ve asked aren’t using AI in their daily work and the studios they work for aren’t using AI commercially. None of the images, shows or films in this article, or worked on by these artists, were created using AI.

“If you click on something 20 times to accomplish a task, an AI tool that comes in to help solve that may require only five clicks instead of 20. Over time, over the development of an asset, that can help be a huge gain in efficiency,” explains AJ. “If using AI to help with creative tasks, what I’ve found is that it helps the artist work at a much higher level more efficiently and faster by removing the tedious start up process and allowing the artist to focus on creative output.”

AJ explains his team has been looking at, and experimenting with AI, but it’s not been used in production. That use case is framed by the need to make artists lives easier. He explains: “Some of the AI tools that we’ve been playing with in testing, to see if it’s worthy to add to a pipeline, has essentially been for the sole purpose of making the artist’s job more efficient, but nothing I’ve seen has replaced an artist yet in any of this pipeline.”

He says: “AI can help to create at least a baseline where it’s like, ‘okay, we’re talking in the same ballpark’. Let’s now take that to the concept artist; the concept artist is still very much in the game, they still have to design the very specific elements that are required for that script that an AI just won’t pick up on. But they’re now going to have less iterations to do because everyone was already on the same page when that got kicked off to them.”

According to AJ there are limitations to AI that can’t match a human artist, such as the subtly of ideas in concepts and designs. He tells me: “It won’t be able to take the nuance from three pages of the script earlier, apply that detail to make that one reference, to that one object that should be in the corner, because it was referenced in the script two pages later; all that needs to still be done by humans.”

Animation is another sector where AI is making a big impression. At the recent The VFX Festival, I met up with Elliot Stammers, lead character TD at DNEG Animation for Paramount’s Under the Boardwalk (again, to clarify, not made with AI). He says, “I find it quite an exciting technology” but clarifies he’s “not an expert” but goes on to describe how just as 3D animation has – “sadly” – risen to dominate traditional animation, “[AI] is just a new tool, we just have to adapt to harness it, to use it effectively”.

Charged with researching how VFX studio Lux Aeterna can make use of AI and machine learning in its workflows, creative technologist and VFX artist James Pollock has been looking into what the future holds, but he says before any AI tools can be used commercially the industry needs to “tackle some of these bigger questions of ethics, the legal standards, and around using some of these tools”. (Read our article on the issues with AI.)

James tells me how the best uses of AI are the ones that integrate into an artist’s workflow but don’t replace an artist (a common theme is emerging). He explains: “So they’re opening up the same software they always open up to do those things, and now there’s something that has been powered-up by AI and is making the act of doing something, like rotoscoping, it’s making that a lot quicker”. (We can see this integrated use with AI tools now in Photoshop.)

The team has experimented with using Stable Diffusion, again not for commercial or client projects, and did so by running it locally on their own machines, building in different extensions and investigating ways to use it. “In a way, that felt like the prototype for what AI needs to be,” comments James. “It should be something where you can customise it and build around it in a way that makes sense for the projects that you’re working on.”

For now, James tells me, this kind of experimentation is where AI is, because until it “matures in an ethical and legal sense” it is useable in a commercial sense for a VFX studio like Lux Aeterna. There are concerns around copyrights, ownership and James reveals many streamers and studios won’t sign off on projects made using broad generative AI, so for now, it’s not being used. “And of course,” says James reflecting on Disney’s AI-created intro to Secret Invasion, “audiences just don’t like it”.