I asked AI to make a Music Video… the results are trippy

In this video, I utilized artificial intelligence to generate an animated music video for the song Canvas by Resonate. This tool allows anyone to generate beautiful images using only text as the input. My question was, what if I used song lyrics as input to the AI, can I make perfect music synchronized videos automatically with the push of a button? Let me know how you think the AI did in this visual interpretation of the song.

After getting caught up in the excitement around DALL·E2 (latest and greatest AI system, it’s INSANE), I searched for any way I could use similar image generation for music synchronization. Since DALL·E2 is not available to the public yet, my search led me to VQGAN + CLIP (Vector Quantized Generative Adversarial Network and Contrastive Language–Image Pre-training), before settling more specifically on Disco Diffusion V5.2 Turbo. If you don’t know what any of these words or acronyms mean, don’t worry, I was just as confused when I first started learning about this technology. I believe we’re reaching a turning point where entire industries are about to shift in reaction to this new process (which is essentially magic!).

Important note:
While this AI is impressive, it still required additional input beyond just the song lyrics to achieve the music video I was looking for. For example, I added keyframes for camera motion throughout the generated world. These keyframes were manually synchronized to the beat by me. I also specified changes to the art style at different moments of the song. Since many of the lyrics are quite non-specific, even a human illustrator would have a hard time making visual representations. To make the lyrics more digestible by the AI, I sometimes modified the phrase to be more coherent, such as specifying a setting or atmosphere.

This was my first time working with DDV5, and I’m very happy with the results! There were many times where my jaw dropped upon seeing what the AI came up with. I haven’t felt this sense of wonder from technology since I first experienced a HD videogame as a child.

If you would like to learn more about how this video was made, try this yourself, or ask me any questions, I’ll post a more detailed explanation of how to get started on Patreon (link below). The post is free to the public, no need to pay. If you do want to support me and become a member that would be much appreciated, you’ll also automatically be entered into the end screen minigames where you earn points on each video and move up the leaderboard!

Join on Patreon to automatically have your name included in the next video: https://www.patreon.com/doodlechaos

Want to add lyrics and color beat blocks to your Disco Diffusion project like I did in my video? Here is my code: https://www.patreon.com/posts/67249569

My social media:
Twitter: https://twitter.com/doodlechaos
Discord: https://discord.com/invite/7FCrWAzDY7
Tiktok: https://www.tiktok.com/@doodlechaos
Shorts Channel: https://www.youtube.com/channel/UCMqgJk1o2eWE7WeNtRIfnpg
Instagram Shorts: https://www.instagram.com/doodlechaos_shorts/
Email: contact@doodlechaos.com

While Disco Diffusion is based on the contributions of many, show some love to the two most prominent contributors:
https://twitter.com/somnai_dreams
https://twitter.com/gandamu_ml

Music:
[Indie Dance] – Rezonate – Canvas [Monstercat EP Release] : https://www.youtube.com/watch?v=i0Ew3cl1gyc

AI Video & Animation, Disco Diffusion, Disco Diffusion, Music Videos May 17, 2022 · 100 Comments

100 Comments

  8. Ask it to map all the stones from PUMA PUNKU then reconstruct it, or any Aincent ruins we don't have the technology to physically reconstruct. It would be FASANATING!!!!!!!!!!!!!!!!!

    Reply

  64. This is the most wicked! piece of art in video form I have ever seen on the entire internet!
    Please I'd like to know who did this…..I want to see more and I want to know how it's done…
    I'm completely blown away!

    Reply

  79. The AI thinks millions of times faster than we do. It's able to show us a far more in depth view of our world and ways, and as we can clearly see here it is a view far superior to any a human would have been able to show because it really understands more.

    Reply

  93. I just thought that maybe, the reason our reaction to this kind of art is so strange, is because unlike humans, maybe this AI doesn't have a point of focus like us, So unlike human art or music videos, they're designed with the same understanding of drawing our attention to particular parts. But with this art, it's similar to meditation or psychedelics because you're in an open or free state where there's no focal point because you are still and free with no intention or in psychedelics case, ability to look at something with out it changing to quick to focus on, where you're kind of forced to go with. Because the AI's in the same state.

    Reply

  99. The art of this is creation and how it follows a pattern that is natural and influenced. Like Earth, Humans, Creatures witnessing and understanding the beauty of nature around them.Then the AI clevery shows its appreciation for the video towards the end of it. Very heart warming stuff.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

© 2025 AI Art Video Tutorials