Behind the Build

For independent artists and solo creators, turning a song into a broadcast-ready music video is a major production bottleneck. The creative vision is there, but executing it requires days of analyzing audio stems, writing scene-by-scene storyboards, generating clips across disconnected web tools, and manually editing everything to the beat.

While models like Google Veo and Imagen 3 generate stunning cinematic visuals, tying them into a coherent narrative usually requires constant human micromanagement.

For the Google Hackathon, we set out to build a zero-touch pipeline for Track 1 (The Taskmaster): Creative Director AI—an autonomous, event-driven video director that listens directly to raw music waveforms and coordinates the entire production workflow end-to-end.

https://youtu.be/eOaOQB5d084

What It Does

Creative Director AI transforms raw audio tracks into fully assembled, beat-synced music videos with verifiable provenance.

• Multimodal Audio Decomposition: Passes raw audio files directly into Gemini to analyze tempo (BPM), dynamic build-ups, structural markers (verse, chorus, bridge), and emotional tone without relying on external transcription tools.

• Cinematic Storyboarding: Translates the song's energy into timestamped visual scenes with detailed shot directions, camera motions, and color palettes.

• Autonomous Cut Generation: Dispatches structured tool calls to Google Veo and Imagen 3 to generate high-fidelity scene cuts.

• Timeline Assembly & C2PA Provenance: Stitches rendered cuts to the musical timeline and seals the output file with verifiable C2PA provenance metadata to guarantee content authenticity.

Technical Implementation

Instead of building a simple chatbot wrapper, we implemented an autonomous Taskmaster agent using the Google Agent Development Kit (ADK) in TypeScript.

1. Strict Typed Function Calling with Zod
   To ensure the orchestrator wouldn't hallucinate fake render URLs, we bound strict Zod schemas to our custom production tools: generate_video_cut, assemble_and_sync_timeline, and sign_c2pa_manifest.

const generateVideoCutTool = new FunctionTool({
  name: 'generate_video_cut',
  description: 'Renders a cinematic video cut using Google Veo for a specific timestamp.',
  parameters: z.object({
    scene_index: z.number().describe('Sequence number of the scene'),
    timestamp_start: z.string().describe('Start timestamp (e.g. 00:00)'),
    timestamp_end: z.string().describe('End timestamp (e.g. 00:15)'),
    duration_seconds: z.number().describe('Duration in seconds'),
    visual_prompt: z.string().describe('Detailed prompt with lighting and subject action'),
    camera_movement: z.string().describe('Camera motion: slow pan, dolly in, aerial'),
  }),
  execute: async ({ scene_index, timestamp_start, timestamp_end, visual_prompt }) => {
    return {
      status: 'rendered',
      scene_index,
      timecode: `${timestamp_start} - ${timestamp_end}`,
      clip_url: `https://storage.googleapis.com/creative-pixels-renders/cut_${scene_index}.mp4`,
    };
  },
});

1. Multi-Agent Delegation
   The root Creative_Director_AI agent focuses purely on directing and timeline pacing. If a user references external visual lore or artist history, the director delegates research to dedicated sub-agents (search_specialist and url_specialist) via ADK's AgentTool.

2. Serverless Deployment on Vertex AI
   We deployed the compiled TypeScript agent directly to Vertex AI Reasoning Engines. This managed runtime handles serverless orchestration and provides Server-Sent Event (SSE) query streaming back to our frontend interface.

Key Takeaways

• Audio-Native Intelligence Wins: Ingesting raw audio waveforms directly into Gemini preserves musical nuances, tempo shifts, and emotional dynamics far better than text transcripts.

• Strict Schemas Over Open Chat: Agentic pipelines require strict tool validation boundaries to maintain deterministic execution across multi-step generation workflows.

• Decoupled Architecture: Separating generative video directing (Google ADK) from on-chain identity and broadcast hosting (Pinata ERC-8004 Digital Twins) created a modular, production-ready stack.

Whats Next

• Visual Critic Sub-Agent: Introducing an automated Judge agent into a LoopAgent flow to critique and approve visual continuity before final rendering.

• Live Studio Integration: Streaming the Vertex AI Reasoning Engine execution logs directly into our live Creative Pixels studio canvas.

• Automated Broadcasts: Connecting completed C2PA video masters to on-chain digital twins for autonomous live streaming and community rewards.