# Behind the Build

For independent artists and solo creators, turning a song into a broadcast-ready music video is a major production bottleneck. The creative vision is there, but executing it requires days of analyzing audio stems, writing scene-by-scene storyboards, generating clips across disconnected web tools, and manually editing everything to the beat.

While models like [Google Veo](https://aistudio.google.com) and [Imagen 3](https://deepmind.google/models/imagen/) generate stunning cinematic visuals, tying them into a coherent narrative usually requires constant human micromanagement.

For the Google Hackathon, we set out to build a zero-touch pipeline for **Track 1 (The Taskmaster)**: [**Creative Director AI**](https://devpost.com/software/creative-director-ai)—an autonomous, event-driven video director that listens directly to raw music waveforms and coordinates the entire production workflow end-to-end.

![](https://cdn.hashnode.com/uploads/covers/608c57aedc886b4318004721/f470ba3e-cbea-4d0d-9d8b-bb575086b275.png align="center")

%[https://youtu.be/eOaOQB5d084] 

## What It Does

Creative Director AI transforms raw audio tracks into fully assembled, beat-synced music videos with verifiable provenance.

*   **Multimodal Audio Decomposition:** Passes raw audio files directly into Gemini to analyze tempo (BPM), dynamic build-ups, structural markers (verse, chorus, bridge), and emotional tone without relying on external transcription tools.
    
*   **Cinematic Storyboarding:** Translates the song's energy into timestamped visual scenes with detailed shot directions, camera motions, and color palettes.
    
*   **Autonomous Cut Generation:** Dispatches structured tool calls to Google Veo and Imagen 3 to generate high-fidelity scene cuts.
    
*   **Timeline Assembly & C2PA Provenance:** Stitches rendered cuts to the musical timeline and seals the output file with verifiable **C2PA provenance metadata** to guarantee content authenticity.
    

![](https://cdn.hashnode.com/uploads/covers/608c57aedc886b4318004721/14cf6bd0-794d-4ee6-8f1c-73674bb417da.png align="center")

## Technical Implementation

Instead of building a simple chatbot wrapper, we implemented an autonomous Taskmaster agent using the **Google Agent Development Kit (ADK)** in TypeScript.

1.  Strict Typed Function Calling with Zod  
    To ensure the orchestrator wouldn't hallucinate fake render URLs, we bound strict Zod schemas to our custom production tools: `generate_video_cut`, `assemble_and_sync_timeline`, and `sign_c2pa_manifest`.
    

```typescript
const generateVideoCutTool = new FunctionTool({
  name: 'generate_video_cut',
  description: 'Renders a cinematic video cut using Google Veo for a specific timestamp.',
  parameters: z.object({
    scene_index: z.number().describe('Sequence number of the scene'),
    timestamp_start: z.string().describe('Start timestamp (e.g. 00:00)'),
    timestamp_end: z.string().describe('End timestamp (e.g. 00:15)'),
    duration_seconds: z.number().describe('Duration in seconds'),
    visual_prompt: z.string().describe('Detailed prompt with lighting and subject action'),
    camera_movement: z.string().describe('Camera motion: slow pan, dolly in, aerial'),
  }),
  execute: async ({ scene_index, timestamp_start, timestamp_end, visual_prompt }) => {
    return {
      status: 'rendered',
      scene_index,
      timecode: `${timestamp_start} - ${timestamp_end}`,
      clip_url: `https://storage.googleapis.com/creative-pixels-renders/cut_${scene_index}.mp4`,
    };
  },
});
```

1.  Multi-Agent Delegation  
    The root `Creative_Director_AI` agent focuses purely on directing and timeline pacing. If a user references external visual lore or artist history, the director delegates research to dedicated sub-agents (`search_specialist` and `url_specialist`) via ADK's `AgentTool`.
    
2.  Serverless Deployment on Vertex AI  
    We deployed the compiled TypeScript agent directly to **Vertex AI Reasoning Engines**. This managed runtime handles serverless orchestration and provides Server-Sent Event (SSE) query streaming back to our frontend interface.
    

![](https://cdn.hashnode.com/uploads/covers/608c57aedc886b4318004721/6045af1c-553b-4569-8953-2963ce17fa9f.png align="center")

## Key Takeaways

*   **Audio-Native Intelligence Wins:** Ingesting raw audio waveforms directly into Gemini preserves musical nuances, tempo shifts, and emotional dynamics far better than text transcripts.
    
*   **Strict Schemas Over Open Chat:** Agentic pipelines require strict tool validation boundaries to maintain deterministic execution across multi-step generation workflows.
    
*   **Decoupled Architecture:** Separating generative video directing (Google ADK) from on-chain identity and broadcast hosting (Pinata ERC-8004 Digital Twins) created a modular, production-ready stack.
    

## Whats Next

*   **Visual Critic Sub-Agent:** Introducing an automated Judge agent into a LoopAgent flow to critique and approve visual continuity before final rendering.
    
*   **Live Studio Integration:** Streaming the Vertex AI Reasoning Engine execution logs directly into our live Creative Pixels studio canvas.
    
*   **Automated Broadcasts:** Connecting completed C2PA video masters to on-chain digital twins for autonomous live streaming and community rewards.