See llms.txt for all machine-readable content.

Back to Templates

Build a multimodal WhatsApp assistant with GPT-5, Gemini, and Wikipedia

Created by

Created by: Dr. Firas || drfiras
Dr. Firas

Last update

Last update 12 hours ago

Categories

Share

Quick overview

This workflow turns a WhatsApp Business number into a multimodal assistant by handling text, voice notes, images, and videos, using OpenAI (GPT-5 for chat/vision and audio transcription) plus Google Gemini for video understanding, and replying back on WhatsApp with per-user conversation memory and Wikipedia lookups.

How it works

  1. Triggers on incoming WhatsApp messages and splits the payload to process each message individually.
  2. Captures sender details and shared configuration (models, memory window, system prompt, and reply length) and routes the message based on whether it is text, audio, image, or video.
  3. For voice notes, it fetches the media from WhatsApp, downloads the audio file, and transcribes it with OpenAI.
  4. For images and videos, it fetches and downloads the media from WhatsApp, then describes images with OpenAI vision and describes/transcribes videos with Google Gemini.
  5. Normalizes the resulting text (original text or media description) along with any caption and the sender phone number into a single message format.
  6. Uses an OpenAI-powered agent with per-sender conversation memory and a Wikipedia tool to generate a concise reply in the user’s language.
  7. Cleans and truncates the reply for WhatsApp-friendly formatting and sends it back to the original sender via WhatsApp.

Setup

  1. Connect WhatsApp Business Cloud credentials for both the WhatsApp Trigger and WhatsApp nodes, then select your WhatsApp Business phone number ID in the send-reply step.
  2. Add an OpenAI API credential for chat, image understanding (vision), and audio transcription.
  3. Add a Google Gemini (PaLM) API credential for the video description request.
  4. Review the Configuration values (models, system prompt, memory context window, and max reply length) to match your needs before activating the workflow.

Additional info

Build a multimodal AI assistant on WhatsApp with GPT-5 and Gemini

📥 Open full documentation on Notion

Need help customizing?

Contact me for consulting and support : Linkedin

MY NEW YOUTUBE CHANNEL

👉 Subscribe to my new YouTube channel. Here I'll share videos and Shorts with practical tutorials and FREE templates for n8n.

The AI Doctor