Week 41: Using AI to analyze marketing data

The Experiment:

 I had a project that required analyzing some seriously gnarly click data—UTM parameters, parsed URLs, click events on modals, the whole messy package. The kind of dataset that makes you wish you'd paid more attention in your analytics courses.

Being both lazy and optimistic about AI, I thought: why not let ChatGPT handle this? Feed it the data, ask for insights, job done. But then a nagging voice in my head said, "What if it's wrong?" So I did what any paranoid person would do—I asked Claude and Gemini to analyze the same dataset.

Turns out, I should have been paranoid.

The Process:

Here's how I tackled it:

1. The Setup I had a CSV file with click data including: UTM parameters (source, medium, campaign, content, term) Click event details (what people actually clicked on) A modal-based interface that made the data extra messy

2. The Upload Struggle All three tools had issues reading the file. I had to convert the CSV to PDF, then finally to .xlsx before they'd cooperate. Even then, getting clean data into these tools was more painful than expected.

3. The Prompt I created a detailed prompt asking each AI to act as a senior marketing data analyst and extract actionable insights about: Traffic sources and campaign performance User intent (informational vs. transactional vs. navigational) Patterns, anomalies, and optimization opportunities and strategic recommendations

4. The Analysis I let each tool run its analysis independently, then compared their findings.

The Outcome

This experiment was both enlightening and unsettling. Here's what I discovered:

The Numbers Were Wildly Different

Despite analyzing the same dataset, the three AIs reported completely different totals:

• ChatGPT: 1,444 total data rows analyzed

• Gemini: 3,962 total clicks

• Claude: 6,016 total clicks

Even more concerning, their intent distributions were all over the map:

They Were Analyzing Different Things

After digging deeper, I realized the contradiction: they weren't actually analyzing the same data. They were each interpreting the dataset differently:

• Claude: Analyzed ALL click events (6,016 total)

• ChatGPT: Analyzed a filtered subset (1,444 rows)

• Gemini: Focused on specific modal click events only (3,962)

The Download That Wasn't

ChatGPT offered to create a downloadable analysis file for me. Great, right? Except when I opened it, the file was completely empty. No data, no analysis, nothing. A perfect metaphor for how this experiment was going.

Contradictory Strategic Recommendations

Most alarming was their contradictory guidance on budget allocation:

Claude recommended: "Reallocate budget from low-performing Search to high-performing Social. Facebook shows 65% transactional vs. Search at 50%."

ChatGPT & Gemini both recommended: The opposite. "Paid Search is superior—90% transactional with far more volume. Social is higher quality but much smaller scale."

If I'd trusted just one tool, I would have made the wrong decision.

Presentation Styles Matter

• Claude: Most polished with emojis, tables, and executive summary format. Best for presenting to leadership, but gave contradictory channel recommendations.

• Gemini: Clean, professional, concise. Best for busy marketing managers who want focused recommendations. Most balanced interpretation.

• ChatGPT: Technically deep but sprawling and dense. Best for technical teams, but hardest to parse quickly.

Which Tools Were Closest?

Gemini and Claude showed similar intent distributions (62-65% transactional) and agreed on most patterns. ChatGPT was the outlier with 81% transactional, likely because it filtered the data differently.

For strategic recommendations, ChatGPT and Gemini aligned on channel performance. Claude contradicted both, suggesting its analysis methodology was fundamentally different.

Which Should You Trust?

• For reliability: Gemini and ChatGPT agreed most often on strategic recommendations

• For presentation: Claude wins for executive audiences

• For technical depth: ChatGPT caught data quality issues the others missed (like 59% of clicks having missing UTM parameters)

• The safest bet: Where Gemini and ChatGPT agree, you're probably on solid ground

Key Takeaway

 AI tools don't just have different presentation styles—they make fundamentally different analytical choices that lead to contradictory recommendations. For marketing data analysis, using multiple AI tools isn't paranoia, it's due diligence.

The most reliable approach: Use 2-3 tools, look for where they agree, and investigate deeply where they contradict each other. Those contradictions often reveal the most important insights about your data (or expose data quality issues you didn't know existed).

Pro Tips for AI-Driven Data Analysis:

1. Never Trust a Single AI Output: Run your analysis through at least two different tools. Where they agree, you're probably safe. Where they contradict, dig deeper.

2. Check the Numbers First: Before diving into insights, confirm each tool is analyzing the same dataset. Look at total row counts—ChatGPT gave me 1,444 rows when I expected 6,000+ clicks.

3. Use Contradictions as Clues: When AIs disagree on channel performance, it often means your data has quality issues. ChatGPT caught missing UTM parameters that explained some contradictions.

4. Beware of Confidence: All three tools presented their findings with equal certainty, despite reaching opposite conclusions. AI doesn't flag when it's uncertain.

Want to Try It Yourself?

• For executive presentations: Start with Claude for polish and strategic framing

• For technical implementation: Use ChatGPT but be prepared for dense output

• For balanced, quick analysis: Gemini offers the best middle ground

• For important decisions: Run the analysis through at least two tools and compare

The real lesson? AI tools are powerful, but they're also opinionated. They make invisible choices about how to interpret your data, and those choices matter. A lot