Published
Aug 15, 2024
Updated
Aug 15, 2024

Do Students Prefer ChatGPT or Google for SQL?

Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queries
By
Harsh Kumar|Mohi Reza|Jeb Mitchell|Ilya Musabirov|Lisa Zhang|Michael Liut

Summary

The rise of large language models (LLMs) like ChatGPT has transformed how students learn to code, particularly when it comes to tackling database queries with SQL. Traditionally, students would hit the web, scouring Stack Overflow and coding blogs for snippets and solutions. But now, they have a powerful new tool at their fingertips. This study explored how students use LLMs versus traditional web search when writing SQL queries. Researchers observed students in a real-world database course, comparing their interactions with standard ChatGPT, a specially tuned instructor LLM (designed to avoid giving direct answers), and good old web search. Surprisingly, the study revealed that students using the instructor-tuned LLM interacted far more with the tool than those using regular ChatGPT or web search. Why? Perhaps the guardrails and prompts encouraged a more conversational, back-and-forth approach. Interestingly, there wasn't a significant difference in the quality of the final queries across the three groups. Though the LLM users edged out the web searchers slightly in terms of correctness. What's even more intriguing is that students using the instructor-tuned LLM reported a lower mental burden. This suggests that while interacting more, the tailored guidance made the process feel less taxing. This research offers a glimpse into how AI is changing the learning landscape. While concerns linger about over-reliance and potential for misuse, LLMs like ChatGPT clearly offer a powerful new way for students to grapple with coding challenges. As these tools evolve and instructors find creative ways to integrate them, we can expect even more engaging and personalized learning experiences. It remains to be seen how these technologies will influence the future of education, but it’s clear that students don't seem to mind having AI tutors in the coding dojo.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team. Get started for free.

Question & Answers

How does the instructor-tuned LLM differ from standard ChatGPT in teaching SQL?
The instructor-tuned LLM was specifically designed to avoid giving direct answers, instead encouraging a more interactive learning process. This system was programmed with guardrails and prompts that facilitated more back-and-forth conversation between students and the AI. The implementation resulted in significantly more interactions compared to standard ChatGPT, while simultaneously reducing students' reported mental burden. For example, instead of providing complete SQL queries, it might ask leading questions or provide partial solutions that encourage students to think through the problem-solving process themselves.
What are the benefits of using AI-powered tools for learning programming?
AI-powered learning tools offer personalized, interactive guidance that adapts to individual learning styles. They provide immediate feedback, reduce mental burden, and create a more engaging learning experience compared to traditional methods. These tools can help break down complex concepts into manageable chunks, offer practice opportunities, and provide context-specific assistance. For instance, students can get instant clarification on coding concepts, debug their code more efficiently, and explore different approaches to problem-solving, all while maintaining an active learning environment.
How is artificial intelligence changing modern education?
Artificial intelligence is transforming education by introducing personalized learning experiences and intelligent tutoring systems. The research shows that AI tools can reduce mental burden while maintaining or improving learning outcomes. AI enables 24/7 access to learning resources, provides immediate feedback, and can adapt to different learning styles. This technology is particularly effective in technical subjects like programming, where students can engage in interactive problem-solving with AI tutors. The trend suggests a future where AI becomes an integral part of educational tools and methods.

PromptLayer Features

  1. A/B Testing
  2. The study's comparison between regular ChatGPT and instructor-tuned LLM variants aligns with PromptLayer's A/B testing capabilities
Implementation Details
Set up parallel prompt variants tracking student interactions, response quality, and engagement metrics using PromptLayer's testing framework
Key Benefits
• Quantitative comparison of different LLM configurations • Systematic evaluation of prompt effectiveness • Data-driven optimization of educational prompts
Potential Improvements
• Add automated quality scoring metrics • Implement real-time performance monitoring • Develop specialized educational success metrics
Business Value
Efficiency Gains
Faster iteration on prompt designs for educational applications
Cost Savings
Reduced time in manual prompt evaluation and optimization
Quality Improvement
More effective educational prompts based on empirical testing
  1. Prompt Management
  2. The instructor-tuned LLM's specialized prompts require version control and systematic management for educational settings
Implementation Details
Create a library of versioned educational prompts with specific guardrails and instructional patterns
Key Benefits
• Consistent educational experience across students • Easy modification and improvement of prompts • Collaborative prompt development among educators
Potential Improvements
• Add educational-specific prompt templates • Implement role-based access for educators • Create prompt performance analytics dashboard
Business Value
Efficiency Gains
Streamlined management of educational prompts across courses
Cost Savings
Reduced overhead in maintaining multiple prompt versions
Quality Improvement
Better standardization of educational AI interactions

The first platform built for prompt engineering