Have you ever felt like your writing style just doesn't *get* the search results you want? New research suggests you might be right. A study presented at WSDM '25 reveals that the writing style of both documents and search queries can significantly impact the fairness and accuracy of information retrieval systems. The study found that state-of-the-art AI models used to understand text, known as "universal text embeddings," exhibit biases toward certain writing styles. For example, many models prefer clear, concise language, similar to Wikipedia articles, while informal or emotive styles are often ranked lower. Surprisingly, even the use of emojis can influence search results! This bias isn't limited to the documents themselves. The study also found that the way you phrase your search query can affect the results. Many models try to match the writing style of the query with the documents they retrieve, meaning a formally worded query is more likely to surface formally written articles. However, this isn't always the case, and certain models consistently prefer some writing styles over others, no matter how the query is phrased. One particularly interesting finding relates to the use of Large Language Models (LLMs) in generating synthetic training data. Models trained on this synthetic data showed a distinct preference for the style of the generated text, raising concerns about the potential for reinforcing existing biases. Even more intriguing, the study explored how different LLMs, like GPT and Llama, have their own unique "answer styles." When these models are used to generate answers, their varying styles can affect how those answers are scored by information retrieval systems. This research has significant implications for the future of search. If search engines are biased toward certain writing styles, it could silence or marginalize particular voices and perspectives online. The challenge now is to develop more balanced and fair text embedding models that don't discriminate against any writing style, ensuring a more equitable and inclusive information landscape for everyone.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team. Get started for free.
Question & Answers
How do universal text embeddings exhibit bias towards different writing styles in search systems?
Universal text embeddings show systematic preferences for certain writing styles through their ranking mechanisms. Technically, these models create vector representations that favor formal, Wikipedia-style content while potentially discriminating against informal or emotive writing styles. The bias manifests in three main ways: 1) Preferential ranking of concise, formal content, 2) Style-matching between queries and retrieved documents, and 3) Reinforcement of biases when trained on LLM-generated synthetic data. For example, if you search for medical information using casual language, the system might overlook valuable content written in a more conversational style, even if it's more accessible to general readers.
How can writing style affect your website's search visibility?
Writing style significantly impacts your website's search visibility through AI-powered ranking systems. These systems tend to favor clear, concise, and formal writing similar to Wikipedia articles, which could affect how your content ranks in search results. To optimize visibility, consider: 1) Using clear, professional language for important pages, 2) Maintaining consistency in writing style across related content, and 3) Balancing formal and informal tones based on your target audience. For instance, a medical website might benefit from using more formal language for clinical information while keeping blog posts conversational.
What role do emojis play in search engine optimization?
Emojis can influence how search engines process and rank your content, though their impact isn't always positive. The research reveals that informal elements like emojis can affect how AI models interpret and rank content, potentially leading to lower rankings in some cases. When using emojis in content, consider: 1) Their appropriateness for your target audience and content type, 2) The platform where the content will appear, and 3) The potential impact on search visibility. For example, while emojis might work well for social media posts or casual blog content, they might be less appropriate for professional or technical documentation.
PromptLayer Features
Testing & Evaluation
Evaluating writing style biases across different prompt formulations and LLM responses requires systematic testing frameworks
Implementation Details
Set up A/B tests comparing response rankings across different writing styles and prompt formulations, implement scoring metrics for style diversity, create regression tests for bias detection
Key Benefits
• Systematic bias detection across different writing styles
• Quantifiable metrics for response diversity
• Reproducible testing across model versions
.png){kind=icon}
.svg)
.svg)
.svg)
.svg)
