Large language models (LLMs) have become incredibly powerful, generating human-like text that's both impressive and, at times, concerning. But how do we truly measure their capabilities and potential pitfalls? Traditional automated metrics fall short, highlighting the crucial role of human evaluation in the age of generative AI. However, human judgment isn't without its flaws. Our perceptions are colored by cognitive biases, influenced by aesthetics, and prone to inconsistencies. This is where the ConSiDERS-The-Human evaluation framework comes in. ConSiDERS emphasizes a multidisciplinary approach, drawing on insights from user experience research and cognitive psychology to create more robust and reliable evaluation methods. The framework's six pillars—Consistency, Scoring Criteria, Differentiating capabilities, User Experience, Responsible AI, and Scalability—provide a roadmap for navigating the complexities of human evaluation. For example, understanding how cognitive biases like the 'halo effect' can lead us to conflate fluency with truthfulness is essential for designing effective scoring criteria. Similarly, recognizing the inherent uncertainty in rating scales like Likert prompts us to explore alternative methods or denoising algorithms. ConSiDERS also stresses the importance of building test sets that truly differentiate LLM capabilities, going beyond standard NLP benchmarks to reflect real-world user scenarios and potential vulnerabilities. Ultimately, the framework aims to make human evaluation more scientific, scalable, and aligned with the responsible development and deployment of increasingly powerful LLMs. This means acknowledging the human element in both the models we evaluate and the evaluators themselves, paving the way for more meaningful insights into the evolving landscape of generative AI.