The rise of large language models (LLMs) has revolutionized how we interact with technology, but this progress isn't evenly distributed. While English-centric LLMs dominate, languages like Bengali face a unique challenge: do they need dedicated LLMs or can they piggyback on advancements in English models? A new research paper dives deep into this issue, assessing existing models on various Bengali NLP tasks, including translation, summarization, and question-answering. The results are a mixed bag. While English LLMs showed surprising strength in understanding Bengali text, they faltered when generating it. This discrepancy highlights the need for improved tokenization methods that handle Bengali script efficiently. Currently, English tokenizers break down Bengali text into smaller units than necessary, hindering efficiency and performance. Furthermore, machine-translated Bengali datasets introduce biases impacting model evaluation. For instance, the study found that models trained on such data overperform in automated metrics, even when their generated paraphrases aren't superior. So, should we build Bengali-specific LLMs? The research suggests a significant need, but acknowledges the high cost and resource-intensive nature of such an endeavor. Training massive models like LLaMA-3 is a multi-million dollar undertaking, and Bengali lacks the colossal datasets needed to match the scale of English corpora. In the interim, the paper proposes a practical approach: harness the power of advanced translation models in conjunction with existing English LLMs. This two-pronged approach can boost Bengali NLP until dedicated Bengali LLMs become a feasible reality. The research also advocates for the expansion of high-quality Bengali datasets through web scraping, media transcription, and strategic translation efforts. This, along with the development of efficient tokenization methods, forms the next crucial steps towards making Bengali a major player in the LLM revolution.