Large Language Models (LLMs) are revolutionizing how we interact with technology, but their immense computational demands present a significant hurdle. Training and running these massive models requires substantial resources, limiting accessibility and hindering further innovation. However, a new approach called GFormer is emerging as a potential game-changer, promising to supercharge LLM performance on specialized hardware like Gaudi processors. Gaudi processors, developed by Habana Labs, offer a unique heterogeneous architecture combining a Matrix Multiplication Engine (MME) and Tensor Processing Cores (TPCs). While seemingly ideal for the matrix-heavy operations within LLMs, traditional Transformer models aren't fully optimized to exploit this hardware blend. Specifically, the Softmax function, essential in attention mechanisms, becomes a bottleneck on Gaudi, particularly with long text sequences. GFormer tackles this challenge by cleverly integrating two different attention mechanisms: sparse attention and linear attention. Sparse attention focuses processing on the most relevant parts of the input, reducing computational load. Linear attention, by approximating the Softmax function, simplifies calculations without significant accuracy loss. This combined approach allows GFormer to effectively distribute the workload between the MME and TPCs, maximizing hardware utilization and minimizing the Softmax bottleneck. The innovation doesn't stop there. GFormer introduces a custom-designed windowed local-context self-attention kernel for the TPCs, ensuring efficient handling of sparse attention. It also implements an optimized outer product kernel, further balancing computations between the MME and TPCs. A sophisticated partitioning algorithm dynamically assigns workloads to each processing unit, ensuring optimal performance. Experimental results on GPT and Vision Transformer (ViT) models demonstrate GFormer's effectiveness. GFormer achieves up to a 2x speedup on GPT models and a 2.2x speedup on ViT models compared to standard implementations on Gaudi. Remarkably, these performance gains come with minimal impact on accuracy. Even compared to GPUs, GFormer on Gaudi demonstrates superior performance in certain scenarios, opening up exciting possibilities for more efficient and accessible LLMs. Future work will focus on extending GFormer to multiple Gaudi processors, enhancing its applicability to other hardware platforms, and exploring its potential in other machine learning tasks. This research signals a significant step towards unlocking the full potential of LLMs, paving the way for even more sophisticated and impactful AI applications.