Large language models (LLMs) have shown promise in various tasks, but their ability to effectively use external APIs remains a challenge. A new benchmark suite, SEAL (Suite for Evaluating API-use of LLMs), aims to thoroughly assess this capability, uncovering the strengths and weaknesses of current LLMs in real-world API interaction scenarios. SEAL addresses limitations of existing benchmarks like ToolBench and APIGen, which suffer from issues like overfitting due to small test sets, lack of multi-step reasoning tasks, and real-time API instability. SEAL standardizes existing benchmarks into a unified format, incorporates an agent system for API retrieval and planning, and tackles API instability by simulating responses with a GPT-4 powered API simulator and caching. This allows for deterministic evaluations and more reliable comparisons. The system evaluates across the entire API-use pipeline, from API retrieval and accurate parameter passing to final response quality and correctness. SEAL's evaluation reveals that while LLMs can perform well on simple API calls, their performance degrades with increasing API pool size and complex queries. Common errors include incorrect API retrieval, particularly in multi-tool scenarios, and inaccurate parameter passing due to difficulties extracting information from complex queries. These findings emphasize the need for better retrieval methods, more robust multi-step planning, and more sophisticated understanding of API documentation by LLMs. SEAL offers a significant advancement in evaluating the practical applicability of LLMs in real-world scenarios, paving the way for more focused research and development to enhance their tool-use capabilities.