Token 计数

AgentScope 在 agentscope.token 模块下提供了 token 计数功能，用于计算给定消息中 的 token 数量，允许开发者在调用 LLM API 前预估 token 数量。

具体而言，可用的 token 计数器如下：

LLM API	类	实现方式	支持图像数据	支持工具
Anthropic	AnthropicTokenCounter	官方 API	✅	✅
OpenAI	OpenAITokenCounter	本地计算	✅	✅
Gemini	GeminiTokenCounter	官方 API	✅	✅
HuggingFace	HuggingFaceTokenCounter	基于Tokenizer计算	取决于模型	取决于模型

小技巧

格式化器模块已集成了 token 计数器以支持提示截断。更多详细信息请参考 提示词格式化 部分。

备注

• 对于 DashScope 模型，目前 dashscope 库不提供 token 计数 API。因此我们建议使用 HuggingFace token 计数器代替。

• 对于 OpenAI 模型，由于官方未提供 token 计数 API，因此可能存在与官方计算结果不一致的情况。

下面展示使用 OpenAI token 计数器计算 token 数量的示例：

import asyncio
from agentscope.token import OpenAITokenCounter


async def example_token_counting():
    # 示例消息
    messages = [
        {"role": "user", "content": "Hello!"},
        {"role": "assistant", "content": "Hi, how can I help you?"},
    ]

    # OpenAI token 计数
    openai_counter = OpenAITokenCounter(model_name="gpt-4.1")
    n_tokens = await openai_counter.count(messages)

    print(f"Token 数量: {n_tokens}")


asyncio.run(example_token_counting())

Traceback (most recent call last):
  File "/Users/hjt0309/Documents/AI_project/agentscope_doc/agentscope-main/docs/tutorial/zh_CN/src/task_token.py", line 69, in <module>
    asyncio.run(example_token_counting())
  File "/opt/miniconda3/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/hjt0309/Documents/AI_project/agentscope_doc/agentscope-main/docs/tutorial/zh_CN/src/task_token.py", line 64, in example_token_counting
    n_tokens = await openai_counter.count(messages)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hjt0309/Documents/AI_project/agentscope_doc/agentscope-main/src/agentscope/token/_openai_token_counter.py", line 332, in count
    encoding = tiktoken.get_encoding("o200k_base")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/tiktoken/registry.py", line 86, in get_encoding
    enc = Encoding(**constructor())
                     ^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/tiktoken_ext/openai_public.py", line 96, in o200k_base
    mergeable_ranks = load_tiktoken_bpe(
                      ^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/tiktoken/load.py", line 144, in load_tiktoken_bpe
    contents = read_file_cached(tiktoken_bpe_file, expected_hash)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/tiktoken/load.py", line 63, in read_file_cached
    contents = read_file(blobpath)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/tiktoken/load.py", line 24, in read_file
    resp = requests.get(blobpath)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

进一步阅读

• 提示词格式化