我正在使用 langchain==0.3.24.

我把一张图片转换成base64，现在想把它放进chatPromptTemplate：

...

    image_data = ... #converting image to base64 and store it in the variable
    generation_prompt = ChatPromptTemplate.from_messages(
     [
            (
                "system",
                """
                You are an expert assistant. Your task is to analyze the food items
displayed in {{im64}} and provide a detailed information about them.
"""

            ),
            MessagesPlaceholder(variable_name = ["im64",str(image_data)]),
        ])

但输出是：

content="Error invoking tool 'tool_x' with kwargs {'image_data': 'description of the image'} with error: 
variable_name: Input should be a valid string 
Please fix the error and try again."

有人能解释这个错误吗？

解决方案

MessagePlaceholder 只是变量名的占位符，
稍后你可以使用这个变量来发送文本或图片。

关于 MessagePlaceholder 的文档显示

MessagesPlaceholder("history", ...)

随后

prompt.format_messages(history=[...])

或

prompt.invoke({"history": [...]})

另一个问题是，字符串 im64:image_data_as_base64 被视为普通文本，而不是图片。Langchain关于 Messages 的文档建议如下：

HumanMessage(
    content=[
        {
            "type": "image",
            "base64": image_data,
            "mime_type": "image/jpeg",
        }
    ]
)

但对于较旧的Langchain 0.3.x，可能需要 image_url

HumanMessage(
    content=[
        {
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{image_data}"},
        },
    ]
)

当然 image_url 在更新的1.x版本中也可以使用

我编写了完整可运行的代码——至少它不会报错（最终会显示一些关于弃用元素的警告），并且它能够识别图片上的对象。

但我用 langchain=1.2.13 进行了测试，因此必须使用 langchain_classic，而不是 langchain（langchain_classic 已经与 langchain 一起预装）

# --- info ---

import langchain
print("langchain:", langchain.__version__)  # 1.2.13

# --- import ---

# --- for older versions 0.3.x ? ---
# from langchain.prompts.chat import ChatPromptTemplate, MessagesPlaceholder
# from langchain.schema import SystemMessage, HumanMessage
# from langchain.chat_models import ChatOpenAI
# from langchain.chat_models import ChatOllama

# --- for newer versions 1.x ---
from langchain_classic.prompts.chat import ChatPromptTemplate, MessagesPlaceholder
from langchain_classic.schema import SystemMessage, HumanMessage
# from langchain_classic.chat_models import ChatOpenAI  # warning to use `langchain_community`
# from langchain_classic.chat_models import ChatOllama  # warning to use `from langchain_ollama import ChatOllama`

# --- suggested by warnings ---
# from langchain_openai import ChatOpenAI
from langchain_ollama import ChatOllama

import base64

print("--- after imports ---")

# --- image ---

with open("food.jpg", "rb") as f:
    image_bytes = f.read()

image_data = base64.b64encode(image_bytes).decode("utf-8")

# --- ChatPromptTemplate ---

prompt = ChatPromptTemplate.from_messages(
    [
        #(
        #    "system",
        #    "You are an expert assistant. Your task is to analyze the food items displayed in {{im64}} and provide a detailed information about them.",
        #),
        SystemMessage("You are an expert assistant. Your task is to analyze the food items displayed in {{im64}} and provide a detailed information about them."),
        MessagesPlaceholder(variable_name="image_messages"),  # placeholder for later
    ]
)

# --- format message with image ---

formatted_prompt = prompt.format_prompt(
    image_messages=[  # <-- the same name as in `MessagesPlaceholder`
        #HumanMessage(content=["im64", image_data])  # it treats it as text, not image
        #HumanMessage(content=f"im64:{image_data}")  # it treats it as text, not image
        HumanMessage(
           content=[
             # {"type: "text", "text": "describe this image in German"},
             # --- for older version 0.3.x but works also with 1.x ---
             # {
             #   "type": "image_url",
             #   "image_url": {"url": f"data:image/jpeg;base64,{image_data}"},
             # },
             # --- only for newer version 1.x ---
             {
               "type": "image",
               "base64": image_data,
               "mime_type": "image/jpeg",
             },
           ]
        )
    ]
)

messages_for_llm = formatted_prompt.to_messages()

# --- invoke ---

# llm = ChatOpenAI(model_name="gpt-4")  # or "gpt-3.5-turbo"
llm = ChatOllama(model="llama3.2-vision:11b")
response = llm.invoke(messages_for_llm)

print(response.content)

似乎提示中的 {{im64}} 没有用处。