我正在尝试为文本创建嵌入并将其存储在Supabase的表中。

列有：

document -- Text（文档中的每段文本）

chunk_id -- UUID

embeddings -- vector

doc_id -- UUID（来自文档表的外键，该文档表包含文档元数据）

这个函数将文本转换为清洗后的文本，以便我可以使用这些清洗后的文本来创建嵌入向量：

这个函数使用 Gemini 模型将文本转换为适合向量化的文本。我也使用了正则表达式，但仍然得到相同的错误。

def genai_text_conversion(text):
     # Remove null characters
    text = text.replace('\u0000', '')
    text = re.sub(r'[\x00-\x1f\x7f-\x9f]', '', text)

    prompt = f"""
    Clean the following text for embedding in pgvector. Remove punctuation, HTML, and control characters.
    Convert to lowercase, remove stop words, and stem words. Return only the cleaned, space-separated text.

    Text: "{text}"
    Cleaned text:
    """

    try:
        response = client.models.generate_content(
            model="gemini-2.5-flash-lite",
            contents=types.Content(
                role="user",
                parts=[types.Part(text=prompt)]
            )
        )
        return response.candidates[0].content.parts[0].text.strip()
    except Exception as e:
        print("Error in Gemini:", e)
        return text  # Fallback to original cleaned text

下面是上面函数的另一个版本，在这里我使用了 nltk、stopwords、porterstemmer、regex。还是不起作用

def clean_text(text):

    text = text.replace('\u0000', '')
    # Or remove all control characters (including nulls)
    text = re.sub(r'[\x00-\x1f\x7f-\x9f]', '', text) 

    words = text.split()
    re_punc = re.compile('[%s]' % re.escape(string.punctuation))
    stripped = [re_punc.sub('', word) for word in words]
# remove the text which are not printable
    re_print = re.compile('[^%s]' % re.escape(string.printable))
    result = [re_print.sub('', word) for word in stripped]
    result = [word.lower() for word in result]
# remove words with one character
    result = [word for word in result if len(word) > 1]

    # remove stop words
    result = [word for word in result if word not in set(stop_words)]

    # stemming
    result = [ps.stem(word) for word in result]

    # remove words, which are not made of alphabet alone
    result = [word for word in result if word.isalpha()]

    cleaned_text = ' '.join(result)


    return cleaned_text

这个函数是在对清洗后的文本创建嵌入并存储到Supabase数据库：

在使用LangChain的文本分割器对文本进行分块之后，我遍历了每段文本。文本随后被转换成清洗后的文本（使用上述函数），然后转换成向量。每一行包含doc_id（每个文档的外键）。一个文档的每个块有多段。每个块都有块ID和嵌入向量。

def create_embeddings_and_store(uploaded_doc_id, chunked_text):
    print("inside embedding creation function")
    try:
        for text in chunked_text:

         # pre embedding text cleaning:
         print("text:", text)
         # convert text , suitable for vectorization
         c_text = genai_text_conversion(text)
         print("cleaned text:", c_text)
    #      cleaned_text = clean_text(text)
    #      print("cleaned text:", cleaned_text)
         # create embeddin
         query_result = embedding.embed_query(c_text)
         # insert into supabase
         chunk_id = str(uuid.uuid4())
         document = text
         doc_id = str(uploaded_doc_id)
         new_row = {'document':str(document),
                'chunk_id':chunk_id,
                'embeddings':query_result,
                'doc_id':doc_id}
         supabase.table('document_vector').insert(new_row).execute()

        response = supabase.table('doc-metadata').select('*').execute()
        if response:
            return response
    except Exception as e:
        print(e)
        return "error in storing embedding"

这是 ERROR：

我认为问题出在这行代码。
Inside function create_embeddings_and_store() --> inside new_row --> 'document':str(document),

图片显示的是 不受支持的Unicode转义错误。

cleaned text: align posit step comput time gener sequen hidden
cleaned text: state ht input posit
{'message': 'unsupported Unicode escape sequence', 'code': '22P05', 'hint': None, 'details': '\\u0000 cannot be converted to text.'}       
error in storing embedding
INFO:     127.0.0.1:59939 - "POST /upload HTTP/1.1" 200 OK
cleaned text: state ht input posit
{'message': 'unsupported Unicode escape sequence', 'code': '22P05', 'hint': None, 'details': '\\u0000 cannot be converted to text.'}       
error in storing embedding

我错在哪里？

解决方案

好，错误出现在这里：

'tocument': str(document)

但在此之前，先解决这个：

text = text.replace('', '')
text = re.sub(r'[x00-x1fx7f-x9f]', '', text)

并非所有内容都是字面字符串。有时它已经是实际字节解码后的形式，而不是 ""。

需要清理任何实际字节：

text = text.encode('utf-8', 'ignore').decode('utf-8')
text = text.replace('\x00', '')

现在让我们回到你最大的问题。这纯粹是逻辑问题。你在清理文本，但你却把脏文本保存了：

c_text = genai_text_conversion(text) # clean text
...
document = text # Here it should be c_text