在筛选后的嵌套文档上进行KNN搜索
我索引中的文档代表各种书籍。一本书的所有段落都存储在名为texts的嵌套文档下。
"texts": {
"type": "nested",
"properties": {
"part": {
"type": "keyword"
},
"text": {
"type": "text"
},
"text_vector": {
"type": "dense_vector",
"element_type": "float",
"dims": 384,
"index": true,
"similarity": "cosine"
},
- part：表示一个小节标题
- text：该小节中的文本
- text_vector：文本的向量表示
我想基于part元数据对嵌套文档进行预过滤/筛选，并在它们上执行knn搜索。使用下面的查询，包含操作工作得非常好
"knn": [
{
"field": "texts.text_vector",
"query_vector": [
-0.00758091127499938,
-0.06187092140316963,
-0.056799083948135376,
........
],
"k": 10,
"num_candidates": 500,
"filter": [
{
"bool": {
"must": [
{
"match_phrase": {
"texts.part": {
"query": "Part1"
}
}
}
]
}
}
],
"inner_hits": {
"size": 10
}
}
]
但排除过滤不会起作用
"knn": [
{
"field": "texts.text_vector",
"query_vector": [
-0.00758091127499938,
-0.06187092140316963,
-0.056799083948135376,
........
],
"k": 10,
"num_candidates": 500,
"filter": [
{
"bool": {
"must_not": [
{
"match_phrase": {
"texts.part": {
"query": "Part1"
}
}
}
]
}
}
],
"inner_hits": {
"size": 10
}
}
]
我的参考资料：kNN search in Elasticsearch。
尝试了term、match等等，而不是match_phrase。没有一个起作用。
###更新###
我在response/inner_hits中查找那些没有part="part1" 的文档。
** 更新 **
我认为术语 nested document 容易让人困惑。这里所说的“嵌套文档”，是指Elasticsearch在我们使用下面的方式建立索引时创建的文档
"texts": [
{ ----- nested document 1
"part": "Part1",
"text": "I dont want this"
},
{ ----- nested document 2
"part": "Part2",
"text": "I need this"
}
]
@Paulo我在这里用你的示例来帮助理解。
我的索引映射包含书（主文档）以及它的文本（嵌套文档）
PUT /book_vector_index
{
"mappings": {
"properties": {
"title": { "type": "keyword" },
"texts": {
"type": "nested",
"properties": {
"part": { "type": "keyword" },
"text": { "type": "text" },
"text_vector": {
"type": "dense_vector",
"dims": 3,
"index": true,
"similarity": "cosine"
}
}
}
}
}
}
我正在向其中索引一些书。
POST /book_vector_index/_bulk
{ "index": { "_id": "1" } }
{ "title": "Book One", "texts": [ { "part": "Part1", "text": "I dont want this", "text_vector": [1.0, 0.0, 0.0] }, { "part": "Part2", "text": "I need this", "text_vector": [0.0, 1.0, 0.0] } ] }
{ "index": { "_id": "2" } }
{ "title": "Book Two", "texts": [ { "part": "Part1", "text": "I dont want this", "text_vector": [0.9, 0.1, 0.0] }, { "part": "Part2", "text": "I need this", "text_vector": [0.0, 1.0, 0.0] } ] }
{ "index": { "_id": "3" } }
{ "title": "Book Three", "texts": [ { "part": "Part1", "text": "I dont want this", "text_vector": [0.9, 0.1, 0.0] }, { "part": "Part2", "text": "I need this", "text_vector": [0.1, 0.9, 0.0] } ] }
{ "index": { "_id": "4" } }
{ "title": "Book Four", "texts": [{ "part": "Part1", "text": "I dont want this", "text_vector": [0.9, 0.1, 0.0] }, { "part": "Part2", "text": "I need this", "text_vector": [0.1, 0.9, 0.0] }, { "part": "Part3", "text": "I am fine", "text_vector": [0.0, 0.0, 1.0] } ] }
{ "index": { "_id": "5" } }
{ "title": "Book Five", "texts": [ { "part": "Part1", "text": "Exclude me", "text_vector": [0.8, 0.2, 0.0] },{ "part": "Part2", "text": "I need this", "text_vector": [0.1, 0.9, 0.0] }, { "part": "Part3", "text": "I am fine", "text_vector": [0.0, 0.0, 1.0] } ] }
我对 title 字段以及在我的knn查询中匹配的书的段落/文本感兴趣。但……我不想对Part = Part 1执行KNN。就像我不关心“前言”中讨论的内容。我只对其实际内容感兴趣。
一种方法是把我感兴趣的 Part 列出来。
GET /book_vector_index/_search
{
"_source": [
"title",
"inner_hits"
],
"knn": {
"field": "texts.text_vector",
"query_vector": [
0,
1,
0
],
"k": 10,
"num_candidates": 100,
"filter": {
"bool": {
"should": [
{
"term": {
"texts.part": "Part2"
}
},
{
"term": {
"texts.part": "Part3"
}
}
]
}
},
"inner_hits": {}
}
}
如果我们查看 inner_hits，可以看到 part 2来自1， part 2来自2， part 2来自3， part 2 * 3来自4 与5 最后，我排除了part 1
{
"took": 9,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 5,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "book_vector_index",
"_id": "1",
"_score": 1,
"_source": {
"title": "Book One"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "book_vector_index",
"_id": "1",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 1,
"_source": {
"part": "Part2",
"text": "I need this"
}
}
]
}
}
}
},
{
"_index": "book_vector_index",
"_id": "2",
"_score": 1,
"_source": {
"title": "Book Two"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "book_vector_index",
"_id": "2",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 1,
"_source": {
"part": "Part2",
"text": "I need this"
}
}
]
}
}
}
},
{
"_index": "book_vector_index",
"_id": "3",
"_score": 0.99694574,
"_source": {
"title": "Book Three"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.99694574,
"hits": [
{
"_index": "book_vector_index",
"_id": "3",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 0.99694574,
"_source": {
"part": "Part2",
"text": "I need this"
}
}
]
}
}
}
},
{
"_index": "book_vector_index",
"_id": "4",
"_score": 0.99694574,
"_source": {
"title": "Book Four"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 0.99694574,
"hits": [
{
"_index": "book_vector_index",
"_id": "4",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 0.99694574,
"_source": {
"part": "Part2",
"text": "I need this"
}
},
{
"_index": "book_vector_index",
"_id": "4",
"_nested": {
"field": "texts",
"offset": 2
},
"_score": 0.5,
"_source": {
"part": "Part3",
"text": "I am fine"
}
}
]
}
}
}
},
{
"_index": "book_vector_index",
"_id": "5",
"_score": 0.99694574,
"_source": {
"title": "Book Five"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 0.99694574,
"hits": [
{
"_index": "book_vector_index",
"_id": "5",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 0.99694574,
"_source": {
"part": "Part2",
"text": "I need this"
}
},
{
"_index": "book_vector_index",
"_id": "5",
"_nested": {
"field": "texts",
"offset": 2
},
"_score": 0.5,
"_source": {
"part": "Part3",
"text": "I am fine"
}
}
]
}
}
}
}
]
}
}
如果我不知道Part的名称怎么办？这时我会使用 must_not
GET /book_vector_index/_search
{
"_source": [
"title",
"inner_hits"
],
"knn": {
"field": "texts.text_vector",
"query_vector": [
0,
1,
0
],
"k": 10,
"num_candidates": 100,
"filter": {
"bool": {
"must_not": [
{
"term": {
"texts.part": "Part1"
}
}
]
}
},
"inner_hits": {}
}
}
但 inner_hits 中有 Part 1
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 5,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "book_vector_index",
"_id": "1",
"_score": 1,
"_source": {
"title": "Book One"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "book_vector_index",
"_id": "1",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 1,
"_source": {
"part": "Part2",
"text": "I need this"
}
},
{
"_index": "book_vector_index",
"_id": "1",
"_nested": {
"field": "texts",
"offset": 0
},
"_score": 0.5,
"_source": {
"part": "Part1",
"text": "I dont want this"
}
}
]
}
}
}
},
{
"_index": "book_vector_index",
"_id": "2",
"_score": 1,
"_source": {
"title": "Book Two"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "book_vector_index",
"_id": "2",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 1,
"_source": {
"part": "Part2",
"text": "I need this"
}
},
{
"_index": "book_vector_index",
"_id": "2",
"_nested": {
"field": "texts",
"offset": 0
},
"_score": 0.5560008,
"_source": {
"part": "Part1",
"text": "I dont want this"
}
}
]
}
}
}
},
{
"_index": "book_vector_index",
"_id": "3",
"_score": 0.99694574,
"_source": {
"title": "Book Three"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 0.99694574,
"hits": [
{
"_index": "book_vector_index",
"_id": "3",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 0.99694574,
"_source": {
"part": "Part2",
"text": "I need this"
}
},
{
"_index": "book_vector_index",
"_id": "3",
"_nested": {
"field": "texts",
"offset": 0
},
"_score": 0.5560008,
"_source": {
"part": "Part1",
"text": "I dont want this"
}
}
]
}
}
}
},
{
"_index": "book_vector_index",
"_id": "4",
"_score": 0.99694574,
"_source": {
"title": "Book Four"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 0.99694574,
"hits": [
{
"_index": "book_vector_index",
"_id": "4",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 0.99694574,
"_source": {
"part": "Part2",
"text": "I need this"
}
},
{
"_index": "book_vector_index",
"_id": "4",
"_nested": {
"field": "texts",
"offset": 0
},
"_score": 0.5560008,
"_source": {
"part": "Part1",
"text": "I dont want this"
}
},
{
"_index": "book_vector_index",
"_id": "4",
"_nested": {
"field": "texts",
"offset": 2
},
"_score": 0.5,
"_source": {
"part": "Part3",
"text": "I am fine"
}
}
]
}
}
}
},
{
"_index": "book_vector_index",
"_id": "5",
"_score": 0.99694574,
"_source": {
"title": "Book Five"
},
"inner_hits": {
"texts": {
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 0.99694574,
"hits": [
{
"_index": "book_vector_index",
"_id": "5",
"_nested": {
"field": "texts",
"offset": 1
},
"_score": 0.99694574,
"_source": {
"part": "Part2",
"text": "I need this"
}
},
{
"_index": "book_vector_index",
"_id": "5",
"_nested": {
"field": "texts",
"offset": 0
},
"_score": 0.62265044,
"_source": {
"part": "Part1",
"text": "Exclude me"
}
},
{
"_index": "book_vector_index",
"_id": "5",
"_nested": {
"field": "texts",
"offset": 2
},
"_score": 0.5,
"_source": {
"part": "Part3",
"text": "I am fine"
}
}
]
}
}
}
}
]
}
}
######## Works -- But need to evaluate
POST /book_vector_index/_search
{
"_source": [
"title"
],
"query": {
"bool": {
"must": [
{
"nested": {
"path": "texts",
"query": {
"bool": {
"must": [
{
"knn": {
"field": "texts.text_vector",
"query_vector": [
0,
1,
0
],
"k": 10,
"num_candidates": 100,
"filter": {
"bool": {
"must_not": [
{
"term": {
"texts.part": "Part1"
}
}
]
}
}
}
}
],
"must_not": [
{
"term": {
"texts.part": {
"value": "Part1"
}
}
}
]
}
},
"inner_hits": {
"size": 10
}
}
}
]
}
}
}
解决方案
TL;DR;
我们应该把问题从 filter out x 重新框定到 match everything but x
这将使x 的分数降低，不会出现在结果中。
Example
Search Query
如果我们想排除 Part1，我们实际上想要搜索 [Part2, Part3 .....]
虽然这可能很繁琐，但我决定把 part 像在 integer 那样存储起来，并使用一个区间。
GET /repro_knn_index/_search
{
"knn": {
"field": "texts.text_vector",
"query_vector": [
0,
0,
1
],
"k": 10,
"num_candidates": 100,
"filter": {
"bool": {
"should": [
{
"range": {
"texts.part": {
"gt": 1
}
}
},
{
"range": {
"texts.part": {
"lt": 1
}
}
}
]
}
},
"inner_hits": {
"_source": false,
"explain": true,
"fields": [
"texts.text",
"texts.part"
]
}
}
}
Set Up
PUT /repro_knn_index
{
"mappings": {
"properties": {
"title": { "type": "keyword" },
"parts": { "type": "integer" },
"texts": {
"type": "nested",
"properties": {
"part": { "type": "integer", "copy_to": "parts" },
"text": { "type": "text" },
"text_vector": {
"type": "dense_vector",
"dims": 3,
"index": true,
"similarity": "cosine"
}
}
}
}
}
}
POST /repro_knn_index/_bulk
{ "index": { "_id": "1" } }
{ "title": "Book One", "texts": [ { "part": 1, "text": "Hello world", "text_vector": [1.0, 0.0, 0.0] }, { "part": 2, "text": "Goodbye world", "text_vector": [0.0, 1.0, 0.0] } ] }
{ "index": { "_id": "2" } }
{ "title": "Book Two", "texts": [ { "part": 1, "text": "Elasticsearch nested", "text_vector": [0.9, 0.1, 0.0] } ] }
{ "index": { "_id": "3" } }
{ "title": "Book Three", "texts": [ { "part": 2, "text": "Knn search", "text_vector": [0.1, 0.9, 0.0] } ] }
{ "index": { "_id": "4" } }
{ "title": "Book Four", "texts": [ { "part": 3, "text": "Filtered vector", "text_vector": [0.0, 0.0, 1.0] } ] }
{ "index": { "_id": "5" } }
{ "title": "Book Five", "texts": [ { "part": 1, "text": "Exclude me", "text_vector": [0.8, 0.2, 0.0] } ] }
{ "index": { "_id": "6" } }
!! Archive !!
Filtering out with nested fields
因为嵌套文档被作为独立文档进行索引，它们只能在嵌套查询、嵌套/反向嵌套聚合，或嵌套的inner hits的作用域内被访问。
嵌套查询会把嵌套字段对象当作独立文档来进行搜索。如果某个对象匹配搜索，嵌套查询会返回根父文档。
这很可能是导致混淆的原因。 对于以下文档
{ "title": "Book One",
"texts": [
{ "part": "Part1", "text": "Hello world", "text_vector": [1.0, 0.0, 0.0] },
{ "part": "Part2", "text": "Goodbye world", "text_vector": [0.0, 1.0, 0.0] }]
}
以及类似如下的查询
{
"query": {
"nested": {
"path": "texts",
"query": {
"bool": {
"must": [
{
"term": {
"texts.part": "Part1"
}
}
]
}
}
}
}
}
我们不应该返回该文档。但是由于elasticsearch "split" 这个文档为2 个子文档进行搜索 independently，主文档确实出现在搜索结果中。
解决此限制的方法？
在字段 texts.part 的映射中使用 copy_to。我们将把一个名为 parts 的字段放在文档根部，用来存放所有包含的parts。
PUT /repro_knn_index
{
"mappings": {
"properties": {
"title": { "type": "keyword" },
"parts": { "type": "keyword" },
"texts": {
"type": "nested",
"properties": {
"part": { "type": "keyword", "copy_to": "parts" },
"text": { "type": "text" },
"text_vector": {
"type": "dense_vector",
"dims": 3,
"index": true,
"similarity": "cosine"
}
}
}
}
}
}
我们就可以通过以下方式简单地过滤掉文档：
GET /repro_knn_index/_search
{
"_source": [
false
],
"knn": {
"field": "texts.text_vector",
"query_vector": [
0,
1,
0
],
"k": 10,
"num_candidates": 100,
"filter": {
"bool": {
"must_not": [
{
"term": {
"parts": "Part1"
}
}
]
}
}
}
}
Example
Query
GET /repro_knn_index/_search
{
"knn": {
"field": "texts.text_vector",
"query_vector": [
0,
1,
0
],
"k": 10,
"num_candidates": 100,
"filter": {
"nested": {
"path": "texts",
"query": {
"bool": {
"must_not": [
{
"match_phrase": {
"texts.part": "Part1"
}
}
]
}
}
}
}
}
}
返回如下：
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "repro_knn_index",
"_id": "1",
"_score": 1,
"_source": {}
},
{
"_index": "repro_knn_index",
"_id": "3",
"_score": 0.99694574,
"_source": {}
},
{
"_index": "repro_knn_index",
"_id": "4",
"_score": 0.5,
"_source": {}
}
]
}
}
这确实排除了所有只有单条条目的文档 Part1。成功排除了文档 2 和 5。
然而 1 仍然存在，因为它有多条条目。（上文已解释）
Set Up
PUT /repro_knn_index
{
"mappings": {
"properties": {
"title": { "type": "keyword" },
"texts": {
"type": "nested",
"properties": {
"part": { "type": "keyword" },
"text": { "type": "text" },
"text_vector": {
"type": "dense_vector",
"dims": 3,
"index": true,
"similarity": "cosine"
}
}
}
}
}
}
POST /repro_knn_index/_bulk
{ "index": { "_id": "1" } }
{ "title": "Book One", "texts": [ { "part": "Part1", "text": "Hello world", "text_vector": [1.0, 0.0, 0.0] }, { "part": "Part2", "text": "Goodbye world", "text_vector": [0.0, 1.0, 0.0] } ] }
{ "index": { "_id": "2" } }
{ "title": "Book Two", "texts": [ { "part": "Part1", "text": "Elasticsearch nested", "text_vector": [0.9, 0.1, 0.0] } ] }
{ "index": { "_id": "3" } }
{ "title": "Book Three", "texts": [ { "part": "Part2", "text": "Knn search", "text_vector": [0.1, 0.9, 0.0] } ] }
{ "index": { "_id": "4" } }
{ "title": "Book Four", "texts": [ { "part": "Part3", "text": "Filtered vector", "text_vector": [0.0, 0.0, 1.0] } ] }
{ "index": { "_id": "5" } }
{ "title": "Book Five", "texts": [ { "part": "Part1", "text": "Exclude me", "text_vector": [0.8, 0.2, 0.0] } ] }
站内所有文章版权归属LeftHeroAI导航站，无授权禁止任何主体转载、抄袭、复制内容，亦不得私自架设镜像站点。一经侵权，本站将通过法律途径追责。
相关文章
汇集全球数千款 AI 工具，每日持续更新，配套 AI 开发学习资源，轻松用 AI 提升工作效率。快捷键 Ctrl+D/⌘+D 一键收藏本站。