Revised VSS RFC #15

allenss-amazon · 2024-11-27T17:22:21Z

Replaces and extends the previous PR for VSS module.

Signed-off-by: Allen Samuels <[email protected]>

madolson

Other open questions that would document for completeness:

Will ACLs be supported?
Is active defragmentation supported?
Is the RDB compatible with redis search.
I assume this a subset of redis search functionality. but would be good to document that along with what is missing.
How will slot migration work, if at all. Will we also copy indexes between nodes when using the cli?
Are any core changes required? We talked about memory sharing and some new keysspace notifications.
How does redirecting read requests to the primary work in cluster mode? Can you force strongly consistent reads? By the consistency model do we care?
Are we allowing searcg in Lua and multi-exec? Will it cause latency issues? Does redis allow it?

VSS.md

madolson · 2024-12-02T03:46:15Z

VSS.md

+
+The command returns either an array if successful or an error.
+
+If `NOCONTENT` is specified, then the output is .....


It's a mystery?

Suggested changes:

On success, the first entry in the response array represents the total number of qualified matching elements, followed by one array entry for each matching element. Note that the amount of response entries may differ from the total number of response elements, captured in the response first entry, just if the LIMIT option is specified.

When NOCONTENT is specified, each entry in the response contains only the matching keys. Otherwise, each entry includes the matching key, followed by an array of the returned fields.

madolson · 2024-12-02T03:49:19Z

VSS.md

+
+If `NOCONTENT` is specified, then the output is .....
+
+If `NOCONTENT` is not specified, then the output is an array of (2\*N)+1 entries, where N is the number of keys output from the search. The first entry in the array is the value N which is followed by N pairs of entries, one per key found. Each pair of entries consists of the key name followed by an array which is the result value for that key.


The protocol has n built in, why is it returned as the first argument?

Right, see my suggested changes above.

compatibility with Redisearch

So, we have no idea why RedisSearch added it?

madolson · 2024-12-02T03:52:34Z

VSS.md

+
+1\. **reader-threads:** (Integer) Controls the amount of threads executing queries.  
+2\. **writer-threads:** (Integer) Controls the amount of threads processing index mutations.  
+3\. **use-coordinator:** (boolean) Cluster mode enabler.


There isn't any discussion of the coordinator here, is that in scope for the initial version?

Yes, it's part of the initial version and it's the way to enable cluster mode.

yes. will add.

Followup, is there a reason to disable the coordinator for cluster mode? Should it just automatically get enabled for cluster mode?

madolson · 2024-12-02T03:54:05Z

VSS.md

+- **search\_total\_indexed\_hash\_keys**	(Integer)	Total count of HASH keys for all indexes  
+- **search\_number\_of\_indexes**	(Integer)	Index schema total count  
+- **search\_number\_of\_attributes**	(Integer)	Total count of attributes for all indexes  
+- **search\_failure\_requests\_count**	(Integer)	A count of all failed requests  


What is the definition of a failed request? One that errors?

This captures any ft.search runtime errors including invalid syntax.

Yeah, I think typically those are called errors throughout the codebase.

madolson · 2024-12-02T03:55:47Z

VSS.md

+- **search\_add\_subscription\_successful\_count**	(Integer)	Count of successfully added subscriptions  
+- **search\_add\_subscription\_failure\_count**	(Integer)	Count of failures of adding subscriptions  
+- **search\_add\_subscription\_skipped\_count**	(Integer)	Count of skipped subscription adding processes  
+- **search\_modify\_subscription\_failure\_count**	(Integer)	Count of failed subscription modifications  


Just making one comment, I don't really understand what most of these info fields are supposed to tell end users. Maybe document them like it's for our public documentation?

Still not sure what these subscriptions are, they are only mentioned here.

VSS.md

yairgott · 2024-12-02T21:33:08Z

VSS.md

+
+The command returns either an array if successful or an error.
+
+If `NOCONTENT` is specified, then the output is .....


Suggested changes:

On success, the first entry in the response array represents the total number of qualified matching elements, followed by one array entry for each matching element. Note that the amount of response entries may differ from the total number of response elements, captured in the response first entry, just if the LIMIT option is specified.

When NOCONTENT is specified, each entry in the response contains only the matching keys. Otherwise, each entry includes the matching key, followed by an array of the returned fields.

yairgott · 2024-12-02T21:42:53Z

VSS.md

+
+- **\<index\>** (required): This index name you want to query.  
+- **\<query\>** (required): The query string, see below for details.  
+- **NOCONTENT** (optional): When present, only the resulting key names are returned, no key values are included.  


Missing:
RETURN (optional): Specifies the fields you want to retrieve from your documents, along with any aliases for the returned values. By default, all fields are returned unless the NOCONTENT option is set, in which case no fields are returned. If num is set to 0, it behaves the same as NOCONTENT.

yairgott · 2024-12-02T21:44:14Z

VSS.md

+
+If `NOCONTENT` is specified, then the output is .....
+
+If `NOCONTENT` is not specified, then the output is an array of (2\*N)+1 entries, where N is the number of keys output from the search. The first entry in the array is the value N which is followed by N pairs of entries, one per key found. Each pair of entries consists of the key name followed by an array which is the result value for that key.


Right, see my suggested changes above.

yairgott · 2024-12-02T21:46:12Z

VSS.md

+
+1\. **reader-threads:** (Integer) Controls the amount of threads executing queries.  
+2\. **writer-threads:** (Integer) Controls the amount of threads processing index mutations.  
+3\. **use-coordinator:** (boolean) Cluster mode enabler.


Yes, it's part of the initial version and it's the way to enable cluster mode.

yairgott · 2024-12-02T21:52:15Z

VSS.md

+- **search\_total\_indexed\_hash\_keys**	(Integer)	Total count of HASH keys for all indexes  
+- **search\_number\_of\_indexes**	(Integer)	Index schema total count  
+- **search\_number\_of\_attributes**	(Integer)	Total count of attributes for all indexes  
+- **search\_failure\_requests\_count**	(Integer)	A count of all failed requests  


This captures any ft.search runtime errors including invalid syntax.

VSS.md

yairgott · 2024-12-03T21:06:55Z

Other open questions that would document for completeness:

Will ACLs be supported?

ACLs are not supported in the current version.

Is active defragmentation supported?

Memory allocated by the module is not subject to active defragmentation.

Is the RDB compatible with redis search.

No, the RDB format is not compatible with RediSearch. The module utilizes a proprietary RDB format based on protobuf, which serializes both index metadata and content. In contrast, RediSearch's RDB format only serializes the index metadata.

I assume this a subset of redis search functionality. but would be good to document that along with what is missing.

Yes, we should have a section which enumerates key missing functionality as bullet points to provide clarity.

How will slot migration work, if at all. Will we also copy indexes between nodes when using the cli?

Slot migration is supported. Indexes are cluster level concepts so they are already replicated across the cluster, and not contained within a slot. Keys from the migrated slot trigger keyspace mutation events, which are processed in the same manner as client-originated mutations, except that they do not block the client during execution.

Are any core changes required? We talked about memory sharing and some new keysspace notifications.

While the module can function without engine changes, two engine changes have been introduced to enhance user experience:

Blocking client on keyspace mutation events: Ensures visibility of mutations in consecutive queries on the same connection. This change extends the behavior of the module API RedisModule_BlockClient.
Memory deduplication: Adds new module APIs to enable memory sharing between the module and the engine, reducing duplication.

How does redirecting read requests to the primary work in cluster mode? Can you force strongly consistent reads? By the consistency model do we care?

Valkey doesn't support distributed transactions so I don't see any consistency guarantee with scatter-gather across multiple shards.
Additional context: In cluster mode, a query can be received by any node, which fans out the request to all shards. Each shard processes the query locally and returns results to the initiating node, which selects the top responses across shards. Currently, forcing reads from primary nodes only are not supported. The current logic for instance selection is designed to improve the performance, increase throughput and lower latency, by load balancing between the instances.

Are we allowing searcg in Lua and multi-exec? Will it cause latency issues? Does redis allow it?
Will ACLs be supported?

Multi-exec: Supported, but it introduces significant performance impacts. Mutation processing cannot be parallelized, and queries inside a multi-exec context must wait for preceding mutations to complete.
Lua scripts: The main constrain is that the engine doesn't allow the client to be blocked when issued by a LUA script. Long term, it makes sense to address this in the engine but in the meantime the current state is as follow:
** Mutation processing: mutations are processed but clients are not blocked.
** Queries: issuing ft.search is the context of LUA script would result in an error due to the fact that the client cannot be blocked.

yairgott · 2024-12-03T23:54:57Z

Made some inline editing to bullet #8 regarding to LUA.

Adding gaps relative to RediSearch section Signed-off-by: yairgott <[email protected]>

Fixing heading of the 'Unsupported knobs and control' section Signed-off-by: yairgott <[email protected]>

zhangh43 · 2024-12-06T22:29:10Z

VSS.md

+**Search Operations CUJ** \- as a user, I want to perform search operations using commonly available clients.
+
+- Requirement: support index insertion/mutation/deletion/query operations  
+- Requirement: maintain compatibility with RediSearch VSS APIs, Memorystore APIs, and MemoryDB APIs as much as possible


RedisSearch API is weird. Pinecone/Milvus API is more natural

# create pinecone.create_index(name=index_name, dimension=dimension) # insert index = pinecone.Index(index_name) index.upsert(vectors) # search results = index.query(queries=[query_vector], top_k=5)

I believe developers prefer the latter.

So to clarify the difference:

In the current implementation, you insert keys with other commands and then create an index ontop of a prefix structure.

In the pinecone world, you create an index and then insert objects into the index explicitly.

?

In the proposed implementation, once an index is created, it always reflects the current state of the keys that the index covers. This means that the order of key insertion vs index creation is arbitrary, i.e., you can do them in any order.

madolson · 2024-12-09T19:42:28Z

VSS.md

+
+The following metrics are added to the INFO command.
+
+- **search\_total\_indexed\_hash\_keys**	(Integer)	Total count of HASH keys for all indexes  


Should there be a corresponding JSON variant?

madolson · 2024-12-09T19:45:11Z

VSS.md

+- **search\_hnsw\_create\_exceptions\_count**	(Integer)	Count of HNSW creation exceptions.  
+- **search\_hnsw\_search\_exceptions\_count**	(Integer)	Count of HNSW search exceptions  
+- **search\_hnsw\_remove\_exceptions\_count**	(Integer)	Count of HNSW removal exceptions.  
+- **search\_hnsw\_add\_exceptions\_count**	(Integer)	Count of HNSW addition exceptions.  
+- **search\_hnsw\_modify\_exceptions\_count**	(Integer)	Count of HNSW modification exceptions  


Still unclear, what are exceptions? How do these fails and what are end users supposed to be doing about these. Are these just syntax errors?

madolson · 2024-12-09T19:45:53Z

VSS.md

+- **search\_add\_subscription\_successful\_count**	(Integer)	Count of successfully added subscriptions  
+- **search\_add\_subscription\_failure\_count**	(Integer)	Count of failures of adding subscriptions  
+- **search\_add\_subscription\_skipped\_count**	(Integer)	Count of skipped subscription adding processes  
+- **search\_modify\_subscription\_failure\_count**	(Integer)	Count of failed subscription modifications  


Still not sure what these subscriptions are, they are only mentioned here.

madolson · 2024-12-09T19:48:09Z

VSS.md

+
+1\. **reader-threads:** (Integer) Controls the amount of threads executing queries.  
+2\. **writer-threads:** (Integer) Controls the amount of threads processing index mutations.  
+3\. **use-coordinator:** (boolean) Cluster mode enabler.


Followup, is there a reason to disable the coordinator for cluster mode? Should it just automatically get enabled for cluster mode?

madolson · 2024-12-09T19:49:02Z

VSS.md

+This means for that for data mutation operations, no cross-cluster communication is required, each node simply updates it's local index to reflect the mutation of the local key.
+
+Query operations are accepted by any node in the cluster and that node is responsible for broadcasting the query across the cluster and merging the results for delivery to the client.
+Cross-client communication uses gRPC and protobufs and does not require mainthread interaction.


Is this on a separate port? Does this need to be configurable so that end users can slot it into their system?

removal of links to redis.io Signed-off-by: yairgott <[email protected]>

Revised VSS RFC

6dfd21e

Signed-off-by: Allen Samuels <[email protected]>

allenss-amazon force-pushed the main branch from 5f46734 to 6dfd21e Compare November 27, 2024 17:24

madolson reviewed Dec 2, 2024

View reviewed changes

yairgott reviewed Dec 2, 2024

View reviewed changes

allenss-amazon and others added 3 commits December 5, 2024 06:39

Revisions

e63290c

Update VSS.md

47d40dd

Adding gaps relative to RediSearch section Signed-off-by: yairgott <[email protected]>

Update VSS.md

8b34c97

Fixing heading of the 'Unsupported knobs and control' section Signed-off-by: yairgott <[email protected]>

zhangh43 reviewed Dec 6, 2024

View reviewed changes

madolson reviewed Dec 9, 2024

View reviewed changes

madolson added the ready-for-discussion label Dec 9, 2024

Update VSS.md

ce08b2b

removal of links to redis.io Signed-off-by: yairgott <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revised VSS RFC #15

Revised VSS RFC #15

allenss-amazon commented Nov 27, 2024

madolson left a comment

madolson Dec 2, 2024

yairgott Dec 2, 2024

madolson Dec 2, 2024

yairgott Dec 2, 2024

allenss-amazon Dec 2, 2024

madolson Dec 9, 2024

madolson Dec 2, 2024 •

edited

Loading

yairgott Dec 2, 2024

allenss-amazon Dec 2, 2024

madolson Dec 9, 2024

madolson Dec 2, 2024

yairgott Dec 2, 2024

madolson Dec 9, 2024

madolson Dec 2, 2024

madolson Dec 9, 2024

yairgott Dec 2, 2024

yairgott Dec 2, 2024

yairgott Dec 2, 2024

yairgott Dec 2, 2024

yairgott Dec 2, 2024

yairgott commented Dec 3, 2024 •

edited

Loading

yairgott commented Dec 3, 2024

zhangh43 Dec 6, 2024

madolson Dec 9, 2024

allenss-amazon Dec 9, 2024

madolson Dec 9, 2024

madolson Dec 9, 2024

madolson Dec 9, 2024

madolson Dec 9, 2024

madolson Dec 9, 2024


		The command returns either an array if successful or an error.

		If `NOCONTENT` is specified, then the output is .....


		If `NOCONTENT` is specified, then the output is .....

		If `NOCONTENT` is not specified, then the output is an array of (2\*N)+1 entries, where N is the number of keys output from the search. The first entry in the array is the value N which is followed by N pairs of entries, one per key found. Each pair of entries consists of the key name followed by an array which is the result value for that key.


		The following metrics are added to the INFO command.

		- search\_total\_indexed\_hash\_keys (Integer) Total count of HASH keys for all indexes

Revised VSS RFC #15

Are you sure you want to change the base?

Revised VSS RFC #15

Conversation

allenss-amazon commented Nov 27, 2024

madolson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

madolson Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yairgott commented Dec 3, 2024 • edited Loading

yairgott commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

madolson Dec 2, 2024 •

edited

Loading

yairgott commented Dec 3, 2024 •

edited

Loading