roadmap: Cortex supports /files

[dan-homebrew]

Architecture for files:

• API should compatible and follow OpenAI: https://platform.openai.com/docs/api-reference/files

Filesystem

~/cortexcpp
├── cortex.db
├── engines
├── files
│   └── Enterprise_Application_Infrastructure_v2_20140903_toCTC_v1.0.pdf
├── logs
├── models
└── threads

Database

• File metadata will be stored in files table in cortex.db

Backward compatible

Provide backward compatible for previous jan, which stores file inside each thread.

./jan_1732370027
├── files
│   └── 01JDMG617BHMPW859VE18BPQ7Y.pdf
├── memory
│   ├── args.json
│   ├── docstore.json
│   └── hnswlib.index
├── messages.jsonl
└── thread.json

Pros and Cons

• User can search for their file using their file name, we don't set their file name to any uuid like before.
• Easy to share files with assistants, tools, etc.

API References

https://c6f778ad.cortex-docs.pages.dev/api-reference

[namchuai]

Architecture for files:

• API should compatible and follow OpenAI: https://platform.openai.com/docs/api-reference/files

Filesystem

~/cortexcpp
├── cortex.db
├── engines
├── files
├── 0001KNE06900GW0X476W5TVBFE
│   ├── AAPB2016_UIImplementation.pdf
│   └── metadata.json
├── 0001KNE3E500GW0X476W5TVBFE
│   ├── AAPB2016_UIImplementation.pdf
│   └── metadata.json
└── 0001KNE3EBC62D620DGYNG2R8H
    ├── AAPB2016_UIImplementation.pdf
    └── metadata.json
├── logs
├── models
└── threads

Sample metadata.json

{
        "bytes" : 2099009,
        "created_at" : 1733755081,
        "filename" : "AAPB2016_UIImplementation.pdf",
        "id" : "0001KNE06900GW0X476W5TVBFE",
        "object" : "file",
        "purpose" : "assistant"
}

• User can search for their file using their file name, we don't set their file name to any uuid like before.
• Easy to share files with assistants, tools, etc.

Backward compatible

We need to provide backward compatible for previous jan, which stores file inside each thread

./jan_1732370027
├── files
│   └── 01JDMG617BHMPW859VE18BPQ7Y.pdf
├── memory
│   ├── args.json
│   ├── docstore.json
│   └── hnswlib.index
├── messages.jsonl
└── thread.json

Cortex will provide same API path as OpenAI's path, but requires additional property thread_id so that we can make cortex API compatible.

[namchuai]

Request for comment: Move the metadata.json to cortex.db

[dan-homebrew]

12 Dec

• Docs
• Migration? -> Files table
• Update Spec for this issue (for future reference)

[louis-jan]

Hi @dan-homebrew @namchuai, now that we’ve flattened the files under /files, which is more of a global storage, it’s not practical to keep the file name. Since file_search can be scoped to a thread or assistant, keeping the file name causes issues where duplicate file names are ignored, leading to incorrect behavior in most application

Case 1:

• Users can have different files with the same name, they can't attach those to a thread.

Case 2:

• Users can upload the same file to a thread to update embeddings after model changes.

Case 3:

• Reinforcing or weighting specific data - increase its weight in search results? In case they ingested multiple files before in the same thread.

Update 1: Attempting to add a postfix on duplicate

[TC117]

• Can Upload files
  
  

• Can get files list
  

• Get file with ID
  

• Get file content
  

• Delete file with ID
  

Still need to check Backward compatible with Jan

[TC117]

Working with Jan

But it will create _1 for files with the same name