User Experience Improvement Summary

[xzdandy]

Search before asking

• I have searched the EvaDB issues and found no similar feature requests.

Description

This is a issue that summarizes the user's feedback on their experience with EvaDB:

SQL Statement related:

• USE statement in EvaQL does not allow parenthesis, and it only supports one native statement.
• LOAD DOCUMENTS and CHUNK SIZE is confusing.
• GROUP BY only works on video and document tables. GROUP BY does not work on traditional columns.
• More documentation on the syntax of creating new tables.
• Drop table throws the table into a limbo realm where it still exists and cannot be effectively dropped or create table. #1320
• SELECT * FROM table does not work for third-party data sources. It is not clear we have to use database.table as table source.

Data related:

• Convert numbers from string (e.g., with comma, scientific numbers) into integer.
• Loading different text file (.pdf and .txt) in the same table is problematic.
• Version incompatibility (table and index are created by an earlier version of EvaDB)
• Data types are not supported in EvaDB when reading from postgres, e.g., array types or uuid.
• Most DB cursors sanitize user-provided data in queries automatically, but EvaDB did not seem to have this functionality.
• Escape all the single apostrophes when insert data into database.
• No string concatenation method
• Can not directly insert dataframes
• On a Windows machine, storing file paths as strings in MySQL caused problems with misidentifying backslashes for
  escape sequences.
• New data sources:
  • Google / Bing as a data source for web scraping (e.g., Serper)
  • Google map
  • Github issues
  • Reddit PRAW
  • Wiki

User-defined function related:

• Sklearn now only supports for linear regression model, more machine learning models can be introduced and supported
• Function only work on table elements. Need to create a table with one tuple.
• For aggregation user defined functions, we need a way to pass the complete table into the function instead of row by row or batch by batch.
• Type definition in Custom AI function's forward decorator is confusing.
  • Input/Output Format of a function (e.g., AWS Rekognition service) can be bytes, while EvaDB's forward function requires input in the form of a numpy array
• Challenge to figure out what is the best algorithm to predict (e.g,, layoff)

LLM related:

• Deal with various constraints of the OpenAI API, such as token restrictions, rate limits, and other limitations.
• ChatGPT deviate from the 'yes' or 'no' responses quite often.
• Convert the ChatGPT response, which in most cases is just some text
• Unable to find the EvaDB implementation of exactly how the OpenAI API was called, I ran into unsatisfactory results with
  certain prompts and was unable to debug them. Certain prompts resulted in a response indicating that no text had been submitted. I assumed this may have been due to certain characters in the string, but without the ability to inspect the source code of how the API request is made, I ended up resorting to simply trying various prompts until I had satisfactory results.

Index related:

• Similarity function is not symmetric.
• Create index does not work on empty table and third party table.
• Vector databases like Milvus and Pinecone have unavoidable setup efforts by users.

Optimization related:

• Web scrape and ChatGPT are expensive and bottleneck of the execution
• There is optimization opportunity for caching, which is not flushed out.

Installation related:

• Documentation of installation of EvaDB doesn't work for Windows. (i.e., Copying the notebook code to run on windows)
• Installation for PyTesseract isn't as simple
• Docker file for EvaDB has been updated for a while.

Use case

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!