BUG : startswith tuples no longer work in pandas.DataFrame.query expr

[steves-dev]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame.from_records([{'title':'abcd','value':1},
                                {'title':'abc','value':2},
                                {'title':'a','value':3},
                                {'title':'bdc','value':4},
                                {'title':'dfg','value':5},
                                {'title':'fgh','value':6},
                                {'title':'a3','value':7},])

#this worked in pandas '1.3.4'; but no longer:
query_str1 = '`title`.str.startswith(tuple(["a","d"]))'
filtered_df1 = df.query(query_str1,engine='python')

#this also worked in pandas '1.3.4'; but no longer:
query_str2 = '`title`.str.startswith(tuple(("a","d")))'
filtered_df2 = df.query(query_str2,engine='python')

#this still works:
t_op = ("a","b")
query_str3 = '`title`.str.startswith(@t_op)'
filtered_df3 = df.query(query_str3,engine='python')

Issue Description

first two patterns for the query expr fail with :
ValueError: "tuple" is not a supported function

Expected Behavior

In earlier versions of pandas (e.g. 1.3.4) they would have returned the same filtered dataframe as the 3rd pattern example (query_str3).

Installed Versions

INSTALLED VERSIONS

commit : e86ed37
python : 3.11.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252

pandas : 2.1.1
numpy : 1.26.0
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.3
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : 1.3.5
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : 2.8.7
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[paulreece]

This seems to be an intentional design change, it doesn't work with any kind of data structure creation because they are not included in MATHOPS

 MATHOPS: tuple[Literal['sin'], Literal['cos'], Literal['exp'], Literal['log'], Literal['expm1'], Literal['log1p'], Literal['sqrt'], Literal['sinh'], Literal['cosh'], Literal['tanh'], Literal['arcsin'], Literal['arccos'], Literal['arctan'], Literal['arccosh'], Literal['arcsinh'], Literal['arctanh'], Literal['abs'], Literal['log10'], Literal['floor'], Literal['ceil'], Literal['arctan2']]

>>> query_str1 = '`title`.str.startswith(list(["a","d"]))'
>>> filtered_df1 = df.query(query_str1,engine='python')

Traceback (most recent call last):
...
ValueError: "list" is not a supported function

It fails here:

pandas/pandas/core/computation/ops.py

Lines 613 to 618 in a39f783

	class FuncNode:
	def __init__(self, name: str) -> None:
	if name not in MATHOPS:
	raise ValueError(f'"{name}" is not a supported function')
	self.name = name
	self.func = getattr(np, name)

As you noted above if you take the extra step and convert to tuple first, it works:

>>> my_list = ["a", "b"]
>>> my_list_tupled = tuple(my_list)
>>> query_str1 = '`title`.str.startswith(@my_list_tupled)'
>>> filtered_df1 = df.query(query_str1,engine='python')

[steves-dev]

Thank you for the attention @paulreece, and for tracing the source.

<removed my reference to docs; I confused pandas.eval() with the base eval() in my test>

The reason that the "@" syntax is sometimes undesirable is if you want to build query strings and store them, but the tuples don't exist in the namespace anymore at the time you call the query. Also if you are building up the query string recursively, keeping track of the swap variables adds a layer of complexity that was unnecessary in earlier versions.

[steves-dev]

Work-around for those 2 cases I described seems to be the "local_dict" arg from pandas.eval:

query_str1 = '`title`.str.startswith(@ref_tuple)'
filtered_df1 = df.query(query_str1,engine='python',local_dict={'ref_tuple':tuple(["a","d"])})

Based on the insight that the change seems intentional I'm closing.