[C++][Parquet] Writing non-nullable field with nulls to Parquet generates invalid Parquet file

[p-ortmann]

Describe the bug, including details regarding any error messages, version, and platform.

Platform MacOs 14.5 (23F79)
Version: 15.0.2 and 16.1.0.

import pyarrow as pa
import pandas as pd
import pyarrow.parquet as pq
data = {
    'column1': [1, 2, None],
    'column2': ['a', None, 'c']
}

schema = pa.schema([
    pa.field('column1', pa.int64(), nullable=True),
    pa.field('column2', pa.string(), nullable=False)  # make column2 not nullable
])

table = pa.Table.from_pydict(data, schema=schema) # set up table with data that doesn't match the schema
assert table.schema.equals(schema)

print('table before writing \n')
print(table.to_pandas())
pq.write_table(table, 'output.parquet')

table = pq.read_table('output.parquet')

print('table after writing and reading \n')
print(table.to_pandas())

yields

table before writing 

   column1 column2
0      1.0       a
1      2.0    None
2      NaN       c
table after writing and reading 

   column1 column2
0      1.0       a
1      2.0       c
2      NaN       a

which is not correct for column 2.

I would expect this to fail on set up of the table, which is what happens if you replace

table = pa.Table.from_pydict(data, schema=schema)

with

dataframe = pd.DataFrame(data)
table = pa.Table.from_pandas(dataframe, schema=schema)

Component(s)

Python

[adhardy]

Hi folks, has anyone had a chance to look at this issue? This is causing data corruption for us. If it is intended behaviour we can work around it, but it doesn't look like it should be.

[adhardy]

Hi, I'd like to try draw attention to this bug again.

This is silently causing data corruption in a way that is easy to accidentally do and very hard to detect.

table before writing 

   column1 column2
0      1.0       a
1      2.0    None
2      NaN       c
table after writing and reading 

   column1 column2
0      1.0       a
1      2.0       c
2      NaN       a

The data in column two is wrong, there should be a null but instead the column leaves it out and starts repeating itself.

This still occurs in pyarrow 18.0.0.

[amoeba]

Hi @adhardy, apologies for this being missed. I've posted this to Zulip and will try to find someone to take a look soon.

[pitrou]

So, the problem is that the Arrow data is invalid (a non-nullable field has nulls), and the Parquet writer doesn't notice the inconsistency, ending up writing invalid data. This was already reported in #31329.

I'm not sure we want to do anything in the Parquet writer to avoid this. However, it would be nice if validation of such incorrect data actually failed: see #31387

[adhardy]

From a user perspective, my expectation is that if I set nullable=False in my schema I am expecting some sort of failure, I guess in a similar way to if I was to pass a mis-matched type.

At the moment if I pass in a None/null to a non-nullable field, my data becomes corrupted and I get no warning about it. I would rather the None gets written, at least my data is "correct".

This is maybe less of a problem I guess in statically typed languages as something would have probably already failed before I got to that point but this is really easy to do in Python.

At present there is really no use in setting nullable=False from the Python API - it does not validate that there are no nulls, and it will just corrupt my data if there is, I may as well always leave nullable=True.

[pitrou]

Issue resolved by pull request 44921
#44921

[pitrou]

From a user perspective, my expectation is that if I set nullable=False in my schema I am expecting some sort of failure, I guess in a similar way to if I was to pass a mis-matched type.

I agree with this, and we have not been consistent in enforcing it. With #44921, at least writing a Parquet file with such data will fail explicitly instead of producing a corrupt file.