BUG: rowspan in read_html failed

[duongkstn]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# html string
s = '<table>\n<caption>Đơn vị tính: Việt Nam đồng</caption>\n<tr>\n<th rowspan="2">Tổng giới hạn bồi thường / năm / người</th>\n<th>Chương trình 1</th>\n<th>Chương trình 2</th>\n<th>Chương trình 3</th>\n</tr>\n<tr>\n<td>50,000,000</td>\n<td>100,000,000</td>\n<td>200,000,000</td>\n</tr>\n<tr>\n<td>Tử vong do tai nạn</td>\n<td>50,000,000</td>\n<td>100,000,000</td>\n<td>200,000,000</td>\n</tr>\n<tr>\n<td>Thương tật toàn bộ vĩnh viễn/ năm</td>\n<td>50,000,000</td>\n<td>100,000,000</td>\n<td>200,000,000</td>\n</tr>\n<tr>\n<td>Thương tật bộ phận vĩnh viễn/ năm</td>\n<td>50,000,000</td>\n<td>100,000,000</td>\n<td>200,000,000</td>\n</tr>\n<tr>\n<td>Chi phí y tế điều trị thương tật do tai nạn/ năm</td>\n<td>10,000,000</td>\n<td>20,000,000</td>\n<td>30,000,000</td>\n</tr>\n</table>'


print(pd.read_html(s))

Issue Description

I have a HTML string (with rowspan attribute), here is how it look when rendering

Then I convert to dataframe by using pd.read_html command (pandas 2.2.3), here is how it looks:

Totally wrong, the row of 50000000, 100000000 and 200000000 should be aligned to the right.

Expected Behavior

the row of 50000000, 100000000 and 200000000 should be aligned to the right

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.10.9
python-bits : 64
OS : Linux
OS-release : 5.15.0-124-generic
Version : #134~20.04.1-Ubuntu SMP Tue Oct 1 15:27:33 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.3
numpy : 1.26.4
pytz : 2022.7
dateutil : 2.8.2
pip : 24.3.1
Cython : None
sphinx : 5.0.2
IPython : 8.10.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.3.1
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.2
lxml.etree : 4.9.1
matplotlib : None
numba : N/A
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.3
pandas_gbq : None
psycopg2 : 2.9.9
pymysql : None
pyarrow : 16.1.0
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
xlsxwriter : 3.2.0
zstandard : 0.19.0
tzdata : 2024.1
qtpy : None
pyqt5 : None

[simonjayhawkins]

Thanks @duongkstn for the report.

For the reproducible example, please follow this guide on how to provide a minimal, copy-pastable example.

I understand this requires additional effort, but it significantly increases the likelihood that a member of the volunteer community will address the issue.

Some preliminary thoughts:

Does the behavior persist without the caption tags?

Does the issue occur with just two rows of data or two columns of data?

Can this be reproduced with regular text?

Additionally, it is worth noting that a warning is issued about passing literal HTML to read_html. In version 3.0.0 (and currently on the development branch), this code sample will result in a FileNotFoundError.

It's possible that some reported issues have already been resolved. Therefore, I suggest updating the code sample to ensure it runs on the latest development version.

If this issue is indeed a bug in pandas and has not been previously reported, any PR with a fix will also need a test. Creating a Minimal Complete Verifiable Example that works on 'main' can help streamline this process, as the test will already be written.

[rhshadrach]

Minimal example:

s = '<table><tr><th rowspan="2">A</th><th>B</th></tr><tr><td>1</td></tr><tr><td>C</td><td>2</td></tr></table>'
buf = io.StringIO(s)
print(pd.read_html(buf)[0])
#    A                  B
#    A Unnamed: 1_level_1
# 0  1                NaN

# Expected:
#    A  B
# 0  A  1
# 1  C  2

Switching the top row from th to td produces the expected results (except that the column labels are now the first row, as they should be). It appears pandas does not handle rowspan="2" correctly in a th.

Browsers do handle this case in the expected manner:

[simonjayhawkins]

Thanks @rhshadrach.

Your expected output is consistent with the given string, but the rendered HTML and actual output aren't quite right.

To confirm, the behavior is the same on the main branch and also back in version 1.5.3.

This seems like a corner case where we need to explicitly define the expected behavior. I understand why we get the current output and could potentially justify it if needed. Conversely, how would you justify that your expected output is correct? The HTML has a header spanning two rows, so why would you expect to get back a DataFrame without a multilevel Index?

[rhshadrach]

The HTML has a header spanning two rows, so why would you expect to get back a DataFrame without a multilevel Index?

In the browser-rendered output, the value A is shared in the top and 2nd row, while the 2nd row is not bold. I take that as an indication that the value "A" belongs to the 2nd row as well as the first, and that the 2nd row is not a header.

[duongkstn]

any updates :) ?

[rhshadrach]

@duongkstn - PRs to fix this issue are welcome.

[snitish]

take

[snitish]

The issue was that the rowspan in the header caused the header cells to overflow into the body rows, which the current logic does not handle due to it independently processing header, body and footer rows. I fixed it passing overflown rows from previous section to the next _expand_colspan_rowspan call and added @rhshadrach's example as a test case. PR: #60464