Skip to content

Raise an error instead of crashing when fetched text cannot be decoded - #1089

Open
rpcope1 wants to merge 1 commit into
ibmdb:masterfrom
rpcope1:fix/fetch-invalid-utf8
Open

rpcope1 wants to merge 1 commit into
ibmdb:masterfrom
rpcope1:fix/fetch-invalid-utf8

Conversation

@rpcope1

@rpcope1 rpcope1 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

When converting a character column to str fails (for example a lone UTF-16 surrogate stored by drivers before #1065), every fetch API now raises the UnicodeDecodeError, as fetch_assoc(), fetch_both(), fetchmany() and fetchall() (and their DBI counterparts) already did:

  • _python_ibm_db_bind_fetch_helper() (fetch_tuple()/fetchone()): return NULL instead of storing the NULL value in the row tuple and returning the tuple with the error pending (SystemError).
  • ibm_db_fetchone(): propagate that error instead of returning None over it.
  • ibm_db_result(): skip the debug log line when the conversion failed. It passed the NULL result to PyUnicode_AsUTF8() and crashed.

No characters are replaced or substituted, and nothing changes for values that decode.

Closes #1088

Added ibm_db_tests/test_6797_FetchUndecodableText.py. It runs against Db2 LUW and needs a UTF-8 database, like the other emoji tests. It inserts the undecodable bytes through a FOR BIT DATA cast, checks fetch_tuple(), fetch_assoc(), fetch_both(), result(), fetchone(), fetchmany(), fetchall() and DBI fetchone()/fetchmany()/fetchall(), and then checks that a valid row on the same connection still reads.

Testing (Linux x86_64, Python 3.13, clidriver v11.5.9, Db2 LUW 11.5.8):

  • On master the new test segfaults the runner (the result() case). With this change it passes.
  • The full sync suite fails the same set of tests before and after this change. Those failures are environment gaps in my test database (missing sample tables).

When a character column holds bytes that cannot be decoded, for example
a lone UTF-16 surrogate stored by older drivers that cut an emoji in
half on insert, the conversion to str fails. fetch_assoc(), fetch_both(),
fetchmany() and fetchall() already raise UnicodeDecodeError in that
case, but fetch_tuple() and fetchone() returned the row with the error
still set, so Python raised SystemError, and result() passed the failed
conversion to a debug log call and crashed. All of them now raise
UnicodeDecodeError.

Add a regression test.

Signed-off-by: Robert Cope <robert@copesystems.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fetch_tuple()/fetchone() raise SystemError and result() segfaults on text that cannot be decoded

2 participants