Skip to content

fix(batch): raise a clear error for unsupported property types - #2264

Open
steven-asearcy wants to merge 2 commits into
weaviate:mainfrom
steven-asearcy:fix/batch-unsupported-type-error
Open

steven-asearcy wants to merge 2 commits into
weaviate:mainfrom
steven-asearcy:fix/batch-unsupported-type-error

Conversation

@steven-asearcy

@steven-asearcy steven-asearcy commented Nov 18, 2026 •

Copy link
Copy Markdown

closes #2263

batch and insert_many passed types they don't handle straight into protobuf's Struct.update, which fails with ValueError: Unexpected type, and in a batch that takes every other object down with it.

now Decimal and numpy numbers get converted before serialization (Decimal and numpy floats to float, numpy ints to int), in both the grpc path and data.insert. converted values that end up NaN or inf raise WeaviateInvalidInputError, and types that still can't be serialized raise the same WeaviateInvalidInputError that data.insert already did. Decimal is added to WeaviateField.

tuples still work, they get turned into lists the same way Struct handled them before.

tests: unit tests through _BatchGRPC.grpc_object in test/collection/test_batch.py, stream() tests in mock_tests/test_batch.py, and integration tests in integration/test_batch_v4.py. pytest test, pytest mock_tests and the batch and collection integration tests pass against 1.40.5 with ci/start_weaviate.sh. pre-commit is clean on the changed files apart from pyright, which reports the same 226 errors on main.

@weaviate-git-bot

Copy link
Copy Markdown

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?

@orca-security-eu orca-security-eu Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@steven-asearcy

Copy link
Copy Markdown
Author

I agree with the CLA

Comment thread test/collection/test_batch.py Resolved
@steven-asearcy
steven-asearcy force-pushed the fix/batch-unsupported-type-error branch from 816c588 to c67bc5c Compare November 18, 2026 16:31
@steven-asearcy

steven-asearcy commented Nov 18, 2026 •

Copy link
Copy Markdown
Author

@dirkkul thanks, good catch on the list test. it was only hitting _serialize_primitive directly so it never went through the branch picking.

went with the conversion in this PR:

  • Decimal and numpy floats go to float and numpy ints to int, before a list picks its array branch
  • if a converted value comes out as NaN or inf it raises. plain floats (and np.float64, since it subclasses float) are left alone like before
  • data.insert does the same now, and Decimal is in WeaviateField
  • other unsupported types still get the "Cannot serialize value of type" error

tests go through grpc_object now (scalar, nested object, Decimal first and second in a list, tuple of uuids, numpy, non-finite, unsupported). added two integration tests in test_batch_v4.py for fixed_size, insert_many and data.insert, and two stream() tests in mock_tests/test_batch.py.

two things i noticed but left alone:

  • mixed lists still pick their branch from the first element in the grpc path. [1, Decimal("2")] fails like [1, 2.0] does on main, and ["a", object()] still gets protobuf's TypeError instead of the new message. data.insert handles both
  • with stream(), if nothing calls add_object after the loop thread hits an error, it never reaches the caller. the thread stores it but nothing raises it on exit, so everything in that batch is dropped and failed_objects is empty. same on main. can open a separate issue for that if you want

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

batch insert fails the whole batch with "ValueError: Unexpected type" when a value is a Decimal

3 participants