Repository navigation
Buffer is not transferable in Node 21/22 #55593
Description
Version
v21.7.3, v22.11.0
Platform
Linux nweiz1 6.9.10-1rodete5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.10-1rodete5 (2024-09-04) x86_64 GNU/Linux
Subsystem
worker_threads
What steps will reproduce the bug?
import {MessageChannel, Worker} from 'worker_threads';
const channel = new MessageChannel();
const buffer = Buffer.from('some text');
channel.port1.postMessage(buffer, [buffer.buffer]);
channel.port2.on('message', (e) => {
console.log(Buffer.from(e).toString());
channel.port1.close();
channel.port2.close();
});This worked in v20 and fails in v21 and v22.
How often does it reproduce? Is there a required condition?
It always reproduces.
What is the expected behavior? Why is that the expected behavior?
The buffer should be transferred through the message channel.
What do you see instead?
node:internal/per_context/domexception:53
ErrorCaptureStackTrace(this);
^
DOMException [DataCloneError]: Cannot transfer object of unsupported type.
at new DOMException (node:internal/per_context/domexception:53:5)
at file:///.../test.mjs:5:15
at ModuleJob.run (node:internal/modules/esm/module_job:268:25)
at async onImport.tracePromise.__proto__ (node:internal/modules/esm/loader:543:26)
at async asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:116:5)
Node.js v22.11.0
Additional information
I didn't see anything in the v21 or v22 changelogs about buffers becoming untransferable, and transferring it worked in v20, so I'm assuming this is unintentional. If it's intentional, I would have expected at least a deprecation message indicating that this was going to start breaking.
Reacted by Juri Gavshin
Reacted by Toni Villena
Activity
nex3 commentedon Oct 29, 2024
Note that
buffer.bufferis documented to be anArrayBuffer, of which MDN says:avivkeller commentedon Oct 30, 2024
Does this also occur in v23?
nex3 commentedon Oct 30, 2024
Yes.
targos commentedon Oct 30, 2024
I did a quick manual bisect using the nightly builds.
good: https://nodejs.org/download/nightly/v21.0.0-nightly20230505af9b48a2f1/
bad: https://nodejs.org/download/nightly/v21.0.0-nightly20230506a90a1459ee/
Diff: af9b48a...a90a145
Most likely caused by #47604
bolt-juri-gavshin commentedon Oct 31, 2024
Some additional context, if you replace this line...
channel.port1.postMessage(buffer);- it workschannel.port1.postMessage("Random string", [buffer.buffer]);- it still throws, buffer doesn't have to be posted to trigger the errorAlso (executed right after
bufferinstantiation)...Does anybody know what's the plan, to make
postMessage(buffer, [buffer.buffer])work again orpostMessage(buffer)will transfer it's buffer implicitly (it doesn't right now)?legendecas commentedon Nov 4, 2024
There is an internal Buffer pool that can not be transfered. Buffers allocated in this pool are not transferable, as documented at https://nodejs.org/api/worker_threads.html#workermarkasuntransferableobject.
#47604 makes the "transfer" intent explicit that if the transfer can not be achieved, an error is thrown, according to the spec. The transfer operation was never completed without the change, instead, an implicit "copy" operation was performed.
I think this difference can be documented at https://nodejs.org/api/buffer.html#buffers-and-typedarrays as well.
101arrowz commentedon Nov 11, 2024
Although the new behavior makes sense, it does make using buffers with worker threads less ergonomic.
When you allocate a
Uint8Arrayusing the de-facto constructornew Uint8Array(size), it's guaranteed to be transferable, but this isn't the case with dynamically allocatedBufferobjects (or the results offs.readFileSync, etc.) With the change that pooledBuffers causeDataCloneErrors on an attempted transfer, it has become more difficult to write code that takes advantage of transferringUint8Arraywithout explicitly handling the case that theUint8Arrayis a pooledBuffer. This is mostly because it is difficult to even know if a givenBufferis pooled or not.I've encountered the effects of this change in 101arrowz/fflate#227. The solution here seems to be to have my library retry the
postMessagewithout any transferables on aDataCloneError, but it would be nice if Node could expose information about whether a givenBufferis transferable or not to users.legendecas commentedon Nov 12, 2024
worker.isMarkedAsUntransferable(object)tells if an array buffer is transferable.101arrowz commentedon Nov 12, 2024
Thanks, that's helpful! Are you familiar with any runtime agnostic way to detect transferability beyond just trying it and failing if it doesn't work? I think there's
ArrayBuffer.prototype.detachedthat can detect if an array buffer has already been transferred, but this isn't available in Node from what I can tell. (Also it's not a perfect proxy for transferability anyway.)targos commentedon Nov 12, 2024
ArrayBuffer.prototype.detachedworks fine in Node.js101arrowz commentedon Nov 12, 2024
OK, MDN is just out of date then; might be worth bringing up with them. I'll just use
isMarkedAsUntransferablefor now though, as it doesn't look likeArrayBuffer.prototype.detached(or any other JS standard property that works across runtimes) specifically checks for non-transferability.2 remaining items
bradzacher commentedon Nov 15, 2024
That's quite interesting! I didn't realise that was the behaviour. When I added it to the transferable I assumed I was being performant 😅
Well then is there any way to get binary data from NodeJS builtin APIs as a
Buffer(eg from fs or stdin) and transfer it to another thread without any copying at all? I suspect that there's a lot of people that want to do this sort of behaviour withBuffer.For context my usecase is that I have a server of sorts which is receiving binary data via stdin and I forward the data to one of the worker threads for handling - so being able to pass the bytes from stdin to whichever worker is free without copying would be ideal to improve throughput.
legendecas commentedon Nov 26, 2024
Buffers from stdin are capable of being transfered.
Only buffers created with
Buffer.allocUnsafe(),Buffer.from(array),Buffer.from(string), andBuffer.concat()may use the buffer pool and not capable of being transfered. This behavior can be disabled withBuffer.poolSize = 0.github-actions commentedon Apr 27, 2026
This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.
nex3 commentedon Apr 27, 2026
Definitely still relevant!
Replace Sharp.clone() with SharpFactory pattern (fixes DataCloneError…
github-actions commentedon Jul 28, 2026
This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.
nex3 commentedon Jul 28, 2026
Stil relevant
github-actions commentedon Oct 28, 2026
This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.
nex3 commentedon Oct 28, 2026
still relevant!