You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jul 12, 2025. It is now read-only.
Repository navigation
This repository was archived by the owner on Jul 12, 2025. It is now read-only.
Unexpected behavior of pgtype.JSONBArray and json.RawMessage #149
I am setting a pgtype.JSONBArray value from a value of type []json.RawMessage, and encountered some unexpected behavior. I am not sure if this is a bug.
What happens is that data is interpreted as an array of single integer values, one integer for each byte in the two json values. So nas.Elements has length 18.
But changing one of the JSON values a bit will make the test pass:
This is with github.com/jackc/pgtype v1.8.1 and go 1.17. Casting the type to [][]byte makes everything work just fine. I assume this is because [][]byte is handled by .Set specifically, where []json.RawMessage will fall through to the default reflection based conversion in .Set.
Would it be possible to add a case for []json.RawMessage in .Set?
As mentioned in the PR, your suggestion will fix the test cases for json.RawMessage. However, the fundamental issue remains, in the sense that defining any other type alias for []byte such as
typesomebytealias []byte
and substituting json.RawMessage with this type, will make the test cases fail once again. So it appears there might be a problem in the reflect based case in .Set?
I am happy to help investigate this, but I do not feel I understand the ramifications of tinkering with the typed_array.go.erb template, as this is used for several types. I have tried to debug these test cases:
And at least I can share some observations. The two first cases are different from the third.
In the third case, the two elements {"foo":1} and {"bar":2} both have length 9, and what is set on dst.Elements is
[12334102111111345849125123349897114345850125]
which is the concatenation of the two inputs.
Changing 1 to 12, which is the first case, gives us this output
["eyJmb28iOjEyfQ==""eyJiYXIiOjJ9"]
which is the base64 encoding of the two inputs as strings. This is probably due to json.Marshal being used in the pgtype.JSON type at some point when calling dst.Elements[index].Set in the setRecursive code, as this is the default behavior on []byte types. The same happens in the second case.
The code that tries to automatically handle unknown types is hard to reason about. The JSON code path is even harder because it automatically calls json.Marshal with anything that isn't explicitly recognized.
That conflicts with the try to figure out if the unknown type can be converted to something known. For example:
Should mytype be marshaled or should it be treated as a []byte? And how to handle that while not breaking any of the other convertible to JSON types.
JSONBArray gets even more complicated because there is the extra layer of the array.
Personally, I'm inclined to be content with the partial solution in your PR rather than delve into that nest of complexity. But your welcome to dig into it further if you want.
FWIW, tricky issues like this were one of the primary motivations behind deciding to start development on pgx v5. I'm hoping to simplify the structure of the type conversion system such that this type issue isn't so complicated.
I think I would be hard pressed to make much progress here :) Seeing as you are working on pgx v5, I think it is better to look for the future. I am curious to look into the new type conversion code.
Activity
jackc commentedon Mar 12, 2022
If that is the solution then it should be possible to add it to typed_array_gen.sh and regenerate the array struct.
EmilLaursen commentedon Mar 20, 2022
As mentioned in the PR, your suggestion will fix the test cases for
json.RawMessage. However, the fundamental issue remains, in the sense that defining any other type alias for[]bytesuch asand substituting
json.RawMessagewith this type, will make the test cases fail once again. So it appears there might be a problem in the reflect based case in.Set?I am happy to help investigate this, but I do not feel I understand the ramifications of tinkering with the
typed_array.go.erbtemplate, as this is used for several types. I have tried to debug these test cases:And at least I can share some observations. The two first cases are different from the third.
{"foo":1}and{"bar":2}both have length 9, and what is set ondst.Elementsiswhich is the concatenation of the two inputs.
1to12, which is the first case, gives us this outputwhich is the base64 encoding of the two inputs as strings. This is probably due to
json.Marshalbeing used in thepgtype.JSONtype at some point when callingdst.Elements[index].Setin the setRecursive code, as this is the default behavior on[]bytetypes. The same happens in the second case.jackc commentedon Mar 22, 2022
The code that tries to automatically handle unknown types is hard to reason about. The JSON code path is even harder because it automatically calls
json.Marshalwith anything that isn't explicitly recognized.That conflicts with the try to figure out if the unknown type can be converted to something known. For example:
Should
mytypebe marshaled or should it be treated as a[]byte? And how to handle that while not breaking any of the other convertible to JSON types.JSONBArray gets even more complicated because there is the extra layer of the array.
Personally, I'm inclined to be content with the partial solution in your PR rather than delve into that nest of complexity. But your welcome to dig into it further if you want.
FWIW, tricky issues like this were one of the primary motivations behind deciding to start development on pgx v5. I'm hoping to simplify the structure of the type conversion system such that this type issue isn't so complicated.
EmilLaursen commentedon Mar 24, 2022
I think I would be hard pressed to make much progress here :) Seeing as you are working on pgx v5, I think it is better to look for the future. I am curious to look into the new type conversion code.