Decode Avro decimal-as-bytearray string
up vote
0
down vote
favorite
I'm using Kafka with Avro messages. One of my fields is defined like this:
{
"name": "a_number",
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 4
}
Using the Avro console consumer, I see a message like this:
{"a_number": "tu0000°"}
Which I expect to equal 59.
Supposedly, the bytearray should be the twos-compliment of the number. I've tried using Python's struct module to decode it, but the values I get don't make any sense:
bs = 'tu0000°'.encode('utf8') # b'tx00xc2xb0'
struct.unpack('>l', bs)[0] / 1e4 # 15104.4784
How can I validate the message? Can I decode the string somehow, or has the Avro console consumer corrupted it?
python python-3.x apache-kafka avro
edited Nov 21 at 22:46
cricket_007
78.3k1142109
asked Nov 21 at 22:14
z0r
4,66513352
add a comment |
up vote
0
down vote
favorite
I'm using Kafka with Avro messages. One of my fields is defined like this:
{
"name": "a_number",
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 4
}
Using the Avro console consumer, I see a message like this:
{"a_number": "tu0000°"}
Which I expect to equal 59.
Supposedly, the bytearray should be the twos-compliment of the number. I've tried using Python's struct module to decode it, but the values I get don't make any sense:
bs = 'tu0000°'.encode('utf8') # b'tx00xc2xb0'
struct.unpack('>l', bs)[0] / 1e4 # 15104.4784
How can I validate the message? Can I decode the string somehow, or has the Avro console consumer corrupted it?
python python-3.x apache-kafka avro
edited Nov 21 at 22:46
cricket_007
78.3k1142109
asked Nov 21 at 22:14
z0r
4,66513352
For comparison:struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'
– z0r
Nov 21 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 at 23:01
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I'm using Kafka with Avro messages. One of my fields is defined like this:
{
"name": "a_number",
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 4
}
Using the Avro console consumer, I see a message like this:
{"a_number": "tu0000°"}
Which I expect to equal 59.
Supposedly, the bytearray should be the twos-compliment of the number. I've tried using Python's struct module to decode it, but the values I get don't make any sense:
bs = 'tu0000°'.encode('utf8') # b'tx00xc2xb0'
struct.unpack('>l', bs)[0] / 1e4 # 15104.4784
How can I validate the message? Can I decode the string somehow, or has the Avro console consumer corrupted it?
python python-3.x apache-kafka avro
edited Nov 21 at 22:46
cricket_007
78.3k1142109
asked Nov 21 at 22:14
z0r
4,66513352
I'm using Kafka with Avro messages. One of my fields is defined like this:
{
"name": "a_number",
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 4
}
Using the Avro console consumer, I see a message like this:
{"a_number": "tu0000°"}
Which I expect to equal 59.
Supposedly, the bytearray should be the twos-compliment of the number. I've tried using Python's struct module to decode it, but the values I get don't make any sense:
bs = 'tu0000°'.encode('utf8') # b'tx00xc2xb0'
struct.unpack('>l', bs)[0] / 1e4 # 15104.4784
How can I validate the message? Can I decode the string somehow, or has the Avro console consumer corrupted it?
python python-3.x apache-kafka avro
python python-3.x apache-kafka avro
edited Nov 21 at 22:46
cricket_007
78.3k1142109
asked Nov 21 at 22:14
z0r
4,66513352
edited Nov 21 at 22:46
cricket_007
78.3k1142109
asked Nov 21 at 22:14
z0r
4,66513352
edited Nov 21 at 22:46
cricket_007
78.3k1142109
edited Nov 21 at 22:46
cricket_007
78.3k1142109
edited Nov 21 at 22:46
cricket_007
78.3k1142109
78.3k1142109
asked Nov 21 at 22:14
z0r
4,66513352
asked Nov 21 at 22:14
z0r
4,66513352
asked Nov 21 at 22:14
z0r
4,66513352
4,66513352
For comparison:struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'
– z0r
Nov 21 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 at 23:01
add a comment |
For comparison:struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'
– z0r
Nov 21 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 at 23:01
For comparison:
struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'– z0r
Nov 21 at 22:16
For comparison:
struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'– z0r
Nov 21 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 at 22:38
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 at 23:01
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 at 23:01
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 at 2:07
J_H
3,1531616
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 at 2:43
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421197%2fdecode-avro-decimal-as-bytearray-string%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 at 2:07
J_H
3,1531616
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 at 2:43
add a comment |
up vote
1
down vote
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 at 2:07
J_H
3,1531616
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 at 2:43
add a comment |
up vote
1
down vote
up vote
1
down vote
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 at 2:07
J_H
3,1531616
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 at 2:07
J_H
3,1531616
answered Nov 22 at 2:07
J_H
3,1531616
answered Nov 22 at 2:07
J_H
3,1531616
answered Nov 22 at 2:07
J_H
3,1531616
3,1531616
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 at 2:43
add a comment |
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 at 2:43
Yep fair point. I don't have a
.avro file to read, but maybe I should just write a little Python script using an AvroConsumer instead of the (presumably) Java-based console consumer to test it.– z0r
Nov 22 at 2:43
Yep fair point. I don't have a
.avro file to read, but maybe I should just write a little Python script using an AvroConsumer instead of the (presumably) Java-based console consumer to test it.– z0r
Nov 22 at 2:43
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421197%2fdecode-avro-decimal-as-bytearray-string%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
For comparison:
struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'– z0r
Nov 21 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 at 23:01