Issue with Retrieving UTF-8 Encoded JSON String Using json_GetStr in NoxDB

[RootDeepak]

Hi Niels,

I'm currently working with NoxDB and encountering an issue when handling UTF-8 encoded JSON data. Specifically, I'm using the following line to parse the JSON string:

The pResponse.records(1).record is a pointer that contains UTF-8 encoded JSON. The hexadecimal representation of this data is:

5B7B2273 686F7274 46697844 65736372
69707469 6F6E223A 22E0A4A6 E0A58BE0
A4B9E0A4 B0E0A4BE 227D5D40 40404040

When converted, this resolves to the following JSON content:

[{"shortFixDescription":"दोहरा"}]

To process this, I'm using an iterator with the following logic:

SQLRecords = json_ParseString(pResponse.records(1).record);
SQLRecord = json_getChild(SQLRecords);
Dow SQLRecord <> *Null;
recordField = Json_NewObject();
iterator = json_setIterator(SQLRecord);

Dow json_ForEach(iterator);
keyName = json_getName(iterator.this);
Json_SetStr(recordField: %Trim(keyName): json_GetStr(iterator.this));
EndDo;
SQLRecord = json_getNext(SQLRecord);
EndDo;

Here’s the issue I’m facing:

The key name (shortFixDescription) is being retrieved correctly.

However, the corresponding value is coming through as blank, even though the original UTF-8 JSON string contains the Hindi word "दोहरा".

My question is:
Is there a known issue with json_GetStr in handling UTF-8/multi-byte characters? Or is there a required conversion or workaround needed to retrieve values like this correctly?

Any help or guidance you can provide would be greatly appreciated.

Thanks & Regards,
Deepak Mishra

[wimjongman]

Hey @RootDeepak. Did you take a look at this example? https://github.com/sitemule/noxDB/blob/master/examples/JSONSQL10B.SQL-Exec-sql-Update-UTF8.rpgle (edited)

Maybe it can give you a clue?

[wimjongman]

Apologies, here is the correct file: https://github.com/sitemule/noxDB/blob/master/examples/JSONSQL10B.SQL-Exec-sql-Update-UTF8.rpgle

[wimjongman]

Can you maybe try this?

SQLRecords = json_ParseString(pResponse.records(1).record);
SQLRecord = json_getChild(SQLRecords);
Dow SQLRecord <> *Null;
   recordField = Json_NewObject();
   iterator = json_setIterator(SQLRecord);

   Dow json_ForEach(iterator);
      keyName = json_getName(iterator.this);
      utfString =  json_GetStr(iterator.this);   <-- put it in a utf string
      Json_SetStr(recordField: %Trim(keyName): utfString);   <-- load it here
   EndDo;
   SQLRecord = json_getNext(SQLRecord);
EndDo;

[RootDeepak]

Can you maybe try this?

SQLRecords = json_ParseString(pResponse.records(1).record);
SQLRecord = json_getChild(SQLRecords);
Dow SQLRecord <> *Null;
   recordField = Json_NewObject();
   iterator = json_setIterator(SQLRecord);

   Dow json_ForEach(iterator);
      keyName = json_getName(iterator.this);
      utfString =  json_GetStr(iterator.this);   <-- put it in a utf string
      Json_SetStr(recordField: %Trim(keyName): utfString);   <-- load it here
   EndDo;
   SQLRecord = json_getNext(SQLRecord);
EndDo;

OKay, I will try this.

[RootDeepak]

Hi Wim,

It's not working.

[wimjongman]

@NielsLiisberg can this only be done with the UTF-8 branch of NOXDB?

[NielsLiisberg]

The utf-8 support in the main branch is quite poor. It will encode / decode data as \u … unicode to get it back and forth between the application. So we made a branch - namely UTF8 where the graph is all in UTF8. However, the interface from ibm does not let you transfer c-strings back and forth leaving up huge limitations. We are planning to make the “master” main branch hybrid to carry both ebcdic and UTF8 with a new set of interfaces procedures. .. it is in mold ons. 16. apr. 2025 kl. 10.13 skrev Wim Jongman ***@***.***>:

…

@NielsLiisberg <https://github.com/NielsLiisberg> can this only be done with the UTF-8 branch of NOXDB? — Reply to this email directly, view it on GitHub <#102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVIPHQXHANFCXKAE4Q2UAD2ZYGKJAVCNFSM6AAAAAB3FAEDM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMBYG43DGOJYGI> . You are receiving this because you were mentioned.Message ID: ***@***.***> *wimjongman* left a comment (sitemule/noxDB#102) <#102 (comment)> @NielsLiisberg <https://github.com/NielsLiisberg> can this only be done with the UTF-8 branch of NOXDB? — Reply to this email directly, view it on GitHub <#102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVIPHQXHANFCXKAE4Q2UAD2ZYGKJAVCNFSM6AAAAAB3FAEDM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMBYG43DGOJYGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[NielsLiisberg]

I think we need a getStrUtf8 so the conversion will be perfect. UTF-8 is stored in the graph as Unicode escapes (in the main branch - most popular) and therefore, you will only be able to retrieve characters that can be mapped to the job CCSID. Now RPG has a real UTF-8 implementation, so it will be natural to have a jx_getStrUtf8 and jx_setStrUtf8