Custom type registration : instrusive API

[pfeatherstone]

Currently, the API for registering custom types is as follows:

using nlohmann::json;

namespace ns {
    void to_json(json& j, const person& p) {
        j = json{{"name", p.name}, {"address", p.address}, {"age", p.age}};
    }

    void from_json(const json& j, person& p) {
        j.at("name").get_to(p.name);
        j.at("address").get_to(p.address);
        j.at("age").get_to(p.age);
    }

It would be great if there was a MACRO-style registration a bit like what msgpack-c uses:

struct person 
{
   std::string name;
   std::string address;
   int age;
   MSGPACK_DEFINE_MAP(name, address, age);
};

or yas:

struct   person
{
   std::string name;
   std::string address;
   int age;
    YAS_DEFINE_STRUCT_SERIALIZE("person", name, address, age);
};

or

struct   person
{
   std::string name;
   std::string address;
   int age;
};
YAS_DEFINE_INTRUSIVE_SERIALIZE("person", name, address, age);

[pfeatherstone]

By the way, if you look at those implementations, they are doing some crazy macro expansions which i'm not brave enough to undertake. But the end result is great, as you can see above. Super clean and easy. It's almost providing compile time reflection in C++. I know this repo isn't a free code service, but if one is looking for suggestions, here is one.

[nlohmann]

I understand the idea. Do I assume correctly that you would assume a "canonic" serialization to

{
    "name": "a name value",
    "address": "an address value",
    "age": 42
}

[pfeatherstone]

I'm not sure what you mean by "canonic" serialization, but in case of JSON serialisation, then yes i would assume a 'person' object would serialise to your example above. What would be great is if there were casting functions like:
nlohmann::json::cast_to : nlohmann::json -> person
nlohmann::json::cast_from: person -> nlohmann::json
I don't know if that's possible. You have the first one already as mentioned here https://github.com/nlohmann/json#arbitrary-types-conversions though the object type has to be in the ns namespace. Is that strictly necessary?

[nlohmann]

With "canonic" I meant that

1. We always serialize to a JSON object.
2. The names of the variables are the names of the keys in that object.

[nlohmann]

If I understand the C++ macro magic correctly, the library needs to make an assumption on the upper bound of member variables.

Assuming three members, a possible implementation would be:

#include <iostream>
#include "json.hpp"

using json = nlohmann::json;

struct person
{
   std::string name;
   std::string address;
   int age;
};

#define YAS_DEFINE_INTRUSIVE_SERIALIZE(type, var1, var2, var3) \
   void to_json(nlohmann::json& j, const type& t)\
   {\
       j[#var1] = t.var1;\
       j[#var2] = t.var2;\
       j[#var3] = t.var3;\
   }\
   void from_json(const nlohmann::json& j, type& t)\
   {\
       j.at(#var1).get_to(t.var1);\
       j.at(#var2).get_to(t.var2);\
       j.at(#var3).get_to(t.var3);\
   }

YAS_DEFINE_INTRUSIVE_SERIALIZE(person, name, address, age)

int main() {
    person p;
    p.name = "Ned Flanders";
    p.age = 65;
    p.address = "Springfield";
    
    std::cout << json(p) << std::endl;

    json j;
    j["name"] = "Homer Simpson";
    j["address"] = "Springfield";
    j["age"] = 42;

    person s = j;

    std::cout << json(s) << std::endl;
}

Output:

{"address":"Springfield","age":65,"name":"Ned Flanders"}
{"address":"Springfield","age":42,"name":"Homer Simpson"}

Is this what you have in mind?

[pfeatherstone]

Yes exactly. I hadn't realised the macro expansion had an upper bound on the number of arguments. I thought it used VA_ARGS and worked automagically :)
I guess it would be nice if it worked with nested custom types (or whatever the correct term is). For example:

struct child
{
   std::string name;
   int age;
};
NLOHMANN_SERIALISE(child, string, age);

struct parent
{
   std::vector<child> children;
};
NLOHMANN_SERIALISE(parent, children);

If that makes sense.
And having conversion functions to_nlohmann and from_nlohmann (instead of to_json and from_json) means it still supports the other binary serialisation protocols (bson, msgpack, etc)
This would be an insanely powerful feature.

[nlohmann]

• I have to look into VA_ARGS, but what I read seems that it only works in printf-like calls. Anything else assumes an upper bound of arguments, but at least the user does not need to be aware of what's going on behind the curtains.
• Your example with nested types works without adjustments.
• What's the point of changing the names of the functions? You can still generate MessagePack from a JSON value by calling json::to_msgpack.

[pfeatherstone]

I have no real opinions on function names. Whatever works and Whatever you think is best. I was just clarifying that conversion functions were to and from the json object, not a json serialised string. What do you think overall ? Do you like the macro expansion API for registering types ?

[FrancoisChabot]

One of the big drawback here is that since json, as a format, is meant to be human-readable, that syntax only looks nice if you have a 1-to-1 match between variable names and the serialized tag, which is inherently precluded by many organizations' coding standards. If such a feature went in, I would expect an immediate followup request to be able to remap names.

At that point, it's worth pointing out that there is a middle ground possible between the current approach and the full macro-based approach:

Something along these lines (with better names):

struct person 
{
   std::string name;
   std::string address;
   int age_;
   
   void as_json(nlohmann::Visitor& v) { 
      v("name", name); 
      v("address", address); 
      v("age", age_); 
   }
};

I think this strikes a good balance between not having to essentially write the same code twice, while remaining debugable and flexible.

[pfeatherstone]

@FrancoisChabot I like your suggestion. It's a good middle ground. I would maybe go a tiny bit further and use something like https://github.com/Neargye/nameof so you don't have to specify the variable name (though please don't use something that depends on C++17 and beyond)

[pfeatherstone]

I still think the macro api is awesome. But looking at how yas and msgpack-c implemented it, it doesn't look debug-friendly that's for sure.

[KonanM]

namespace detail{
    constexpr size_t nArgs(std::string_view str) {
        //constexpr count_if is C++20
        size_t nargs = 1;
        for (auto c : str) if (c == ',') ++nargs;
        return nargs;
    }
    template<typename T, size_t... Is>
    void to_json_impl(nlohmann::json::json& j, const T& t, std::index_sequence<Is...>) {
        ((j[std::get<Is>(t._memberNames)] = std::get<Is>(t.members())), ...);
    }
    template<typename T, size_t... Is>
    void from_json_impl(const nlohmann::json::json& j, T& t, std::index_sequence<Is...>) {
        (j.at(std::get<Is>(t._memberNames)).get_to(std::get<Is>(t.members())), ...);
    }
}

#define NLOHMANN_DEFINE_STRUCT_SERIALIZE(Type, ...) \
constexpr inline decltype(auto) members() const { return std::tie(__VA_ARGS__); } \
constexpr inline decltype(auto) members() { return std::tie(__VA_ARGS__); }  \
static constexpr std::string_view _typeName = #Type;\
static constexpr std::array<std::string_view, detail::nArgs(#__VA_ARGS__)> _memberNames = \
[](){ std::array<std::string_view, detail::nArgs(#__VA_ARGS__)> out{}; \
size_t off = 0, next = 0; std::string_view ini(#__VA_ARGS__); \
for(auto& strV : out){ next = ini.find_first_of(',', off); strV = ini.substr(off, next - off); off = next + 1;} return out;}();\
friend void to_json(nlohmann::json::json& j, const Type& t) { detail::to_json_impl(j, t, std::make_index_sequence<_memberNames.size()>{}); } \
friend void from_json(const nlohmann::json::json& j, const Type& t) { detail::from_json_impl(j, t, std::make_index_sequence<_memberNames.size()>{}); } 

struct person{
    std::string name;
    std::string address;
    int age;
    NLOHMANN_DEFINE_STRUCT_SERIALIZE(person, name, address, age)
};

If you don't like macro heavy implementations, something along this lines might be an option (this is implemented using C++17, but should be easy to port to C++11). If something like this is viable then I could also try to rewrite it for C++11.

[pfeatherstone]

Yeah that looks great but very much a C++17 implementation. std::string_view is available in the nonstd libraries, std::tie isn't constexpr in C++11 and std::index_sequence is C++14. Do you think it's possible as a C++11 implementation?

[pfeatherstone]

As a user i don't care about the implementation, that's hidden away from me. So long as it works i'm happy. Personally though, i am restricted to C++11 due to what i develop for. But that's just me, maybe this API could be enabled in C++17 only. Tough if you can't use it.