Parsing a particular dialogue format #431

IngwiePhoenix · 2025-03-06T08:30:17Z

IngwiePhoenix
Mar 6, 2025

Hello there!

I am working together with some friends to AI-translate the text contents of Persona 4 Golden's dialogues.

The format is rather easy, but I want to use an AST for several reasons (keeping the extra metadata, positions, etc.).

In fact, this is technically all that it should take:

type DialogDatabase struct {
	Dialogues []*Dialogue
}

type Dialogue struct {
	Type string
	Identifier string
	Speaker string `@@`
	Boxes []*TextBox
}

type TextBox struct {
	Segments []*DialogueElement
}

type DialogueElement struct {
	Type string
	Content string
}

However, I am considerably new to writing lexers and parsers... So, here is a snippet:

[msg MSG_004 [Nanako]]
[f 0 5 65278][f 2 1][f 4 6 10 1 65535 1][f 3 1 1 0 0 6]I can toast bread... and cook sunny-side[n]up eggs in the morning.[n][f 1 3 65535][f 1 1][e]
[f 0 5 65278][f 2 1][f 3 1 1 0 0 7]Dad can't cook, so I buy dinner.[n][f 1 3 65535][f 1 1][e]

This should deserialize into:

DialogDatabase{
    Dialogues: [
        Dialogue{
            Type: "msg",
            Identifier: "MSG_004",
            Speaker: "[Nanako]",
            Boxes: [
                TextBox{
                    Segments: [
                        DialogueElement{Type: "metadata", Content: "[f 0 5 65278]"},
                        DialogueElement{Type: "metadata", Content: "[f 2 1]"},
                        DialogueElement{Type: "metadata", Content: "[f 4 6 10 1 65535 1]"},
                        DialogueElement{Type: "metadata", Content: "[f 3 1 1 0 0 6]"},
                        DialogueElement{Type: "text", Content: "I can toast bread... and cook sunny-side"},
                        DialogueElement{Type: "text", Content: "up eggs in the morning."},
                        DialogueElement{Type: "metadata", Content: "[f 1 3 65535]"},
                        DialogueElement{Type: "metadata", Content: "[f 1 1]"},
                        DialogueElement{Type: "boxend", Content: "[e]"},
                        // ... repeat untill:
                        // - Either '\n\n' (two consequtive newlines) followed by next Dialogue{...}
                        // - EOF
                    ]
                }
            ]
        }
    ]
}

How would I realize that? Do I need to write a custom Lexer as well or would the default one work?

Thank you very much and kind regards,
Ingwie

alecthomas · 2025-03-06T08:40:00Z

alecthomas
Mar 6, 2025
Maintainer

It's a bit hard to tell how this is structured, but here are a couple of observations:

You may not need a custom lexer. You capture up until some terminal, you can use ~<expr>.
You won't be able to create a "Type" field. What you'll need to do is use Participle's union type support to parse each into their own Go type. This will be better too, as you can capture the metadata integers into a slice of ints directly.

1 reply

IngwiePhoenix Mar 6, 2025
Author

Thanks for the quick reply!

I'm playing around to see how I can get this to work - but I hadn't thought of using the unions. That said, I do need to capture the MSG_004 as a whole ID (string) since I do plan to walk the AST and write everything back out once I am done translating the text-segments (those are the only ones I am genuenly interested in, but in order to write out a valid file, I need to also capture and thus store the metadata).

Initially, I attempted to use a RegExp for this... perhaps this makes it a little clearer: /((\[(?<category>msg|sel) (?<typ>(MSG|SEL)_(?<id>\d+)+)(\s(?<speaker>\S*))?\])\n(?<body>[\s\S]*?)(?:\n\n|$))/giu (JS/ES6 in Node). But since this RegExp is kinda spaghetti, I am trying to port this into a proper parser...this parser here. :) And learn about writing them while I am at it.

Again, thanks for the pointers! =)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Parsing a particular dialogue format #431

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Parsing a particular dialogue format #431

Uh oh!

IngwiePhoenix Mar 6, 2025

Replies: 1 comment · 1 reply

Uh oh!

alecthomas Mar 6, 2025 Maintainer

Uh oh!

IngwiePhoenix Mar 6, 2025 Author

IngwiePhoenix
Mar 6, 2025

Replies: 1 comment 1 reply

alecthomas
Mar 6, 2025
Maintainer

IngwiePhoenix Mar 6, 2025
Author