500 error when editing

[FP234]

Hello,

I've been starting to use MerMEId for Elgar works records, but have been encountering an error (three times in the past two days, on two different records). I also had the same error when using the Demo version a couple of months ago. I don't think there's been any patten in terms of when/why it happens (I was editing a text field the most recent time), but it does mean I lose recent changes as I can't press save.

The error is: "Submission response status: 500 Error type: resource-error URI: http://127.0.0.2:8080/apps/mermeid/data/elgar_inthesouth.xml"

David Lewis suggested this looks like a similar bug to #28 or #88? I've copied and pasted the log extract from around the time of the error (12:17:35) below.

Many thanks for any ideas! I'll keep an eye on if/when it happens again as I continue.
Frankie

2025-01-23 12:17:35,041 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "limiter", "message": "start: filter", "path": "/xforms-server", "method": "POST"}
2025-01-23 12:17:35,044 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "limiter", "message": "start: chain", "path": "/xforms-server", "method": "POST", "wait": "1"}
2025-01-23 12:17:35,045 [qtp881977454-33] DEBUG (FormRunnerAuth.scala [findCredentialsFromContainerOrHeaders]:134) - using Container method
2025-01-23 12:17:35,046 [qtp881977454-33] DEBUG (FormRunnerAuth.scala [findCredentialsFromContainerOrHeaders]:139) - usernameOpt: None, roles: None
2025-01-23 12:17:35,046 [qtp881977454-33] WARN (FormRunnerAuth.scala [getCredentialsAsHeadersUseSession]:86) - not setting credentials headers because credentials are not found
2025-01-23 12:17:35,048 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "service", "message": "start: handle"}
2025-01-23 12:17:35,081 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "xforms", "message": "ajax with update events", "uuid": "828fe0b34fbc510f2a713564bae2c360cce74efc"}
2025-01-23 12:17:35,081 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "xforms", "message": "before document lock", "uuid": "828fe0b34fbc510f2a713564bae2c360cce74efc"}
2025-01-23 12:17:35,081 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "xforms", "message": "got document lock", "path": "/xforms-server", "method": "POST", "uuid": "828fe0b34fbc510f2a713564bae2c360cce74efc", "wait": "0"}
2025-01-23 12:17:35,136 [qtp881977454-502] INFO (Transform.java [extractSerializationProperties]:414) - Setting serialization property: method = xml
2025-01-23 12:17:35,136 [qtp881977454-502] INFO (Transform.java [extractSerializationProperties]:414) - Setting serialization property: media-type = application/xml
2025-01-23 12:17:35,189 [qtp881977454-502] WARN (TransactionManager.java [close]:421) - Transaction was not committed or aborted, auto aborting!
2025-01-23 12:17:35,190 [qtp881977454-502] ERROR (XMLDBStore.java [evalWithCollection]:230) - A resource with the same name already exists in the target collection '/db/apps/mermeid-data', and you do not have write access on that resource.
org.xmldb.api.base.XMLDBException: A resource with the same name already exists in the target collection '/db/apps/mermeid-data', and you do not have write access on that resource.
      at org.exist.xmldb.function.LocalXmldbCollectionFunction.apply(LocalXmldbCollectionFunction.java:52) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.function.LocalXmldbCollectionFunction.apply(LocalXmldbCollectionFunction.java:50) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.AbstractLocal.lambda$6(AbstractLocal.java:218) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.AbstractLocal.lambda$5(AbstractLocal.java:152) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.function.LocalXmldbFunction.apply(LocalXmldbFunction.java:48) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.txn.bridge.InTxnLocalCollection.withDb(InTxnLocalCollection.java:59) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.txn.bridge.InTxnLocalCollection.withDb(InTxnLocalCollection.java:53) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.AbstractLocal.lambda$4(AbstractLocal.java:152) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.LocalCollection.lambda$29(LocalCollection.java:823) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.LocalCollection.storeXMLResource(LocalCollection.java:622) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.LocalCollection.storeResource(LocalCollection.java:572) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.LocalCollection.storeResource(LocalCollection.java:561) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.functions.xmldb.XMLDBStore.evalWithCollection(XMLDBStore.java:226) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.functions.xmldb.XMLDBAbstractCollectionManipulator.eval(XMLDBAbstractCollectionManipulator.java:169) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.BasicFunction.eval(BasicFunction.java:73) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:62) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.LetExpr.eval(LetExpr.java:98) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.LetExpr.eval(LetExpr.java:110) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.LetExpr.eval(LetExpr.java:110) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.PathExpr.eval(PathExpr.java:280) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.TryCatchExpression.eval(TryCatchExpression.java:148) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.PathExpr.eval(PathExpr.java:280) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.SwitchExpression.eval(SwitchExpression.java:109) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.PathExpr.eval(PathExpr.java:280) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.SequenceConstructor.eval(SequenceConstructor.java:82) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:98) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.PathExpr.eval(PathExpr.java:280) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.SequenceConstructor.eval(SequenceConstructor.java:82) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.PathExpr.eval(PathExpr.java:280) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.PathExpr.eval(PathExpr.java:280) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.XQuery.execute(XQuery.java:445) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.XQuery.execute(XQuery.java:348) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xquery.XQuery.execute(XQuery.java:335) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.http.urlrewrite.XQueryURLRewrite.runQuery(XQueryURLRewrite.java:677) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.http.urlrewrite.XQueryURLRewrite.service(XQueryURLRewrite.java:239) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:292) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:571) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [exist.uber.jar:6.3.0-SNAPSHOT]
      at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [exist.uber.jar:6.3.0-SNAPSHOT]
      at java.lang.Thread.run(Thread.java:750) [?:1.8.0_432]
Caused by: org.exist.security.PermissionDeniedException: A resource with the same name already exists in the target collection '/db/apps/mermeid-data', and you do not have write access on that resource.
      at org.exist.collections.MutableCollection.checkPermissionsForAddDocument(MutableCollection.java:1707) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.collections.MutableCollection.validateXMLResourceInternal(MutableCollection.java:1542) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.collections.MutableCollection.storeXmlDocument(MutableCollection.java:1207) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.collections.MutableCollection.storeDocument(MutableCollection.java:1148) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.collections.LockedCollection.storeDocument(LockedCollection.java:367) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.storage.NativeBroker.storeDocument(NativeBroker.java:2154) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.LocalCollection.lambda$19(LocalCollection.java:657) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      at org.exist.xmldb.function.LocalXmldbCollectionFunction.apply(LocalXmldbCollectionFunction.java:50) ~[exist.uber.jar:6.3.0-SNAPSHOT]
      ... 75 more
2025-01-23 12:17:35,235 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "xforms", "message": "after cacheOrStore", "document cache current size": "1", "document cache max size": "50"}
2025-01-23 12:17:35,235 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "service", "message": "end: handle", "time": "187 ms"}
2025-01-23 12:17:35,235 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "limiter", "message": "end: chain", "time": "192 ms"}
2025-01-23 12:17:35,235 [qtp881977454-33] INFO (LifecycleLogger.scala [event]:102) - event: {"request": "190", "session": "node0180624o6fabsq1646dplpk7ezw2", "source": "limiter", "message": "end: filter", "time": "199 ms"}

[DILewis]

How are rich text fields cleaned up and displayed? In the examples that break for @FP234 , it looks like entities in the rich text are causing the breakage. So for example, in one file that causes the reported error, we have:

<notesStmt xml:id="notesStmt_77b7a0f5">
  <annot type="links" xml:id="annot_f3212f50">
    <ptr mimetype="" xml:id="ptr_d6e131"/>
  </annot>
  <annot type="general_description" xml:id="annot_f93a11e3">
    <p>A song for voice and piano adapted from the 'Canto popolare' theme from <rend fontstyle="italic">In the South</rend>, Op. 50. In a letter to Alfred Littleton on 2 August 1904, Elgar wrote: 'Here is an adaptation of the melody which people liked so much from the Overture. The words just suit the mood &amp; are, of course (?) non copyright. I wish you wd. get it out quickly as I think it should be quite popular: it wd. be necessary to issue several keys at first. I am also arranging, quite easily, the same air for Violin, Cello, Viola &amp;c. Let me know what you think of them'. Novello issued 'In moonlight' in G, F, and E-flat, and eight arrangements for various instruments of the 'Canto popolare'.</p>
  </annot>
</notesStmt>

When I load this into a fresh MerMEId instance, the entry is shown, but is read only. If I delete the text to remove the &s, the file loads successfully.
My guess would be that somewhere a conversion pipeline is getting confused, but don't know/remember enough to be able to follow it. If this is an existing library, there may be a big report there.

[peterstadler]

Yes, there is quite some preprocessing involved for converting the rich text HTML to MEI:

MerMEId/filter/xsl/filter_put.xsl

Lines 288 to 295 in 871a01c

	<xsl:template name="replace_nodes">
	<xsl:param name="text"/>
	<!-- Wrap contents in a temporary <div> root element and parse XML -->
	<xsl:variable name="fragment">&lt;div xmlns="http://www.w3.org/1999/xhtml"&gt;<xsl:value-of select="replace(replace(replace($text, '&amp;(\s)', '&amp;amp;$1'), '&amp;nbsp;', ' '), '&lt;br&gt;', '&lt;br/&gt;')"/>&lt;/div&gt;</xsl:variable>
	<xsl:for-each select="parse-xml($fragment)/h:div">
	<xsl:apply-templates/>
	</xsl:for-each>
	</xsl:template>

As you can see it's some awkward string replacing for several entities. Yet & should be properly treated already - I would have expected such errors only for other entities? What MerMEId version are you running?

[DILewis]

Ooh, gnarly!
I could reproduce the error on a fresh checkout of the github develop branch yesterday (6 Feb, 2025). If I get the chance, I'll try running that transform on the single file from the command line and see what comes out...

[DILewis]

Ok, this is fun. Running neither filter_put.xsl nor filter_get.xsl causes any error, and the results look sane. Running the get then transform and then the put transform causes an error, though.
The specific error, on parse-xml() is about the use of &c. in the 'general description' string: "The reference to entity "c." must end with the ';' delimiter." If I delete the characters after the entity, then there is no error.
I can't immediately see what's changed – I can run the same transform on the source file, without &c. causing it to break.
Unless anyone has any bright ideas, I'll try for a minimal example and dig a bit deeper into what has changed.

[imranasif87]

I've been working on this bug and have been debugging it to determine its cause. I discovered that the issue arises when the TinyMCE editor is not focused, and there are special characters (like '&' or others) within the editor. In this case, the editor fails to parse them into a valid XML format. However, when the editor is focused and then loses focus, it properly encodes the special characters into a valid XML format, allowing us to save the XML file. The video below should help clarify the bug.

Screencast.from.2025-05-15.12-50-04.mp4

I implemented several solutions to address the issue. One of the approaches I tried was to serialize the TinyMCE editor into valid XML content before saving.

1.

<xf:iterate nodeset="instance('data-instance')//m:annot[@type='general_description']" index="i">
                <xf:setvalue ref="." value="xxf:serialize(., 'oxf:xhtml-sanitizer')"/>
             </xf:iterate>

2.

<xf:iterate nodeset="instance('data-instance')//m:annot[@type='general_description']" index="i">
            <xf:setvalue 
                ref=".[position() = xxf:index('i')]" 
                value="xxf:serialize(.[position() = xxf:index('i')])"/>
        </xf:iterate>

3.

<xf:setvalue ref="." value="xxf:serialize(., 'oxf:xhtml-sanitizer')"/>

4.

<xf:setvalue 
  ref="instance('data-instance')//m:annot[@type='general_description']" 
  value="
    for $annot in instance('data-instance')//m:annot[@type='general_description']
    return xxf:serialize($annot, 'oxf:xhtml-sanitizer')
  "/>

The solutions above don't resolve the issue. When I click the save button, it freezes and doesn't save the file. After analyzing the situation, I realized I should check the Orbeon version (currently 2018.2.1.201902072242 CE), as this might be causing the problem. The xf:iterate feature is available starting in Orbeon Form 4.8.0. Do I need to upgrade to the latest version of Orbeon?

See the TinyMCE editor Limitation: https://doc.orbeon.com/form-runner/component/rich-text-editor#limitations

[imranasif87]

I found the solution and tested it using all of the following special characters.

! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~ À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï Ñ Ò Ó Ô Õ Ö Ø Ù Ú Û Ü Ý ß à á â ã ä å æ ç è é ê ë ì í î ï ñ ò ó ô õ ö ø ù ú û ü ý ÿ ± × ÷ ∓ ∑ ∏ √ ∫ ∞ ∇ ≈ ≠ ≤ ≥ ∂ ∆ ∩ ∪ ∈ ∉ ∅ ∃ ∀ ∧ ∨ ∴ ∵ ∠ ∞ ∝ ∞ $ ¢ £ ¥ € ₩ ₹ ₱ ฿ ₽ ₴ ₪ ¡ ¿ ‽ • ′ ″ ‡ † ‾ ⁂ § ¶ ‡ ‣ ‣ ‣ ← ↑ → ↓ ↔ ↕ ↖ ↗ ↘ ↙ ⇐ ⇑ ⇒ ⇓ ⇔ ⇕ ⟵ ⟶ ⟷ ⟹ © ® ™ ° ¬ µ ¶ † ‡ ‰ ‱ • ‣ ※ ☑ ✓ ✔ ✕ ✖ ✗ ✘ ✚ ✛ ✜ ✝ ✞ ✟ ─ │ ┌ ┐ └ ┘ ├ ┤ ┬ ┴ ┼ ░ ▒ ▓ █ ▀ ▄ ▌ ▐ ▓ ■ □ ▲ △ ▼ ▽ ◆ ◇ ○ ● ◯ ◐ ◑ ◒ ◓ ◔ ◕ ◖ ◗ ⁰ ¹ ² ³ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹ ⁺ ⁻ ⁼ ⁽ ⁾ ₀ ₁ ₂ ₃ ₄ ₅ ₆ ₇ ₈ ₉ ₊ ₋ ₌ ₍ ₎

The solution is
<xf:setvalue ref="instance('data-instance')//m:annot[@type='general_description']" value="replace(., '&amp;', '&amp;amp;')" />

The above solution specifically replaces the '&' character with its XML-parsable equivalent '&', as XML cannot parse the '&' symbol directly. The TinyMCE editor does not automatically handle this conversion when the text is modified.

The problem with that solution is that it only works for a single TinyMCE editor. If there are multiple editors, it won’t be effective. An alternative solution is to explicitly handle each editor individually, as shown below.

<xf:setvalue ref="instance('data-instance')//m:annot[@type='general_description'][1]"
                     value="replace(., '&amp;', '&amp;amp;')" />

<xf:setvalue ref="instance('data-instance')//m:annot[@type='general_description'][2]"
                     value="replace(., '&amp;', '&amp;amp;')" />

<xf:setvalue ref="instance('data-instance')//m:annot[@type='general_description'][3]"
                     value="replace(., '&amp;', '&amp;amp;')" />

and so on...

We could accomplish the above task using xf:repeat or xf:iterate, but these elements are not functioning in the current version of Orbeon Forms.

[krpage]

To clarify: should the immediate 'minimal viable' workaround be to submit a PR with the fix above for a single TinyMCE instance? If we can move this forward I think we should (perfection being the enemy of the good).

But also to have a broader discussion (in issue #256) about broader workarounds via component upgrades or replacements (e.g. orbeon).

[imranasif87]

I've resolved the issue for multiple editors. Here's the updated code:

<xf:var name="nodes" 
             select="instance('data-instance')//m:annot[@type='general_description']"/>
        <xf:var name="count" select="count($nodes)"/>

         <xf:setvalue ref="instance('tempeditor')/index" value="1"/>
         <xf:action while="instance('tempeditor')/index &lt;= $count">
            <!-- Get current position -->
            <xf:var name="pos" select="instance('tempeditor')/index"/>
            
            <!-- Apply replacement to current annotation -->
            <xf:setvalue
                ref="$nodes[position() = $pos]"
                value="replace($nodes[position() = $pos], '&amp;', '&amp;amp;')" />

                 <!-- Increment counter -->
            <xf:setvalue ref="instance('tempeditor')/index" value="$pos + 1"/>
         </xf:action>

After fixing this, I noticed that the & character is now being saved as & in the XML file. Please see the screenshot below for reference.

and it's rendered on the HTML page as shown below. This happens because the & character is encoded twice in the XML file due to XML formatting rules. We can't use the & symbol directly in an .xml file.

However, it displays correctly in the TinyMCE editor after the file is loaded. Is this expected behaviour?

[peterstadler]

Have you tried with disabling the replacements from the filter_put.xsl at

MerMEId/filter/xsl/filter_put.xsl

Lines 288 to 295 in 871a01c

	<xsl:template name="replace_nodes">
	<xsl:param name="text"/>
	<!-- Wrap contents in a temporary <div> root element and parse XML -->
	<xsl:variable name="fragment">&lt;div xmlns="http://www.w3.org/1999/xhtml"&gt;<xsl:value-of select="replace(replace(replace($text, '&amp;(\s)', '&amp;amp;$1'), '&amp;nbsp;', ' '), '&lt;br&gt;', '&lt;br/&gt;')"/>&lt;/div&gt;</xsl:variable>
	<xsl:for-each select="parse-xml($fragment)/h:div">
	<xsl:apply-templates/>
	</xsl:for-each>
	</xsl:template>

This might introduce the double escaping of the &.

Regardless, would you mind sending in a Pull Request so you get the proper credits for the code—and we can both check the resulting Docker image?

[imranasif87]

Thanks, @peterstadler, for your comment. The issue was that ampersands (&) were being saved in the XML file as double-escaped (&), which then rendered in HTML as &. I ended up combining both approaches: I made a small update to the filter_put.xsl file, and also modified my solution to convert double ampersands back to single ones just before submission.

I made some adjustments in the filter_put.xsl file using the replace function. I updated the expression from:

replace($text, '&(\s)', '&$1')

to

replace($text, '&', '&')

This is my revised solution, incorporating a small change from the earlier version.

<xf:var name="nodes" 
             select="instance('data-instance')//m:annot[@type='general_description']"/>
        <xf:var name="count" select="count($nodes)"/>

         <xf:setvalue ref="instance('tempeditor')/index" value="1"/>
         <xf:action while="instance('tempeditor')/index &lt;= $count">
            <!-- Get current position -->
            <xf:var name="pos" select="instance('tempeditor')/index"/>
            
            <!-- Apply replacement to current annotation -->
            <xf:setvalue
                ref="$nodes[position() = $pos]"
                value="replace($nodes[position() = $pos], '&amp;amp;', '&amp;')" />

                 <!-- Increment counter -->
            <xf:setvalue ref="instance('tempeditor')/index" value="$pos + 1"/>
         </xf:action>

I've tested the solution against all edge cases, and it works as expected. I'll be submitting the pull request shortly.