Fix issue with 411 Length Required HTTP error

[vkuznet]

Fixes #12348

Status

ready

Description

We observed in reqmgr2 logs the following:

[19/May/2025:12:10:10] reqmgr2-64d558846-2ckpr 137.138.154.184 "DELETE /reqmgr2/data/transferinfo/wangz_r-4-Run2024C_JetMET0_2024CDEReprocessing_250211_195607_8586 HTTP/1.1" 411 Length Required [data: 1964 in - out 244 us ] [auth: ok "xxx" "" ] [ref: "https://xxx.cern.ch" "WMCore.Services.Requests/v002" ]

This messages appears upon reqmgr2-tasks HTTP client calls /reqmgr2/data/transferinfo end-point which is served by CherryPy server. If DELETE request does not contain payload body the CherryPy complains with 411 error code. This PR provides a fix to setup Content-Length HTTP header for DELETE method within CherryPy server before processing this request. For more details please see #12348 (comment)

Notes:
through comprehensive R&D I found the following:

• some frontends, e.g. Go reverse proxy, drop Content-Length HTTP header if body is nil. At the same time it preserve ContentLength within HTTP request itself (not as separate HTTP header).
• different backend servers like CherryPy or Flask can handle or not such scenario internally
• I found that older version CherryPy server requires explicitly such header, but newer one may not

Based on my findings I opted to use solution to modify CherryPy server with proper configuration to gracefully handle this case.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

External dependencies / deployment changes

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 2 tests no longer failing
  • 1 changes in unstable tests
• Python3 Pylint check: failed
  • 20 warnings and errors that must be fixed
  • 6 warnings
  • 224 comments to review
• Pycodestyle check: succeeded
  • 59 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/709/artifact/artifacts/PullRequestReport.html

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 2 tests no longer failing
• Python3 Pylint check: failed
  • 20 warnings and errors that must be fixed
  • 6 warnings
  • 224 comments to review
• Pycodestyle check: succeeded
  • 59 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/710/artifact/artifacts/PullRequestReport.html

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 2 changes in unstable tests
• Python3 Pylint check: failed
  • 16 warnings and errors that must be fixed
  • 4 warnings
  • 108 comments to review
• Pycodestyle check: succeeded
  • 38 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/717/artifact/artifacts/PullRequestReport.html

[mapellidario]

thanks for the thorough investigation, i agree with the solution proposed. we have to live with what we have. thanks valentin

[amaltaro]

Even though this might be a working solution, I wonder why APS drops an HTTP header provided by the client. If there is no strong reason for that, then the correct way to fix it would be updating the APS code to stop doing that.

[vkuznet]

@amaltaro , it is not issue with APS but it is how GoLang team interpret HTTP standard. The standard does not specify if this header is mandatory or not in this particular situation, and therefore it is up to dev teams to treat it as they think the best. Because of that some frameworks discard it, others requests its explicitly. Bottom line, nothing I can do within APS itself. If you are not satisfied with it, we'll need to raise issue with GoLang development team.

[amaltaro]

I see. I thought it had been an implementation decision in APS. If it is in the GoLang HTTP library, then yes, I am afraid that there is no other place to make this change.

What we can do though is to upgrade the CherryPy version in WMCore - which is already in the plans before we freeze the code. So I'd suggest to come back to this in a week or so and test with a new version of CherryPy. Or perhaps we could even pip upgrade CherryPy in our reqmgr2 container and see if it will work out of the box.

[anpicci]

Thanks @vkuznet for sharing what you have found, and for the fix you have provided.
That's approved from my side.

Regarding @amaltaro 's comment, I won't keep this hanging until we update the CherryPy version we use, as the latter is not part of our Q2 priority plan. I would rather suggest merging this PR and coming back to this topic with a new issue once we have a working transition to CherryPy.

[vkuznet]

After further R&D I found that the 411 HTTP error code literally comes from WM stack rather the version of CherryPy.

First, I confirmed that using Flask Python framework to proof that using DELETE method and Content-Length:0 just works. For instance using this server codebase:

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route('/dump', methods=['GET', 'POST', 'PUT', 'DELETE', 'PATCH'])
def dump_request():
    """
    Endpoint to dump incoming HTTP request info.
    """
    request_info = {
        "method": request.method,
        "headers": dict(request.headers),
        "args": request.args.to_dict(),        # Query parameters
        "form": request.form.to_dict(),        # Form data
        "json": request.get_json(silent=True), # JSON body
        "data": request.get_data(as_text=True) # Raw body
    }

    # Optional: print to stdout
    print("Received Request:")
    for key, value in request_info.items():
        print(f"{key}: {value}")

    return jsonify(request_info), 200

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=5050)

I can perform the following curl call to it and it passes just fine:

curl -v -X DELETE http://localhost:5050/dump
...
* Connected to localhost (127.0.0.1) port 5050
* using HTTP/1.x
> DELETE /dump HTTP/1.1
> Host: localhost:5050
> User-Agent: curl/8.13.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Server: Werkzeug/3.1.3 Python/3.11.12
< Date: Mon, 26 May 2025 13:44:34 GMT
< Content-Type: application/json
< Content-Length: 186
< Connection: close
<
{
  "args": {},
  "data": "",
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "localhost:5050",
    "User-Agent": "curl/8.13.0"
  },
  "json": null,
  "method": "DELETE"
}

So, the Content-Length HTTP header is not required in Flask web server with DELETE method and no payload.

Then, I installed two CherryPy versions, one we are using and another using latest release. For testing I used the following web server:

import cherrypy

class RequestDumper:
    @cherrypy.expose
    @cherrypy.tools.json_out()
    def dump(self, **params):
        request = cherrypy.request
        headers = dict(request.headers)
        method = request.method
        try:
            body = request.body.read().decode('utf-8')
        except Exception as exp:
            print(f"HTTP request with method {method} does not allow to read body:\n{str(exp)}")
            body = "" # body is not sent

        return {
            "method": method,
            "headers": headers,
            "params": params,
            "body": body
        }

if __name__ == '__main__':
    cherrypy.config.update({
        'server.socket_host': '0.0.0.0',
        'server.socket_port': 5050,
        'log.screen': True
    })
    cherrypy.quickstart(RequestDumper())

With this server we get the following

curl -v -X DELETE http://localhost:5050/dump
...
* Connected to localhost (127.0.0.1) port 5050
* using HTTP/1.x
> DELETE /dump HTTP/1.1
> Host: localhost:5050
> User-Agent: curl/8.13.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: application/json
< Server: CherryPy/18.10.0
< Date: Mon, 26 May 2025 13:57:26 GMT
< Content-Length: 159
<
* Connection #0 to host localhost left intact
{"method": "DELETE", "headers": {"Remote-Addr": "127.0.0.1", "Host": "localhost:5050", "User-Agent": "curl/8.13.0", "Accept": "*/*"}, "params": {}, "body": ""}

But in a server log I see the caught exception:

HTTP request with method DELETE does not allow to read body:
KnownLengthRFile.read() takes from 1 to 2 positional arguments but 3 were given
127.0.0.1 - - [26/May/2025:09:57:26] "DELETE /dump HTTP/1.1" 200 159 "" "curl/8.13.0"

The internal CherryPy server error (if exception is not caught), is 500 Server error rather the 411 one. Therefore, using stock CherryPy server does not lead to 411 HTTP error when we send DELETE method and do not provide Content-Length:0. It will lead to 500 server error if we try to read the body.

Instead, the WM web framework somewhere sets constrain that DELETE (and most likely any othe HTTP ) method must be accomplished with Content-Length because it tries to prevent to read the body and therefore protect against possible failure.

Therefore, my conclusion remains the same:

• we don't need to supply neither body payload or Content-Length to web framework using DELETE method (regardless of the language)
• Python frameworks align with others (Go web framework) where no requirement is set on Content-Length HTTP header using DELETE method
• the 411 error is triggered by some place in WM web framework. Unfortunately, I can't find exact place in WM framework where this requirements is set due to complexity and multiple layers/wrappers used within WM web framework. But I scanned WMCore code and didn't find any explicit set of 411 code even though there are multiple places where Content-Length is checked/used.

For instance, the 411 error occurs from WTLogger class where in access method we read cherrypy.response, see https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WebTools/Root.py#L89-L126 (lines 89-126) since this block matches the log output. Therefore, if we require to read body of HTTP request we must supply the Content-Length. Please note, there are other places where WM framework where Content-Length is used.

Therefore, the easiest way to fix reported issue is what I provided in this PR, i.e. before internal web server will process incoming HTTP request we set up Content-Length:0 header for DELETE method if body is not provided.

I hope this further clarifies the underlying issue.

[anpicci]

@vkuznet may you squash the commits, and we eventually merge them?

[vkuznet]

It was only one commit, but I took liberty to improve docstring for proposed function to reflect my later findings.

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: failed
  • 1 new failures
  • 4 changes in unstable tests
• Python3 Pylint check: failed
  • 16 warnings and errors that must be fixed
  • 4 warnings
  • 108 comments to review
• Pycodestyle check: succeeded
  • 38 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/725/artifact/artifacts/PullRequestReport.html

[vkuznet]

I saw that new unit test failed

WMCore_t.Storage_t.StageInMgr_t.StageInMgrUnitTest:testStageInMgr changed from success to error

But it is unstable or rather based from its log I see it is not unrelated:

Unable to find site local config file:
/cvmfs/cms.cern.ch/SITECONF/T1_US_FNAL/JobConfig/site-local-config.xml

[anpicci]

Thanks @vkuznet, I'm going to retrigger Jenkins to check that it's a temporary issue

[anpicci]

test this please

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 1 changes in unstable tests
• Python3 Pylint check: failed
  • 16 warnings and errors that must be fixed
  • 4 warnings
  • 108 comments to review
• Pycodestyle check: succeeded
  • 38 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/726/artifact/artifacts/PullRequestReport.html

[anpicci]

I'm merging the PR