Adding pilot registrations and authentification (Router)

[Robin-Van-de-Merghel]

Changes

Endpoints

Adding a pilot service with some endpoints:

• POST / creates a pilot with (if not prevented) a secret
• DELETE / deletes pilots by stamp
• DELETE /interval deletes pilots that lived more than n days
• POST /token exchanges a pilot secret for a token
• POST /refresh-token refresh a pilot token
• POST /fields/secrets creates secrets
• PATCH /fields/secrets associates a pilot with a secret
• PATCH /fields/jobs associates a pilot with jobs
• PATCH /fields helps modifying pilot fields (benchmark, gridsite, ...)
• GET /search searchs for pilots with parameters

Note

The DELETE /interval is there because we need it directly and because it is faster, but we can simplify it with GET /search then DELETE /.

Security Model

As the security model dictates, pilot secrets are strings, and hashed in the db itself.

Important

For the JWT perspective, we need to chose whether a pilot will need refresh tokens or not, and how long a token will live to implement it.

These changes are mandatory for this PR.

After offline discussions: A pilot will have a different token (refresh and access), and with a different duration.

[Robin-Van-de-Merghel]

The failed CI i'm not sure if I have to regenerate the client manually.

[aldbr]

The failed CI i'm not sure if I have to regenerate the client manually.

Yes, you need to regenerate the client manually, here is the documentation: https://github.com/DIRACGrid/diracx/blob/main/docs/CLIENT.md#updating-the-client

If you have any trouble, please let me know

[fstagni]

These look a bit fragile (e.g. at the moment we are effectively only supporting MySQL, but what if we add support also for e.g. PG?).
Maybe there's nothing different that can be done, but worth checking.

[Robin-Van-de-Merghel]

Just went to the code of SQLAlchemy, there's indeed an IntegrityError, but nothing is generic. We have to get some db-specific error: psycopg2.errors.ForeignKeyViolation for postgres, if error_code == 2291: for oracle, ...

[aldbr]

Can't you rely on an error code instead of relying on a string at least?
Also, it seems you are not using and testing the case where PilotAlreadyExistsError is raised (or I possibly missed it)

[Robin-Van-de-Merghel]

If we check if an error is an instance of another module pymysql we could potentially catch some errors as code that are specific on a db. And even with that, I saw errors where people had to use both IntegrityError from sql-alchemy and pymy integrity error because of a bad handling..

It is not pretty, and you can read this response: https://stackoverflow.com/a/70714697

Also, it seems you are not using and testing the case where PilotAlreadyExistsError is raised (or I possibly missed it)

This part add_pilot_credentials is not used yet but soon will be when Dirac or another entity will register pilots on DiracX and add credentials. I currently didn't catch it, because HTTPExceptions are to be raised on a router, and in the logic it will be automatically raised.
I don't know if it is fine to raise an error from the logic and raise the same one to the router: in a way it helps understand from the logic the potential, in another, it adds code...

[Robin-Van-de-Merghel]

I'll open an issue for this, to later fix this

[Robin-Van-de-Merghel]

Modified from (PilotID, secret) login request to (PilotRef, secret), see this issue I opened about it.

[Robin-Van-de-Merghel]

Tested with this Pilot PR version and worked successfully. Could retrieve a DiracX token from a Pilot.

[Robin-Van-de-Merghel]

If someone has a solution for this CI, I'm all ears.

I moved a function as suggested above to diracx.logic, and it seems to have destroyed OSDB? (I don't use OpenSearch).

[aldbr]

Can't you rely on an error code instead of relying on a string at least?
Also, it seems you are not using and testing the case where PilotAlreadyExistsError is raised (or I possibly missed it)

[Robin-Van-de-Merghel]

[DB Specific bug:]

(pymysql.err.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'RETURNING `PilotAgents`.`PilotID`' at line 1")
[SQL: INSERT INTO `PilotAgents` (`InitialJobID`, `CurrentJobID`, `PilotJobReference`, `PilotStamp`, `DestinationSite`, `Queue`, `GridSite`, `VO`, `GridType`, `BenchMark`, `SubmissionTime`, `LastUpdateTime`, `Status`, `StatusReason`, `AccountingSent`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING `PilotAgents`.`PilotID`]
[parameters: (0, 0, 'aa', '', 'NotAssigned', 'Unknown', 'Unknown', 'diracAdmin', 'DIRAC', 0.0, datetime.datetime(2025, 4, 8, 8, 27, 35, 874664, tzinfo=datetime.timezone.utc), datetime.datetime(2025, 4, 8, 8, 27, 35, 874664, tzinfo=datetime.timezone.utc), 'Submitted', 'Unknown', 'False')]

insert(PilotAgents).values(values).returning(PilotAgents.pilot_id) is not supported in mysql, but the CI passes.