common : refactor arg parser

[ngxson]

TL;DR for breaking change

This PR has only some small breaking changes to the environment variable system introduced in #9105 :

• To disable continuous batching, add LLAMA_ARG_NO_CONT_BATCHING=1 (instead of LLAMA_ARG_CONT_BATCHING=0)
• To disable slots endpoint, add LLAMA_ARG_NO_ENDPOINT_SLOTS=1 (instead of LLAMA_ARG_ENDPOINT_SLOTS=0)
• If both command line argument and environment variable are both set for the same param, the argument will take precedence over env var.
  In such case, you will also see a warning, for example: warn: LLAMA_ARG_CTX_SIZE environment variable is set, but will be overwritten by command line argument -c

In this PR

The goals of this PR are:

• Refactor & make the arg parser code more intuitive, tailored to llama.cpp's usage
• Ability to auto-generate documentation (markdown content) from code
• Better support for multiple examples (plus, one arg used by multiple examples or different purposes)
• Unifying env variable & arguments logic into one place

To generate markdown, run: make llama-export-docs
Output file will be in the format of autogen-{EXAMPLE_NAME}.md

TODO:

• migrate all args to this new format
• migrate env variables
• add some ctests
• add binary target to export markdown
• update exiting markdown docs

---

List of removed args (this is not breaking change, since no where in the code base handles these args):

--priority
--priority-batch
--priority-draft
-Cbd
--priority-batch-draft

---

• I have read the contributing guidelines
• Self-reported review complexity:
  • Medium

[ngxson]

@ggerganov Before proceeding further, I would like to ask for you opinion about this subject. Do you think this is a good way to have code-as-documentations? And if so, do you have any idea to add to this? Thank you.

[ggerganov]

Seems ok to me 👍

[ngxson]

@ggerganov Thank you for the initial review. This PR is now ready.

Here is a quick recap on what I've done:

• All CLI args & env var are migrated to the new format
• tests/test-arg-parser is added to test this new system
• llama-export-docs target is added, which can export list of arguments to markdown table (instead of markdown list in my initial demo). Here is an example for server docs

[ggerganov]

Functionality-wise, this is great. However the build time of libcommon increases on my machine from ~3s to ~12s:

ccache -C && touch ../common/common.cpp && time make -j common

Should we try to reduce it in some way? I suppose the culprit is in the lambda handlers in gpt_params_parser_init.

[ngxson]

Quick question: because sparams is already included inside params, should we get rid of passing sparams as argument of handler function? (and replace it with params.sparams

[ngxson]

Alright, I change some std::string to const char * and it brought down the build time a little bit. Keep in mind that the 5.2s result above was with only one arg repeated 190 times.

The build time of the latest commit e625f5f :

$ make clean && time make -j common/common.o
make -j common/common.o  5.70s user 0.16s system 99% cpu 5.889 total

# versus master
# make -j common/common.o  5.48s user 0.16s system 99% cpu 5.674 total

So that's 0.2s slower compared to master. It can be further reduced if std::string help can be changed to const char *, but currently it's a quite tricky to have format(...) function to return const char *, so I think we can consider it later.

[ggerganov]

Should move the method implementations in the .cpp, to avoid building the same code in all examples.

Can also move all llama_arg related stuff in common/arg.h,.cpp. Can be in a follow-up PR

[ngxson]

Currently doing this will break gen-docs, since it reads the data directly from class member.

But yes I will do a follow up to split it into common/arg.h,.cpp