Repository navigation
Conversation
|
I've been wanting a way to build utf8proc as a constexpr lib as well, but this PR is utterly massive and I don't like this codegen approach to it. I think it should be possible to just define a valueless macro and include the .c file in a C++ source file to enable constexpr support. No need for C++ namespaces either. |
|
What is the motivation for making a constexpr version of utf8proc? Do you have benchmarks of a real application where it is beneficial? I would think that most realistic applications of utf8proc would apply its functions mainly to strings only available at runtime, and hence not benefiting much from compile-time evaluation. |
|
It's not about performance, it's about being able to normalize embedded strings during compile time. Alternatives are preprocessing (e.g. host build tool step, git hooks/filters) or normalizing during runtime (e.g. during global variable initialization or first use). Both have undesirable drawbacks (build system/source control complications, unnecessary runtime performance loss, etc). For strings known at build time anyway it's simpler to just make the compiler double as the preprocessor for them, using the same code as what runtime would use. EDIT: It's also useful for running tests at compile time, which simplifies testing in many cases. |
Immediately afterwards, however, you identify the main drawback of normalizing at runtime as "unnecessary runtime performance loss". In other words, it is about performance? Can you sketch a realistic application where static normalization is important? (I would think that most real applications working with normalization-sensitive Unicode strings would want to load them at runtime anyway for i18n/l10n purposes.) |
|
My "not about performance" comment meant that Plenty of apps embed text into the final binary for standalone purposes, e.g. with resources on Windows, or If normalization was possible at compile time, the "naive" approach of just declaring static variables could be used to store the already-normalized strings in the built binary, with the type system affirming that runtime normalization isn't needed. No complex runtime machinery is needed in that case, at worst the strings would just be copied as-is. |
As I understand it, these are separate files attached to the application, similar to files in app bundles on macos, not part of the C++ source code.
Can you give an example, perhaps linking an existing project on github, of a practical example of an application with a large number of embedded non-ASCII strings in its source code that need to be normalized, and for which this takes a noticeable portion of startup time? |
The link you provided is for UWP's external resource files, which isn't what I was referring to. Win32 resources are documented separately: As explained in the first link, resources are compiled by the resource compiler and then passed to the linker with the other object files, so they by design must be available at build time. The vast majority of Win32 apps use resource files to at least some extent, since it's how icons, version, and copyright information is embedded. Using them for other text strings or embedded files is common mainly only in apps that only target Win32 though, since it's obviously not portable to other platforms. There are many examples of people searching for similar functionality for other platforms though, many libraries and scripts exist specifically to help with embedding files in C or C++ sources (e.g. corrade-rc), and the culmination of those efforts was
There is a chicken and egg problem here. Most existing projects don't even use If we remove or broaden the "source code" requirement you stipulated, the number of matching projects explodes. There are countless apps and games that don't offer support for adding additional languages post-install but which do support non-ASCII languages, for example because they are installed in read-only linux distributions or read-only game console filesystems. Therefore, I don't understand the purpose behind the "source code" requirement. As far as I am concerned, anything which is available at build time and not expected to change at runtime can technically be considered to be part of the source code. Whether it's actually passed through a C or C++ compiler at some point before runtime is just down to tooling ergonomics. I was intending to contribute |
Because non-source files can easily be processed separately, and don't require compiler support. e.g. if you have XML files you can pass them through a normalization pass as a build step. |
|
Yes, I mentioned that during my second comment in this thread, that's what I consider a "build system complication". That sort of processing during build typically requires use of a host tool, and if the files have a complicated schema where some parts should not be normalized or need different normalizations, you usually have to make a custom host tool for that. If you're cross compiling, using a custom host tool becomes more tedious since it can't be built in the same pass with the rest of the code anymore, unless your build system supports it and you have a host compiler toolchain at the ready too. It's a whole lot of extra complexity for something that could be done in one pass through the compiler. |
What?
This pull request contains a generator (python3 script) to parse utf8prop.c and convert it to a constexpr C++ header-only library.
The resulting library is placed in the
utf8procnamespace, andutf8proc_prefixes are removed.Private entities are in the namespace
utf8proc::detail.Why?
In C++, constexpr allows programmers to evaluate functions at compile-time (before the linker).
This is a big part of C++ meta-programming.
I could not find a constexpr library for Unicode, so here is my solution.
I used
utf8procas a basis because I do not want to implement these algorithms myself, and other libraries such asICUare way too big for me to dive into and produce a generator in a fairly short period of time.Modifications on existing code?
When defined, it removes the functions that are calling malloc/realloc.
It was needed because such functions cannot be marked
constexpr.static const char s[][3]->static const char * const s[]This does not modify the syntax anywhere, but it changes the string storage from
static storage to literal storage (needed in constexpr because static storage is not allowed).
All the tests of the C library passes correctly after these modifications.
License
The generation script is under the Boost Software License 1.0.
This is a more permissive license than the "expat" license:
The only requirement is to let the copyright notice inside the
gen_hpp.pyscript.Nowhere else.