Python: Fix enabling of UTF-8 in some situations.

There was a couple of reports of Python applications failing due to the
following type of error

File "/opt/netbox/netbox/netbox/configuration.py", line 25, in _import
     print(f"\U0001f9ec loaded config '{path}'")
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f9ec' in
position 0: ordinal not in range(128)

due to the use of Unicode text in the print() statement.

This only happened for python 3.8+ when using the "home" configuration
option as this meant we were going through the new PyConfig
configuration.

When using this new configuration method with the 'isolated' specific
API (for embedded Python) UTF-8 is disabled by default,
PyPreConfig->utf8_mode = 0.

To fix this we need to setup the Python pre config and enable utf-8
mode. However rather than enable utf-8 unconditionally we can set to it
to -1 so that it will use the LC_CTYPE environment variable to determine
whether to enable utf-8 mode or not. utf-8 mode will be enabled if
LC_CTYPE is either: C, POSIX or some specific UTF-8 locale. This is the
default utf8_mode setting when using the non-isolated PyPreConfig API.

Reported-by: Tobias Genannt <tobias.genannt@kappa-velorum.net>
Tested-by: Tobias Genannt <tobias.genannt@kappa-velorum.net>
Link: <https://peps.python.org/pep-0587/>
Link: <https://docs.python.org/3/c-api/init_config.html#c.PyPreConfig.utf8_mode>
Fixes: 491d0f70 ("Python: Added support for Python 3.11.")
Closes: <https://github.com/nginx/unit/issues/817>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>

1 files changed, 14 insertions, 0 deletions

diff --git a/src/python/nxt_python.c b/src/python/nxt_python.c
index 83f2544e..7c059649 100644
--- a/src/python/nxt_python.c
+++ b/src/python/nxt_python.c
@@ -78,9 +78,23 @@ nxt_python3_init_config(nxt_int_t pep405)
     PyConfig     config;
     PyStatus     status;
     nxt_int_t    ret;
+    PyPreConfig  preconfig;
 
     ret = NXT_ERROR;
 
+    PyPreConfig_InitIsolatedConfig(&preconfig);
+    /*
+     * Determine whether to use UTF-8 mode or not, UTF-8
+     * will be enabled if LC_CTYPE is C, POSIX or some
+     * specific UTF-8 locale.
+     */
+    preconfig.utf8_mode = -1;
+
+    status = Py_PreInitialize(&preconfig);
+    if (PyStatus_Exception(status)) {
+        return ret;
+    }
+
     PyConfig_InitIsolatedConfig(&config);
 
     if (pep405) {