| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
This was inadvertently removed in 76086d6d ("Wasm: Allow to set the HTTP
response status.")
Fixes: 76086d6d ("Wasm: Allow to set the HTTP response status.")
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
|
|
On Github, @rlandgrebe reported an issue when trying to rewrite URLs
that contained query strings.
With the PHP language module we were in fact segfaulting (SIGSEGV) in
libphp
[93960.462952] unitd[20940]: segfault at 7f307cef6476 ip 00007f2f81a94577 sp 00007fff28a777d0 error 4 in libphp-8.2.so[7f2f818df000+2fd000] likely on CPU 0 (core 0, socket 0)
#0 0x00007f2abd494577 in php_default_treat_data (arg=1, str=0x0,
destArray=<optimized out>)
at /usr/src/debug/php-8.2.10-1.fc38.x86_64/main/php_variables.c:488
488 if (c_var && *c_var) {
(gdb) p c_var
$1 = 0x7f2bb8880676 <error: Cannot access memory at address 0x7f2bb8880676>
This was when trying to get the query string which somehow is pointing
off into the woods.
This gdb debug session when doing rewrite basically shows the core of
the issue
(gdb) x /64bs req->fields
...
0x7f7eaaaa8090: "GET"
0x7f7eaaaa8094: "HTTP/1.1"
0x7f7eaaaa809d: "::1"
0x7f7eaaaa80a1: "::1"
0x7f7eaaaa80a5: "8080"
0x7f7eaaaa80aa: "localhost"
0x7f7eaaaa80b4: "/test?q=a"
0x7f7eaaaa80be: "/test"
...
(gdb) p target_pos
$4 = (void *) 0x7f7eaaaa80b4
(gdb) p query_pos
$6 = (void *) 0x7f7eaaaa6af6
(gdb) p r->args->start
$8 = (u_char *) 0x7f7ea4002b02 "q=a HTTP/1.1\r\nHost: localhost:8080\r\nUser-Agent: curl/8.0.1\r\nAccept: */*\r\n\r\n"
(gdb) p r->target.start
$9 = (u_char *) 0x7f7ea40040c0 "/test?q=a"
That last address, 0x7f7ea40040c0, looks out of wack, it should be
smaller than r->args->start.
That results in a calculation in nxt_router_prepare_msg()
if (r->args->start != NULL) {
query_pos = nxt_pointer_to(target_pos,
r->args->start - r->target.start);
nxt_unit_sptr_set(&req->query, query_pos);
} else {
that goes negative that then is stored in req->query.offset which is a
uint32_t and so wraps backwards from UINT_MAX to give us an offset of a
little under 4GiB, hence the above invalid memory access.
All this happens due to in nxt_http_rewrite() if we have a URL with a
query string, we create a new memory allocation to store the transformed
URL and query string.
We set r->target to point to this new allocation, but we also need to
point r->args->start to the start of the query string in this new
allocation.
Reported-by: René Landgrebe <https://github.com/rlandgrebe>
Tested-by: René Landgrebe <https://github.com/rlandgrebe>
Tested-by: Liam Crilly <liam.crilly@nginx.com>
Fixes: 14d6d97b ("HTTP: added basic URI rewrite.")
Closes: <https://github.com/nginx/unit/issues/964>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
The default configuration previously contained just a listeners and
applications object. Since routes is now a principle configuration object,
and a recommended way of configurating Unit, it is now included in the
default configuration.
This change benefits new users because it explicitly introduces the three
principle configuration objects which leads more intuitively to the
documentation. Experienced users may choose to ignore or delete routes.
routes is defined as an array instead of an object because this change
is designed to assist new users, where the simpler form of routes is
easier to understand.
|
|
We need to take into account the size of the nxt_unit_response_t
structure itself when calculating where to start appending data to in
memory.
Closes: <https://github.com/nginx/unit/issues/923>
Reported-by: Alejandro Colomar <alx@kernel.org>
Reviewed-by: Andrew Clayton <a.clayton@nginx.org>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This closes #871 issue on GitHub.
|
|
Starting from Node.js 15.0.0 the chunk parameter of the response.write()
can be a Uint8Array.
This closes #870 issue on GitHub.
|
|
Currently Wasm modules are limited to a 32bit address space (until at
least the memory64 work is completed). All the counters etc in the
request structure were u32's. Which matched with 32bit memory
limitation.
However there is really no need to not allow >4GiB uploads that can be
saved off to disk or some such.
To do this we just need to increase the ->content_len &
->total_content_sent members to u64's.
However because we need the request structure to have the exact same
layout on 32bit (for Wasm modules) as it does on 64bit we need to re-jig
the order of some of these members and add a four-byte padding member.
Thus the request structure now looks like on 64bit (as shown by
pahole(1))
struct nxt_wasm_request_s {
uint32_t method_off; /* 0 4 */
uint32_t method_len; /* 4 4 */
uint32_t version_off; /* 8 4 */
uint32_t version_len; /* 12 4 */
uint32_t path_off; /* 16 4 */
uint32_t path_len; /* 20 4 */
uint32_t query_off; /* 24 4 */
uint32_t query_len; /* 28 4 */
uint32_t remote_off; /* 32 4 */
uint32_t remote_len; /* 36 4 */
uint32_t local_addr_off; /* 40 4 */
uint32_t local_addr_len; /* 44 4 */
uint32_t local_port_off; /* 48 4 */
uint32_t local_port_len; /* 52 4 */
uint32_t server_name_off; /* 56 4 */
uint32_t server_name_len; /* 60 4 */
/* --- cacheline 1 boundary (64 bytes) --- */
uint64_t content_len; /* 64 8 */
uint64_t total_content_sent; /* 72 8 */
uint32_t content_sent; /* 80 4 */
uint32_t content_off; /* 84 4 */
uint32_t request_size; /* 88 4 */
uint32_t nfields; /* 92 4 */
uint32_t tls; /* 96 4 */
char __pad[4]; /* 100 4 */
nxt_wasm_http_field_t fields[]; /* 104 0 */
/* size: 104, cachelines: 2, members: 25 */
/* last cacheline: 40 bytes */
};
and the same structure (taken from unit-wasm) compiled as 32bit
struct luw_req {
u32 method_off; /* 0 4 */
u32 method_len; /* 4 4 */
u32 version_off; /* 8 4 */
u32 version_len; /* 12 4 */
u32 path_off; /* 16 4 */
u32 path_len; /* 20 4 */
u32 query_off; /* 24 4 */
u32 query_len; /* 28 4 */
u32 remote_off; /* 32 4 */
u32 remote_len; /* 36 4 */
u32 local_addr_off; /* 40 4 */
u32 local_addr_len; /* 44 4 */
u32 local_port_off; /* 48 4 */
u32 local_port_len; /* 52 4 */
u32 server_name_off; /* 56 4 */
u32 server_name_len; /* 60 4 */
/* --- cacheline 1 boundary (64 bytes) --- */
u64 content_len; /* 64 8 */
u64 total_content_sent; /* 72 8 */
u32 content_sent; /* 80 4 */
u32 content_off; /* 84 4 */
u32 request_size; /* 88 4 */
u32 nr_fields; /* 92 4 */
u32 tls; /* 96 4 */
char __pad[4]; /* 100 4 */
struct luw_hdr_field fields[]; /* 104 0 */
/* size: 104, cachelines: 2, members: 25 */
/* last cacheline: 40 bytes */
};
We can see the structures have the same layout, same size and no
padding.
We need the __pad member as otherwise I saw gcc and clang on Alpine
Linux automatically add the 'packed' attribute to the structure which
made the two structures not match.
Link: <https://github.com/WebAssembly/memory64>
Link: <https://github.com/nginx/unit-wasm>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
When trying to upload files to the luw-upload-reflector demo[0] above a
certain size that would mean Unit would need to make more than two calls
to the request_handler function in the Wasm module we would get the
following error from wasmtime and the upload would stall on the third
call to the request_handler
WASMTIME ERROR: failed to call function [->wasm_request_handler]
error while executing at wasm backtrace:
0: 0x5ce2 - <unknown>!memcpy
1: 0x7df - luw_req_buf_append
at /home/andrew/src/unit-wasm/src/c/libunit-wasm.c:308:14
2: 0x3a1 - luw_request_handler
at /home/andrew/src/unit-wasm/examples/c/luw-upload-reflector.c:110:3
Caused by:
wasm trap: out of bounds memory access
This was due to ->content_off (the offset of where the actual body
content starts in the request structure/memory) being some overly large
value.
This was largely down to me being an idiot!
Before calling the loop that makes the calls to the request_handler we
would calculate the new offset, which is now just the size of the
request structure as we don't re-send all the HTTP meta data and headers
etc. However because this value is in the request structure which is in
the shared memory and we use this same memory for requests and
responses, when we make a response we overwrite this request structure
with the response structure, so our ->content_off is now some wacked out
value when we make the next call to the request_handler.
To fix this we just need to reset ->content_off each time round the
loop.
There's also no point in setting ->nfields to 0, it has the same issue
as above, but doesn't get re-used by the Wasm module anyway.
[0]: <https://github.com/nginx/unit-wasm/blob/main/examples/c/luw-upload-reflector.c>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This commit enables WebAssembly modules to set the HTTP response status
to something other than the previously hard coded '200 OK'.
To do this they can make a call to nxt_wasm_set_resp_status() providing
the required status code.
If this function isn't called the status code defaults to '200 OK'. The
WebAssembly module can also return -1 from the request_handler function
as a short cut to signal a '500 Internal Server Error'.
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
As reported by @andypost on GitHub, if you try to build Unit on a system
that uses musl libc (such as Alpine Linux) with clang then you get the
following
clang -c -pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -fstrict-aliasing -Wstrict-overflow=5 -Wmissing-prototypes -Werror -g -I src -I build/include \
\
\
-o build/src/nxt_socketpair.o \
-MMD -MF build/src/nxt_socketpair.dep -MT build/src/nxt_socketpair.o \
src/nxt_socketpair.c
In file included from src/nxt_socketpair.c:8:
src/nxt_socket_msg.h:138:17: error: comparison of integers of different signs: 'unsigned long' and 'long' [-Werror,-Wsign-compare]
cmsg = CMSG_NXTHDR(&msg, cmsg))
^~~~~~~~~~~~~~~~~~~~~~~
/usr/include/sys/socket.h:358:44: note: expanded from macro 'CMSG_NXTHDR'
__CMSG_LEN(cmsg) + sizeof(struct cmsghdr) >= __MHDR_END(mhdr) - (unsigned char *)(cmsg) \
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from src/nxt_socketpair.c:8:
src/nxt_socket_msg.h:177:17: error: comparison of integers of different signs: 'unsigned long' and 'long' [-Werror,-Wsign-compare]
cmsg = CMSG_NXTHDR(&msg, cmsg))
^~~~~~~~~~~~~~~~~~~~~~~
/usr/include/sys/socket.h:358:44: note: expanded from macro 'CMSG_NXTHDR'
__CMSG_LEN(cmsg) + sizeof(struct cmsghdr) >= __MHDR_END(mhdr) - (unsigned char *)(cmsg) \
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.
make: *** [build/Makefile:261: build/src/nxt_socketpair.o] Error 1
GCC works fine, it seems to have some smarts so that it doesn't give
warnings on system header files.
This seems to be a long standing issue with musl libc (bad casting in
the CMSG_NXTHDR macro) and the workaround employed by several projects
is to disable the -Wsign-compare clang warning for the code in question.
So, that's what we do. We wrap the CMSG_NXTHDR macro in a function, so
we can use the pre-processor in it to selectively disable the warning.
Link: <https://github.com/dotnet/runtime/issues/16438>
Link: <https://git.openembedded.org/meta-openembedded/tree/meta-oe/recipes-devtools/breakpad/breakpad/0001-Turn-off-sign-compare-for-musl-libc.patch>
Link: <https://github.com/dotnet/corefx/blob/57ff88bb75a0/src/Native/Unix/System.Native/pal_networking.c#L811-L829>
Link: <https://patchwork.yoctoproject.org/project/oe/patch/20220407191438.3696227-1-stefan@datenfreihafen.org/>
Closes: <https://github.com/nginx/unit/issues/936>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
Scripted change:
$ grep -ril recevied src/ | xargs sed -i s/recevied/received/
Reported-by: <https://github.com/jeffdafoe>
Closes: <https://github.com/nginx/unit/issues/920>
Cc: <https://github.com/meezaan>
Cc: Timo Stark <t.stark@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
|
|
Due to the sandboxed nature of WebAssembly, by default WASM modules
don't have any access to the underlying filesystem.
There is however a capabilities based mechanism[0] for allowing such
access.
This adds a config option to the 'wasm' application type;
'access.filesystem' which takes an array of directory paths that are
then made available to the WASM module. This access works recursively,
i.e everything under a specific path is allowed access to.
Example config might look like
"access" {
"filesystem": [
"/tmp",
"/var/tmp"
]
}
The actual mechanism used allows directories to be mapped differently in
the guest. But at the moment we don't support that and just map say /tmp
to /tmp. This can be revisited if it's something users clamour for.
Network sockets are another resource that may be controlled in this
manner, for example there is a wasi_config_preopen_socket() function,
however this requires the runtime to open the network socket then
effectively pass this through to the guest.
This is something that can be revisited in the future if users desire
it.
[0]:
<https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-capabilities.md>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This exposes various WebAssembly language module specific options.
The application type is "wasm".
There is a "module" option that is required, this specifies the full
path to the WebAssembly module to be run. This module should be in
binary format, i.e a .wasm file.
There are also currently eight function handlers that can be specified.
Three of them are _required_
1) request_handler
The main driving function. This may be called multiple times for a
single HTTP request if the request is larger than the shared memory.
2) malloc_handler
Used to allocate a chunk of memory at language module startup. This
memory is allocated from the WASM modules address space and is what is
sued for communicating between the WASM module (the guest) and Unit (the
host).
3) free_handler
Used to free the memory from above at language module shutdown.
Then there are the following five _optional_ handlers
1) module_init_handler
If set, called at language module startup.
2) module_end_handler
If set, called at language module shutdown.
3) request_init_handler
If set, called at the start of request. Called only once per HTTP
request.
4) request_end_handler
If set, called once all of a request has been sent to the WASM module.
5) response_end_handler
If set, called at the end of a request, once the WASM module has sent
all its headers and data.
Example config
"applications": {
"luw-echo-request": {
"type": "wasm",
"module": "/path/to/unit-wasm/examples/c/luw-echo-request.wasm",
"request_handler": "luw_request_handler",
"malloc_handler": "luw_malloc_handler",
"free_handler": "luw_free_handler",
"module_init_handler": "luw_module_init_handler",
"module_end_handler": "luw_module_end_handler",
}
}
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This adds the core of runtime WebAssembly[0] support. Future commits
will enable this in the Unit core and expose the configuration.
This introduces a new src/wasm directory for storing this source.
We are initially using Wasmtime[0] as the WebAssembly runtime, however
this has been designed with the ability to use different runtimes in
mind.
src/wasm/nxt_wasm.[ch] is the main interface to Unit.
src/wasm/nxt_rt_wasmtime.c is the Wasmtime runtime support. This is
nicely insulated from any knowledge of internal Unit workings.
Wasmtime is what loads and runs the Wasm modules.
The Wasm modules can export functions Wasmtime can call and Wasmtime can
export functions that the module can call.
We make use of both. The terminology used is that function exports are
what the Wasm module exports and function imports are what the Wasm
runtime exports to the module.
We currently have four function imports (functions exported by the
runtime to be called by the Wasm module).
1) nxt_wasm_get_init_mem_size
This allows Wasm modules to get the size of the initially allocated
shared memory. This is the size allocated at Unit startup and what the
Wasm modules can assume they have access to (in reality this shared
memory will likely be larger).
The amount of memory allocated at startup is NXT_WASM_MEM_SIZE which as
of this commit is 32MiB.
We do actually allocate NXT_WASM_MEM_SIZE + NXT_WASM_PAGE_SIZE at
startup which is an extra 64KiB (the smallest allocation unit), this is
to allow room for the response structure and so module developers can
just assume they have the full 32MiB for their actual response.
2) nxt_wasm_send_headers
This allows WASM modules to send their headers.
3) nxt_wasm_send_response
This allows WASM modules to send their response.
4) nxt_wasm_response_end
This allows WASM modules to inform Unit they have finished sending their
response. This calls nxt_unit_request_done()
Then there are currently up to eight functions that a module can export.
Three of which are required. These function can be named anything. I'll
use the Unit configuration names to refer to them
1) request_handler
The main driving function. This may be called multiple times for a
single HTTP request if the request is larger than the shared memory.
2) malloc_handler
Used to allocate a chunk of memory at language module startup. This
memory is allocated from the WASM modules address space and is what is
sued for communicating between the WASM module (the guest) and Unit (the
host).
3) free_handler
Used to free the memory from above at language module shutdown.
Then there are the following optional handlers
1) module_init_handler
If set, called at language module startup.
2) module_end_handler
If set, called at language module shutdown.
3) request_init_handler
If set, called at the start of request. Called only once per HTTP
request.
4) request_end_handler
If set, called once all of a request has been sent to the WASM module.
5) response_end_handler
If set, called at the end of a request, once the WASM module has sent
all its headers and data.
32bits
We currently support 32bit WASM modules, I.e wasm32-wasi. Newer version
of clang, 13+[2], do seem to have support for wasm64 as a target (which
uses a LP64 model). However it's not entirely clear if the WASI SDK
fully supports[3] this and by extension WASI libc/wasi-sysroot.
64bit support is something than can be explored more thoroughly in the
future.
As such in structures that are used to communicate between the host and
guest we use 32bit ints. Even when a single byte might be enough. This
is to avoid issues with structure layout differences between a 64bit
host and 32bit guest (I.e WASM module) and the need for various bits of
structure padding depending on host architecture. Instead everything is
4-byte aligned.
[0]: <https://webassembly.org/>
[1]: <https://wasmtime.dev/>
[2]: <https://reviews.llvm.org/rG670944fb20b226fc22fa993ab521125f9adbd30a>
[3]: <https://github.com/WebAssembly/wasi-sdk/issues/185>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This is required to actually _build_ the Wasm language module.
The nxt_wasm_app_conf_t structure consists of the modules name, e.g
wasm, then the three required function handlers followed by the five
optional function handlers.
See the next commit for details of these function handlers.
We also need to include the u.wasm union entry that provides access to
the above structure.
The bulk of the configuration infrastructure will be added in a
subsequent commit.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This is the first patch in adding WebAssembly language module support.
This just adds a new NXT_APP_WASM type, required by subsequent commits.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This makes it much more clear what's what.
This is in preparation for adding WebAssembly language module support.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
|
|
No functional changes.
|
|
|
|
This commit adds the variable $response_header_NAME.
|
|
This commit is to reimplement the variables with an unknown field
such as $header_{name} to make the parsing more generic,
it's a preparation for supporting response header variables.
|
|
|
|
Signed-off-by: Alejandro Colomar <alx@nginx.com>
|
|
When a variable is accessed in the Unit configuration, the value is cached.
This was useful prior to the URI rewrite feature, but now that the URI (more
precisely, the request target) can be rewritten, the contents of the variable
$uri (which contains the path part of the request target, and is decoded)
should not be cached anymore, or at least the cached value should be invalidated
after a URI rewrite.
Example:
{
"rewrite": "/prefix$uri",
"share": "$uri"
}
For a request line like GET /foo?bar=baz HTTP/1.1\r\n, the expected file
served in the response would be /prefix/foo, but due to the caching issue,
Unit currently serves /foo.
|
|
Signed-off-by: synodriver <diguohuangjiajinweijun@gmail.com>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
[ Re-word commit subject - Andrew ]
Fixes: c4c2f90c5b53 ("Python: ASGI server introduced.")
Closes: <https://github.com/nginx/unit/issues/895>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
Lifespan state is a special dict in asgi lifespan scope, which allow
applications to persist data from the lifespan cycle to request/response
handling. The scope["state"] namespace provides a place to store these
sorts of things. The server will ensure that a shallow copy of the
namespace is passed into each subsequent request/response call into the
application.
Some frameworks are already taking advantage of this feature, for
example, starlette, and without this feature they wouldn't work
properly.
Signed-off-by: synodriver <diguohuangjiajinweijun@gmail.com>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
[ Minor code tweaks to avoid lines > 80 chars, static a function and
re-work the PyMemberDef structure initialisation for Python <3.7
and -Wwrite-strings compatibility - Andrew ]
Tested-by: <https://github.com/synodriver>
Tested-by: <https://github.com/hawiliali>
Closes: <https://github.com/nginx/unit/issues/864>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
If we don't update the pointer before copying the request body, then we
get the behavior shown below. After this patch, "foo\n" is rightly
appended at the end of the response body.
Request:
"GET / HTTP/1.1\r\nHost: _\nContent-Length: 4\n\nfoo\n"
Response body:
"""
Hello world!
foo
est data:
Method: GET
Protocol: HTTP/1.1
Remote addr: 127.0.0.1
Local addr: 127.0.0.1
Target: /
Path: /
Fields:
Host: _
Content-Length: 4
Body:
"""
Fixes: 1bb22d1e922c ("Unit application library.")
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
|
|
We renamed the options recently, with the intention of keeping the old
names as supported but deprecated for some time, before removal. This
was done with the configure script options, but in the unitd binary, we
accidentally removed the old names, causing some unintended breakage.
Keep support for the old names, albeit with a deprecation message to
stderr, for some time, until we decide to remove them.
Fixes: 5a37171f733f ("Added default values for pathnames.")
Closes: <https://github.com/nginx/unit/issues/876>
Reported-by: El RIDO <elrido@gmx.net>
Acked-by: Liam Crilly <liam@nginx.com>
Acked-by: Artem Konev <a.konev@f5.com>
Acked-by: Timo Stark <t.stark@nginx.com>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Cc: Andrei Zeliankou <zelenkov@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
|
|
There are a couple of reports on GitHub about issues accessing Python
ASGI based applications over IPv6.
A request over IPv6 would result in an error like
2023/05/13 17:49:12 [alert] 47202#47202 [unit] #10: Python failed to create 'client' pair
2023/05/13 17:49:12 [alert] 47202#47202 [unit] Python failed to call 'loop.call_soon'
ValueError: invalid literal for int() with base 10: 'db8:1:1:1ee7:dead:beef:cafe'
The above error was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib64/python3.11/asyncio/base_events.py", line 765, in call_soon
handle = self._call_soon(callback, args, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/asyncio/base_events.py", line 781, in _call_soon
handle = events.Handle(callback, args, self, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SystemError: <class 'asyncio.events.Handle'> returned a result with an exception set
This issue occurred in the nxt_py_asgi_create_ip_address() function
where it tries to create an IP address / port number pair.
It does this by looking for the first ':' in the address and taking
everything after it as the port number. Like in the above error message,
if we tried to access the server @ 2001:db8:1:1:1ee7:dead:beef:cafe,
then we'd end up with the port number as 'db8:1:1:1ee7:dead:beef:cafe'.
There are two issues with this
1) The IP address and port number are already flowed through
separately.
2) Even if (1) wasn't true, it would still be broken for IPv6 as we'd
expect to a get an address literal like
[2001:db8:1:1:1ee7:dead:beef:cafe]:8080, however there was no code to
handle the []'s.
The fix is to simply not try looking for a port number. We pass a port
number into this function to use in the case where we don't find a port
number, we never will...
A further cleanup would be to flow through the server port number when
creating the 'server pair' PyTuple, rather than just using the hard
coded 80.
Closes: <https://github.com/nginx/unit/issues/793>
Closes: <https://github.com/nginx/unit/issues/874>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
|
|
This commit introduced the basic URI rewrite. It allows users to change request URI. Note the "rewrite" option ignores the contained query if any and the query from the request is preserverd.
An example:
"routes": [
{
"match": {
"uri": "/v1/test"
},
"action": {
"return": 200
}
},
{
"action": {
"rewrite": "/v1$uri",
"pass": "routes"
}
}
]
Reviewed-by: Alejandro Colomar <alx@nginx.com>
|
|
Normally Unit responds to HTTP requests by including a header like
Server: Unit/1.30.0
however it can sometimes be beneficial to withhold the version
information and in this case just respond with
Server: Unit
This patch adds a new "settings.http" boolean option called
server_version, which defaults to true, in which case the full version
information is sent. However this can be set to false, e.g
"settings": {
"http": {
"server_version": false
}
},
in which case Unit responds without the version information as the
latter example above shows.
Link: <https://www.ietf.org/rfc/rfc9110.html#section-10.2.4>
Closes: <https://github.com/nginx/unit/issues/158>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
Split out the "Unit" name from the NXT_SERVER #define into its own
NXT_NAME #define, then make NXT_SERVER a combination of that and
NXT_VERSION.
This is required for a subsequent commit where we may want the server
name on its own.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
In nxt_kqueue_poll() error is declared as a nxt_bool_t aka unsigned int
(on x86-64 anyway).
It is used both as a boolean and as the return storage for a bitwise AND
operation.
This has potential to go awry.
If nxt_bool_t was changed to be a u8 then we would have the following
issue
gcc12 -c -pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-prototypes -Werror -g -O2 -I src -I build -I/usr/local/include -o build/src/nxt_kqueue_engine.o -MMD -MF build/src/nxt_kqueue_engine.dep -MT build/src/nxt_kqueue_engine.o src/nxt_kqueue_engine.c
src/nxt_kqueue_engine.c: In function 'nxt_kqueue_poll':
src/nxt_kqueue_engine.c:728:17: error: overflow in conversion from 'int' to 'nxt_bool_t' {aka 'unsigned char'} changes value from '(int)kev->flags & 16384' to '0' [-Werror=overflow]
728 | error = (kev->flags & EV_ERROR);
| ^
cc1: all warnings being treated as errors
EV_ERROR has the value 16384, after the AND operation error holds 16384,
however this overflows and wraps around (64 times) exactly to 0.
With nxt_bool_t defined as a u32, we would have a similar issue if
EV_ERROR ever became UINT_MAX + 1 (or a multiple thereof)...
Rather than conflating the use of error, keep error as a boolean (it is
used further down the function) but do the AND operation inside the
if ().
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This removes a bunch of unused files that would have been touched by
subsequent commits that switch to using nxt_bool_t (AKA unit6_t) in
structures.
In auto/sources we have
NXT_LIB_SRC0=" \
src/nxt_buf_filter.c \
src/nxt_job_file.c \
src/nxt_stream_module.c \
src/nxt_stream_source.c \
src/nxt_upstream_source.c \
src/nxt_http_source.c \
src/nxt_fastcgi_source.c \
src/nxt_fastcgi_record_parse.c \
\
src/nxt_mem_pool_cleanup.h \
src/nxt_mem_pool_cleanup.c \
"
None of these seem to actually be used anywhere (other than within
themselves). That variable is _not_ referenced anywhere else.
Also remove the unused related header files: src/nxt_buf_filter.h,
src/nxt_fastcgi_source.h, src/nxt_http_source.h, src/nxt_job_file.h,
src/nxt_stream_source.h and src/nxt_upstream_source.h
Also, these files do not seem to be used, no mention under auto/ or build/
src/nxt_file_cache.c
src/nxt_cache.c
src/nxt_job_file_cache.c
src/nxt_cache.h is #included in src/nxt_main.h, but AFAICT is not
actually used.
With all the above removed
$ ./configure --openssl --debug --tests && make -j && make -j tests &&
make libnxt
all builds.
Buildbot passes.
NOTE: You may need to do a 'make clean' before the next build attempt.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
Don't reconstruct a new string for the $request_line from the parsed
method, target, and HTTP version, but rather keep a pointer to the
original memory where the request line was received.
This will be necessary for implementing URI rewrites, since we want to
log the original request line, and not one constructed from the
rewritten target.
This implementation changes behavior (only for invalid requests) in the
following way:
Previous behavior was to log as many tokens from the request line as
were parsed validly, thus:
Request -> access log ; error log
"GET / HTTP/1.1" -> "GET / HTTP/1.1" OK ; =
"GET / HTTP/1.1" -> "GET / HTTP/1.1" [1] ; =
"GET / HTTP/2.1" -> "GET / HTTP/2.1" OK ; =
"GET / HTTP/1." -> "GET / HTTP/1." [2] ; "GET / HTTP/1. [null]"
"GET / food" -> "GET / food" [2] ; "GET / food [null]"
"GET / / HTTP/1.1" -> "GET / / HTTP/1.1" [2] ; =
"GET / / HTTP/1.1" -> "GET / / HTTP/1.1" [2] ; =
"GET food HTTP/1.1" -> "GET" ; "GET [null] [null]"
"OPTIONS * HTTP/1.1" -> "OPTIONS" [3] ; "OPTIONS [null] [null]"
"FOOBAR baz HTTP/1.1"-> "FOOBAR" ; "FOOBAR [null] [null]"
"FOOBAR / HTTP/1.1" -> "FOOBAR / HTTP/1.1" ; =
"get / HTTP/1.1" -> "-" ; " [null] [null]"
"" -> "-" ; " [null] [null]"
This behavior was rather inconsistent. We have several options to go
forward with this patch:
- NGINX behavior.
Log the entire request line, up to '\r' | '\n', even if it was
invalid.
This is the most informative alternative. However, RFC-complying
requests will probably not send invalid requests.
This information would be interesting to users where debugging
requests constructed manually via netcat(1) or a similar tool, or
maybe for debugging a client, are important. It might be interesting
to support this in the future if our users are interested; for now,
since this approach requires looping over invalid requests twice,
that's an overhead that we better avoid.
- Previous Unit behavior
This is relatively fast (almost as fast as the next alternative, the
one we chose), but the implementation is ugly, in that we need to
perform the same operation in many places around the code.
If we want performance, probably the next alternative is better; if
we want to be informative, then the first one is better (maybe in
combination with the third one too).
- Chosen behavior
Only logging request lines when the request is valid. For any
invalid request, or even unsupported ones, the request line will be
logged as "-". Thus:
Request -> access log [4]
"GET / HTTP/1.1" -> "GET / HTTP/1.1" OK
"GET / HTTP/1.1" -> "GET / HTTP/1.1" [1]
"GET / HTTP/2.1" -> "-" [3]
"GET / HTTP/1." -> "-"
"GET / food" -> "-"
"GET / / HTTP/1.1" -> "GET / / HTTP/1.1" [2]
"GET / / HTTP/1.1" -> "GET / / HTTP/1.1" [2]
"GET food HTTP/1.1" -> "-"
"OPTIONS * HTTP/1.1" -> "-"
"FOOBAR baz HTTP/1.1"-> "-"
"FOOBAR / HTTP/1.1" -> "FOOBAR / HTTP/1.1"
"get / HTTP/1.1" -> "-"
"" -> "-"
This is less informative than previous behavior, but considering how
inconsistent it was, and that RFC-complying agents will probably not
send us such requests, we're ready to lose that information in the
log. This is of course the fastest and simplest implementation we
can get.
We've chosen to implement this alternative in this patch. Since we
modified the behavior, this patch also changes the affected tests.
[1]: Multiple successive spaces as a token delimiter is allowed by the
RFC, but it is discouraged, and considered a security risk. It is
currently supported by Unit, but we will probably drop support for
it in the future.
[2]: Unit currently supports spaces in the request-target. This is
a violation of the relevant RFC (linked below), and will be fixed
in the future, and consider those targets as invalid, returning
a 400 (Bad Request), and thus the log lines with the previous
inconsistent behavior would be changed.
[3]: Not yet supported.
[4]: In the error log, regarding the "log_routes" conditional logging
of the request line, we only need to log the request line if it
was valid. It doesn't make sense to log "" or "-" in case that
the request was invalid, since this is only useful for
understanding decisions of the router. In this case, the access
log is more appropriate, which shows that the request was invalid,
and a 400 was returned. When the request line is valid, it is
printed in the error log exactly as in the access log.
Link: <https://datatracker.ietf.org/doc/html/rfc9112#section-3>
Suggested-by: Liam Crilly <liam@nginx.com>
Reviewed-by: Zhidao Hong <z.hong@f5.com>
Cc: Timo Stark <t.stark@nginx.com>
Cc: Andrei Zeliankou <zelenkov@nginx.com>
Cc: Andrew Clayton <a.clayton@nginx.com>
Cc: Artem Konev <a.konev@f5.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
|
|
Currently when running in the foreground, unit application processes
will send stdout to the current TTY and stderr to the unit log file.
That behaviour won't change.
When running as a daemon, unit application processes will send stdout to
/dev/null and stderr to the unit log file.
This commit allows to alter the latter case of unit running as a daemon,
by allowing applications to redirect stdout and/or stderr to specific
log files. This is done via two new application options, 'stdout' &
'stderr', e.g
"applications": {
"myapp": {
...
"stdout": "/path/to/log/unit/app/stdout.log",
"stderr": "/path/to/log/unit/app/stderr.log"
}
}
These log files are created by the application processes themselves and
thus the log directories need to be writable by the user (and or group)
of the application processes.
E.g
$ sudo mkdir -p /path/to/log/unit/app
$ sudo chown APP_USER /path/to/log/unit/app
These need to be setup before starting unit with the above config.
Currently these log files do not participate in log-file rotation
(SIGUSR1), that may change in a future commit. In the meantime these
logs can be rotated using the traditional copy/truncate method.
NOTE:
You may or may not see stuff printed to stdout as stdout was
traditionally used by CGI applications to communicate with the
webserver.
Closes: <https://github.com/nginx/unit/issues/197>
Closes: <https://github.com/nginx/unit/issues/846>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This is analogous to the nxt_file_stderr() function and will be used in
a subsequent commit.
This function redirects stdout to a given file descriptor.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
On GitHub, @jamesRUS52 reported that the PHP filter_input()[0] function
would just return NULL.
To enable this function we need to run the variables through the
sapi_module.input_filter() function when we call
php_register_variable_safe().
In PHP versions prior to 7.0.0, input_filter() takes 'len' as an
unsigned int, while later versions take it as a size_t.
Now, with this commit and the following PHP
<?php
var_dump(filter_input(INPUT_SERVER, 'REMOTE_ADDR'));
var_dump(filter_input(INPUT_SERVER, 'REQUEST_URI'));
var_dump(filter_input(INPUT_GET, 'get', FILTER_SANITIZE_SPECIAL_CHARS));
?>
you get
$ curl 'http://localhost:8080/854.php?get=foo<>'
string(3) "::1"
string(18) "/854.php?get=foo<>"
string(13) "foo<>"
[0]: <https://www.php.net/manual/en/function.filter-input.php>
Tested-by: <https://github.com/jamesRUS52>
Closes: <https://github.com/nginx/unit/issues/854>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This was reported by the 'Clang Static Analyzer' as a 'dead nested
assignment'.
We assign prev_size then check if it's != 0 and if true we then set
prev_pages to page_size right shifted by two at the same time setting
prev_size to be right shifted by two (>>=), however page_size is never
used again so no need to set it here.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
We allocate 'job' we then have a check if it's not NULL and do stuff
with it, but then we accessed it outside this check.
Simply return if job is NULL.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This was reported by the 'Clang Static Analyzer' as a 'dead nested
assignment'.
We set end outside the loop but the first time we use it is to assign it
in the loop (not used anywhere else).
Further cleanup could be to reduce the scope of end by moving its
declaration inside the loop.
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
|
|
This makes the build tree more organized, which is good for adding new
stuff. Now, it's useful for example for adding manual pages in man3/,
but it may be useful in the future for example for extending the build
system to run linters (e.g., clang-tidy(1), Clang analyzer, ...) on the
C source code.
Previously, the build tree was quite flat, and looked like this (after
`./configure && make`):
$ tree -I src build
build
├── Makefile
├── autoconf.data
├── autoconf.err
├── echo
├── libnxt.a
├── nxt_auto_config.h
├── nxt_version.h
├── unitd
└── unitd.8
1 directory, 9 files
And after this patch, it looks like this:
$ tree -I src build
build
├── Makefile
├── autoconf.data
├── autoconf.err
├── bin
│ └── echo
├── include
│ ├── nxt_auto_config.h
│ └── nxt_version.h
├── lib
│ ├── libnxt.a
│ └── unit
│ └── modules
├── sbin
│ └── unitd
├── share
│ └── man
│ └── man8
│ └── unitd.8
└── var
├── lib
│ └── unit
├── log
│ └── unit
└── run
└── unit
17 directories, 9 files
It also solves one issue introduced in
5a37171f733f ("Added default values for pathnames."). Before that
commit, it was possible to run unitd from the build system
(`./build/unitd`). Now, since it expects files in a very specific
location, that has been broken. By having a directory structure that
mirrors the installation, it's possible to trick it to believe it's
installed, and run it from there:
$ ./configure --prefix=./build
$ make
$ ./build/sbin/unitd
Fixes: 5a37171f733f ("Added default values for pathnames.")
Reported-by: Liam Crilly <liam@nginx.com>
Reviewed-by: Konstantin Pavlov <thresh@nginx.com>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Cc: Andrei Zeliankou <zelenkov@nginx.com>
Cc: Zhidao Hong <z.hong@f5.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
|
|
In BSD systems, it's usually </var/db> or some other dir under </var>
that is not </var/lib>, so $statedir is a more generic name. See
hier(7).
Reported-by: Andrei Zeliankou <zelenkov@nginx.com>
Reported-by: Zhidao Hong <z.hong@f5.com>
Reviewed-by: Konstantin Pavlov <thresh@nginx.com>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Cc: Liam Crilly <liam@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
|
|
- Configuration: added "/config/settings/http/log_route".
Type: bool
Default: false
This adds configurability to the error log. It allows enabling and
disabling logs related to how the router performs selection of the
routes.
- HTTP: logging request line.
Log level: [notice]
The request line is essential to understand which logs correspond to
which request when reading the logs.
- HTTP: logging route that's been discarded.
Log level: [info]
- HTTP: logging route whose action is selected.
Log level: [notice]
- HTTP: logging when "fallback" action is taken.
Log level: [notice]
Closes: <https://github.com/nginx/unit/issues/758>
Link: <https://github.com/nginx/unit/pull/824>
Link: <https://github.com/nginx/unit/pull/839>
Suggested-by: Timo Stark <t.stark@nginx.com>
Suggested-by: Mark L Wood-Patrick <mwoodpatrick@gmail.com>
Suggested-by: Liam Crilly <liam@nginx.com>
Tested-by: Liam Crilly <liam@nginx.com>
Acked-by: Artem Konev <a.konev@f5.com>
Cc: Andrew Clayton <a.clayton@nginx.com>
Cc: Andrei Zeliankou <zelenkov@nginx.com>
Reviewed-by: Zhidao Hong <z.hong@f5.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
|
|
This considerably simplifies the function, and will also help log the
iteration in which we are, which corresponds to the route array element.
Link: <https://github.com/nginx/unit/pull/824>
Link: <https://github.com/nginx/unit/pull/839>
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Tested-by: Liam Crilly <liam@nginx.com>
Reviewed-by: Zhidao Hong <z.hong@f5.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
|
