Daniel Borkmann 39f1735c81 bpf: Fix tail_call_reachable rejection for interpreter when jit failed ... [ Upstream commit 5dd0a6b8582ffbfa88351949d50eccd5b6694ade ] During testing of f263a81451c1 ("bpf: Track subprog poke descriptors correctly and fix use-after-free") under various failure conditions, for example, when jit_subprogs() fails and tries to clean up the program to be run under the interpreter, we ran into the following freeze: [...] #127/8 tailcall_bpf2bpf_3:FAIL [...] [ 92.041251] BUG: KASAN: slab-out-of-bounds in ___bpf_prog_run+0x1b9d/0x2e20 [ 92.042408] Read of size 8 at addr ffff88800da67f68 by task test_progs/682 [ 92.043707] [ 92.044030] CPU: 1 PID: 682 Comm: test_progs Tainted: G O 5.13.0-53301-ge6c08cb33a30-dirty #87 [ 92.045542] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1 04/01/2014 [ 92.046785] Call Trace: [ 92.047171] ? __bpf_prog_run_args64+0xc0/0xc0 [ 92.047773] ? __bpf_prog_run_args32+0x8b/0xb0 [ 92.048389] ? __bpf_prog_run_args64+0xc0/0xc0 [ 92.049019] ? ktime_get+0x117/0x130 [...] // few hundred [similar] lines more [ 92.659025] ? ktime_get+0x117/0x130 [ 92.659845] ? __bpf_prog_run_args64+0xc0/0xc0 [ 92.660738] ? __bpf_prog_run_args32+0x8b/0xb0 [ 92.661528] ? __bpf_prog_run_args64+0xc0/0xc0 [ 92.662378] ? print_usage_bug+0x50/0x50 [ 92.663221] ? print_usage_bug+0x50/0x50 [ 92.664077] ? bpf_ksym_find+0x9c/0xe0 [ 92.664887] ? ktime_get+0x117/0x130 [ 92.665624] ? kernel_text_address+0xf5/0x100 [ 92.666529] ? __kernel_text_address+0xe/0x30 [ 92.667725] ? unwind_get_return_address+0x2f/0x50 [ 92.668854] ? ___bpf_prog_run+0x15d4/0x2e20 [ 92.670185] ? ktime_get+0x117/0x130 [ 92.671130] ? __bpf_prog_run_args64+0xc0/0xc0 [ 92.672020] ? __bpf_prog_run_args32+0x8b/0xb0 [ 92.672860] ? __bpf_prog_run_args64+0xc0/0xc0 [ 92.675159] ? ktime_get+0x117/0x130 [ 92.677074] ? lock_is_held_type+0xd5/0x130 [ 92.678662] ? ___bpf_prog_run+0x15d4/0x2e20 [ 92.680046] ? ktime_get+0x117/0x130 [ 92.681285] ? __bpf_prog_run32+0x6b/0x90 [ 92.682601] ? __bpf_prog_run64+0x90/0x90 [ 92.683636] ? lock_downgrade+0x370/0x370 [ 92.684647] ? mark_held_locks+0x44/0x90 [ 92.685652] ? ktime_get+0x117/0x130 [ 92.686752] ? lockdep_hardirqs_on+0x79/0x100 [ 92.688004] ? ktime_get+0x117/0x130 [ 92.688573] ? __cant_migrate+0x2b/0x80 [ 92.689192] ? bpf_test_run+0x2f4/0x510 [ 92.689869] ? bpf_test_timer_continue+0x1c0/0x1c0 [ 92.690856] ? rcu_read_lock_bh_held+0x90/0x90 [ 92.691506] ? __kasan_slab_alloc+0x61/0x80 [ 92.692128] ? eth_type_trans+0x128/0x240 [ 92.692737] ? __build_skb+0x46/0x50 [ 92.693252] ? bpf_prog_test_run_skb+0x65e/0xc50 [ 92.693954] ? bpf_prog_test_run_raw_tp+0x2d0/0x2d0 [ 92.694639] ? __fget_light+0xa1/0x100 [ 92.695162] ? bpf_prog_inc+0x23/0x30 [ 92.695685] ? __sys_bpf+0xb40/0x2c80 [ 92.696324] ? bpf_link_get_from_fd+0x90/0x90 [ 92.697150] ? mark_held_locks+0x24/0x90 [ 92.698007] ? lockdep_hardirqs_on_prepare+0x124/0x220 [ 92.699045] ? finish_task_switch+0xe6/0x370 [ 92.700072] ? lockdep_hardirqs_on+0x79/0x100 [ 92.701233] ? finish_task_switch+0x11d/0x370 [ 92.702264] ? __switch_to+0x2c0/0x740 [ 92.703148] ? mark_held_locks+0x24/0x90 [ 92.704155] ? __x64_sys_bpf+0x45/0x50 [ 92.705146] ? do_syscall_64+0x35/0x80 [ 92.706953] ? entry_SYSCALL_64_after_hwframe+0x44/0xae [...] Turns out that the program rejection from e411901c0b ("bpf: allow for tailcalls in BPF subprograms for x64 JIT") is buggy since env->prog->aux->tail_call_reachable is never true. Commit ebf7d1f508 ("bpf, x64: rework pro/epilogue and tailcall handling in JIT") added a tracker into check_max_stack_depth() which propagates the tail_call_reachable condition throughout the subprograms. This info is then assigned to the subprogram's func[i]->aux->tail_call_reachable. However, in the case of the rejection check upon JIT failure, env->prog->aux->tail_call_reachable is used. func[0]->aux->tail_call_reachable which represents the main program's information did not propagate this to the outer env->prog->aux, though. Add this propagation into check_max_stack_depth() where it needs to belong so that the check can be done reliably. Fixes: ebf7d1f508 ("bpf, x64: rework pro/epilogue and tailcall handling in JIT") Fixes: e411901c0b ("bpf: allow for tailcalls in BPF subprograms for x64 JIT") Co-developed-by: John Fastabend <john.fastabend@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: John Fastabend <john.fastabend@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com> Link: https://lore.kernel.org/bpf/618c34e3163ad1a36b1e82377576a6081e182f25.1626123173.git.daniel@iogearbox.net Signed-off-by: Sasha Levin <sashal@kernel.org>		2021-07-28 14:35:37 +02:00
..
preload	bpf: Fix umd memory leak in copy_process()	2021-03-30 14:32:03 +02:00
arraymap.c	bpf: Allow for map-in-map with dynamic inner array map entries	2020-10-11 10:21:04 -07:00
bpf_inode_storage.c	bpf: Change inode_storage's lookup_elem return value from NULL to -EBADF	2021-03-30 14:31:56 +02:00
bpf_iter.c	bpf: Fix an unitialized value in bpf_iter	2021-03-04 11:37:33 +01:00
bpf_local_storage.c	bpf: Use hlist_add_head_rcu when linking to local_storage	2020-09-19 01:12:35 +02:00
bpf_lru_list.c	bpf_lru_list: Read double-checked variable once without lock	2021-03-04 11:37:29 +01:00
bpf_lru_list.h
bpf_lsm.c	bpf: Update verification logic for LSM programs	2020-11-06 13:15:21 -08:00
bpf_struct_ops_types.h
bpf_struct_ops.c	bpf: Fix fexit trampoline.	2021-04-07 15:00:03 +02:00
btf.c	bpf: Forbid trampoline attach for functions with variable arguments	2021-06-16 12:01:35 +02:00
cgroup.c	bpf, cgroup: Fix problematic bounds check	2021-02-10 09:29:12 +01:00
core.c	bpf: Track subprog poke descriptors correctly and fix use-after-free	2021-07-25 14:36:21 +02:00
cpumap.c	bpf, cpumap: Remove rcpu pointer from cpu_map_build_skb signature	2020-09-28 23:30:42 +02:00
devmap.c	bpf, devmap: Use GFP_KERNEL for xdp bulk queue allocation	2021-03-04 11:37:33 +01:00
disasm.c
disasm.h
dispatcher.c
hashtab.c	bpf: Zero-fill re-used per-cpu map element	2020-11-05 19:55:57 -08:00
helpers.c	bpf, lockdown, audit: Fix buggy SELinux lockdown permission checks	2021-06-10 13:39:19 +02:00
inode.c	bpf: link: Refuse non-O_RDWR flags in BPF_OBJ_GET	2021-04-14 08:42:00 +02:00
local_storage.c	bpf/local_storage: Fix build without CONFIG_CGROUP	2020-07-25 20:16:36 -07:00
lpm_trie.c	bpf: Add map_meta_equal map ops	2020-08-28 15:41:30 +02:00
Makefile	bpf: Don't rely on GCC __attribute__((optimize)) to disable GCSE	2020-10-29 20:01:46 -07:00
map_in_map.c	bpf: Relax max_entries check for most of the inner map types	2020-08-28 15:41:30 +02:00
map_in_map.h	bpf: Add map_meta_equal map ops	2020-08-28 15:41:30 +02:00
map_iter.c	bpf: Implement link_query callbacks in map element iterators	2020-08-21 14:01:39 -07:00
net_namespace.c	bpf: Add support for forced LINK_DETACH command	2020-08-01 20:38:28 -07:00
offload.c
percpu_freelist.c	bpf: Use raw_spin_trylock() for pcpu_freelist_push/pop in NMI	2020-10-06 00:04:11 +02:00
percpu_freelist.h	bpf: Use raw_spin_trylock() for pcpu_freelist_push/pop in NMI	2020-10-06 00:04:11 +02:00
prog_iter.c	bpf: Refactor bpf_iter_reg to have separate seq_info member	2020-07-25 20:16:32 -07:00
queue_stack_maps.c	bpf: Add map_meta_equal map ops	2020-08-28 15:41:30 +02:00
reuseport_array.c	bpf, net: Rework cookie generator as per-cpu one	2020-09-30 11:50:35 -07:00
ringbuf.c	bpf: Fix false positive kmemleak report in bpf_ringbuf_area_alloc()	2021-07-19 09:44:54 +02:00
stackmap.c	bpf: Refcount task stack in bpf_get_task_stack	2021-04-14 08:42:01 +02:00
syscall.c	bpf: Prevent double bpf_prog_put call from bpf_tracing_prog_attach	2021-01-27 11:55:07 +01:00
sysfs_btf.c	bpf: Fix sysfs export of empty BTF section	2020-09-21 21:50:24 +02:00
task_iter.c	bpf: Save correct stopping point in file seq iteration	2021-01-19 18:27:28 +01:00
tnum.c
trampoline.c	bpf: Fix fexit trampoline.	2021-04-07 15:00:03 +02:00
verifier.c	bpf: Fix tail_call_reachable rejection for interpreter when jit failed	2021-07-28 14:35:37 +02:00