Author:
Date:
Subject:
Ingo Molnar
2008-07-18 01:56:57
Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison
overwritten
* Vegard Nossum <[email protected]> wrote:
> On Thu, Jul 17, 2008 at 11:42 PM, Ingo Molnar <[email protected]> wrote:
> > With this config:
> >
> > http://redhat.com/~mingo/misc/config-Thu_Jul_17_20_24_45_CEST_2008.bad
> >
>
> It doesn't actually work. The config says
>
> # head: 088fcf34
>
> and I checked out this from the tip tree. But kernel-config still
> complains about unknown config options... What went wrong?
that's ok - i've got some local qa helpers that have config options.
Things like making SMP bootups more likely in randconfig, adding various
boot parameters to the bootup via .config methods (so that boot
parameters can be randomized via make randconfig), etc.
these:
CONFIG_BOOTPARAM_SUPPORT=y
CONFIG_BOOTPARAM_NO_HZ_OFF=y
CONFIG_BOOTPARAM_NMI_WATCHDOG_BIT_0=y
CONFIG_BOOTPARAM_LAPIC=y
CONFIG_BOOTPARAM_IDLE_MWAIT=y
CONFIG_BOOTPARAM_NOPAT=y
CONFIG_BOOTPARAM_NOTSC=y
are equivalent to adding this to the boot line:
nohz=off nmi_watchdog=1 lapic idle=mwait nopat notsc
although i dont think they are normally material to netconsole workings.
Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Author:
Date:
Subject:
Vegard Nossum
2008-07-21 13:41:54
Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten
On Fri, Jul 18, 2008 at 1:56 AM, Ingo Molnar <[email protected]> wrote:
>
> * Vegard Nossum <[email protected]> wrote:
>
>> On Thu, Jul 17, 2008 at 11:42 PM, Ingo Molnar <[email protected]> wrote:
>> > With this config:
>> >
>> > http://redhat.com/~mingo/misc/config-Thu_Jul_17_20_24_45_CEST_2008.bad
>> >
>>
>> It doesn't actually work. The config says
>>
>> # head: 088fcf34
>>
>> and I checked out this from the tip tree. But kernel-config still
>> complains about unknown config options... What went wrong?
>
> that's ok - i've got some local qa helpers that have config options.
>
> Things like making SMP bootups more likely in randconfig, adding various
> boot parameters to the bootup via .config methods (so that boot
> parameters can be randomized via make randconfig), etc.
Hm, your config doesn't contain SLUB_DEBUG either. Isn't that a
requirement for seeing the BUG you're getting?
$ grep SLUB config-Thu_Jul_17_20_24_45_CEST_2008.bad
# CONFIG_SLUB_DEBUG is not set
CONFIG_SLUB=y
Did you post the wrong config?
Vegard
--
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
-- E. W. Dijkstra, EWD1036
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Author:
Date:
Subject:
Evgeniy Polyakov
2008-07-18 09:46:26
Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten
Hi Ingo.
On Thu, Jul 17, 2008 at 11:42:22PM +0200, Ingo Molnar ([email protected]) wrote:
> Pid: 5098, comm: gdm-binary Not tainted 2.6.26-tip #3094
> [<c0186f99>] print_trailer+0xa9/0xf0
> [<c018707b>] check_bytes_and_report+0x9b/0xc0
> [<c01874ae>] check_object+0x19e/0x1e0
> [<c0187ef1>] __slab_alloc+0x371/0x4e0
> [<c0188552>] kmem_cache_alloc+0xb2/0xc0
> [<c06732dc>] ? __alloc_skb+0x2c/0x110
Out of curiosity, why does it scream at allocation time?
Does SLUB have a debug check at freeing time? If so, how does it work
and why didn't it caught use after free there?
--
Evgeniy Polyakov
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Author:
Date:
Subject:
Pekka Enberg
2008-07-18 12:02:26
Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten
Hi Evgeniy,
On Fri, Jul 18, 2008 at 8:46 AM, Evgeniy Polyakov <[email protected]> wrote:
> Hi Ingo.
>
> On Thu, Jul 17, 2008 at 11:42:22PM +0200, Ingo Molnar ([email protected]) wrote:
>> Pid: 5098, comm: gdm-binary Not tainted 2.6.26-tip #3094
>> [<c0186f99>] print_trailer+0xa9/0xf0
>> [<c018707b>] check_bytes_and_report+0x9b/0xc0
>> [<c01874ae>] check_object+0x19e/0x1e0
>> [<c0187ef1>] __slab_alloc+0x371/0x4e0
>> [<c0188552>] kmem_cache_alloc+0xb2/0xc0
>> [<c06732dc>] ? __alloc_skb+0x2c/0x110
>
> Out of curiosity, why does it scream at allocation time?
Because it's checking for use-after-free errors. The object is
poisoned with POISON_FREE when it's free'd and we verify the poison
values at allocation time.
On Fri, Jul 18, 2008 at 8:46 AM, Evgeniy Polyakov <[email protected]> wrote:
> Does SLUB have a debug check at freeing time? If so, how does it work
> and why didn't it caught use after free there?
You can't detect use after free before the object is actually free'd ;-)
Pekka
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Author:
Date:
Subject:
Ingo Molnar
2008-07-18 11:09:51
Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison
overwritten
* Pekka Enberg <[email protected]> wrote:
> On Fri, Jul 18, 2008 at 8:46 AM, Evgeniy Polyakov <[email protected]> wrote:
> > Does SLUB have a debug check at freeing time? If so, how does it work
> > and why didn't it caught use after free there?
>
> You can't detect use after free before the object is actually free'd
> ;-)
yeah, we want to check use-after free at the next allocation point -
i.e. as late as possible to gather all corruptions that happened
meanwhile.
We could in theory have a SLUB debug mode where a SCHED_IDLE kernel
thread would periodically check all free objects (of that CPU) in the
background to ensure their integrity. That would catch corruptions
sooner, with a possibly still meaningful context to print out. [right
after the IRQ or process that corrupts them finishes running]
It could also be hooked into ftrace to print out the last few hundred
kernel function calls executed prior any corruption. ftrace/slub-debug
plugin perhaps?
Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Author:
Date:
Subject:
Pekka Enberg
2008-07-18 12:15:26
Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten
Hi Ingo,
On Fri, Jul 18, 2008 at 12:09 PM, Ingo Molnar <[email protected]> wrote:
> yeah, we want to check use-after free at the next allocation point -
> i.e. as late as possible to gather all corruptions that happened
> meanwhile.
>
> We could in theory have a SLUB debug mode where a SCHED_IDLE kernel
> thread would periodically check all free objects (of that CPU) in the
> background to ensure their integrity. That would catch corruptions
> sooner, with a possibly still meaningful context to print out. [right
> after the IRQ or process that corrupts them finishes running]
>
> It could also be hooked into ftrace to print out the last few hundred
> kernel function calls executed prior any corruption. ftrace/slub-debug
> plugin perhaps?
Well, there's this Norwegian guy, Vegard, who has written a small
piece of code that can detect use-after-free _as it happens_. I think
he calls the thing kmemcheck :-).
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Author:
Date:
Subject:
Evgeniy Polyakov
2008-07-18 14:16:24
Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten
Hi Pekka.
On Fri, Jul 18, 2008 at 12:02:26PM +0300, Pekka Enberg ([email protected]) wrote:
> > Out of curiosity, why does it scream at allocation time?
>
> Because it's checking for use-after-free errors. The object is
> poisoned with POISON_FREE when it's free'd and we verify the poison
> values at allocation time.
Does it also scream on double free event? Just to closer guilty
circles... 0x9c offset is somewhere at the very end of the skbuff
structure, likely skb->users.
Can you also check in some kind of this patch to catch freed skb freeing
for testing?
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 3666216..dda96bf 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -419,6 +419,14 @@ void kfree_skb(struct sk_buff *skb)
{
if (unlikely(!skb))
return;
+
+ {
+ u8 *ptr = (u8 *)(&skb->users);
+
+ if (*ptr == POISON_FREE || *ptr == POISON_INUSE || *ptr == POISON_END)
+ BUG();
+ }
+
if (likely(atomic_read(&skb->users) == 1))
smp_rmb();
else if (likely(!atomic_dec_and_test(&skb->users)))
--
Evgeniy Polyakov
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
|
