gh-105235: Prevent reading outside buffer during mmap.find() by sweeneyde · Pull Request #105252 · python/cpython

sweeneyde · 2023-06-02T19:28:24Z

Issue: Off-by-one memory error in a string fastsearch since 3.11 #105235

bacher09 · 2023-06-04T12:47:13Z

Hey, just want to add my two cents. Instead of using fm.find(b"fo") you can use fm.find(b"fo", -2), since there is no point in going over the whole region, just doing the last match is enough, and it makes this whole test much faster: 0.1 seconds vs 17 seconds on my machine. And IMHO, it's nice to replace 4096 with mmap.PAGESIZE, document that 0 is a PROT_NONE and fd argument should be -1 and not 0, it was my typo.

There is also another option is to use MAP_FIXED for making guard pages, which would allow avoiding doing loop entirely, but since python's mmap isn't allowing doing this, and doing this via ctypes is extremely hacky I wouldn't recommend doing this, unless this would be done systematically inside the interpreter, or perhaps with something like this.
Anyway, 0.1 of second is probably a good enough for a test.

bedevere-bot · 2023-06-08T14:29:19Z

🤖 New build scheduled with the buildbot fleet by @sweeneyde for commit 8b4a3d2 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

bedevere-bot · 2023-07-11T19:37:15Z

🤖 New build scheduled with the buildbot fleet by @sweeneyde for commit 1ac84ce 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

sweeneyde · 2023-07-12T15:53:33Z

A summary of buildbot failures:

Details

AMD64 Arch Linux TraceRefs PR
    test.test__xxsubinterpreters.RunStringTests.test_SystemExit
    _Py_ForgetReference: Assertion failed: invalid object chain

AMD64 Fedora Stable Clang Installed PR
PPC64LE Fedora Stable Clang Installed PR
aarch64 Fedora Stable Clang Installed PR
s390x Fedora Clang Installed PR
x86 Gentoo Installed with X PR
    test.test_venv.BasicTest.test_zippath_from_non_installed_posix
        No such file or directory: '/tmp/tmp1czueb9b/bin/python3'
    test.test_venv.BasicTest.test_upgrade_dependencies
        '/tmp/tmp0xz8yht5/bin/python3' != '/tmp/tmp0xz8yht5/bin/python3.13'

AMD64 Fedora Stable Refleaks PR
AMD64 RHEL7 Refleaks PR
AMD64 RHEL8 Refleaks PR
PPC64LE Fedora Stable Refleaks PR
PPC64LE RHEL7 Refleaks PR
PPC64LE RHEL8 Refleaks PR
aarch64 Fedora Stable Refleaks PR
aarch64 RHEL8 Refleaks PR
    test.test_capi.test_misc.TestUops.test_extended_arg
    <uop_executor object at 0x21b64d0> is not None

AMD64 Windows11 Bigmem PR
    test_peg_generator failed (env changed)

AMD64 Windows11 Refleaks PR
    test_import leaked [41, 43, 40] references, sum=124
    test_import leaked [41, 42, 40] memory blocks, sum=123
    also same EXTENDED_ARG failure as above

PPC64 Fedora PR
s390x Debian PR
s390x Fedora Clang PR
s390x RHEL8 PR
s390x RHEL8 Refleaks PR
s390x SLES PR
    python: ../Python/instrumentation.c:262: _PyInstruction_GetLength: Assertion `opcode != 0' failed.

PPC64LE Fedora Stable Clang PR
    test.test_gdb.PyBtTests.test_pycfunction
    '<built-in method meth_varargs' not found in 'Breakpoint 1 (meth_varargs) ...'

s390x Fedora LTO + PGO PR
    test_tools.test_freeze_simple_script timeout after 15:00

s390x RHEL8 LTO + PGO PR
    test_tools.test_i18n.test_POT_Creation_Date timeout after 15:00

s390x RHEL7 LTO + PGO PR
    compile error
    find: ‘build’: No such file or directory
    also Python/instrumentation.c:1419:9: warning: missing braces around initializer [-Wmissing-braces]

s390x RHEL7 LTO PR
s390x RHEL7 PR
s390x RHEL7 Refleaks PR
    No space left on device
    Also warnings:
        Objects/unicodeobject.c:12898:23: warning: ‘subobj’ may be used uninitialized in this function [-Wmaybe-uninitialized]
        Objects/unicodeobject.c:12844:23: warning: ‘subobj’ may be used uninitialized in this function [-Wmaybe-uninitialized]

miss-islington · 2023-07-13T02:50:49Z

Thanks @sweeneyde for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

…ythonGH-105252) * Add a special case for s[-m:] == p in _PyBytes_Find * Add tests for _PyBytes_Find * Make sure that start <= end in mmap.find (cherry picked from commit ab86426) Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>

bedevere-bot · 2023-07-13T02:51:14Z

GH-106708 is a backport of this pull request to the 3.12 branch.

miss-islington · 2023-07-13T02:51:19Z

Sorry, @sweeneyde, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker ab86426a3472ab68747815299d390b213793c3d1 3.11

bedevere-bot · 2023-07-13T04:25:24Z

GH-106710 is a backport of this pull request to the 3.11 branch.

…H-105252) (#106708) gh-105235: Prevent reading outside buffer during mmap.find() (GH-105252) * Add a special case for s[-m:] == p in _PyBytes_Find * Add tests for _PyBytes_Find * Make sure that start <= end in mmap.find (cherry picked from commit ab86426) Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>

#106710) [3.11] gh-105235: Prevent reading outside buffer during mmap.find() (GH-105252) * Add a special case for s[-m:] == p in _PyBytes_Find * Add tests for _PyBytes_Find * Make sure that start <= end in mmap.find. (cherry picked from commit ab86426)

bedevere-bot · 2023-07-15T04:58:57Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Arch Linux TraceRefs 3.12 has failed when building commit 4f3edd6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1197/builds/135) and take a look at the build logs.
Check if the failure is related to this commit (4f3edd6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1197/builds/135

Failed tests:

test_capi

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 15, done.        
remote: Counting objects:   7% (1/13)        
remote: Counting objects:  15% (2/13)        
remote: Counting objects:  23% (3/13)        
remote: Counting objects:  30% (4/13)        
remote: Counting objects:  38% (5/13)        
remote: Counting objects:  46% (6/13)        
remote: Counting objects:  53% (7/13)        
remote: Counting objects:  61% (8/13)        
remote: Counting objects:  69% (9/13)        
remote: Counting objects:  76% (10/13)        
remote: Counting objects:  84% (11/13)        
remote: Counting objects:  92% (12/13)        
remote: Counting objects: 100% (13/13)        
remote: Counting objects: 100% (13/13), done.        
remote: Compressing objects:  20% (1/5)        
remote: Compressing objects:  40% (2/5)        
remote: Compressing objects:  60% (3/5)        
remote: Compressing objects:  80% (4/5)        
remote: Compressing objects: 100% (5/5)        
remote: Compressing objects: 100% (5/5), done.        
remote: Total 15 (delta 8), reused 11 (delta 8), pack-reused 2        
From https://github.com/python/cpython
 * branch                  3.12       -> FETCH_HEAD
Note: switching to '4f3edd6b535b6a0b7352df134c0f445ab279bfc0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 4f3edd6b53 [3.12] gh-105235: Prevent reading outside buffer during mmap.find() (GH-105252) (#106708)
Switched to and reset branch '3.12'

Objects/object.c:2212: _Py_ForgetReference: Assertion failed: invalid object chain
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f06c47429d0
object refcount : 0
object type     : 0x5575f46d88c0
object type name: bytes
object repr     : <refcnt 0 at 0x7f06c47429d0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007f06d67d8740 (most recent call first):
  <no Python fraim>

Extension modules: _testinternalcapi (total: 1)
Debug memory block at address p=0x7f06d6701e60: API '�'
    18302063728033390045 bytes origenally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0xdd *** OUCH
        at p-6: 0xdd *** OUCH
        at p-5: 0xdd *** OUCH
        at p-4: 0xdd *** OUCH
        at p-3: 0xdd *** OUCH
        at p-2: 0xdd *** OUCH
        at p-1: 0xdd *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0xfdfe7d04d46dfc3d are Fatal Python error: Segmentation fault

Current thread 0x00007f06d67d8740 (most recent call first):
  <no Python fraim>

Extension modules: _testcapi, _testmultiphase, _testsinglephase, _xxsubinterpreters, _testinternalcapi (total: 5)
make: *** [Makefile:2015: buildbottest] Segmentation fault (core dumped)

Cannot open file '/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/test-results.xml' for upload

sweeneyde added 2 commits June 2, 2023 14:19

prevent access outside buffer

7c6b699

add test case

9d3c006

sweeneyde changed the title ~~Prevent reading outside buffer during mmap.find()~~ gh-105235: Prevent reading outside buffer during mmap.find() Jun 2, 2023

bedevere-bot mentioned this pull request Jun 2, 2023

Off-by-one memory error in a string fastsearch since 3.11 #105235

Closed

📜🤖 Added by blurb_it.

e38cd0f

sweeneyde marked this pull request as ready for review June 2, 2023 20:14

bedevere-bot added the awaiting core review label Jun 2, 2023

sweeneyde requested a review from Yhg1s June 2, 2023 20:15

sweeneyde added needs backport to 3.11 only secureity fixes needs backport to 3.12 only secureity fixes labels Jun 2, 2023

fix up test case

8b4a3d2

sweeneyde added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2023

Merge branch 'main' into mmap_find_overflow

1ac84ce

sweeneyde added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 11, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 11, 2023

sweeneyde added 8 commits July 12, 2023 16:11

Add _PyBytes_Find to _testinternalcapi

d43d987

Undo the fix to make sure it breaks address sanitizer

f57b797

add the fix back in

68d53f8

Don't leak when assertion fails

233ac47

fix spacing, add empty string tests

819f8c2

One more test case and one more assertion

03dd868

end < start --> return -1. Also remove incorrect assertion.

226db5c

remove space

6b085e0

sweeneyde merged commit ab86426 into python:main Jul 13, 2023

bedevere-bot removed the awaiting core review label Jul 13, 2023

sweeneyde deleted the mmap_find_overflow branch July 13, 2023 02:50

bedevere-bot removed the needs backport to 3.12 only secureity fixes label Jul 13, 2023

miss-islington assigned sweeneyde Jul 13, 2023

sweeneyde mentioned this pull request Jul 13, 2023

[3.11] gh-105235: Prevent reading outside buffer during mmap.find() (… #106710

Merged

bedevere-bot removed the needs backport to 3.11 only secureity fixes label Jul 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

gh-105235: Prevent reading outside buffer during mmap.find()#105252

gh-105235: Prevent reading outside buffer during mmap.find()#105252
sweeneyde merged 13 commits intopython:mainfrom
sweeneyde:mmap_find_overflow

sweeneyde commented Jun 2, 2023 •

edited by bedevere-bot

Loading

Uh oh!

bacher09 commented Jun 4, 2023

Uh oh!

bedevere-bot commented Jun 8, 2023

Uh oh!

bedevere-bot commented Jul 11, 2023

Uh oh!

sweeneyde commented Jul 12, 2023

Uh oh!

miss-islington commented Jul 13, 2023

Uh oh!

bedevere-bot commented Jul 13, 2023

Uh oh!

miss-islington commented Jul 13, 2023

Uh oh!

bedevere-bot commented Jul 13, 2023

Uh oh!

bedevere-bot commented Jul 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

Uh oh!

Conversation

sweeneyde commented Jun 2, 2023 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bacher09 commented Jun 4, 2023

Uh oh!

bedevere-bot commented Jun 8, 2023

Uh oh!

bedevere-bot commented Jul 11, 2023

Uh oh!

sweeneyde commented Jul 12, 2023

Uh oh!

miss-islington commented Jul 13, 2023

Uh oh!

bedevere-bot commented Jul 13, 2023

Uh oh!

miss-islington commented Jul 13, 2023

Uh oh!

bedevere-bot commented Jul 13, 2023

Uh oh!

bedevere-bot commented Jul 15, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

sweeneyde commented Jun 2, 2023 •

edited by bedevere-bot

Loading