| Age | Commit message (Collapse) | Author |
|---|
|
Summary:
This is an attempt at making `PopulateFreeArray` less obscure, more consistent,
and a tiny bit faster in some circumstances:
- use more consistent variable names, that work both for the user & the metadata
portions of the code; the purpose of the code is mostly the same for both
regions, so it makes sense that the code should be mostly similar as well;
- replace the while sum loops with a single `RoundUpTo`;
- mask most of the metadata computations behind kMetadataSize, allowing some
blocks to be completely optimized out if not use metadata;
- `const` the constant variables;
- add a `LIKELY` as the branch it applies to will almost always be taken.
Reviewers: alekseyshl, flowerhack
Reviewed By: alekseyshl
Subscribers: kubamracek, llvm-commits
Differential Revision: https://reviews.llvm.org/D40754
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@319673 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
Update sanitizer_allocator to use new API.
Second patch in a series. First patch https://reviews.llvm.org/D39072
Updates MmapNoAccess / MmapFixed call sites in the saniziter_allocator
to use the new Init/Map APIs instead.
Reviewers: alekseyshl, cryptoad, phosek, mcgrathr, dvyukov
Reviewed By: alekseyshl, cryptoad
Subscribers: dvyukov, mcgrathr, kubamracek
Differential Revision: https://reviews.llvm.org/D38592
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@317586 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
Call NanoTime() in primary 64 bit allocator only when necessary,
otherwise the unwarranted syscall causes problems in sandbox environments.
ReleaseToOSIntervalMs() conditional allows them to turn the feature off
with allocator_release_to_os_interval_ms=-1 flag.
Reviewers: eugenis
Subscribers: kubamracek, llvm-commits
Differential Revision: https://reviews.llvm.org/D39624
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@317386 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
With new release to OS approach (see D38245) it's reasonable to enable
it by default. Setting allocator_release_to_os_interval_ms to 5000 seems
to be a reasonable default (might be tuned later, based on the
feedback).
Also delaying the first release to OS in each bucket for at least
allocator_release_to_os_interval_ms after the first allocation to
prevent just allocated memory to be madvised back to OS and let short
lived processes to avoid release to OS overhead altogether.
Reviewers: cryptoad
Subscribers: kubamracek, llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D39318
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@316683 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
The 64-bit primary has had random shuffling of chunks for a while, this
implements it for the 32-bit primary. Scudo is currently the only user of
`kRandomShuffleChunks`.
This change consists of a few modifications:
- move the random shuffling functions out of the 64-bit primary to
`sanitizer_common.h`. Alternatively I could move them to
`sanitizer_allocator.h` as they are only used in the allocator, I don't feel
strongly either way;
- small change in the 64-bit primary to make the `rand_state` initialization
`UNLIKELY`;
- addition of a `rand_state` in the 32-bit primary's `SizeClassInfo` and
shuffling of chunks when populating the free list.
- enabling the `random_shuffle.cpp` test on platforms using the 32-bit primary
for Scudo.
Some comments on why the shuffling is done that way. Initially I just
implemented a `Shuffle` function in the `TransferBatch` which was simpler but I
came to realize this wasn't good enough: for chunks of 10000 bytes for example,
with a `CompactSizeClassMap`, a batch holds only 1 chunk, meaning shuffling the
batch has no effect, while a region is usually 1MB, eg: 104 chunks of that size.
So I decided to "stage" the newly gathered chunks in a temporary array that
would be shuffled prior to placing the chunks in batches.
The result is looping twice through n_chunks even if shuffling is not enabled,
but I didn't notice any significant significant performance impact.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: srhines, llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D39244
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@316596 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary: .
Reviewers: cryptoad
Subscribers: llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D39131
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@316356 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
Purging allocator quarantine and returning memory to OS might be desired
between fuzzer iterations since, most likely, the quarantine is not
going to catch bugs in the code under fuzz, but reducing RSS might
significantly prolong the fuzzing session.
Reviewers: cryptoad
Subscribers: kubamracek, llvm-commits
Differential Revision: https://reviews.llvm.org/D39153
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@316347 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary: .
Reviewers: cryptoad
Subscribers: kubamracek, llvm-commits
Differential Revision: https://reviews.llvm.org/D38834
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@315730 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
The current implementation of the allocator returning freed memory
back to OS (controlled by allocator_release_to_os_interval_ms flag)
requires sorting of the free chunks list, which has two major issues,
first, when free list grows to millions of chunks, sorting, even the
fastest one, is just too slow, and second, sorting chunks in place
is unacceptable for Scudo allocator as it makes allocations more
predictable and less secure.
The proposed approach is linear in complexity (altough requires quite
a bit more temporary memory). The idea is to count the number of free
chunks on each memory page and release pages containing free chunks
only. It requires one iteration over the free list of chunks and one
iteration over the array of page counters. The obvious disadvantage
is the allocation of the array of the counters, but even in the worst
case we support (4T allocator space, 64 buckets, 16 bytes bucket size,
full free list, which leads to 2 bytes per page counter and ~17M page
counters), requires just about 34Mb of the intermediate buffer (comparing
to ~64Gb of actually allocated chunks) and usually it stays under 100K
and released after each use. It is expected to be a relatively rare event,
releasing memory back to OS, keeping the buffer between those runs
and added complexity of the bookkeeping seems unnesessary here (it can
always be improved later, though, never say never).
The most interesting problem here is how to calculate the number of chunks
falling into each memory page in the bucket. Skipping all the details,
there are three cases when the number of chunks per page is constant:
1) P >= C, P % C == 0 --> N = P / C
2) C > P , C % P == 0 --> N = 1
3) C <= P, P % C != 0 && C % (P % C) == 0 --> N = P / C + 1
where P is page size, C is chunk size and N is the number of chunks per
page and the rest of the cases, where the number of chunks per page is
calculated on the go, during the page counter array iteration.
Among the rest, there are still cases where N can be deduced from the
page index, but they require not that much less calculations per page
than the current "brute force" way and 2/3 of the buckets fall into
the first three categories anyway, so, for the sake of simplicity,
it was decided to stick to those two variations. It can always be
refined and improved later, should we see that brute force way slows
us down unacceptably.
Reviewers: eugenis, cryptoad, dvyukov
Subscribers: kubamracek, mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D38245
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@314311 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
In `sanitizer_allocator_primary32.h`:
- rounding up in `MapWithCallback` is not needed as `MmapOrDie` does it. Note
that the 64-bit counterpart doesn't round up, this keeps the behavior
consistent;
- since `IsAligned` exists, use it in `AllocateRegion`;
- in `PopulateFreeList`:
- checking `b->Count` to be greater than 0 when `b->Count() == max_count` is
redundant when done more than once. Just check that `max_count` is greater
than 0 out of the loop; the compiler (at least on ARM) didn't optimize it;
- mark the batch creation failure as `UNLIKELY`;
In `sanitizer_allocator_primary64.h`:
- in `MapWithCallback`, mark the failure condition as `UNLIKELY`;
In `sanitizer_posix.h`:
- mark a bunch of Mmap related failure conditions as `UNLIKELY`;
- in `MmapAlignedOrDieOnFatalError`, we have `IsAligned`, so use it; rearrange
the conditions as one test was redudant;
- in `MmapFixedImpl`, 30 chars was not large enough to hold the message and a
full 64-bit address (or at least a 48-bit usermode address), increase to 40.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: aemerson, kubamracek, kristof.beyls, llvm-commits
Differential Revision: https://reviews.llvm.org/D34840
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@306834 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
Make SizeClassAllocator64 return nullptr when it encounters OOM, which
allows the entire sanitizer's allocator to follow
allocator_may_return_null=1 policy
(LargeMmapAllocator: D34243, SizeClassAllocator64: D34433).
Reviewers: eugenis
Subscribers: srhines, kubamracek, llvm-commits
Differential Revision: https://reviews.llvm.org/D34540
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@306342 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary: Improved ASAN allocator and quarantine stats.
Reviewers: eugenis
Patch by Alex Shlyapnikov.
Subscribers: llvm-commits, kubabrecka
Differential Revision: https://reviews.llvm.org/D28333
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@291183 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
The current code was sometimes attempting to release huge chunks of
memory due to undesired RoundUp/RoundDown interaction when the requested
range is fully contained within one memory page.
Reviewers: eugenis
Subscribers: kubabrecka, llvm-commits
Patch by Aleksey Shlyapnikov.
Differential Revision: https://reviews.llvm.org/D27228
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@288271 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
In order to avoid starting a separate thread to return unused memory to
the system (the thread interferes with process startup on Android,
Zygota waits for all threads to exit before fork, but this thread never
exits), try to return it right after free.
Reviewers: eugenis
Subscribers: cryptoad, filcab, danalbert, kubabrecka, llvm-commits
Patch by Aleksey Shlyapnikov.
Differential Revision: https://reviews.llvm.org/D27003
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@288091 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
the free array (kudos to Kostya Korcthinsky). Also make sure that the allocator does not mmap more than requested. Test both.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@281103 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
allocated chunks
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@279983 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
madvise. Requires quite some tuning.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@279887 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
under a flag. Set this flag for the scudo allocator, add a test.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@279793 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
parameter instead of 5. First, this will make the mangled names shorter. Second, this will make adding more parameters simpler.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@279771 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
64-bit allocator to use a single array for free-d chunks instead of a lock-free linked list of tranfer batches. This change simplifies the code, makes the allocator more 'hardened', and will allow simpler code to release RAM to OS. This may also slowdown malloc stress tests due to lock contension, but I did not observe noticeable slowdown on various real multi-threaded benchmarks.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@279664 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
This reverts commit r279572 and r279595.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@279643 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@279595 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
chunks instead of a lock-free linked list of tranfer batches. This change simplifies the code, makes the allocator more 'hardened', and will allow simpler code to release RAM to OS. This may also slowdown malloc stress tests due to lock contension, but I did not observe noticeable slowdown on various real multi-threaded benchmarks.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@279572 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
allocator and one for 64-bit one. NFC. The two imlementations will diverge in the following changes.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@279495 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
allocator's transfer batches. This saves 2x memory for the transfer batches (up to ~1.5% overall in some cases)
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@278179 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@278163 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
SizeClassAllocator64/SizeClassAllocator32 because we actually need different iplementations for the 64- and 32-bit case. NFC; the following patches will make the TransferBatch implementations differ
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@277899 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@277746 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Summary:
The sanitizer allocators can works with a dynamic address space
(i.e. specified with ~0ULL).
Unfortunately, the code was broken on GetMetadata and GetChunkIdx.
The current patch is moving the Win64 memory test to a dynamic
address space. There is a migration to move every concept to a
dynamic address space on windows.
To have a better coverage, the unittest are now testing
dynamic address space on other platforms too.
Reviewers: rnk, kcc
Subscribers: kubabrecka, dberris, llvm-commits, chrisha
Differential Revision: https://reviews.llvm.org/D23170
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@277745 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
Second attempt after failed r276383 which was reverted.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@277554 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@276456 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@276383 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@276343 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
SizeClassMap::MaxCached; ensure that TransferBatch size is a power of two, refactor TransferBatch creation/destruction into separate functions.
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@276318 91177308-0d34-0410-b5e6-96231b3b80d8
|
|
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@276195 91177308-0d34-0410-b5e6-96231b3b80d8
|
