summaryrefslogtreecommitdiff
path: root/lib/Target
AgeCommit message (Collapse)Author
2018-04-11Merging r329359 and r329363:Tom Stellard
------------------------------------------------------------------------ r329359 | manojgupta | 2018-04-05 15:47:25 -0700 (Thu, 05 Apr 2018) | 11 lines Attempt to fix Mips breakages. Summary: Replace ArrayRefs by actual std::array objects so that there are no dangling references. Reviewers: rsmith, gkistanova Subscribers: sdardis, arichardson, llvm-commits Differential Revision: https://reviews.llvm.org/D45338 ------------------------------------------------------------------------ ------------------------------------------------------------------------ r329363 | manojgupta | 2018-04-05 16:23:29 -0700 (Thu, 05 Apr 2018) | 5 lines Fix lld-x86_64-darwin13 build fails. Use double braces in std::array initialization to keep Darwin builders happy. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@329859 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-11Merging r325653 with test fixups:Simon Dardis
------------------------------------------------------------------------ r325653 | sdardis | 2018-02-21 00:06:53 +0000 (Wed, 21 Feb 2018) | 31 lines [mips] Spectre variant two mitigation for MIPSR2 This patch provides mitigation for CVE-2017-5715, Spectre variant two, which affects the P5600 and P6600. It implements the LLVM part of -mindirect-jump=hazard. It is _not_ enabled by default for the P5600. The migitation strategy suggested by MIPS for these processors is to use hazard barrier instructions. 'jalr.hb' and 'jr.hb' are hazard barrier variants of the 'jalr' and 'jr' instructions respectively. These instructions impede the execution of instruction stream until architecturally defined hazards (changes to the instruction stream, privileged registers which may affect execution) are cleared. These instructions in MIPS' designs are not speculated past. These instructions are used with the attribute +use-indirect-jump-hazard when branching indirectly and for indirect function calls. These instructions are defined by the MIPS32R2 ISA, so this mitigation method is not compatible with processors which implement an earlier revision of the MIPS ISA. Performance benchmarking of this option with -fpic and lld using -z hazardplt shows a difference of overall 10%~ time increase for the LLVM testsuite. Certain benchmarks such as methcall show a substantially larger increase in time due to their nature. Reviewers: atanasyan, zoran.jovanovic Differential Revision: https://reviews.llvm.org/D43486 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@329798 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-10Merging r329588:Tom Stellard
------------------------------------------------------------------------ r329588 | tstellar | 2018-04-09 09:09:13 -0700 (Mon, 09 Apr 2018) | 11 lines AMDGPU: Initialize GlobalISel passes Summary: This fixes AMDGPU GlobalISel test failures when enabling the AMDGPU target without any other targets that use GlobalISel. Reviewers: arsenm Subscribers: kzhuravl, wdng, nhaehnle, yaxunl, rovka, kristof.beyls, dstuttard, tpr, llvm-commits, t-tye Differential Revision: https://reviews.llvm.org/D45353 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@329662 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-09Merging r327651:Tom Stellard
------------------------------------------------------------------------ r327651 | carrot | 2018-03-15 10:49:12 -0700 (Thu, 15 Mar 2018) | 9 lines [PPC] Avoid non-simple MVT in STBRX optimization PR35402 triggered this case. It bswap and stores a 48bit value, current STBRX optimization transforms it into STBRX. Unfortunately 48bit is not a simple MVT, there is no PPC instruction to support it, and it can't be automatically expanded by llvm, so caused a crash. This patch detects the non-simple MVT and returns early. Differential Revision: https://reviews.llvm.org/D44500 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@329641 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-09Merging r322319:Tom Stellard
------------------------------------------------------------------------ r322319 | matze | 2018-01-11 14:30:43 -0800 (Thu, 11 Jan 2018) | 7 lines PeepholeOptimizer: Fix for vregs without defs The PeepholeOptimizer would fail for vregs without a definition. If this was caused by an undef operand abort to keep the code simple (so we don't need to add logic everywhere to replicate the undef flag). Differential Revision: https://reviews.llvm.org/D40763 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@329619 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-09Merging r326535:Tom Stellard
------------------------------------------------------------------------ r326535 | jvesely | 2018-03-01 18:50:22 -0800 (Thu, 01 Mar 2018) | 6 lines AMDGPU/GCN: Promote i16 ctpop i16 capable ASICs do not support i16 operands for this instruction. Add tablegen pattern to merge chained i16 additions. Differential Revision: https://reviews.llvm.org/D43985 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@329589 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-07Merging r328341:Tom Stellard
------------------------------------------------------------------------ r328341 | apazos | 2018-03-23 10:53:27 -0700 (Fri, 23 Mar 2018) | 16 lines [ARM] Fix "Constant pool entry out of range!" in Thumb1 mode This patch fixes PR36658, "Constant pool entry out of range!" in Thumb1 mode. In ARMConstantIslands::optimizeThumb2JumpTables() in Thumb1 mode, adjustBBOffsetsAfter() is not calculating postOffset correctly by properly accounting for the padding that is required for the constant pool that immediately follows the jump table branch instruction. Reviewers: t.p.northover, eli.friedman Reviewed By: t.p.northover Subscribers: chrib, tstellar, javed.absar, kristof.beyls, llvm-commits Differential Revision: https://reviews.llvm.org/D44709 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@329487 91177308-0d34-0410-b5e6-96231b3b80d8
2018-03-01Merging r326393:Hans Wennborg
------------------------------------------------------------------------ r326393 | ctopper | 2018-03-01 01:08:38 +0100 (Thu, 01 Mar 2018) | 5 lines [X86] Make sure we don't combine (fneg (fma X, Y, Z)) to a target specific node when there are no FMA instructions. This would cause a 'cannot select' error at isel when we should have emitted a lib call and an xor. Fixes PR36553. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@326423 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-26Fix an unused variable warning in non-assert buildsHans Wennborg
git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@326075 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-22Merging r325739:Hans Wennborg
------------------------------------------------------------------------ r325739 | nemanjai | 2018-02-22 04:02:41 +0100 (Thu, 22 Feb 2018) | 9 lines [PowerPC] Do not produce invalid CTR loop with an FRem An FRem instruction inside a loop should prevent the loop from being converted into a CTR loop since this is not an operation that is legal on any PPC subtarget. This will always be a call to a library function which means the loop will be invalid if this instruction is in the body. Fixes PR36292. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325767 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-21Merging r325654:Hans Wennborg
------------------------------------------------------------------------ r325654 | ctopper | 2018-02-21 01:15:48 +0100 (Wed, 21 Feb 2018) | 10 lines [X86] Disable CLWB for Cannon Lake Cannon Lake does not support CLWB, therefore it does not include all features listed under SKX anymore. Instead, enumerate all SKX features with the exception of CLWB. Patch by Gabor Buella Differential Revision: https://reviews.llvm.org/D43380 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325671 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-21[AArch64][GlobalISel] Support G_INSERT/G_EXTRACT of types < s32 bits.Hans Wennborg
These are needed for operations on fp16 types in a later patch. This also re-instates the test/CodeGen/AArch64/GlobalISel/fp16-copy-gpr.mir test that was deleted which depended on this patch. (See PR36345.) git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325669 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-20Merging r325550:Hans Wennborg
I couldn't get fp16-copy-gpr.mir to pass after merging so I removed it until aemerson; the other test I re-generated and it seems to work. ------------------------------------------------------------------------ r325550 | aemerson | 2018-02-20 06:11:57 +0100 (Tue, 20 Feb 2018) | 7 lines [AArch64][GlobalISel] When copying from a gpr32 to an fpr16 reg, convert to fpr32 first. This is a follow on commit to r[x] where we fix the other direction of copy. For this case, after converting the source from gpr32 -> fpr32, we use a subregister copy, which is essentially what EXTRACT_SUBREG does in SDAG land. https://reviews.llvm.org/D43444 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325591 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-20Merging r325463:Hans Wennborg
(I had to re-generate the test and manually update to handle the r323922 MIR physical register sigil. ------------------------------------------------------------------------ r325463 | aemerson | 2018-02-18 18:10:49 +0100 (Sun, 18 Feb 2018) | 8 lines [AArch64][GlobalISel] Fix an assert fail/miscompile when fp16 types are copied to gpr register banks. PR36345. rdar://36478867 Differential Revision: https://reviews.llvm.org/D43310 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325586 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-20Merging r324110:Hans Wennborg
------------------------------------------------------------------------ r324110 | aemerson | 2018-02-02 19:03:30 +0100 (Fri, 02 Feb 2018) | 3 lines [AArch64][GlobalISel] Use getRegClassForTypeOnBank() in selectCopy. Differential Revision: https://reviews.llvm.org/D42832 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325584 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-19Merging r324353:Hans Wennborg
------------------------------------------------------------------------ r324353 | mareko | 2018-02-06 16:17:55 +0100 (Tue, 06 Feb 2018) | 5 lines AMDGPU: Fix S_BUFFER_LOAD_DWORD_SGPR moveToVALU Author: Bas Nieuwenhuizen https://reviews.llvm.org/D42881 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325497 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-14Revert r319778 (and r319911) due to PR36357Hans Wennborg
git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325112 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-14Revert r320917 for PR36357Hans Wennborg
git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325111 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-14Merging r324576:Hans Wennborg
------------------------------------------------------------------------ r324576 | ctopper | 2018-02-08 08:45:55 +0100 (Thu, 08 Feb 2018) | 20 lines [X86] Don't emit KTEST instructions unless only the Z flag is being used Summary: KTEST has weird flag behavior. The Z flag is set for all bits in the AND of the k-registers being 0, and the C flag is set for all bits being 1. All other flags are cleared. We currently emit this instruction in EmitTEST and don't check the condition code. This can lead to strange things like using the S flag after a KTEST for a signed compare. The domain reassignment pass can also transform TEST instructions into KTEST and is not protected against the flag usage either. For now I've disabled this part of the domain reassignment pass. I tried to comment out the checks in the mir test so that we could recover them later, but I couldn't figure out how to get that to work. This patch moves the KTEST handling into LowerSETCC and now creates a ktest+x86setcc. I've chosen this approach because I'd like to add support for the C flag for all ones in a followup patch. To do that requires that I can rewrite the condition code going in the x86setcc to be different than the original SETCC condition code. This fixes PR36182. I'll file a PR to fix domain reassignment once this goes in. Should this be merged to 6.0? Reviewers: spatel, guyblank, RKSimon, zvi Reviewed By: guyblank Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D42770 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325106 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-14Merging r325085:Reid Kleckner
------------------------------------------------------------------------ r325085 | rnk | 2018-02-13 16:24:29 -0800 (Tue, 13 Feb 2018) | 3 lines [X86] Remove dead code from retpoline thunk generation Follow-up to r325049 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325086 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-14Merging r325049:Reid Kleckner
------------------------------------------------------------------------ r325049 | rnk | 2018-02-13 12:47:49 -0800 (Tue, 13 Feb 2018) | 17 lines [X86] Use EDI for retpoline when no scratch regs are left Summary: Instead of solving the hard problem of how to pass the callee to the indirect jump thunk without a register, just use a CSR. At a call boundary, there's nothing stopping us from using a CSR to hold the callee as long as we save and restore it in the prologue. Also, add tests for this mregparm=3 case. I wrote execution tests for __llvm_retpoline_push, but they never got committed as lit tests, either because I never rewrote them or because they got lost in merge conflicts. Reviewers: chandlerc, dwmw2 Subscribers: javed.absar, kristof.beyls, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D43214 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325084 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-14Merging r324645:Reid Kleckner
------------------------------------------------------------------------ r324645 | dwmw2 | 2018-02-08 12:06:05 -0800 (Thu, 08 Feb 2018) | 5 lines [X86] Support 'V' register operand modifier This allows the register name to be printed without the leading '%'. This can be used for emitting calls to the retpoline thunks from inline asm. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325083 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-14Merging r324449:Reid Kleckner
------------------------------------------------------------------------ r324449 | chandlerc | 2018-02-06 22:16:24 -0800 (Tue, 06 Feb 2018) | 15 lines [x86/retpoline] Make the external thunk names exactly match the names that happened to end up in GCC. This is really unfortunate, as the names don't have much rhyme or reason to them. Originally in the discussions it seemed fine to rely on aliases to map different names to whatever external thunk code developers wished to use but there are practical problems with that in the kernel it turns out. And since we're discovering this practical problems late and since GCC has already shipped a release with one set of names, we are forced, yet again, to blindly match what is there. Somewhat rushing this patch out for the Linux kernel folks to test and so we can get it patched into our releases. Differential Revision: https://reviews.llvm.org/D42998 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325082 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-13Merging r324746:Hans Wennborg
------------------------------------------------------------------------ r324746 | arsenm | 2018-02-09 17:57:48 +0100 (Fri, 09 Feb 2018) | 4 lines AMDGPU: Fix layering issue Move utility function that depends on codegen. Fixes build with r324487 reapplied. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325007 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-12Merging r324772:Hans Wennborg
------------------------------------------------------------------------ r324772 | thegameg | 2018-02-09 22:47:07 +0100 (Fri, 09 Feb 2018) | 25 lines [X86][MC] Fix assembling rip-relative addressing + immediate displacements In the rare case where the input contains rip-relative addressing with immediate displacements, *and* the instruction ends with an immediate, we encode the instruction in the wrong way: movl $12345678, 0x400(%rdi) // all good, no rip-relative addr movl %eax, 0x400(%rip) // all good, no immediate at the end of the instruction movl $12345678, 0x400(%rip) // fails, encodes address as 0x3fc(%rip) Offset is a label: movl $12345678, foo(%rip) we want to account for the size of the immediate (in this case, $12345678, 4 bytes). Offset is an immediate: movl $12345678, 0x400(%rip) we should not account for the size of the immediate, assuming the immediate offset is what the user wanted. Differential Revision: https://reviews.llvm.org/D43050 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@324875 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-02Merging r323908:Hans Wennborg
------------------------------------------------------------------------ r323908 | mareko | 2018-01-31 21:18:04 +0100 (Wed, 31 Jan 2018) | 7 lines AMDGPU: Add intrinsics llvm.amdgcn.cvt.{pknorm.i16, pknorm.u16, pk.i16, pk.u16} Reviewers: arsenm, nhaehnle Subscribers: kzhuravl, wdng, yaxunl, dstuttard, tpr, t-tye Differential Revision: https://reviews.llvm.org/D41663 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@324103 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-02Merging r323643:Hans Wennborg
------------------------------------------------------------------------ r323643 | jdevlieghere | 2018-01-29 13:10:32 +0100 (Mon, 29 Jan 2018) | 16 lines [Sparc] Account for bias in stack readjustment Summary: This was broken long ago in D12208, which failed to account for the fact that 64-bit SPARC uses a stack bias of 2047, and it is the *unbiased* value which should be aligned, not the biased one. This was seen to be an issue with Rust. Patch by: jrtc27 (James Clarke) Reviewers: jyknight, venkatra Reviewed By: jyknight Subscribers: jacob_hansen, JDevlieghere, fhahn, fedor.sergeev, llvm-commits Differential Revision: https://reviews.llvm.org/D39425 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@324090 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-02Merging r323909:Hans Wennborg
------------------------------------------------------------------------ r323909 | mareko | 2018-01-31 21:18:11 +0100 (Wed, 31 Jan 2018) | 13 lines AMDGPU: Fold inline offset for loads properly in moveToVALU on GFX9 Summary: This enables load merging into x2, x4, which is driven by inline offsets. 6500 shaders are affected: Code Size in affected shaders: -15.14 % Reviewers: arsenm, nhaehnle Subscribers: kzhuravl, wdng, yaxunl, dstuttard, tpr, t-tye, llvm-commits Differential Revision: https://reviews.llvm.org/D42078 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@324089 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-02Merging r323536:Hans Wennborg
------------------------------------------------------------------------ r323536 | arichardson | 2018-01-26 16:56:14 +0100 (Fri, 26 Jan 2018) | 11 lines [MIPS] Don't crash on unsized extern types with -mgpopt Summary: This fixes an assertion when building the FreeBSD MIPS64 kernel. Reviewers: atanasyan, sdardis, emaste Reviewed By: sdardis Subscribers: krytarowski, llvm-commits Differential Revision: https://reviews.llvm.org/D42571 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@324087 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-02Merging r323781:Hans Wennborg
------------------------------------------------------------------------ r323781 | sdardis | 2018-01-30 17:24:10 +0100 (Tue, 30 Jan 2018) | 15 lines [mips] Fix incorrect sign extension for fpowi libcall PR36061 showed that during the expansion of ISD::FPOWI, that there was an incorrect zero extension of the integer argument which for MIPS64 would then give incorrect results. Address this with the existing mechanism for correcting sign extensions. This resolves PR36061. Thanks to James Cowgill for reporting the issue! Reviewers: atanasyan, hfinkel Differential Revision: https://reviews.llvm.org/D42537 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@324085 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-02Merging r323857:Hans Wennborg
------------------------------------------------------------------------ r323857 | rogfer01 | 2018-01-31 10:23:43 +0100 (Wed, 31 Jan 2018) | 19 lines [ARM] Allow the scheduler to clone a node with glue to avoid a copy CPSR ↔ GPR. In Thumb 1, with the new ADDCARRY / SUBCARRY the scheduler may need to do copies CPSR ↔ GPR but not all Thumb1 targets implement them. The schedule can attempt, before attempting a copy, to clone the instructions but it does not currently do that for nodes with input glue. In this patch we introduce a target-hook to let the hook decide if a glued machinenode is still eligible for copying. In this case these are ARM::tADCS and ARM::tSBCS . As a follow-up of this change we should actually implement the copies for the Thumb1 targets that do implement them and restrict the hook to the targets that can't really do such copy as these clones are not ideal. This change fixes PR35836. Differential Revision: https://reviews.llvm.org/D42051 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@324082 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-02Merging r323915:Hans Wennborg
------------------------------------------------------------------------ r323915 | chandlerc | 2018-01-31 21:56:37 +0100 (Wed, 31 Jan 2018) | 17 lines [x86] Make the retpoline thunk insertion a machine function pass. Summary: This removes the need for a machine module pass using some deeply questionable hacks. This should address PR36123 which is a case where in full LTO the memory usage of a machine module pass actually ended up being significant. We should revert this on trunk as soon as we understand and fix the memory usage issue, but we should include this in any backports of retpolines themselves. Reviewers: echristo, MatzeB Subscribers: sanjoy, mcrosier, mehdi_amini, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D42726 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@324071 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-02Merging r323155:Hans Wennborg
------------------------------------------------------------------------ r323155 | chandlerc | 2018-01-22 23:05:25 +0100 (Mon, 22 Jan 2018) | 133 lines Introduce the "retpoline" x86 mitigation technique for variant #2 of the speculative execution vulnerabilities disclosed today, specifically identified by CVE-2017-5715, "Branch Target Injection", and is one of the two halves to Spectre.. Summary: First, we need to explain the core of the vulnerability. Note that this is a very incomplete description, please see the Project Zero blog post for details: https://googleprojectzero.blogspot.com/2018/01/reading-privileged-memory-with-side.html The basis for branch target injection is to direct speculative execution of the processor to some "gadget" of executable code by poisoning the prediction of indirect branches with the address of that gadget. The gadget in turn contains an operation that provides a side channel for reading data. Most commonly, this will look like a load of secret data followed by a branch on the loaded value and then a load of some predictable cache line. The attacker then uses timing of the processors cache to determine which direction the branch took *in the speculative execution*, and in turn what one bit of the loaded value was. Due to the nature of these timing side channels and the branch predictor on Intel processors, this allows an attacker to leak data only accessible to a privileged domain (like the kernel) back into an unprivileged domain. The goal is simple: avoid generating code which contains an indirect branch that could have its prediction poisoned by an attacker. In many cases, the compiler can simply use directed conditional branches and a small search tree. LLVM already has support for lowering switches in this way and the first step of this patch is to disable jump-table lowering of switches and introduce a pass to rewrite explicit indirectbr sequences into a switch over integers. However, there is no fully general alternative to indirect calls. We introduce a new construct we call a "retpoline" to implement indirect calls in a non-speculatable way. It can be thought of loosely as a trampoline for indirect calls which uses the RET instruction on x86. Further, we arrange for a specific call->ret sequence which ensures the processor predicts the return to go to a controlled, known location. The retpoline then "smashes" the return address pushed onto the stack by the call with the desired target of the original indirect call. The result is a predicted return to the next instruction after a call (which can be used to trap speculative execution within an infinite loop) and an actual indirect branch to an arbitrary address. On 64-bit x86 ABIs, this is especially easily done in the compiler by using a guaranteed scratch register to pass the target into this device. For 32-bit ABIs there isn't a guaranteed scratch register and so several different retpoline variants are introduced to use a scratch register if one is available in the calling convention and to otherwise use direct stack push/pop sequences to pass the target address. This "retpoline" mitigation is fully described in the following blog post: https://support.google.com/faqs/answer/7625886 We also support a target feature that disables emission of the retpoline thunk by the compiler to allow for custom thunks if users want them. These are particularly useful in environments like kernels that routinely do hot-patching on boot and want to hot-patch their thunk to different code sequences. They can write this custom thunk and use `-mretpoline-external-thunk` *in addition* to `-mretpoline`. In this case, on x86-64 thu thunk names must be: ``` __llvm_external_retpoline_r11 ``` or on 32-bit: ``` __llvm_external_retpoline_eax __llvm_external_retpoline_ecx __llvm_external_retpoline_edx __llvm_external_retpoline_push ``` And the target of the retpoline is passed in the named register, or in the case of the `push` suffix on the top of the stack via a `pushl` instruction. There is one other important source of indirect branches in x86 ELF binaries: the PLT. These patches also include support for LLD to generate PLT entries that perform a retpoline-style indirection. The only other indirect branches remaining that we are aware of are from precompiled runtimes (such as crt0.o and similar). The ones we have found are not really attackable, and so we have not focused on them here, but eventually these runtimes should also be replicated for retpoline-ed configurations for completeness. For kernels or other freestanding or fully static executables, the compiler switch `-mretpoline` is sufficient to fully mitigate this particular attack. For dynamic executables, you must compile *all* libraries with `-mretpoline` and additionally link the dynamic executable and all shared libraries with LLD and pass `-z retpolineplt` (or use similar functionality from some other linker). We strongly recommend also using `-z now` as non-lazy binding allows the retpoline-mitigated PLT to be substantially smaller. When manually apply similar transformations to `-mretpoline` to the Linux kernel we observed very small performance hits to applications running typical workloads, and relatively minor hits (approximately 2%) even for extremely syscall-heavy applications. This is largely due to the small number of indirect branches that occur in performance sensitive paths of the kernel. When using these patches on statically linked applications, especially C++ applications, you should expect to see a much more dramatic performance hit. For microbenchmarks that are switch, indirect-, or virtual-call heavy we have seen overheads ranging from 10% to 50%. However, real-world workloads exhibit substantially lower performance impact. Notably, techniques such as PGO and ThinLTO dramatically reduce the impact of hot indirect calls (by speculatively promoting them to direct calls) and allow optimized search trees to be used to lower switches. If you need to deploy these techniques in C++ applications, we *strongly* recommend that you ensure all hot call targets are statically linked (avoiding PLT indirection) and use both PGO and ThinLTO. Well tuned servers using all of these techniques saw 5% - 10% overhead from the use of retpoline. We will add detailed documentation covering these components in subsequent patches, but wanted to make the core functionality available as soon as possible. Happy for more code review, but we'd really like to get these patches landed and backported ASAP for obvious reasons. We're planning to backport this to both 6.0 and 5.0 release streams and get a 5.0 release with just this cherry picked ASAP for distros and vendors. This patch is the work of a number of people over the past month: Eric, Reid, Rui, and myself. I'm mailing it out as a single commit due to the time sensitive nature of landing this and the need to backport it. Huge thanks to everyone who helped out here, and everyone at Intel who helped out in discussions about how to craft this. Also, credit goes to Paul Turner (at Google, but not an LLVM contributor) for much of the underlying retpoline design. Reviewers: echristo, rnk, ruiu, craig.topper, DavidKreitzer Subscribers: sanjoy, emaste, mcrosier, mgorny, mehdi_amini, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D41723 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@324067 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-31Merging r323810:Hans Wennborg
------------------------------------------------------------------------ r323810 | mstorsjo | 2018-01-30 20:50:51 +0100 (Tue, 30 Jan 2018) | 3 lines [AArch64] Properly handle dllimport of variables when using fast-isel Differential Revision: https://reviews.llvm.org/D42567 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@323852 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-30Merging r323706:Hans Wennborg
------------------------------------------------------------------------ r323706 | mareko | 2018-01-30 00:19:10 +0100 (Tue, 30 Jan 2018) | 15 lines AMDGPU: Allow a SGPR for the conditional KILL operand Patch by: Bas Nieuwenhuizen Just use the _e64 variant if needed. This should be possible as per def : Pat < (int_amdgcn_kill (i1 (setcc f32:$src, InlineFPImm<f32>:$imm, cond:$cond))), (SI_KILL_F32_COND_IMM_PSEUDO $src, (bitcast_fpimm_to_i32 $imm), (cond_as_i32imm $cond)) > ; I don't think we can get an immediate for the other operand for which we need the second 32-bit word. https://reviews.llvm.org/D42302 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@323772 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-30Merging r323469:Hans Wennborg
------------------------------------------------------------------------ r323469 | ctopper | 2018-01-25 22:23:57 +0100 (Thu, 25 Jan 2018) | 3 lines [X86] Teach Intel syntax InstPrinter to print lock prefixes that have been parsed from the asm parser. The asm parser puts the lock prefix in the MCInst flags so we need to check that in addition to TSFlags. This matches what the ATT printer does. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@323770 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-30Merging r323672: (test-case re-generated)Hans Wennborg
------------------------------------------------------------------------ r323672 | ctopper | 2018-01-29 18:56:57 +0100 (Mon, 29 Jan 2018) | 5 lines [X86] Don't create SHRUNKBLEND when the condition is used by the true or false operand of the vselect. Fixes PR34592. Differential Revision: https://reviews.llvm.org/D42628 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@323743 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-25Merging r323369 and r323371:Hans Wennborg
------------------------------------------------------------------------ r323369 | aemerson | 2018-01-24 20:59:29 +0100 (Wed, 24 Jan 2018) | 4 lines [GlobalISel] Don't fall back to FastISel. Apparently checking the pass structure isn't enough to ensure that we don't fall back to FastISel, as it's set up as part of the SelectionDAGISel. ------------------------------------------------------------------------ ------------------------------------------------------------------------ r323371 | aemerson | 2018-01-24 21:35:37 +0100 (Wed, 24 Jan 2018) | 12 lines [AArch64][GlobalISel] Fall back during AArch64 isel if we have a volatile load. The tablegen imported patterns for sext(load(a)) don't check for single uses of the load or delete the original after matching. As a result two loads are left in the generated code. This particular issue will be fixed by adding support for a G_SEXTLOAD opcode in future. There are however other potential issues around this that wouldn't be fixed by a G_SEXTLOAD, so until we have a proper solution we don't try to handle volatile loads at all in the AArch64 selector. Fixes/works around PR36018. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@323434 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-24Merging r323190:Hans Wennborg
------------------------------------------------------------------------ r323190 | rksimon | 2018-01-23 12:39:06 +0100 (Tue, 23 Jan 2018) | 5 lines [X86][SSE] LowerBUILD_VECTORAsVariablePermute - fix PSHUFB source/index operand ordering As detailed in rL317463, PSHUFB (like most variable shuffle instructions) uses Op[0] for the source vector and Op[1] for the shuffle index vector, VPERMV works in reverse which is probably where the confusion comes from. Differential Revision: https://reviews.llvm.org/D42380 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@323335 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-24Merging r322372 and r322767:Hans Wennborg
------------------------------------------------------------------------ r322372 | nemanjai | 2018-01-12 15:58:41 +0100 (Fri, 12 Jan 2018) | 10 lines [PowerPC] Zero-extend the compare operand for ATOMIC_CMP_SWAP Part of the fix for https://bugs.llvm.org/show_bug.cgi?id=35812. This patch ensures that the compare operand for the atomic compare and swap is properly zero-extended to 32 bits if applicable. A follow-up commit will fix the extension for the SETCC node generated when expanding an ATOMIC_CMP_SWAP_WITH_SUCCESS. That will complete the bug fix. Differential Revision: https://reviews.llvm.org/D41856 ------------------------------------------------------------------------ ------------------------------------------------------------------------ r322767 | efriedma | 2018-01-17 23:04:36 +0100 (Wed, 17 Jan 2018) | 12 lines [LegalizeDAG] Fix ATOMIC_CMP_SWAP_WITH_SUCCESS legalization. The code wasn't zero-extending correctly, so the comparison could spuriously fail. Adds some AArch64 tests to cover this case. Inspired by D41791. Differential Revision: https://reviews.llvm.org/D41798 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@323334 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-22Merging r322878:Hans Wennborg
------------------------------------------------------------------------ r322878 | aemerson | 2018-01-18 20:21:27 +0100 (Thu, 18 Jan 2018) | 5 lines [AArch64][GlobalISel] Add isel support for global values in the large code model. Fixes PR35958. Differential Revision: https://reviews.llvm.org/D42175 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@323103 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-19Merging r322053:Hans Wennborg
------------------------------------------------------------------------ r322053 | echristo | 2018-01-09 03:38:17 +0100 (Tue, 09 Jan 2018) | 1 line Remove unused function HvxSelector::zerous. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@322953 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-18Merging r322644:Hans Wennborg
------------------------------------------------------------------------ r322644 | d0k | 2018-01-17 05:01:06 -0800 (Wed, 17 Jan 2018) | 7 lines [X86] Don't mutate shuffle arguments after early-out for AVX512 The match* functions have the annoying behavior of modifying its inputs. Save and restore the inputs, just in case the early out for AVX512 is hit. This is still not great and its only a matter of time this kind of bug happens again, but I couldn't come up with a better pattern without rewriting significant chunks of this code. Fixes PR35977. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@322840 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-18Merging r322724:Hans Wennborg
------------------------------------------------------------------------ r322724 | ctopper | 2018-01-17 10:46:01 -0800 (Wed, 17 Jan 2018) | 7 lines [X86] When legalizing (v64i1 select i8, v64i1, v64i1) make sure not to introduce bitcasts to i64 in 32-bit mode We legalize selects of masks with scalar conditions using a bitcast to an integer type. But if we are in 32-bit mode we can't convert v64i1 to i64. So instead split the v64i1 to v32i1 and concat it back together. Each half will then be legalized by bitcasting to i32 which is fine. The test case is a little indirect. If we have the v64i1 select in IR it will get legalized by legalize vector ops which has a run of type legalization after it. That type legalization run is able to fix this i64 bitcast. So in order to avoid that we need a build_vector of a splat which legalize vector ops will ignore. Legalize DAG will then turn that into a select via LowerBUILD_VECTORvXi1. And the select will get legalized. In this case there is no type legalizer run to cleanup the bitcast. This fixes pr35972. ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@322835 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-17Merging r322106:Hans Wennborg
------------------------------------------------------------------------ r322106 | abataev | 2018-01-09 11:08:22 -0800 (Tue, 09 Jan 2018) | 11 lines [COST]Fix PR35865: Fix cost model evaluation for shuffle on X86. Summary: If the vector type is transformed to non-vector single type, the compile may crash trying to get vector information about non-vector type. Reviewers: RKSimon, spatel, mkuper, hfinkel Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D41862 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@322680 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-17Merging r322272:Hans Wennborg
------------------------------------------------------------------------ r322272 | zvi | 2018-01-11 04:26:52 -0800 (Thu, 11 Jan 2018) | 15 lines X86: Fix LowerBUILD_VECTORAsVariablePermute for case Src is smaller than Indices Summary: As RKSimon suggested in pr35820, in the case that Src is smaller in bit-size than Indices, need to widen Src to avoid type mismatch. Fixes pr35820 Reviewers: RKSimon, craig.topper Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D41865 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@322679 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-17Merging r322623:Hans Wennborg
------------------------------------------------------------------------ r322623 | avt77 | 2018-01-17 02:12:06 -0800 (Wed, 17 Jan 2018) | 3 lines Allow usage of X86-prefixes as separate instrs. Differential Revision: https://reviews.llvm.org/D42102 ------------------------------------------------------------------------ git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@322654 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-03Remove left-over debug printout from r321692Hans Wennborg
Besides the unsightly print-out, it was causing some buildbots to fail, e.g. http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/9311 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321711 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-03[ARM][NFC] Avoid recreating MCSubtargetInfo in ARMAsmBackendAlex Bradbury
After D41349, we can now directly access MCSubtargetInfo from createARM*AsmBackend. This patch makes use of this, avoiding the need to create a fresh MCSubtargetInfo (which was previously always done with a blank CPU and feature string). Given the total size of the change remains pretty tiny and we're removing the old explicit destructor, I changed the STI field to a reference rather than a pointer. Differential Revision: https://reviews.llvm.org/D41693 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321707 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-03[AArch64][SVE] Asm: Add restricted register classes for SVE predicate vectors.Sander de Smalen
Summary: Add a register class for SVE predicate operands that can only be p0-p7 (as opposed to p0-p15) Patch [1/3] in a series to add predicated ADD/SUB instructions for SVE. Reviewers: rengolin, mcrosier, evandro, fhahn, echristo, olista01, SjoerdMeijer, javed.absar Reviewed By: fhahn Subscribers: aemerson, javed.absar, tschuett, kristof.beyls, llvm-commits Differential Revision: https://reviews.llvm.org/D41441 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321699 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-15 21:59:25 +0000