[IfConversion] Maintain the CFG when predicating/merging blocks in IfConvert*

Summary:
This fixes PR32721 in IfConvertTriangle and possible similar problems in
IfConvertSimple, IfConvertDiamond and IfConvertForkedDiamond.

In PR32721 we had a triangle

   EBB
   | \
   |  |
   | TBB
   |  /
   FBB

where FBB didn't have any successors at all since it ended with an
unconditional return. Then TBB and FBB were be merged into EBB, but EBB
would still keep its successors, and the use of analyzeBranch and
CorrectExtraCFGEdges wouldn't help to remove them since the return
instruction is not analyzable (at least not on ARM).

The edge updating code and branch probability updating code is now pushed
into MergeBlocks() which allows us to share the same update logic between
more callsites. This lets us remove several dependencies on analyzeBranch
and completely eliminate RemoveExtraEdges.

One thing that showed up with this patch was that IfConversion sometimes
left a successor with 0% probability even if there was no branch or
fallthrough to the successor.

One such example from the test case ifcvt_bad_zero_prob_succ.mir. The
indirect branch tBRIND can only jump to bb.1, but without the patch we
got:

  bb.0:
    successors: %bb.1(0x80000000)

  bb.1:
    successors: %bb.1(0x80000000), %bb.2(0x00000000)
    tBRIND %r1, 1, %cpsr
    B %bb.1

  bb.2:

There is no way to jump from bb.1 to bb2, but still there is a 0% edge
from bb.1 to bb.2.

With the patch applied we instead get the expected:

  bb.0:
    successors: %bb.1(0x80000000)

  bb.1:
    successors: %bb.1(0x80000000)
    tBRIND %r1, 1, %cpsr
    B %bb.1

Since bb.2 had no predecessor at all, it was removed.

Several testcases had to be updated due to this since the removed
successor made the "Branch Probability Basic Block Placement" pass
sometimes place blocks in a different order.

Finally added a couple of new test cases:

* PR32721_ifcvt_triangle_unanalyzable.mir:
  Regression test for the original problem dexcribed in PR 32721.

* ifcvt_triangleWoCvtToNextEdge.mir:
  Regression test for problem that caused a revert of my first attempt
  to solve PR 32721.

* ifcvt_simple_bad_zero_prob_succ.mir:
  Test case showing the problem where a wrong successor with 0% probability
  was previously left.

* ifcvt_[diamond|forked_diamond|simple]_unanalyzable.mir
  Very simple test cases for the simple and (forked) diamond cases
  involving unanalyzable branches that can be nice to have as a base if
  wanting to write more complicated tests.

Reviewers: iteratee, MatzeB, grosser, kparzysz

Reviewed By: kparzysz

Subscribers: kbarton, davide, aemerson, nemanjai, javed.absar, kristof.beyls, llvm-commits

Differential Revision: https://reviews.llvm.org/D34099

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@310697 91177308-0d34-0410-b5e6-96231b3b80d8

8 files changed, 102 insertions, 89 deletions

diff --git a/lib/CodeGen/IfConversion.cpp b/lib/CodeGen/IfConversion.cpp
index ff840536617..7d152b10753 100644
--- a/lib/CodeGen/IfConversion.cpp
+++ b/lib/CodeGen/IfConversion.cpp
@@ -242,7 +242,6 @@ namespace {
     void AnalyzeBlocks(MachineFunction &MF,
                        std::vector<std::unique_ptr<IfcvtToken>> &Tokens);
     void InvalidatePreds(MachineBasicBlock &MBB);
-    void RemoveExtraEdges(BBInfo &BBI);
     bool IfConvertSimple(BBInfo &BBI, IfcvtKind Kind);
     bool IfConvertTriangle(BBInfo &BBI, IfcvtKind Kind);
     bool IfConvertDiamondCommon(BBInfo &BBI, BBInfo &TrueBBI, BBInfo &FalseBBI,
@@ -1342,14 +1341,6 @@ static void InsertUncondBranch(MachineBasicBlock &MBB, MachineBasicBlock &ToMBB,
   TII->insertBranch(MBB, &ToMBB, nullptr, NoCond, dl);
 }
 
-/// Remove true / false edges if either / both are no longer successors.
-void IfConverter::RemoveExtraEdges(BBInfo &BBI) {
-  MachineBasicBlock *TBB = nullptr, *FBB = nullptr;
-  SmallVector<MachineOperand, 4> Cond;
-  if (!TII->analyzeBranch(*BBI.BB, TBB, FBB, Cond))
-    BBI.BB->CorrectExtraCFGEdges(TBB, FBB, !Cond.empty());
-}
-
 /// Behaves like LiveRegUnits::StepForward() but also adds implicit uses to all
 /// values defined in MI which are also live/used by MI.
 static void UpdatePredRedefs(MachineInstr &MI, LivePhysRegs &Redefs) {
@@ -1483,15 +1474,15 @@ bool IfConverter::IfConvertSimple(BBInfo &BBI, IfcvtKind Kind) {
     // the entry block.
     CopyAndPredicateBlock(BBI, *CvtBBI, Cond);
 
-    // RemoveExtraEdges won't work if the block has an unanalyzable branch, so
-    // explicitly remove CvtBBI as a successor.
+    // Keep the CFG updated.
     BBI.BB->removeSuccessor(&CvtMBB, true);
   } else {
     // Predicate the instructions in the true block.
     RemoveKills(CvtMBB.begin(), CvtMBB.end(), DontKill, *TRI);
     PredicateBlock(*CvtBBI, CvtMBB.end(), Cond);
 
-    // Merge converted block into entry block.
+    // Merge converted block into entry block. The BB to Cvt edge is removed
+    // by MergeBlocks.
     MergeBlocks(BBI, *CvtBBI);
   }
 
@@ -1512,8 +1503,6 @@ bool IfConverter::IfConvertSimple(BBInfo &BBI, IfcvtKind Kind) {
     IterIfcvt = false;
   }
 
-  RemoveExtraEdges(BBI);
-
   // Update block info. BB can be iteratively if-converted.
   if (!IterIfcvt)
     BBI.IsDone = true;
@@ -1599,10 +1588,6 @@ bool IfConverter::IfConvertTriangle(BBInfo &BBI, IfcvtKind Kind) {
     // Copy instructions in the true block, predicate them, and add them to
     // the entry block.
     CopyAndPredicateBlock(BBI, *CvtBBI, Cond, true);
-
-    // RemoveExtraEdges won't work if the block has an unanalyzable branch, so
-    // explicitly remove CvtBBI as a successor.
-    BBI.BB->removeSuccessor(&CvtMBB, true);
   } else {
     // Predicate the 'true' block after removing its branch.
     CvtBBI->NonPredSize -= TII->removeBranch(CvtMBB);
@@ -1612,6 +1597,9 @@ bool IfConverter::IfConvertTriangle(BBInfo &BBI, IfcvtKind Kind) {
     MergeBlocks(BBI, *CvtBBI, false);
   }
 
+  // Keep the CFG updated.
+  BBI.BB->removeSuccessor(&CvtMBB, true);
+
   // If 'true' block has a 'false' successor, add an exit branch to it.
   if (HasEarlyExit) {
     SmallVector<MachineOperand, 4> RevCond(CvtBBI->BrCond.begin(),
@@ -1659,8 +1647,6 @@ bool IfConverter::IfConvertTriangle(BBInfo &BBI, IfcvtKind Kind) {
     IterIfcvt = false;
   }
 
-  RemoveExtraEdges(BBI);
-
   // Update block info. BB can be iteratively if-converted.
   if (!IterIfcvt)
     BBI.IsDone = true;
@@ -1923,8 +1909,6 @@ bool IfConverter::IfConvertForkedDiamond(
   TII->insertBranch(*BBI.BB, TrueBBI.TrueBB, TrueBBI.FalseBB,
                     TrueBBI.BrCond, dl);
 
-  RemoveExtraEdges(BBI);
-
   // Update block info.
   BBI.IsDone = TrueBBI.IsDone = FalseBBI.IsDone = true;
   InvalidatePreds(*BBI.BB);
@@ -1961,6 +1945,11 @@ bool IfConverter::IfConvertDiamond(BBInfo &BBI, IfcvtKind Kind,
   // fold the tail block in as well. Otherwise, unless it falls through to the
   // tail, add a unconditional branch to it.
   if (TailBB) {
+    // We need to remove the edges to the true and false blocks manually since
+    // we didn't let IfConvertDiamondCommon update the CFG.
+    BBI.BB->removeSuccessor(TrueBBI.BB);
+    BBI.BB->removeSuccessor(FalseBBI.BB, true);
+
     BBInfo &TailBBI = BBAnalysis[TailBB->getNumber()];
     bool CanMergeTail = !TailBBI.HasFallThrough &&
       !TailBBI.BB->hasAddressTaken();
@@ -1990,13 +1979,6 @@ bool IfConverter::IfConvertDiamond(BBInfo &BBI, IfcvtKind Kind,
     }
   }
 
-  // RemoveExtraEdges won't work if the block has an unanalyzable branch,
-  // which can happen here if TailBB is unanalyzable and is merged, so
-  // explicitly remove BBI1 and BBI2 as successors.
-  BBI.BB->removeSuccessor(TrueBBI.BB);
-  BBI.BB->removeSuccessor(FalseBBI.BB, /* NormalizeSuccessProbs */ true);
-  RemoveExtraEdges(BBI);
-
   // Update block info.
   BBI.IsDone = TrueBBI.IsDone = FalseBBI.IsDone = true;
   InvalidatePreds(*BBI.BB);
@@ -2133,7 +2115,8 @@ void IfConverter::CopyAndPredicateBlock(BBInfo &ToBBI, BBInfo &FromBBI,
 /// Move all instructions from FromBB to the end of ToBB.  This will leave
 /// FromBB as an empty block, so remove all of its successor edges except for
 /// the fall-through edge.  If AddEdges is true, i.e., when FromBBI's branch is
-/// being moved, add those successor edges to ToBBI.
+/// being moved, add those successor edges to ToBBI and remove the old edge
+/// from ToBBI to FromBBI.
 void IfConverter::MergeBlocks(BBInfo &ToBBI, BBInfo &FromBBI, bool AddEdges) {
   MachineBasicBlock &FromMBB = *FromBBI.BB;
   assert(!FromMBB.hasAddressTaken() &&
@@ -2165,12 +2148,10 @@ void IfConverter::MergeBlocks(BBInfo &ToBBI, BBInfo &FromBBI, bool AddEdges) {
   // AddEdges is true and FromMBB is a successor of ToBBI.BB.
   auto To2FromProb = BranchProbability::getZero();
   if (AddEdges && ToBBI.BB->isSuccessor(&FromMBB)) {
+    // Remove the old edge but remember the edge probability so we can calculate
+    // the correct weights on the new edges being added further down.
     To2FromProb = MBPI->getEdgeProbability(ToBBI.BB, &FromMBB);
-    // Set the edge probability from ToBBI.BB to FromMBB to zero to avoid the
-    // edge probability being merged to other edges when this edge is removed
-    // later.
-    ToBBI.BB->setSuccProbability(find(ToBBI.BB->successors(), &FromMBB),
-                                 BranchProbability::getZero());
+    ToBBI.BB->removeSuccessor(&FromMBB);
   }
 
   for (MachineBasicBlock *Succ : FromSuccs) {
@@ -2229,9 +2210,11 @@ void IfConverter::MergeBlocks(BBInfo &ToBBI, BBInfo &FromBBI, bool AddEdges) {
     }
   }
 
-  // Now FromBBI always falls through to the next block!
-  if (NBB && !FromMBB.isSuccessor(NBB))
-    FromMBB.addSuccessor(NBB);
+  // Move the now empty FromMBB out of the way to the end of the function so
+  // it doesn't interfere with fallthrough checks done by canFallThroughTo().
+  MachineBasicBlock *Last = &*FromMBB.getParent()->rbegin();
+  if (Last != &FromMBB)
+    FromMBB.moveAfter(Last);
 
   // Normalize the probabilities of ToBBI.BB's successors with all adjustment
   // we've done above.
diff --git a/test/CodeGen/ARM/ifcvt-branch-weight.ll b/test/CodeGen/ARM/ifcvt-branch-weight.ll
index 0de039cde23..5c39d63fda1 100644
--- a/test/CodeGen/ARM/ifcvt-branch-weight.ll
+++ b/test/CodeGen/ARM/ifcvt-branch-weight.ll
@@ -19,7 +19,7 @@ bb:
   br i1 %9, label %return, label %bb2
 
 ; CHECK: BB#2: derived from LLVM BB %bb2
-; CHECK: Successors according to CFG: BB#3({{[0-9a-fx/= ]+}}50.00%) BB#4({{[0-9a-fx/= ]+}}50.00%)
+; CHECK: Successors according to CFG: BB#4({{[0-9a-fx/= ]+}}50.00%) BB#3({{[0-9a-fx/= ]+}}50.00%)
 
 bb2:
   %v10 = icmp eq i32 %3, 16
diff --git a/test/CodeGen/ARM/indirectbr-3.ll b/test/CodeGen/ARM/indirectbr-3.ll
index 1d8a24577f0..12666331600 100644
--- a/test/CodeGen/ARM/indirectbr-3.ll
+++ b/test/CodeGen/ARM/indirectbr-3.ll
@@ -10,9 +10,11 @@
 define i32 @preserve_blocks(i32 %x) {
 ; preserve_blocks:
 ; CHECK: Block address taken
-; CHECK: movs r0, #1
-; CHECK: Block address taken
+; CHECK: %ibt1
 ; CHECK: movs r0, #2
+; CHECK: Block address taken
+; CHECK: %ibt2
+; CHECK: movs r0, #1
 ; CHECK-NOT: Address of block that was removed by CodeGen
 
 ; Separate bug. There are no valid diamonds to if-convert in this file.
diff --git a/test/CodeGen/ARM/struct-byval-frame-index.ll b/test/CodeGen/ARM/struct-byval-frame-index.ll
index 52f70fe1e0f..68629d3581d 100644
--- a/test/CodeGen/ARM/struct-byval-frame-index.ll
+++ b/test/CodeGen/ARM/struct-byval-frame-index.ll
@@ -4,11 +4,21 @@
 ; generated.
 ; PR16393
 
+; We expect the spill to be generated in %if.end230 and the reloads in
+; %if.end249 and %for.body285.
+
 ; CHECK: set_stored_macroblock_parameters
+; CHECK: @ %if.end230
+; CHECK-NOT:@ %if.
+; CHECK-NOT:@ %for.
 ; CHECK: str r{{.*}}, [sp, [[SLOT:#[0-9]+]]] @ 4-byte Spill
-; CHECK: bl RestoreMVBlock8x8
-; CHECK: bl RestoreMVBlock8x8
-; CHECK: bl RestoreMVBlock8x8
+; CHECK: @ %if.end249
+; CHECK-NOT:@ %if.
+; CHECK-NOT:@ %for.
+; CHECK: ldr r{{.*}}, [sp, [[SLOT]]] @ 4-byte Reload
+; CHECK: @ %for.body285
+; CHECK-NOT:@ %if.
+; CHECK-NOT:@ %for.
 ; CHECK: ldr r{{.*}}, [sp, [[SLOT]]] @ 4-byte Reload
 
 target triple = "armv7l-unknown-linux-gnueabihf"
diff --git a/test/CodeGen/PowerPC/logic-ops-on-compares.ll b/test/CodeGen/PowerPC/logic-ops-on-compares.ll
index 7da978990ba..7996262dc8d 100644
--- a/test/CodeGen/PowerPC/logic-ops-on-compares.ll
+++ b/test/CodeGen/PowerPC/logic-ops-on-compares.ll
@@ -47,7 +47,6 @@ define void @neg_truncate_i32_eq(i32 *%ptr) {
 ; CHECK-NEXT:    rldicl. r3, r3, 0, 63
 ; CHECK-NEXT:    bclr 12, eq, 0
 ; CHECK-NEXT:  # BB#1: # %if.end29.thread136
-; CHECK-NEXT:  .LBB1_2: # %if.end29
 entry:
   %0 = load i32, i32* %ptr, align 4
   %rem17127 = and i32 %0, 1
@@ -106,7 +105,6 @@ define void @neg_truncate_i64_eq(i64 *%ptr) {
 ; CHECK-NEXT:    rldicl. r3, r3, 0, 63
 ; CHECK-NEXT:    bclr 12, eq, 0
 ; CHECK-NEXT:  # BB#1: # %if.end29.thread136
-; CHECK-NEXT:  .LBB3_2: # %if.end29
 entry:
   %0 = load i64, i64* %ptr, align 4
   %rem17127 = and i64 %0, 1
@@ -167,7 +165,6 @@ define void @neg_truncate_i64_ne(i64 *%ptr) {
 ; CHECK-NEXT:    andi. r3, r3, 1
 ; CHECK-NEXT:    bclr 12, gt, 0
 ; CHECK-NEXT:  # BB#1: # %if.end29.thread136
-; CHECK-NEXT:  .LBB5_2: # %if.end29
 entry:
   %0 = load i64, i64* %ptr, align 4
   %rem17127 = and i64 %0, 1
diff --git a/test/CodeGen/PowerPC/ppc-shrink-wrapping.ll b/test/CodeGen/PowerPC/ppc-shrink-wrapping.ll
index 10edefb2e21..2bf4b0722f9 100644
--- a/test/CodeGen/PowerPC/ppc-shrink-wrapping.ll
+++ b/test/CodeGen/PowerPC/ppc-shrink-wrapping.ll
@@ -591,7 +591,9 @@ if.end:
 
 ; Another infinite loop test this time with two nested infinite loop.
 ; CHECK-LABEL: infiniteloop3
-; CHECK: # %end
+; CHECK: Lfunc_begin[[FUNCNUM:[0-9]+]]
+; CHECK: bclr
+; CHECK: Lfunc_end[[FUNCNUM]]
 define void @infiniteloop3() {
 entry:
   br i1 undef, label %loop2a, label %body
diff --git a/test/CodeGen/SystemZ/atomicrmw-minmax-03.ll b/test/CodeGen/SystemZ/atomicrmw-minmax-03.ll
index b53633a5e06..362a4f4ec49 100644
--- a/test/CodeGen/SystemZ/atomicrmw-minmax-03.ll
+++ b/test/CodeGen/SystemZ/atomicrmw-minmax-03.ll
@@ -7,13 +7,15 @@
 define i32 @f1(i32 %dummy, i32 *%src, i32 %b) {
 ; CHECK-LABEL: f1:
 ; CHECK: l %r2, 0(%r3)
-; CHECK: [[LOOP:\.[^:]*]]:
-; CHECK: lr [[NEW:%r[0-9]+]], %r2
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: cs %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
+; CHECK: [[LOOP]]:
+; CHECK: lr [[NEW]], %r2
 ; CHECK: crjle %r2, %r4, [[KEEP:\..*]]
 ; CHECK: lr [[NEW]], %r4
-; CHECK: cs %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw min i32 *%src, i32 %b seq_cst
   ret i32 %res
 }
@@ -22,13 +24,15 @@ define i32 @f1(i32 %dummy, i32 *%src, i32 %b) {
 define i32 @f2(i32 %dummy, i32 *%src, i32 %b) {
 ; CHECK-LABEL: f2:
 ; CHECK: l %r2, 0(%r3)
-; CHECK: [[LOOP:\.[^:]*]]:
-; CHECK: lr [[NEW:%r[0-9]+]], %r2
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: cs %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
+; CHECK: [[LOOP]]:
+; CHECK: lr [[NEW]], %r2
 ; CHECK: crjhe %r2, %r4, [[KEEP:\..*]]
 ; CHECK: lr [[NEW]], %r4
-; CHECK: cs %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw max i32 *%src, i32 %b seq_cst
   ret i32 %res
 }
@@ -37,13 +41,15 @@ define i32 @f2(i32 %dummy, i32 *%src, i32 %b) {
 define i32 @f3(i32 %dummy, i32 *%src, i32 %b) {
 ; CHECK-LABEL: f3:
 ; CHECK: l %r2, 0(%r3)
-; CHECK: [[LOOP:\.[^:]*]]:
-; CHECK: lr [[NEW:%r[0-9]+]], %r2
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: cs %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
+; CHECK: [[LOOP]]:
+; CHECK: lr [[NEW]], %r2
 ; CHECK: clrjle %r2, %r4, [[KEEP:\..*]]
 ; CHECK: lr [[NEW]], %r4
-; CHECK: cs %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw umin i32 *%src, i32 %b seq_cst
   ret i32 %res
 }
@@ -52,13 +58,15 @@ define i32 @f3(i32 %dummy, i32 *%src, i32 %b) {
 define i32 @f4(i32 %dummy, i32 *%src, i32 %b) {
 ; CHECK-LABEL: f4:
 ; CHECK: l %r2, 0(%r3)
-; CHECK: [[LOOP:\.[^:]*]]:
-; CHECK: lr [[NEW:%r[0-9]+]], %r2
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: cs %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
+; CHECK: [[LOOP]]:
+; CHECK: lr [[NEW]], %r2
 ; CHECK: clrjhe %r2, %r4, [[KEEP:\..*]]
 ; CHECK: lr [[NEW]], %r4
-; CHECK: cs %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw umax i32 *%src, i32 %b seq_cst
   ret i32 %res
 }
@@ -159,14 +167,15 @@ define i32 @f12(i32 %dummy, i64 %base, i64 %index, i32 %b) {
 define i32 @f13(i32 %dummy, i32 *%ptr) {
 ; CHECK-LABEL: f13:
 ; CHECK: lhi [[LIMIT:%r[0-9]+]], 42
-; CHECK: l %r2, 0(%r3)
-; CHECK: [[LOOP:\.[^:]*]]:
-; CHECK: lr [[NEW:%r[0-9]+]], %r2
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: cs %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
+; CHECK: [[LOOP]]:
+; CHECK: lr [[NEW]], %r2
 ; CHECK: crjle %r2, [[LIMIT]], [[KEEP:\..*]]
 ; CHECK: lhi [[NEW]], 42
-; CHECK: cs %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw min i32 *%ptr, i32 42 seq_cst
   ret i32 %res
 }
diff --git a/test/CodeGen/SystemZ/atomicrmw-minmax-04.ll b/test/CodeGen/SystemZ/atomicrmw-minmax-04.ll
index 444dc915c0f..5e43edf21bc 100644
--- a/test/CodeGen/SystemZ/atomicrmw-minmax-04.ll
+++ b/test/CodeGen/SystemZ/atomicrmw-minmax-04.ll
@@ -7,13 +7,15 @@
 define i64 @f1(i64 %dummy, i64 *%src, i64 %b) {
 ; CHECK-LABEL: f1:
 ; CHECK: lg %r2, 0(%r3)
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: csg %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
 ; CHECK: [[LOOP:\.[^:]*]]:
 ; CHECK: lgr [[NEW:%r[0-9]+]], %r2
 ; CHECK: cgrjle %r2, %r4, [[KEEP:\..*]]
 ; CHECK: lgr [[NEW]], %r4
-; CHECK: csg %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw min i64 *%src, i64 %b seq_cst
   ret i64 %res
 }
@@ -22,13 +24,15 @@ define i64 @f1(i64 %dummy, i64 *%src, i64 %b) {
 define i64 @f2(i64 %dummy, i64 *%src, i64 %b) {
 ; CHECK-LABEL: f2:
 ; CHECK: lg %r2, 0(%r3)
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: csg %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
 ; CHECK: [[LOOP:\.[^:]*]]:
 ; CHECK: lgr [[NEW:%r[0-9]+]], %r2
 ; CHECK: cgrjhe %r2, %r4, [[KEEP:\..*]]
 ; CHECK: lgr [[NEW]], %r4
-; CHECK: csg %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw max i64 *%src, i64 %b seq_cst
   ret i64 %res
 }
@@ -37,13 +41,15 @@ define i64 @f2(i64 %dummy, i64 *%src, i64 %b) {
 define i64 @f3(i64 %dummy, i64 *%src, i64 %b) {
 ; CHECK-LABEL: f3:
 ; CHECK: lg %r2, 0(%r3)
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: csg %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
 ; CHECK: [[LOOP:\.[^:]*]]:
 ; CHECK: lgr [[NEW:%r[0-9]+]], %r2
 ; CHECK: clgrjle %r2, %r4, [[KEEP:\..*]]
 ; CHECK: lgr [[NEW]], %r4
-; CHECK: csg %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw umin i64 *%src, i64 %b seq_cst
   ret i64 %res
 }
@@ -52,13 +58,15 @@ define i64 @f3(i64 %dummy, i64 *%src, i64 %b) {
 define i64 @f4(i64 %dummy, i64 *%src, i64 %b) {
 ; CHECK-LABEL: f4:
 ; CHECK: lg %r2, 0(%r3)
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: csg %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
 ; CHECK: [[LOOP:\.[^:]*]]:
 ; CHECK: lgr [[NEW:%r[0-9]+]], %r2
 ; CHECK: clgrjhe %r2, %r4, [[KEEP:\..*]]
 ; CHECK: lgr [[NEW]], %r4
-; CHECK: csg %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw umax i64 *%src, i64 %b seq_cst
   ret i64 %res
 }
@@ -127,13 +135,15 @@ define i64 @f10(i64 %dummy, i64 *%ptr) {
 ; CHECK-LABEL: f10:
 ; CHECK: lghi [[LIMIT:%r[0-9]+]], 42
 ; CHECK: lg %r2, 0(%r3)
+; CHECK: j [[LOOP:\.[^:]*]]
+; CHECK: [[BB1:\.[^:]*]]:
+; CHECK: csg %r2, [[NEW:%r[0-9]+]], 0(%r3)
+; CHECK: ber %r14
 ; CHECK: [[LOOP:\.[^:]*]]:
 ; CHECK: lgr [[NEW:%r[0-9]+]], %r2
 ; CHECK: cgrjle %r2, [[LIMIT]], [[KEEP:\..*]]
 ; CHECK: lghi [[NEW]], 42
-; CHECK: csg %r2, [[NEW]], 0(%r3)
-; CHECK: ber %r14
-; CHECK: j [[LOOP]]
+; CHECK: j [[BB1]]
   %res = atomicrmw min i64 *%ptr, i64 42 seq_cst
   ret i64 %res
 }