8283091: Support type conversion between different data sizes in SLP

After JDK-8275317, C2's SLP vectorizer has supported type conversion
between the same data size. We can also support conversions between
different data sizes like:
int <-> double
float <-> long
int <-> long
float <-> double

A typical test case:

int[] a;
double[] b;
for (int i = start; i < limit; i++) {
    b[i] = (double) a[i];
}

Our expected OptoAssembly code for one iteration is like below:

add R12, R2, R11, LShiftL #2
vector_load   V16,[R12, #16]
vectorcast_i2d  V16, V16  # convert I to D vector
add R11, R1, R11, LShiftL #3	# ptr
add R13, R11, #16	# ptr
vector_store [R13], V16

To enable the vectorization, the patch solves the following problems
in the SLP.

There are three main operations in the case above, LoadI, ConvI2D and
StoreD. Assuming that the vector length is 128 bits, how many scalar
nodes should be packed together to a vector? If we decide it
separately for each operation node, like what we did before the patch
in SuperWord::combine_packs(), a 128-bit vector will support 4 LoadI
or 2 ConvI2D or 2 StoreD nodes. However, if we put these packed nodes
in a vector node sequence, like loading 4 elements to a vector, then
typecasting 2 elements and lastly storing these 2 elements, they become
invalid. As a result, we should look through the whole def-use chain
and then pick up the minimum of these element sizes, like function
SuperWord::max_vector_size_in_ud_chain() do in the superword.cpp.
In this case, we pack 2 LoadI, 2 ConvI2D and 2 StoreD nodes, and then
generate valid vector node sequence, like loading 2 elements,
converting the 2 elements to another type and storing the 2 elements
with new type.

After this, LoadI nodes don't make full use of the whole vector and
only occupy part of it. So we adapt the code in
SuperWord::get_vw_bytes_special() to the situation.

In SLP, we calculate a kind of alignment as position trace for each
scalar node in the whole vector. In this case, the alignments for 2
LoadI nodes are 0, 4 while the alignment for 2 ConvI2D nodes are 0, 8.
Sometimes, 4 for LoadI and 8 for ConvI2D work the same, both of which
mark that this node is the second node in the whole vector, while the
difference between 4 and 8 are just because of their own data sizes. In
this situation, we should try to remove the impact caused by different
data size in SLP. For example, in the stage of
SuperWord::extend_packlist(), while determining if it's potential to
pack a pair of def nodes in the function SuperWord::follow_use_defs(),
we remove the side effect of different data size by transforming the
target alignment from the use node. Because we believe that, assuming
that the vector length is 512 bits, if the ConvI2D use nodes have
alignments of 16-24 and their def nodes, LoadI, have alignments of 8-12,
these two LoadI nodes should be packed as a pair as well.

Similarly, when determining if the vectorization is profitable, type
conversion between different data size takes a type of one size and
produces a type of another size, hence the special checks on alignment
and size should be applied, like what we do in SuperWord::is_vector_use.

After solving these problems, we successfully implemented the
vectorization of type conversion between different data sizes.

Here is the test data on NEON:

Before the patch:
Benchmark              (length)  Mode  Cnt    Score   Error  Units
  VectorLoop.convertD2F       523  avgt   15  216.431 ± 0.131  ns/op
  VectorLoop.convertD2I       523  avgt   15  220.522 ± 0.311  ns/op
  VectorLoop.convertF2D       523  avgt   15  217.034 ± 0.292  ns/op
  VectorLoop.convertF2L       523  avgt   15  231.634 ± 1.881  ns/op
  VectorLoop.convertI2D       523  avgt   15  229.538 ± 0.095  ns/op
  VectorLoop.convertI2L       523  avgt   15  214.822 ± 0.131  ns/op
  VectorLoop.convertL2F       523  avgt   15  230.188 ± 0.217  ns/op
  VectorLoop.convertL2I       523  avgt   15  162.234 ± 0.235  ns/op

After the patch:
Benchmark              (length)  Mode  Cnt    Score    Error  Units
  VectorLoop.convertD2F       523  avgt   15  124.352 ±  1.079  ns/op
  VectorLoop.convertD2I       523  avgt   15  557.388 ±  8.166  ns/op
  VectorLoop.convertF2D       523  avgt   15  118.082 ±  4.026  ns/op
  VectorLoop.convertF2L       523  avgt   15  225.810 ± 11.180  ns/op
  VectorLoop.convertI2D       523  avgt   15  166.247 ±  0.120  ns/op
  VectorLoop.convertI2L       523  avgt   15  119.699 ±  2.925  ns/op
  VectorLoop.convertL2F       523  avgt   15  220.847 ±  0.053  ns/op
  VectorLoop.convertL2I       523  avgt   15  122.339 ±  2.738  ns/op

perf data on X86:
Before the patch:
Benchmark              (length)  Mode  Cnt    Score   Error  Units
  VectorLoop.convertD2F       523  avgt   15  279.466 ± 0.069  ns/op
  VectorLoop.convertD2I       523  avgt   15  551.009 ± 7.459  ns/op
  VectorLoop.convertF2D       523  avgt   15  276.066 ± 0.117  ns/op
  VectorLoop.convertF2L       523  avgt   15  545.108 ± 5.697  ns/op
  VectorLoop.convertI2D       523  avgt   15  745.303 ± 0.185  ns/op
  VectorLoop.convertI2L       523  avgt   15  260.878 ± 0.044  ns/op
  VectorLoop.convertL2F       523  avgt   15  502.016 ± 0.172  ns/op
  VectorLoop.convertL2I       523  avgt   15  261.654 ± 3.326  ns/op

After the patch:
Benchmark              (length)  Mode  Cnt    Score   Error  Units
  VectorLoop.convertD2F       523  avgt   15  106.975 ± 0.045  ns/op
  VectorLoop.convertD2I       523  avgt   15  546.866 ± 9.287  ns/op
  VectorLoop.convertF2D       523  avgt   15   82.414 ± 0.340  ns/op
  VectorLoop.convertF2L       523  avgt   15  542.235 ± 2.785  ns/op
  VectorLoop.convertI2D       523  avgt   15   92.966 ± 1.400  ns/op
  VectorLoop.convertI2L       523  avgt   15   79.960 ± 0.528  ns/op
  VectorLoop.convertL2F       523  avgt   15  504.712 ± 4.794  ns/op
  VectorLoop.convertL2I       523  avgt   15  129.753 ± 0.094  ns/op

perf data on AVX512:
Before the patch:
Benchmark              (length)  Mode  Cnt    Score   Error  Units
  VectorLoop.convertD2F       523  avgt   15  282.984 ± 4.022  ns/op
  VectorLoop.convertD2I       523  avgt   15  543.080 ± 3.873  ns/op
  VectorLoop.convertF2D       523  avgt   15  273.950 ± 0.131  ns/op
  VectorLoop.convertF2L       523  avgt   15  539.568 ± 2.747  ns/op
  VectorLoop.convertI2D       523  avgt   15  745.238 ± 0.069  ns/op
  VectorLoop.convertI2L       523  avgt   15  260.935 ± 0.169  ns/op
  VectorLoop.convertL2F       523  avgt   15  501.870 ± 0.359  ns/op
  VectorLoop.convertL2I       523  avgt   15  257.508 ± 0.174  ns/op

After the patch:
Benchmark              (length)  Mode  Cnt    Score   Error  Units
  VectorLoop.convertD2F       523  avgt   15   76.687 ± 0.530  ns/op
  VectorLoop.convertD2I       523  avgt   15  545.408 ± 4.657  ns/op
  VectorLoop.convertF2D       523  avgt   15  273.935 ± 0.099  ns/op
  VectorLoop.convertF2L       523  avgt   15  540.534 ± 3.032  ns/op
  VectorLoop.convertI2D       523  avgt   15  745.234 ± 0.053  ns/op
  VectorLoop.convertI2L       523  avgt   15  260.865 ± 0.104  ns/op
  VectorLoop.convertL2F       523  avgt   15   63.834 ± 4.777  ns/op
  VectorLoop.convertL2I       523  avgt   15   48.183 ± 0.990  ns/op

Change-Id: I93e60fd956547dad9204ceec90220145c58a72ef

Author: Faye Gao

src/hotspot/share/opto/superword.cpp

@@ -999,6 +999,12 @@ int SuperWord::get_vw_bytes_special(MemNode* s) {
     }
   }
 
+  // Check for special case where there is a type conversion between different data size.
+  int vectsize = max_vector_size_in_ud_chain(s);
+  if (vectsize < Matcher::max_vector_size(btype)) {
+    vw = MIN2(vectsize * type2aelembytes(btype), vw);
+  }
+
   return vw;
 }
 
@@ -1187,7 +1193,9 @@ bool SuperWord::stmts_can_pack(Node* s1, Node* s2, int align) {
   BasicType bt2 = velt_basic_type(s2);
   if(!is_java_primitive(bt1) || !is_java_primitive(bt2))
     return false;
-  if (Matcher::max_vector_size(bt1) < 2) {
+  BasicType longer_bt = longer_type_for_conversion(s1);
+  if (Matcher::max_vector_size(bt1) < 2 ||
+      (longer_bt != T_ILLEGAL && Matcher::max_vector_size(longer_bt) < 2)) {
     return false; // No vectors for this type
   }
 
@@ -1432,16 +1440,20 @@ bool SuperWord::follow_use_defs(Node_List* p) {
 
   if (s1->is_Load()) return false;
 
-  int align = alignment(s1);
-  NOT_PRODUCT(if(is_trace_alignment()) tty->print_cr("SuperWord::follow_use_defs: s1 %d, align %d", s1->_idx, align);)
+  NOT_PRODUCT(if(is_trace_alignment()) tty->print_cr("SuperWord::follow_use_defs: s1 %d, align %d", s1->_idx, alignment(s1));)
   bool changed = false;
   int start = s1->is_Store() ? MemNode::ValueIn   : 1;
   int end   = s1->is_Store() ? MemNode::ValueIn+1 : s1->req();
   for (int j = start; j < end; j++) {
+    int align = alignment(s1);
     Node* t1 = s1->in(j);
     Node* t2 = s2->in(j);
     if (!in_bb(t1) || !in_bb(t2))
       continue;
+    if (longer_type_for_conversion(s1) != T_ILLEGAL ||
+        longer_type_for_conversion(t1) != T_ILLEGAL) {
+      align = align / data_size(s1) * data_size(t1);
+    }
     if (stmts_can_pack(t1, t2, align)) {
       if (est_savings(t1, t2) >= 0) {
         Node_List* pair = new Node_List();
@@ -1485,12 +1497,18 @@ bool SuperWord::follow_def_uses(Node_List* p) {
       if (t2->Opcode() == Op_AddI && t2 == _lp->as_CountedLoop()->incr()) continue; // don't mess with the iv
       if (!opnd_positions_match(s1, t1, s2, t2))
         continue;
-      if (stmts_can_pack(t1, t2, align)) {
+      int adjusted_align = alignment(s1);
+      if (longer_type_for_conversion(s1) != T_ILLEGAL ||
+          longer_type_for_conversion(t1) != T_ILLEGAL) {
+        adjusted_align = adjusted_align / data_size(s1) * data_size(t1);
+      }
+      if (stmts_can_pack(t1, t2, adjusted_align)) {
         int my_savings = est_savings(t1, t2);
         if (my_savings > savings) {
           savings = my_savings;
           u1 = t1;
           u2 = t2;
+          align = adjusted_align;
         }
       }
     }
@@ -1683,8 +1701,7 @@ void SuperWord::combine_packs() {
   for (int i = 0; i < _packset.length(); i++) {
     Node_List* p1 = _packset.at(i);
     if (p1 != NULL) {
-      BasicType bt = velt_basic_type(p1->at(0));
-      uint max_vlen = Matcher::max_vector_size(bt); // Max elements in vector
+      uint max_vlen = max_vector_size_in_ud_chain(p1->at(0)); // Max elements in vector
       assert(is_power_of_2(max_vlen), "sanity");
       uint psize = p1->size();
       if (!is_power_of_2(psize)) {
@@ -2007,6 +2024,8 @@ bool SuperWord::implemented(Node_List* p) {
       } else {
         retValue = ReductionNode::implemented(opc, size, arith_type->basic_type());
       }
+    } else if (VectorNode::is_convert_opcode(opc)) {
+      retValue = VectorCastNode::implemented(opc, size, velt_basic_type(p0->in(1)), velt_basic_type(p0));
     } else {
       retValue = VectorNode::implemented(opc, size, velt_basic_type(p0));
     }
@@ -2558,12 +2577,11 @@ void SuperWord::output() {
         Node* in = vector_opd(p, 1);
         vn = VectorNode::make(opc, in, NULL, vlen, velt_basic_type(n));
         vlen_in_bytes = vn->as_Vector()->length_in_bytes();
-      } else if (opc == Op_ConvI2F || opc == Op_ConvL2D ||
-                 opc == Op_ConvF2I || opc == Op_ConvD2L) {
+      } else if (VectorNode::is_convert_opcode(opc)) {
         assert(n->req() == 2, "only one input expected");
         BasicType bt = velt_basic_type(n);
-        int vopc = VectorNode::opcode(opc, bt);
         Node* in = vector_opd(p, 1);
+        int vopc = VectorCastNode::opcode(in->bottom_type()->is_vect()->element_basic_type());
         vn = VectorCastNode::make(vopc, in, bt, vlen);
         vlen_in_bytes = vn->as_Vector()->length_in_bytes();
       } else if (is_cmov_pack(p)) {
@@ -2961,9 +2979,26 @@ bool SuperWord::is_vector_use(Node* use, int u_idx) {
     return true;
   }
 
-
   if (u_pk->size() != d_pk->size())
     return false;
+
+  if (longer_type_for_conversion(use) != T_ILLEGAL) {
+    // type conversion takes a type of a kind of size and produces a type of
+    // another size - hence the special checks on alignment and size.
+    for (uint i = 0; i < u_pk->size(); i++) {
+      Node* ui = u_pk->at(i);
+      Node* di = d_pk->at(i);
+      if (ui->in(u_idx) != di) {
+        return false;
+      }
+      if (alignment(ui) / type2aelembytes(velt_basic_type(ui)) !=
+          alignment(di) / type2aelembytes(velt_basic_type(di))) {
+        return false;
+      }
+    }
+    return true;
+  }
+
   for (uint i = 0; i < u_pk->size(); i++) {
     Node* ui = u_pk->at(i);
     Node* di = d_pk->at(i);
@@ -3196,6 +3231,63 @@ void SuperWord::compute_max_depth() {
   }
 }
 
+BasicType SuperWord::longer_type_for_conversion(Node* n) {
+  int opcode = n->Opcode();
+  switch (opcode) {
+    case Op_ConvD2I:
+    case Op_ConvI2D:
+    case Op_ConvF2D:
+    case Op_ConvD2F: return T_DOUBLE;
+    case Op_ConvF2L:
+    case Op_ConvL2F:
+    case Op_ConvL2I:
+    case Op_ConvI2L: return T_LONG;
+    case Op_ConvI2F: {
+      BasicType src_t = velt_basic_type(n->in(1));
+      if (src_t == T_BYTE || src_t == T_SHORT) {
+        return T_FLOAT;
+      }
+      return T_ILLEGAL;
+    }
+    case Op_ConvF2I: {
+      BasicType dst_t = velt_basic_type(n);
+      if (dst_t == T_BYTE || dst_t == T_SHORT) {
+        return T_FLOAT;
+      }
+      return T_ILLEGAL;
+    }
+  }
+  return T_ILLEGAL;
+}
+
+int SuperWord::max_vector_size_in_ud_chain(Node* n) {
+  BasicType bt = velt_basic_type(n);
+  BasicType vt = bt;
+
+  // find the longest type among def nodes.
+  uint start, end;
+  VectorNode::vector_operands(n, &start, &end);
+  for (uint i = start; i < end; ++i) {
+    Node* input = n->in(i);
+    if (!in_bb(input)) continue;
+    BasicType newt = longer_type_for_conversion(input);
+    vt = (newt == T_ILLEGAL) ? vt : newt;
+  }
+
+  // find the longest type among use nodes.
+  for (uint i = 0; i < n->outcnt(); ++i) {
+    Node* output = n->raw_out(i);
+    if (!in_bb(output)) continue;
+    BasicType newt = longer_type_for_conversion(output);
+    vt = (newt == T_ILLEGAL) ? vt : newt;
+  }
+
+  int max = Matcher::max_vector_size(vt);
+  // If now there is no vectors for the longest type, the nodes with the longest
+  // type in the def-use chain are not packed in SuperWord::stmts_can_pack.
+  return max < 2 ? Matcher::max_vector_size(bt) : max;
+}
+
 //-------------------------compute_vector_element_type-----------------------
 // Compute necessary vector element type for expressions
 // This propagates backwards a narrower integer type when the

src/hotspot/share/opto/superword.hpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2007, 2020, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2007, 2022, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * This code is free software; you can redistribute it and/or modify it
@@ -528,6 +528,10 @@ class SuperWord : public ResourceObj {
   void bb_insert_after(Node* n, int pos);
   // Compute max depth for expressions from beginning of block
   void compute_max_depth();
+  // Return the longer type for type-conversion node and return illegal type for other nodes.
+  BasicType longer_type_for_conversion(Node* n);
+  // Find the longest type in def-use chain for packed nodes, and then compute the max vector size.
+  int max_vector_size_in_ud_chain(Node* n);
   // Compute necessary vector element type for expressions
   void compute_vector_element_type();
   // Are s1 and s2 in a pack pair and ordered as s1,s2?

src/hotspot/share/opto/vectornode.cpp

@@ -226,15 +226,6 @@ int VectorNode::opcode(int sopc, BasicType bt) {
     return Op_StoreVector;
   case Op_MulAddS2I:
     return Op_MulAddVS2VI;
-  case Op_ConvI2F:
-    return Op_VectorCastI2X;
-  case Op_ConvL2D:
-    return Op_VectorCastL2X;
-  case Op_ConvF2I:
-    return Op_VectorCastF2X;
-  case Op_ConvD2L:
-    return Op_VectorCastD2X;
-
   default:
     return 0; // Unimplemented
   }
@@ -365,6 +356,26 @@ bool VectorNode::is_shift_opcode(int opc) {
   }
 }
 
+bool VectorNode::is_convert_opcode(int opc) {
+  switch (opc) {
+    case Op_ConvI2F:
+    case Op_ConvL2D:
+    case Op_ConvF2I:
+    case Op_ConvD2L:
+    case Op_ConvI2D:
+    case Op_ConvL2F:
+    case Op_ConvL2I:
+    case Op_ConvI2L:
+    case Op_ConvF2L:
+    case Op_ConvD2F:
+    case Op_ConvF2D:
+    case Op_ConvD2I:
+      return true;
+    default:
+      return false;
+  }
+}
+
 bool VectorNode::is_shift(Node* n) {
   return is_shift_opcode(n->Opcode());
 }
@@ -1118,11 +1129,21 @@ int VectorCastNode::opcode(BasicType bt, bool is_signed) {
     case T_FLOAT:  return Op_VectorCastF2X;
     case T_DOUBLE: return Op_VectorCastD2X;
     default:
-      assert(false, "unknown type: %s", type2name(bt));
+      assert(bt == T_CHAR || bt == T_BOOLEAN, "unknown type: %s", type2name(bt));
       return 0;
   }
 }
 
+bool VectorCastNode::implemented(int opc, uint vlen, BasicType src_type, BasicType dst_type) {
+  if (is_java_primitive(dst_type) &&
+      (vlen > 1) && is_power_of_2(vlen) &&
+      Matcher::vector_size_supported(dst_type, vlen)) {
+    int vopc = VectorCastNode::opcode(src_type);
+    return vopc > 0 && Matcher::match_rule_supported_vector(vopc, vlen, dst_type);
+  }
+  return false;
+}
+
 Node* VectorCastNode::Identity(PhaseGVN* phase) {
   if (!in(1)->is_top()) {
     BasicType  in_bt = in(1)->bottom_type()->is_vect()->element_basic_type();

src/hotspot/share/opto/vectornode.hpp

@@ -80,6 +80,7 @@ class VectorNode : public TypeNode {
   static VectorNode* make_mask_node(int vopc, Node* n1, Node* n2, uint vlen, BasicType bt);
 
   static bool is_shift_opcode(int opc);
+  static bool is_convert_opcode(int opc);
 
   static bool is_vshift_cnt_opcode(int opc);
 
@@ -1469,7 +1470,7 @@ class VectorCastNode : public VectorNode {
 
   static VectorCastNode* make(int vopc, Node* n1, BasicType bt, uint vlen);
   static int  opcode(BasicType bt, bool is_signed = true);
-  static bool implemented(BasicType bt, uint vlen);
+  static bool implemented(int opc, uint vlen, BasicType src_type, BasicType dst_type);
 
   virtual Node* Identity(PhaseGVN* phase);
 };