无语的index hint：手工分配哈希区，5小时不出结果，优化后20分钟

同事说,有个语句5个小时不出结果，叫我帮忙看看，于是叫同事发过来。不看不知道，一看吓一跳，3个表关联，强制使用了2个index hint，当中一个表9g，一个表67g，另一个小表40Mb。开发者，总以为走index就是快的，所以使用了index hint，终于导致走得很慢。
以下是同事发来的语句：
select /*+  parallel(t,4) index(a,IDX_COMMBASUBSHIST_1) index(b,IDX_COMMCMSERVHIST_1)*/
    1,
    t.DISC_ID,
    t.DISC_LEV,
    to_date(20140117082042, 'yyyymmddhh24miss'),
    t.MSINFO_ID,
    t.ORG_ID,
    t.SERV_ID,
    t.SUBS_ID,
    t.OBJ_GRP_ID,
    a.SUBS_CODE,
    a.SUBS_STAT,
    a.SUBS_STAT_REASON,
    a.SUBS_STAT_DATE,
    a.ACTION_ID,
    a.ACTION_TYPE,
    a.ACTION_EX_TYPE,
    a.ACT_DATE,
    a.REQ_ID,
    a.STAFF_ID,
    a.CMMS_CUST_CODE,
    a.SPEED_VALUE,
    b.ACC_NBR,
    b.CUST_ID,
    b.SERV_NBR,
    b.CONSUME_GRADE,
    b.SERV_LEV,
    b.ACCOUNT_NBR,
    b.CITY_VILLAGE_ID,
    b.SERV_CHANNEL_ID,
    b.SERV_STAT_ID,
    b.CUST_CLASS_DL,
    b.CUST_TYPE_ID,
    b.USER_TYPE,
    b.USER_CHAR,
    b.PAYMENT_TYPE,
    b.BILLING_TYPE,
    b.PROD_ID,
    b.PROD_CAT_ID,
    b.EXCHANGE_ID,
    b.SERV_COL1,
    b.SERV_COL2,
    b.AREA_ID,
    b.SUBST_ID,
    b.BRANCH_ID,
    b.STOP_TYPE,
    b.CUST_MANAGER_ID,
    b.CREATE_DATE,
    b.ADDRESS_ID,
    b.SUBS_DATE,
    b.OPEN_DATE,
    b.MODI_STAFF_ID,
    b.CMMS_CUST_ID,
    b.CUST_NAME,
    b.SALES_ID,
    b.SALES_TYPE_ID,
    b.SERV_ADDR_ID,
    t.HIST_CREATE_DATE,
    b.ARREAR_MONTH,
    b.ARREAR_MONTH_LAST,
    t.SALESTAFF_ID,
    t.EHOME_TYPE,
    t.EHOME_CLASS,
    b.strat_grp_dl,
    b.sale_org1,
    b.sale_org2,
    b.sale_org3,
    b.location_type,
    b.region_flag,
    b.terminal_id,
    b.pstn_id,
    b.fee_id,
    b.payment_id,
    b.billing_id,
    b.strat_grp_xl,
    b.fld1,
    b.fld3,
    b.cust_level,
    b.group_cust_type,
    b.cust_region,
    b.group_cust_grade,
    b.control_level,
    b.net_connect_type,
    b.trade_type_id,
    b.acc_nbr2,
    b.cdma_class_id,
    b.phone_number_id,
    b.develop_channel,
    b.online_time,
    t.wireless_type,
    b.new_serv_stat_id,
    b.is_phs_tk,
    b.serv_grp_type,
    b.state,
    t.cdma_disc_type,
    b.mix_disc,
    b.is_3g,
    t.add_disc_type,
    to_number(nvl(b.business_type, '-1')),
    nvl(t.label_num, -1),
    b.is_mix_prod,
    t.price_id,
    t.disc_item_id,
    b.STD_SUBST_ID,
    b.STD_BRANCH_ID,
    t.DISC_ITEM_ID_OP,
    t.PRICE_ID_OP,
    t.business_type,
    b.new_prod_id,
    b.BOARD_SUBST_ID,
    b.BOARD_BRANCH_ID
     from AAA  a,
          BBB  b,
          CCC t
    where a.subs_id = t.subs_id
      and b.serv_id = t.serv_id    

--同事说开销比較大。有450W。。以下是运行计划：

涉及的表大小：
OWNER   SEGMENT_NAME    SEGMENT_TYPE    Size(Mb)
SUMMARY_SJZ_GZ  CCC   TABLE   40
SUMMARY_SJZ_GZ  BBB   TABLE PARTITION 9016.1875
SUMMARY_SJZ_GZ  AAA   TABLE PARTITION 67330.25    

下面是优化思路：
强制使用索引，导致当中9g的表走了index full scan，然后回表。由于除了index fast scan以外，其它索引扫描都是单块读，回表又是单块读。导致速度很慢。优化时考虑使用哈希连接，40Mb的小表作为驱动表，连接9g的表，最后连接超大的67G的表。
优化时使用的技术：
1.  use_hash（a，b），使用哈希表关联方式
2.  /*+parallel(a 5)*/;并行处理
3.  db_file_multiblock_read_count多块读參数设置为最大
4.  workarea_size_policy设置为手工管理
5.  sort_area_size设为接近最大
6.  hash_area_size设为接近最大    

alter session enable parallel dml;
alter session set workarea_size_policy=manual;
alter session set sort_area_size=2100000000;
alter session set hash_area_size=2100000000;
alter session set db_file_multiblock_read_count=128;    

select  /*+parallel(a,5) parallel(b,5) parallel(t,5) leading(t) use_hash(t,b) user_hash(b,a)*/
     1,
    t.DISC_ID,
    t.DISC_LEV,
    to_date(20140117082042, 'yyyymmddhh24miss'),
    t.MSINFO_ID,
    t.ORG_ID,
    t.SERV_ID,
    t.SUBS_ID,
    t.OBJ_GRP_ID,
    a.SUBS_CODE,
    a.SUBS_STAT,
    a.SUBS_STAT_REASON,
    a.SUBS_STAT_DATE,
    a.ACTION_ID,
    a.ACTION_TYPE,
    a.ACTION_EX_TYPE,
    a.ACT_DATE,
    a.REQ_ID,
    a.STAFF_ID,
    a.CMMS_CUST_CODE,
    a.SPEED_VALUE,
    b.ACC_NBR,
    b.CUST_ID,
    b.SERV_NBR,
    b.CONSUME_GRADE,
    b.SERV_LEV,
    b.ACCOUNT_NBR,
    b.CITY_VILLAGE_ID,
    b.SERV_CHANNEL_ID,
    b.SERV_STAT_ID,
    b.CUST_CLASS_DL,
    b.CUST_TYPE_ID,
    b.USER_TYPE,
    b.USER_CHAR,
    b.PAYMENT_TYPE,
    b.BILLING_TYPE,
    b.PROD_ID,
    b.PROD_CAT_ID,
    b.EXCHANGE_ID,
    b.SERV_COL1,
    b.SERV_COL2,
    b.AREA_ID,
    b.SUBST_ID,
    b.BRANCH_ID,
    b.STOP_TYPE,
    b.CUST_MANAGER_ID,
    b.CREATE_DATE,
    b.ADDRESS_ID,
    b.SUBS_DATE,
    b.OPEN_DATE,
    b.MODI_STAFF_ID,
    b.CMMS_CUST_ID,
    b.CUST_NAME,
    b.SALES_ID,
    b.SALES_TYPE_ID,
    b.SERV_ADDR_ID,
    t.HIST_CREATE_DATE,
    b.ARREAR_MONTH,
    b.ARREAR_MONTH_LAST,
    t.SALESTAFF_ID,
    t.EHOME_TYPE,
    t.EHOME_CLASS,
    b.strat_grp_dl,
    b.sale_org1,
    b.sale_org2,
    b.sale_org3,
    b.location_type,
    b.region_flag,
    b.terminal_id,
    b.pstn_id,
    b.fee_id,
    b.payment_id,
    b.billing_id,
    b.strat_grp_xl,
    b.fld1,
    b.fld3,
    b.cust_level,
    b.group_cust_type,
    b.cust_region,
    b.group_cust_grade,
    b.control_level,
    b.net_connect_type,
    b.trade_type_id,
    b.acc_nbr2,
    b.cdma_class_id,
    b.phone_number_id,
    b.develop_channel,
    b.online_time,
    t.wireless_type,
    b.new_serv_stat_id,
    b.is_phs_tk,
    b.serv_grp_type,
    b.state,
    t.cdma_disc_type,
    b.mix_disc,
    b.is_3g,
    t.add_disc_type,
    to_number(nvl(b.business_type, '-1')),
    nvl(t.label_num, -1),
    b.is_mix_prod,
    t.price_id,
    t.disc_item_id,
    b.STD_SUBST_ID,
    b.STD_BRANCH_ID,
    t.DISC_ITEM_ID_OP,
    t.PRICE_ID_OP,
    t.business_type,
    b.new_prod_id,
    b.BOARD_SUBST_ID,
    b.BOARD_BRANCH_ID
     from SUMMARY_SJZ_GZ.AAA  a,
          SUMMARY_SJZ_GZ.BBB  b,
          SUMMARY_SJZ_GZ.CCC t
    where a.subs_id = t.subs_id
      and b.serv_id = t.serv_id   ;  

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 382646192  

--------------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                  | Name                   | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT           |                        |   781K|   453M|   260K  (1)| 01:18:03 |       |       |        |      |            |
|   1 |  PX COORDINATOR            |                        |       |       |            |          |       |       |        |      |            |
|   2 |   PX SEND QC (RANDOM)      | :TQ10002               |   781K|   453M|   260K  (1)| 01:18:03 |       |       |  Q1,02 | P->S | QC (RAND)  |
|*  3 |    HASH JOIN               |                        |   781K|   453M|   260K  (1)| 01:18:03 |       |       |  Q1,02 | PCWP |            |
|   4 |     PX RECEIVE             |                        |   781K|   379M| 26771   (2)| 00:08:02 |       |       |  Q1,02 | PCWP |            |
|   5 |      PX SEND BROADCAST     | :TQ10001               |   781K|   379M| 26771   (2)| 00:08:02 |       |       |  Q1,01 | P->P | BROADCAST  |
|*  6 |       HASH JOIN            |                        |   781K|   379M| 26771   (2)| 00:08:02 |       |       |  Q1,01 | PCWP |            |
|   7 |        PX RECEIVE          |                        |   781K|    73M|   327   (2)| 00:00:06 |       |       |  Q1,01 | PCWP |            |
|   8 |         PX SEND BROADCAST  | :TQ10000               |   781K|    73M|   327   (2)| 00:00:06 |       |       |  Q1,00 | P->P | BROADCAST  |
|   9 |          PX BLOCK ITERATOR |                        |   781K|    73M|   327   (2)| 00:00:06 |       |       |  Q1,00 | PCWC |            |
|  10 |           TABLE ACCESS FULL| CCC                    |   781K|    73M|   327   (2)| 00:00:06 |       |       |  Q1,00 | PCWP |            |
|  11 |        PX BLOCK ITERATOR   |                        |    23M|  9096M| 26427   (2)| 00:07:56 |     1 |    17 |  Q1,01 | PCWC |            |
|  12 |         TABLE ACCESS FULL  | BBB                    |    23M|  9096M| 26427   (2)| 00:07:56 |     1 |    17 |  Q1,01 | PCWP |            |
|  13 |     PX BLOCK ITERATOR      |                        |   204M|    18G|   233K  (1)| 01:09:58 |     1 |    20 |  Q1,02 | PCWC |            |
|  14 |      TABLE ACCESS FULL     | AAA                    |   204M|    18G|   233K  (1)| 01:09:58 |     1 |    20 |  Q1,02 | PCWP |            |
--------------------------------------------------------------------------------------------------------------------------------------------------  

Predicate Information (identified by operation id):
---------------------------------------------------  

   3 - access("A"."SUBS_ID"="T"."SUBS_ID")
   6 - access("B"."SERV_ID"="T"."SERV_ID")  

--5小时不出结果，优化后终于20分钟不到就出结果。