聊聊flink Table的groupBy操作
本文主要研究一下flink Table的groupBy操作
Table.groupBy
flink-table_2.11-1.7.0-sources.jar!/org/apache/flink/table/api/table.scala
class Table(
private[flink] val tableEnv: TableEnvironment,
private[flink] val logicalPlan: LogicalNode) {
//......
def groupBy(fields: String):www.lezongyule.com GroupedTable = {
val fieldsExpr = ExpressionParser.parseExpressionList(fields)
groupBy(fieldsExpr: _*)
}
def groupBy(fields: Expression*): GroupedTable = {
new GroupedTable(this, fields)
}
//......
}
Table的groupBy操作支持两种参数，一种是String类型，一种是Expression类型；String参数的方法是将String转换为Expression，最后调用的Expression参数的groupBy方法，该方法创建了GroupedTable
GroupedTable
flink-table_2.11-1.7.0-sources.jar!/org/apache/flink/table/api/table.scala
class GroupedTable(
private[flink] val table: Table,
private[flink] val groupKey: Seq[Expression]) {
def select(fields: Expression*): Table = {
val expandedFields = expandProjectList(fields, table.logicalPlan, table.tableEnv)
val (aggNames, propNames) = extractAggregationsAndProperties(expandedFields, table.tableEnv)
if (propNames.nonEmpty) {
throw new ValidationException(www.mingheyl178.com/"Window properties can only be used on windowed tables.")
}
val projectsOnAgg = replaceAggregationsAndProperties(
expandedFields, table.tableEnv, aggNames, propNames)
val projectFields = extractFieldReferences(expandedFields ++ groupKey)
new Table(table.tableEnv,
Project(projectsOnAgg,
Aggregate(groupKey, aggNames.map(a www.dfgjpt.com/=> Alias(a._1, a._2)).toSeq,
Project(projectFields, table.logicalPlan).validate(table.tableEnv)
).validate(table.tableEnv)
).validate(table.tableEnv))
}
def select(fields: String): Table = {
val fieldExprs = ExpressionParser.parseExpressionList(fields)
//get the correct expression for AggFunctionCall
val withResolvedAggFunctionCall = fieldExprs.map(replaceAggFunctionCall(_, table.tableEnv))
select(withResolvedAggFunctionCall: _*)
}
}
GroupedTable有两个属性，一个是原始的Table，一个是Seq[Expression]类型的groupKey
GroupedTable提供两个select方法，参数类型分别为String、Expression，String类型的参数最后也是转为Expression类型
select方法使用Project创建新的Table，而Project则是通过Aggregate来创建
Aggregate
flink-table_2.11-1.7.0-sources.jar!/org/apache/flink/table/plan/logical/operators.scala
case class Aggregate(
groupingExpressions: Seq[Expression],
aggregateExpressions: Seq[NamedExpression],
child: LogicalNode) extends UnaryNode {
override def output: Seq[Attribute] = {
(groupingExpressions ++ aggregateExpressions) map {
case ne: NamedExpression => ne.toAttribute
case e => Alias(e, e.toString).toAttribute
}
}
override protected[logical] def construct(relBuilder: RelBuilder): RelBuilder = {
child.construct(relBuilder)
relBuilder.aggregate(
relBuilder.groupKey(groupingExpressions.map(_.toRexNode(relBuilder)).asJava),
aggregateExpressions.map {
case Alias(agg: Aggregation, name, _www.078881.cn) => agg.toAggCall(name)(relBuilder)
case _ => throw new RuntimeException("This should never happen.")
}.asJava)
}
override def validate(tableEnv: TableEnvironment): LogicalNode = {
implicit val relBuilder: RelBuilder = tableEnv.getRelBuilder
val resolvedAggregate = super.validate(tableEnv).asInstanceOf[Aggregate]
val groupingExprs = resolvedAggregate.groupingExpressions
val aggregateExprs = resolvedAggregate.aggregateExpressions
aggregateExprs.foreach(validateAggregateExpression)
groupingExprs.foreach(validateGroupingExpression)
def validateAggregateExpression(expr: Expression): Unit = expr match {
case distinctExpr: DistinctAgg =>
distinctExpr.child match {
case _: DistinctAgg => failValidation(
"Chained distinct operators are not supported!")
case aggExpr: Aggregation => validateAggregateExpression(aggExpr)
case _ => failValidation(
"Distinct operator can only be applied to aggregation expressions!")
}
// check aggregate function
case aggExpr: Aggregation
if aggExpr.getSqlAggFunction.requiresOver =>
failValidation(s"OVER clause is necessary for window functions: [${aggExpr.getClass}].")
// check no nested aggregation exists.
case aggExpr: Aggregation =>
aggExpr.children.foreach {www.fengshen157.com/ child =>
child.preOrderVisit {
case agg: Aggregation =>
failValidation(
"It's not allowed to use an aggregate function as " +
"input of another aggregate function")
case _ => // OK
}
}
case a: Attribute if !groupingExprs.exists(_.checkEquals(a)) =>
failValidation(
s"expression '$a' is invalid because it is neither" +
" present in group by nor an aggregate function")
case e if groupingExprs.exists(_.checkEquals(e)) => // OK
case e => e.children.foreach(validateAggregateExpression)
}
def validateGroupingExpression(expr: Expression): Unit = {
if (!expr.resultType.isKeyType) {
failValidation(
s"expression $expr cannot be used as a grouping expression " +
"because it's not a valid key type which must be hashable and comparable")
}
}
resolvedAggregate
}
}
Aggregate继承了UnaryNode，它接收三个参数，一个是Seq[Expression]类型的groupingExpressions，一个是Seq[NamedExpression]类型的aggregateExpressions，一个是LogicalNode类型的child；construct方法调用了relBuilder.aggregate，传入的RelBuilder.GroupKey参数是通过relBuilder.groupKey构建，而传入的RelBuilder.AggCall参数则是通过aggregateExpressions.map构造而来
RelBuilder.groupKey
calcite-core-1.18.0-sources.jar!/org/apache/calcite/tools/RelBuilder.java
public class RelBuilder {
protected final RelOptCluster cluster;
protected final RelOptSchema relOptSchema;
private final RelFactories.FilterFactory filterFactory;
private final RelFactories.ProjectFactory projectFactory;
private final RelFactories.AggregateFactory aggregateFactory;
private final RelFactories.SortFactory sortFactory;
private final RelFactories.ExchangeFactory exchangeFactory;
private final RelFactories.SortExchangeFactory sortExchangeFactory;
private final RelFactories.SetOpFactory setOpFactory;
private final RelFactories.JoinFactory joinFactory;
private final RelFactories.SemiJoinFactory semiJoinFactory;
private final RelFactories.CorrelateFactory correlateFactory;
private final RelFactories.ValuesFactory valuesFactory;
private final RelFactories.TableScanFactory scanFactory;
private final RelFactories.MatchFactory matchFactory;
private final Deque<Frame> stack = new ArrayDeque<>();
private final boolean simplify;
private final RexSimplify simplifier;
//......
/** Creates an empty group key. */
public GroupKey groupKey() {
return groupKey(ImmutableList.of());
}
/** Creates a group key. */
public GroupKey groupKey(RexNode... nodes) {
return groupKey(ImmutableList.copyOf(nodes));
}
/** Creates a group key. */
public GroupKey groupKey(Iterable<? extends RexNode> nodes) {
return new GroupKeyImpl(ImmutableList.copyOf(nodes), false, null, null);
}
/** Creates a group key with grouping sets. */
public GroupKey groupKey(Iterable<? extends RexNode> nodes,
Iterable<? extends Iterable<? extends RexNode>> nodeLists) {
return groupKey_(nodes, false, nodeLists);
}
/** Creates a group key of fields identified by ordinal. */
public GroupKey groupKey(int... fieldOrdinals) {
return groupKey(fields(ImmutableIntList.of(fieldOrdinals)));
}
/** Creates a group key of fields identified by name. */
public GroupKey groupKey(String... fieldNames) {
return groupKey(fields(ImmutableList.copyOf(fieldNames)));
}
public GroupKey groupKey(@Nonnull ImmutableBitSet groupSet) {
return groupKey(groupSet, ImmutableList.of(groupSet));
}
public GroupKey groupKey(ImmutableBitSet groupSet,
@Nonnull Iterable<? extends ImmutableBitSet> groupSets) {
return groupKey_(groupSet, false, ImmutableList.copyOf(groupSets));
}
private GroupKey groupKey_(ImmutableBitSet groupSet, boolean indicator,
@Nonnull ImmutableList<ImmutableBitSet> groupSets) {
if (groupSet.length() > peek().getRowType().getFieldCount()) {
throw new IllegalArgumentException("out of bounds: " + groupSet);
}
Objects.requireNonNull(groupSets);
final ImmutableList<RexNode> nodes =
fields(ImmutableIntList.of(groupSet.toArray()));
final List<ImmutableList<RexNode>> nodeLists =
Util.transform(groupSets,
bitSet -> fields(ImmutableIntList.of(bitSet.toArray())));
return groupKey_(nodes, indicator, nodeLists);
}
private GroupKey groupKey_(Iterable<? extends RexNode> nodes,
boolean indicator,
Iterable<? extends Iterable<? extends RexNode>> nodeLists) {
final ImmutableList.Builder<ImmutableList<RexNode>> builder =
ImmutableList.builder();
for (Iterable<? extends RexNode> nodeList : nodeLists) {
builder.add(ImmutableList.copyOf(nodeList));
}
return new GroupKeyImpl(ImmutableList.copyOf(nodes), indicator, builder.build(), null);
}
//......
}
RelBuilder提供了诸多groupKey方法用于创建GroupKey，其最后调用的是私有方法groupKey_，该方法创建了GroupKeyImpl
GroupKey
calcite-core-1.18.0-sources.jar!/org/apache/calcite/tools/RelBuilder.java
public interface GroupKey {
/** Assigns an alias to this group key.
*
* <p>Used to assign field names in the {@code group} operation. */
GroupKey alias(String alias);
}
/** Implementation of {@link GroupKey}. */
protected static class GroupKeyImpl implements GroupKey {
final ImmutableList<RexNode> nodes;
final boolean indicator;
final ImmutableList<ImmutableList<RexNode>> nodeLists;
final String alias;
GroupKeyImpl(ImmutableList<RexNode> nodes, boolean indicator,
ImmutableList<ImmutableList<RexNode>> nodeLists, String alias) {
this.nodes = Objects.requireNonNull(nodes);
assert !indicator;
this.indicator = indicator;
this.nodeLists = nodeLists;
this.alias = alias;
}
@Override public String toString() {
return alias == null ? nodes.toString() : nodes + " as " + alias;
}
public GroupKey alias(String alias) {
return Objects.equals(this.alias, alias)
? this
: new GroupKeyImpl(nodes, indicator, nodeLists, alias);
}
}
GroupKey接口定义了alias方法，用于给group操作的字段别名；GroupKeyImpl是GroupKey接口的实现类，其alias返回的是GroupKeyImpl
RelBuilder.aggregate
calcite-core-1.18.0-sources.jar!/org/apache/calcite/tools/RelBuilder.java
public class RelBuilder {
protected final RelOptCluster cluster;
protected final RelOptSchema relOptSchema;
private final RelFactories.FilterFactory filterFactory;
private final RelFactories.ProjectFactory projectFactory;
private final RelFactories.AggregateFactory aggregateFactory;
private final RelFactories.SortFactory sortFactory;
private final RelFactories.ExchangeFactory exchangeFactory;
private final RelFactories.SortExchangeFactory sortExchangeFactory;
private final RelFactories.SetOpFactory setOpFactory;
private final RelFactories.JoinFactory joinFactory;
private final RelFactories.SemiJoinFactory semiJoinFactory;
private final RelFactories.CorrelateFactory correlateFactory;
private final RelFactories.ValuesFactory valuesFactory;
private final RelFactories.TableScanFactory scanFactory;
private final RelFactories.MatchFactory matchFactory;
private final Deque<Frame> stack = new ArrayDeque<>();
private final boolean simplify;
private final RexSimplify simplifier;
//......
/** Creates an {@link Aggregate} with an array of
* calls. */
public RelBuilder aggregate(GroupKey groupKey, AggCall... aggCalls) {
return aggregate(groupKey, ImmutableList.copyOf(aggCalls));
}
public RelBuilder aggregate(GroupKey groupKey,
List<AggregateCall> aggregateCalls) {
return aggregate(groupKey,
Lists.transform(aggregateCalls, AggCallImpl2::new));
}
/** Creates an {@link Aggregate} with a list of
* calls. */
public RelBuilder aggregate(GroupKey groupKey, Iterable<AggCall> aggCalls) {
final Registrar registrar = new Registrar();
registrar.extraNodes.addAll(fields());
registrar.names.addAll(peek().getRowType().getFieldNames());
final GroupKeyImpl groupKey_ = (GroupKeyImpl) groupKey;
final ImmutableBitSet groupSet =
ImmutableBitSet.of(registrar.registerExpressions(groupKey_.nodes));
label:
if (Iterables.isEmpty(aggCalls) && !groupKey_.indicator) {
final RelMetadataQuery mq = peek().getCluster().getMetadataQuery();
if (groupSet.isEmpty()) {
final Double minRowCount = mq.getMinRowCount(peek());
if (minRowCount == null || minRowCount < 1D) {
// We can't remove "GROUP BY ()" if there's a chance the rel could be
// empty.
break label;
}
}
if (registrar.extraNodes.size() == fields().size()) {
final Boolean unique = mq.areColumnsUnique(peek(), groupSet);
if (unique != null && unique) {
// Rel is already unique.
return project(fields(groupSet.asList()));
}
}
final Double maxRowCount = mq.getMaxRowCount(peek());
if (maxRowCount != null && maxRowCount <= 1D) {
// If there is at most one row, rel is already unique.
return this;
}
}
final ImmutableList<ImmutableBitSet> groupSets;
if (groupKey_.nodeLists != null) {
final int sizeBefore = registrar.extraNodes.size();
final SortedSet<ImmutableBitSet> groupSetSet =
new TreeSet<>(ImmutableBitSet.ORDERING);
for (ImmutableList<RexNode> nodeList : groupKey_.nodeLists) {
final ImmutableBitSet groupSet2 =
ImmutableBitSet.of(registrar.registerExpressions(nodeList));
if (!groupSet.contains(groupSet2)) {
throw new IllegalArgumentException("group set element " + nodeList
+ " must be a subset of group key");
}
groupSetSet.add(groupSet2);
}
groupSets = ImmutableList.copyOf(groupSetSet);
if (registrar.extraNodes.size() > sizeBefore) {
throw new IllegalArgumentException(
"group sets contained expressions not in group key: "
+ registrar.extraNodes.subList(sizeBefore,
registrar.extraNodes.size()));
}
} else {
groupSets = ImmutableList.of(groupSet);
}
for (AggCall aggCall : aggCalls) {
if (aggCall instanceof AggCallImpl) {
final AggCallImpl aggCall1 = (AggCallImpl) aggCall;
registrar.registerExpressions(aggCall1.operands);
if (aggCall1.filter != null) {
registrar.registerExpression(aggCall1.filter);
}
}
}
project(registrar.extraNodes);
rename(registrar.names);
final Frame frame = stack.pop();
final RelNode r = frame.rel;
final List<AggregateCall> aggregateCalls = new ArrayList<>();
for (AggCall aggCall : aggCalls) {
final AggregateCall aggregateCall;
if (aggCall instanceof AggCallImpl) {
final AggCallImpl aggCall1 = (AggCallImpl) aggCall;
final List<Integer> args =
registrar.registerExpressions(aggCall1.operands);
final int filterArg = aggCall1.filter == null ? -1
: registrar.registerExpression(aggCall1.filter);
if (aggCall1.distinct && !aggCall1.aggFunction.isQuantifierAllowed()) {
throw new IllegalArgumentException("DISTINCT not allowed");
}
if (aggCall1.filter != null && !aggCall1.aggFunction.allowsFilter()) {
throw new IllegalArgumentException("FILTER not allowed");
}
RelCollation collation =
RelCollations.of(aggCall1.orderKeys
.stream()
.map(orderKey ->
collation(orderKey, RelFieldCollation.Direction.ASCENDING,
null, Collections.emptyList()))
.collect(Collectors.toList()));
aggregateCall =
AggregateCall.create(aggCall1.aggFunction, aggCall1.distinct,
aggCall1.approximate, args, filterArg, collation,
groupSet.cardinality(), r, null, aggCall1.alias);
} else {
aggregateCall = ((AggCallImpl2) aggCall).aggregateCall;
}
aggregateCalls.add(aggregateCall);
}
assert ImmutableBitSet.ORDERING.isStrictlyOrdered(groupSets) : groupSets;
for (ImmutableBitSet set : groupSets) {
assert groupSet.contains(set);
}
RelNode aggregate = aggregateFactory.createAggregate(r,
groupKey_.indicator, groupSet, groupSets, aggregateCalls);
// build field list
final ImmutableList.Builder<Field> fields = ImmutableList.builder();
final List<RelDataTypeField> aggregateFields =
aggregate.getRowType().getFieldList();
int i = 0;
// first, group fields
for (Integer groupField : groupSet.asList()) {
RexNode node = registrar.extraNodes.get(groupField);
final SqlKind kind = node.getKind();
switch (kind) {
case INPUT_REF:
fields.add(frame.fields.get(((RexInputRef) node).getIndex()));
break;
default:
String name = aggregateFields.get(i).getName();
RelDataTypeField fieldType =
new RelDataTypeFieldImpl(name, i, node.getType());
fields.add(new Field(ImmutableSet.of(), fieldType));
break;
}
i++;
}
// second, indicator fields (copy from aggregate rel type)
if (groupKey_.indicator) {
for (int j = 0; j < groupSet.cardinality(); ++j) {
final RelDataTypeField field = aggregateFields.get(i);
final RelDataTypeField fieldType =
new RelDataTypeFieldImpl(field.getName(), i, field.getType());
fields.add(new Field(ImmutableSet.of(), fieldType));
i++;
}
}
// third, aggregate fields. retain `i' as field index
for (int j = 0; j < aggregateCalls.size(); ++j) {
final AggregateCall call = aggregateCalls.get(j);
final RelDataTypeField fieldType =
new RelDataTypeFieldImpl(aggregateFields.get(i + j).getName(), i + j,
call.getType());
fields.add(new Field(ImmutableSet.of(), fieldType));
}
stack.push(new Frame(aggregate, fields.build()));
return this;
}
//......
}
RelBuilder的aggregate操作接收两个参数，一个是GroupKey，一个是集合类型的AggCall；其中AggCall最后是转换为AggregateCall，然后通过aggregateFactory.createAggregate方法取出stack队首的Frame，创建新的RelNode，构造新的Frame，然后重新放入stack的队首
RelFactories.AggregateFactory.createAggregate
calcite-core-1.18.0-sources.jar!/org/apache/calcite/rel/core/RelFactories.java
public class RelFactories {
//......
public static final AggregateFactory DEFAULT_AGGREGATE_FACTORY =
new AggregateFactoryImpl();
public interface AggregateFactory {
/** Creates an aggregate. */
RelNode createAggregate(RelNode input, boolean indicator,
ImmutableBitSet groupSet, ImmutableList<ImmutableBitSet> groupSets,
List<AggregateCall> aggCalls);
}
private static class AggregateFactoryImpl implements AggregateFactory {
@SuppressWarnings("deprecation")
public RelNode createAggregate(RelNode input, boolean indicator,
ImmutableBitSet groupSet, ImmutableList<ImmutableBitSet> groupSets,
List<AggregateCall> aggCalls) {
return LogicalAggregate.create(input, indicator,
groupSet, groupSets, aggCalls);
}
}
//......
}
RelFactories定义了AggregateFactory接口，该接口定义了createAggregate方法，用于将一系列的AggregateCall操作转为新的RelNode；AggregateFactoryImpl是AggregateFactory接口的实现类，它的createAggregate方法调用的是LogicalAggregate.create方法
LogicalAggregate.create
calcite-core-1.18.0-sources.jar!/org/apache/calcite/rel/logical/LogicalAggregate.java
public final class LogicalAggregate extends Aggregate {
//......
public static LogicalAggregate create(final RelNode input,
ImmutableBitSet groupSet, List<ImmutableBitSet> groupSets,
List<AggregateCall> aggCalls) {
return create_(input, false, groupSet, groupSets, aggCalls);
}
@Deprecated // to be removed before 2.0
public static LogicalAggregate create(final RelNode input,
boolean indicator,
ImmutableBitSet groupSet,
List<ImmutableBitSet> groupSets,
List<AggregateCall> aggCalls) {
return create_(input, indicator, groupSet, groupSets, aggCalls);
}
private static LogicalAggregate create_(final RelNode input,
boolean indicator,
ImmutableBitSet groupSet,
List<ImmutableBitSet> groupSets,
List<AggregateCall> aggCalls) {
final RelOptCluster cluster = input.getCluster();
final RelTraitSet traitSet = cluster.traitSetOf(Convention.NONE);
return new LogicalAggregate(cluster, traitSet, input, indicator, groupSet,
groupSets, aggCalls);
}
//......
}
LogicalAggregate的create方法创建的是LogicalAggregate
小结
Table的groupBy操作支持两种参数，一种是String类型，一种是Expression类型；String参数的方法是将String转换为Expression，最后调用的Expression参数的groupBy方法，该方法创建了GroupedTable
GroupedTable有两个属性，一个是原始的Table，一个是Seq[Expression]类型的groupKey；它提供两个select方法，参数类型分别为String、Expression，String类型的参数最后也是转为Expression类型；select方法使用Project创建新的Table，而Project则是通过Aggregate来创建
Aggregate继承了UnaryNode，它接收三个参数，一个是Seq[Expression]类型的groupingExpressions，一个是Seq[NamedExpression]类型的aggregateExpressions，一个是LogicalNode类型的child；construct方法调用了relBuilder.aggregate，传入的RelBuilder.GroupKey参数是通过relBuilder.groupKey构建，而传入的RelBuilder.AggCall参数则是通过aggregateExpressions.map构造而来
RelBuilder的aggregate操作接收两个参数，一个是GroupKey(GroupKey接口定义了alias方法，用于给group操作的字段别名；GroupKeyImpl是GroupKey接口的实现类，其alias返回的是GroupKeyImpl)，一个是集合类型的AggCall；其中AggCall最后是转换为AggregateCall，然后通过aggregateFactory.createAggregate方法取出stack队首的Frame，创建新的RelNode，构造新的Frame，然后重新放入stack的队首
RelFactories定义了AggregateFactory接口，该接口定义了createAggregate方法，用于将一系列的AggregateCall操作转为新的RelNode；AggregateFactoryImpl是AggregateFactory接口的实现类，它的createAggregate方法调用的是LogicalAggregate.create方法，创建的是LogicalAggregate
聊聊flink Table的groupBy操作的更多相关文章
- Flink table&Sql中使用Calcite
Apache Calcite是什么东东 Apache Calcite面向Hadoop新的sql引擎,它提供了标准的SQL语言.多种查询优化和连接各种数据源的能力.除此之外,Calcite还提供了OLA ...
- 使用flink Table &Sql api来构建批量和流式应用(2)Table API概述
从flink的官方文档,我们知道flink的编程模型分为四层,sql层是最高层的api,Table api是中间层,DataStream/DataSet Api 是核心,stateful Stream ...
- 使用flink Table &Sql api来构建批量和流式应用(3)Flink Sql 使用
从flink的官方文档,我们知道flink的编程模型分为四层,sql层是最高层的api,Table api是中间层,DataStream/DataSet Api 是核心,stateful Stream ...
- 【翻译】Flink Table Api & SQL —— 概念与通用API
本文翻译自官网:https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html Flink Tabl ...
- 【翻译】Flink Table Api & SQL —— Table API
本文翻译自官网:Table API https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/tableApi.ht ...
- 【翻译】Flink Table Api & SQL — SQL
本文翻译自官网:SQL https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/sql.html Flink Tab ...
- 【翻译】Flink Table Api & SQL — 内置函数
本文翻译自官网:Built-In Functions https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/fu ...
- 8、Flink Table API & Flink Sql API
一.概述 上图是flink的分层模型,Table API 和 SQL 处于最顶端,是 Flink 提供的高级 API 操作.Flink SQL 是 Flink 实时计算为简化计算模型,降低用户使用实时 ...
- Flink 报错 "Could not find a suitable table factory for 'org.apache.flink.table.factories.StreamTableSourceFactory' in the classpath"
先上代码: table = tablexx.select('*).tablexx.groupBy('x).select('x, xx.count ) tableEnvironment // decla ...
随机推荐
- JS中的eval函数
最近开始慢慢学习前端的脚本了,上次碰到了一个问题,需要通过一个对象的属性名称来获得这个对象这个属性的值.如果在C#中,那么直接通过反射就可以了.而在js中,也有类似的函数,那就是eval ...
- hive 日志
hive中日志分为两种: 1 系统日志,记录hive运行情况,错误状态 2 job日志 , 记录hive中 job执行的历史过程 系统日志存储位置: 配置在 hive/conf/hive-log4j. ...
- FTP如何通过Windows防火墙
将C:\Windows\SysWOW64\ftp.exe(64位系统)C:\Windows\system32\ftp.exe(32位系统)文件添加到防火墙的允许列表中即可. 注:“控制面板” -> ...
- Consul 架构（译）
Consul 架构 此篇文章主要对consul的相关内部技术细节进行简要概述. »术语 代理 - 代理是指consul集群中运行的consul实例,通过执行 consul agent 命令来启动. 代 ...
- 概念这种东西--node.js
概念是一个既简单又复杂.既招人爱又招人恨的东西.概念是对一事务或现象的抽象.抽象好了,那就太方便问题的解决了,抽象坏了,那就驴唇不对马嘴,反而会让逻辑一塌糊涂.现实中经常有这样的概念:东北人怎么怎么样 ...
- 1.6 JAVA高并发之线程池
一.JAVA高级并发 1.5JDK之后引入高级并发特性,大多数的特性在java.util.concurrent 包中,是专门用于多线程发编程的,充分利用了现代多处理器和多核心系统的功能以编写大规模并发 ...
- 3星|《十大全球CEO亲授企业高速成长的关键战略》:作为CEO，我也非常坦率地表明过家庭优先于工作
十大全球CEO亲授 企业高速成长的关键战略(<哈佛商业评论>增刊) <哈佛商业评论>上的10来篇文章合集.大部分都看过,除了一篇中信的访谈,其他大部分是美国的有点旧的案例. 总 ...
- 基于腾讯云CLB实现K8S v1.10.1集群高可用+负载均衡
概述: 最近对K8S非常感兴趣,同时对容器的管理等方面非常出色,是一款非常开源,强大的容器管理方案,最后经过1个月的本地实验,最终决定在腾讯云平台搭建属于我们的K8S集群管理平台~ 采购之后已经在本地 ...
- Mysql 单表主从同步
先配主从同步,后将主库表老数据传输到从库 说明:api-server的数据库为主,其他harbor为从 1.master 配置文件更改 [mysqld] log-bin = mysql-bin ser ...
- 微软职位内部推荐-Software Engineer II-Data Mining
微软近期Open的职位: Are you looking for a big challenge? Do you know why Big Data is the next frontier for ...