继上篇第一阶段清洗数据并导入hive

本篇是剩下的两阶段

2、数据处理：

·统计最受欢迎的视频/文章的Top10访问次数 （video/article）

·按照地市统计最受欢迎的Top10课程 （ip）

·按照流量统计最受欢迎的Top10课程 （traffic）

3、数据可视化：将统计结果倒入MySql数据库中，通过图形化展示的方式展现出来。

2、

·统计最受欢迎的视频/文章的Top10访问次数 （video/article）

package mapreduce;

import java.io.IOException;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.io.WritableComparable;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.Reducer.Context;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class GetVideoResult {

    public static void main(String[] args) {

        try {

            Job job = Job.getInstance();

            job.setJobName("GetVideoResult");

            job.setJarByClass(GetVideoResult.class);

            job.setMapperClass(doMapper.class);

            job.setReducerClass(doReducer.class);

            job.setOutputKeyClass(Text.class);

            job.setOutputValueClass(IntWritable.class);

            job.setInputFormatClass(TextInputFormat.class);

            job.setOutputFormatClass(TextOutputFormat.class);

            Path in = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out1/part-r-00000");

            Path out = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out1.2");

            FileInputFormat.addInputPath(job,in);

            FileOutputFormat.setOutputPath(job,out);

            //System.exit(job.waitForCompletion(true) ? 0:1);

            if(job.waitForCompletion(true))

            {

                Job job1 = Job.getInstance();

                job1.setJobName("Sort");

                job1.setJarByClass(GetVideoResult.class);

                job1.setMapperClass(doMapper1.class);

                job1.setReducerClass(doReducer1.class);

                job1.setOutputKeyClass(IntWritable.class);

                job1.setOutputValueClass(Text.class);

                job1.setSortComparatorClass(IntWritableDecreasingComparator.class);

                job1.setInputFormatClass(TextInputFormat.class);

                job1.setOutputFormatClass(TextOutputFormat.class);

                Path in1 = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out1.2/part-r-00000");

                Path out1 = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out1.3");

                FileInputFormat.addInputPath(job1,in1);

                FileOutputFormat.setOutputPath(job1,out1);

                System.exit(job1.waitForCompletion(true) ? 0:1);

            }

        } catch (Exception e) {

            e.printStackTrace();

        }

    }

    public static class doMapper extends Mapper<Object,Text,Text,IntWritable>{

        public static Text word = new Text();

        public static final IntWritable id = new IntWritable(1);

        @Override

        protected void map(Object key,Text value,Context context) throws IOException,InterruptedException{

            String[] data = value.toString().split("\t");

            word.set(data[5]);

            //id.set(Integer.parseInt(data[5]));

            context.write(word,id);

        }

    }

    public static class doReducer extends Reducer< Text, IntWritable, IntWritable, Text>{

        private static IntWritable result= new IntWritable();  

        public void reduce(Text key,Iterable<IntWritable> values,Context context) throws IOException, InterruptedException{

            int sum = 0;

            for(IntWritable value:values){

                sum += value.get();

            }

            result.set(sum);

            context.write(result,key);

            }

    }

    public static class doMapper1 extends Mapper<Object , Text , IntWritable,Text >{

        private static Text goods=new Text();

        private static IntWritable num=new IntWritable();

        public void map(Object key,Text value,Context context) throws IOException, InterruptedException{

            String line=value.toString();

            String arr[]=line.split("\t");

            num.set(Integer.parseInt(arr[0]));

            goods.set(arr[1]);

            context.write(num,goods);

        }

    }

    public static class doReducer1 extends Reducer< IntWritable, Text, IntWritable, Text>{

        private static IntWritable result= new IntWritable();

        int i=0;

        public void reduce(IntWritable key,Iterable<Text> values,Context context) throws IOException, InterruptedException{

            for(Text value:values){

                if(i<10) {

                context.write(key,value);

                i++;

                }

            }

            }

    }

    private static class IntWritableDecreasingComparator extends IntWritable.Comparator {

         public int compare(WritableComparable a, WritableComparable b) {

             return -super.compare(a, b);

          }

         public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {

                    return -super.compare(b1, s1, l1, b2, s2, l2);

           }

    }

}

自己一开始使用两个类完成的，先求和在排序，在网上查阅资料后发现可以有两个job，然后就在一个类中完成，然后MapReduce本来的排序是升序，而我们需要的是降序，所以在此引入了一个比较器。

按照地市统计最受欢迎的Top10课程 （ip）

package mapreduce;

import java.io.IOException;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.io.WritableComparable;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.Reducer.Context;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class GetVideoResultip {

    public static void main(String[] args) {

        try {

            Job job = Job.getInstance();

            job.setJobName("GetVideoResult");

            job.setJarByClass(GetVideoResultip.class);

            job.setMapperClass(doMapper.class);

            job.setReducerClass(doReducer.class);

            job.setOutputKeyClass(Text.class);

            job.setOutputValueClass(IntWritable.class);

            job.setInputFormatClass(TextInputFormat.class);

            job.setOutputFormatClass(TextOutputFormat.class);

            Path in = new Path("hdfs://192.168.137.67:9000/mymapreducel/in/result.txt");

            Path out = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out2.1");

            FileInputFormat.addInputPath(job,in);

            FileOutputFormat.setOutputPath(job,out);

            //System.exit(job.waitForCompletion(true) ? 0:1);

            if(job.waitForCompletion(true))

            {

                Job job1 = Job.getInstance();

                job1.setJobName("Sort");

                job1.setJarByClass(GetVideoResult.class);

                job1.setMapperClass(doMapper1.class);

                job1.setReducerClass(doReducer1.class);

                job1.setOutputKeyClass(IntWritable.class);

                job1.setOutputValueClass(Text.class);

                job1.setSortComparatorClass(IntWritableDecreasingComparator.class);

                job1.setInputFormatClass(TextInputFormat.class);

                job1.setOutputFormatClass(TextOutputFormat.class);

                Path in1 = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out2.1/part-r-00000");

                Path out1 = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out2.2");

                FileInputFormat.addInputPath(job1,in1);

                FileOutputFormat.setOutputPath(job1,out1);

                System.exit(job1.waitForCompletion(true) ? 0:1);

            }

        } catch (Exception e) {

            e.printStackTrace();

        }

    }

    public static class doMapper extends Mapper<Object,Text,Text,IntWritable>{

        public static Text word = new Text();

        public static final IntWritable id = new IntWritable(1);

        @Override

        protected void map(Object key,Text value,Context context) throws IOException,InterruptedException{

            String[] data = value.toString().split(",");

            String str=data[0]+"\t"+data[5];

            System.out.println(str);

            word.set(str);

            //id.set(Integer.parseInt(data[5]));

            context.write(word,id);

        }

    }

    public static class doReducer extends Reducer< Text, IntWritable, IntWritable, Text>{

        private static IntWritable result= new IntWritable();

        public void reduce(Text key,Iterable<IntWritable> values,Context context) throws IOException, InterruptedException{

            int sum = 0;

            for(IntWritable value:values){

                sum += value.get();

            }

            result.set(sum);

            context.write(result,key);

            }

    }

    public static class doMapper1 extends Mapper<Object , Text , IntWritable,Text >{

        private static Text goods=new Text();

        private static IntWritable num=new IntWritable();

        public void map(Object key,Text value,Context context) throws IOException, InterruptedException{

            String line=value.toString();

            String arr[]=line.split("\t");

            String str=arr[1]+"\t"+arr[2];

            num.set(Integer.parseInt(arr[0]));

            goods.set(str);

            context.write(num,goods);

        }

    }

    public static class doReducer1 extends Reducer< IntWritable, Text, IntWritable, Text>{

        private static IntWritable result= new IntWritable();

        int i=0;

        public void reduce(IntWritable key,Iterable<Text> values,Context context) throws IOException, InterruptedException{

            for(Text value:values){

                if(i<10) {

                context.write(key,value);

                i++;

                }

            }

            }

    }

    private static class IntWritableDecreasingComparator extends IntWritable.Comparator {

         public int compare(WritableComparable a, WritableComparable b) {

             return -super.compare(a, b);

          }

         public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {

                    return -super.compare(b1, s1, l1, b2, s2, l2);

           }

    }

}

·按照流量统计最受欢迎的Top10课程 （traffic）

package mapreduce;

import java.io.IOException;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.io.WritableComparable;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.Reducer.Context;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class GetVideoResulttraffic {

    public static void main(String[] args) {

        try {

            Job job = Job.getInstance();

            job.setJobName("GetVideoResult");

            job.setJarByClass(GetVideoResultip.class);

            job.setMapperClass(doMapper.class);

            job.setReducerClass(doReducer.class);

            job.setOutputKeyClass(Text.class);

            job.setOutputValueClass(IntWritable.class);

            job.setInputFormatClass(TextInputFormat.class);

            job.setOutputFormatClass(TextOutputFormat.class);

            Path in = new Path("hdfs://192.168.137.67:9000/mymapreducel/in/result.txt");

            Path out = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out3.1");

            FileInputFormat.addInputPath(job,in);

            FileOutputFormat.setOutputPath(job,out);

            //System.exit(job.waitForCompletion(true) ? 0:1);

            if(job.waitForCompletion(true))

            {

                Job job1 = Job.getInstance();

                job1.setJobName("Sort");

                job1.setJarByClass(GetVideoResult.class);

                job1.setMapperClass(doMapper1.class);

                job1.setReducerClass(doReducer1.class);

                job1.setOutputKeyClass(IntWritable.class);

                job1.setOutputValueClass(Text.class);

                job1.setSortComparatorClass(IntWritableDecreasingComparator.class);

                job1.setInputFormatClass(TextInputFormat.class);

                job1.setOutputFormatClass(TextOutputFormat.class);

                Path in1 = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out3.1/part-r-00000");

                Path out1 = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out3.2");

                FileInputFormat.addInputPath(job1,in1);

                FileOutputFormat.setOutputPath(job1,out1);

                System.exit(job1.waitForCompletion(true) ? 0:1);

            }

        } catch (Exception e) {

            e.printStackTrace();

        }

    }

    public static class doMapper extends Mapper<Object,Text,Text,IntWritable>{

        public static Text word = new Text();

        public static final IntWritable id = new IntWritable();

        @Override

        protected void map(Object key,Text value,Context context) throws IOException,InterruptedException{

            String[] data = value.toString().split(",");

            //String str=data[0]+"\t"+data[5];

            data[3] = data[3].substring(0, data[3].length()-1);

            word.set(data[5]);

            id.set(Integer.parseInt(data[3]));

            context.write(word,id);

        }

    }

    public static class doReducer extends Reducer< Text, IntWritable, IntWritable, Text>{

        private static IntWritable result= new IntWritable();

        public void reduce(Text key,Iterable<IntWritable> values,Context context) throws IOException, InterruptedException{

            int sum = 0;

            for(IntWritable value:values){

                sum += value.get();

            }

            result.set(sum);

            context.write(result,key);

            }

    }

    public static class doMapper1 extends Mapper<Object , Text , IntWritable,Text >{

        private static Text goods=new Text();

        private static IntWritable num=new IntWritable();

        public void map(Object key,Text value,Context context) throws IOException, InterruptedException{

            String line=value.toString();

            String arr[]=line.split("\t");

            num.set(Integer.parseInt(arr[0]));

            goods.set(arr[1]);

            context.write(num,goods);

        }

    }

    public static class doReducer1 extends Reducer< IntWritable, Text, IntWritable, Text>{

        private static IntWritable result= new IntWritable();

        int i=0;

        public void reduce(IntWritable key,Iterable<Text> values,Context context) throws IOException, InterruptedException{

            for(Text value:values){

                if(i<10) {

                context.write(key,value);

                i++;

                }

            }

            }

    }

    private static class IntWritableDecreasingComparator extends IntWritable.Comparator {

         public int compare(WritableComparable a, WritableComparable b) {

             return -super.compare(a, b);

          }

         public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {

                    return -super.compare(b1, s1, l1, b2, s2, l2);

           }

    }

}

3、数据没有导入到mysql中，但是通过MapReduce进行了echarts可视化

先通过MapReduce进行清洗数据，然后在jsp中进行可视化

package mapreduce3;

import java.io.IOException;

import java.util.ArrayList;

import java.util.List;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.io.WritableComparable;

import org.apache.hadoop.io.WritableComparator;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.Reducer.Context;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class Pai {

    public static List<String> Names=new ArrayList<String>();

    public static  List<String> Values=new ArrayList<String>();

    public static class Sort extends WritableComparator

    {

        public Sort()

        {

            super(IntWritable.class,true);

        }

        @Override

        public int compare(WritableComparable a, WritableComparable b)

        {

            return -a.compareTo(b);

        }

    }

    public static class Map extends Mapper<Object , Text , IntWritable,Text >{

        private static Text Name=new Text();

        private static IntWritable num=new IntWritable();

        public void map(Object key,Text value,Context context)throws IOException, InterruptedException

        {

            String line=value.toString();

            String arr[]=line.split("\t");

            if(!arr[0].startsWith(" "))

            {

                  num.set(Integer.parseInt(arr[0]));

                  Name.set(arr[1]);

                  context.write(num, Name);

            }

        }

    }

    public static class Reduce extends Reducer< IntWritable, Text, IntWritable, Text>{

        private static IntWritable result= new IntWritable();

        int i=0;

         public void reduce(IntWritable key,Iterable<Text> values,Context context) throws IOException, InterruptedException{

                for(Text val:values)

                {

                if(i<10)

                {i=i+1;

                    Names.add(val.toString());

                    Values.add(key.toString());

                }

                context.write(key,val);

                }

         }

    }

    public static int run()throws IOException, ClassNotFoundException, InterruptedException{

        Configuration conf=new Configuration();

        conf.set("fs.defaultFS", "hdfs://192.168.137.67:9000");

        FileSystem fs =FileSystem.get(conf);

        Job job =new Job(conf,"OneSort");

        job.setJarByClass(Pai.class);

        job.setMapperClass(Map.class);

        job.setReducerClass(Reduce.class);

        job.setSortComparatorClass(Sort.class);

        job.setOutputKeyClass(IntWritable.class);

        job.setOutputValueClass(Text.class);

        job.setInputFormatClass(TextInputFormat.class);

        job.setOutputFormatClass(TextOutputFormat.class);

        Path in = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out1.2/part-r-00000");

        Path out = new Path("hdfs://192.168.137.67:9000/mymapreducelShiYan/out1.4");

        FileInputFormat.addInputPath(job,in);

        fs.delete(out,true);

        FileOutputFormat.setOutputPath(job,out);

        return(job.waitForCompletion(true) ? 0 : 1);  

        }

}

zhu.jsp

<%@page import="mapreduce3.Pai"%>

<%@page import="mapreduce3.GetVideoResult"%>

<%@ page language="java" import="java.util.*" contentType="text/html; charset=UTF-8"

    pageEncoding="UTF-8"%>

<!DOCTYPE html>

<html>

<head>

<meta charset="UTF-8">

<title>Insert title here</title>

<script src="${pageContext.request.contextPath}/resource/echarts.js"></script>

</head>

<body>

<%

    Pai ss=    new Pai();

    ss.run();

    String[] a=new String[11];

    String[] b=new String[11];

    int i=0,j=0;

        for(i = 0 ; i < 10 ; i++)

        {

            a[i] = ss.Values.get(i);

            b[i] = ss.Names.get(i);

        }

%>

<div id="main" style="width: 600px;height:400px;"></div>

  <script type="text/javascript">

      // 基于准备好的dom，初始化echarts实例

      var myChart = echarts.init(document.getElementById('main'));

      // 指定图表的配置项和数据

      var option = {

            title: {

                text: '最受欢迎的文章/视频 TOP10'

            },

            tooltip: {},

            legend: {

                data:['统计']

            },

            xAxis: {

                data: [   <%

                          for( i=0;i<10;i++)

                          {

                          %><%=b[i]%>,<%

                          }

                          %>]

            },

            yAxis: {},

            series: [{

                name: '最受欢迎的文章',

                type: 'bar',

                data: [

                <%

                for( i=0;i<10;i++)

                {

                %><%=a[i]%>,<%

                }

                %>

                        ]

            }]

        };

      // 使用刚指定的配置项和数据显示图表。

      myChart.setOption(option);

      </script>

</body>

</html>

因为其他的数据清洗上边有，代码就不一一展示，只贴出jsp文件，如果想要改变可视化团，在echarts官网中直接复制代码到jsp中进行修改即可。

zhe.jsp

<%@page import="mapreduce3.Pai1"%>

<%@page import="mapreduce3.GetVideoResult"%>

<%@ page language="java" import="java.util.*" contentType="text/html; charset=UTF-8"

    pageEncoding="UTF-8"%>

<!DOCTYPE html>

<html>

<head>

<meta charset="UTF-8">

<title>Insert title here</title>

<script src="${pageContext.request.contextPath}/resource/echarts.js"></script>

</head>

<body>

<%

    Pai1 ss=    new Pai1();

    ss.run();

    String[] a=new String[11];

    String[] b=new String[11];

    int i=0,j=0;

        for(i = 0 ; i < 10 ; i++)

        {

            a[i] = ss.Values.get(i);

            b[i] = ss.Names.get(i);

        }

%>

<div id="main" style="width: 600px;height:400px;"></div>

  <script type="text/javascript">

      // 基于准备好的dom，初始化echarts实例

      var myChart = echarts.init(document.getElementById('main'));

      // 指定图表的配置项和数据

      var option = {

            title: {

                text: '按照地市最受欢迎'

            },

            tooltip: {},

            legend: {

                data:['统计']

            },

            xAxis: {

                data: [

　　　　　　　　　　　　　　<%
　　　　　　　　　　　　　　　　for( i=0;i<10;i++)
　　　　　　　　　　　　　　　　{
　　　　　　　　　　　　　　%>'<%=b[i]%>',

　　　　　　　　　　　　　　　　<%
　　　　　　　　　　　　　　　　}
　　　　　　　　　　　　　　　　%>

]

            },

            yAxis: {},

            series: [{

                name: '最受欢迎的文章',

                type: 'line',

                data: [

                <%

                for( i=0;i<10;i++)

                {

                %><%=a[i]%>,<%

                }

                %>

                        ]

            }]

        };

      // 使用刚指定的配置项和数据显示图表。

      myChart.setOption(option);

      </script>

</body>

</html>

tu.jsp

<%@page import="mapreduce3.Pai2"%>

<%@page import="mapreduce3.GetVideoResult"%>

<%@ page language="java" import="java.util.*" contentType="text/html; charset=UTF-8"

    pageEncoding="UTF-8"%>

<!DOCTYPE html>

<html>

<head>

<meta charset="UTF-8">

<title>Insert title here</title>

<script src="${pageContext.request.contextPath}/resource/echarts.js"></script>

</head>

<body>

<%

    Pai2 ss=    new Pai2();

    ss.run();

    String[] a=new String[11];

    String[] b=new String[11];

    int i=0,j=0;

        for(i = 0 ; i < 10 ; i++)

        {

            a[i] = ss.Values.get(i);

            b[i] = ss.Names.get(i);

        }

%>

<div id="main" style="width: 600px;height:400px;"></div>

  <script type="text/javascript">

      // 基于准备好的dom，初始化echarts实例

      var myChart = echarts.init(document.getElementById('main'));

      // 指定图表的配置项和数据

      option = {

            title : {

                text: '按照流量最受欢迎',

                x:'center'

            },

            tooltip : {

        trigger: 'item',

        formatter: "{a} <br/>{b} : {c} ({d}%)"

    },

    legend: {

        orient: 'vertical',

        left: 'left',

        data: [

            <%

            for( i=0;i<10;i++)

            {

            %>'<%=b[i]%>',

            <%

            }

            %>

        ]

    },

    series : [

        {

            name: '访问来源',

            type: 'pie',

            radius : '55%',

            center: ['50%', '60%'],

            data:[

                <%

                for( i=0;i<10;i++)

                {

                %>{value:<%=a[i]%>,name:'<%=b[i]%>'},

                <%

                }

                %>

            ],

            itemStyle: {

                emphasis: {

                    shadowBlur: 10,

                    shadowOffsetX: 0,

                    shadowColor: 'rgba(0, 0, 0, 0.5)'

                }

            }

        }

    ]

};

      // 使用刚指定的配置项和数据显示图表。

      myChart.setOption(option);

      </script>

</body>

</html>

MapReduce清洗数据进行可视化的更多相关文章

  1. mapreduce清洗数据

    继上篇 MapReduce清洗数据 package mapreduce; import java.io.IOException; import org.apache.hadoop.conf.Confi ...

  2. 视频网站数据MapReduce清洗及Hive数据分析

    一.需求描述 利用MapReduce清洗视频网站的原数据,用Hive统计出各种TopN常规指标: 视频观看数 Top10 视频类别热度 Top10 视频观看数 Top20 所属类别包含这 Top20 ...

  3. MapReduce处理数据1

    学了一段时间的hadoop了,一直没有什么正经练手的机会,今天老师给了一个课堂测试来进行练手,正好试一下. 项目已上传至github:https://github.com/yandashan/MapR ...

  4. discuz论坛apache日志hadoop大数据分析项目：清洗数据核心功能解说及代码实现

    discuz论坛apache日志hadoop大数据分析项目:清洗数据核心功能解说及代码实现http://www.aboutyun.com/thread-8637-1-1.html(出处: about云 ...

  5. 做Data Mining，其实大部分时间都花在清洗数据

    做Data Mining,其实大部分时间都花在清洗数据 时间 2016-12-12 18:45:50  51CTO 原文  http://bigdata.51cto.com/art/201612/52 ...

  6. 11，SFDC 管理员篇 - 报表和数据的可视化

    1,Report Builder 1,每一个report type 都有一个 primay object 和多个相关的object 2,Primary object with related obje ...

  7. MetricGraphics.js – 时间序列数据的可视化

    MetricsGraphics.js 是建立在D3的基础上,被用于可视化和布局的时间序列数据进行了优化.它提供以产生一个原则性的,一致的和响应式的方式的图形常见类型的简单方法.该库目前支持折线图,散点 ...

  8. 利用 t-SNE 高维数据的可视化

    利用 t-SNE 高维数据的可视化  具体软件和教程见: http://lvdmaaten.github.io/tsne/  简要介绍下用法: % Load data load ’mnist_trai ...

  9. MapReduce的数据流程、执行流程

    MapReduce的数据流程: 预先加载本地的输入文件 经过MAP处理产生中间结果 经过shuffle程序将相同key的中间结果分发到同一节点上处理 Recude处理产生结果输出 将结果输出保存在hd ...

随机推荐

  1. 解决Maven项目中的无故报错的方法

    解决Eclipse+maven中的无故报错 错误: One or more constraints have not been satisfied. Deployment Assembly跟java版 ...

  2. 初始 Kafka Consumer 消费者

    温馨提示:整个 Kafka 专栏基于 kafka-2.2.1 版本. 1.KafkaConsumer 概述 根据 KafkaConsumer 类上的注释上来看 KafkaConsumer 具有如下特征 ...

  3. 【转】在MyEclipse 8.6上搭建Android开发环境

    内容导航 第 1 页:基本环境准备 第 2 页:下载Android SDK 第 3 页:配置SDK环境变量 第 4 页:给MyEclipse安装ADT插件 第 5 页:配置MyEclipse 第 6 ...

  4. cogs 1440. [NOIP2013]积木大赛 贪心水题

    1440. [NOIP2013]积木大赛 ★★   输入文件:BlockNOIP2013.in   输出文件:BlockNOIP2013.out   简单对比时间限制:1 s   内存限制:128 M ...

  5. Python解析json字符串，json字符串用法

    json数据简介 json数据是一个轻量级的数据交换格式,采用完全独立于语言的文本格式,这些特性使json称为理想的数据交换语言,易于人阅读和编写,同时易于机器解析和生成. json中的字符集必须是U ...

  6. Qt Installer Framework翻译(3-5)

    指定设置选项 设置页面使用户可以指定代理设置或安装附加组件. 用户在简介页面上点击"设置"来指定设置选项. 指定代理设置 默认情况下,安装程序使用系统代理设置.用户可以选择不使用或 ...

  7. 简单理解设计模式——享元模式-线程池-任务(tesk)

    前面在写到多线程的文章的时候,一直想写一篇关于线程池等一系列的文章,做一下记录,本篇博客记录一下设计模式中享元模式的设计思想,以及使用享元模式的实现案例——线程池,以及线程池的简化版——任务(tesk ...

  8. 进击.net 三大框架

    spring mybatis NHibernate

  9. 51Nod 1238 - 最小公倍数之和 V3（毒瘤数学+杜教筛）

    题目 戳这里 推导 ∑i=1n∑j=1nlcm(i,j)~~~\sum_{i=1}^{n}\sum_{j=1}^{n}lcm(i,j)   ∑i=1n​∑j=1n​lcm(i,j) =∑i=1n∑j= ...

  10. Ubuntu学习之路2

    由于数据越来越多,学习的也稍微多了点,故将一步一步学习到的命令和快捷键继续到这篇博客. 1. 查看隐藏文件和文件夹 Ctrl+H 2. 打开搜索管理器 Alt+F2 3. 查看系统内存 free -h ...