使用java操作elasticsearch(1)
1.安装elasticsearch
这儿用的是5.6.9的版本，下载安装过程较为简单,在官网上下载好后解压到文件夹。需要注意的是在elasticsearch-5.6.9\config下的elasticsearch.yml文件夹中
配置data与log的地址，自己创建文件夹后如下配置
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: D:/java/soft/elasticsearch/dev/data
#
# Path to log files:
#
path.logs: D:/java/soft/elasticsearch/dev/log
#
启动elasticsearch 点击elasticksearch.bat即可启动 注意电脑上要安装1.8以上的jdk
出现以下消息则说明启动成功
上面红圈的9300端口则是与es进行数据交互时的端口，下面的9200则是查看数据的端口，ip为安装es的ip
启动项目后，访问http://ip:9200 ip是安装elasticsearch服务的机器ip，如果出现以下信息即代表安装成功
{
"name": "kSs9lbx",
"cluster_name": "elasticsearch",
"cluster_uuid": "Wu6n-WzvQuecDEMWspm_aA",
"version": {
"number": "5.6.9",
"build_hash": "877a590",
"build_date": "2018-04-12T16:25:14.838Z",
"build_snapshot": false,
"lucene_version": "6.6.1"
},
"tagline": "You Know, for Search"
}
2.创建java的springboot工程
2.1pom.xml文件依赖
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.2.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent> <properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<fastjson.version>1.2.31</fastjson.version>
<jackson.version>2.8.7</jackson.version>
</properties> <dependencies>
<!-- set jetty server -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
<!-- aop -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<!-- json处理 -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>${fastjson.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/net.sourceforge.javacsv/javacsv -->
<dependency>
<groupId>net.sourceforge.javacsv</groupId>
<artifactId>javacsv</artifactId>
<version>2.0</version>
</dependency>
<!-- jackson json begin -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-jaxb-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
<!-- jackson json end -->
<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.3.2</version>
</dependency>
<!-- druid datasource -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<version>1.0.27</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
<dependency>
<groupId>com.github.theborakompanioni</groupId>
<artifactId>thymeleaf-extras-shiro</artifactId>
<version>${thymeleaf-shiro.version}</version>
</dependency>
<!-- mysql驱动 -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
<!-- spring boot 开发工具 -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<optional>true</optional>
</dependency>
<!-- 测试 -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!--elasticsearrch-->
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>5.6.9</version>
</dependency> <dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>5.6.9</version>
</dependency> <dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency> </dependencies> <repositories>
<repository>
<id>aliyun</id>
<name>aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</repository>
</repositories> <build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<includeSystemScope>true</includeSystemScope>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
</plugins>
</build>
2.2application.yml文件配置
server:
port: 9999
es:
host: 127.0.0.1
port: 9300
spring:
datasource:
initialize: false #false时不进行数据库表结构和数据初始化
username: root
#password: root
password: root
url: jdbc:mysql://localhost:3306/estest?createDatabaseIfNotExist=true&autoReconnect=true&useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true
type: com.alibaba.druid.pool.DruidDataSource
#连接池配置
driverClassName: com.mysql.jdbc.Driver
# 初始化大小，最小，最大
initialSize: 5
minIdle: 5
maxActive: 50
# 配置获取连接等待超时的时间
maxWait: 60000
# 配置间隔多久才进行一次检测，检测需要关闭的空闲连接，单位是毫秒
timeBetweenEvictionRunsMillis: 60000
# 配置一个连接在池中最小生存的时间，单位是毫秒
minEvictableIdleTimeMillis: 300000
validationQuery: SELECT 1 FROM DUAL
testWhileIdle: true
testOnBorrow: false
testOnReturn: false
# 打开PSCache，并且指定每个连接上PSCache的大小
poolPreparedStatements: true
maxPoolPreparedStatementPerConnectionSize: 20
# 配置监控统计拦截的filters，去掉后监控界面sql无法统计，'wall'用于防火墙
filters: stat,log4j
# 通过connectProperties属性来打开mergeSql功能；慢SQL记录
connectionProperties: druid.stat.mergeSql=true;druid.stat.slowSqlMillis=5000
2.3 创建elasticsearch的配置类
package spakt.demo.config; import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;
import org.elasticsearch.transport.client.PreBuiltTransportClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration; import java.net.InetAddress;
import java.net.UnknownHostException; @Configuration
public class EsConfiguration { private String clusterName = "elasticsearch";
@Value("${es.host}")
private String esHost; @Value("${es.port}")
private String esPort; @Bean
public TransportClient getTransportClient() throws UnknownHostException { Settings sets = Settings.builder().put("cluster.name", clusterName) //集群名字
.put("client.transport.sniff", true)
.build();
InetSocketTransportAddress inet = new InetSocketTransportAddress(InetAddress.getByName(esHost), Integer.parseInt(esPort));
TransportClient transportClient = new PreBuiltTransportClient(sets).addTransportAddress(inet);
return transportClient;
}
}
这儿注意 如果是es6.x以后的版本 ，在引入es包之后，还要引入netty插件包，这儿以6.7.2为例
<!-- https://mvnrepository.com/artifact/org.elasticsearch.plugin/transport-netty4-client -->
<dependency>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>transport-netty4-client</artifactId>
<version>6.7.2</version>
</dependency>
并且上面的配置中InetSocketTransportAddress 修改为TransportAddress即可 具体如下
@Bean
public TransportClient initTransportClient() throws UnknownHostException {
Settings build = Settings.builder().put("cluster.name", clusterName).put("client.transport.sniff", true).build();
TransportAddress transportAddress = new TransportAddress(InetAddress.getByName(address),port);
TransportClient transportClient = new PreBuiltTransportClient(build).addTransportAddress(transportAddress);
return transportClient;
}
2.4 创建测试的controller类 包含添加，查询，删除方法
package spakt.demo.elasticsearch; import org.apache.commons.lang3.StringUtils;
import org.elasticsearch.action.bulk.BulkRequestBuilder;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.delete.DeleteRequestBuilder;
import org.elasticsearch.action.index.IndexRequestBuilder;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.action.search.SearchRequestBuilder;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.text.Text;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import spakt.demo.bean.Prisoner; import javax.annotation.Resource;
import java.text.DecimalFormat;
import java.util.*; /**
*
*/
@Controller
public class EsController { // @Resource(name = "jdbc2")
@Autowired
private JdbcTemplate jdbcTemplate; @Autowired
private TransportClient transportClient; /**
* 插入数据方法
*
* @return
*/
@RequestMapping("/es")
@ResponseBody
public Object addEs() { //要插入的数据
List<Map<String, Object>> prisonerListList = jdbcTemplate.
queryForList("select id,name,fact,family_address as address from da_prisoner ");
//批处理类
BulkRequestBuilder bulkRequestBuilder = transportClient.prepareBulk();
int count = 0;
for (Map<String, Object> prisoner : prisonerListList) { //插入方法
//jyplatform 代表index 理解为数据库中的库
//prisoner 代表type 理解为数据库总的表
//UUID 个人理解是为数据库中的主键
String uuid = UUID.randomUUID().toString().replace("-", "");
IndexRequestBuilder indexRequestBuilder = transportClient.
prepareIndex("jyplatform", "prisoner",uuid)
.setSource(prisoner);
//单个执行
//IndexResponse indexResponse = indexRequestBuilder.get(); //批量执行 将添加器加入批处理中
bulkRequestBuilder.add(indexRequestBuilder);
if (count % 5000 == 0) {
//每隔5000条执行一次批量插入的操作
BulkResponse bulkItemResponses = bulkRequestBuilder.execute().actionGet();
System.err.println(bulkItemResponses.getTook());
float num = (float) count / prisonerListList.size();
DecimalFormat df = new DecimalFormat("0.00");
String s = df.format(num * 100);
System.err.println("================================" + s + "%");
}
count++;
}
return prisonerListList;
} /**
* 查询数据
*
* @return
*/
@RequestMapping("/get")
@ResponseBody
public Object getMap(String content) {
List<Prisoner> prisonerList = new ArrayList<>();
// 构造搜索条件
BoolQueryBuilder query = QueryBuilders.boolQuery();
//fact 即是field 理解为表中的字段
query.filter(QueryBuilders.matchPhraseQuery("fact", content));
//开始查询
SearchRequestBuilder searchRequestBuilder = transportClient
.prepareSearch("jyplatform") //确定index
.setTypes("prisoner") //确定type
.setFrom(0) //从多少条开始 分页使用
.setSize(10000) //查多少条 分页使用 size-from>10000会报异常
.setQuery(query); //加入查询条件
//结果高亮显示
HighlightBuilder highlightBuilder = new HighlightBuilder();
highlightBuilder.field("fact");
highlightBuilder.preTags("<span style=\"color:red\">");
highlightBuilder.postTags("</span>");
searchRequestBuilder.highlighter(highlightBuilder);
//获取查询结果
SearchResponse searchResponse = searchRequestBuilder.get();
if (searchResponse.status() != RestStatus.OK) {
System.err.println("查询结果错误");
return "error";
}
//得到查询的结果
SearchHits hits = searchResponse.getHits();
for (SearchHit hit : hits) {
//进行高亮处理
String fact = "";
Map<String, HighlightField> highlightFields = hit.getHighlightFields();
HighlightField field = highlightFields.get("fact");
if(field!=null){
Text[] fragments = field.fragments();
for (Text fragment : fragments) {
fact+=fragment.toString();
}
} prisonerList.add(new Prisoner(hit.getSource().get("id").toString(),
hit.getSource().get("name").toString(),
StringUtils.isBlank(fact)?hit.getSource().get("fact").toString():fact,
hit.getSource().get("address").toString()));
}
return prisonerList;
} /**
* 删除数据
*/
@RequestMapping("/delete")
@ResponseBody
public String delete() {
//开始查询
SearchRequestBuilder searchRequestBuilder = transportClient
.prepareSearch("jyplatform") //确定index
.setTypes("prisoner") //确定type
.setFrom(0) // begin
.setSize(10000); //size size-from>10000会报异常
//获取查询结果
SearchHits hits = searchRequestBuilder.get().getHits();
//批量操作
BulkRequestBuilder bulkRequestBuilder = transportClient.prepareBulk();
for (SearchHit hit : hits) {
//删除方法
//jyplatform 代表index 理解为数据库中的库
//prisoner 代表type 理解为数据库总的表
// hit.getId() 这条数据的id
DeleteRequestBuilder deleteRequestBuilder = transportClient.
prepareDelete("jyplatform", "prisoner", hit.getId());
//批量删除
// bulkRequestBuilder.add(deleteRequestBuilder);
// if(i%5000 ==0){
// bulkRequestBuilder.execute();
// }
//单个操作
deleteRequestBuilder.get();
}
return "OK";
} }
2.5 创建启动类启动测试
package spakt.demo; import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ImportResource; @SpringBootApplication
@ImportResource(value = "classpath:spring-*.xml")
public class DemoApplication { public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
总结
目前只学习了下初步的使用，复杂的应用待后面继续挖掘。集群也，还有分词器一大堆东西待待填坑
使用java操作elasticsearch(1)的更多相关文章
- java操作elasticsearch实现组合桶聚合
1.terms分组查询 //分组聚合 @Test public void test40() throws UnknownHostException{ //1.指定es集群 cluster.name 是 ...
- java操作elasticsearch实现query String
1.CommonTersQuery: 指定字段进行模糊查询 //commonTermsQuery @Test public void test35() throws UnknownHostExcept ...
- java操作elasticsearch实现聚合查询
1.max 最大值 //max 求最大值 @Test public void test30() throws UnknownHostException{ //1.指定es集群 cluster.name ...
- java操作elasticsearch实现前缀查询、wildcard、fuzzy模糊查询、ids查询
1.前缀查询(prefix) //prefix前缀查询 @Test public void test15() throws UnknownHostException { //1.指定es集群 clus ...
- java操作elasticsearch实现条件查询（match、multiMatch、term、terms、reange）
1.条件match query查询 //条件查询match query @Test public void test10() throws UnknownHostException { //1.指定e ...
- java操作elasticsearch实现查询删除和查询所有
后期博客本人都只给出代码,具体的说明在代码中也有注释. 1.查询删除 //查询删除:将查询到的数据进行删除 @Test public void test8() throws UnknownHostEx ...
- java操作elasticsearch实现批量添加数据（bulk）
java操作elasticsearch实现批量添加主要使用了bulk 代码如下: //bulk批量操作(批量添加) @Test public void test7() throws IOExcepti ...
- java操作elasticsearch实现基本的增删改查操作
一.在进行java操作elasticsearch之前,请确认好集群的名称及对应的ES节点ip和端口 1.查看ES的集群名称 #进入elasticsearch.yml配置文件/opt/elasticse ...
- Java操作ElasticSearch之创建客户端连接
Java操作ElasticSearch之创建客户端连接 3 发布时间:『 2017-09-11 17:02』 博客类别:elasticsearch 阅读(3157) Java操作ElasticSe ...
随机推荐
- 爬虫框架 Scrapy
一 介绍 crapy一个开源和协作的框架,其最初是为了页面抓取 (更确切来说, 网络抓取 )所设计的,使用它可以以快速.简单.可扩展的方式从网站中提取所需的数据.但目前Scrapy的用途十分广泛,可用 ...
- jQuery 第六章 jQuery在Ajax应用
1.本章目标 ajax 2.ajax 异步刷新技术,我们的网页不需要全部刷新,按需实现局部刷新,上线后台的交互 用户体验好 地图,前台验证,表单提交,修改,查询等等 原生的js和ajax packag ...
- ASP.NET 多环境下配置文件web.config的灵活配置
调试,发布Asp.net程序的时候,开发环境和发布环境的Web.Config往往不同,比如connectionstring等.如果常常有调试,发布的需求,就需要常常修改web.config文件,这往往 ...
- php树形结构数组转化
/** * @param array $list 要转换的结果集 * @param string $pid parent标记字段 * @param string $level level标记字段 */ ...
- 理解WidowManager
--摘自<Android进阶解密> *WindowManger的关联类* 1.Window是一个抽象类,具体的实现类为PhoneWindow,它对View进行管理.WindowManage ...
- 【java】-- 多线程快速入门
1.什么是进程?什么是线程?两者区别? 1.每个正在系统上运行的程序都是一个进程,每个进程包含一到多个线程,多线程处理就是允许一个进程中在同一时刻执行多个任务. 2.线程是一组指令的集合,或者是程序的 ...
- C#中抽象类和接口的区别(二)
一.抽象类: 抽象类是特殊的类,只是不能被实例化:除此以外,具有类的其他特性:重要的是抽象类可以包括抽象方法,这是普通类所不能的.抽象方法只能声明于抽象类中,且不包含任何实现,派生类必须覆盖它们.另外 ...
- 4. Spring 如何通过 XML 文件配置Bean，以及如何获取Bean
在 Spring 容器内拼凑 bean 叫做装配.装配 bean 的时候,你是在告诉容器,需要哪些 bean ,以及容器如何使用依赖注入将它们配合在一起. 理论上,bean 装配的信息可以从任何资源获 ...
- idea 自定义注释模板
一.类注释模板 打开Preferences Editor -> File and Code Templates -> Files -> Class 效果图: 注释模板 /** * @ ...
- vue PC端项目中解决userinfo问题
在vue2 中用脚手架建立的项目,后端提供接口获取数据.在公司做第一个项目的时候不清楚公司里的对接流程,结果后续代码被一个接口整的乱七八糟,这个接口是获取用户信息的接口——'usre/info'. 如 ...