解决异常断电导致的： CorruptSSTableException: java.io.EOFException

问题产生

服务器重启，导致cassandra损坏，整个集群不可用。所使用的cassandra为2.1.9版本。

问题描述

运行启动命令，报错如下：

DEBUG :: All segments have been unmapped successfully
INFO  :: Opening ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka- ( bytes)
ERROR :: Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at org.apache.cassandra.io.sstable.SSTableReader$.run(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:) [na:1.8.0_45]
    at java.util.concurrent.FutureTask.run(FutureTask.java:) [na:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:) [na:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:) [na:1.8.0_45]
    at java.lang.Thread.run(Thread.java:) [na:1.8.0_45]
Caused by: java.io.EOFException: null
    at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:) ~[na:1.8.0_45]
    at java.io.DataInputStream.readUTF(DataInputStream.java:) ~[na:1.8.0_45]
    at java.io.DataInputStream.readUTF(DataInputStream.java:) ~[na:1.8.0_45]
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:) ~[apache-cassandra-2.1..jar:2.1.]
    ...  common frames omitted
DEBUG :: INDEX LOAD TIME for ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-:  ms.
DEBUG :: Load metadata for ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-
INFO  :: Opening ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka- ( bytes)
DEBUG :: INDEX LOAD TIME for ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-:  ms.

解决方案

1. 在正常节点上执行（节点需要启动）

./nodetool ring | grep 192.168.66.149 | awk '{print $NF ","}' | xargs

 

返回如下：

-, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -,

2. 在损坏节点上修改配置文件：cassandra.yaml，并加入如下配置项：

auto_bootstrap: false

initial_token: -, -, -, -, -, -, -, -, -, -,......（后面的省略...）

 

注意：initial_token的取值为上一步骤的返回值。

3. 删除数据盘下system目录

如/usr/local/cassandra2/apache-cassandra-2.1.9/data/data/system

4. 启动cassandra

./cassandra

./cassandra

 

启动过程可能会报错，但会继续重建system库，只要能启动成功加入集群就算正常。

5. 修复数据

运行nodetool工具：

nodetool repair

6. 将配置项改回原样并重启

本解决方案参考自：/usr/local/cassandra2/apache-cassandra-2.1.9/data/data/system