반응형
1. 증상 : call tree - method 에 아래 그림과 같이 "API-METADATA-NOT-DOUND" 출력으로 추적불가
2. 확인 : hbase → ApiMetaData 테이블을 조회하면 실제 기록된 agent별 metadata를 볼수 있음
## hbase_home/bin 에 hbase shell을 실행하여 접속 한다.
[root@PINPOINT bin]# ./hbase shell
2020-07-15 14:28:50,761 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.7, rac57c51f7ad25e312b4275665d62b34a5945422f, Fri Sep 7 16:11:05 CDT 2018
## snapshot 목록 출력
hbase(main):001:0> list_snapshots
SNAPSHOT TABLE + CREATION TIME
0 row(s) in 0.4340 seconds
=> []
hbase(main):002:0> list
TABLE
AgentEvent
AgentInfo
AgentLifeCycle
AgentStatV2
ApiMetaData
ApplicationIndex
ApplicationMapStatisticsCallee_Ver2
ApplicationMapStatisticsCaller_Ver2
ApplicationMapStatisticsSelf_Ver2
ApplicationStatAggre
ApplicationTraceIndex
HostApplicationMap_Ver2
SqlMetaData_Ver2
StringMetaData
TraceV2
15 row(s) in 0.0530 seconds
=> ["AgentEvent", "AgentInfo", "AgentLifeCycle", "AgentStatV2", "ApiMetaData", "ApplicationIndex", "ApplicationMapStatisticsCallee_Ver2", "ApplicationMapStatisticsCaller_Ver2", "ApplicationMapStatisticsSelf_Ver2", "ApplicationStatAggre", "ApplicationTraceIndex", "HostApplicationMap_Ver2", "SqlMetaData_Ver2", "StringMetaData", "TraceV2"]
## hbase 구성 상태 정보
hbase(main):007:0> status
1 active master, 0 backup masters, 1 servers, 0 dead, 482.0000 average load
## user 출력
hbase(main):008:0> whoami
root (auth:SIMPLE)
groups: root
## 테이블 목록 조회
## 기본적으로 1000단위로 끊어 출력 하며 싱글쓰레드를 사용하므로 오랜시간이걸리고 부하가 가므로 주의한다.
## 해당 목록에 agent별 기록된 metadata 정보를 확인가능한데 "API_METADATA-NOT_FOUNT" 로 오류를 일으키는 agent metadata 정보는 없음.
hbase(main):009:0> count 'ApiMetaData'
Current count: 1000, row: \x01web3__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1Bmj\x00\x00\x00\xF7
Current count: 2000, row: \x01web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xE6\x11g\x00\x00\x01\xB5
Current count: 3000, row: \x01web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xE6\x11g\x00\x00\x05\x9D
Current count: 4000, row: \x01web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xB3\xF0\x15\x1C\xFF\xFF\xFF\x97
Current count: 5000, row: \x02web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE8\xE8\xEE\x00\x00\x00\x94
Current count: 6000, row: \x02web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE8\xE8\xEE\x00\x00\x04|
Current count: 7000, row: \x02web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE8\xE8\xEE\x00\x00\x08d
Current count: 8000, row: \x03web8__cc_login_tc\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB3\xF2\xB9\xE6\xFF\xFF\xFF\xF5
Current count: 9000, row: \x03web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\x00\xBE\x00\x00\x02T
Current count: 10000, row: \x04web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xF0\x86\x96\x00\x00\x01\x91
Current count: 11000, row: \x04web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xF0\x86\x96\x00\x00\x05y
Current count: 12000, row: \x04web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xF0\x86\x96\x00\x00\x09a
Current count: 13000, row: \x04web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xAF\xED\xB6o\x00\x00\x02N
Current count: 14000, row: \x05web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB3\xF4NG\x00\x00\x03\x00
Current count: 15000, row: \x05web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB3\xF4NG\xFF\xFF\xFD\xB4
Current count: 16000, row: \x05web8__slide_login_tc\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB3\xF1F/\xFF\xFF\xFE\xA8
Current count: 17000, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\x00\x00\x02\x91
Current count: 18000, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\xFF\xFF\xFF\xA2
Current count: 19000, row: \x06web8__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xAF\xE9\x94-\x00\x00\x00\x9F
Current count: 20000, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\x00\x00\x00\x02
Current count: 21000, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\x00\x00\x03\xEA
Current count: 22000, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\x00\x00\x07\xD2
Current count: 23000, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\xFF\xFF\xF9\xB3
Current count: 24000, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\xFF\xFF\xFD\x9B
Current count: 25000, row: \x07web2__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1EEV\x00\x00\x01\x84
Current count: 26000, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x03\xD7
26746 row(s) in 4.7680 seconds
=> 26746
## 테이블 정보
hbase(main):010:0> describe 'ApiMetaData'
Table ApiMetaData is ENABLED
ApiMetaData
COLUMN FAMILIES DESCRIPTION
{NAME => 'Api', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'PREFIX', TTL => '604800 SECONDS (7 DAYS)', COMPRE
SSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
1 row(s) in 0.0900 seconds
## 100단위로 끊어 출력
hbase(main):011:0> count 'ApiMetaData',INTERVAL=>100
Current count: 100, row: \x00web8__slide_login_tc\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\xA1\x9B\x00\x00\x00d
Current count: 200, row: \x00web8__slide_login_tc\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\xA1\x9B\x00\x00\x00\xC8
Current count: 300, row: \x00web8__slide_login_tc\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\xA1\x9B\x00\x00\x01,
Current count: 400, row: \x00web8__slide_login_tc\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\xA1\x9B\x00\x00\x01\x90
Current count: 500, row: \x00web8__slide_login_tc\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\xA1\x9B\x00\x00\x01\xF4
Current count: 600, row: \x00web8__slide_login_tc\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\xA1\x9B\x00\x00\x02X
Current count: 700, row: \x00web8__slide_login_tc\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\xA1\x9B\x00\x00\x02\xBC
Current count: 800, row: \x01web3__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1Bmj\x00\x00\x00/
Current count: 900, row: \x01web3__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1Bmj\x00\x00\x00\x93
Current count: 1000, row: \x01web3__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1Bmj\x00\x00\x00\xF7
Current count: 1100, row: \x01web3__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1Bmj\x00\x00\x01[
Current count: 1200, row: \x01web8__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1Bm-\x00\x00\x00*
Current count: 1300, row: \x01web8__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1Bm-\x00\x00\x00\x8E
Current count: 1400, row: \x01web8__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1Bm-\x00\x00\x00\xF2
Current count: 1500, row: \x01web8__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1Bm-\x00\x00\x01V
Current count: 1600, row: \x01web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xE6\x11g\x00\x00\x00%
Current count: 1700, row: \x01web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xE6\x11g\x00\x00\x00\x89
Current count: 1800, row: \x01web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xE6\x11g\x00\x00\x00\xED
Current count: 1900, row: \x01web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xE6\x11g\x00\x00\x01Q
Current count: 8400, row: \x03web8__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB3\xF6~k\xFF\xFF\xFF\xFB
Current count: 8500, row: \x03web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\x00\xBE\x00\x00\x00`
Current count: 8600, row: \x03web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\x00\xBE\x00\x00\x00\xC4
Current count: 8700, row: \x03web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\x00\xBE\x00\x00\x01(
Current count: 8800, row: \x03web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\x00\xBE\x00\x00\x01\x8C
Current count: 8900, row: \x03web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\x00\xBE\x00\x00\x01\xF0
Current count: 9000, row: \x03web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\x00\xBE\x00\x00\x02T
Current count: 9100, row: \x03web8__styleshop_web_tc\x00\x00\x7F\xFF\xFE\x8C\xAF\xE7\x00\xBE\x00\x00\x02\xB8
Current count: 9200, row: \x04web1__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1ED\xC3\x00\x00\x00\x06
Current count: 9300, row: \x04web1__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1ED\xC3\x00\x00\x00j
Current count: 9400, row: \x04web1__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1ED\xC3\x00\x00\x00\xCE
Current count: 9500, row: \x04web1__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1ED\xC3\x00\x00\x012
Current count: 9600, row: \x04web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xF0\x86\x96\x00\x00\x00\x01
Current count: 9700, row: \x04web8__cc_mweb_tc\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xAF\xF0\x86\x96\x00\x00\x00e
-- 중략 --
Current count: 16900, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\x00\x00\x02-
Current count: 17000, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\x00\x00\x02\x91
Current count: 17100, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\x00\x00\x02\xF5
Current count: 17200, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\x00\x00\x03Y
Current count: 17300, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\xFF\xFF\xFC\xE6
Current count: 17400, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\xFF\xFF\xFDJ
Current count: 17500, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\xFF\xFF\xFD\xAE
Current count: 17600, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\xFF\xFF\xFE\x12
Current count: 17700, row: \x05web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xB3\xEFGT\xFF\xFF\xFEv
Current count: 24000, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\xFF\xFF\xFD\x9B
Current count: 24100, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\xFF\xFF\xFD\xFF
Current count: 24200, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\xFF\xFF\xFEc
Current count: 24300, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\xFF\xFF\xFE\xC7
Current count: 24400, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\xFF\xFF\xFF+
Current count: 24500, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\xFF\xFF\xFF\x8F
Current count: 24600, row: \x07was6__mapi_tomcat\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFE\x8C\xB4)\x9E\xBA\xFF\xFF\xFF\xF3
Current count: 24700, row: \x07web2__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1EEV\x00\x00\x00X
Current count: 24800, row: \x07web2__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1EEV\x00\x00\x00\xBC
Current count: 24900, row: \x07web2__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1EEV\x00\x00\x01
Current count: 25000, row: \x07web2__cc_web_renewal_tc\x00\x7F\xFF\xFE\x8C\xB5\x1EEV\x00\x00\x01\x84
Current count: 25100, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x00S
Current count: 25200, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x00\xB7
Current count: 25300, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x01\x1B
Current count: 25400, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x01\x7F
Current count: 25500, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x01\xE3
Current count: 25600, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x02G
Current count: 25700, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x02\xAB
Current count: 25800, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x03\x0F
Current count: 25900, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x03s
Current count: 26000, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x03\xD7
Current count: 26100, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x04;
Current count: 26200, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x04\x9F
Current count: 26300, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x05\x03
Current count: 26400, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x05g
Current count: 26500, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x05\xCB
Current count: 26600, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x06/
Current count: 26700, row: \x07web8__styleshop_mweb_tc\x00\x7F\xFF\xFE\x8C\xAF\xEC\xCB\xF9\x00\x00\x06\x93
26746 row(s) in 2.9080 seconds
=> 26746
hbase(main):012:0>
3. 해결 : 해당 증상이 나타나는 agent를 재기동 필요
- 참조 : https://github.com/naver/pinpoint/issues/6496
- 재기동 시 관리 - agent management 에서 해당 agent를 삭제하고 재기동 하는 것이 좋음.
4. 시도 : hbase의 data 보존 기간인 7일 이후 "ApiMetaData" 테이블 또한 삭제되는 것이 아닌가 생각이 듬
따라서 아래와 같이 "ApiMetaData" 테이블을 백업 하고 동일 증상이 타나날시 백업한 table을 import 해보자
Export
dump the contents of table to HDFS
$ hbase org.apache.hadoop.hbase.mapreduce.Export <tablename> <outputdir> <versions> <starttime> <endtime>
Import
load data that has been exported back into HBase.
$ hbase org.apache.hadoop.hbase.mapreduce.Import <tablename> <inputdir>
## 실제 작업
## 백업할 폴더를 미리 생성 해놓으면 다음과 같은 에러가 발생 하므로 미리 생성 하지 않는다.
[root@PINPOINT bin]# ./hbase org.apache.hadoop.hbase.mapreduce.Export ApiMetaData ./
2020-07-27 08:57:56,435 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-07-27 08:57:56,576 INFO [main] mapreduce.Export: versions=1, starttime=0, endtime=9223372036854775807, keepDeletedCells=false
2020-07-27 08:57:57,172 INFO [main] Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2020-07-27 08:57:57,174 INFO [main] jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/usr/local/pinpoint/hbase/hbase-1.2.7/bin already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:188)
## 다시 실행
[root@CC-MON-PINPOINT bin]# ./hbase org.apache.hadoop.hbase.mapreduce.Export ApiMetaData ./backup
2020-07-27 08:58:17,113 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-07-27 08:58:17,252 INFO [main] mapreduce.Export: versions=1, starttime=0, endtime=9223372036854775807, keepDeletedCells=false
2020-07-27 08:58:17,853 INFO [main] Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2020-07-27 08:58:17,855 INFO [main] jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2020-07-27 08:58:18,769 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1757cd72 connecting to ZooKeeper ensemble=localhost:2181
2020-07-27 08:58:18,782 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2020-07-27 08:58:18,782 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=CC-MON-PINPOINT
2020-07-27 08:58:18,782 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_241
2020-07-27 08:58:18,782 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
-- 중략 --
2020-07-27 08:58:22,949 INFO [main] mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=209297456
FILE: Number of bytes written=227107390
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=11557
Map output records=11557
Input split bytes=863
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=15
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=3000500224
HBase Counters
BYTES_IN_REMOTE_RESULTS=0
BYTES_IN_RESULTS=2330525
MILLIS_BETWEEN_NEXTS=762
NOT_SERVING_REGION_EXCEPTION=0
NUM_SCANNER_RESTARTS=0
NUM_SCAN_RESULTS_STALE=0
REGIONS_SCANNED=8
REMOTE_RPC_CALLS=0
REMOTE_RPC_RETRIES=0
ROWS_FILTERED=60392
ROWS_SCANNED=71949
RPC_CALLS=24
RPC_RETRIES=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=2972976
## 확인
[root@PINPOINT bin]# cd backup/
[root@PINPOINT backup]# ll
total 2900
-rw-r--r-- 1 root root 495685 Jul 27 08:58 part-m-00000
-rw-r--r-- 1 root root 9077 Jul 27 08:58 part-m-00001
-rw-r--r-- 1 root root 472643 Jul 27 08:58 part-m-00002
-rw-r--r-- 1 root root 470471 Jul 27 08:58 part-m-00003
-rw-r--r-- 1 root root 498675 Jul 27 08:58 part-m-00004
-rw-r--r-- 1 root root 507498 Jul 27 08:58 part-m-00005
-rw-r--r-- 1 root root 495685 Jul 27 08:58 part-m-00006
-rw-r--r-- 1 root root 114 Jul 27 08:58 part-m-00007
-rw-r--r-- 1 root root 0 Jul 27 08:58 _SUCCESS
## 복원
[root@PINPOINT bin]# ./hbase org.apache.hadoop.hbase.mapreduce.Import ApiMetaData ./backup/
2020-07-27 08:59:46,605 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-07-27 08:59:47,090 INFO [main] Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2020-07-27 08:59:47,092 INFO [main] jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2020-07-27 08:59:47,966 INFO [main] input.FileInputFormat: Total input paths to process : 8
2020-07-27 08:59:48,004 INFO [main] mapreduce.JobSubmitter: number of splits:8
2020-07-27 08:59:48,176 INFO [main] mapreduce.JobSubmitter: Submitting tokens for job: job_local1408655420_0001
-- 중략 --
2020-07-27 08:59:52,483 INFO [LocalJobRunner Map Task Executor #0-EventThread] zookeeper.ClientCnxn: EventThread shut down
2020-07-27 08:59:52,483 INFO [LocalJobRunner Map Task Executor #0] mapred.Task: Task:attempt_local1408655420_0001_m_000007_0 is done. And is in the process of committing
2020-07-27 08:59:52,486 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner: map
2020-07-27 08:59:52,487 INFO [LocalJobRunner Map Task Executor #0] mapred.Task: Task 'attempt_local1408655420_0001_m_000007_0' done.
2020-07-27 08:59:52,487 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner: Finishing task: attempt_local1408655420_0001_m_000007_0
2020-07-27 08:59:52,487 INFO [Thread-44] mapred.LocalJobRunner: map task executor complete.
2020-07-27 08:59:53,126 INFO [main] mapreduce.Job: map 100% reduce 0%
2020-07-27 08:59:53,127 INFO [main] mapreduce.Job: Job job_local1408655420_0001 completed successfully
2020-07-27 08:59:53,189 INFO [main] mapreduce.Job: Counters: 18
File System Counters
FILE: Number of bytes read=225753281
FILE: Number of bytes written=213278864
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=11557
Map output records=11557
Input split bytes=1048
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=33
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=3000500224
File Input Format Counters
Bytes Read=2972976
File Output Format Counters
Bytes Written=0
## 복원 후에도 TTL값을 그대로 가지고 복원이 되므로 데이터는 삭제 되는 것 같음.
5. 시도 2 : "ApiMetaData" table의 TTL값을 변경
## "ApiMetaData" 정보 조회
## table의 description 정보는 웹에서도 확인 가능 "http://{pinpoint ip}:16010/tablesDetailed.jsp"
hbase(main):011:0> describe 'ApiMetaData'
Table ApiMetaData is ENABLED
ApiMetaData
COLUMN FAMILIES DESCRIPTION
{NAME => 'Api', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'PREFIX', TTL => '604800 SECONDS (7 DAYS)', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE =
> 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
1 row(s) in 0.0460 seconds
## TTL 값을 FOREVER로 변경
hbase(main):014:0> alter 'ApiMetaData',NAME=>'Api',TTL=>'2147483647'
Updating all regions with the new schema...
4/8 regions updated.
8/8 regions updated.
Done.
0 row(s) in 3.0590 seconds
## 확인
hbase(main):015:0> describe 'ApiMetaData'
Table ApiMetaData is ENABLED
ApiMetaData
COLUMN FAMILIES DESCRIPTION
{NAME => 'Api', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'PREFIX', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKS
IZE => '65536', REPLICATION_SCOPE => '0'}
1 row(s) in 0.0430 seconds반응형
'Monitoring Tools > Pinpoint' 카테고리의 다른 글
| 18. Real Remote Address (reverse proxy 사용) (0) | 2021.03.17 |
|---|---|
| 17. pinpoint | Nginx Http Header 추가 (0) | 2021.03.17 |
| pinpoint | 15. Alarm -> Slack Push 설정 (0) | 2021.03.17 |
| pinpoint | 14. 로그인 페이지 생성 (0) | 2021.03.17 |
| 13.[기타] WEB - API ERROR Check (0) | 2021.03.17 |
