14. ELK

ELK
작은 사이즈의 데이터는 키바나의 파일 업로드 기능을 이용하는 게 편하다

1.키바나 첫페이지(192.168.2.180:5601)에 upload a file로 가서 csv를 넣음
2.machine learning에서 visual

csv형태로 되어있는걸 알아서 쪼개서 필드를 만들어줌
필드가 어떤 타입인지도 알아서 셋팅해줌
ingestor파이프라인도 알아서 생성해줌

Dev tools에서
csv파일 업로드로 만들어진 인덱스를 조회할 수 있음
GET Seoul-metro-station-info-temp/_search


데이터가 잘 들어가도
매핑을 손봐야한다
인덱스를 민들면서 매핑을 추가한다
PUT seoul-metro-station-info
{
  "mappings": {
    "properties": {
      "geo": {
        "properties": {
          "addres_road": { "type": "text" },
          "address_land": { "type": "text" },
          "latitude": { "type": "float" },
          "longitude": { "type": "float" },
          "phone": { "type": "text" },
          "sigungu_code": { "type": "keyword" },
          "sigungu_name": { "type": "keyword" },
          "location": { "type": "geo_point" }
        }
      },
      "line": {
        "properties": {
          "name": { "type": "keyword" },
          "name_sub": { "type": "keyword" },
          "num": { "type": "byte" },
          "station_seq": { "type": "byte" }
        }
      },
      "station": {
        "properties": {
          "code": { "type": "short" },
          "fr_code": { "type": "keyword" },
          "name": { "type": "keyword" },
          "name_chc": { "type": "keyword" },
          "name_chn": { "type": "keyword" },
          "name_en": { "type": "text" },
          "name_full": { "type": "keyword" },
          "name_jp": { "type": "keyword" }
        }
      }
    }
  }
}

geo포인트 타입으로 할거임

reindex허가 전에 ingest pipline을 통해
Geo, line, station 필드의 하위필드로 쪼개 저장함.
PUT _ingest/pipeline/seoul-metro-station-pipe
{
  "processors": [
    { "set": { "field": "_id", "value": "{{station_code}}" } },
    { "set": { "field": "geo.location.lon", "value": "{{geo_longitude}}" } },
    { "set": { "field": "geo.location.lat", "value": "{{geo_latitude}}" } },
    { "convert": { "field": "geo.location.lon", "type": "float" } },
    { "convert": { "field": "geo.location.lat", "type": "float" } },
    { "split": { "field": "station_name", "separator": "\\|" } },
    { "split": { "field": "line_name_sub", "separator": "\\|" } },
    {"rename": { "field": "geo_addres_road", "target_field": "geo.addres_road" } },
    {"rename": { "field": "geo_address_land", "target_field": "geo.address_land" } },
    {"rename": { "field": "geo_latitude", "target_field": "geo.latitude" } },
    {"rename": { "field": "geo_longitude", "target_field": "geo.longitude" } },
    {"rename": { "field": "geo_phone", "target_field": "geo.phone" } },
    {"rename": { "field": "geo_sigungu_code", "target_field": "geo.sigungu_code" } },
    {"rename": { "field": "geo_sigungu_name", "target_field": "geo.sigungu_name" } },
    {"rename": { "field": "line_name", "target_field": "line.name" } },
    {"rename": { "field": "line_name_sub", "target_field": "line.name_sub" } },
    {"rename": { "field": "line_num", "target_field": "line.num" } },
    {"rename": { "field": "line_station_seq", "target_field": "line.station_seq" } },
    {"rename": { "field": "station_code", "target_field": "station.code" } },
    {"rename": { "field": "station_fr_code", "target_field": "station.fr_code" } },
    {"rename": { "field": "station_name", "target_field": "station.name" } },
    {"rename": { "field": "station_name_chc", "target_field": "station.name_chc" } },
    {"rename": { "field": "station_name_chn", "target_field": "station.name_chn" } },
    {"rename": { "field": "station_name_en", "target_field": "station.name_en" } },
    {"rename": { "field": "station_name_jp", "target_field": "station.name_jp" } },
    {"rename": { "field": "station_name_full", "target_field": "station.name_full" } }
  ]
}

station_name에서 spilit으로 띄어쓰기| 단위로 쪼개서 [] 형태로 저장시킬 수 있음
그리고 station_name을 target_field로 station.name으로 station 하위에 name : [] 형태로 들어가게 파이프라인을 구성함
set은 칼럼 하나 더 구성 하겠다 라는 것
geo_longtitude 라는 필드 그대로 get.location.lon안에 넣음
convert는 데이터 타입 변환 같음…
geo_logtitude는 rename을 통해 geo.longtitude안에도 있고 set을 통해 geo.location.lon 안에도 존재함


POST _reindex
{
  "source": {
    "index": "seoul-metro-station-info-temp"
  },
  "dest": {
    "index": "seoul-metro-station-info",
    "pipeline": "seoul-metro-station-pipe"
  }
}
reindex로 index를 바꿔줌

승하차 인원 집계 로그 파일이 색인될 때
인덱스 정보에 맞는 데이터와 조인되어서 데이터가 들어가야함.(하위 필드로 넣을 애들)
enrich프로세서를 포함하는 invgestor파이프라인을 만들어야 함

PUT /_enrich/policy/seoul-metro-info_policy
{
  "match": {
    "indices": "seoul-metro-station-info",
    "match_field": "station.code",
    "enrich_fields": [ "line", "station", "geo" ]
  }
}

_enrich/policy/seoul-metro-info_policy
match_field는 인덱스의 필드와 로그 파일의 필드 사이에 조인 key라고 생각하면됨
station_code로 조인하겠다.
enrich_fields는 인덱스 안에 있는 line, geo, station필드를 가져 오도록 한다
즉, station_code를 기준으로 line, geo,station은 하위 필드로 들어가게 만들어준다
enrich : seoul-metro-info_policy 인덱스에서 가져온 정보들을 info 필드의 하위에 넣습니다.
Infof라는 필드는 pipline만들때 생성
 

POST /_enrich/policy/seoul-metro-info_policy/_execute

그리고 실행해줘야함
execute안해주면 활성화 안됨


PUT _ingest/pipeline/seoul-metro-logs-pipe
{
  "processors": [
    {
      "enrich": {
        "policy_name": "seoul-metro-info_policy",
        "field": "station_code",
        "target_field": "info"
      }
    }
  ]
}
Enrich/policy만든거(조인방법)을 ingestor pipeline pocessot안에 
enruch에 policy 이름 넣고
field는 조인키
target_field는 조인된 데이터(geo, station, line)가 들어갈 필드임


POST _ingest/pipeline/seoul-metro-logs-pipe/_simulate
{
  "docs": [
    {
      "_source": {
        "@timestamp": "2015-01-01T05:00:00.000+09:00",
        "station_code": 150,
        "people_in": 441,
        "people_out": 392
      }
    }
  ]
}
데이터를 넣어준다 _simuldate로 샘플 doc하나만 넣어준다