ansible | [AWX] logstash(elasticsearch) 를 이용한 로그 수집

AWX 로그설정

---

• AWX 버전 : 17.1.0
• Settings - Logging settings - Edit

 

• Logging Aggregator : Logstash Host IP
• Logging Aggregator Port : Logstash Listen Port (기 사용중이단 5044포트와 다른 5045로 별도 지정하였음)
• Logging Aggregator Type : logstash
• Logging Aggregator Username : Logstash user(여기서는 사용하지 않음.)
• Logging Aggregator Password/Token : Logstash password (여기서는 사용하지 않음)
• Logging Aggregator Protocol : TCP
• Logging Aggregator Level Threshold : log level info 이상 설정 (job_events의 경우 info level로 로그 이벤트가 생성)
• Loggers Sending Data to Log Aggregator Form
  • job_events: Provides data returned from the Ansible callback module
  • activity_stream: Displays the record of changes to the objects within the Ansible Tower application
  • system_tracking: Provides fact data gathered by Ansible setup module (i.e. gather_facts: True) when job templates are ran with Enable Fact Cache selected
  • awx: Provides generic server logs, which include logs that would normally be written to a file. It contains the standard metadata that all logs have, except it only has the message from the log statement.
  • 참조 : https://docs.ansible.com/ansible-tower/latest/html/administration/logging.html

 

 

AWX Log Test

---

• Logstash 대신 간단히 nc를 사용하여 awx에서 인입되는 로그정보를 확인할수 있다.
• Logstash Host 에서 nc 로 5045 port Listen 하고 json 표출로 jq를 사용.

> nc -kl 5045 | jq

 

 

 

기본 스키마

• cluster_host_id: Unique identifier of the host within the Tower cluster
  • awx의 클러스터 호스트 ID
• level: Standard python log level, roughly reflecting the significance of the event All of the data loggers as a part of this feature use INFO level, but the other Tower logs will use different levels as appropriate
  • 로그 레벨
• logger_name: Name of the logger we use in the settings, for example, “activity_stream”
  • Loggers Sending Data to Log Aggregator Form 에서 설정했던 항목의 구분이 가능하다. 예를 들면 "logger_name": "awx.analytics.job_events"
• @timestamp: Time of log
• path: File path in code where the log was generated

{
  "@timestamp": "2022-06-16T06:40:21.866Z",
  "message": "scaling down worker pid:9469",
  "host": "awx",
  "level": "WARNING",
  "logger_name": "awx.main.dispatch",
  "stack_info": null,
  "cluster_host_id": "awx",
  "tower_uuid": null
}

 

 

 

job_events

• template 등의 job이 실행 될때 생성되는 로그 이벤트로 "event_data" 의 sub-dict 을 갖는다.

{
  "@timestamp": "2022-06-16T06:40:21.591Z",
  "message": "Event data saved.",
  "host": "",
  "level": "INFO",
  "logger_name": "awx.analytics.job_events",
  "id": null,
  "modified": null,
  "event": "playbook_on_task_start",
  "event_data": {
    "playbook": "playbook/04.windows_vss_backup.yml",
    "playbook_uuid": "9743065e-2c57-4629-b95f-6f2049dba3b6",
    "play": "all",
    "play_uuid": "0242ac13-0005-3fe4-c86f-000000000006",
    "play_pattern": "all",
    "task": "debug",
    "task_uuid": "0242ac13-0005-3fe4-c86f-000000000009",
    "task_action": "debug",
    "task_args": "",
    "task_path": "/tmp/bwrap_113_8rx1p1t5/awx_113_dc8ouo_m/project/playbook/04.windows_vss_backup.yml:10",
    "name": "debug",
    "is_conditional": false,
    "uuid": "0242ac13-0005-3fe4-c86f-000000000009"
  },
  "failed": false,
  "changed": false,
  "uuid": "0242ac13-0005-3fe4-c86f-000000000009",
  "playbook": "playbook/04.windows_vss_backup.yml",
  "play": "all",
  "role": "",
  "task": "debug",
  "counter": 6,
  "stdout": "\r\nTASK [debug] *******************************************************************",
  "verbosity": 0,
  "start_line": 5,
  "end_line": 7,
  "created": "2022-06-16T06:40:21.583Z",
  "job": 113,
  "host_name": "",
  "parent_uuid": "0242ac13-0005-3fe4-c86f-000000000006",
  "event_display": "Task Started (debug)",
  "cluster_host_id": "awx",
  "tower_uuid": null
}

 

 

 

activity_stream

• awx의 설정이 변경 될때마다 생성되는 로그 이벤트로 아래와 같은 스키마를 갖는다.
  • actor: Username of the user who took the action documented in the log
  • changes: JSON summary of what fields changed, and their old/new values.
  • operation: The basic category of the changed logged in the activity stream, for instance, “associate”.
  • object1: Information about the primary object being operated on, consistent with what we show in the activity stream
  • object2: If applicable, the second object involved in the action
• 아래는 inventory 내의 remote_server host를 비활성화(enable false) 했을때 생성되는 로그 이벤트이다.

{
  "@timestamp": "2022-06-17T02:07:44.926Z",
  "message": "Activity Stream update entry for host",
  "host": "awxweb",
  "level": "INFO",
  "logger_name": "awx.analytics.activity_stream",
  "stack_info": null,
  "changes": {
    "enabled": [
      true,
      false
    ]
  },
  "relationship": "",
  "actor": "admin",
  "operation": "update",
  "object1": "host",
  "object2": "",
  "summary_fields": {
    "host": [
      {
        "id": 11,
        "name": "remote_server",
        "description": "imported"
      }
    ],
    "actor": {
      "id": 1,
      "username": "admin",
      "first_name": "",
      "last_name": ""
    }
  },
  "cluster_host_id": "awx",
  "tower_uuid": null
}

 

 

Logstash Pipeline 설정

---

 

• Logstash에 기본 5044 포트를 사용하는 다른 설정들이 존재하여 input tags를 사용하여 filter, output에서 사용하기 편하게 구분지어 주었다.

input {
...
        tcp {
                port => 5045
                host => "0.0.0.0"
                tags => ["awx"]
        }
}

filter {
...
        if "awx" in [tags] {
                json {
                        source => "message"
                }
        }
...
}

output {
...
		if [tags] == "awx"{
        		elasticsearch {
                		hosts => "http://localhost:9200"
                		index => "awx-%{+YYYY.MM.dd}"
                }
        }
...
}

 

Elasticsearch Discover

• awx template job_events 가 실행되었을때 
  • inventory host "test_win", "remote_server"
  • playbook "04.windows_vss_backup.yml" task(once VSS backup) 실행