HomeAboutMeBlogGuest
© 2025 Sejin Cha. All rights reserved.
Built with Next.js, deployed on Vercel
✍🏻
Learnary (learn - diary)/
ec2가 다운된 이유를 알아보기

ec2가 다운된 이유를 알아보기

progress
Done
Tags
DevOps
  1. 시스템 로그 확인하기 (>> /var/log)
      2. notion image
      • SSH 세션 연결 로그 확인해보기
        • /var/log/auth.log
      • 시스템 시간 이벤트 로그 확인해보기
        • /var/log/syslog
       
      로그 검색 명령어
      sudo "${grep or zgrep}" "${key_word}" ${file_name}

dmesg | grep -i "oom"
      추가적으로 kern.log, dmesg 확인
 
1번에서 SSH, 시스템 이벤트 로그 확인 시 크론잡 세션이 열리고 닫히는 부분 확인
ex) pam_unix(cron:session) 관련 메시지가 많이 보임
 
커널로그 OOM발생
sudo grep -E -i "oom|memory|cpu" /var/log/kern.log.1
Sep 30 02:53:03 ip-172-26-10-138 kernel: [956040.859812] H2 Log Writer W invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-1000
Sep 30 02:53:03 ip-172-26-10-138 kernel: [956040.859819] CPU: 3 PID: 2258 Comm: H2 Log Writer W Tainted: G        W         5.15.0-1063-aws #69~20.04.1-Ubuntu
Sep 30 02:53:03 ip-172-26-10-138 kernel: [956040.859838]  oom_kill_process.cold+0xb/0x10
Sep 30 02:53:03 ip-172-26-10-138 kernel: [956040.859840]  out_of_memory+0x1dc/0x530
Sep 30 02:53:03 ip-172-26-10-138 kernel: [956040.859963] Tasks state (memory values in pages):
Sep 30 02:53:03 ip-172-26-10-138 kernel: [956040.859964] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sep 30 02:53:03 ip-172-26-10-138 kernel: [956040.860176] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/af13515ef59c2268f77db0c07ea2d9c936b8c41c3bd8f132a96403a1de2808db,task=uvicorn,pid=2828925,uid=0
Sep 30 02:53:03 ip-172-26-10-138 kernel: [956040.860286] Out of memory: Killed process 2828925 (uvicorn) total-vm:6182800kB, anon-rss:1401972kB, file-rss:0kB, shmem-rss:380kB, UID:0 pgtables:3600kB oom_score_adj:0
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.000000] KERNEL supported cpus:
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.010910] ACPI: Reserving FACP table memory at [mem 0xfc00d3f0-0xfc00d4e3]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.010911] ACPI: Reserving DSDT table memory at [mem 0xfc001000-0xfc00cd13]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.010912] ACPI: Reserving FACS table memory at [mem 0xfc00d2f0-0xfc00d32f]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.010913] ACPI: Reserving FACS table memory at [mem 0xfc00d2f0-0xfc00d32f]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.010914] ACPI: Reserving SSDT table memory at [mem 0xfc00cd20-0xfc00cd52]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.010915] ACPI: Reserving SSDT table memory at [mem 0xfc00cd60-0xfc00cd90]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.010916] ACPI: Reserving APIC table memory at [mem 0xfc00cda0-0xfc00ce77]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.010917] ACPI: Reserving HPET table memory at [mem 0xfc00d280-0xfc00d2b7]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.010918] ACPI: Reserving WAET table memory at [mem 0xfc00d2c0-0xfc00d2e7]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.011482] Early memory node ranges
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044171] smpboot: Allowing 15 CPUs, 11 hotplug CPUs
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044180] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044182] PM: hibernation: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044183] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000dffff]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044184] PM: hibernation: Registered nosave memory: [mem 0x000e0000-0x000fffff]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044185] PM: hibernation: Registered nosave memory: [mem 0xf0000000-0xfbffffff]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044186] PM: hibernation: Registered nosave memory: [mem 0xfc000000-0xffffffff]
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044198] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:15 nr_cpu_ids:15 nr_node_ids:1
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044743] percpu: Embedded 62 pages/cpu s217088 r8192 d28672 u262144
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044749] pcpu-alloc: s217088 r8192 d28672 u262144 alloc=1*2097152
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.044751] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 -- 
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.098190] Memory: 16353572K/16776820K available (16393K kernel code, 4399K rwdata, 10936K rodata, 3368K init, 18700K bss, 422988K reserved, 0K cma-reserved)
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.098407] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=15, Nodes=1
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.125993] rcu:    RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=15.
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.125997] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=15
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.441849] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.442554] MMIO Stale Data: Vulnerable: Clear CPU buffers attempted, no microcode
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.473981] Freeing SMP alternatives memory: 44K
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.489719] Xen: using vcpuop timer interface
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.490555] installing Xen timer for CPU 0
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.492002] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz (family: 0x6, model: 0x4f, stepping: 0x1)
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.494553] cpu 0 spinlock event irq 53
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.494673] Performance Events: unsupported p6 CPU model 79 no PMU driver, software events only.
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.502253] smp: Bringing up secondary CPUs ...
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.503814] installing Xen timer for CPU 1
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.506563] .... node  #0, CPUs:        #1
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.509397] cpu 1 spinlock event irq 59
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.511927] installing Xen timer for CPU 2
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.517688] cpu 2 spinlock event irq 65
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.520211] installing Xen timer for CPU 3
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.525680] cpu 3 spinlock event irq 71
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.527642] smp: Brought up 1 node, 4 CPUs
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.535799] x86/mm: Memory block size: 128MB
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.565532] cpuidle: using governor ladder
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.566560] cpuidle: using governor menu
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.788021] hpet: 3 channels of 0 reserved for per-cpu timers
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    0.917647] Freeing initrd memory: 7856K
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    1.092305] intel_pstate: CPU model not supported
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    1.094060] ledtrig-cpu: registered to indicate activity on CPUs
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    1.700351] Freeing unused decrypted memory: 2036K
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    1.703028] Freeing unused kernel image (initmem) memory: 3368K
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    1.707534] Freeing unused kernel image (text/rodata gap) memory: 2036K
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    1.709773] Freeing unused kernel image (rodata/data gap) memory: 1352K
Sep 30 14:19:15 ip-172-26-10-138 kernel: [    3.055721] cryptd: max_cpu_qlen set to 1000
Oct  1 17:09:43 ip-172-26-10-138 kernel: [96639.467529] clocksource: timekeeping watchdog on CPU3: xen wd-wd read-back delay of 152351ns
Oct  1 22:53:54 ip-172-26-10-138 kernel: [117290.243153] clocksource: timekeeping watchdog on CPU0: xen wd-wd read-back delay of 70706ns
Oct  2 15:37:42 ip-172-26-10-138 kernel: [177519.543147] clocksource: timekeeping watchdog on CPU1: xen wd-wd read-back delay of 994773ns
Oct  5 19:18:45 ip-172-26-10-138 kernel: [449984.060550] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Oct  5 19:18:45 ip-172-26-10-138 kernel: [449984.060557] CPU: 1 PID: 1 Comm: systemd Not tainted 5.15.0-1070-aws #76~20.04.1-Ubuntu
Oct  5 19:18:45 ip-172-26-10-138 kernel: [449984.060577]  oom_kill_process.cold+0xb/0x10
Oct  5 19:18:45 ip-172-26-10-138 kernel: [449984.060579]  out_of_memory+0x1dc/0x530
Oct  5 19:18:45 ip-172-26-10-138 kernel: [449984.060691] Tasks state (memory values in pages):
Oct  5 19:18:45 ip-172-26-10-138 kernel: [449984.060692] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Oct  5 19:18:45 ip-172-26-10-138 kernel: [449984.060918] oom-kill:constraint=CONSTRAI
 
 
1. 첫 번째 OOM 발생 (Sep 30 02:53:03)메모리 부족 상황
  • 로그에 따르면, uvicorn이라는 프로세스가 종료되었으며, 이는 메모리를 많이 사용한 프로세스로 보인다 (total-vm:6182800kB, anon-rss:1401972kB).
2. 두 번째 OOM 발생 (Oct 5 19:18:45)
  • 프로세스: node (Docker 컨테이너에서 실행 중인 프로세스) 커널 로그에서 이 프로세스가 메모리를 과도하게 사용한 것으로 나타남. (total-vm:8748712kB, anon-rss:7606104kB). 이 프로세스가 시스템 메모리의 대부분을 차지하고 있어, 커널이 이를 종료함으로써 메모리 자원을 확보하려 한 것.
 
 

크론잡 확인

  • 시스템 크론잡 확인
sudo ls /etc/cron.hourly/
sudo ls /etc/cron.daily/
sudo ls /etc/cron.weekly/
sudo ls /etc/cron.monthly/
sudo crontab -l
  • 사용자 크론잡 확인
sudo crontab -u root -l
sudo crontab -u ubuntu -l
  • 크론 로그 확인
      • sudo grep CRON /var/log/syslog
  • 특정 시간대로 로그 목록 좁혀서 보기
      • sudo grep "Oct  8 00:28" /var/log/syslog
 
 
 
해결방안
  • Docker 컨테이너 리소스 제한
      • 메모리
      docker run -m 512m --memory-swap 512m your_container_image
      CPU
      docker run -d --name myapp-container --network hm-network --memory 512m --cpus 1.0 myapp
  • 스왑메모리 사용 설정
      • 메모리 정보 확인 → >> free -h
      시스템 메모리 실제 확인 → >> cat /proc/meminfo
      실시간 모니터링 인터페이스 → >> top
      sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# /etc/fstab에 추가하여 재부팅 후 스왑 유지되도록 설정
/swapfile none swap sw 0 0
 
 

문제발생

10월 5일 17시부터 SSH 접속이 끊어지기 시작됬다.
  • 갑자기 17시에서 19시로 건너뛰는 시점이다.
auth.log
>> cat -n  /var/log/auth.log.1
8966  Oct  5 17:00:01 ip-172-26-10-138 CRON[1516474]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
  8967  Oct  5 17:03:31 ip-172-26-10-138 CRON[1516474]: pam_unix(cron:session): session closed for user ubuntu
  8968  Oct  5 17:05:01 ip-172-26-10-138 CRON[1517445]: pam_unix(cron:session): session opened for user root by (uid=0)
  8969  Oct  5 17:05:01 ip-172-26-10-138 CRON[1517445]: pam_unix(cron:session): session closed for user root
  8970  Oct  5 17:06:18 ip-172-26-10-138 sshd[1517733]: Connection closed by 64.120.116.26 port 33764 [preauth]
  8971  Oct  5 17:15:01 ip-172-26-10-138 CRON[1519157]: pam_unix(cron:session): session opened for user root by (uid=0)
  8972  Oct  5 17:15:01 ip-172-26-10-138 CRON[1519157]: pam_unix(cron:session): session closed for user root
  8973  Oct  5 17:17:01 ip-172-26-10-138 CRON[1519503]: pam_unix(cron:session): session opened for user root by (uid=0)
  8974  Oct  5 17:17:01 ip-172-26-10-138 CRON[1519503]: pam_unix(cron:session): session closed for user root
  8975  Oct  5 17:19:37 ip-172-26-10-138 sshd[1519991]: Accepted publickey for ubuntu from 115.138.69.42 port 62296 ssh2: RSA SHA256:UNmT/mWkwP8zD4F/f0uLh2I5LFT5sm3muDOwhQ0taDo
  8976  Oct  5 17:19:37 ip-172-26-10-138 sshd[1519991]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)
  8977  Oct  5 17:19:37 ip-172-26-10-138 systemd-logind[580]: New session 959 of user ubuntu.
  8978  Oct  5 17:19:48 ip-172-26-10-138 sshd[1520088]: Received disconnect from 115.138.69.42 port 62296:11: disconnected by user
  8979  Oct  5 17:19:48 ip-172-26-10-138 sshd[1520088]: Disconnected from user ubuntu 115.138.69.42 port 62296
  8980  Oct  5 17:19:48 ip-172-26-10-138 sshd[1519991]: pam_unix(sshd:session): session closed for user ubuntu
  8981  Oct  5 17:19:48 ip-172-26-10-138 systemd-logind[580]: Session 959 logged out. Waiting for processes to exit.
  8982  Oct  5 17:19:55 ip-172-26-10-138 sshd[1520275]: Accepted publickey for ubuntu from 115.138.69.42 port 62324 ssh2: RSA SHA256:UNmT/mWkwP8zD4F/f0uLh2I5LFT5sm3muDOwhQ0taDo
  8983  Oct  5 17:19:55 ip-172-26-10-138 sshd[1520275]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)
  8984  Oct  5 17:19:55 ip-172-26-10-138 systemd-logind[580]: New session 960 of user ubuntu.
  8985  Oct  5 17:25:01 ip-172-26-10-138 CRON[1521656]: pam_unix(cron:session): session opened for user root by (uid=0)
  8986  Oct  5 17:25:01 ip-172-26-10-138 CRON[1521656]: pam_unix(cron:session): session closed for user root
  8987  Oct  5 17:35:01 ip-172-26-10-138 CRON[1527129]: pam_unix(cron:session): session opened for user root by (uid=0)
  8988  Oct  5 17:35:01 ip-172-26-10-138 CRON[1527129]: pam_unix(cron:session): session closed for user root
  8989  Oct  5 17:38:33 ip-172-26-10-138 sshd[1530132]: error: kex_exchange_identification: Connection closed by remote host
  8990  Oct  5 17:38:33 ip-172-26-10-138 sshd[1530168]: Connection closed by 165.154.41.47 port 46970 [preauth]
  8991  Oct  5 17:38:34 ip-172-26-10-138 sshd[1530177]: Connection closed by 165.154.41.47 port 47350 [preauth]
  8992  Oct  5 17:41:25 ip-172-26-10-138 sudo:   ubuntu : TTY=pts/2 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/sbin/ufw status
  8993  Oct  5 17:41:25 ip-172-26-10-138 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
  8994  Oct  5 17:41:26 ip-172-26-10-138 sudo: pam_unix(sudo:session): session closed for user root
  8995  Oct  5 19:18:50 ip-172-26-10-138 sshd[1504311]: Received disconnect from 115.138.69.42 port 58179:11: disconnected by user
  8996  Oct  5 19:18:50 ip-172-26-10-138 systemd-logind[580]: Failed to abandon session scope, ignoring: Connection timed out
  8997  Oct  5 19:18:51 ip-172-26-10-138 sshd[1510211]: Received disconnect from 115.138.69.42 port 60929:11: disconnected by user
  8998  Oct  5 19:18:51 ip-172-26-10-138 systemd-logind[580]: Session 951 logged out. Waiting for processes to exit.
  8999  Oct  5 19:18:51 ip-172-26-10-138 sshd[1504311]: Disconnected from user ubuntu 115.138.69.42 port 58179
  9000  Oct  5 19:18:51 ip-172-26-10-138 systemd-logind[580]: Failed to abandon session scope, ignoring: Connection timed out
  9001  Oct  5 19:18:51 ip-172-26-10-138 sshd[1510211]: Disconnected from user ubuntu 115.138.69.42 port 60929
  9002  Oct  5 19:18:51 ip-172-26-10-138 systemd-logind[580]: Session 945 logged out. Waiting for processes to exit.
  9003  Oct  5 19:18:51 ip-172-26-10-138 sshd[1510136]: pam_unix(sshd:session): session closed for user ubuntu
  9004  Oct  5 19:18:51 ip-172-26-10-138 systemd-logind[580]: Failed to abandon session scope, ignoring: Connection timed out
  9005  Oct  5 19:18:51 ip-172-26-10-138 sshd[1504091]: pam_unix(sshd:session): session closed for user ubuntu
  9006  Oct  5 19:18:51 ip-172-26-10-138 systemd-logind[580]: Session 960 logged out. Waiting for processes to exit.
  9007  Oct  5 19:18:51 ip-172-26-10-138 sshd[1520339]: Received disconnect from 115.138.69.42 port 62324:11: disconnected by user
  9008  Oct  5 19:18:51 ip-172-26-10-138 sshd[1520339]: Disconnected from user ubuntu 115.138.69.42 port 62324
  9009  Oct  5 19:18:51 ip-172-26-10-138 sshd[1520275]: pam_unix(sshd:session): session closed for user ubuntu
  9010  Oct  5 19:18:51 ip-172-26-10-138 sshd[1510136]: pam_systemd(sshd:session): Failed to release session: Connection reset by peer
  9011  Oct  5 19:18:51 ip-172-26-10-138 sshd[1504091]: pam_systemd(sshd:session): Failed to release session: Connection timed out
  9012  Oct  5 19:18:51 ip-172-26-10-138 sshd[1520275]: pam_systemd(sshd:session): Failed to release session: Connection timed out
  9013  Oct  5 19:18:51 ip-172-26-10-138 systemd-logind[580]: Removed session 945.
  9014  Oct  5 19:18:51 ip-172-26-10-138 CRON[1532263]: pam_unix(cron:session): session opened for user root by (uid=0)
  9015  Oct  5 19:18:51 ip-172-26-10-138 systemd-logind[580]: Removed session 951.
  9016  Oct  5 19:18:51 ip-172-26-10-138 CRON[1532263]: pam_unix(cron:session): session closed for user root
  9017  Oct  5 19:18:51 ip-172-26-10-138 CRON[1532276]: pam_unix(cron:session): session opened for user root by (uid=0)
  9018  Oct  5 19:18:51 ip-172-26-10-138 CRON[1532276]: pam_unix(cron:session): session closed for user root
  9019  Oct  5 19:18:51 ip-172-26-10-138 CRON[1532288]: pam_unix(cron:session): session opened for user root by (uid=0)
  9020  Oct  5 19:18:51 ip-172-26-10-138 CRON[1532281]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
  9021  Oct  5 19:18:51 ip-172-26-10-138 CRON[1532288]: pam_unix(cron:session): session closed for user root
 
SSH도 결국 메모리 점유가 필요하기 때문에 이 부분을 의심해보고 디버깅하기 시작했다.
>> sudo grep -i "out of memory" /var/log/kern.log.1
Sep 30 02:53:03 ip-172-26-10-138 kernel: [956040.860286] Out of memory: Killed process 2828925 (uvicorn) total-vm:6182800kB, anon-rss:1401972kB, file-rss:0kB, shmem-rss:380kB, UID:0 pgtables:3600kB oom_score_adj:0
Oct  5 19:18:45 ip-172-26-10-138 kernel: [449984.060940] Out of memory: Killed process 1524663 (uvicorn) total-vm:8748712kB, anon-rss:7606104kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:101908kB oom_score_adj:0

sudo zcat /var/log/syslog.2.gz | grep -i "out of memory"
Oct  5 19:18:45 ip-172-26-10-138 kernel: [449984.060940] Out of memory: Killed process 1524663 (uvicorn) total-vm:8748712kB, anon-rss:7606104kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:101908kB oom_score_adj:0
AI서버 1대가 실행될때 마다 CPU 리소스가 4코어 밖에 없는데, ram 16기가 스펙에서 oom이 발생했다
 
유비콘! FAST api에서 발생한것 같다
매번 크롤링함에 있어 파일을 읽어와 쓰고 하는 양이 믾아서 메모리에 적재하고 하드디스크로 넣는 작업이 꽤나 큰 리소스가 들어 oom이 발생하는 것 같다
 
해당 부분을 해결하기 위해 파이썬 크롤링을 파일을 읽고 쓰고 저장 이 흐름은 하나위 파이프라인으로 처리를 해야할것같다
모두 메모리에 로드하는게 아니라 로드되면서 바로바로 퍼다 날라줘야 한다고 생각했다
 

2차 죽음

 
Oct  7 20:26:22 ip-172-26-10-138 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct  7 20:26:22 ip-172-26-10-138 sudo: pam_unix(sudo:session): session closed for user root
Oct  8 00:28:25 ip-172-26-10-138 sshd[205513]: Received disconnect from 115.138.69.42 port 59784:11: disconnected by user
Oct  8 00:28:25 ip-172-26-10-138 systemd-logind[579]: Failed to abandon session scope, ignoring: Connection timed out
Oct  8 00:28:25 ip-172-26-10-138 sshd[205513]: Disconnected from user ubuntu 115.138.69.42 port 59784
Oct  8 00:28:25 ip-172-26-10-138 systemd-logind[579]: Session 125 logged out. Waiting for processes to exit.
Oct  8 00:28:25 ip-172-26-10-138 sshd[205416]: pam_unix(sshd:session): session closed for user ubuntu
Oct  8 00:28:26 ip-172-26-10-138 sshd[205416]: pam_systemd(sshd:session): Failed to release session: Connection timed out
Oct  8 00:28:26 ip-172-26-10-138 CRON[207589]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207589]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207610]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207610]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207626]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207626]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 systemd-logind[579]: Removed session 125.
Oct  8 00:28:26 ip-172-26-10-138 CRON[207637]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207645]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207645]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207663]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207663]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207670]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207670]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207674]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207674]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207686]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207686]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207698]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207698]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207709]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207709]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207714]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207720]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207720]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207734]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207732]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207732]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207734]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207740]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207747]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207740]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207747]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207750]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207750]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207758]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207758]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207762]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207762]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207768]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207769]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207775]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207768]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207769]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207775]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207761]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207779]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207779]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207785]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207794]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207790]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207796]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207807]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207814]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207803]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207809]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207796]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207790]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207794]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207807]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207814]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207803]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207785]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 CRON[207797]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207809]: pam_unix(cron:session): session closed for user root
Oct  8 00:28:26 ip-172-26-10-138 sshd[207671]: Connection closed by 172.104.11.46 port 27906 [preauth]
Oct  8 00:28:26 ip-172-26-10-138 sshd[207668]: Connection closed by 172.104.11.46 port 3774 [preauth]
Oct  8 00:28:26 ip-172-26-10-138 sshd[207664]: Connection closed by 172.104.11.46 port 3772 [preauth]
Oct  8 00:28:26 ip-172-26-10-138 CRON[207714]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 00:28:26 ip-172-26-10-138 CRON[207797]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 00:28:26 ip-172-26-10-138 CRON[207761]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 00:28:26 ip-172-26-10-138 CRON[207637]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 00:35:01 ip-172-26-10-138 CRON[209220]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:35:01 ip-172-26-10-138 CRON[209220]: pam_unix(cron:session): session closed for user root
Oct  8 00:45:01 ip-172-26-10-138 CRON[210957]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:45:01 ip-172-26-10-138 CRON[210957]: pam_unix(cron:session): session closed for user root
Oct  8 00:55:01 ip-172-26-10-138 CRON[212710]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 00:55:01 ip-172-26-10-138 CRON[212710]: pam_unix(cron:session): session closed for user root
Oct  8 01:00:01 ip-172-26-10-138 CRON[213592]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 01:00:01 ip-172-26-10-138 CRON[213592]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 01:05:01 ip-172-26-10-138 CRON[214478]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 01:05:01 ip-172-26-10-138 CRON[214478]: pam_unix(cron:session): session closed for user root
Oct  8 01:15:01 ip-172-26-10-138 CRON[216235]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 01:15:01 ip-172-26-10-138 CRON[216235]: pam_unix(cron:session): session closed for user root
Oct  8 01:17:01 ip-172-26-10-138 CRON[216589]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 01:17:01 ip-172-26-10-138 CRON[216589]: pam_unix(cron:session): session closed for user root
Oct  8 01:25:01 ip-172-26-10-138 CRON[217980]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 01:25:01 ip-172-26-10-138 CRON[217980]: pam_unix(cron:session): session closed for user root
Oct  8 01:35:01 ip-172-26-10-138 CRON[219629]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 01:35:01 ip-172-26-10-138 CRON[219629]: pam_unix(cron:session): session closed for user root
Oct  8 01:45:01 ip-172-26-10-138 CRON[221362]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 01:45:01 ip-172-26-10-138 CRON[221362]: pam_unix(cron:session): session closed for user root
Oct  8 01:55:01 ip-172-26-10-138 CRON[223121]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 01:55:01 ip-172-26-10-138 CRON[223121]: pam_unix(cron:session): session closed for user root
Oct  8 02:00:01 ip-172-26-10-138 CRON[223997]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 02:00:01 ip-172-26-10-138 CRON[223997]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 02:05:01 ip-172-26-10-138 CRON[224866]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 02:05:01 ip-172-26-10-138 CRON[224866]: pam_unix(cron:session): session closed for user root
Oct  8 02:15:01 ip-172-26-10-138 CRON[226602]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 02:15:01 ip-172-26-10-138 CRON[226602]: pam_unix(cron:session): session closed for user root
Oct  8 02:17:01 ip-172-26-10-138 CRON[226954]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 02:17:01 ip-172-26-10-138 CRON[226954]: pam_unix(cron:session): session closed for user root
Oct  8 02:25:01 ip-172-26-10-138 CRON[228346]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 02:25:01 ip-172-26-10-138 CRON[228346]: pam_unix(cron:session): session closed for user root
Oct  8 02:35:01 ip-172-26-10-138 CRON[230106]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 02:35:01 ip-172-26-10-138 CRON[230106]: pam_unix(cron:session): session closed for user root
Oct  8 02:45:01 ip-172-26-10-138 CRON[231832]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 02:45:01 ip-172-26-10-138 CRON[231832]: pam_unix(cron:session): session closed for user root
Oct  8 02:55:01 ip-172-26-10-138 CRON[233593]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 02:55:01 ip-172-26-10-138 CRON[233593]: pam_unix(cron:session): session closed for user root
Oct  8 03:00:01 ip-172-26-10-138 CRON[234482]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 03:00:02 ip-172-26-10-138 CRON[234482]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 03:05:01 ip-172-26-10-138 CRON[235370]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 03:05:01 ip-172-26-10-138 CRON[235370]: pam_unix(cron:session): session closed for user root
Oct  8 03:10:01 ip-172-26-10-138 CRON[236250]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 03:10:01 ip-172-26-10-138 CRON[236250]: pam_unix(cron:session): session closed for user root
Oct  8 03:15:01 ip-172-26-10-138 CRON[237111]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 03:15:01 ip-172-26-10-138 CRON[237111]: pam_unix(cron:session): session closed for user root
Oct  8 03:17:01 ip-172-26-10-138 CRON[237466]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 03:17:01 ip-172-26-10-138 CRON[237466]: pam_unix(cron:session): session closed for user root
Oct  8 03:25:01 ip-172-26-10-138 CRON[238744]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 03:25:01 ip-172-26-10-138 CRON[238744]: pam_unix(cron:session): session closed for user root
Oct  8 03:33:31 ip-172-26-10-138 systemd-logind[579]: Removed session 68.
Oct  8 03:35:01 ip-172-26-10-138 CRON[240481]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 03:35:01 ip-172-26-10-138 CRON[240481]: pam_unix(cron:session): session closed for user root
Oct  8 03:45:01 ip-172-26-10-138 CRON[242212]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 03:45:01 ip-172-26-10-138 CRON[242212]: pam_unix(cron:session): session closed for user root
Oct  8 03:55:01 ip-172-26-10-138 CRON[243907]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 03:55:01 ip-172-26-10-138 CRON[243907]: pam_unix(cron:session): session closed for user root
Oct  8 04:00:01 ip-172-26-10-138 CRON[244805]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 04:00:01 ip-172-26-10-138 CRON[244805]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 04:05:01 ip-172-26-10-138 CRON[245649]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 04:05:01 ip-172-26-10-138 CRON[245649]: pam_unix(cron:session): session closed for user root
Oct  8 04:15:01 ip-172-26-10-138 CRON[247339]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 04:15:01 ip-172-26-10-138 CRON[247339]: pam_unix(cron:session): session closed for user root
Oct  8 04:17:01 ip-172-26-10-138 CRON[247685]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 04:17:01 ip-172-26-10-138 CRON[247685]: pam_unix(cron:session): session closed for user root
Oct  8 04:25:01 ip-172-26-10-138 CRON[249079]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 04:25:01 ip-172-26-10-138 CRON[249079]: pam_unix(cron:session): session closed for user root
Oct  8 04:35:01 ip-172-26-10-138 CRON[250805]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 04:35:01 ip-172-26-10-138 CRON[250805]: pam_unix(cron:session): session closed for user root
Oct  8 04:45:01 ip-172-26-10-138 CRON[252508]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 04:45:01 ip-172-26-10-138 CRON[252508]: pam_unix(cron:session): session closed for user root
Oct  8 04:55:01 ip-172-26-10-138 CRON[254247]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 04:55:01 ip-172-26-10-138 CRON[254247]: pam_unix(cron:session): session closed for user root
Oct  8 05:00:01 ip-172-26-10-138 CRON[255108]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 05:00:01 ip-172-26-10-138 CRON[255108]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 05:05:01 ip-172-26-10-138 CRON[255958]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 05:05:01 ip-172-26-10-138 CRON[255958]: pam_unix(cron:session): session closed for user root
Oct  8 05:15:01 ip-172-26-10-138 CRON[257567]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 05:15:01 ip-172-26-10-138 CRON[257567]: pam_unix(cron:session): session closed for user root
Oct  8 05:17:01 ip-172-26-10-138 CRON[257910]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 05:17:01 ip-172-26-10-138 CRON[257910]: pam_unix(cron:session): session closed for user root
Oct  8 05:25:01 ip-172-26-10-138 CRON[259283]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 05:25:01 ip-172-26-10-138 CRON[259283]: pam_unix(cron:session): session closed for user root
Oct  8 05:35:01 ip-172-26-10-138 CRON[261005]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 05:35:01 ip-172-26-10-138 CRON[261005]: pam_unix(cron:session): session closed for user root
Oct  8 05:45:01 ip-172-26-10-138 CRON[262714]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 05:45:01 ip-172-26-10-138 CRON[262714]: pam_unix(cron:session): session closed for user root
Oct  8 05:55:01 ip-172-26-10-138 CRON[264446]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 05:55:01 ip-172-26-10-138 CRON[264446]: pam_unix(cron:session): session closed for user root
Oct  8 06:00:01 ip-172-26-10-138 CRON[265320]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 06:00:01 ip-172-26-10-138 CRON[265320]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 06:05:01 ip-172-26-10-138 CRON[266170]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 06:05:01 ip-172-26-10-138 CRON[266170]: pam_unix(cron:session): session closed for user root
Oct  8 06:15:01 ip-172-26-10-138 CRON[267877]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 06:15:01 ip-172-26-10-138 CRON[267877]: pam_unix(cron:session): session closed for user root
Oct  8 06:17:01 ip-172-26-10-138 CRON[268290]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 06:17:01 ip-172-26-10-138 CRON[268290]: pam_unix(cron:session): session closed for user root
Oct  8 06:25:01 ip-172-26-10-138 CRON[269657]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 06:25:01 ip-172-26-10-138 CRON[269658]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 06:25:01 ip-172-26-10-138 CRON[269658]: pam_unix(cron:session): session closed for user root
Oct  8 06:25:01 ip-172-26-10-138 CRON[269657]: pam_unix(cron:session): session closed for user root
Oct  8 06:35:01 ip-172-26-10-138 CRON[271438]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 06:35:01 ip-172-26-10-138 CRON[271438]: pam_unix(cron:session): session closed for user root
Oct  8 06:45:01 ip-172-26-10-138 CRON[273131]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 06:45:01 ip-172-26-10-138 CRON[273131]: pam_unix(cron:session): session closed for user root
Oct  8 06:55:01 ip-172-26-10-138 CRON[274833]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 06:55:01 ip-172-26-10-138 CRON[274833]: pam_unix(cron:session): session closed for user root
Oct  8 07:00:01 ip-172-26-10-138 CRON[275694]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 07:00:01 ip-172-26-10-138 CRON[275694]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 07:05:01 ip-172-26-10-138 CRON[276426]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 07:05:01 ip-172-26-10-138 CRON[276426]: pam_unix(cron:session): session closed for user root
Oct  8 07:15:01 ip-172-26-10-138 CRON[278156]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 07:15:01 ip-172-26-10-138 CRON[278156]: pam_unix(cron:session): session closed for user root
Oct  8 07:17:01 ip-172-26-10-138 CRON[278492]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 07:17:01 ip-172-26-10-138 CRON[278492]: pam_unix(cron:session): session closed for user root
Oct  8 07:25:01 ip-172-26-10-138 CRON[279887]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 07:25:01 ip-172-26-10-138 CRON[279887]: pam_unix(cron:session): session closed for user root
Oct  8 07:35:01 ip-172-26-10-138 CRON[281628]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 07:35:01 ip-172-26-10-138 CRON[281628]: pam_unix(cron:session): session closed for user root
Oct  8 07:45:01 ip-172-26-10-138 CRON[283343]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 07:45:01 ip-172-26-10-138 CRON[283343]: pam_unix(cron:session): session closed for user root
Oct  8 07:55:01 ip-172-26-10-138 CRON[285060]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 07:55:01 ip-172-26-10-138 CRON[285060]: pam_unix(cron:session): session closed for user root
Oct  8 08:00:01 ip-172-26-10-138 CRON[285906]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 08:00:01 ip-172-26-10-138 CRON[285906]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 08:05:01 ip-172-26-10-138 CRON[286763]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 08:05:01 ip-172-26-10-138 CRON[286763]: pam_unix(cron:session): session closed for user root
Oct  8 08:15:01 ip-172-26-10-138 CRON[288474]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 08:15:01 ip-172-26-10-138 CRON[288474]: pam_unix(cron:session): session closed for user root
Oct  8 08:17:01 ip-172-26-10-138 CRON[288825]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 08:17:01 ip-172-26-10-138 CRON[288825]: pam_unix(cron:session): session closed for user root
Oct  8 08:25:01 ip-172-26-10-138 CRON[290207]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 08:25:01 ip-172-26-10-138 CRON[290207]: pam_unix(cron:session): session closed for user root
Oct  8 08:35:01 ip-172-26-10-138 CRON[291909]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 08:35:01 ip-172-26-10-138 CRON[291909]: pam_unix(cron:session): session closed for user root
Oct  8 08:45:01 ip-172-26-10-138 CRON[293607]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 08:45:01 ip-172-26-10-138 CRON[293607]: pam_unix(cron:session): session closed for user root
Oct  8 08:55:01 ip-172-26-10-138 CRON[295190]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  8 08:55:01 ip-172-26-10-138 CRON[295190]: pam_unix(cron:session): session closed for user root
Oct  8 09:00:01 ip-172-26-10-138 CRON[296035]: pam_unix(cron:session): session opened for user ubuntu by (uid=0)
Oct  8 09:00:01 ip-172-26-10-138 CRON[296035]: pam_unix(cron:session): session closed for user ubuntu
Oct  8 09:02:49 ip-172-26-10-138 systemd-logind[576]: Watching system buttons on /dev/input/event0 (Power Button)
Oct  8 09:02:49 ip-172-26-10-138 systemd-logind[576]: Watching system buttons on /dev/input/event1 (Sleep Button)
Oct  8 09:02:49 ip-172-26-10-138 systemd-logind[576]: Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard)
Oct  8 09:02:49 ip-172-26-10-138 systemd-logind[576]: New seat seat0.
 
SSH 연결 종료 메시지
Oct  8 00:28:25 ip-172-26-10-138 sshd[205513]: Received disconnect from 115.138.69.42 port 59784:11: disconnected by user
Oct  8 00:28:25 ip-172-26-10-138 sshd[205513]: Disconnected from user ubuntu 115.138.69.42 port 59784
Oct  8 00:28:25 ip-172-26-10-138 sshd[205416]: pam_unix(sshd:session): session closed for user ubuntu
 
systemd-logind 관련 오류 메시지
Oct  8 00:28:25 ip-172-26-10-138 systemd-logind[579]: Failed to abandon session scope, ignoring: Connection timed out
Oct  8 00:28:25 ip-172-26-10-138 systemd-logind[579]: Session 125 logged out. Waiting for processes to exit.
Oct  8 00:28:26 ip-172-26-10-138 sshd[205416]: pam_systemd(sshd:session): Failed to release session: Connection timed out
 
 
⇒ systemd-logind에서 세션 해제에 실패했다는 메시지가 반복적으로 나타난ㄷ.
 
CRON 세션이 다수 열리고 닫힘 ⇒ CRON 세션 폭주
로그의 대부분이 CRON 세션이 열리고 닫히는 내용으로 크론 잡이 매우 빈번하게 실행되고 있음을 나타낸다.
 
정상적인 동작으로 보임과 동시에 시스템에 부하를 줄 수 있다고 생각했다.
systemd-logind의 타임아웃 오류
systemd-logind에서 세션 해제에 실패하고 타임아웃이 발생하는 것은 시스템 리소스가 부족하거나 프로세스가 정상적으로 종료되지 않을 때 발생하는 것 같다.
 
 
ubuntu 사용자 크론 잡
sudo crontab -u ubuntu -l
# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command
0 * * * * docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling
 
 
크론잡 로그 확인
sudo grep CRON /var/log/syslog
Oct  8 00:28:26 ip-172-26-10-138 CRON[207603]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207620]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207638]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207650]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 00:28:26 ip-172-26-10-138 CRON[207657]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207679]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207684]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207693]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207705]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207715]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207719]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207729]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 00:28:26 ip-172-26-10-138 CRON[207741]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207754]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207755]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207763]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207767]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207774]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207780]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207784]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207793]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207795]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207801]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207806]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 00:28:26 ip-172-26-10-138 CRON[207808]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207816]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207838]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207839]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 60 2)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207840]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207859]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207860]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207861]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207862]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207870]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 00:28:26 ip-172-26-10-138 CRON[207714]: (CRON) info (No MTA installed, discarding output)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207797]: (CRON) info (No MTA installed, discarding output)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207761]: (CRON) info (No MTA installed, discarding output)
Oct  8 00:28:26 ip-172-26-10-138 CRON[207637]: (CRON) info (No MTA installed, discarding output)
Oct  8 00:35:01 ip-172-26-10-138 CRON[209221]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:45:01 ip-172-26-10-138 CRON[210958]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 00:55:01 ip-172-26-10-138 CRON[212711]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 01:00:01 ip-172-26-10-138 CRON[213593]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 01:00:01 ip-172-26-10-138 CRON[213592]: (CRON) info (No MTA installed, discarding output)
Oct  8 01:05:01 ip-172-26-10-138 CRON[214479]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 01:15:01 ip-172-26-10-138 CRON[216236]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 01:17:01 ip-172-26-10-138 CRON[216590]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 01:25:01 ip-172-26-10-138 CRON[217981]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 01:35:01 ip-172-26-10-138 CRON[219630]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 01:45:01 ip-172-26-10-138 CRON[221363]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 01:55:01 ip-172-26-10-138 CRON[223122]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 02:00:01 ip-172-26-10-138 CRON[223998]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 02:00:01 ip-172-26-10-138 CRON[223997]: (CRON) info (No MTA installed, discarding output)
Oct  8 02:05:01 ip-172-26-10-138 CRON[224867]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 02:15:01 ip-172-26-10-138 CRON[226603]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 02:17:01 ip-172-26-10-138 CRON[226955]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 02:25:01 ip-172-26-10-138 CRON[228347]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 02:35:01 ip-172-26-10-138 CRON[230107]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 02:45:01 ip-172-26-10-138 CRON[231833]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 02:55:01 ip-172-26-10-138 CRON[233594]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 03:00:01 ip-172-26-10-138 CRON[234483]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 03:00:02 ip-172-26-10-138 CRON[234482]: (CRON) info (No MTA installed, discarding output)
Oct  8 03:05:01 ip-172-26-10-138 CRON[235371]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 03:10:01 ip-172-26-10-138 CRON[236251]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Oct  8 03:15:01 ip-172-26-10-138 CRON[237112]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 03:17:01 ip-172-26-10-138 CRON[237467]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 03:25:01 ip-172-26-10-138 CRON[238745]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 03:35:01 ip-172-26-10-138 CRON[240482]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 03:45:01 ip-172-26-10-138 CRON[242213]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 03:55:01 ip-172-26-10-138 CRON[243908]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 04:00:01 ip-172-26-10-138 CRON[244806]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 04:00:01 ip-172-26-10-138 CRON[244805]: (CRON) info (No MTA installed, discarding output)
Oct  8 04:05:01 ip-172-26-10-138 CRON[245650]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 04:15:01 ip-172-26-10-138 CRON[247340]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 04:17:01 ip-172-26-10-138 CRON[247686]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 04:25:01 ip-172-26-10-138 CRON[249080]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 04:35:01 ip-172-26-10-138 CRON[250806]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 04:45:01 ip-172-26-10-138 CRON[252509]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 04:55:01 ip-172-26-10-138 CRON[254248]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 05:00:01 ip-172-26-10-138 CRON[255109]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 05:00:01 ip-172-26-10-138 CRON[255108]: (CRON) info (No MTA installed, discarding output)
Oct  8 05:05:01 ip-172-26-10-138 CRON[255959]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 05:15:01 ip-172-26-10-138 CRON[257568]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 05:17:01 ip-172-26-10-138 CRON[257911]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 05:25:01 ip-172-26-10-138 CRON[259284]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 05:35:01 ip-172-26-10-138 CRON[261006]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 05:45:01 ip-172-26-10-138 CRON[262715]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 05:55:01 ip-172-26-10-138 CRON[264447]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 06:00:01 ip-172-26-10-138 CRON[265321]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 06:00:01 ip-172-26-10-138 CRON[265320]: (CRON) info (No MTA installed, discarding output)
Oct  8 06:05:01 ip-172-26-10-138 CRON[266171]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 06:15:01 ip-172-26-10-138 CRON[267878]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 06:17:01 ip-172-26-10-138 CRON[268291]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 06:25:01 ip-172-26-10-138 CRON[269660]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Oct  8 06:25:01 ip-172-26-10-138 CRON[269659]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 06:35:01 ip-172-26-10-138 CRON[271439]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 06:45:01 ip-172-26-10-138 CRON[273132]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 06:55:01 ip-172-26-10-138 CRON[274834]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 07:00:01 ip-172-26-10-138 CRON[275695]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 07:00:01 ip-172-26-10-138 CRON[275694]: (CRON) info (No MTA installed, discarding output)
Oct  8 07:05:01 ip-172-26-10-138 CRON[276427]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 07:15:01 ip-172-26-10-138 CRON[278157]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 07:17:01 ip-172-26-10-138 CRON[278493]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 07:25:01 ip-172-26-10-138 CRON[279888]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 07:35:01 ip-172-26-10-138 CRON[281629]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 07:45:01 ip-172-26-10-138 CRON[283344]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 07:55:01 ip-172-26-10-138 CRON[285061]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 08:00:01 ip-172-26-10-138 CRON[285907]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 08:00:01 ip-172-26-10-138 CRON[285906]: (CRON) info (No MTA installed, discarding output)
Oct  8 08:05:01 ip-172-26-10-138 CRON[286764]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 08:15:01 ip-172-26-10-138 CRON[288475]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 08:17:01 ip-172-26-10-138 CRON[288826]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 08:25:01 ip-172-26-10-138 CRON[290208]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 08:35:01 ip-172-26-10-138 CRON[291910]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 08:45:01 ip-172-26-10-138 CRON[293608]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 08:55:01 ip-172-26-10-138 CRON[295191]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 09:00:01 ip-172-26-10-138 CRON[296036]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 09:00:01 ip-172-26-10-138 CRON[296035]: (CRON) info (No MTA installed, discarding output)
Oct  8 09:02:49 ip-172-26-10-138 cron[532]: (CRON) INFO (pidfile fd = 3)
Oct  8 09:02:49 ip-172-26-10-138 cron[532]: (CRON) INFO (Running @reboot jobs)
Oct  8 09:05:01 ip-172-26-10-138 CRON[4925]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 09:15:01 ip-172-26-10-138 CRON[8789]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 09:17:01 ip-172-26-10-138 CRON[9129]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 09:25:01 ip-172-26-10-138 CRON[11546]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 09:35:01 ip-172-26-10-138 CRON[14043]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 09:45:01 ip-172-26-10-138 CRON[15822]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 09:55:01 ip-172-26-10-138 CRON[19593]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 10:00:01 ip-172-26-10-138 CRON[22363]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  8 10:05:01 ip-172-26-10-138 CRON[24348]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 10:05:06 ip-172-26-10-138 CRON[22362]: (CRON) info (No MTA installed, discarding output)
Oct  8 10:15:01 ip-172-26-10-138 CRON[26705]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  8 10:17:01 ip-172-26-10-138 CRON[27691]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  8 10:25:01 ip-172-26-10-138 CRON[30830]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
 
 
syslog1 죽은 시간 이후 크롤링이 안돌아간 로그
 
sudo grep CRON /var/log/syslog.1
Oct  7 00:00:03 ip-172-26-10-138 CRON[1829187]: (CRON) info (No MTA installed, discarding output)
Oct  7 00:05:01 ip-172-26-10-138 CRON[1830162]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 00:15:01 ip-172-26-10-138 CRON[1831763]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 00:17:01 ip-172-26-10-138 CRON[1832116]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 00:25:01 ip-172-26-10-138 CRON[1833475]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 00:35:01 ip-172-26-10-138 CRON[1835172]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 00:45:01 ip-172-26-10-138 CRON[1836877]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 00:55:01 ip-172-26-10-138 CRON[1838552]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 01:00:01 ip-172-26-10-138 CRON[1839410]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 01:00:02 ip-172-26-10-138 CRON[1839409]: (CRON) info (No MTA installed, discarding output)
Oct  7 01:05:01 ip-172-26-10-138 CRON[1840382]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 01:15:01 ip-172-26-10-138 CRON[1842078]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 01:17:01 ip-172-26-10-138 CRON[1842422]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 01:25:01 ip-172-26-10-138 CRON[1843777]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 01:35:01 ip-172-26-10-138 CRON[1845469]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 01:45:01 ip-172-26-10-138 CRON[1847176]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 01:55:01 ip-172-26-10-138 CRON[1848773]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 02:00:01 ip-172-26-10-138 CRON[1849641]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 02:00:03 ip-172-26-10-138 CRON[1849640]: (CRON) info (No MTA installed, discarding output)
Oct  7 02:05:01 ip-172-26-10-138 CRON[1850608]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 02:15:01 ip-172-26-10-138 CRON[1852298]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 02:17:01 ip-172-26-10-138 CRON[1852633]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 02:25:01 ip-172-26-10-138 CRON[1854007]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 02:35:01 ip-172-26-10-138 CRON[1855706]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 02:45:01 ip-172-26-10-138 CRON[1857395]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 02:55:01 ip-172-26-10-138 CRON[1859089]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 03:00:01 ip-172-26-10-138 CRON[1859952]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 03:00:03 ip-172-26-10-138 CRON[1859951]: (CRON) info (No MTA installed, discarding output)
Oct  7 03:05:01 ip-172-26-10-138 CRON[1860939]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 03:10:01 ip-172-26-10-138 CRON[1861795]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Oct  7 03:15:01 ip-172-26-10-138 CRON[1862639]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 03:17:01 ip-172-26-10-138 CRON[1862981]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 03:25:01 ip-172-26-10-138 CRON[1864342]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 03:35:01 ip-172-26-10-138 CRON[1866039]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 03:45:01 ip-172-26-10-138 CRON[1867627]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 03:55:01 ip-172-26-10-138 CRON[1869326]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 04:00:01 ip-172-26-10-138 CRON[1870178]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 04:00:03 ip-172-26-10-138 CRON[1870177]: (CRON) info (No MTA installed, discarding output)
Oct  7 04:05:01 ip-172-26-10-138 CRON[1871144]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 04:15:01 ip-172-26-10-138 CRON[1872864]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 04:17:01 ip-172-26-10-138 CRON[1873212]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 04:25:01 ip-172-26-10-138 CRON[1874581]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 04:35:01 ip-172-26-10-138 CRON[1876294]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 04:45:01 ip-172-26-10-138 CRON[1877995]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 04:55:01 ip-172-26-10-138 CRON[1879722]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 05:00:01 ip-172-26-10-138 CRON[1880596]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 05:00:02 ip-172-26-10-138 CRON[1880595]: (CRON) info (No MTA installed, discarding output)
Oct  7 05:05:01 ip-172-26-10-138 CRON[1881572]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 05:15:01 ip-172-26-10-138 CRON[1883275]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 05:17:01 ip-172-26-10-138 CRON[1883619]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 05:25:01 ip-172-26-10-138 CRON[1884856]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 05:35:01 ip-172-26-10-138 CRON[1886539]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 05:45:01 ip-172-26-10-138 CRON[1888250]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 05:55:01 ip-172-26-10-138 CRON[1889957]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 06:00:02 ip-172-26-10-138 CRON[1890809]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 06:00:03 ip-172-26-10-138 CRON[1890808]: (CRON) info (No MTA installed, discarding output)
Oct  7 06:05:01 ip-172-26-10-138 CRON[1891782]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 06:15:01 ip-172-26-10-138 CRON[1893474]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 06:17:01 ip-172-26-10-138 CRON[1893824]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 06:25:01 ip-172-26-10-138 CRON[1895191]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 06:25:01 ip-172-26-10-138 CRON[1895190]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Oct  7 06:35:01 ip-172-26-10-138 CRON[1896958]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 06:45:01 ip-172-26-10-138 CRON[1898668]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 06:55:01 ip-172-26-10-138 CRON[1900456]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 07:00:01 ip-172-26-10-138 CRON[1901304]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 07:00:03 ip-172-26-10-138 CRON[1901303]: (CRON) info (No MTA installed, discarding output)
Oct  7 07:05:01 ip-172-26-10-138 CRON[1902276]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 07:15:01 ip-172-26-10-138 CRON[1903861]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 07:17:01 ip-172-26-10-138 CRON[1904194]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 07:25:01 ip-172-26-10-138 CRON[1905567]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 07:35:01 ip-172-26-10-138 CRON[1907261]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 07:45:01 ip-172-26-10-138 CRON[1908974]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 07:55:01 ip-172-26-10-138 CRON[1910679]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 08:00:01 ip-172-26-10-138 CRON[1911515]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 08:00:03 ip-172-26-10-138 CRON[1911514]: (CRON) info (No MTA installed, discarding output)
Oct  7 08:05:01 ip-172-26-10-138 CRON[1912474]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 08:15:01 ip-172-26-10-138 CRON[1914162]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 08:17:01 ip-172-26-10-138 CRON[1914520]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 08:25:01 ip-172-26-10-138 CRON[1915887]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 08:35:01 ip-172-26-10-138 CRON[1917596]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 08:45:01 ip-172-26-10-138 CRON[1919311]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 08:55:01 ip-172-26-10-138 CRON[1920902]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 08:56:08 ip-172-26-10-138 cron[534]: (CRON) INFO (pidfile fd = 3)
Oct  7 08:56:08 ip-172-26-10-138 cron[534]: (CRON) INFO (Running @reboot jobs)
Oct  7 09:00:01 ip-172-26-10-138 CRON[5557]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 09:02:43 ip-172-26-10-138 CRON[5556]: (CRON) info (No MTA installed, discarding output)
Oct  7 09:05:01 ip-172-26-10-138 CRON[9869]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 09:15:01 ip-172-26-10-138 CRON[14086]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 09:17:01 ip-172-26-10-138 CRON[14894]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 09:25:01 ip-172-26-10-138 CRON[18196]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 09:35:01 ip-172-26-10-138 CRON[28821]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 09:45:01 ip-172-26-10-138 CRON[31980]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 09:55:01 ip-172-26-10-138 CRON[37432]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 10:05:01 ip-172-26-10-138 CRON[40676]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 10:15:01 ip-172-26-10-138 CRON[43372]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 10:17:01 ip-172-26-10-138 CRON[43778]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 10:25:01 ip-172-26-10-138 CRON[46105]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 10:35:01 ip-172-26-10-138 CRON[47935]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 10:45:01 ip-172-26-10-138 CRON[49644]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 10:55:01 ip-172-26-10-138 CRON[51732]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 11:05:01 ip-172-26-10-138 CRON[53552]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 11:15:01 ip-172-26-10-138 CRON[55624]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 11:17:01 ip-172-26-10-138 CRON[56038]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 11:25:01 ip-172-26-10-138 CRON[57598]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 11:35:01 ip-172-26-10-138 CRON[60005]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 11:45:01 ip-172-26-10-138 CRON[61689]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 11:55:01 ip-172-26-10-138 CRON[63412]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 12:00:01 ip-172-26-10-138 CRON[64267]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)
Oct  7 12:05:01 ip-172-26-10-138 CRON[65115]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 12:15:01 ip-172-26-10-138 CRON[66799]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 12:17:01 ip-172-26-10-138 CRON[67154]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 12:25:01 ip-172-26-10-138 CRON[69584]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 12:35:01 ip-172-26-10-138 CRON[72642]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 12:45:01 ip-172-26-10-138 CRON[75761]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 12:55:01 ip-172-26-10-138 CRON[77642]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 13:05:01 ip-172-26-10-138 CRON[80408]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 13:15:01 ip-172-26-10-138 CRON[83625]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 13:17:01 ip-172-26-10-138 CRON[83987]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 13:25:01 ip-172-26-10-138 CRON[86498]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 13:35:01 ip-172-26-10-138 CRON[88231]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 13:45:01 ip-172-26-10-138 CRON[92017]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 13:55:01 ip-172-26-10-138 CRON[96946]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 14:05:01 ip-172-26-10-138 CRON[101978]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 14:15:01 ip-172-26-10-138 CRON[106192]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 14:17:01 ip-172-26-10-138 CRON[107741]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 14:25:01 ip-172-26-10-138 CRON[112242]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 14:35:01 ip-172-26-10-138 CRON[114099]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 14:45:01 ip-172-26-10-138 CRON[116302]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 14:55:01 ip-172-26-10-138 CRON[119715]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 15:05:01 ip-172-26-10-138 CRON[121477]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 15:15:01 ip-172-26-10-138 CRON[124418]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 15:17:01 ip-172-26-10-138 CRON[124966]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 15:25:01 ip-172-26-10-138 CRON[126518]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 15:35:01 ip-172-26-10-138 CRON[128254]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 15:45:01 ip-172-26-10-138 CRON[132998]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 15:55:01 ip-172-26-10-138 CRON[138203]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 16:05:01 ip-172-26-10-138 CRON[139945]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 16:15:01 ip-172-26-10-138 CRON[141646]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 16:17:01 ip-172-26-10-138 CRON[141985]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 16:25:01 ip-172-26-10-138 CRON[143297]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 16:35:01 ip-172-26-10-138 CRON[145962]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 16:45:01 ip-172-26-10-138 CRON[148319]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 16:55:01 ip-172-26-10-138 CRON[152303]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 17:05:01 ip-172-26-10-138 CRON[155221]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 17:15:01 ip-172-26-10-138 CRON[159236]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 17:17:01 ip-172-26-10-138 CRON[161360]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 17:25:01 ip-172-26-10-138 CRON[165399]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 17:35:01 ip-172-26-10-138 CRON[167990]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 17:45:01 ip-172-26-10-138 CRON[171103]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 17:55:01 ip-172-26-10-138 CRON[175485]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 18:00:01 ip-172-26-10-138 CRON[178361]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 18:00:10 ip-172-26-10-138 CRON[178360]: (CRON) info (No MTA installed, discarding output)
Oct  7 18:05:01 ip-172-26-10-138 CRON[181585]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 18:15:01 ip-172-26-10-138 CRON[183863]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 18:17:01 ip-172-26-10-138 CRON[184208]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 18:25:01 ip-172-26-10-138 CRON[185563]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 18:35:01 ip-172-26-10-138 CRON[187296]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 18:45:01 ip-172-26-10-138 CRON[188984]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 18:55:01 ip-172-26-10-138 CRON[190710]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 19:00:01 ip-172-26-10-138 CRON[191575]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 19:00:06 ip-172-26-10-138 CRON[191574]: (CRON) info (No MTA installed, discarding output)
Oct  7 19:05:01 ip-172-26-10-138 CRON[192451]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 19:15:01 ip-172-26-10-138 CRON[194188]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 19:17:01 ip-172-26-10-138 CRON[194544]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 19:25:01 ip-172-26-10-138 CRON[195927]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 19:35:01 ip-172-26-10-138 CRON[197665]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 19:45:02 ip-172-26-10-138 CRON[199363]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 19:55:01 ip-172-26-10-138 CRON[200954]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 20:00:01 ip-172-26-10-138 CRON[201807]: (ubuntu) CMD (docker run --rm --name hm-ai-crawling --network hm-network -v ~/project_data/crawling_data:/var/data -v /etc/localtime:/etc/localtime hm-ai-crawling )
Oct  7 20:00:09 ip-172-26-10-138 CRON[201806]: (CRON) info (No MTA installed, discarding output)
Oct  7 20:05:01 ip-172-26-10-138 CRON[202691]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 20:15:01 ip-172-26-10-138 CRON[204375]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 20:17:01 ip-172-26-10-138 CRON[204718]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 20:25:01 ip-172-26-10-138 CRON[207038]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct  7 20:25:01 ip-172-26-10-138 CRON[207039]: (root) CMD (   test -x /etc/cron.daily/popularity-contest && /etc/cron.daily/popularity-contest --crond)
 
 
10.11 OOM 재발생.
 
zgrep "Out" /var/log/syslog.5.gz
Oct 11 18:10:43 ip-172-26-15-51 kernel: [121583.825830] Out of memory: Killed process 241161 (node) total-vm:16238732kB, anon-rss:13423580kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:130256kB oom_score_adj:0
 
 
dmesg | grep -i "Killed process”
dmesg | grep -i "Out of memory”
zgrep "Out of memory" /var/log/dmesg.*
 
로그 파일 롤링 설정 → cat /etc/logrotate.conf
 
해결방법 : 지속적인 모니터링을 통한 시스템 사용량 파악해야함
 
docker run --memory="1g" --memory-swap="1g" …