Class: Karafka::Web::Tracking::Consumers::Sampler::Metrics::Os

Inherits:
Base
  • Object
show all
Defined in:
lib/karafka/web/tracking/consumers/sampler/metrics/os.rb

Overview

Collects OS-level metrics from /proc filesystem and system commands Used when running directly on a host OS (not in containers)

Direct Known Subclasses

Container

Instance Method Summary collapse

Constructor Details

#initialize(shell) ⇒ Os

Returns a new instance of Os.

Parameters:

  • shell (MemoizedShell) —

    shell executor for running system commands



13
14
15
16
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 13

def initialize(shell)
  super()
  @shell = shell
end

Instance Method Details

#cpu_usage ⇒ Array<Float>

Returns load averages for last 1, 5 and 15 minutes.

Returns:

  • (Array<Float>) —

    load averages for last 1, 5 and 15 minutes



91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 91

def cpu_usage
  case RUBY_PLATFORM
  when /linux/
    File
      .read('/proc/loadavg')
      .split
      .first(3)
      .map(&:to_f)
  when /darwin|bsd/
    shell
      .call('w | head -1')
      .strip
      .split
      .map(&:to_f)
      .last(3)
  else
    [-1, -1, -1]
  end
end

#cpus ⇒ Integer

Returns CPU count.

Returns:

  • (Integer) —

    CPU count



112
113
114
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 112

def cpus
  @cpus ||= Etc.nprocessors
end

#memory_size ⇒ Integer

Note:

This is a STATIC value (system RAM capacity), memoized for performance

Note:

Used in Web UI to show “OS memory available” metric

This is the total physical memory available to the system/container. On Linux: reads MemTotal from /proc/meminfo On macOS: uses sysctl hw.memsize In containers: Container class overrides this to return cgroup memory limit

Returns:

  • (Integer) —

    total amount of available memory in kilobytes



68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 68

def memory_size
  return @memory_size if instance_variable_defined?(:@memory_size)

  @memory_size = case RUBY_PLATFORM
                 when /linux/
                   mem_info = File.read('/proc/meminfo')
                   mem_total_line = mem_info.match(/MemTotal:\s*(?<total>\d+)/)
                   mem_total_line['total'].to_i
                 when /darwin|bsd/
                   shell
                 .call('sysctl -a')
                 .split("\n")
                 .find { |line| line.start_with?('hw.memsize:') }
                 .to_s
                 .split
                 .last
                 .to_i
                 else
                   0
                 end
end

#memory_threads_ps ⇒ Array<Array<Integer, Integer, Integer>>, false

Note:

Sampler calls this once per sample cycle (every ~5 seconds) and caches the result in @memory_threads_ps to ensure consistent data within a single sample snapshot

Note:

The cache is refreshed on EVERY sample cycle, so data stays current

Note:

On Linux, thread count is only extracted for the current process to optimize performance

Loads process information for all running processes This method reads information about ALL processes on the system (or in the container). The data is used by multiple metrics: - memory_total_usage: sums RSS across all processes - threads: extracts thread count for current process

Format of each array element: [memory_in_kb, thread_count, process_id] - memory_in_kb: RSS (Resident Set Size) in kilobytes - thread_count: Number of threads (only populated for current process, 0 for others) - process_id: Process ID

Platform behavior: - Linux: Reads /proc/[0-9]*/statm for ALL processes on host/container - macOS: Uses ps -A to get all processes - Containers: Due to PID namespaces, only sees processes within the container

Returns:

  • (Array<Array<Integer, Integer, Integer>>, false) —

    array of [rss_kb, threads, pid] for each process, or false if unavailable



149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 149

def memory_threads_ps
  case RUBY_PLATFORM
  when /linux/
    page_size = Helpers::Sysconf.page_size
    current_pid = ::Process.pid

    # Read all processes from /proc
    Dir.glob('/proc/[0-9]*/statm').filter_map do |statm_file|
      pid = statm_file.match(%r{/proc/(\d+)/statm})[1].to_i
      status_file = "/proc/#{pid}/status"

      # Extract RSS from /proc/<pid>/statm (second field)
      rss_pages = begin
        File.read(statm_file).split[1].to_i
      rescue StandardError
        next # Process may have exited
      end

      # Extract thread count from /proc/<pid>/status (only for current process)
      thcount = if pid == current_pid
                  begin
                    File.read(status_file)[/^Threads:\s+(\d+)/, 1].to_i
                  rescue StandardError
                    0
                  end
                else
                  0
                end

      # Convert RSS from pages to kilobytes
      rss_kb = (rss_pages * page_size) / 1024

      [rss_kb, thcount, pid]
    end
  # thcount is not available on macos ps
  # because of that we inject 0 as threads count similar to how
  # we do on windows
  when /darwin|bsd/
    shell
      .call('ps -A -o rss=,pid=')
      .split("\n")
      .map { |row| row.strip.split.map(&:to_i) }
      .map { |row| [row.first, 0, row.last] }
  else
    false
  end
end

#memory_total_usage(memory_threads_ps) ⇒ Integer

Note:

This is DIFFERENT from memory_usage which only shows current process memory

Note:

Used in Web UI to show “OS memory used” metric

This represents system-wide (or container-wide) memory usage by summing RSS across all processes. On bare metal: sums memory for all processes on the host In containers: sums memory for all processes within the container (due to PID namespace)

Parameters:

  • memory_threads_ps (Array, false) —

    parsed ps/proc output for all processes

Returns:

  • (Integer) —

    total memory used by all processes in the system (or container)



55
56
57
58
59
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 55

def memory_total_usage(memory_threads_ps)
  return 0 unless memory_threads_ps

  memory_threads_ps.sum(&:first)
end

#memory_usage ⇒ Integer

Note:

This represents ONLY the current Karafka process memory usage

This is the amount of physical memory currently used by the Karafka process. On Linux: reads VmRSS from /proc/pid/status On macOS: uses ps command to get RSS for current process

Returns:

  • (Integer) —

    memory used by this process in kilobytes (RSS - Resident Set Size)



23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 23

def memory_usage
  pid = ::Process.pid

  case RUBY_PLATFORM
  # Reading this that way is cheaper than running a shell command
  when /linux/
    File.readlines("/proc/#{pid}/status").each do |line|
      next unless line.start_with?('VmRSS:')

      break line.split[1].to_i
    end
  when /darwin|bsd/
    shell
      .call("ps -o pid,rss -p #{pid}")
      .lines
      .last
      .split
      .last
      .to_i
  else
    0
  end
end

#threads(memory_threads_ps) ⇒ Integer

Note:

This returns total number of threads from the OS perspective including native extensions threads, etc.

Returns number of process threads.

Parameters:

  • memory_threads_ps (Array, false) —

    parsed ps output

Returns:

  • (Integer) —

    number of process threads



120
121
122
123
124
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 120

def threads(memory_threads_ps)
  return 0 unless memory_threads_ps

  memory_threads_ps.find { |row| row.last == ::Process.pid }[1]
end