Shihta's Note

© Copyright 2023 | Shihta's Note | All rights reserved

Building collectd

14 Jun 2018

Building

  • ubuntu cosmic collectd 5.8 @ bionic (18.04)

    • Commands

      1
2
3
4
5
6

      apt-get build-dep collectd
apt-get install intel-cmt-cat libgrpc++-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc
wget http://archive.ubuntu.com/ubuntu/pool/universe/c/collectd/collectd_5.8.0-5.1.dsc
wget http://archive.ubuntu.com/ubuntu/pool/universe/c/collectd/collectd_5.8.0-5.1.debian.tar.xz
wget http://archive.ubuntu.com/ubuntu/pool/universe/c/collectd/collectd_5.8.0-5.debian.tar.xz
dpkg-buildpackage -us -uc -j8

  • ubuntu cosmic collectd 5.8 @ artful (17.10)

    • Build libi2c first (the version is too old)

      1
2
3
4
5
6

      apt-get build-dep libi2c-dev
apt-get install python3-all-dev
wget http://archive.ubuntu.com/ubuntu/pool/universe/i/i2c-tools/i2c-tools_4.0-2.dsc
wget http://archive.ubuntu.com/ubuntu/pool/universe/i/i2c-tools/i2c-tools_4.0.orig.tar.xz
wget http://archive.ubuntu.com/ubuntu/pool/universe/i/i2c-tools/i2c-tools_4.0-2.debian.tar.xz
dpkg-source -x i2c-tools_4.0-2.dsc

    • Fix libgrpc++-dev (revert a openssl patch), refer to Shihta.Kuan/grpc

      1
2

      apt-get build-dep libgrpc-dev
apt-get install libprotoc-dev libprotobuf-dev

    • the original source:

      1
2
3

      wget http://archive.ubuntu.com/ubuntu/pool/universe/c/collectd/collectd_5.8.0-5.dsc
wget http://archive.ubuntu.com/ubuntu/pool/universe/c/collectd/collectd_5.8.0.orig.tar.bz2
wget http://archive.ubuntu.com/ubuntu/pool/universe/c/collectd/collectd_5.8.0-5.debian.tar.xz

    • the original source used python3 by default, refer to Shihta.Kuan/collectd for python2 version.(shihta branch)

    • prepare the building environment:

      1
2

      apt-get install intel-cmt-cat libgrpc++-dev libi2c-dev (>= 4.0~) libmongoc-dev libprotobuf-dev protobuf-compiler (>= 3.0.0) protobuf-compiler-grpc
apt-get install protobuf-compiler-grpc libgrpc++-dev

    • build info

      1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244

      Configuration:
  Build:
    Platform  . . . . . . Linux
    Compiler vendor . . . gnu
    CC  . . . . . . . . . x86_64-linux-gnu-gcc
    CFLAGS  . . . . . . . -Wall -Werror -g -O2 -fdebug-prefix-map=/root/collectd/cosmic/build/collectd-5.8.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wno-error=deprecated-declarations
    CXXFLAGS  . . . . . . -Wall -Werror -g -O2
    CPP . . . . . . . . . x86_64-linux-gnu-gcc -E
    CPPFLAGS  . . . . . . -Wdate-time -D_FORTIFY_SOURCE=2 -I/root/collectd/cosmic/build/collectd-5.8.0/debian/include -UCONFIGFILE -DCONFIGFILE='"/etc/collectd/collectd.conf"'
    GRPC_CPP_PLUGIN . . . /usr/bin/grpc_cpp_plugin
    LD  . . . . . . . . . /usr/bin/ld -m elf_x86_64
    LDFLAGS . . . . . . . -Wl,-Bsymbolic-functions -Wl,-z,relro
    PROTOC  . . . . . . . /usr/bin/protoc
    YACC  . . . . . . . . bison -y
    YFLAGS  . . . . . . .

  Libraries:
    intel mic . . . . . . no (MicAccessApi not found)
    libaquaero5 . . . . . no (libaquaero5.h not found)
    libatasmart . . . . . yes
    libcurl . . . . . . . yes
    libdbi  . . . . . . . yes
    libdpdk . . . . . . . yes
    libesmtp  . . . . . . yes
    libganglia  . . . . . yes
    libgcrypt . . . . . . yes
    libgps  . . . . . . . yes
    libgrpc++ . . . . . . no (libgrpc++ not found)
    libhiredis  . . . . . yes
    libi2c-dev  . . . . . yes
    libiokit  . . . . . . no
    libiptc . . . . . . . yes
    libjevents  . . . . . no (jevents.h not found)
    libjvm  . . . . . . . yes
    libkstat  . . . . . . no (Solaris only)
    libkvm  . . . . . . . no
    libldap . . . . . . . yes
    liblua  . . . . . . . yes
    liblvm2app  . . . . . yes
    libmemcached  . . . . yes
    libmicrohttpd . . . . yes
    libmnl  . . . . . . . yes
    libmodbus . . . . . . yes
    libmongoc . . . . . . yes
    libmosquitto  . . . . yes
    libmysql  . . . . . . yes
    libnetapp . . . . . . no (netapp_api.h not found)
    libnetsnmp  . . . . . yes
    libnetsnmpagent . . . yes
    libnotify . . . . . . yes
    libopenipmi . . . . . yes
    liboping  . . . . . . yes
    libowcapi . . . . . . yes
    libpcap . . . . . . . yes
    libperfstat . . . . . no (AIX only)
    libperl . . . . . . . yes (version 5.26.0)
    libpq . . . . . . . . yes
    libpqos . . . . . . . yes
    libprotobuf . . . . . yes
    libprotobuf-c . . . . yes
    libpython . . . . . . yes
    librabbitmq . . . . . yes
    libriemann-client . . yes
    librdkafka  . . . . . yes
    librouteros . . . . . no (routeros_api.h not found)
    librrd  . . . . . . . yes
    libsensors  . . . . . yes
    libsigrok   . . . . . yes
    libstatgrab . . . . . no
    libtokyotyrant  . . . yes
    libudev . . . . . . . yes
    libupsclient  . . . . yes
    libvarnish  . . . . . yes
    libvirt . . . . . . . yes
    libxenctrl  . . . . . yes
    libxml2 . . . . . . . yes
    libxmms . . . . . . . no
    libyajl . . . . . . . yes
    oracle  . . . . . . . no (ORACLE_HOME is not set)
    protobuf-c  . . . . . yes
    protoc 3  . . . . . . yes

  Features:
    daemon mode . . . . . yes
    debug . . . . . . . . no

  Bindings:
    perl  . . . . . . . . yes (INSTALLDIRS=vendor INSTALL_BASE=)

  Modules:
    aggregation . . . . . yes
    amqp    . . . . . . . yes
    apache  . . . . . . . yes
    apcups  . . . . . . . yes
    apple_sensors . . . . no (disabled on command line)
    aquaero . . . . . . . no (disabled on command line)
    ascent  . . . . . . . yes
    barometer . . . . . . yes
    battery . . . . . . . yes
    bind  . . . . . . . . yes
    ceph  . . . . . . . . yes
    cgroups . . . . . . . yes
    chrony. . . . . . . . yes
    conntrack . . . . . . yes
    contextswitch . . . . yes
    cpu . . . . . . . . . yes
    cpufreq . . . . . . . yes
    cpusleep  . . . . . . yes
    csv . . . . . . . . . yes
    curl  . . . . . . . . yes
    curl_json . . . . . . yes
    curl_xml  . . . . . . yes
    dbi . . . . . . . . . yes
    df  . . . . . . . . . yes
    disk  . . . . . . . . yes
    dns . . . . . . . . . yes
    dpdkevents. . . . . . yes
    dpdkstat  . . . . . . yes
    drbd  . . . . . . . . yes
    email . . . . . . . . yes
    entropy . . . . . . . yes
    ethstat . . . . . . . yes
    exec  . . . . . . . . yes
    fhcount . . . . . . . yes
    filecount . . . . . . yes
    fscache . . . . . . . yes
    gmond . . . . . . . . yes
    gps . . . . . . . . . yes
    grpc  . . . . . . . . no (dependency error)
    hddtemp . . . . . . . yes
    hugepages . . . . . . yes
    intel_pmu . . . . . . no (disabled on command line)
    intel_rdt . . . . . . yes
    interface . . . . . . yes
    ipc . . . . . . . . . yes
    ipmi  . . . . . . . . yes
    iptables  . . . . . . yes
    ipvs  . . . . . . . . yes
    irq . . . . . . . . . yes
    java  . . . . . . . . yes
    load  . . . . . . . . yes
    logfile . . . . . . . yes
    log_logstash  . . . . yes
    lpar  . . . . . . . . no (disabled on command line)
    lua . . . . . . . . . yes
    lvm . . . . . . . . . yes
    madwifi . . . . . . . yes
    match_empty_counter . yes
    match_hashed  . . . . yes
    match_regex . . . . . yes
    match_timediff  . . . yes
    match_value . . . . . yes
    mbmon . . . . . . . . yes
    mcelog  . . . . . . . yes
    md  . . . . . . . . . yes
    memcachec . . . . . . yes
    memcached . . . . . . yes
    memory  . . . . . . . yes
    mic . . . . . . . . . no (disabled on command line)
    modbus  . . . . . . . yes
    mqtt  . . . . . . . . yes
    multimeter  . . . . . yes
    mysql . . . . . . . . yes
    netapp  . . . . . . . no (disabled on command line)
    netlink . . . . . . . yes
    network . . . . . . . yes
    nfs . . . . . . . . . yes
    nginx . . . . . . . . yes
    notify_desktop  . . . yes
    notify_email  . . . . yes
    notify_nagios . . . . yes
    ntpd  . . . . . . . . yes
    numa  . . . . . . . . yes
    nut . . . . . . . . . yes
    olsrd . . . . . . . . yes
    onewire . . . . . . . yes
    openldap  . . . . . . yes
    openvpn . . . . . . . yes
    oracle  . . . . . . . no (disabled on command line)
    ovs_events  . . . . . yes
    ovs_stats . . . . . . yes
    perl  . . . . . . . . yes
    pf  . . . . . . . . . no (disabled on command line)
    pinba . . . . . . . . yes
    ping  . . . . . . . . yes
    postgresql  . . . . . yes
    powerdns  . . . . . . yes
    processes . . . . . . yes
    protocols . . . . . . yes
    python  . . . . . . . yes
    redis . . . . . . . . yes
    routeros  . . . . . . no (disabled on command line)
    rrdcached . . . . . . yes
    rrdtool . . . . . . . yes
    sensors . . . . . . . yes
    serial  . . . . . . . yes
    sigrok  . . . . . . . no (disabled on command line)
    smart . . . . . . . . yes
    snmp  . . . . . . . . yes
    snmp_agent  . . . . . yes
    statsd  . . . . . . . yes
    swap  . . . . . . . . yes
    synproxy  . . . . . . yes
    syslog  . . . . . . . yes
    table . . . . . . . . yes
    tail_csv  . . . . . . yes
    tail  . . . . . . . . yes
    tape  . . . . . . . . no (disabled on command line)
    target_notification . yes
    target_replace  . . . yes
    target_scale  . . . . yes
    target_set  . . . . . yes
    target_v5upgrade  . . yes
    tcpconns  . . . . . . yes
    teamspeak2  . . . . . yes
    ted . . . . . . . . . yes
    thermal . . . . . . . yes
    threshold . . . . . . yes
    tokyotyrant . . . . . yes
    turbostat . . . . . . yes
    unixsock  . . . . . . yes
    uptime  . . . . . . . yes
    users . . . . . . . . yes
    uuid  . . . . . . . . yes
    varnish . . . . . . . no (disabled on command line)
    virt  . . . . . . . . yes
    vmem  . . . . . . . . yes
    vserver . . . . . . . yes
    wireless  . . . . . . yes
    write_graphite  . . . yes
    write_http  . . . . . yes
    write_kafka . . . . . yes
    write_log . . . . . . yes
    write_mongodb . . . . yes
    write_prometheus. . . yes
    write_redis . . . . . yes
    write_riemann . . . . yes
    write_sensu . . . . . yes
    write_tsdb  . . . . . yes
    xencpu  . . . . . . . yes
    xmms  . . . . . . . . no (disabled on command line)
    zfs_arc . . . . . . . yes
    zone  . . . . . . . . no (disabled on command line)
    zookeeper . . . . . . yes

Others

  • use date -R to generate correct date format

collectd config

  • /etc/collectd/collectd.conf.d/my.conf

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

    <LoadPlugin "python">
    Globals true
</LoadPlugin>

<Plugin "python">
    ModulePath "/root/collectd-openstack/plugins"

    Import "keystone_plugin"

    <Module "keystone_plugin">
        Username "admin"
        Password "secret"
        ProjectName "admin"
        AuthURL "http://10.101.5.9/identity/v3"
        KeystoneVersion "v3"
        ProjectDomain "default"
        UserDomain "default"
#        Debug "True"
        Verbose "False"
    </Module>
</Plugin>

<Plugin "python">
    ModulePath "/root/collectd-openstack/plugins"

    Import "project_plugin"

    <Module "project_plugin">
        Username "admin"
        Password "secret"
        ProjectName "admin"
        AuthURL "http://10.101.5.9/identity/v3"
        KeystoneVersion "v3"
        ProjectDomain "default"
        UserDomain "default"
#        Debug "True"
        Verbose "False"
    </Module>
</Plugin>

<Plugin "python">
    ModulePath "/root/collectd-openstack/plugins"

    Import "hypervisor_plugin"

    <Module "hypervisor_plugin">
        Username "admin"
        Password "secret"
        ProjectName "admin"
        AuthURL "http://10.101.5.9/identity/v3"
        KeystoneVersion "v3"
        ProjectDomain "default"
        UserDomain "default"
#        Debug "True"
        Verbose "False"
    </Module>
</Plugin>

  • /etc/collectd/collectd.conf.d/01-logfile.conf

    1
2
3
4
5
6
7
8
9
10
11

    LoadPlugin "logfile"
<Plugin "logfile">
  LogLevel "info"
  File "/var/log/collectd.log"
  Timestamp true
</Plugin>

# Decrease syslog verbosity, to avoid duplicate logging
<Plugin "syslog">
  LogLevel "err"
</Plugin>

collectd-openstack

  • Source

  • Sample collectd prometheus output

  • Data source types

    There are four data source types which are basically identical to the data source types of the same name in RRDtool:

    • GAUGE

      A GAUGE value is simply stored as-is. This is the right choice for values which may increase as well as decrease, such as temperatures or the amount of memory used.

    • DERIVE

      These data sources assume that the change of the value is interesting, i.e. the derivative. Such data sources are very common with events that can be counted, for example the number of emails that have been received by an MTA since it was started. The total number of emails is not interesting, but the change since the value has been read the last time. The value is therefore converted to a rate using the following formula:

      rate = (value_new - value_old) / (time_new - time_old)

      Please note that if value_new < value_old, the resulting rate will be negative. If you set the minimum value to zero, such data points will be discarded. Using DERIVE data sources and a minimum value of zero is recommended for counters that rarely overflow, i.e. wrap-around after their maximum value has been reached. This data source type is available since version 4.8.

    • COUNTER

      These data sources behave exactly like DERIVE data sources in the “normal” case. Their behavior differs when value_new < value_old, i.e. when the new value is smaller than the previous value. In this case, COUNTER data sources will assume the counter “wrapped around” and take this into account. The formula for wrap-around cases is:

      rate = (2**width - value_old + value_new) / (time_new - time_old) width = value_old < 2**32 ? 32 : 64

      Please note that the rate of a COUNTER data source is never negative. If a counter is reset to zero, for example because an application was restarted, the wrap-around calculation may result in a huge rate. Thus setting a reasonable maximum value is essential when using COUNTER data sources. Because of this, COUNTER data sources are only recommended for counters that wrap-around often, for example 32 bit octet counters of a busy switch port.

    • ABSOLUTE

      This is probably the most exotic type: It is intended for counters which are reset upon reading. In effect, the type is very similar to GAUGE except that the value is an (unsigned) integer and will be divided by the time since the last reading. This data source type is available since version 4.8 and has been added mainly for consistency with the data source types available in RRDtool.

  • 3 layers architecture

    1
2
3
4
5
6
7
8
9
10

    data = {
  "testprefix": {
    "nameA": {
      "THE_A_A": 100
    },
    "nameB": {
      "THE_B_B": 12
    }
  }
}
: collectd

Related Posts

comments powered by Disqus