sam/obmp-docker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updates to dev-image and added postgres

Browse Source
This commit is contained in:
Tim Evens 2021-03-29 11:13:57 -07:00
parent 068537b740
commit 8b3356086b
8 changed files with 336 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
README.md
View File

@ -1,12 +1,11 @@
# OpenBMP docker files
Docker files for OpenBMP.
(Prerequisite) Platform Docker Install
--------------------------------------
## (Prerequisite) Platform Docker Install
> Ignore this step if you already have a current docker install
> ####NOTE
> **NOTE**
> You should use the latest docker version, documented in this section.
Follow the instructions on https://docs.docker.com/get-docker/
@ -26,20 +25,22 @@ Follow the instructions on https://docs.docker.com/get-docker/
Make sure you can run '**docker run hello-world**' successfully.
Install OpenBMP using Docker
----------------------------
## OpenBMP Docker Files
Each docker file contains a readme file, see below:
* [Collector](collector/README.md)
* [PostgreSQL](postgres/README.md)
* [PSQL Consumer](psql-consumer/README.md)
Install OpenBMP using docker-compose
----------------------------
[Docker Compose](https://docs.docker.com/compose/install/) is used to run several containers. It also handles restarting containers on reboot/restart.
## Using Docker Compose to run everything
### Install Docker Compose
You will need docker-compose. You can install that via [Docker Compose](https://docs.docker.com/compose/install/)
instructions. Docker compose will run everything, including handling restarts of containers.
```
docker-compose up
docker-compose -p obmp up
```

74
collector/Dockerfile Normal file
View File

@ -0,0 +1,74 @@
# Collector: openbmp/collector
#
# Copyright (c) 2021 Cisco Systems, Inc. and Tim Evens. All rights reserved.
#
# Author: Tim Evens <tim@openbmp.org>
#
# BUILD:
#
# Docker context does not support multiple paths or mounting volumes for builds.
# In effort to build the container from local git clones, we dynamically build a context
#
# Clone the obmp-collector and obmp-docker repos into the same directory.
# Change directories to obmp-docker/collector and run the below from that
# directory.
#
# Example docker build:
# tar -cL -C ../../ ./obmp-collector ./obmp-docker/collector \
# | docker build --build-arg BUILD_NUMBER=50 \
# -f obmp-docker/collector/Dockerfile -t openbmp/collector:build-50 -
#
# -----------------------------------------------
# stage: Build collector
# -----------------------------------------------
FROM openbmp/dev-image:latest AS build
COPY obmp-collector/ /ws
WORKDIR /ws
RUN mkdir -p build && cd build \
&& cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr ../ \
&& make \
&& make install
# -----------------------------------------------
# stage: Final container
# -----------------------------------------------
# Pull base image.
FROM debian:bullseye-slim
# Add files.
ADD obmp-docker/collector/scripts/install /tmp/
ADD obmp-docker/collector/scripts/run /usr/sbin/
ARG BUILD_NUMBER=0
# Copy files from previous stages
COPY --from=build /usr/bin/openbmpd /usr/bin/
COPY --from=build /usr/etc/openbmp/openbmpd.conf /usr/etc/openbmp/openbmpd.conf
COPY --from=build /etc/init/openbmpd.conf /etc/init/openbmpd.conf
COPY --from=build /etc/default/openbmpd.new /etc/default/openbmpd
COPY --from=build /etc/logrotate.d/openbmpd /etc/logrotate.d/openbmpd
# Proxy servers
#ENV http_proxy http://proxy:80
#ENV https_proxy http://proxy:80
#ENV no_proxy "domain.com"
# Run Install script
RUN /tmp/install
# Define mount points.
VOLUME ["/config"]
# Define working directory.
WORKDIR /tmp
# Define default command.
CMD ["/usr/sbin/run"]
# Expose ports.
# openbmpd/collector
EXPOSE 5000

78
collector/README.md Normal file
View File

@ -0,0 +1,78 @@
# OpenBMP Collector Container Image
Collector is the container for collecting BMP messages from BMP senders, e.g. routers.
## Container Includes
* The Latest collector (listening port is TCP 5000)
## Building the container
See [Dockerfile] notes.
## Steps to Use the container
### 1) Install docker
Follow the [Docker Instructions](https://docs.docker.com/installation/) to install docker.
### 2) Download the docker image
docker pull openbmp/collector
### 3) [OPTIONAL] Add persistent configs
mkdir -p /var/openbmp/config
chmod 777 /var/openbmp/config
#### config/hosts
You can add custom host entries so that the collector will reverse lookup IP addresses
using a persistent hosts file.
Run docker with ```-v /var/openbmp/config:/config``` to make use of the persistent config files.
#### config/openbmpd.conf
You can provide a customized **openbmpd.conf**. See [Config Example](https://github.com/OpenBMP/obmp-collector/blob/main/Server/openbmpd.conf)
### 4) Run docker container
#### Environment Variables
Below table lists the environment variables that can be used with ``docker run -e <name=value>``
NAME | Value | Details
:---- | ----- |: -------
KAFKA\_FQDN | hostanme or IP | Kafka broker hostname[:port]. Hostname can be an IP address
OPENBMP\_ADMIN\_ID | name or IP | Name or IP of the collector, default is the docker hostname
OPENBMP\_BUFFER | Size in MB | Defines the openbmpd buffer per router for BMP messages. Default is 16 MB.
#### Run normally
> ##### IMPORTANT
> You must define the **KAFKA_FQDN** as a 'hostname'. If all containers are running on the same node, this
> hostname can be local specific, such as 'localhost' or 'myhost'. If Kafka is running on a different server,
> than the consumers and producers, then the KAFKA_FQDN should be a valid hostname that can be resolved using DNS.
> This can be internal DNS or manually done by updating the /etc/hosts file on each machine.
docker run -d --name=obmp_collector -e KAFKA_FQDN=localhost \
--sysctl net.ipv4.tcp_keepalive_intvl=30 \
--sysctl net.ipv4.tcp_keepalive_probes=5 \
--sysctl net.ipv4.tcp_keepalive_time=180 \
-v /var/openbmp/config:/config \
-p 5000:5000 \
openbmp/collector
### Monitoring/Troubleshooting
You can use standard docker exec commands to monitor the log files. To monitor
openbmp, use ```docker exec obmp_collector tail -f /var/log/openbmpd.log```
Alternatively, it can be easier at times to navigate all the log files from within the container. You can do so using:
docker exec -it obmp_collector bash
#### docker logs
You can use ```docker logs obmp_collector``` to get the console logs. This is useful if the container exits due to
invalid start or for another reason.

40
collector/scripts/install Executable file
View File

@ -0,0 +1,40 @@
#!/bin/bash
# Collector install script
#
# Copyright (c) 2021 Cisco Systems, Inc. and Tim Evens. All rights reserved.
#
# Author: Tim Evens <tim@openbmp.org>
# Add build details
touch /usr/local/build-${BUILD_NUMBER}
#
# Defaults
#
# Disable interactive
export DEBIAN_FRONTEND=noninteractive
# Install base packages
apt-get update
# Fix ubuntu docker install
#sed -i 's/exit 101/exit 0/' /usr/sbin/policy-rc.d
# General depend install
apt-get install -y iproute2 wget zlib1g libssl1.1 libsasl2-2
# --
# -- Add host entries for reverse PTR lookups
# --
if [[ -f /config/hosts ]]; then
cat /config/hosts >> /etc/hosts
fi
# --
# -- Clean up
# --
apt-get clean
rm -rf /var/lib/apt/lists/* /var/tmp/*
rm -f /tmp/install

65
collector/scripts/run Executable file
View File

@ -0,0 +1,65 @@
#!/bin/bash
# All-in-One run script
#
# Copyright (c) 2021 Cisco Systems, Inc. and Tim Evens. All rights reserved.
#
# Author: Tim Evens <tim@openbmp.org>
#
ADMIN_ID=${ADMIN_ID:="collector"}
DOCKER_HOST_IP=$(ip route | grep default | head -1 | awk '{ print $3}')
if [[ ${KAFKA_FQDN:-""} == "" ]]; then
echo "ERROR: Missing ENV KAFKA_FQDN. Cannot proceed until you add that in docker run -e KAFKA_FQDN=<...>"
exit 1
else
if [[ ${KAFKA_FQDN} == "localhost" ]]; then
KAFKA_FQDN="docker-localhost"
elif [[ ${KAFKA_FQDN} == "127.0.0.1" ]]; then
KAFKA_FQDN="docker-localhost"
elif [[ ${KAFKA_FQDN} == "::1" ]]; then
KAFKA_FQDN="docker-localhost"
fi
fi
if [[ -f /config/openbmpd ]]; then
source /config/openbmpd
else
source /etc/default/openbmpd
fi
#
# System info
#
if [[ ${MEM:-""} = "" ]]; then
SYS_TOTAL_MEM=$(grep MemTotal /proc/meminfo | awk '{print int($2 / 1000)}')
else
SYS_TOTAL_MEM=$(($MEM * 1024))
fi
SYS_NUM_CPU=$(grep processor /proc/cpuinfo | wc -l)
# Update the hosts file
echo "$DOCKER_HOST_IP docker-localhost" >> /etc/hosts
# Update the etc hosts file
if [[ -f /config/hosts ]]; then
cat /config/hosts >> /etc/hosts
fi
# Update openbmpd config file
OPENBMP_CFG_FILE=/usr/etc/openbmp/openbmpd.conf
sed -r -i "s/admin_id:.*/admin_id: ${ADMIN_ID}/" /usr/etc/openbmp/openbmpd.conf
sed -r -i "s/localhost:9092/${KAFKA_FQDN}:9092/" /usr/etc/openbmp/openbmpd.conf
if [[ -f /config/openbmpd.conf ]]; then
OPENBMP_CFG_FILE=/config/openbmpd.conf
fi
# Start openbmpd and wait - openbmpd runs in foreground
echo "Running openbmpd collector, see /var/log/openbmpd.log"
/usr/bin/openbmpd -f -l /var/log/openbmpd.log -c ${OPENBMP_CFG_FILE}

6
dev-image/README.md
View File

@ -22,3 +22,9 @@ docker tag openbmp/dev-image:build-NNN openbmp/dev-image:latest
docker push openbmp/dev-image:build-NNN
docker push openbmp/dev-image:latest
```
### Running
```
docker run --rm -v $(PWD):/ws -it openbmp/dev-image /bin/bash
```

40
postgres/Dockerfile Normal file
View File

@ -0,0 +1,40 @@
# Postgres Container
#
# Copyright (c) 2021 Cisco Systems, Inc. and Tim Evens. All rights reserved.
#
# Build:
# docker build -t openbmp/postgres:build-NNN .
#
# Run:
# docker run --rm -it -p 5432:5432 \
# -e POSTGRES_PASSWORD=openbmp \
# -e POSTGRES_USER=openbmp \
# -e POSTGRES_DB=openbmp \
# openbmp/postgres:build-NNN
FROM timescale/timescaledb:2.1.0-pg13
# Current/working dir
VOLUME ["/ws"]
WORKDIR /ws
RUN apk update \
&& apk add openssl \
&& openssl req -x509 -newkey rsa:4096 -nodes -subj "/C=US/ST=CA/L=Seattle/O=OpenBMP/CN=localhost" \
-keyout /psql_server.key -out /psql_server.crt -days 365 \
&& chown postgres /psql_server.* \
&& mkdir -p /var/lib/postgresql/ts \
&& chown postgres /var/lib/postgresql/ts \
&& egrep -q -e '^hostssl( |\t)+all' /usr/local/share/postgresql/pg_hba.conf.sample || \
echo 'hostssl all all 0.0.0.0/0 md5' >> /usr/local/share/postgresql/pg_hba.conf.sample \
&& sed -i -e "s/^\#*listen_addresses.*=.*/listen_addresses = '*'/" /usr/local/share/postgresql/postgresql.conf.sample \
&& sed -i -e "s/^\#*ssl[ ]*=.*/ssl = on/" /usr/local/share/postgresql/postgresql.conf.sample \
&& sed -i -e "s/^\#*ssl_cert_file.*=.*/ssl_cert_file = \'\/psql_server.crt\'/" /usr/local/share/postgresql/postgresql.conf.sample \
&& sed -i -e "s/^\#*ssl_key_file.*=.*/ssl_key_file = \'\/psql_server.key\'/" /usr/local/share/postgresql/postgresql.conf.sample
#RUN echo 'psql -U $POSTGRES_USER -c "CREATE TABLESPACE timeseries LOCATION '\''/var/lib/postgresql/ts'\'';" $POSTGRES_DB' > /docker-entrypoint-initdb.d/0_cnis_schema.sh \
# && echo 'psql -U $POSTGRES_USER $POSTGRES_DB < /tmp/cnis-mq.sql' >> /docker-entrypoint-initdb.d/0_cnis_schema.sh \
# && echo 'psql -U $POSTGRES_USER $POSTGRES_DB < /tmp/cnis-mq-functions.sql' >> /docker-entrypoint-initdb.d/0_cnis_schema.sh \
# && echo 'psql -U $POSTGRES_USER $POSTGRES_DB < /tmp/dev-psql.sql' >> /docker-entrypoint-initdb.d/0_cnis_schema.sh

22
postgres/README.md Normal file
View File

@ -0,0 +1,22 @@
# OpenBMP Postgres
The postgres container is a plain postgres/timescaleDB container with
some modifications to support OpenBMP. Any postgres install will work as long as
they have similar changes as shown in [Dockerfile](Dockerfile).
## Building
See the [Dockerfile](Dockerfile) notes for build instructions.
## Running
```
docker run --rm -it -p 5432:5432 \
-e POSTGRES_PASSWORD=openbmp \
-e POSTGRES_USER=openbmp \
-e POSTGRES_DB=openbmp \
openbmp/postgres:build-NNN
```
### Configuration/Environment Variables
See both [Postgres](https://hub.docker.com/_/postgres) and
[TimescaleDB](https://hub.docker.com/r/timescale/timescaledb) documentation for more
information on how to configure/run the docker container.