Saturday, April 12, 2014

FreeBSD: make config does not show config option

Today was update-time for my FreeBSD machines. editors/vim required some changes to the options. However, "make config" did not open the usual config dialog.

The config dialog requires ports-mgmt/dialog4ports to be installed. While the port was installed on my machine, it also was due to be updated. The stale version of dialog4ports prohibited the config dialog from appearing. After an successful update, I was able to change my vim options again:




So, if the config dialog does not appear on your system:


  1.  check if ports-mgmt/dialog4ports is installed
  2. make sure it is updated to the newest version



Thursday, May 02, 2013

Setting up Flow-Inspector on Your System

Some time ago, our group at TUM created a web frontend that allows users to visualize network flow data. It's based on Javascript and employs the D3Js library, which allows for dynamic rendering of arbitrary data. We used it to create some visualizations like this one:




If you're interested in how the system works, then have a look at our publication that was published at WIV 2012.

Setting up flow-inspector can be a bit tricky, as you first need to have a data source that generates your flow data. We can import data using a variety of ways, but most of them require you to set up our network monitoring toolkit VERMONT. Furthermore, there is a certain lack of documentation ...

This blog posts will try to walk you through the process of setting up VERMONT and flow-inspector on your system. After following this blog post, you should have a working installation of VERMONT that monitors the interfaces of your system, generates flow data from these observations, and pushes the data into the visualization system.

Preparing Your System

In order to install any of the components, you need to prepare your local system by installing some packages. As we currently do not ship any binaries for either VERMONT or flow-inspector, you need to build them from the sources.

In order to do this, you need to be have a unix-based system such as Linux (any modern distribution should work), FreeBSD, or Mac OS X. Install a development environment on your system and make sure the development packages for the dependencies that are documented in the READMEs of VERMONT and Flow-Inspector are installed. 

This is the current list for VERMONT:

  • cmake 
  • git
  • build-essentials (XCode on Mac OS X)
  • libboost-filesystem-dev 
  • libboost-regex-dev 
  • libboost-test-dev 
  • libxml2-dev 
  • libpcap-dev 
  • hiredis-dev

For flow-inspector you need

  • python (>= 2.6)
  • Jinja2 (tested with v2.6) pip install Jinja2
  • ujson (tested with 1.18) (optional but recommended) pip install ujson
  • Redis (tested with v2.4.3) pip install redis
  • MySQLdb (tested with v1.2.3) pip install mysql-python

For this example, we will use MySQL as our backend for storing the flow data. You can choose between the MySQL, Oracle and MongoDB backends. Currently, I recommend using the MySQL backend. However, you might want to check the flow-inspector page as this recommendation might change over time.
Create a database called "flowinspector" in your DB, create a MySQL user and grant him all permissions on the flowinspector DB.

Installing and Configuring Flow-Inspector 


After you installed all base components into your system, you are now ready to get the latest flow-inspector version:

git clone https://github.com/constcast/flow-inspector.git

After you retrieved the source code, you need to adopt the configuration for your system.  In order to do this, you need to create a file in the directory "flow-inspector/config/":


cd flow-inspector/config
cp config.default.py config.py


Then use your favorite editor to change the configuration. You need to change the destination flow backend part of the configuration in order to make sure flow-inspector can connect to your data base:

db_backend = "mysql"
db_host = "127.0.0.1"
db_port = 3306
db_user = ""
db_password = ""
db_name = "flowinspector"


Make sure you enter the username and password of the MySQL user that you created and granted the rights on the database flowinspector. 

After you did this, you should try to start flow-inspector by running the command 


app/app.py

from the flow-inspector base directory. You should receive a message that looks like:

willet:flow-inspector braun$ ./app/app.py
Bottle v0.11.3 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit. 

Running this script starts a webserver on port 8080 that you can use to connect your browser to. At the time of this writeup, only Chrome and Safari have all the features implemented that are required to run flow-inspector.  

If you successfully managed to perform all the steps above, your browser should show you a screen that looks like this:



As you can see, there is some kind of user interface, but it doesn't show any data. This is what we would expect because we did not yet import any data into flow-inspector. Please note that the application script will, at this moment, throw some exceptions whenever you point your web browser at the system. This is flow-inspector's way to complain about missing data. Ignore these messages until we actually have some data to display. 

In order to make flow-inspector insert data into the system, you need to start the preprocessor that you can find in the directory preprocess/. You can start the preprocessor by changing into the directory preprocess and running:

./preprocess.py

It is necessary to keep this script running whenever you want to import data.  If the script is not running, no new data will be imported into flow-inspector. 

If you want to end the importing process, you can finish the script using the key combination "STRG+C". The script will not finish immediately, but it will perform some cleanups and commit some cached data to the database. Please be patient while the script finishes and do not abort the processing. If you abort the script by pressing "STRG+C" for a second time, you will certainly see some data loss. So please be patient while the script finishes :)

Now that we have the importing process running, we need to make sure that we generate some data that can be imported. 

Compiling and Configuring VERMONT

Make sure you installed all the packages that are required for VERMONT. Then get the latest development version from the repository:

git clone https://github.com/constcast/vermont.git

VERMONT can directly insert flows into the preprocessing system of flow-inspector. However, the module that is able to perform this task is still under active development at the time this blog post was written. As we aim at keeping our master development branch stable (since we only rarely create releases), this feature will only be merged into the main line after it received some more testing. 

As I'm not sure that I'll update this blog post when we merge the feature into the main branch, the flow-inspector connection might already be included into the master branch. You can check this by having a look at the documentation at https://github.com/constcast/vermont/wiki/Moduleconfiguration

If you can find documentation for the module ipfixFlowInspectorExporter on this site, then you can safely choose to build the master branch. Otherwise, you should checkout the branch "merge-features"

cd vermont
git branch merge-features origin/merge-features
git checkout merge-features 
If you checked out the proper branch, then you need to configure the system:

cmake -DSUPPORT_SCTP=OFF -DSUPPORT_REDIS=ON -DWITH_TOOLS=OFF .
If you successfully installed all the dependencies, you should get some output that looks similar to this one:

-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
-- Found hiredis: /usr/local/include, /usr/local/lib/libhiredis.dylib
-- Boost version: 1.53.0
-- Found boost libraries
-- Found LibXml2: /usr/lib/libxml2.dylib (found version "2.7.8")
-- Found libxml2 libraries
-- Found libxml2 libraries
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/braun/code/vermont 
If you don't get the message that the build files have been written, then you should check the error messages and install the missing packages. In case you encounter any problems that you cannot fix, just open an issue at our tracker and someone (probably me) will have a look at your problems.

Once you successfully ran cmake, try to build the system by running

make

This process might take a while. After it finished, you should see a binary file named "vermont" in the base source directory.  This file is the monitoring probe that can be run in order to generate the flow data. It requires a configuration file which defines the flow generation and export process. A sample file is shipped with vermont:

configs/flowinspector_exporter.xml

You need to change the configuration file using your favorite text editor. Open the file, search for the tag inside the module, and change the default interface to the main interface of your machine (e.g. eth0, eth1, wlan0, ...). You can keep all other configuration options as they are supplied in the file. Please note that we don't yet support IPv6 :(

Afterwards, you need to start vermont. As vermont needs to have the right to set your interface into promiscuous mode, you probably need to run it as root:

sudo ./vermont -f configs/flowinspector_exporter.xml
Make sure you have your preprocessor running. If VERMONT did not give you any error message, then it's time to do some browsing or any other kind of network-related activities that generate network traffic. 

Waiting for Data ...


It might be a good idea to get some coffee at this time, as it will take at least 10 minutes until some data will be imported into the database with the default configuration. 

The default configuration uses an inactive timeout of 5 minutes and an active timeout of 10 minutes for the flow generation process. In addition, flow-inspector will cache data for five minutes before committing it to the database. 

You can change this behavior, but I'd not recommend to do this unless you really know what you're doing. You can monitor the preprocessor in order to determine when the first data was inserted into the database. The preprocessor will output the number of flows it has processed every ten seconds. This number will (and should) be zero throughout the first five minutes, as VERMONT will not produce any flow data unless a flow timeout occurs (and this will not happen before at least 5 minutes have elapsed). 

Afterwards, you should see some flows being processed (if you generated some traffic in the meantime). Flow-inspector will cache the incoming data for 5 additional minutes before committing it to the database. Flow-inspector will also commit data if at least 100,000 flows have been seen (whatever happens first). It will inform you that about this commit process by printing the line

Live import. Flushing caches ...

After you have seen this line, you can start looking at your flow data in the web frontend.  

You can also force flow-inspector to commit the cached data by terminating the script (press STRG+C once). Wait for the preprocessor to finish, and you should see some flows in the frontend. You can afterwards start the preprocessor again to consume more flows.

 




Saturday, November 13, 2010

tftp-hpa setup instructions on mac

Just for backup purposes as I need this tftp server only on-demand:


***** Setup Instructions *****

NOTE: By default, tftp-hpa listens to the tftp port specified in /etc/services (port 69)
on all local addresses.

To run tftpd manually for download only access, use this command:
sudo tftpd -L -s

To run tftpd manually and support tftp uploads, add "-c" to the command:
sudo tftpd -L -c -s

You may run tftpd at system boot using the startupitem if you installed tftp-hpa
using the server variant. To load the startupitem using launchctl:
sudo launchctl load -w /Library/LaunchDaemons/org.macports.tftpd.plist

NOTE: When loading tftp-hpa using launchctl, make sure to place the files you want to serve
in /opt/local/var/tftp-hpa/, because that is the location set in the StartupItem.

*******************************

If you wish to run tftpd in inetd mode, you may make an inetd compatible .plist
file and replace the one installed by MacPorts in /Library/LaunchDaemons. You
may use /System/Library/LaunchDaemons/tftp.plist as a template.

Sunday, May 23, 2010

cpufreq on debian lenny

First, check whether you already have cpu frequency scaling enabled by running cpufreq-info:

# cpufreq-info
cpufrequtils 004: cpufreq-info (C) Dominik Brodowski 2004-2006
Report errors and bugs to cpufreq@lists.linux.org.uk, please.
analyzing CPU 0:
driver: acpi-cpufreq
CPUs which need to switch frequency at the same time: 0
hardware limits: 600 MHz - 1.70 GHz
available frequency steps: 1.70 GHz, 1.40 GHz, 1.20 GHz, 1000 MHz, 800 MHz, 600 MHz
available cpufreq governors: conservative, userspace, powersave, ondemand, performance
current policy: frequency should be within 600 MHz and 1.70 GHz.
The governor "ondemand" may decide which speed to use
within this range.
current CPU frequency is 600 MHz (asserted by call to hardware).
cpufreq stats: 1.70 GHz:4.56%, 1.40 GHz:0.18%, 1.20 GHz:0.00%, 1000 MHz:0.21%, 800 MHz:0.18%, 600 MHz:94.86% (10)


If you do not have an output like the one above, you need to configure your cpufreq-stuff. If you do not have cpufreq-info installed, run

apt-get install cpufrequtils sysfsutils


Afterwards, you have to load your cpu scaling manager. I usually use acpi_cpufreq:


modprobe acpi_cpufreq


Afterwards, you need to decided the policy manager. See this previous entry for more information.

You can use cpufreq_ondemand for example:

modprobe cpufreq_ondemand


Finally, you have to enable the frequency scaling by enabling the governor:

echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor


To make things permanent, you can run:

echo acpi_cpufreq >> /etc/modules
echo cpufreq_ondemand >> /etc/modules
echo "devices/system/cpu/cpu0/cpufreq/scaling_governor = ondemand" >> /etc/sysfs

Saturday, March 06, 2010

XCode: 1 source file parsed, no classes found or changed

The XCode Cocoa Application Tutorial states that one has to create a controller class (ConverterController in this tutorial). This controller needs to be included into the Interface Builder (IB) by "Open->Read Class File".

If this is done, there will be an error in the MainMenu XIB/NIB-Window that says: "1 source file parsed, no classes found or changed".

I'm not sure what the problem is (maybe there was a change for Snow Leopards and XCode 3.2.1), but I found a way on how to import the Class file. You need to select "Tools->Library" and select the "Classes" tab in the Library window. You will find the ConverterController class in this list. Double-Click it and the class will appear in your MainMenu.xib

Wednesday, February 03, 2010

New Window, New Frame

Create a new window with emacs:


C-x 4 b


Close the window:


C-x 4 0


New frames can be created with C-x 5 b ...

Tuesday, February 02, 2010

Emacs: Evaluating expressions

Use M-: to evaluate a single elisp expression. You can use this for example for setting variables. For example: If you enabled the mode auto-fill (M-x auto-fill-mode), the default wrap value for your lines is set to 70.

You can then use M-: to reset this to a more sane value by entering

(setq fill-column)

into the minibuffer.