--- /dev/null
+.. _tsdr-hbase-install-guide:
+
+TSDR HBase Data Store Installation Guide
+========================================
+
+This document is for the user to install the artifacts that are needed
+for using HBase Data Store in Time Series Data Repository, which is
+a new feature available in OpenDaylight Lithium release.
+
+Overview
+--------
+
+The Time Series Data Repository (TSDR) project in OpenDaylight (ODL) creates a
+framework for collecting, storing, querying, and maintaining time series data
+in the OpenDaylight SDN controller. It contains the following services and
+components:
+
+ Data Collection Service
+ Data Storage Service
+ TSDR Persistence Layer with data stores as plugins
+ TSDR Data Stores
+ Data Query Service
+ Data Aggregation Service
+ Data Purging Service
+
+Data Collection Service handles the collection of time series data into TSDR
+and hands it over to Data Storage Service. Data Storage Service stores the data
+into TSDR through TSDR Persistence Layer. TSDR Persistence Layer provides
+generic Service APIs allowing various data stores to be plugged in. Data
+Aggregation Service aggregates time series fine-grained raw data into
+course-grained roll-up data to control the size of the data. Data Purging
+Service periodically purges both fine-grained raw data and course-granined
+aggregated data according to user-defined schedules.
+
+
+In Lithium, we implemented Data Collection Service, Data Storage Service, TSDR
+Persistence Layer, TSDR HBase Data Store, and TSDR H2 Data Store. Among these
+services and components, time series data is communicated using a common TSDR
+data model, which is designed and implemented for the abstraction of the time
+series data commonalities. With these functions, TSDR will be able to collect
+the data from the data sources and store them into one of the TSDR data stores:
+either HBase Data Store or H2 Data Store. We also provided a simple query
+command from Karaf console for the user to retrieve TSDR data from the data
+stores.
+
+A future release will contain Data Aggregation service, Data Purging Service,
+and a full-fledged Data Query Service with Norghbound APIs.
+
+
+Prerequisites for Installing TSDR HBase Data Store
+--------------------------------------------------
+
+The hardware requirements are the same as those for standard ODL controller
+installation.
+
+The software requirements for TSDR HBase Data Store are as follows:
+
+ The supported operating system for TSDR HBase Data Store is Linux. We do not
+ support TSDR HBase Data Store on Windows.
+ Besides the software that ODL requires, we also require HBase database
+ running on top of Hadoop single node.
+
+Preparing for Installation
+--------------------------
+
+Download HBase (version number to be finalized) from the following website.
+
+http://archive.apache.org/dist/hbase/hbase-0.94.15/
+
+
+Installing TSDR HBase Data Store
+--------------------------------
+
+Installing TSDR HBase Data Store contains two steps:
+
+ Installing HBase server, and
+ Installing TSDR HBase Data Store features from ODL Karaf console.
+
+This installation guide will only cover the first step. For installing TSDR
+HBase Data Store features, please refer to TSDR HBase Data Store User Guide.
+
+In Lithium, we only support HBase single node running together on the same
+machine as ODL controller. Therefore, follow the steps to download and install
+HBase server onto the same box as where ODL controller is running:
+
+ Create a folder in Linux operating system for the HBase server.
+
+For example, create an hbase directory under /usr/lib:
+
+ ::
+
+ mkdir /usr/lib/hbase
+
+Unzip the downloaded HBase server tar file.
+
+Run the following command to unzip the installation package:
+
+ ::
+
+ tar xvf <hbase-installer-name> /usr/lib/hbase
+
+Make proper changes in hbase-site.xml
+
+ .. Under <hbase-install-directory>/conf/, there is a hbase-site.xml.
+ .. Although it is not recommended, an experience user with HBase canmodify
+ .. the data directory for hbase server to store the data.
+
+ .. Modify the value of the property with name "hbase.rootdir" in the file
+ .. to reflect the desired file directory for storing hbase data.
+
+The following is an example of the file:
+
+ ::
+
+ <configuration>
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///usr/lib/hbase/data</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.dataDir</name>
+ <value>/usr/lib/hbase/zookeeper</value>
+ </property>
+ </configuration>
+
+Verifying your Installation
+---------------------------
+
+After the HBase server is properly installed, start hbase server and hbase shell.
+
+ ::
+
+ start hbase server
+ cd <hbase-installation-directory>
+ ./start-hbase.sh
+
+ start hbase shell
+ cd <hbase-insatllation-directory>
+ ./hbase shell
+
+Post Installation Configuration
+-------------------------------
+
+Please refer to HBase Data Store User Guide.
+
+Upgrading From a Previous Release
+---------------------------------
+
+Lithium is the first release of TSDR. Upgrading is not applicable for TSDR Lithium release.
+
+Uninstalling HBase Data Store
+-----------------------------
+
+To uninstall TSDR HBase Data Store,
+
+.. code-block:: bash
+
+ stop hbase server
+ cd <hbase-installation-directory>
+ ./stop-hbase.sh
+
+To remove the file directory that contains the HBase server installation.
+
+.. code-block:: bash
+
+ rm -r <hbase-installation-directory>
--- /dev/null
+.. _tsdr-hsqldb-install-guide:
+
+TSDR HSQLDB Default Datastore Installation Guide
+================================================
+
+This document is for the user to install the artifacts that are needed
+for using Time Series Data Repository (TSDR) functionality in the ODL
+Controller by enabling the default HSQLDB Datastore. TSDR is new functionality
+added in OpenDaylight in Lithium Release.
+
+Overview
+--------
+
+In Lithium Release the time series data records of OpenFlow statistics are
+collected periodically and stored in a persistent store. For non-production
+usage, the bundled default datastore (HSQLDB) is utilized based on odl-tsdr-all
+feature installation. The TSDR records get persisted in HSQLDB store in
+<install folder>/tsdr/ folder by default.
+
+Installing TSDR with default HSQLDB datastore
+---------------------------------------------
+
+Once OpenDaylight distribution is up, from karaf console install the TSDR
+feature with default datastore (HSQLDB store used) can be installed by::
+
+ feature:install odl-tsdr-all
+
+
+This will install all dependency features (and can take sometime) before
+returning control to the console.
+
+Verifying your Installation
+---------------------------
+
+If the feature install was successful you should be able to see the following
+tsdr commands added::
+
+ tsdr:list
+
+
+Troubleshooting
+---------------
+
+Check the ../data/log/karaf.log for any exception related to TSDR or HSQLDB
+related features.
+
+Post Installation Configuration
+-------------------------------
+
+The feature installation takes care of automated configuration of the
+datasource by installing a file in
+<install folder>/etc named org.ops4j.datasource-metric.cfg.
+This contains the default location of <install folder>/tsdr where the HSQLDB
+datastore files are stored. If you want to change the default location of the
+datastore files to some other location update the last portion of the url
+property in the org.ops4j.datasource-metric.cfg and then restart the karaf
+container
+
+Upgrading From a Previous Release
+---------------------------------
+
+Lithium being the first release supporting TSDR functionality, only fresh
+installation is possible. However if you want to move to production usage by
+enabling the store HBase for TSDR usage, you can do it by uninstalling the TSDR
+with default HSQLDB datastore, restarting the Karaf container and then enabling
+the TSDR with HBase store as documented in tsdr-hbase-install.doc
+
+Uninstalling TSDR with default HSQLDB datastore
+-----------------------------------------------
+
+To uninstall the TSDR functionality with the default store, you need to do the
+following from karaf console
+
+.. code-block:: bash
+
+ feature:uninstall odl-tsdr-all
+ feature:uninstall odl-tsdr-core
+ feature:uninstall odl-tsdr-HSQLDB-persistence
+
+
+It's recommended to restart the Karaf container after uninstallation of the TSDR
+functionality with the default store
--- /dev/null
+.. _tsdr-install-guide:
+
+TSDR Installation Guide
+=======================
+
+This document is for the user to install the artifacts that are needed
+for using Time Series Data Repository (TSDR) functionality in the ODL
+Controller by enabling either an HSQLDB, HBase, or Cassandra Data Store.
+
+
+Overview
+--------
+
+The Time Series Data Repository (TSDR) project in OpenDaylight (ODL) creates a
+framework for collecting, storing, querying, and maintaining time series data
+in the OpenDaylight SDN controller. Please refer to the User Guide for the
+detailed description of the functionality of the project and how to use the
+corresponding features provided in TSDR.
+
+Pre Requisites for Installing TSDR
+----------------------------------
+
+The software requirements for TSDR HBase Data Store are as follows:
+
+* In the case when the user chooses HBase or Cassandra data store, besides the
+software that ODL requires, we also require HBase and Cassandra database
+running in single node deployment scenario.
+
+No additional software is required for the HSQLDB Data Stores.
+
+Preparing for Installation
+--------------------------
+
+* When using HBase data store, download HBase from the following website:
+
+ http://archive.apache.org/dist/hbase/hbase-0.94.15/
+
+* When using Cassandra data store, download Cassandra from the following website:
+
+ http://www.eu.apache.org/dist/cassandra/2.1.10/
+
+* No additional steps are required to install the TSDR HSQL Data Store.
+
+Installing TSDR Data Stores
+---------------------------
+
+Installing HSQLDB Data Store
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Once OpenDaylight distribution is up, from karaf console install the HSQLDB
+data store using the following command:
+
+ ::
+
+ feature:install odl-tsdr-hsqldb-all
+
+
+This will install hsqldb related dependency features (and can take sometime) as
+well as OpenFlow statistics collector before returning control to the console.
+
+
+Installing HBase Data Store
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Installing TSDR HBase Data Store contains two steps:
+
+#. Installing HBase server, and
+#. Installing TSDR HBase Data Store features from ODL Karaf console.
+
+In this release, we only support HBase single node running together on the same
+machine as OpenDaylight. Therefore, follow the steps to download and install
+HBase server onto the same machine as where OpenDaylight is running:
+
+#. Create a folder in Linux operating system for the HBase server
+
+ For example, create an hbase directory under ``/usr/lib``::
+
+ mkdir /usr/lib/hbase
+
+
+#. Unzip the downloaded HBase server tar file
+
+ Run the following command to unzip the installation package::
+
+ tar xvf <hbase-installer-name> /usr/lib/hbase
+
+
+#. Make proper changes in hbase-site.xml
+
+ #. Under <hbase-install-directory>/conf/, there is a ``hbase-site.xml``
+
+ Although it is not recommended, an experienced user with HBase can modify
+ the data directory for hbase server to store the data.
+
+ #. Modify the value of the property with name "hbase.rootdir" in the file to
+ reflect the desired file directory for storing hbase data.
+
+ The following is an example of the file::
+
+ <configuration>
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///usr/lib/hbase/data</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.dataDir</name>
+ <value>/usr/lib/hbase/zookeeper</value>
+ </property>
+ </configuration>
+
+
+#. Start hbase server
+
+ .. code-block:: bash
+
+ cd <hbase-installation-directory>
+ ./start-hbase.sh
+
+#. Start hbase shell
+
+ .. code-block:: bash
+
+ cd <hbase-insatllation-directory>
+ ./hbase shell
+
+#. Start Karaf console
+
+#. Install hbase data store feature from Karaf console:
+
+ .. code-block:: bash
+
+ feature:install odl-tsdr-hbase
+
+
+Installing Cassandra Data Store
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Installing TSDR Cassandra Data Store contains two steps:
+
+#. Installing Cassandra server, and
+#. Installing TSDR Cassandra Data Store features from ODL Karaf console.
+
+ In this release, we only support Cassadra single node running together on the
+ same machine as OpenDaylight. Therefore, follow these steps to download and
+ install Cassandra server onto the same machine as where OpenDaylight is
+ running.
+
+#. Install Cassandra (latest stable version) by downloading the zip file and
+ untar the tar ball to cassandra/ directory on the testing machine.
+
+ .. code-block:: bash
+
+ mkdir cassandra
+ wget http://www.eu.apache.org/dist/cassandra/2.1.10/apache-cassandra-2.1.10-bin.tar.gz[2.1.10 is current stable version, it can vary]
+ mv apache-cassandra-2.1.10-bin.tar.gz cassandra/
+ cd cassandra
+ tar -xvzf apache-cassandra-2.1.10-bin.tar.gz
+
+
+#. Start Cassandra from cassandra directory
+
+ .. code-block:: bash
+
+ ./apache-cassandra-2.1.10/bin/cassandra
+
+#. Start cassandra shell
+
+ .. code-block:: bash
+
+ ./apache-cassandra-2.1.10/bin/cqlsh
+
+#. Install Cassandra data store feature from Karaf console
+
+ .. code-block::
+
+ feature:install odl-tsdr-cassandra
+
+Verifying your Installation
+---------------------------
+
+After the TSDR data store is installed, no matter whether it is HBase data
+store, Cassandra data store, or HSQLDB data store, the user can verify the
+installation with the following steps.
+
+#. Verify if the following two TSDR commands are available from Karaf console:
+
+ .. code-block:: bash
+
+ tsdr:list
+ tsdr:purgeAll
+
+
+#. Verify if OpenFlow statistics data can be received successfully:
+
+ .. code-block:: bash
+
+ feature:install odl-tsdr-openflow-statistics-collector
+
+#. Run mininet to connect to ODL controller. For example, use the following
+ command to start a three node topology:
+
+ .. code-block:: bash
+
+ mn --topo single,3 --controller 'remote,ip=172.17.252.210,port=6653' --switch ovsk,protocols=OpenFlow13
+
+
+From Karaf console, the user should be able to retrieve the statistics data of
+OpenFlow statistics data from the console::
+
+ tsdr:list FLOWSTATS
+
+Troubleshooting
+^^^^^^^^^^^^^^^
+
+Check the ``../data/log/karaf.log`` for any exception related to TSDR features.
+
+Post Installation Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Post Installation Configuration for HSQLDB Data Store
+"""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The feature installation takes care of automated configuration of the
+datasource by installing a file in <install folder>/etc named
+org.ops4j.datasource-metric.cfg. This contains the default location of
+<install folder>/tsdr where the HSQLDB datastore files are stored. If you want
+to change the default location of the datastore files to some other location
+update the last portion of the url property in the
+org.ops4j.datasource-metric.cfg and then restart the Karaf container.
+
+Post Installation Configuration for HBase Data Store
+""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Please refer to HBase Data Store User Guide.
+
+Post Installation Configuration for Cassandra Data Store
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+There is no post configuration for TSDR Cassandra data store.
+
+Upgrading From a Previous Release
+---------------------------------
+
+The HBase data store was supported in the previous release as well as in this
+release. However, we do not support data store upgrade for HBase data store.
+The user needs to reinstall TSDR and start to collect data in TSDR HBase
+datastore after the installation.
+
+HSQLDB and Cassandra are new data stores introduced in this release.
+Therefore, upgrading from previous release does not apply in these two data
+store scenarios.
+
+Uninstalling TSDR Data Stores
+-----------------------------
+
+To uninstall TSDR HSQLDB data store
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To uninstall the TSDR functionality with the default store, you need to do the
+following from karaf console:
+
+.. code-block:: bash
+
+ feature:uninstall odl-tsdr-hsqldb-all
+ feature:uninstall odl-tsdr-core
+ feature:uninstall odl-tsdr-hsqldb
+ feature:uninstall odl-tsdr-openflow-statistics-collector
+
+It is recommended to restart the Karaf container after the uninstallation of
+the TSDR functionality with the default store.
+
+To uninstall TSDR HBase Data Store
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To uninstall the TSDR functionality with the HBase data store,
+
+* Uninstall HBase data store related features from karaf console
+
+ .. code-block:: bash
+
+ feature:uninstall odl-tsdr-hbase
+ feature:uninstall odl-tsdr-core
+
+* Stop hbase server
+
+ .. code-block:: bash
+
+ cd <hbase-installation-directory>
+ ./stop-hbase.sh
+
+* Remove the file directory that contains the HBase server installation:
+
+ .. code-block:: bash
+
+ rm -r <hbase-installation-directory>
+
+
+It is recommended to restart the Karaf container after the uninstallation of
+the TSDR data store.
+
+To uninstall TSDR Cassandra Data Store
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To uninstall the TSDR functionality with the Cassandra store,
+
+* uninstall cassandra data store related features following from karaf console:
+
+ .. code-block:: bash
+
+ feature:uninstall odl-tsdr-cassandra
+ feature:uninstall odl-tsdr-core
+
+* stop cassandra database::
+
+ ps auwx | grep cassandra
+ sudo kill pid
+
+* remove the cassandra installation files::
+
+ rm <cassandra-installation-directory>
+
+It is recommended to restart the Karaf container after uninstallation of the
+TSDR data store.
==============
The Time Series Data Repository (TSDR) project in OpenDaylight (ODL)
creates a framework for collecting, storing, querying, and maintaining
-time series data from multiple similar and disparate data sources.
+time series data from many similar and disparate data sources.
* **TSDR Features URL:** https://git.opendaylight.org/gerrit/gitweb?p=tsdr.git;a=blob;f=features/features-tsdr/pom.xml
* **Installation Guide(s):**
- * :ref:`TSDR Installation Guide <tsdr-install-guide>`
+ * `TSDR Installation Guide <https://github.com/opendaylight/docs/blob/master/docs/getting-started-guide/project-specific-guides/tsdr-install-guide.rst>`
+ * `TSDR HBase Installation Guide <https://github.com/opendaylight/docs/blob/master/docs/getting-started-guide/project-specific-guides/tsdr-hbase-install-guide.rst>`
+ * `TSDR HSQLDB Installation Guide <https://github.com/opendaylight/docs/blob/master/docs/getting-started-guide/project-specific-guides/tsdr-hsqldb-install-guide.rst>`
* **User Guide(s):**
- * :ref:`TSDR User Guide <tsdr-user-guide>`
-
-* **HSQLDB TSDR User Guide:** https://github.com/opendaylight/docs/blob/stable/lithium/manuals/user-guide/src/main/asciidoc/tsdr/tsdr-hsqldb-user.adoc
-* **HBase TSDR User Guide:** https://github.com/opendaylight/docs/blob/stable/lithium/manuals/user-guide/src/main/asciidoc/tsdr/tsdr-hbase-user.adoc
+ * `TSDR User Guide <https://github.com/opendaylight/docs/blob/master/docs/user-guide/tsdr-user-guide.rst>`
+ * `TSDR HSQLDB User Guide <https://github.com/opendaylight/docs/blob/master/docs/user-guide/tsdr-hsqldb-user-guide.rst>`
+ * `TSDR HBase User Guide <https://github.com/opendaylight/docs/blob/master/docs/user-guide/tsdr-hbase-user-guide.rst>`
+ * `TSDR ElasticSearch User Guide <https://github.com/opendaylight/docs/blob/master/docs/user-guide/tsdr-elastic-search-user-guide.rst>`
+ * `<coming soon> TSDR Grafana User Guide <tsdr-grafana-user-guide>`
Security Considerations
=======================
* List of features/APIs which are EOLed, deprecated, and/or removed in this release
- * SNMP data collector was temporarily removed.
+ * experimental SNMP data collector was temporarily removed until further support is found.
+ * experimental sFlow data collector was temporarily removed until further support is found.
Standards
=========
=================
* `Link to release plan <https://wiki.opendaylight.org/view/TSDR:TSDR_Oxygen_Release_Plan>`_
-* Describe any major shifts in release schedule from the release plan
- * N/A.
+* Describe any major shifts in release schedule from the release plan. None
+
--- /dev/null
+.. _tsdr-elasticsearch-user-guide:
+
+ElasticSearch
+=============
+
+Setting Up the environment
+--------------------------
+
+To setup and run the TSDR data store ElasticSearch feature, you need to have
+an ElasticSearch node (or a cluster of such nodes) running. You can use a
+customized ElasticSearch docker image for this purpose.
+
+Your ElasticSearch (ES) setup must have the "Delete By Query Plugin" installed.
+Without this, some of the ES functionality won't work properly.
+
+
+Creating a custom ElasticSearch docker image
+--------------------------------------------
+
+(You can skip this section if you already have an instance of ElasticSearch running)
+
+Run the following set of commands:
+
+ ::
+
+ cat << EOF > Dockerfile
+ FROM elasticsearch:2
+ RUN /usr/share/elasticsearch/bin/plugin install --batch delete-by-query
+ EOF
+
+
+To build the image, run the following command in the directory where the
+Dockerfile was created:
+
+ ::
+
+ docker build . -t elasticsearch-dd
+
+
+You can check whether the image was properly created by running:
+
+ ::
+
+ docker images
+
+
+This should print all your container images including the elasticsearch-dd.
+
+Now we can create and run a container from our image by typing:
+
+ ::
+
+ docker run -d -p 9200:9200 -p 9300:9300 --name elasticsearch-dd elasticsearch-dd
+
+
+To see whether the container is running, run the following command:
+
+ ::
+
+ docker ps
+
+
+The output should include a row with elasticsearch-dd in the NAMES column.
+To check the std out of this container use
+
+ ::
+
+ docker logs elasticsearch-dd
+
+
+Running the ElasticSearch feature
+---------------------------------
+
+Once the features have been installed, you can change some of its properties. For
+example, to setup the URL where your ElasticSearch installation runs,
+change the *serverUrl* parameter in tsdr/persistence-elasticsearch/src/main/resources/configuration/initial/:
+
+ tsdr-persistence-elasticsearch.properties
+
+
+All the data are stored into the TSDR index under a type. The metric data are
+stored under the metric type and the log data are store under the log type.
+You can modify the files in tsdr/persistence-elasticsearch/src/main/resources/configuration/initial/:
+
+ tsdr-persistence-elasticsearch_metric_mapping.json
+ tsdr-persistence-elasticsearch_log_mapping.json
+
+to change or tune the mapping for those types. The changes in those files will be promoted after
+the feature is reloaded or the distribution is restarted.
+
+
+Testing the setup
+^^^^^^^^^^^^^^^^^
+
+We can now test whether the setup is correct by downloading and installing mininet,
+which we use to send some data to the running ElasticSearch instance.
+
+Installing the necessary features:
+
+Start OpenDaylight
+
+ ::
+
+ feature:install odl-restconf odl-l2switch-switch odl-tsdr-core odl-tsdr-openflow-statistics-collector
+ feature:install odl-tsdr-elasticsearch
+
+We can check whether the distribution is now listening on port 6653:
+
+ ::
+
+ netstat -an | grep 6653
+
+Run mininet
+
+ ::
+
+ sudo mn --topo single,3 --controller 'remote,ip=distro_ip,port=6653' --switch ovsk,protocols=OpenFlow13
+
+
+where the distro_ip is the IP address of the machine where the OpenDaylight distribution
+is running. This command will create three hosts connected to one OpenFlow capable
+switch.
+
+We can check if data was stored by ElasticSearch in TSDR by running the
+following command:
+
+ ::
+
+ tsdr:list FLOWTABLESTATS
+
+The output should look similar to the following::
+
+ [NID=openflow:1][DC=FLOWTABLESTATS][MN=ActiveFlows][RK=Node:openflow:1,Table:50][TS=1473427383598][3]
+ [NID=openflow:1][DC=FLOWTABLESTATS][MN=PacketMatch][RK=Node:openflow:1,Table:50][TS=1473427383598][12]
+ [NID=openflow:1][DC=FLOWTABLESTATS][MN=PacketLookup][RK=Node:openflow:1,Table:50][TS=1473427383598][12]
+ [NID=openflow:1][DC=FLOWTABLESTATS][MN=ActiveFlows][RK=Node:openflow:1,Table:80][TS=1473427383598][3]
+ [NID=openflow:1][DC=FLOWTABLESTATS][MN=PacketMatch][RK=Node:openflow:1,Table:80][TS=1473427383598][17]
+ [NID=openflow:1][DC=FLOWTABLESTATS][MN=PacketMatch][RK=Node:openflow:1,Table:246][TS=1473427383598][19]
+ ...
+
+Or you can query your ElasticSearch instance:
+
+ ::
+
+ curl -XPOST "http://elasticseach_ip:9200/_search?pretty" -d'{ "from": 0, "size": 10000, "query": { "match_all": {} } }'
+
+The elasticseach_ip is the IP address of the server where the ElasticSearch is running.
+
+
+Web Activity Collector
+======================
+
+The Web Activity Collector records the meaningful REST requests made through the
+OpenDaylight RESTCONF interface.
+
+
+How to test the RESTCONF Collector
+----------------------------------
+
+- Install some other feature that has a RESTCONF interface, for example. "odl-tsdr-syslog-collector"
+- Issue a RESTCONF command that uses either POST,PUT or DELETE.
+ For example, you could call the register-filter RPC of tsdr-syslog-collector.
+- Look up data in TSDR database from Karaf.
+
+ ::
+
+ tsdr:list RESTCONF
+
+
+- You should see the request that you have sent, along with its information
+ (URL, HTTP method, requesting IP address and request body)
+- Try to send a GET request, then check again, your request should not be
+ registered, because the collector does not register GET requests by default.
+- Open the file: "etc/tsdr.restconf.collector.cfg", and add GET to the list of
+ METHODS_TO_LOG, so that it becomes:
+
+
+ METHODS_TO_LOG=POST,PUT,DELETE,GET
+
+ - Try again to issue your GET request, and check if it was recorded this time,
+ it should be recorder.
+ - Try manipulating the other properties (PATHS_TO_LOG (which URLs do we want
+ to log from), REMOTE_ADDRESSES_TO_LOG (which requesting IP addresses do we
+ want to log from) and CONTENT_TO_LOG (what should be in the request's body
+ in order to log it)), and see if the requests are getting logged.
+ - Try providing invalid properties (unknown methods for the METHODS_TO_LOG
+ parameter, or the same method repeated multiple times, and invalid regular
+ expressions for the other parameters), then check karaf's log using
+ "log:display". It should tell you that the value is invalid, and that it
+ will use the default value instead.
+
--- /dev/null
+.. _tsdr-hbase-user-guide:
+
+TSDR HBase Data Store User Guide
+================================
+
+This document describes how to use HBase to capture time series data
+using Time Series Data Repository (TSDR) feature in the OpenDaylight
+Lithium release. This document contains configuration, administration,
+management, usage, and troubleshooting sections for the feature.
+
+Overview
+--------
+
+The Time Series Data Repository (TSDR) project in OpenDaylight (ODL) creates a framework for collecting, storing, querying, and maintaining time series data in then OpenDaylight SDN controller. TSDR provides the framework for plugging in proper data collectors to collect various time series data and store into TSDR data stores. With a common data model and generic TSDR data persistence APIs, the user could choose various data stores to be plugged into TSDR Persistence framework. In Lithium, two types of data stores are supported: HBase NoSQL database and H2 relational database.
+
+With the capabilities of data collection, storage, query, aggregation, and purging provided by TSDR, network administrators could leverage various data driven appliations built on top of TSDR for security risk detection, performance analysis, operational configuration optimization, traffic engineering, and network analytics with automated intelligence.
+
+TSDR with HBase Data Store Architecture
+---------------------------------------
+
+TSDR contains the following services and components
+
+- Data Collection Service
+- Data Storage Service
+- TSDR Persistence Layer with data stores as plugins
+- TSDR Data Stores
+- Data Query Service
+- Data Aggregation Service
+- Data Purging Service
+
+Data Collection Service handles the collection of time series data into TSDR and hands it over to Data Storage Service. Data Storage Service stores the data into TSDR through TSDR Persistence Layer. TSDR Persistence Layer provides generic Service APIs allowing various data stores to be plugged in. Data Aggregation Service aggregates time series fine-grained raw data into course-grained roll-up data to control the size of the data. Data Purging Service periodically purges both fine-grained raw data and course-granined aggregated data according to user-defined schedules.
+
+
+In Lithium, we implemented Data Collection Service, Data Storage Service, TSDR Persistence Layer, TSDR HBase Data Store, and TSDR H2 Data Store. Among these services and components, time series data is communicated using a common TSDR data model, which is designed and implemented for the abstraction of time series data commonalities. With these functions, TSDR will be able to collect the data from the data sources and store them into one of the TSDR data stores: either HBase Data Store or H2 Data Store. We also provided a simple query command from Karaf console for the user to retrieve TSDR data from the data stores.
+
+
+A future release will contain Data Aggregation service, Data Purging Service, and a full-fledged Data Query Service with Norghbound APIs.
+
+Configuring TSDR with HBase Data Store
+--------------------------------------
+
+After installing HBase Server on the same VM as the OpenDaylight Controller, if the user accepts the default configuration of the HBase Data Store, the user can directly proceed with the installation of HBase Data Store from Karaf console.
+
+Optionally, the user can configure TSDR HBase Data Store following HBase Data Store Configuration Procedure.
+
+HBase Data Store Configuration Steps
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+- Open the file etc/tsdr-persistence-hbase.peroperties under karaf distribution directory.
+- Edit the following parameters
+ - HBase server name
+ - HBase server port
+ - HBase client connection pool size
+ - HBase client write buffer size
+
+After the configuration of HBase Data Store is complete, proceed with the installation of HBase Data Store from Karaf console.
+
+- HBase Data Store Installation Steps
+ - Start Karaf Console
+ - Run the following commands from Karaf Console:
+
+ ::
+
+ feature:install odl-tsdr-hbase
+
+
+Administering or Managing TSDR HBase Data Store
+-----------------------------------------------
+
+Using Karaf Command to retrieve data from HBase Data Store
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The user can retrieve the data from HBase data store using the following commands from Karaf console:
+
+ ::
+
+ tsdr:list
+
+ ::
+
+ tsdr:list <CategoryName> <StartTime> <EndTime>
+
+Typing tab will get the context prompt of the arguments when typeing the command in Karaf console.
+
+Troubleshooting issues with log files
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+- Karaf logs
+
+Similar to other OpenDaylight components and features, TSDR HBase Data Store writes logging information to Karaf logs. All the information messages, warnings, error messages, and debug messages are written to Karaf logs.
+
+- HBase logs
+
+For HBase system level logs, the user can check standard HBase server logs, which is under <HBase-installation-directory>/logs.
+
+Tutorials
+=========
+
+How to use TSDR to collect, store, and view OpenFlow Interface Statistics
+
+Overview
+--------
+
+This tutorial describes an example of using TSDR to collect, store, and view one type of time series data in OpenDaylight environment.
+
+
+Prerequisites
+-------------
+
+You would need to have the following as prerequisits:
+ - One or multiple OpenFlow enabled switches. Alternatively, you can use mininet to simulate such a switch.
+ - Successfully installed OpenDaylight Controller.
+ - Successfully installed HBase Data Store following TSDR HBase Data Store Installation Guide.
+ - Connect the OpenFlow enabled switch(es) to OpenDaylight Controller.
+
+Target Environment
+------------------
+
+HBase data store is only supported in Linux operation system.
+
+Instructions
+------------
+
+- Start OpenDaylight controller.
+
+- Connect OpenFlow enabled switch(es) to the controller. If using mininet, run the following commands from mininet command line:
+
+ ::
+
+ mn --topo single,3 --controller 'remote,ip=172.17.252.210,port=6653' --switch ovsk,protocols=OpenFlow13
+
+- If using real switch(es), the OpenDaylight controller should be able to discover the network toplogy containing the switches.
+
+- Install tsdr hbase feature from Karaf:
+
+ ::
+
+ feature:install odl-tsdr-hbase
+
+- run the following command from Karaf console:
+
+ ::
+
+ tsdr:list InterfaceStats
+
+You should be able to see the interface statistics of the switch(es) from the HBase Data Store. If there are too many rows, you can use "tsdr:list InterfaceStats|more" to view it page by page.
+
+By tabbing after "tsdr:list", you will see all the supported data categories. For example, "tsdr:list FlowStats" will output the Flow statistics data collected from the switch(es).
+
--- /dev/null
+.. _tsdr-hsqldb-user-guide:
+
+TSDR HSQLDB Datastore User Guide
+================================
+
+This document describes how to use the embedded datastore HSQLDB, which is the default datastore (recommended for non-production use case) introduced as part of OpenDaylight Time Series Data Respository (TSDR) project. This store captures the time series data. This document contains configuration, administration, management, using, and troubleshooting sections for the feature.
+
+Overview
+--------
+
+In the Lithium Release of Time Series Data Repository (TSDR), time series metrics corresponding to the OpenFlow statistics are captured. For users trying out things on their laptop or non-production environment, TSDR functionality can be enabled with default datastore HSQLDB by installing the feature odl-tsdr-all.
+
+TSDR Architecture
+-----------------
+
+The following wiki pages capture the TSDR Model/Architecture
+
+a. https://wiki.opendaylight.org/view/TSDR_Data_Storage_Service_and_Persistence_with_HBase_Plugin
+b. https://wiki.opendaylight.org/view/TSDR_Data_Collection_Service
+
+Note in Lithium the DataCollection Service is implemented just for OpenFlow Statistics metrics.
+
+Administering or Managing TSDR with default datastore HSQLDB
+------------------------------------------------------------
+
+Once the TSDR default datastore feature (odl-tsdr-all) is enabled, the TSDR captured OpenFlow statistics metrics can be accessed from Karaf Console by executing the command
+
+ ::
+
+ tsdr:list <metric-category> <starttimestamp> <endtimestamp>
+
+wherein
+
+ ::
+
+ <metric-category> = any one of the following categories FlowGroupStats, FlowMeterStats, FlowStats, FlowTableStats, PortStats, QueueStats
+ <starttimestamp> = to filter the list of metrics starting this timestamp
+ <endtimestamp> = to filter the list of metrics ending this timestamp
+
+If either of <starttimestamp> or <endtimestamp> is not specified, this command displays the latest 1000 metrics captured.
+
- Start Karaf Console
- - Run the following commands from Karaf Console: feature:install
- odl-tsdr-hbase
+ - Run the following commands from Karaf Console:
+
+ ::
+
+ feature:install odl-tsdr-hbase
To Configure Cassandra Data Store
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
enabled, the TSDR captured OpenFlow statistics metrics can be accessed
from Karaf Console by executing the command
-::
+ ::
- tsdr:list <metric-category> <starttimestamp> <endtimestamp>
+ tsdr:list <metric-category> <starttimestamp> <endtimestamp>
wherein
To Administer HBase Data Store
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Using Karaf Command to retrieve data from HBase Data Store
+Using Karaf Command to retrieve data from HBase Data Store
The user first need to install hbase data store from karaf console:
-feature:install odl-tsdr-hbase
+ ::
+
+ feature:install odl-tsdr-hbase
The user can retrieve the data from HBase data store using the following
commands from Karaf console:
-::
+ ::
- tsdr:list
- tsdr:list <CategoryName> <StartTime> <EndTime>
+ tsdr:list
+
+ ::
+
+ tsdr:list <CategoryName> <StartTime> <EndTime>
Typing tab will get the context prompt of the arguments when typeing the
command in Karaf console.
The user first needs to install Cassandra data store from Karaf console:
-::
+ ::
- feature:install odl-tsdr-cassandra
+ feature:install odl-tsdr-cassandra
Then the user can retrieve the data from Cassandra data store using the
following commands from Karaf console:
-::
+ ::
+
+ tsdr:list
- tsdr:list
- tsdr:list <CategoryName> <StartTime> <EndTime>
+ ::
+
+ tsdr:list <CategoryName> <StartTime> <EndTime>
Typing tab will get the context prompt of the arguments when typeing the
command in Karaf console.
feature:install odl-tsdr-netflow-statistics-collector
-- sFlow Data Collector
+- Syslog Data Collector
::
- feature:install odl-tsdr-sflow-statistics-colletor
+ feature:install odl-tsdr-syslog-collector
-- SNMP Data Collector
+- Controller Metrics Collector
::
- feature:install odl-tsdr-snmp-data-collector
+ feature:install odl-tsdr-controller-metrics-collector
-- Syslog Data Collector
+- Web Activity Collector
::
- feature:install odl-tsdr-syslog-collector
+ feature:install odl-tsdr-restconf-collector
-- Controller Metrics Collector
+- sFlow Data Collector (experimental)
::
- feature:install odl-tsdr-controller-metrics-collector
+ feature:install odl-tsdr-sflow-statistics-colletor
-- Web Activity Collector
+- SNMP Data Collector (experimental)
::
- feature:install odl-tsdr-restconf-collector
+ feature:install odl-tsdr-snmp-data-collector
In order to use controller metrics collector, the user needs to install
directory in your controller home directory and place the
"sigar-1.6.4.jar" there
+
+Querying TSDR from REST APIs
+----------------------------
+
+TSDR provides two REST APIs for querying data stored in TSDR data
+stores.
+
+- Query of TSDR Metrics
+
+ - URL: http://localhost:8181/tsdr/metrics/query
+
+ - Verb: GET
+
+ - Parameters:
+
+ - tsdrkey=[NID=][DC=][MN=][RK=]
+
+ ::
+
+ The TSDRKey format indicates the NodeID(NID), DataCategory(DC), MetricName(MN), and RecordKey(RK) of the monitored objects.
+ For example, the following is a valid tsdrkey:
+ [NID=openflow:1][DC=FLOWSTATS][MN=PacketCount][RK=Node:openflow:1,Table:0,Flow:3]
+ The following is also a valid tsdrkey:
+ tsdrkey=[NID=][DC=FLOWSTATS][MN=][RK=]
+ In the case when the sections in the tsdrkey is empty, the query will return all the records in the TSDR data store that matches the filled tsdrkey. In the above example, the query will return all the data in FLOWSTATS data category.
+ The query will return only the first 1000 records that match the query criteria.
+
+ - from=<time\_in\_seconds>
+
+ - until=<time\_in\_seconds>
+
+The following is an example curl command for querying metric data from
+TSDR data store:
+
+ ::
+
+ curl -G -v -H "Accept: application/json" -H "Content-Type:
+ application/json" "http://localhost:8181/tsdr/metrics/query"
+ --data-urlencode "tsdrkey=[NID=][DC=FLOWSTATS][MN=][RK=]"
+ --data-urlencode "from=0" --data-urlencode "until=240000000000"\|more
+
+- Query of TSDR Log type of data
+
+ - URL:http://localhost:8181/tsdr/logs/query
+
+ - Verb: GET
+
+ - Parameters:
+
+ - tsdrkey=tsdrkey=[NID=][DC=][RK=]
+
+ - The TSDRKey format indicates the NodeID(NID), DataCategory(DC), and RecordKey(RK) of the monitored objects.
+ For example, the following is a valid tsdrkey:
+ [NID=openflow:1][DC=NETFLOW][RK]
+ The query will return only the first 1000 records that match the query criteria.
+
+ - from=<time\_in\_seconds>
+
+ - until=<time\_in\_seconds>
+
+The following is an example curl command for querying log type of data
+from TSDR data store:
+
+ ::
+
+ curl -G -v -H "Accept: application/json" -H "Content-Type: application/json" "http://localhost:8181/tsdr/logs/query"
+ --data-urlencode "tsdrkey=[NID=][DC=NETFLOW][RK=]" --data-urlencode
+ "from=0" --data-urlencode "until=240000000000"\|more
+
+Grafana integration with TSDR
+-----------------------------
+
+TSDR provides northbound integration with Grafana time series data
+visualization tool. All the metric type of data stored in TSDR data
+store can be visualized using Grafana.
+
+For the detailed instruction about how to install and configure Grafana
+to work with TSDR, please refer to the following link:
+
+https://wiki.opendaylight.org/view/Grafana_Integration_with_TSDR_Step-by-Step
+
Configuring TSDR Data Collectors
--------------------------------
-- SNMP Data Collector Device Credential Configuration
+SNMP Data Collector Device Credential Configuration (experimental)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
After installing SNMP Data Collector, a configuration file under etc/
directory of ODL distribution is generated: etc/tsdr.snmp.cfg is
}
}
-Querying TSDR from REST APIs
-----------------------------
-
-TSDR provides two REST APIs for querying data stored in TSDR data
-stores.
-
-- Query of TSDR Metrics
-
- - URL: http://localhost:8181/tsdr/metrics/query
-
- - Verb: GET
-
- - Parameters:
-
- - tsdrkey=[NID=][DC=][MN=][RK=]
-
- ::
-
- The TSDRKey format indicates the NodeID(NID), DataCategory(DC), MetricName(MN), and RecordKey(RK) of the monitored objects.
- For example, the following is a valid tsdrkey:
- [NID=openflow:1][DC=FLOWSTATS][MN=PacketCount][RK=Node:openflow:1,Table:0,Flow:3]
- The following is also a valid tsdrkey:
- tsdrkey=[NID=][DC=FLOWSTATS][MN=][RK=]
- In the case when the sections in the tsdrkey is empty, the query will return all the records in the TSDR data store that matches the filled tsdrkey. In the above example, the query will return all the data in FLOWSTATS data category.
- The query will return only the first 1000 records that match the query criteria.
-
- - from=<time\_in\_seconds>
-
- - until=<time\_in\_seconds>
-
-The following is an example curl command for querying metric data from
-TSDR data store:
-
-curl -G -v -H "Accept: application/json" -H "Content-Type:
-application/json" "http://localhost:8181/tsdr/metrics/query"
---data-urlencode "tsdrkey=[NID=][DC=FLOWSTATS][MN=][RK=]"
---data-urlencode "from=0" --data-urlencode "until=240000000000"\|more
-
-- Query of TSDR Log type of data
-
- - URL:http://localhost:8181/tsdr/logs/query
-
- - Verb: GET
-
- - Parameters:
-
- - tsdrkey=tsdrkey=[NID=][DC=][RK=]
-
- ::
-
- The TSDRKey format indicates the NodeID(NID), DataCategory(DC), and RecordKey(RK) of the monitored objects.
- For example, the following is a valid tsdrkey:
- [NID=openflow:1][DC=NETFLOW][RK]
- The query will return only the first 1000 records that match the query criteria.
-
- - from=<time\_in\_seconds>
-
- - until=<time\_in\_seconds>
-
-The following is an example curl command for querying log type of data
-from TSDR data store:
-
-curl -G -v -H "Accept: application/json" -H "Content-Type:
-application/json" "http://localhost:8181/tsdr/logs/query"
---data-urlencode "tsdrkey=[NID=][DC=NETFLOW][RK=]" --data-urlencode
-"from=0" --data-urlencode "until=240000000000"\|more
-
-Grafana integration with TSDR
------------------------------
-
-TSDR provides northbound integration with Grafana time series data
-visualization tool. All the metric type of data stored in TSDR data
-store can be visualized using Grafana.
-
-For the detailed instruction about how to install and configure Grafana
-to work with TSDR, please refer to the following link:
-
-https://wiki.opendaylight.org/view/Grafana_Integration_with_TSDR_Step-by-Step
-
Purging Service configuration
-----------------------------
- If using mininet, run the following commands from mininet command
line:
- - mn --topo single,3 --controller
- *remote,ip=172.17.252.210,port=6653* --switch
- ovsk,protocols=OpenFlow13
+ ::
+
+ mn --topo single,3 --controller *remote,ip=172.17.252.210,port=6653* --switch
+ ovsk,protocols=OpenFlow13
- Install TSDR hbase feature from Karaf:
- - feature:install odl-tsdr-hbase
+ ::
+
+ feature:install odl-tsdr-hbase
- Install OpenFlow Statistics Collector from Karaf:
- - feature:install odl-tsdr-openflow-statistics-collector
+ ::
+
+ feature:install odl-tsdr-openflow-statistics-collector
- run the following command from Karaf console:
- - tsdr:list PORTSTATS
+ ::
+
+ tsdr:list PORTSTATS
You should be able to see the interface statistics of the switch(es)
from the HBase Data Store. If there are too many rows, you can use
statistics data collected from the switch(es).
-.. include:: tsdr-elastic-search.rst
-
-
Troubleshooting
---------------
(TLS) since the OpenFlow Plugin that TSDR depends on provides this
security support.
-- SNMP Security
-
- - The SNMP version3 has security support. However, since ODL SNMP
- Plugin that TSDR depends on does not support version 3, we (TSDR)
- will not have security support at this moment.
-
- NetFlow Security
- NetFlow, which cannot be configured with security so we recommend
- Syslog, which cannot be configured with security so we recommend
making sure it flows only over a secured management network.
+- SNMP Security
+
+ - The SNMP version3 has security support. However, since ODL SNMP
+ Plugin that TSDR depends on does not support version 3, we (TSDR)
+ will not have security support at this moment.
+
+- sFlow Security
+
+ - The sflow has security support.
+
+
Support multiple data stores simultaneously at runtime
------------------------------------------------------
example, the default content of tsdr-persistence-hsqldb.properties is as
follows:
-::
+ ::
- metric-persistency=true
- log-persistency=true
- binary-persistency=true
+ metric-persistency=true
+ log-persistency=true
+ binary-persistency=true
When the user would like to use different data stores to support
different types of data, he/she could enable or disable a particular
metric-psersistency=true
log-persistency=false
binary-persistency=false
+
Code Review / docs.git / commitdiff