Difference between revisions of "CCDB 1.00"

From GlueXWiki
Jump to: navigation, search
 
(One intermediate revision by the same user not shown)
Line 2: Line 2:
  
  
== Creating tables right from file and information about files ==
+
== Command line tools ==
It is usual situation when one already has text data file and want to create table to store the data.  
+
 
 +
=== Creating tables from text files ===
 +
CCDB now can help to create a table from text file with data. It infers number of columns, rows and column names (if file provides it)
 +
 
 +
It is usual situation when one already has text data file and want to create table to store the data.
 +
 
 
<code>mktbl -f</code> helps with this
 
<code>mktbl -f</code> helps with this
 
<pre>
 
<pre>
Line 17: Line 22:
 
Now one can copy it and run. It is not done automatically to be sure the values are reviewed.  
 
Now one can copy it and run. It is not done automatically to be sure the values are reviewed.  
  
The file is:
+
The file format is:
 
<pre>
 
<pre>
 
#Test assignment for software tests
 
#Test assignment for software tests
Line 25: Line 30:
 
</pre>
 
</pre>
 
You may notice that line with column names starts with '''#&''' while normal comments starts with plain '''#'''
 
You may notice that line with column names starts with '''#&''' while normal comments starts with plain '''#'''
 +
 +
(Please, read ccdb_file_format.pdf in doc directory of CCDB for more info about how CCDB reads text files)
  
 
One can use column types right there:
 
One can use column types right there:
Line 30: Line 37:
 
#& X=int            Y=int            NOT_Z=string
 
#& X=int            Y=int            NOT_Z=string
 
</pre>
 
</pre>
 +
 +
 +
 +
=== information about files ===
  
 
'''-f''' flag added to Info command also. <code> info -f </code> gives the info of text files, how CCDB sees it.  
 
'''-f''' flag added to Info command also. <code> info -f </code> gives the info of text files, how CCDB sees it.  
Line 51: Line 62:
  
  
== 'ls' command update ==
+
=== 'ls' command update ===
 
When ls is used directly on a table name, it gives data versions (as vers command):
 
When ls is used directly on a table name, it gives data versions (as vers command):
 
<pre>
 
<pre>
Line 84: Line 95:
 
     -l or --extended    - shows extended info when is used on table
 
     -l or --extended    - shows extended info when is used on table
 
</pre>
 
</pre>
 +
 +
  
 
== JAVA API ==
 
== JAVA API ==
Line 111: Line 124:
 
$CCDB_HOME/java/out/artifacts
 
$CCDB_HOME/java/out/artifacts
 
</pre>
 
</pre>
 +
 +
  
 
== C++ API update ==
 
== C++ API update ==
Line 152: Line 167:
 
a->GetValueInt(101, "x")
 
a->GetValueInt(101, "x")
 
</pre>
 
</pre>
 +
 +
  
 
=== Examples ===
 
=== Examples ===
Line 162: Line 179:
 
</pre>
 
</pre>
 
After the compilation examples can be run from console as ''example_ccdb_(example name)''
 
After the compilation examples can be run from console as ''example_ccdb_(example name)''
 +
 +
== Change log ==
 +
* Removed PyQt import. Why was it there?
 +
* CCDB databases update for 1.00 version
 +
* CCDB sqlite shcema update for next version
 +
* users_create script to create or reacreate users lists (for example during cron job)
 +
* Create and delete users
 +
* Fix - Perl script to get all halld database users, users_create.py doesn't stop on users that are already exists.
 +
* Fixed bug in my original script. Insert missing comma between group lines.
 +
* Fix - mkdir comment
 +
* Tests - Error regression test for comments in mkdir error
 +
* Doc - Update
 +
* Fix - Inactivity time
 +
* Doc - update
 +
* Update - SqlAlchemy update to 9.2
 +
* Fix - A  bug with doublication of directory in ls is fixed
 +
* Fix - parse string bag,
 +
* Update - empty utility example update
 +
* Doc - help updated (connection strings examples added)
 +
* Doc - update on requests
 +
* Update - info command
 +
* Fix - small bugs in provider fixed
 +
* Tests - unit tests update
 +
* Update - Schema version is set to 4
 +
* Fix - Is deleted flag for autogenerated users is set to 0
 +
* Doc - CCDB text update
 +
* Update - ccdb.sqlite schema is updated to version 4
 +
* Fix - dump command fixed
 +
* Doc - Documentation update
 +
* Doc - example how to stroe xml in ccdb
 +
* Tests - update assignment add
 +
* Update - AlchemyProvider improvements
 +
* Tests - AlchemyProvider new tests
 +
* Doc - Better structuring
 +
* Fix - Socns SQLite isn't recompiled each time
 +
* Fix - get single value test failed
 +
* Doc - Simple example created
 +
* Update - Basic java implementation
 +
* Doc - Pep8 improvements
 +
* Update - When adding data, types are checked now
 +
* Fix - Fixed bug with parsing of doubles and ints
 +
* Update - SQLite Sconscript in separate folder
 +
* Fix - Time parsing bugs changed
 +
* Fix - Time parsing bugs
 +
* Doc - requests example
 +
* Update - -m32 forced flag added to scons options
 +
* Update - Catch unit test framework updated to 1.01
 +
* Tests - Time parsing issues update
 +
* Update - New catch framework tests update
 +
* Tests - Tests for time parse update
 +
* Fix - sqlite dummy database corresponds to mysql
 +
* Fix - Time shift between mysql and sqlite
 +
* Update - -f flag for info command
 +
* Tests - mktbl -f unit tests update.
 +
* Update - ls remade
 +
* Update - Infer status by file
 +
* Update - Comments update
 +
* Tests - ls unit tests update
 +
* Update - Java version MySQL and SQLite works
 +
* Update - mysql jar added by default
 +
* Fix - Patch for situation when ':' is found in time string
 +
* Update - Better parse of mysql connection string
 +
* Update - SQLite JDBC drivers added to lib folder
 +
* Update - ccdb.jar added to the project
 +
* svn cleanup
 +
* Update - C++ API Calibration returns Assignment now. Assignment have GetValue and GetValueType functions
 +
* Tests - New C++ API unit tests
 +
* Fix - bug with variation select fixed,
 +
* Fix - Bug with deleting assignment before using result fixed
 +
* Fix - typos
 +
* Svn - cleanup
 +
* Update - Default values for CreateCalibration added
 +
* Doc - simple example update
 +
* Update - GetRowsCount() and GetColumnsCount() functions added to Assignment object
 +
* Fix - Type table GetVelue is changed to GetValue
 +
* Doc - Assignments example added. It describes how to get extended information about particular ccdb data
 +
* Doc - Scons description of how to compile CCDB examples added
 +
* Doc - Java example update
 +
* Update - Java getColumnValuesString function introduced to get all values of one column
 +
* Update - Java. Class.forName("com.mysql.jdbc.Driver") added so mysql driver loads automatically
 +
* Update - Function isTableAvailable added
 +
* Update - Function overloads for getting data from assignment as double, int, long, boolean added
 +
* Doc - Comments update

Latest revision as of 15:58, 9 May 2014

CCDB 1.00 release notes


Command line tools

Creating tables from text files

CCDB now can help to create a table from text file with data. It infers number of columns, rows and column names (if file provides it)

It is usual situation when one already has text data file and want to create table to store the data.

mktbl -f helps with this


> mktbl /test/haha -f python/tests/test_table.txt #my lonely comment
The command to create table: 

mktbl /test/haha -r 2 X Y Z #my lonely comment

Comments in file: 
Test assignment for software tests

Now one can copy it and run. It is not done automatically to be sure the values are reviewed.

The file format is:

#Test assignment for software tests
#& X            Y            Z
  2.2          2.3          2.4         
  2.5          2.6          2.7     

You may notice that line with column names starts with #& while normal comments starts with plain #

(Please, read ccdb_file_format.pdf in doc directory of CCDB for more info about how CCDB reads text files)

One can use column types right there:

#& X=int            Y=int            NOT_Z=string


information about files

-f flag added to Info command also. info -f gives the info of text files, how CCDB sees it.

 ~/halld/ccdb/trunk $ ccdb info -f $CCDB_HOME/python/tests/test_table.txt
Rows: 2
Columns: 3
Column names:
    X
    Y
    Z

Comments in file: 
Test assignment for software unit tests

Type 'ccdb mktbl -f /home/romanov/halld/ccdb/trunk/python/tests/test_table.txt' to see how to create a table for the file
Type 'ccdb add <table name> /home/romanov/halld/ccdb/trunk/python/tests/test_table.txt #<comments>' to add the file to existing table (rows and columns must consist)


'ls' command update

When ls is used directly on a table name, it gives data versions (as vers command):

 ccdb ls /test/test_vars/test_table
/test/test_vars/test_table
(ID)   (Created)              (Modified)              (variation)     (run range)      (comments)
 5      2012-10-30 23-48-43    2012-10-30 23-48-43     subtest         0-inf           Test assignment for 
 4      2012-10-30 23-48-42    2012-10-30 23-48-42     default         0-inf           Test assignment for 
 1      2012-07-30 23-48-42    2012-07-30 23-48-42     default         0-inf           Test assignment for 


-l flag adds info about table like (info) command.

ccdb ls -l /test/test_vars/test_table

The flags of ls command changed too:

Lists directories and tables for current directory

- Accepts wildcards symbols '*', and '?'

- When used on single table name, gives table data version as 'vers <table name>' command

keys:
    -v or --variations   - prints all variations
    -t or --tables       - prints full path for all tables
    -d or --directories  - prints all directories
    -x or --dtree        - draws directory tree

    -l or --extended     - shows extended info when is used on table


JAVA API

Users can use JAVA to access CCDB data. Or even Kotlin JVM language.

That is how it looks:

        JDBCProvider provider = CcdbPackage.createProvider("mysql://localhost")  ;
        provider.connect();
        Assignment asgmt = provider.getData("/test/test_vars/test_table");
        
        // gets data represented as number of columns
        for(Vector<String> row : asgmt.getStringTable()){
            for(String cell: row){
				System.out.print(cell + " ");
			}
			System.out.println(); //next line after a row
        }
        
        //gets some extended info about data
        System.out.println(asgmt.getTypeTable().getFullPath());


CCDB is provided as jar file including all required database drivers. The file could be found in:

$CCDB_HOME/java/out/artifacts


C++ API update

Cell by cell readout and extended data inforation

In 0.9 users can read data like:

vector<vector<int> > tabledValues;
calib->GetCalib(tabledValues, "/test/test_vars/test_table2"));

So there is no extended information about the data, like column types or date. In CCDB 1.00 new GetAssignment function is introduced, which returns Assignment object. Assignment holds information about data and its type table. For example that is the way of getting columns information:

auto_ptr<Assignment> a(calib->GetAssignment("/test/test_vars/test_table"));    // auto_ptr to delete assignment after
                                                                               // auto_ptr is deprecated in C++11 btw

//Get type table and it holds information about columns 
const vector<ConstantsTypeColumn *> &columns = a->GetTypeTable()->GetColumns();

cout<<"Columns:"<<endl;
for(size_t i=0; i< columns.size(); i++)
{
    cout<<"   name: '"<< columns[i]->GetName() 
        <<"'  type: '"<< columns[i]->GetTypeString() <<"'"
        <<endl;
}

But Assignment has even more handy feature. CCDB can have columns of different types in one table. For example: 2 columns of ints, 1 column of doubles and 1 column of strings (with some descriptions). There was no direct way to read it that way in 0.9. But with assignment you can do:

a->GetValueInt(0, 0)
a->GetValueInt(0, 1)
a->GetValueDouble(0, 2)
a->GetValue(0, 3)       //string

Or even better! If column names are "x", "y", "gain", "description" one can use

a->GetValueInt("x")
a->GetValueInt("y")
a->GetValueDouble("gain")
a->GetValue("description")       //string
//assuming row is 0. if it is not 0, one can use:
a->GetValueInt(101, "x")


Examples

CCDB now ships examples folder:

 $CCDB_HOME/examples 

Where you can find some simple examples of how to use CCDB C++ user api. Add 'with-examples=true' flag to scons to compile the examples.

   scons with-examples=true

After the compilation examples can be run from console as example_ccdb_(example name)

Change log

  • Removed PyQt import. Why was it there?
  • CCDB databases update for 1.00 version
  • CCDB sqlite shcema update for next version
  • users_create script to create or reacreate users lists (for example during cron job)
  • Create and delete users
  • Fix - Perl script to get all halld database users, users_create.py doesn't stop on users that are already exists.
  • Fixed bug in my original script. Insert missing comma between group lines.
  • Fix - mkdir comment
  • Tests - Error regression test for comments in mkdir error
  • Doc - Update
  • Fix - Inactivity time
  • Doc - update
  • Update - SqlAlchemy update to 9.2
  • Fix - A bug with doublication of directory in ls is fixed
  • Fix - parse string bag,
  • Update - empty utility example update
  • Doc - help updated (connection strings examples added)
  • Doc - update on requests
  • Update - info command
  • Fix - small bugs in provider fixed
  • Tests - unit tests update
  • Update - Schema version is set to 4
  • Fix - Is deleted flag for autogenerated users is set to 0
  • Doc - CCDB text update
  • Update - ccdb.sqlite schema is updated to version 4
  • Fix - dump command fixed
  • Doc - Documentation update
  • Doc - example how to stroe xml in ccdb
  • Tests - update assignment add
  • Update - AlchemyProvider improvements
  • Tests - AlchemyProvider new tests
  • Doc - Better structuring
  • Fix - Socns SQLite isn't recompiled each time
  • Fix - get single value test failed
  • Doc - Simple example created
  • Update - Basic java implementation
  • Doc - Pep8 improvements
  • Update - When adding data, types are checked now
  • Fix - Fixed bug with parsing of doubles and ints
  • Update - SQLite Sconscript in separate folder
  • Fix - Time parsing bugs changed
  • Fix - Time parsing bugs
  • Doc - requests example
  • Update - -m32 forced flag added to scons options
  • Update - Catch unit test framework updated to 1.01
  • Tests - Time parsing issues update
  • Update - New catch framework tests update
  • Tests - Tests for time parse update
  • Fix - sqlite dummy database corresponds to mysql
  • Fix - Time shift between mysql and sqlite
  • Update - -f flag for info command
  • Tests - mktbl -f unit tests update.
  • Update - ls remade
  • Update - Infer status by file
  • Update - Comments update
  • Tests - ls unit tests update
  • Update - Java version MySQL and SQLite works
  • Update - mysql jar added by default
  • Fix - Patch for situation when ':' is found in time string
  • Update - Better parse of mysql connection string
  • Update - SQLite JDBC drivers added to lib folder
  • Update - ccdb.jar added to the project
  • svn cleanup
  • Update - C++ API Calibration returns Assignment now. Assignment have GetValue and GetValueType functions
  • Tests - New C++ API unit tests
  • Fix - bug with variation select fixed,
  • Fix - Bug with deleting assignment before using result fixed
  • Fix - typos
  • Svn - cleanup
  • Update - Default values for CreateCalibration added
  • Doc - simple example update
  • Update - GetRowsCount() and GetColumnsCount() functions added to Assignment object
  • Fix - Type table GetVelue is changed to GetValue
  • Doc - Assignments example added. It describes how to get extended information about particular ccdb data
  • Doc - Scons description of how to compile CCDB examples added
  • Doc - Java example update
  • Update - Java getColumnValuesString function introduced to get all values of one column
  • Update - Java. Class.forName("com.mysql.jdbc.Driver") added so mysql driver loads automatically
  • Update - Function isTableAvailable added
  • Update - Function overloads for getting data from assignment as double, int, long, boolean added
  • Doc - Comments update