The Labs.Com Web Lab MyDBase

The Labs - Design & Functionality For The Net

MyDBase, The Alternative Kind of Database

1. Introduction
2. Concept
3. Download
4. Server
5. Commands
6. Clients
7. Getting Started
8. Design Hints
9. Intern Data-Structure
10. Performance Considerations
11. Icons
12. Further Resources

Another database? Yes, another one. :-) I started to write is as I didn't like the immense overhead to use SQL based systems. For the few web-tasks I required a small and stable database which source-code is short enough to debug easily by myself within a reasonable time. Another requirement was that the actual data stored should be ASCII-based and easy to change with a simple text-editor.

The main different to common databases: the tables usually don't grow horizontally (new fields) but vertically with new entries, MyDBase is the oposite: it's optimized to grow horizontally mainly, vertically you have to look carefully how to do since there are several approaches possible, ie. hierachical tables will help you to grow vertically, more below. MyDBase doesn't provide any kind of "query language" to combine a query or so, it's a simple database, that's it. We may later provide a query feature, but for now it's just store & fetch.

MyDBase 2. Concept

In general, a database handles multiple so called "tables", they are named by an ordinary string (A-Za-z_-). Within the table there are fields and their field-values can be stored, retrieved or deleted. Unlike in SQL, the MyDBase is optimized to handle growing table-fields. Each field-name within a table must be unique:

MyDBase Concept:

fullname Webmaster email webmaster@here.com age 25 : :

SQL Concept:

fullname email age Webmaster webmaster@here.com 25 Root root@here.com 32 Me me@here.com 48 : : :

Since MyDBase's tables are compared to SQL-based databases just one entry within a table. You have to rethink your design when you come from there, one advantage of MyDBase are hierachical tables like this:

Sample Design:
Members is an empty table, but has multiple nodes (for each user), additional each user (another table) has sub-tables contain the list of Interest.

Lists is node table, with multiple sub-nodes for each list, ie. perl-l, cgi-l etc, each list has another sub-table called dist which stands for distribution. Each dist contains the list of the users, use the user-IDs as unique and assigned how they are subscribed.

For sake of speed up you may even add the email-address there as well, ie. root=subscribed,root@me.com; that way the list software requires only to call the database once to check if a user is permitted to submit. In that case whenever a Member table is updated (email-address) the related dist of the subscribed lists requires also an update. Usually the update is quite rare measured to queries a list-software has to do.

MyDBase 3. Download

$MyVersion: 0.053 - Wed Oct 6 04:55:40 MEST 1999 - kiwi$

mydb.tar.gz (perl-sources)

mydbd is the server, it runs in the background as daemon, and the mydb-client (standalone prg) or mydb.pl and TieMyDB.pm (Perl API) are the clients.

$MyHistory$
	14 Aug 1999: 0.051: MyDatabase renamed to MyDBase due TM infringement .
	14 Aug 1999: 0.051: query("path.*") and query("path:*") implemented .
	19 Jun 1999: 0.050: bug fix, when field was 0 ( get ) .
	05 Feb 1999: 0.046: delete field in table fixed .
	29 Jan 1999: 0.044: delete field implemented finally (was missed) .
	17 Jan 1999: 0.043: rudimentary web-interface mydb.cgi .
	09 Jan 1999: 0.042: bug fix in-queue .
	07 Jan 1999: 0.041: dump included .
	05 Oct 1998: 0.040: true user identification .
	09 Sep 1998: 0.037: output queue fixed, no blocking anymore .
	07 Sep 1998: 0.035: misc bug fixed in mydbd, TieMyDB.pm .
	06 Sep 1998: 0.033: command create dropped, put can do it .
	05 Sep 1998: 0.032: put can force creation of table (for tie perl5) .
	04 Sep 1998: 0.031: bug-fix: inqueue checked too .
	04 Sep 1998: 0.030: allow/deny of ip support .
	02 Sep 1998: 0.028: hierachical tables possible, get recursive .
	02 Sep 1998: 0.027: client can sent statistic infos .
	29 Aug 1998: 0.025: queued writing .
	27 Aug 1998: 0.024: delete recursively implemented .
	26 Aug 1998: 0.021: non-blocking implemented .
	25 Aug 1998: 0.017: authentication included .

MyDBase 4. Server

mydbd you can start right away, default port is 4700.

% mydbd &

mydbd: --- MyDBase, written by Rene K. Mueller <kiwi at the-labs dot com>

mydbd:     $MyVersion: 0.016 - Tue Aug 25 03:28:22 EDT 1998 - kiwi$

mydbd: For further info look at http://the-labs.com/MyDBase/

mydbd: Accepting connections (port 4700)

You can run multiple mydbd just call them with unique port-numbers:

% mydbd -p 4701 &

Additionally, you can use the option -name name... to title the session:

% mydbd -name "Customer Support" &

You can view the status of each mydbd session via with your browser: ./status.html (updated every 10 secs), the html-file is created where mydbd was started.

Note: If you start multiple mydbd with different port-numbers, create several directories and start them in there to avoid they all use the name log-file and status.html.

MyDBase 5. Commands

The server commands:

Command Feedback Description authen <user> <passwd> <status> <comment> Authenticate yourself put <table> <field> <data> <status> <comment> Put a field & value into a table push <field> <data> <status> <comment> Push data on stack dump <table> <status> <comment> Dump data of stack into table get <table> <status> <comment> <n-data> <field1> <data1> .. ... <fieldN> <dataN> Get table content; if the table has subnodes then you retrieve all sub-tables as well. If not table is given, the entire database is retrieves which can be huge get <table> <field> <status> <comment> 1 <field> <data> Get a table field delete <table> <status> <comment> Delete a table and all sub-tables delete <table> <field> <status> <comment> Delete a field within a table mode <field> <data> none put = force; when you use put, the table will be created if required statistic <name> <data> none Send back statistic infos: <name> = "delay: your program-name" <data> = delay-of-seconds (delay from 'get'-commend til all data arrived)

The separator is <tab>, each line is ended with \n.

Status Comment 100 Successful 200 General Error Occured 201 Could not find table 202 Could not delete table 300 Bad Authentication 301 Authentication Required 500 Syntax Error

Comment usually contains a more detail description of the action (success and failure).

MyDBase 6. Clients

Standalone

mydb-client is the client, you can call direct all syntax supported (assuming no ./passwd exist and therefore no authentication is required):  % ./mydb-client put example name "Ann Smith"   % ./mydb-client get example   If you have passwd created, use  % ./mydb-client -u user passwd get  lists all content of the database (can be huge).

Perl API

mydb.pl and TieMyDB.pm gives perl-interface to use within a CGI or other perl-scripts. Perl4 Perl5 $handle = My'Open(host,port [,user,passwd]); $t = tie %handle,host,port [,user,passwd]; MyDB'Put($handle,table,field,data); $handle{"table:field"} = data; MyDB'Push($handle,field,data); MyDB'Dump($handle,table); $t->dump("table",%data); ($status,@data) = MyDB'Get($handle,table); @_ = $t->query("table"); %h = $t->query("table"); ($status,$field,$data) = MyDB'Get($handle,table,field); data = $handle{"table:field"}; MyDB'Delete($handle,table); delete $handle{"table"}; MyDB'Delete($handle,table,field); delete $handle{"table:field"}; MyDB'Close($handle); untie %handle; Use it like this:  require "mydb.pl"      $handle = MyDB'Connect('localhost',4700,'default','none');   open(D,"data.txt");   $n = 0;   while(<D>) {      local($name,$street,$city) = split(/\t/);      MyDB'Put($handle,"Members.$n","name",$name);      MyDB'Put($handle,"Members.$n","street",$street);      MyDB'Put($handle,"Members.$n","city",$city);      $n++;   }   close(D);   MyDB'Close($handle);  Note: If you use MyDB'Delete($handle,table,field) and field is empty so the entire table will be deleted, make sure field isn't empty unless you really like to delete the table. TieMyDB.pm gives perl5-interface using tie command:  use TieMyDB;      $t = tie %db,'TieMyDB','localhost',4700,'default','none';      die "couldn't connect: $!\n" if(!$t);      open(D,"data.txt");   $n = 0;   while(<D>) {      local($name,$street,$city) = split(/\t/);      $db{"Members.$n:name"} = $name;      $db{"Members.$n:street"} = $street;      $db{"Members.$n:city"} = $city;      $n++;   }   close(D);      untie %db;  Note: the hash-key between { } must contain at least one '.', otherwise no assigned can be done (table & field required always). query(table) copies a portion of the db into a hash, like:  %users = $t->query("Members");   foreach $u (sort keys %users) {      print "$u = '$users{$_}'\n";      # --- Note: $users{$_} eq $db{"Members.$_"}   }  Note: If you call delete $db{"$table:$field"} and $field is empty, then no action is done, in order to remove the entire table call delete $db{"$table"} without ':'.

Shell

A very ALPHA version of a simple shell for MyDBase you get with mydbsh:  % ./mydbsh    --- MyDB-Shell written by Rene K. Mueller <kiwi at the-labs dot com>       ALPHA $MyVersion: 0.034 - Sun Sep  6 02:20:41 EDT 1998 - kiwi$       See also at http://the-labs.com/MyDBase/                  Connecting localhost (4700) with mydbsh ...      mydbsh@> ls              Following commands are available: ls list table of current level cd <table> change table, or .. put <field> <data> put data into table As command line mydbsh takes: [-u <login> <password>] [-p <port>] <host> In order to run mydbsh you require to add a new user in ./passwd:  mydbsh:none  replace 'none' by another more thoughtful password. Call then  % mydbsh -u mydhsh passwd

Read more at MyDB Cookbook how to use this all.

MyDBase 7. Getting Started

Here the few steps to get all running right away within few minutes:

1. Run make install

2. Edit ./passwd file:

   default:none

   Of course you use another user-id and password when you do this, if the passwd file is omitted, then no user-authentication of the client is required.

3. Start mydbd & (maybe with -v option also)

4. Study and run the mess-it (raw access), mess-it2 (perl4), and mess-it3 (perl5, most elegant approach)

5. Use mydbsh and play a bit.

6. Read more at MyDB Cookbook.

7. Write your perl-application using mydb.pl or TieMyDB.

Additionally you should create ./allowip:

127.0.0.1

to allow only localhost to connect. You can also write ./denyip file of those IP-no you don't want to have connected.

For both allowip and denyip list each IP on one line, * allowed.

127.0.0.1

192.123.15.*

128.12.*.*

192.*.*.*

The files passwd, allowip, and denyip are reloaded every 30 secs, so you can change them on the fly while mydbd is running.

MyDBase 8. Design Hints

The table-name can contain '.', like Members.root, or Members.webmaster, with the get query you can access it via

% ./mydb-client put Members.root fullname "Superuser"

% ./mydb-client put Members.webmaster fullname "Webmaster"

% ./mydb-client get Members

mydb: received "Members.root:fullname  Superuser"

mydb: received "Members.webmaster:fullname  Webmaster"

% ./mydb-client get Members.root

mydb: received "fullname  Superuser"

and you get all Members.* fields, this is the easy way to make linear-lists.

Additional you can hierachical put more deeper tables:

% ./mydb-client put Members.webmaster.Interest java low

% ./mydb-client put Members.webmaster.Interest perl high

% ./mydb-client put Members.webmaster.Interest c average

Note: If the table doesn't exist when you use put, the table is created automatically.

Create tables upto 100-200 field-entries, otherwise try to split them up using the dot so a table remains that small, that way the database remains fast.

MyDBase 9. Intern Data-Structure

Internally all tables are files, line-wise oriented field and data separated by <tab>. Tables with hierachical structure are also directories on the UNIX system.

Note: Be careful when you manipulate the files or directories without mydbd.

Looking at the file-system from the example from above:

./data/

<user>/

Members.d/

webmaster

webmaster.d/

Interest

root

root.d/

Interest

So it should be very simple to backup or restore the database in its entirety or parts of it.

MyDBase 10. Performance Considerations

The connection-time for mydbd is relatively long since it requires to establish a tcp-connection. Once the connection is established the transactions are done quite fast. For that reason, open as few sessions as possible, connect to db, make all requests and receive the data, and when all is done, close the session.

A hint is the connection vs requests ratio (seen in the ./status.html of the running mydbd), the smaller it is the better your design is, also the better the performance usage can be.

MyDBase 11. Icons

MyDBase 12. Further Resources

MyDB Cookbook Further description how to use the packages

                                                                                                                                   

WebDesign

Web Lab

WebSherpa