Tik-76.115 Individual Project: Guinea Pig
$Id: file_parser.html,v 1.2 1996/04/22 18:22:00 kepa Exp $

Test Creation - File parser module

This document is the technical documentation for the file parser module. This module implements parsing different configuration and result files used in the Guinea Pig project. It is used in the Test Creation module, Test Logic and Result processing. It loads the necessary file and extracts the tags and tag data values from it. To be able to parse the file correctly, file parser needs to know the tags used in the file. The result of parsing is a Python dictionary with tag names as keys and a list of tag data values as the data.

Python interface

To use the Python file parser module file_parser.py, you have to first import it to your Python program.
import file_parser
After this you must first initialize the file parser with the description of the tags you use in your configuration file. The different types of tags will be discussed below. Here is how you do the initialization for the test.config file:
p = file_parser.file_parser(tags={'TestConfigVersion': 'f',
                                  'TestType': 's',
                                  'TestRootDir': 's',
                                  'SamplesDir': 's',
                                  'GuineaPigDir': 's',
                                  'TestItemsFile': 's',
                                  'PlayListFile': 's',
                                  'SampleRate': 'i',
                                  'OutputChannels': 'i',
                                  'OutputLevel': 'f',
                                  'UseMCLL': 'b',
                                  'MCLLParams': 'iiii',
                                  'MCLLShowvalue': 'b',
                                  'SoundPlayerOption': '[s]',
                                  'SampleList': ('D', {'File': '[ss]',
			          'Null': '[si]'}),
	                          'ABTestGlobal': ('D', {'WarnTime': 'f',
                                                         'AnswerTime': 'f',
                                                         'TestSequence': ('D', {'Item': '[ss]'}),
                                                         'AB_SCALETest': ('D', {'ScaleParams': 'iiii'})
				                        })
                                 })

If you compare this tag definition with the format description of the test.config file, you get the basic idea how the parsing works. We will discuss the meaning of the various tags below.

Now you are ready to parse the test.config file. You should check that the file exists before calling the parse method.

filename = 'test.config'
result = p.parse(filename)

# Do what you want with the result
print result
The result will be None if the parsing failed. The file parser will print the reason for failure and the line it occured on to stderr. If the parsing was succesful, the file parser will return a python directory containing the parsed tags. Below is the output of the print command for one test.items file (indentation added to make the output clearer).
{'MCLLShowvalue': [0],
 'TestType': ['ab'],
 'PlayListFile': ['playlist'],
 'TestConfigVersion': [1.0],
 'TestRootDir': ['.'],
 'MCLLParams': [-60, 20, -20, 10],
 'TestItemsFile': ['test.items'],
 'ABTestGlobal': [{'AnswerTime': [10.0],
                   'WarnTime': [5.0],
                   'TestSequence': [{}],
                   'AB_SCALETest': [{'ScaleParams': [1, 5, 1, 0]}]}],
 'SamplesDir': ['samples'],
 'SampleRate': [11025],
 'SampleList': [{'Null': [['pa1', 11025],
                          ['pa5', 77175],
                          ['pa7', 77175],
                          ['pa8', 33075],
                          ['pa9', 187425]],
                 'File': [['smpl1', 'file1.aiff'],
                          ['smpl2', 'file2.aiff'],
                          ['smpl3', 'file3.aiff'],
                          ['smpl4', 'file4.aiff'],
                          ['smpl5', 'file5.aiff']]}],
 'GuineaPigDir': ['guineapigs'],
 'UseMCLL': [1],
 'OutputChannels': [2]}
The dictionary contains the logical structure of the tags in the file, but not the order in which tags appear in the file. Now you can access the various tags in the dictionary. If the result of the tag is a list with several items, the items will be in the order they were found in the parsed file.

If some tag is not found, it won't exist in the result dictionary and accessing it will be an error. If there is some tag in the file which is not listed in the tag description, it is skipped and not added to the result dictionary. Here is an example how to use the result dictionary:

testtype = result['TestType'][0]
files = result['SampleList'][0]['File']

Tags description dictionary

The tags dictionary used to initialize the file parser contains the description of the file to parsed. First we need some basics about how tags are used.

A simple tag is one row of text ending in a newline '\n' character. The name of the tag ends with a douple column and the tag arguments are separated from each other by whitespace.

Tagname: arg1 arg2 arg3

A dictionary tag is a tag which implements its own new tag namespace. They are used to create hierarchical file format definitions. Dictionary tags begin with the tagname followed by a douple column on a line by itself, and they end with a line containing a slash followed by the tagname.

Dictionary tags can contain any tags that the main tag definitions can contain. Here is a simple dictionary tag:

Tagname:

Tag2: arg1 arg2

/Tagname

The tags description dictionary contains items with the tagname as the key and the tag description as the data of the item. Let's take the test.items file tag description as an example:

tags = {'TestItemsVersion': 'f',
	'TestItem': ('[D]', {'Id': 's',
                             'Sample': '[ss]'})}

On the top level we have one simple tag, the 'TestItemsVersion' tag. It has one argument which is a float (marked by 'f'). The tag could look like this in the parsed file:
TestItemsVersion: 1.0

There is also another tag on the top level. It is a dictionary tag called 'TestItem' (marked by 'D'). You should also notice that because the dictionary tag contains another tag description inside it, we must use a tuple to describe it. The first item of the tuple is the tagtype '[D]' and the second is another dictionary containing another tag description.

Yet one thing we haven't discussed si why there are the '[]' characters surrounding 'D'. This notes the fact that it is not an error if there are several 'TestItem' tags present in the file to be parsed. This can be done with a simple tag also.

Here is what the 'TestItem' tag looks like in the file:

TestItem:
Id: id1
Sample: A smpl1
Sample: B smpl2
/TestItem

TestItem:
Id: id2
Sample: A smpl1
Sample: B smpl3
/TestItem

Now as can be seen from above the 'TestItem' tag contains two simple tags: 'Id' and 'Sample'. There can be several 'Sample' tags inside one 'TestItem' tag, but only one 'Id' tag. The 'Sample' tag contains two string arguments and the 'Id' one correspondingly.

Now, here are the argument types:

'D'        Dictionary tag. Used as the first item of the tuple ('D', tags),
           where tags is a tag description dictionary
'b'        Boolean, 1 ('yes') or 0 ('no')
'i'        Integer
'f'        Float
's'        String. Strings can contain spaces if in quotes "like this"
'[xxx]'    Tag can appear multiple times. (xxx is any argumentype string)
'xyz+'     The tag contains x and y type arguments followed by
           one or more z type arguments. (x, y and z are argment types)

If we return to the example at hand, here is what the file parser module returns from the example file above:

{'TestItem': [[{'Sample': [['A', 'smpl1'],
                           ['B', 'smpl2']],
                'Id': ['id1']}],
              [{'Sample': [['A', 'smpl1'],
                           ['B', 'smpl3']],
                'Id': ['id2']}]],
 'TestItemsVersion': [1.0]}
There are many examples in Guinea Pig project sources that use the file parser module. They should give you further insight on the usage of the file parser module.


· Test Creation Index · Document index · Guinea Pig ·