| Author: | Jaap Murre |
|---|---|
| Version: | 1.0b |
| Date: | April-June, 2011 |
This is a JavaScript library for logging events. Logged events are buffered and will be sent to the server at intervals set by time and number of events. The logger will buffer events in local storage thus preserving them also if the server is down, the client’s computer is offline for a while, or if the page (i.e., app) is accidentally clicked away. As soon as the connection (or app) is restored, the data will be retrieved from local storage and sent to the server. Once the data has been saved at the server, they will be removed from local storage. By default, data in local storage are encrypted using advanced encryption: AES. The data are decrypted before they are sent to the server.
In order to work with the logger, you must set up server-side data processing that is capable of receiving XHR put events. Each of these save attempts (XHR put events) has attributes:
(string) Describes the type of the event, e.g., ‘page_navigation’.
(string) This could for example identify the user or application. Usually, this is not need on the server; it is mainly important for local storage at the client.
(any) The data attributed contains an array of event objects, each of which has attributes:
(string) The name of the event. E.g., ‘next_page’, ‘lesson_2’.
(integer) Timestamp in ms since epoch (JavaScript timestamp, created with new Date()).
(json string) This is anything sent to the logger, JSON encoded.
The logger works most reliably with dojo 1.6, which received an important upgrade of the local storage providers in dojox.storage.
We assume that the the nm directory, with the file logger.js underneath it, resides itself at the same level as the dojoroot:
dojoroot/
dojo/
dojox/
nm/
logger/_base.js
docs/
Register the nm module with:
dojo.registerModulePath("nm","../../nm"); // path is relative to dojo.js file
Now, the logger can easily be dojo.required(). After doing this, register one or more event types, e.g., ‘event’, ‘trial’, ‘keyboard’, ‘login_attempts’, etc. Any string can be used. Registration is as follows:
dojo.require("nm.logger");
nm.logger.register("event",{url:"process_data.php",password:"pietje"});
Warning
It is advisable to ask the password directly from the user and only keep it in memory, not hardcoded in the text as is shown here. HTML and JS files may be left in the cache so that passwords can easily be retrieved.
Now, you can log events with dojo.publish(), like this:
dojo.publish("event",[{
name: "Next",
data: currentPage
}]);
Notice the array brackets, [], which are mandatory with dojo.publish().
Standard events use the format:
{
name: "the_name_of_your_event",
data: "string, or integer or object or whatever"
}
If a different format is used, the logger will turn such a non-standard event into a standard event (with name and data attribute). A timestamp with attributed name created is always added (in ms since epoch).
So, it is also possible to write:
dojo.publish("trialno",[34]);
or:
dojo.publish("trial",[{no: 34, score: 98}]);
or:
dojo.publish("logged in",["jaap"]);
If you do not give an event name (i.e., in addition to an event type such as ‘trial’), the logger will assign the name ‘default’.
It is also possible to call the logger directly, with:
nm.logger.store("trial",{name: 'exp1', data: {block: 3, score: 23}});
(no [] brackets necessary here)
or store another type, e.g., a string or array:
nm.logger.store("scores",[34,29,35,43,51]);
Registration allows setting of various details using a details object. The details object must be provided, because saveToServer and encrypt are true by default and hence url and password must be specified. If you don’t want saving or encryption these must be turned off explicitly. The following attributes may be set:
Interval in ms between save-to-server attempts, set to 30,000 ms (i.e., 30 s) by default. Save attempts occur even if the the repeatSaveEvents number has not been reached.
Interval in number of events until next save, e.g. 20 means that the system will wait until 20 events have happened and will then save those to server, unless the repeatSaveMs is reached before that. Default is 10.
If the save fails, it will count from 0 to repeatSaveEvents before another save-to-server is attempted based on counted events.
Maximum number of times save-to-server will be tried (with repeatedSaveMs) in case of failure (e.g., due to server or connection problems). Default is 3.
true (default) or false . Attempt to post data to server. If true, you must provide a url
Url where to post the data.
true (default) or false. Encrypt local data using AES, If true, you must provide a password. For non-critical applications, encryption may be turned off for increased performance, e.g., on mobile platforms.
Note
A simplified version of the AES implementation is used here.
Password used for encryption of local storage
It is possible to set the current namespace with nm.logger.setNamespace("subj134")(). This is only relevant for local storage and it is often useful to mark the ID of the subject in the namespace. If not set, it is likely that multiple users of the same application running on the same computer (sharing an account) will get mixed up in local storage.
Note that all in-memory events are assumed to belong to the current user (only one namespace is active at a time).
The following methods are available (for event_type it is best to always use a string):
Send data to server, via local storage. See above for legal values of event.
| Arguments: |
|
|---|
Retrieves (and decrypts) data from local storage and in-memory data, returns these as a possibly empty array. Data from server is not retrieved. This function is useful if the logger is used without server-side storage (i.e., saveToServer is false).
| Arguments: |
|
|---|
As logger.getData() but only returns events with name atribute event_name.
| Arguments: |
|
|---|
Clear in-memory and local storage of a specific event type.
| Arguments: |
|
|---|
Clear in-memory and local storage of a specific event type. Only events with name atribute event_name are removed. Attempts to store remaining events are to local storage.
| Arguments: |
|
|---|
Clear all in-memory and local storage of all event types.
See section Register settings, for an explanation of details.
| Arguments: |
|
|---|
See explanation in section Namespace.
| Arguments: |
|
|---|
| Returns: | the currently set namespace (= "default", if none has been specified). |
|---|
If there are any existing events of type ‘event_type’ in local storage (i.e., the file system), it will try to save these to the server with any in-memory events of ‘event_type’. This function is called in function logger.register().
| Arguments: |
|
|---|
< ?php // Remove space before ? in actual implementation
function insert_data()
{
$method = $_SERVER['REQUEST_METHOD'];
if ($method == 'POST' || $method == 'PUT')
{
$input_data = json_decode(file_get_contents('php://input'));
}
else
{
$input_data = $_GET;
}
$event_type = $input_data->event_type;
$namespace = $input_data->namespace;
$contents = $input_data->data;
$con = mysql_connect("localhost","logger_user");
if (!$con)
{
die('Could not connect: ' . mysql_error());
}
mysql_select_db("logged_events", $con);
$count = 0;
foreach ($contents as $event)
{
$sql="INSERT INTO events (event_type, namespace, event_name, data, created)
VALUES
('$event_type','$namespace','$event->name','$event->data',$event->created)";
if (!mysql_query($sql,$con))
{
die('MySQL error: ' . mysql_error());
}
++$count;
}
echo "$count records inserted";
mysql_close($con);
}
insert_data();
--
-- Table structure for table `events`
--
CREATE TABLE IF NOT EXISTS `events` (
`index` int(11) NOT NULL AUTO_INCREMENT,
`event_type` text NOT NULL,
`namespace` text NOT NULL,
`event_name` text NOT NULL,
`data` longtext NOT NULL,
`created` bigint(20) NOT NULL,
PRIMARY KEY (`index`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=4 ;
Updated documentation.
Found a bug where events were stored twice in the database. Traced it to the results handler of dojox.storage.put(), which somehow did not work in the global context. Therefore clearing of in memory events did not succeed and they were thus stored twice (in memory and local storage). This happened using the WhatWGStorageProvider (though using FF3). I then noticed I was still working with dojo 1.5. After moving to 1.6, the problem disappeared and storage provider is now upgraded to one that does not have this problem: dojox.storage.LocalStorageProvider.
Using dojo 1.6 is thus required. Added this in the documentation.
In the process of tracing this bug, also added a new function clearInMemory(). For now, only intended for private use.
Updated the file so that the reStructured documentation can be used directly from the .js file, except for the /* comment at the beginning.
Did several things today:
- Changed fatal bug with _checkEventRegistration(), which was not a function yet
- Turned logger.js into nm/logger/_base.js construction
- Changed ‘doc’ directory name into ‘docs’
- Updated documentation to reflect this. Also, updated documentation in various other places.
- Found bug with unicode (e.g., Chinese) characters with _encrypt/_decrypt. Apparently the algorithms expect single-byte (i.e., ASCII) characters. Solved this by wrapping the data with simple encode_utf8/decode_utf8 functions. Works for Pinyin at least. For a more complete solution, see http://www.webtoolkit.info/javascript-utf8.html.
Added two public functions that allow getting and clearing by event name (i.e., as in the name atribute):
- getDataByName()
- clearByName()
- Also updated documentation to reflect this.
Added tests directory and wrote test_logger.html. Discovered a large bug that caused server access for each additional logged event once the repeatSaveEvents maximum had been reached, because the event counter was never reset. Changed the logic. Now, the counter is set to 0 before a save to server is attempted. If the save fails, it is not set back to its original value. This is deliberate to prevent save-to-server attempts with each new event in case of a connection failure.