=========== logger.js =========== :Author: Jaap Murre :Version: 1.0b :Date: April, 2011 *********** User Manual *********** Introduction ------------ This is a JavaScript library for logging events. Logged events are buffered and will be sent to the server at intervals set by time and number of events. The logger will buffer events in local storage thus preserving them also if the server is down, the client's computer is offline for a while, or if the page (i.e., app) is accidentally clicked away. As soon as the connection (or app) is restored, the data will be retrieved from local storage and sent to the server. Once the data has been saved at the server, they will be removed from local storage. By default, data in local storage are encrypted using advanced encryption: AES_. The data are decrypted before they are sent to the server. In order to work with the logger, you must set up server-side data processing that is capable of receiving XHR put events. Each of these save attempts (XHR put events) has attributes: .. attribute:: event_type (string) Describes the type of the event, e.g., 'page_navigation'. .. attribute:: name_space (string) This could for example identify the user or application. Usually, this is not need on the server; it is mainly important for local storage at the client. .. attribute:: data (any) The :attr:`data` attributed contains an array of event objects, each of which has attributes: .. attribute:: name (string) The name of the event. E.g., 'next_page', 'lesson_2'. .. attribute:: created (integer) Timestamp in ms since epoch (JavaScript timestamp, created with ``new Date()``). .. attribute:: data (json string) This is anything sent to the logger, JSON encoded. Usage ----- We assume that the the ``nm`` directory, with the file ``logger.js`` underneath it, resides itself underneath the the dojoroot:: dojoroot/ dojo/ dojox/ nm/ logger/_base.js docs/ Register the ``nm`` module with:: dojo.registerModulePath("nm","../../nm"); // path is relative to dojo.js file Now, the logger can easily be ``dojo.required()``. After doing this, register one or more event types, e.g., 'event', 'trial', 'keyboard', 'login_attempts', etc. Any string can be used. Registration is as follows:: dojo.require("nm.logger"); nm.logger.register("event",{url:"process_data.php",password:"pietje"}); .. WARNING:: It is advisable to ask the password directly from the user and only keep it in memory, not hardcoded in the text as is shown here. HTML and JS files may be left in the cache so that passwords can easily be retrieved. Now, you can log events with ``dojo.publish()``, like this:: dojo.publish("event",[{ name: "Next", data: currentPage }]); Notice the array brackets, ``[]``, which are mandatory with ``dojo.publish()``. .. _event_structure: Event structure +++++++++++++++ Standard events use the format:: { name: "the_name_of_your_event", data: "string, or integer or object or whatever" } If a different format is used, the logger will turn such a non-standard event into a standard event (with :attr:`name` and :attr:`data` attribute). A timestamp with attributed name :attr:`created` is always added (in ms since epoch). So, it is also possible to write:: dojo.publish("trialno",[34]); or:: dojo.publish("trial",[{no: 34, score: 98}]); or:: dojo.publish("logged in",["jaap"]); If you do not give an event name (i.e., in addition to an event *type* such as 'trial'), the logger will assign the name 'default'. It is also possible to call the logger directly, with:: nm.logger.store("trial",{name: 'exp1', data: {block: 3, score: 23}}); (no ``[]`` brackets necessary here) or store another type, e.g., a string or array:: nm.logger.store("scores",[34,29,35,43,51]); Register settings ----------------- Registration allows setting of various details using a ``details`` object. The ``details`` object must be provided, because :attr:`saveToServer` and :attr:`encrypt` are true by default and hence :attr:`url` and :attr:`password` must be specified. If you don't want saving or encryption these must be turned off explicitly. The following attributes may be set: .. attribute:: repeatSaveMs Interval in ms between save-to-server attempts, set to 30,000 ms (i.e., 30 s) by default. Save attempts occur even if the the :attr:`repeatSaveEvents` number has not been reached. .. attribute:: repeatSaveEvents Interval in number of events until next save, e.g. 20 means that the system will wait until 20 events have happened and will then save those to server, unless the :attr:`repeatSaveMs` is reached before that. Default is 10. If the save fails, it will count from 0 to :attr:`repeatSaveEvents` before another save-to-server is attempted based on counted events. .. attribute:: maxSaveAttempts Maximum number of times save-to-server will be tried (with :attr:`repeatedSaveMs`) in case of failure (e.g., due to server or connection problems). Default is 3. .. attribute:: saveToServer true (default) or false . Attempt to post data to server. If true, you *must* provide a :attr:`url` .. attribute:: url Url where to post the data. .. attribute:: encrypt true (default) or false. Encrypt local data using AES_, If true, you must provide a password. For non-critical applications, encryption may be turned off for increased performance, e.g., on mobile platforms. .. NOTE:: A `simplified version`_ of the AES implementation is used here. .. _simplified version: http://dojotoolkit.org/reference-guide/dojox/encoding/crypto/SimpleAES.html .. _AES: http://en.wikipedia.org/wiki/Advanced_Encryption_Standard .. attribute:: password Password used for encryption of local storage Namespace --------- It is possible to set the current namespace with :func:`nm.logger.setNamespace("subj134")``. This is only relevant for local storage and it is often useful to mark the ID of the sujbect in the namespace. If not, it is likely that multiple users of the same application will get mixed up in local storage. Note, that all in-memory events are assumed to belong to the current user (one namespace active at a time). Methods ------- The following methods are available (for ``event_type`` it is best to always use a string): .. function:: logger.store(event_type,event) Send data to server, via local storage. See above for legal values of ``event``. :param event_type (string): event type :param event_type (string): see :ref:`event_structure` .. function:: logger.getData(event_type) Retrieves (and decrypts) data from local storage and in-memory data, returns these as a possibly empty array. Data from server is not retrieved. This function is useful if the logger is used without server-side storage (i.e., :attr:`saveToServer` is ``false``). :param event_type (string): event type .. function:: logger.getDataByName(event_type,event_name) As :func:`logger.getData` but only returns events with :attr:`name` atribute ``event_name``. :param event_type (string): event type .. function:: logger.clear(event_type) Clear in-memory and local storage of a specific event type. :param event_type (string): event type .. function:: logger.clearByName(event_type,event_name) Clear in-memory and local storage of a specific event type. Only events with :attr:`name` atribute ``event_name`` are removed. Attempts to store remaining events are to local storage. :param event_type (string): event type .. function:: logger.clearAll() Clear all in-memory and local storage of all event types. .. function:: logger.register(event_type,details) See section `Register settings`_, for an explanation of ``details``. :param event_type (string): event type :param details (object): properties overriding default settings .. function:: logger.setNamespace(namespace) See explanation in section Namespace_. .. function:: logger.getNamespace() :returns: the currently set namespace (= ``"default"``, if none has been specified). .. function:: logger.saveOldEvents(event_type) :param event_type (string): event type If there are any existing events of type 'event_type' in local storage (i.e., the file system), it will try to save these to the server with any in-memory events of 'event_type'. This function is called in function :func:`logger.register`. ------------------------------------------------------------------------------------- ***************************************** Appendix A: Server-side Processing in PHP ***************************************** .. code-block:: js+php < ?php // Remove space before ? in actual implementation function insert_data() { $method = $_SERVER['REQUEST_METHOD']; if ($method == 'POST' || $method == 'PUT') { $input_data = json_decode(file_get_contents('php://input')); } else { $input_data = $_GET; } $event_type = $input_data->event_type; $namespace = $input_data->namespace; $contents = $input_data->data; $con = mysql_connect("localhost","logger_user"); if (!$con) { die('Could not connect: ' . mysql_error()); } mysql_select_db("logged_events", $con); $count = 0; foreach ($contents as $event) { $sql="INSERT INTO events (event_type, namespace, event_name, data, created) VALUES ('$event_type','$namespace','$event->name','$event->data',$event->created)"; if (!mysql_query($sql,$con)) { die('MySQL error: ' . mysql_error()); } ++$count; } echo "$count records inserted"; mysql_close($con); } insert_data(); ------------------------------------------------------------------------------------- ********************************************* Appendix B: MySQL Structure of Database Table ********************************************* .. code-block:: mysql -- -- Table structure for table `events` -- CREATE TABLE IF NOT EXISTS `events` ( `index` int(11) NOT NULL AUTO_INCREMENT, `event_type` text NOT NULL, `namespace` text NOT NULL, `event_name` text NOT NULL, `data` longtext NOT NULL, `created` bigint(20) NOT NULL, PRIMARY KEY (`index`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=4 ; ------------------------------------------------------------------------------------- ********** Change Log ********** 2011-05-06 - Jaap ----------------- Did several things today: - Changed fatal bug with _checkEventRegistration(), which was not a function yet - Turned logger.js into nm/logger/_base.js construction - Changed 'doc' directory name into 'docs' - Updated documentation to reflect this. Also, updated documentation in various other places. - Found bug with unicode (e.g., Chinese) characters with _encrypt/_decrypt. Apparently the algorithms expect single-byte (i.e., ASCII) characters. Solved this by wrapping the data with simple encode_utf8/decode_utf8 functions. Works for Pinyin at least. For a more complete solution, see http://www.webtoolkit.info/javascript-utf8.html. 2011-05-07 - Jaap ----------------- Added two public functions that allow getting and clearing by event name (i.e., as in the ``name`` atribute): - :func:`getDataByName` - :func:`clearByName` - Also updated documentation to reflect this. Added tests directory and wrote ``test_logger.html``. Discovered a large bug that caused server access for each additional logged event once the ``repeatSaveEvents`` maximum had been reached, because the event counter was never reset. Changed the logic. Now, the counter is set to 0 before a save to server is attempted. If the save fails, it is *not* set back to its original value. This is deliberate to prevent save-to-server attempts with each new event in case of a connection failure.