Data Integration: Quotefeeds

Pulling Data From a Server

The easiest way to provide data to an interactive chart is by responding to the chart's requests for data, through callbacks. We call this the "pull" method of data integration.

Interactive charts require data to be fetched for multiple symbols, various ranges and various periodicities. This data may be needed at any moment, for instance when a user adds a comparison symbol or scrolls back in time. When these actions occur, the chart makes asynchronous requests for data which can be fulfilled by a "quotefeed".

A "quotefeed" is an object that responds to a chart's requests for data. It acts as a "bridge" between the chart and your data server. Typically, developers build their own quotefeed to integrate with their firm's proprietary data server. This tutorial explains how.

Data Integration: Considerations and Overview covers the broader topic of data-integration with the library, including a brief introduction to quotefeeds.

Other supporting tutorials include Getting Started, The ChartIQ OHLC Data Format for Charts and Periodicity and Your Master Data.

Note: ChartIQ provides ready-made quotefeeds for many market data vendors including Xignite and Thomson Financial.

How a quotefeed works:

How a quotefeed works

The General Structure of a Quotefeed

A quotefeed is a JavaScript object that a developer builds and then attaches to the charting engine (stxx). After a quotefeed is attached, the chart will use the quotefeed's functions to fetch data. The quotefeed's functions fetch data from your server, then they return that data to the charting library.

The library expects a quotefeed to provide the following interface:

  • quotefeed.fetchInitialData() - This function is called by the library to request an array of data for a specific range of datetimes. This happens when:

    • A chart is initialized with a new symbol newChart(),importLayout()

    • When a new "series" is added (e.g. comparison) addSeries()

    • When periodicity (interval) changes (i.e. moving from 1 minute bars to 5 minute bars) setPeriodicity()

  • quotefeed.fetchUpdateData() - This function is called by the library to request a data update - the most recent data. As time progresses, new data is continually appended to the right side of the chart (streaming data).

  • quotefeed.fetchPaginationData() - This function is called by the library to request older data (pagination). Pagination occurs when a user scrolls past the first chart bar (left edge). Pagination data is "prepended" to the beginning of the chart.

The response from each of these methods modifies the underlying masterData and dataSet in the chart, and triggers a redraw.

Note: when one of these methods is not provided, the chart will continue to operate but skip the associated events. For instance, to disable pagination simply don't provide fetchPaginationData.

Here's what a quotefeed looks like. We will refer to these as "fetch..." functions:


{
    fetchInitialData: function(...){...your code...},
    fetchUpdateData: function(...){...your code...},
    fetchPaginationData: function(...){...your code...}
}

Implementing the "fetch..." functions

A quotefeed cannot directly (i.e. synchronously) return the requested data to the library. It must query a data server and wait for a response. The quotefeed always returns data to the library through a callback function. Callbacks are passed into the fetch request as a cb parameter. (If this isn't clear now, the example below will clarify.)

Also keep in mind that when the charting engine is displaying multiple symbols (i.e. the main symbol plus one or more comparison symbols), it will call the same "fetch.." function multiple times - once for each symbol. A "fetch..." function always handles a request for one symbol at a time. For example, if a chart for IBM is displayed with two comparison symbols, CSCO and APPL, then the chart engine calls fetchUndateDate() for IBM, a fetchUpdate() for CSCO, and a fetchUpdate() for APPL (3 times).

To summarize, the key quotefeed concepts are:

  • In order to interface with a proprietary data server, you attach to the library a custom-built quotefeed.

  • When the library needs new data, it calls the attached quotefeed's "fetch..." functions.

  • Typically a quotefeed defines three distinct "fetch..." functions; however they aren't all required -- the library skips over data requests that don't have a corresponding fetch function defined.

  • Within each fetch function, this is bound to the quotefeed object.

  • The quotefeed asynchronously returns data to the library using the callback function ("cb") passed in from the library to the "fetch..." functions.

  • The charting engine uses the quotefeed to fetch data for every symbol being displayed

  • All "fetch..." functions provide data for one symbol at a time.

Note, within the "fetch..." functions, this will be bound to your quotefeed object.

A Quotefeed Example

Step 1: Setting up the "fetch..." functions

The below code snippet outlines a quotefeed and shows how it's attached to the chart engine. Once the quotefeed is attached, its methods called as needed by the library to supply chart data.


    var quotefeed = {}; // create quotefeed object

    quotefeed.fetchInitialData = function (symbol, startDate, endDate, params, cb) {
        // custom code to fetch initial data goes here -- data is returned using callback parameter cb
    };

    quotefeed.fetchUpdateData = function (symbol, startdate, params, cb) {
        // custom code to fetch update data goes here -- data is returned using callback parameter cb
    };

    quotefeed.fetchPaginationData = function (symbol, suggestedStartDate, enddate, params, cb) {
        // custom code to fetch pagination data goes here -- data is returned using callback parameter cb
    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")}); // get chart engine instance
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1}); // ** attach quotefeed to chart engine **
    ciq.newChart("SPY"); // create and display the chart for "SPY" -- the quotefeed will supply the data
Step 2: Code to fetch data from server

Using the quotefeed outline, we can now implement code to fetch data from your server. For now, we will pretend that your server can provide the requested data by checking its query string for startDate and/or endDate (we'll ignore errors for now). We're also going to pretend that your server returns data in the exact format that the chart engine requires.

Typically, HTTP is used to communicate with a data server; therefore, JQuery's AJAX Promises are used to send data requests and receive data responses from the server.

Here's what your quotefeed will look like (Note: pagination is not supported yet (i.e. fetchPaginationData is not defined.)):


    var quotefeed = {};

    // Return an array of all bars between startDate and endDate

    quotefeed.fetchInitialData = function (symbol, startDate, endDate, params, cb) {
        var url="http://yourserver.yourdomain.com/fetch?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&startDate=" + startDate;
        url+="&endDate" + endDate;

        $(ajax)({url:url})
        .done(function(response){ // on  successful response
            cb({quotes:response}); // pass the fetched data back to the library
        });
    };

    // Note how fetchUpdateData takes *only* a startDate. Return an array
    // of all bars beginning with startDate

    quotefeed.fetchUpdateData = function (symbol, startdate, params, cb) {
        var url="http://yourserver.yourdomain.com/fetch?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&startDate=" + startDate;

        $(ajax)({url:url})
        .done(function(response){ // on successful response
            cb({quotes:response}); // pass the fetched data back to the library
        });
    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")}); // get library instance
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1}); // attach quotefeed to library
    ciq.newChart("SPY"); // create and display new chart -- quotefeed will supply the data

Sidebar: streaming data is often thought to update a single bar at a time. In reality, a stream will sometimes update two bars. This happens whenever the chart advances (by one interval bar). Consider a system that fetches a fresh bar every 5 seconds. From :00-:59 the chart will only require a single bar ever 5 seconds, updating the most recent bar with the most recent closing price.

However, when a request is made at :03 seconds we need to get two bars. We need the final bar from :00.000 - :59.999; and we need a new bar that represents the current interval - from :00 - :03.

It's not necessary to think too hard about it. The important thing to remember is that your quote server should always respond with all of the bars beginning at a requested startDate. That is, your server should always return an array of bars, even for updates.

Step 3: Error checking

With a basic quotefeed in place, let's add a couple of enhancements.

1) We'll set the URL as a member of the quotefeed object (and reference in the fetch functions with this).

2) We'll add some error checking to the ajax response.


    var quotefeed = {};
    quotefeed.url = "http://yourserver.yourdomain.com/fetch"; // ** save URL with quotefeed **

    quotefeed.fetchInitialData = function (symbol, startDate, endDate, params, cb) {
        var url=this.url + "?symbol" + symbol; // ** start building query using URL **
        url+="&interval=" + params.interval;
        url+="&startDate=" + startDate;
        url+="&endDate" + endDate;

        $(ajax)({url:url})
        .done(function(response){
            newQuotes = this.translate(response);
            cb({quotes:newquotes});
        })
        .fail(function(status){
            cb({error:status}); // ** error response added **
        });
    };

    quotefeed.fetchUpdateData = function (symbol, startdate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&startDate=" + startDate;

        $(ajax)({url:url})
        .done(function(response){
            newQuotes = this.translate(response);
            cb({quotes:newquotes});
        })
        .fail(function(status){
            cb({error:status}); // ** error response added **
        });
    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")});
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1});
    ciq.newChart("SPY");
Step 4: Pagination

Now we can complete our quotefeed with a few more enhancements.

1) We'll implement a pagination function to support the continuous display of historical data when the user scrolls left.

2) Also, when no more pagination data is available, we will signal the charting engine to stop making pagination requests by setting moreAvailable=false in the callback parameter.

3) Finally, given that you are most likely working with an existing quote server, you'll have to provide some translation code -- let's assume now that we're getting a comma-separated response value from the data server.


    var quotefeed = {};
    quotefeed.url = "http://yourserver.yourdomain.com/fetch";

    // utility function to translate data server response into library's required format
    quotefeed.translate = function (response, cb) {
        var lines=response.split('\n');
        var newQuotes=[];
        for(var i=0;i<lines.length;i++){
            var fields=lines[i].split(',');
            newQuotes[i]={};
            newQuotes[i].Date=fields[0]; // Or set newQuotes[i].DT if you have a JS Date
            newQuotes[i].Open=fields[1];
            newQuotes[i].High=fields[2];
            newQuotes[i].Low=fields[3];
            newQuotes[i].Close=fields[4];
            newQuotes[i].Volume=fields[5];
        }
        return newQuotes;
   };

    quotefeed.fetchInitialData = function (symbol, startDate, endDate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&startDate=" + startDate;
        url+="&endDate" + endDate;

        $(ajax)({url:url})
        .done(function(response){
            newQuotes = this.translate(response);
            cb({quotes:newquotes});
        })
        .fail(function(status){
            cb({error:status});
        });
    };

    quotefeed.fetchUpdateData = function (symbol, startdate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&startDate=" + startDate;

        $(ajax)({url:url})
        .done(function(response){
            newQuotes = this.translate(response);
            cb({quotes:newquotes});
        })
        .fail(function(status){
            cb({error:status});
        });
    };

    quotefeed.fetchPaginationData = function (symbol, suggestedStartdate, enddata, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&startDate=" + suggestedStartDate;
        url+="&endDate" + endDate;

        $(ajax)({url:url})
        .done(function(response){
                newQuotes = this.translate(response);
                cb({quotes:newquotes});
        })
        .fail(function(status){
            if (status==404) { // signals no more data is available from our particular data server
                cb({error:status, moreAvailable:false});
            else {
                  cb({error:status});
            }
        });
    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")});
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1});
    ciq.newChart("SPY");

From the examples you can see that your "fetch..." methods must always call the provided callback ("cb") method. An object should be set as the sole parameter to the callback. That object must have either (1) a "quotes" parameter that is an array of properly formatted data or (2) an "error" parameter that is a string error message.

This completes our first example, showing one way a quotefeed might be built.

Building a Quotefeed Without JQuery

The previous example relied on jQuery's AJAX functions to communicate with the data server. Now let's show the same example without jQuery -- this time the charting library's built-in postAjax function (CIQ.postAjax) is used to communicate with the server.


    var quotefeed = {};
    quotefeed.url = "http://yourserver.yourdomain.com/fetch";

    // utility function to translate data server response into library's required format
    quotefeed.translate = function (response, cb) {
         var lines=response.split('\n');
        var newQuotes=[];
        for(var i=0;i<lines.length;i++){
            var fields=lines[i].split(',');
            newQuotes[i]={};
            newQuotes[i].Date=fields[0]; // Or set newQuotes[i].DT if you have a JS Date
            newQuotes[i].Open=fields[1];
            newQuotes[i].High=fields[2];
            newQuotes[i].Low=fields[3];
            newQuotes[i].Close=fields[4];
            newQuotes[i].Volume=fields[5];
        }
        return newQuotes;
   };

    quotefeed.fetchInitialData = function (symbol, startDate, endDate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&startDate=" + startDate;
        url+="&endDate" + endDate;

        CIQ.postAjax(url, null, function(status, response){
            // process the HTTP response from the datafeed
            if(status==200){ // if successful response from datafeed
                   newQuotes = this.translate(response);
                   cb({quotes:newquotes});
            } else { // else error response from datafeed
                cb({error:status});
            }
            return;
        });
    };

    quotefeed.fetchUpdateData = function (symbol, startdate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&startDate=" + startDate;

        CIQ.postAjax(url, null, function(status, response){
            // process the HTTP response from the datafeed
            if(status==200){ // if successful response from datafeed
                newQuotes = this.translate(response);
                cb({quotes:newquotes});
            } else { // else error response from datafeed
                cb({error:status});
            }
            return;
        });
    };

    quotefeed.fetchPaginationData = function (symbol, suggestedStartdate, enddata, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&startDate=" + suggestedStartDate;
        url+="&endDate" + endDate;

        CIQ.postAjax(url, null, function(status, response){
            // process the HTTP response from the datafeed
            if(status==200){ // if successful response from datafeed
                newQuotes = this.translate(response);
                cb({quotes:newquotes});
              else if (status==404) { // signals no more data is available from our data server
                  cb({error:status, moreAvailable:false});
               } else { // else error response from datafeed
                cb({error:status});
            }
            return;
        });

    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")});
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1});
    ciq.newChart("SPY");

Other Quotefeed Examples

Another example can be found in in the library's quoteFeedSimulator.js file. This example quotefeed connects to ChartIQ's data simulator.

Note: Other quotefeeds found in the library (e.g. quoteFeedXignite.js) are based on the legacy quotefeed interface, which is still supported but no longer recommended.

A Live QuoteFeed

The following jsfiddle shows a live quotefeed to ChartIQ's data-simulator server:

Example:

  • Click on the 'JavaScript' tab to see the code.

Additional Notes and FAQ

1) Does my server need to provide weekly and monthly data?

Your quote server only needs to support (1D) daily data. The charting engine can automatically roll up weekly and monthly bars. See the 'Periodicity and Your MasterData' for more information.

2) What intraday (minute, hour) intervals must my server support?

The charting engine can roll up intraday intervals to support "compounded" intervals. For instance, if your server can provide 1 minute bars then the charting engine can product 2 minute, 3 minute, 4 minute bars, etc. However, we recommend that, at a minimum, your server should support 1 minute, 5 minute and 30 minute bars. This will provide users with reasonably quick charts for any selected interval.

3) Array Order:

Items in the response data array must be ordered from oldest to newest. This means that your data[0] element will be the oldest in the set and your data[length-1] element will be the most current.

4) Filtering Data

Your feed will probably not return a response that contains just the quote data in perfect form.

For example a typical server response may look like this JSON:


{
   "status": "Success",
   "myserverdata":
   [{"Date": "2015-05-27T15:00:00+00:00", "Close": 42.49},
    {"Date": "2015-05-27T15:30:00+00:00", "Close": 42.45}]
}

Don't just return the server's response to the callback! You must format the data correctly and you must assign the correct object members before you make the callback. Generally this means calling JSON.parse() on your response and then reformatting the results.

Your response array must contain Close and Date -or DT. Please note that Open, High, Low, Close, Date and Volume are capitalized.


[{"Date": "2015-05-27T15:00:00+00:00", "Close": 42.49}, {"Date": "2015-05-27T15:30:00+00:00", "Close": 42.45}]

Finally, as long as you return properly formated OHLC objects, you can also include extra fields for use in studies or series. For example:


[{"Date": "2015-05-27T15:00:00+00:00", "Close": 42.49, "Extra1": 37.31, "yourStudy": 84.57}, {"Date": "2015-05-27T15:30:00+00:00", "Close": 42.45, "Extra1": 84.39, "yourStudy": 16.82}]

All data, including errors, must be returned using the callback. This is mandatory. Failure to do so will prevent the feed from making subsequent update calls (since it will still be waiting for the prior callback).

5) The Callback Format

Following is the format for your callback function:


    cb({
        error:status, // only include when there was an error
        quotes:newQuotes, // an array of properly formatted ohlc data
        moreAvailable:false // set this to false if you know you are at the end of the data
    });

6) Defining the refresh interval

A chart's refresh (polling) interval is specified as refreshInterval in the second argument to the CIQ.ChartEngine#attachQuoteFeed method. Refresh interval is specified in seconds. You should set this to a value that is appropriate for your user base and which can be safely handled by your server. Generally, refresh intervals of 5 seconds or lower qualify as "real-time" from user's perspective.


    stxx.attachQuoteFeed(quotefeed, {
        refreshInterval: 5    // check for updates every 5 seconds
    });

Next Steps: