Data Integration: Quotefeeds

Pulling Data From a Server

The easiest way to provide data to an interactive chart is by responding to the chart's requests for data, through callbacks. We call this the "pull" method of data integration.

Interactive charts require data to be fetched for multiple symbols, various ranges and various periodicities. This data may be needed at any moment, for instance when a user adds a comparison symbol or scrolls back in time. When these actions occur, the chart makes asynchronous requests for data which can be fulfilled by a "quotefeed".

A "quotefeed" is an object that responds to a chart's requests for data. It acts as a "bridge" between the chart and your data server. Typically, developers build their own quotefeed to integrate with their firm's proprietary data server. This tutorial explains how.

Data Integration: Considerations and Overview covers the broader topic of data-integration with the library, including a brief introduction to quotefeeds.

Other supporting tutorials include Getting Started, The ChartIQ OHLC Data Format for Charts and Periodicity and Your Master Data.

Note: ChartIQ provides ready-made quotefeeds for many market data vendors including Xignite and Thomson Financial.

How a quotefeed works:

How a quotefeed works

The General Structure of a Quotefeed

A quotefeed is a JavaScript object that a developer builds and then attaches to the charting engine (stxx). After a quotefeed is attached, the chart will use the quotefeed's functions to fetch data. The quotefeed's functions fetch data from your server, then they return that data to the charting library.

The library expects a quotefeed to provide the following interface:

  • quotefeed.fetchInitialData() - This function is called by the library to request an array of initial data for for a particular instrument/symbol. This happens when:

    -A chart is initialized with a new symbol newChart(),importLayout()

    -When a new "series" is added (e.g. comparison) addSeries()

    -When periodicity (interval and/or timeUnit) changes (i.e. moving from 1 minute bars to 5 minute bars) setPeriodicity()

    Parameters with a suggested date range (suggestedStartDate, suggestedEndDate) are provided, but the actual range of data returned is entirely up to your particular implementation and data availability.

    This especially important to keep in mind when loading tick periodicity data, for which data fluency can drastically vary between instruments causing the suggested dates to ask for too much or too little data. The chart will know to ask for more again if not enough to fill the screen, so feel free to add any additional logic you need in the fetch function to adjust the range before you make a request into your quote feed. Or even better, use the params. ticks parameter instead. Which may be more appropriate for tick periodicity.

    If less than the requested data range is available from the feed, and there is no longer a need to try any additional pagination requests, set moreAvailable=false in the callback to prevent the chart from continuing to ask for the same ranges.

    Remember that requested data ranges will include the additional period required to also gather data for the buffer size set in CIQ.ChartEngine#attachQuoteFeed

  • quotefeed.fetchUpdateData() - This function is called by the library to request a data update - the most recent data not yet loaded into the chart; NOT changes to data already loaded.

    As time progresses, new data is continually appended to the right side of the chart by this method to keep the display up to date.

    The interval in which this call is made will be dictated by the refresh interval value set in CIQ.ChartEngine#attachQuoteFeed. And can be set to '0' if no updates are required or will be pushed-in using a streaming interface.

    Be sure to never return data older than the start date requested (startDate).

  • quotefeed.fetchPaginationData() - This function is called by the library to add more data into the chart in response to user interactions; such as zooming and scrolling.

    As a user scrolls past the currently loaded data on either edge of the chart, a pagination call will be made and the returned data appended to the appropriate side of the masterData array.

    Pagination is sometimes also referred to as 'Lazy Loading'.

    It is important to note that there are 2 modes of pagination requests.

    • Standard: Standard pagination triggers if your initial load has provided data that covers a period all the way to the current candle (right now). In which case, there will only be pagination requests being made as you scroll the chart to the right to see older data on the left. You will only see pagination requests for older data.
    • Historical: Historical pagination triggers if you have used ranges or spans to see a time period far into the past. In this case, only the requested range will be immediately loaded into masterData and additional bars will be paginated forwards or backwards as needed to comply with user panning and zooming requests. Once the user has once again reached the current bar, the chart will automatically switch from 'Historical' pagination to 'Standard' pagination until a new range or span is set. This provides a huge performance boost and prevents large back-end data requests, which can produce large delays. This feature works automatically but only when using ranges or spans, and will not be enabled even if your initial load does not contain current data.

    Be sure to never return data newer than the end date requested, or older than the start date requested as this can produce overlaps and errors. And if less than the requested data is available, set moreAvailable=false in the callback to prevent the chart from continuing to ask for the same ranges.

    Remember that requested data ranges will include the additional period required to also gather data for the buffer size set in CIQ.ChartEngine#attachQuoteFeed. In historical mode, this will include buffering on both sides of the required range.

The response from each of these methods modifies the underlying masterData and dataSet in the chart, and triggers a redraw.

Quote Feeds request -and expect to be returned- full OHLC candles with time stamps matching the periodicity selected. As such, ticks are not augmented in any way, but rather inserted or replaced. When requesting updates for a bar that is still growing (the last bar on the chart), the feed should return whatever data is available for that bar time frame, up to that point, with a date matching the beginning of the current bar; not the current time. Therefore, if the chart is displaying bars, each representing 1 day, every single update response for that day must have the same exact date value corresponding to the current day.

To stream last sale updates instead see the Steaming-Data Integration Tutorial.

If your implementation needs to change existing data, manually call ChartEngine#updateChartData.

Note: when one of these methods is not provided, the chart will continue to operate but skip the associated events. For instance, to disable pagination simply don't provide fetchPaginationData.

Here's what a quotefeed looks like. We will refer to these as "fetch..." functions:


{
    fetchInitialData: function(...){...your code...},
    fetchUpdateData: function(...){...your code...},
    fetchPaginationData: function(...){...your code...}
}

Implementing the "fetch..." functions

A quotefeed cannot directly (i.e. synchronously) return the requested data to the library. It must query a data server and wait for a response. The quotefeed must always returns data to the library through a callback function. Callbacks are passed into the fetch request as a cb parameter. (If this isn't clear now, the example below will clarify.)

Also keep in mind that when the charting engine is displaying multiple symbols (i.e. the main symbol plus one or more comparison symbols), it will call the same "fetch.." function multiple times - once for each symbol. A "fetch..." function always handles a request for one symbol at a time. For example, if a chart for IBM is displayed with two comparison symbols, CSCO and APPL, then the chart engine calls fetchUndateDate() for IBM, a fetchUpdate() for CSCO, and a fetchUpdate() for APPL (3 times).

To summarize, the key quotefeed concepts are:

  • In order to interface with a proprietary data server, you attach to the library a custom-built quotefeed.

  • When the library needs new data, it calls the attached quotefeed's "fetch..." functions.

  • Typically a quotefeed defines three distinct "fetch..." functions; however they aren't all required -- the library skips over data requests that don't have a corresponding fetch function defined.

  • The quotefeed asynchronously returns data to the library using the callback function ("cb") passed in from the library to the "fetch..." functions.

  • The charting engine uses the quotefeed to fetch data for every symbol being displayed

  • All "fetch..." functions provide data for one symbol at a time.

Note, within the "fetch..." functions, this will be bound to your quotefeed object.

A Quotefeed Example

Step 1: Setting up the "fetch..." functions

The below code snippet outlines a quotefeed and shows how it's attached to the chart engine. Once the quotefeed is attached, its methods called as needed by the library to supply chart data.


    var quotefeed = {}; // create quotefeed object

    quotefeed.fetchInitialData = function (symbol, suggestedStartDate, suggestedEndDate, params, cb) {
        // custom code to fetch initial data goes here -- data is returned using callback parameter cb
    };

    quotefeed.fetchUpdateData = function (symbol, startDate, params, cb) {
        // custom code to fetch update data goes here -- data is returned using callback parameter cb
    };

    quotefeed.fetchPaginationData = function (symbol, suggestedStartDate, endDate, params, cb) {
        // custom code to fetch pagination data goes here -- data is returned using callback parameter cb
    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")}); // get chart engine instance
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1}); // ** attach quotefeed to chart engine **
    ciq.newChart("SPY"); // create and display the chart for "SPY" -- the quotefeed will supply the data
Step 2: Code to fetch data from server

Using the quotefeed outline, we can now implement code to fetch data from your server. For simplicity, let's assume, that your server can provide the requested data by checking its query string for period, interval, startDate and/or endDate. For now, we will ignore connectivity errors and assume that your server returns data in the exact format required by the chart engine. We will get back to handling that in the following sections.

Typically, HTTP is used to communicate with a data server; therefore, JQuery's AJAX Promises are used to send data requests and receive data responses from the server.

It is important that your code returns the right amount of data in the requested interval (minutes, days, weeks, months). You must always evaluate all pertinent params, as well as symbol, startDate and endDate.

  • Returning less than requested data, may cause unnecessary retries that can cause performance issues.
  • Returning data in the wrong periodicity will cause the chart to malfunction.

See STX.QuoteFeed for a listing and explanation of all available params.

Here's what your quotefeed may look like (Note: pagination will be covered in the following section):


    var quotefeed = {};

    // Return an array of all bars between startDate and endDate

    quotefeed.fetchInitialData = function (symbol, startDate, endDate, params, cb) {
        var url="https://yourserver.yourdomain.com/fetch?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + suggestedStartDate;
        url+="&endDate" + suggestedEndDate;

        $(ajax)({url:url})
        .done(function(response){ // on  successful response
            cb({quotes:response}); // pass the fetched data back to the library
        });
    };

    // Note how fetchUpdateData takes *only* a startDate. Return an array
    // of all bars newer than startDate

    quotefeed.fetchUpdateData = function (symbol, startDate, params, cb) {
        var url="https://yourserver.yourdomain.com/fetch?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + startDate;

        $(ajax)({url:url})
        .done(function(response){ // on successful response
            cb({quotes:response}); // pass the fetched data back to the library
        });
    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")}); // get library instance
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1}); // attach quotefeed to library
    ciq.newChart("SPY"); // create and display new chart -- quotefeed will supply the data

Sidebar: a chart update is often thought to just modify or add the last bar. In reality, an update request may sometimes need two bars to keep the chart current. This happens whenever the chart needs to advance (by one interval bar), but has not yet received the final update for the current bar. Consider a system that fetches a fresh bar every 5 seconds to update 10-minute candles. If the last fetch took place at 14:57, for example, the next fetch will take place at 15:03. As such, the response from the 15:03 request should include both, a record for the 14:50 bar, as well as the 15:00 bar.

The good news is that the feed will be smart enough to know this, and make a fetch request with 14:50 as the startDate.

It's not necessary to think too hard about it. The important thing to remember is that your quote server should always respond with all of the bars beginning at a requested startDate and return an array of bars, even for updates.

Step 3: Error checking

With a basic quotefeed in place, let's add a couple of enhancements.

1) We'll set the URL as a member of the quotefeed object (and reference in the fetch functions with this).

2) We'll add some error checking to the ajax response.


    var quotefeed = {};
    quotefeed.url = "https://yourserver.yourdomain.com/fetch"; // ** save URL with quotefeed **

    quotefeed.fetchInitialData = function (symbol, suggestedStartDate, suggestedEndDate, params, cb) {
        var url=this.url + "?symbol" + symbol; // ** start building query using URL **
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + suggestedStartDate;
        url+="&endDate" + suggestedEndDate;

        $(ajax)({url:url})
        .done(function(response){
            newQuotes = this.translate(response);
            cb({quotes:newquotes});
        })
        .fail(function(status){
            cb({error:status}); // ** error response added **
        });
    };

    quotefeed.fetchUpdateData = function (symbol, startDate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + startDate;

        $(ajax)({url:url})
        .done(function(response){
            newQuotes = this.translate(response);
            cb({quotes:newquotes});
        })
        .fail(function(status){
            cb({error:status}); // ** error response added **
        });
    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")});
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1});
    ciq.newChart("SPY");
Step 4: Pagination

Now we can complete our quotefeed with a few more enhancements.

1) We'll implement a pagination function to support the continuous display of historical data when the user scrolls left, or also when the user scrolls right if in 'historical mode'.

2) When no more pagination data is available, we will signal the charting engine to stop making pagination requests by setting moreAvailable=false in the callback parameter.

3) Given that you are most likely working with an existing quote server, you'll have to provide some translation code -- in this example we assume now that we're getting a comma-separated response value from the data server.


    var quotefeed = {};
    quotefeed.url = "https://yourserver.yourdomain.com/fetch";

    // utility function to translate data server response into library's required format
    quotefeed.translate = function (response, cb) {
        var lines=response.split('\n');
        var newQuotes=[];
        for(var i=0;i<lines.length;i++){
            var fields=lines[i].split(',');
            newQuotes[i]={};
            newQuotes[i].Date=fields[0]; // Or set newQuotes[i].DT if you have a JS Date
            newQuotes[i].Open=fields[1];
            newQuotes[i].High=fields[2];
            newQuotes[i].Low=fields[3];
            newQuotes[i].Close=fields[4];
            newQuotes[i].Volume=fields[5];
        }
        return newQuotes;
   };

    quotefeed.fetchInitialData = function (symbol, suggestedStartDate, suggestedEndDate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + suggestedStartDate;
        url+="&endDate" + suggestedEndDate;

        $(ajax)({url:url})
        .done(function(response){
            newQuotes = this.translate(response);
            cb({quotes:newquotes});
        })
        .fail(function(status){
            cb({error:status});
        });
    };

    quotefeed.fetchUpdateData = function (symbol, startdate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + startDate;

        $(ajax)({url:url})
        .done(function(response){
            newQuotes = this.translate(response);
            cb({quotes:newquotes});
        })
        .fail(function(status){
            cb({error:status});
        });
    };

    quotefeed.fetchPaginationData = function (symbol, suggestedStartDate, enddata, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + suggestedStartDate;
        url+="&endDate" + endDate;

        $(ajax)({url:url})
        .done(function(response){
                newQuotes = this.translate(response);
                cb({quotes:newquotes});
        })
        .fail(function(status){
            if (status==404) { // in your case use whatever flags or status codes you feed returns when there is no more data.
                cb({quotes:[], moreAvailable:false});
            else {
                  cb({error:status});
            }
        });
    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")});
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1});
    ciq.newChart("SPY");

From the examples you can see that your "fetch..." methods must always call the provided callback ("cb") method. An object should be set as the sole parameter to the callback. That object must have either (1) a "quotes" parameter that is an array of properly formatted data or (2) an "error" parameter that is a string error message.

This completes our first example, showing one way a quotefeed might be built.

Building a Quotefeed Without JQuery

The previous example relied on jQuery's AJAX functions to communicate with the data server. Now let's show the same example without jQuery -- this time the charting library's built-in postAjax function (CIQ.postAjax) is used to communicate with the server.


    var quotefeed = {};
    quotefeed.url = "https://yourserver.yourdomain.com/fetch";

    // utility function to translate data server response into library's required format
    quotefeed.translate = function (response, cb) {
         var lines=response.split('\n');
        var newQuotes=[];
        for(var i=0;i<lines.length;i++){
            var fields=lines[i].split(',');
            newQuotes[i]={};
            newQuotes[i].Date=fields[0]; // Or set newQuotes[i].DT if you have a JS Date
            newQuotes[i].Open=fields[1];
            newQuotes[i].High=fields[2];
            newQuotes[i].Low=fields[3];
            newQuotes[i].Close=fields[4];
            newQuotes[i].Volume=fields[5];
        }
        return newQuotes;
   };

    quotefeed.fetchInitialData = function (symbol, suggestedStartDate, suggestedEndDate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + suggestedStartDate;
        url+="&endDate" + suggestedEndDate;

        CIQ.postAjax(url, null, function(status, response){
            // process the HTTP response from the datafeed
            if(status==200){ // if successful response from datafeed
                   newQuotes = this.translate(response);
                   cb({quotes:newquotes});
            } else { // else error response from datafeed
                cb({error:status});
            }
            return;
        });
    };

    quotefeed.fetchUpdateData = function (symbol, startDate, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + startDate;

        CIQ.postAjax(url, null, function(status, response){
            // process the HTTP response from the datafeed
            if(status==200){ // if successful response from datafeed
                newQuotes = this.translate(response);
                cb({quotes:newquotes});
            } else { // else error response from datafeed
                cb({error:status});
            }
            return;
        });
    };

    quotefeed.fetchPaginationData = function (symbol, suggestedStartdate, enddata, params, cb) {
        var url=this.url + "?symbol" + symbol;
        url+="&interval=" + params.interval;
        url+="&period=" + params.period;
        url+="&startDate=" + suggestedStartDate;
        url+="&endDate" + endDate;

        CIQ.postAjax(url, null, function(status, response){
            // process the HTTP response from the datafeed
            if(status==200){ // if successful response from datafeed
                newQuotes = this.translate(response);
                cb({quotes:newquotes});
              else if (status==404) { // in your case use whatever flags or status codes you feed returns when there is no more data.
                  cb({quotes:[], moreAvailable:false});
               } else { // else error response from datafeed
                cb({error:status});
            }
            return;
        });

    };

    ciq=new CIQ.ChartEngine({container:$$$(".chartContainer")});
    ciq.attachQuoteFeed(quotefeed, {refreshInterval:1});
    ciq.newChart("SPY");

Symbol Lookup

CIQ.ChartEngine.Driver.Lookup provides a default implementation that queries against ChartIQ's symbol server to perform a lookup of closely matching securities.

To create your own driver, simply derive a new class from CIQ.ChartEngine.Driver.Lookup and implement an acceptText() method. This method will be passed the text entered by the user, and any lookup filters that the user has selected. Results should be returned as an array of objects, each containing a data item (in whatever format you wish) and a display array with displayable values.

Example, sample Lookup.Driver :

CIQ.ChartEngine.Driver.Lookup.MyDriver=function(){};
CIQ.ChartEngine.Driver.Lookup.MyDriver.ciqInheritsFrom(CIQ.ChartEngine.Driver.Lookup);

CIQ.ChartEngine.Driver.Lookup.MyDriver.prototype.acceptText=function(text, filter, maxResults, cb){
    var myCB=function(data){
        var results=[];
        for(var d in data){
            results.push({
              data:{symbol:d}, // your result symbolObject
              display:[d, data[d].description, data[d].exchange]
            });
        }
        cb(results);
    };
    myFetchFunction(text, filter, maxResults, myCB);
};

Other Quotefeed Examples

Another example can be found in in the library's quoteFeedSimulator.js file. This example quotefeed connects to ChartIQ's data simulator.

Note: Other quotefeeds found in the library (e.g. quoteFeedXignite.js) are based on the legacy quotefeed interface, which is still supported but no longer recommended.

A Live QuoteFeed

The following jsfiddle shows a live quotefeed to ChartIQ's data-simulator server:

Example:

  • Click on the 'JavaScript' tab to see the code.

Additional Notes and FAQ

1) Does my server need to provide weekly and monthly data?

Your quote server only needs to support (1D) daily data. The charting engine can automatically roll up weekly and monthly bars. See the 'Periodicity and Your MasterData' for more information.

2) What intraday (minute, hour) intervals must my server support?

The charting engine can roll up intraday intervals to support "compounded" intervals. For instance, if your server can provide 1 minute bars then the charting engine can product 2 minute, 3 minute, 4 minute bars, etc. However, we recommend that, at a minimum, your server should support 1 minute, 5 minute and 30 minute bars. This will provide users with reasonably quick charts for any selected interval.

3) Array Order:

Items in the response data array must be ordered from oldest to newest. This means that your data[0] element will be the oldest in the set and your data[length-1] element will be the most current.

4) Filtering Data

Your feed will probably not return a response that contains just the quote data in perfect form.

For example a typical server response may look like this JSON:


{
   "status": "Success",
   "myserverdata":
   [{"Date": "2015-05-27T15:00:00+00:00", "Close": 42.49},
    {"Date": "2015-05-27T15:30:00+00:00", "Close": 42.45}]
}

Don't just return the server's response to the callback! You must format the data correctly and you must assign the correct object members before you make the callback. Generally this means calling JSON.parse() on your response and then reformatting the results.

Your response array must contain Close and Date -or DT. Please note that Open, High, Low, Close, Date and Volume are capitalized.


[{"Date": "2015-05-27T15:00:00+00:00", "Close": 42.49}, {"Date": "2015-05-27T15:30:00+00:00", "Close": 42.45}]

Finally, as long as you return properly formated OHLC objects, you can also include extra fields for use in studies or series. For example:


[{"Date": "2015-05-27T15:00:00+00:00", "Close": 42.49, "Extra1": 37.31, "yourStudy": 84.57}, {"Date": "2015-05-27T15:30:00+00:00", "Close": 42.45, "Extra1": 84.39, "yourStudy": 16.82}]

All data, including errors, must be returned using the callback. This is mandatory. Failure to do so will prevent the feed from making subsequent update calls (since it will still be waiting for the prior callback).

5) The Callback Format

Following is the format for your callback function:


    cb({
        error:status, // only include when there was an error
        quotes:newQuotes, // an array of properly formatted ohlc data
        moreAvailable:false // set this to false if you know you are at the end of the data
    });

6) Defining the refresh interval

A chart's refresh (polling) interval is specified as refreshInterval in the second argument to the CIQ.ChartEngine#attachQuoteFeed method. Refresh interval is specified in seconds. You should set this to a value that is appropriate for your user base and which can be safely handled by your server. Generally, refresh intervals of 5 seconds or lower qualify as "real-time" from user's perspective.


    stxx.attachQuoteFeed(quotefeed, {
        refreshInterval: 5    // check for updates every 5 seconds
    });

Next Steps: