Data Integration: Quotefeeds

Pulling data from a server

The easiest way to provide data to an interactive chart is by responding to the chart's requests for data, through callbacks. We call this the "pull" method of data integration.

Interactive charts require data to be fetched for multiple symbols, various ranges and various periodicities. This data may be needed at any moment, for instance when a user adds a comparison symbol or scrolls back in time. When these actions occur, the chart makes asynchronous requests for data which can be fulfilled by a "quotefeed".

A "quotefeed" is an object that responds to a chart's requests for data. It acts as a "bridge" between the chart and your data server. The chart can be considered a client of the bridge code you provide. A design pattern close to a quotefeed is the Bridge pattern. Typically, developers build their own quotefeed to integrate with their firm's proprietary data server. This tutorial explains how.

Note: ChartIQ provides ready-made quotefeeds for many market data vendors, including Xignite and FIS (SunGard). As of 8.4.0, prebuilt quotefeeds are available in a dedicated GitHub repo for real-data quotefeeds at https://github.com/chartiq/quotefeeds.

How a quotefeed works

The general structure of a quotefeed

A quotefeed is a JavaScript object that a developer builds and then attaches to the charting engine (stxx). After a quotefeed is attached, the chart will use the quotefeed's functions to fetch data. The quotefeed's functions fetch data from your server, then they return that data to the charting library.

The library expects a quotefeed to provide the following interface:

quotefeed.fetchInitialData() - This function is called by the library to request an array of initial data for for a particular instrument/symbol. This happens when:
- A chart is initialized with a new symbol loadChart(),importLayout()
- When a new "series" is added (e.g. comparison) addSeries()
- When periodicity (interval and/or timeUnit) changes (i.e. moving from 1 minute bars to 5 minute bars) setPeriodicity()
Parameters with a suggested date range (startDate, endDate) are provided. But initial data is a requirement when using a quotefeed. So if the suggested start and end dates return no data, you must continue to query your feed until something is returned. Then send that data back in your initial response.

Each initial load request attempts to fetch enough data to fill two full screens as well as a data buffer. The purpose is to pre-fetch enough data to minimize subsequent pagination requests. If no data is returned for the suggested date range, the chart engine incrementally increases the range by a factor of 2 until data is returned or moreAvailable is set to false.

This is especially important to keep in mind when loading tick periodicity data, for which data fluency can drastically vary between instruments causing the suggested dates to ask for too much or too little data. The chart will know to ask for more again if not enough to fill the screen, so feel free to add any additional logic you need in the fetch function to adjust the range before you make a request into your quotefeed. Or even better, use the params.ticks parameter instead. Which may be more appropriate for tick periodicity.

If less than the requested data range is available from the feed, and there is no longer a need to try any additional pagination requests, set moreAvailable = false in the callback to prevent the chart from continuing to ask for the same ranges.

Remember that requested data ranges will include the additional period required to also gather data for the buffer size set in CIQ.ChartEngine#attachQuoteFeed
quotefeed.fetchUpdateData() - This function is called by the library to request a data update - the most recent data not yet loaded into the chart; NOT changes to data already loaded.

As time progresses, new data is continually appended to the right side of the chart by this method to keep the display up to date.

The interval in which this call is made will be dictated by the refresh interval value set in CIQ.ChartEngine#attachQuoteFeed. And can be set to '0' if no updates are required or will be pushed-in using a streaming interface.

Be sure to never return data older than the start date requested (startDate).
quotefeed.fetchPaginationData() - This function is called by the library to add more data into the chart in response to user interactions; such as zooming and scrolling.
As a user scrolls past the currently loaded data on either edge of the chart, a pagination call will be made, and the returned data appended to the appropriate side of the masterData array.

Pagination is sometimes also referred to as 'Lazy Loading'.

It is important to note that there are 2 modes of pagination requests.
- Standard: Standard mode triggers if your initial load has provided data that covers a period all the way to the current candle (right now). In which case, there will only be pagination requests being made as you scroll the chart to the right to see older data on the left. You will only see pagination requests for older data.
- Historical: Historical mode triggers if you have used ranges or spans to see a time period far into the past. In this case, only the requested range will be immediately loaded into masterData and additional bars will be paginated forwards or backwards as needed to comply with user panning and zooming requests. Once the user has once again reached the current bar, the chart will automatically switch from 'Historical' pagination to 'Standard' pagination until a new range or span is set. This provides a huge performance boost and prevents large back-end data requests, which can produce large delays. This feature works automatically but only when using ranges or spans, and will not be enabled even if your initial load does not contain current data.
Be sure to never return data newer than the end date requested, or older than the start date requested as this can produce overlaps and errors. And if less than the requested data is available, set moreAvailable = false in the callback to prevent the chart from continuing to ask for the same ranges.

Remember that requested data ranges will include the additional period required to also gather data for the buffer size set in CIQ.ChartEngine#attachQuoteFeed. In historical mode, this will include buffering on both sides of the required range.

To permanently disable historical mode, use the following override:
```
CIQ.ChartEngine.prototype.isHistoricalMode=function(){return false};
```
This may be required if your implementation only displays delayed or historical data and there is never a need to query current data even when the most current candle is being shown on the chart.

The response from each of these methods modifies the underlying masterData and dataSet in the chart and triggers a redraw.

Quote Feeds request -and expect to be returned- full OHLC candles with time stamps matching the periodicity selected. As such, ticks are not augmented in any way, but rather inserted or replaced. When requesting updates for a bar that is still growing (the last bar on the chart), the feed should return whatever data is available for that bar time frame, up to that point, with a date matching the beginning of the current bar; not the current time. Therefore, if the chart is displaying bars, each representing 1 day, every single update response for that day must have the same exact date value corresponding to the current day.

To stream last sale updates instead see the Steaming-Data Integration Tutorial.

If your implementation needs to change existing data, manually call ChartEngine#updateChartData.

Note: when one of these methods is not provided, the chart will continue to operate but skip the associated events. For instance, to disable pagination simply don't provide fetchPaginationData.

Here's what a quotefeed looks like. We will refer to these as "fetch..." functions:

{
	fetchInitialData: function(...) {...your code...},
	fetchUpdateData: function(...) {...your code...},
	fetchPaginationData: function(...) {...your code...}
}

Implementing the "fetch..." functions

A quotefeed cannot directly (i.e., synchronously) return the requested data to the library. It must query a data server and wait for a response. The quotefeed must always return data to the library through a callback function. Callbacks are passed into the fetch request as a cb parameter. (If this isn't clear now, the example below will clarify.)

Also keep in mind that when the charting engine is displaying multiple symbols (i.e. the main symbol plus one or more comparison symbols), it will call the same "fetch.." function multiple times - once for each symbol. A "fetch..." function always handles a request for one symbol at a time. For example, if a chart for IBM is displayed with two comparison symbols, CSCO and APPL, then the chart engine calls fetchUndateDate() for IBM, a fetchUpdate() for CSCO, and a fetchUpdate() for APPL (3 times).

To summarize, the key quotefeed concepts are:

In order to interface with a proprietary data server, you attach to the library a custom-built quotefeed.
When the library needs new data, it calls the attached quotefeed's "fetch..." functions.
Typically a quotefeed defines three distinct "fetch..." functions; however they aren't all required -- the library skips over data requests that don't have a corresponding fetch function defined.
The quotefeed asynchronously returns data to the library using the callback function ("cb") passed in from the library to the "fetch..." functions.
The charting engine uses the quotefeed to fetch data for every symbol being displayed
All "fetch..." functions provide data for one symbol at a time.

Note: Within the "fetch..." functions, this will be bound to your quotefeed object.

Important differences between "fetch..." functions and setPeriodicity parameters

Note that period and interval parameters have different purposes in "fetch..." functions and not interchangeable with parameters of the same name on other API calls. See quotefeed for exact details.

Preventing infinite backwards pagination requests

Sometimes no historical data is returned by the quotefeed for a requested range. This of course can happen if there really is no more historical data, in which case just use the noLoadMore flag. But sometimes, this could happen due to thinly traded stocks, a weekend, or a holiday.

If you continue to return an empty array on the pagination callback, without setting the noLoadMore, the chart will continue to ask for the same range over and over again causing an infinite loop.

So how do you mitigate this scenario?

Here are two approaches that can be used:

Return a single data object in the callback with just the 'suggestedStartDate' in the DT field.
Example: [{DT:suggestedStartDate}];
This will preserve that date to be used as the starting point for the next pagination request.
You will have to eventually decide if you want to continue moving backwards, or stop. To stop just set noLoadMore to true
Simply make a subsequent request for more data, before returning an empty array. Then, if that additional request also returns no data, set noLoadMore to true, informing the chart that's all there is. If the additional request returns data, return that instead of the empty array.

Special requirements when not using a primary instrument on your chart

Typically, in technical analysis, a chart will first load a primary symbol using CIQ.ChartEngine#loadChart. This will become the anchor for the rest of the data loaded on the chart; such as comparisons against it or studies/indicators for it.

But sometimes an implantation will only require a group relative series, where none have to take the role of a primary anchor, and likely no studies will be applied.

To do this using a quotefeed, you have to ensure that your quotefeed will never return any pagination or update data for the primary instrument. You will also need to use a special syntax when calling loadChart; specifically:

Set an empty symbol (make sure it has at least one character)
Set an empty data array

See the live QuoteFeed example later in this tutorial, complete with all code and instructions.

Building a quotefeed

In this example we use the charting library's built-in postAjax function (CIQ.postAjax) to communicate with the data server. Other AJAX functions, such as those provided by jQuery, can be used instead.

var quotefeed = {};
quotefeed.url = "https://yourserver.yourdomain.com/fetch";

// utility function to translate data server response into library's required format
quotefeed.translate = function(response, cb) {
	var lines = response.split("\n");
	var newQuotes = [];
	for (var i = 0; i < lines.length; i++) {
		var fields = lines[i].split(",");
		newQuotes[i] = {};
		newQuotes[i].Date = fields[0]; // Or set newQuotes[i].DT if you have a JS Date
		newQuotes[i].Open = fields[1];
		newQuotes[i].High = fields[2];
		newQuotes[i].Low = fields[3];
		newQuotes[i].Close = fields[4];
		newQuotes[i].Volume = fields[5];
	}
	return newQuotes;
};

quotefeed.fetchInitialData = function(symbol, startDate, endDate, params, cb) {
	var url = this.url + "?symbol" + symbol;
	url += "&interval=" + params.interval;
	url += "&period=" + params.period;
	url += "&startDate=" + startDate;
	url += "&endDate" + endDate;

	CIQ.postAjax(url, null, function(status, response) {
		// process the HTTP response from the datafeed
		if (status == 200) {
			// if successful response from datafeed
			newQuotes = this.translate(response);
			cb({ quotes: newquotes });
		} else {
			// else error response from datafeed
			cb({ error: status });
		}
		return;
	});
};

quotefeed.fetchUpdateData = function(symbol, startDate, params, cb) {
	var url = this.url + "?symbol" + symbol;
	url += "&interval=" + params.interval;
	url += "&period=" + params.period;
	url += "&startDate=" + startDate;

	CIQ.postAjax(url, null, function(status, response) {
		// process the HTTP response from the datafeed
		if (status == 200) {
			// if successful response from datafeed
			newQuotes = this.translate(response);
			cb({ quotes: newquotes });
		} else {
			// else error response from datafeed
			cb({ error: status });
		}
		return;
	});
};

quotefeed.fetchPaginationData = function(symbol, startDate, endDate, params, cb) {
	var url = this.url + "?symbol" + symbol;
	url += "&interval=" + params.interval;
	url += "&period=" + params.period;
	url += "&startDate=" + startDate;
	url += "&endDate" + endDate;

	CIQ.postAjax(url, null, function(status, response) {
		// process the HTTP response from the datafeed
		if (status == 200) {
			// if successful response from datafeed
			newQuotes = this.translate(response);
			cb({ quotes: newquotes });
		} else if (status == 404) {
			// use whatever flags or status codes you feed returns when there is no more data.
			cb({ quotes: [], moreAvailable: false });
		} else {
			// else error response from datafeed
			cb({ error: status });
		}
		return;
	});
};

ciq = new CIQ.ChartEngine({ container: document.querySelector(".chartContainer") });
ciq.attachQuoteFeed(quotefeed, { refreshInterval: 1 });
ciq.loadChart("SPY");

Lookup (symbol search)

CIQ.ChartEngine.Driver.Lookup provides a default implementation that queries against ChartIQ's symbol server to perform a lookup of closely matching securities.

To create your own driver, simply derive a new class from CIQ.ChartEngine.Driver.Lookup and implement an acceptText() method. This method will be passed the text entered by the user, and any lookup filters that the user has selected. Results should be returned as an array of objects, each containing a data item (in whatever format you wish) and a display array with displayable values.

Example, sample Lookup.Driver :

CIQ.ChartEngine.Driver.Lookup.MyDriver = function() {};
CIQ.inheritsFrom(CIQ.ChartEngine.Driver.Lookup.MyDriver, CIQ.ChartEngine.Driver.Lookup);

CIQ.ChartEngine.Driver.Lookup.MyDriver.prototype.acceptText = function(
	text,
	filter,
	maxResults,
	cb
) {
	var myCB = function(data) {
		var results = [];
		for (var d in data) {
			results.push({
				data: { symbol: d }, // your result symbolObject
				display: [d, data[d].description, data[d].exchange]
			});
		}
		cb(results);
	};
	myFetchFunction(text, filter, maxResults, myCB);
};

Other quotefeed examples

Another example can be found in in the library's quoteFeedSimulator.js file. This example quotefeed connects to ChartIQ's data simulator.

A live quotefeed with a primary instrument

The following jsfiddle shows a quotefeed attached to ChartIQ's data-simulator server:

Example:

Click on the 'JavaScript' tab to see the code.

A live quotefeed without a primary instrument

The following jsfiddle shows how to use a quotefeed when no primary instrument is defined. It leverages the standard code used to attach to ChartIQ's data-simulator server; outlining the additional requirements.

Note that no data should ever be returned for the string matching the primary instrument placeholder used in CIQ.ChartEngine#loadChart

It will load three series, each one using its own y-axis:

Example:

Click on the 'JavaScript' tab to see the code.

Additional notes and FAQ

Does my server need to provide weekly and monthly data?

Your quote server only needs to support (1D) daily data. The charting engine can automatically roll up weekly and monthly bars. See the 'Periodicity and Your MasterData' for more information.
What intraday (minute, hour) intervals must my server support?

The charting engine can roll up intraday intervals to support "compounded" intervals. For instance, if your server can provide 1 minute bars then the charting engine can product 2 minute, 3 minute, 4 minute bars, etc. However, we recommend that, at a minimum, your server should support 1 minute, 5 minute and 30 minute bars. This will provide users with reasonably quick charts for any selected interval.
Array order

Items in the response data array must be ordered from oldest to newest. This means that your data[0] element will be the oldest in the set and your data[length-1] element will be the most current.
Filtering data

Your feed will probably not return a response that contains just the quote data in perfect form.

For example, a typical server response may look like this JSON:
```
{
	"status": "Success",
	"myserverdata": [
		{ "Date": "2015-05-27T15:00:00+00:00", "Close": 42.49 },
		{ "Date": "2015-05-27T15:30:00+00:00", "Close": 42.45 }
	]
}
```
Don't just return the server's response to the callback! You must format the data correctly and you must assign the correct object members before you make the callback. Generally this means calling JSON.parse() on your response and then reformatting the results.

Your response array must contain Close and Date -or DT. Please note that Open, High, Low, Close, Date and Volume are capitalized.
```
[
	{ "Date": "2015-05-27T15:00:00+00:00", "Close": 42.49 },
	{ "Date": "2015-05-27T15:30:00+00:00", "Close": 42.45 }
]
```
Finally, as long as you return properly formated OHLC objects, you can also include extra fields for use in studies or series. For example:
```
[
	{ "Date": "2015-05-27T15:00:00+00:00", "Close": 42.49, "Extra1": 37.31, "yourStudy": 84.57 },
	{ "Date": "2015-05-27T15:30:00+00:00", "Close": 42.45, "Extra1": 84.39, "yourStudy": 16.82 }
]
```
All data, including errors, must be returned using the callback. This is mandatory. Failure to do so will prevent the feed from making subsequent update calls (since it will still be waiting for the prior callback).
The callback format

Following is the format for your callback function:
```
cb({
	error: status, // only include when there was an error
	quotes: newQuotes, // an array of properly formatted ohlc data
	moreAvailable: suggestedStartDate.getTime() > 0, // set this to false if you know you are at the end of the data
	upToDate: true, // set to true in the initial and pagination fetch callbacks if the responses do not require more "future" data (i.e., forward pagination)
	beginDate: suggestedStartDate, // set this to indicate the beginning date of the fetch response
	endDate: suggestedEndDate // set this to indicate the ending date of the fetch response
}); // return fetched data
```
Note: As of 8.0.0, upToDate indicates whether there is more data available into the future; basically the opposite of moreAvailable, which indicates whether there is more data into the past (i.e., backwards pagination). This is necessary when the chart is displaying forecasted data that spans past the current date, or when the date range is set into the distant past, which causes the masterData object to contain non-current data elements so that when a user zooms or scrolls, the chart needs to know when to stop making forward pagination requests. Note: As of 9.0.0, beginDate and endDate indicate dates other than those of the first and last records' date. This is useful when fetching sparse data, so the same dates aren't refetched on pagination.
Defining the refresh interval

A chart's refresh (polling) interval is specified as refreshInterval in the second argument to the CIQ.ChartEngine#attachQuoteFeed method. Refresh interval is specified in seconds. You should set this to a value that is appropriate for your user base and which can be safely handled by your server. Generally, refresh intervals of 5 seconds or lower qualify as "real-time" from the user's perspective.
```
stxx.attachQuoteFeed(quotefeed, {
	refreshInterval: 5 // check for updates every 5 seconds
});
```

Next steps

Learn About Asynchronous Streaming Data - If you are receiving data asynchronously through a WebSocket