Understanding "Periodicity" and "masterData"

What is "Periodicity"?

See CIQ.ChartEngine#setPeriodicity for API usage.

Understanding periodicity is fundamental to time series charting. Simply put, periodicity is the amount of time represented by one horizontal data point on the chart. For instance, on a candlestick chart, each candle represents the movement of price between a start time and an end time. That timeframe, for one candle, is what we call the "periodicity" of the chart.

Example: This chart has a periodicity of 30 minutes. Notice how the x-axis increases by 30 minutes with each candle:

Chart with 30m Candles

Periodicity for a chart can be any timeframe but is always denominated in a specific "time unit". ChartIQ supports the following time units: milliseconds, seconds, minutes, hours, days, weeks or months. The "interval" is the numeric portion of the time unit. For instance, a "17 minute" chart has an interval of 17 and a time unit of "minute".

"Rolling" Periodicity

Often, an application needs to display periodicities that aren't supported by the data source. Many data sources can only provide 1 minute, 5 minute, 30 minute and daily bars. The ChartIQ library supports rolling, or "composing", periodicity using a multiplier. For instance, if your data server provides 5 minute bars then the chart will be capable of display 5, 10, 15, 20... bars (any multiple of 5). We refer to this multiplier as the "period".

We tell the chart to roll periodicity with the first argument to CIQ.ChartEngine#setPeriodicity.

Example: Rolling periodicity:

// set the chart with a periodicity of 10 minutes. We are telling the chart that each data item represents 5 minutes, but to combine two 5 minute data items when creating each 10 minute bar.
stxx.setPeriodicity({period:2, interval:5, timeUnit:"minute"});

// Six 5 minute raw data items.
  {"Date":"2016-01-01 05:00:00", "Open":36.09, "Close":36.10},
  {"Date":"2016-01-01 05:05:00", "Open":36.10, "Close":36.12},
  {"Date":"2016-01-01 05:10:00", "Open":36.12, "Close":36.11},
  {"Date":"2016-01-01 05:15:00", "Open":36.11, "Close":36.10},
  {"Date":"2016-01-01 05:20:00", "Open":36.10, "Close":36.09},
  {"Date":"2016-01-01 05:25:00", "Open":36.09, "Close":35.99},

// Chart will display three 10 minute candles. The data will be "rolled" into this:
  {"Date":"2016-01-01 05:00:00", "Open":36.09, "Close":36.12},
  {"Date":"2016-01-01 05:10:00", "Open":36.12, "Close":36.10},
  {"Date":"2016-01-01 05:20:00", "Open":36.10, "Close":35.99},

The most frequent use case for rolling is to provide weekly or monthly charts. Very few data servers provide weekly or monthly data, but almost all of them provide daily data. The charting engine can "roll" daily OHLC bars into weekly or monthly charts. In fact, it does this by default!

See CIQ.ChartEngine.dontRoll to override this behavior if your feed already aggregates weekly and monthly bars.

Example: Displaying a weekly chart from daily data:

// Tell the chart to display weekly bar ( assumes dontRoll still defaults to false )
stxx.setPeriodicity({period:1, interval:'week'});

  {"Date":"2016-01-01", "Close":36.10},
  {"Date":"2016-01-02", "Close":36.11},
  {"Date":"2016-01-03", "Close":36.16},
  {"Date":"2016-01-04", "Close":36.01},
  {"Date":"2016-01-05", "Close":35.80},
  {"Date":"2016-01-08", "Close":35.62},
  {"Date":"2016-01-09", "Close":34.10},
  {"Date":"2016-01-10", "Close":33.10},
  {"Date":"2016-01-11", "Close":33.50},
  {"Date":"2016-01-12", "Close":33.90},
  {"Date":"2016-01-15", "Close":35.10},
  {"Date":"2016-01-16", "Close":35.30},
Rolling Periodicity: Performance considerations

Extra bandwidth and horsepower are required whenever rolling is required. Developers should be careful to ensure that rollup settings are reasonable.

The most common mistake is to use 1 minute bars to roll up very large intervals. A typical chart will require from 100 to 1,000 bars of data to display on the screen (depending on the user's screen size and zoom level). Generally we recommend trying to keep the amount of data under 20,000 data points. If you were to roll up a 30 minute chart using 1 minute bars, then you could easily exceed 20,000 data points which would negatively impact usability. It could take many seconds to transfer such a large set of data over the Internet to the browser, and another second to process the data.

Recommendation: Although not required; if you plan on supporting intraday charts, your data server should deliver at least: 1 minute, 5 minute and 30 minute bars. It is not efficient to produce large charts based on 1 minute bars because of the bandwidth necessary to transmit data. On the other hand, it is generally unnecessary for your server to support weekly or monthly data (these can be generated automatically for you by the chart).

Non-Linear Periodicities ("Tick" charts)

Some data sets consist of points that are not in even increments. A common example would be "tick" charts. A tick chart displays a horizontal data point for every trade that executes. A tick chart may display 20 points during one second and then not display a point for many seconds later.

The charting library gracefully handles these so called "non-linear" periodicities.

Example: Set the chart to a non-linear periodicity

stxx.setPeriodicity({period:1, interval:'tick'});
How Do I Create a Periodicity Menu?

Making a periodicity menu is easy. Simply set up some DOM objects to call CIQ.ChartEngine#setPeriodicity with values that make sense for your app!

Example: Menu that lets a user change periodicity. Note how we use a rollup for the 10 minute option.

    <span onclick="stxx.setPeriodicity({period:1, interval:1, timeUnit:'minute'})">1 Minute</span>
      <span onclick="stxx.setPeriodicity({period:1, interval:5, timeUnit:'minute'})">5 Minute</span>
      <span onclick="stxx.setPeriodicity({period:1, interval:30, timeUnit:'minute'})">30 Minute</span>
      <span onclick="stxx.setPeriodicity({period:2, interval:30, timeUnit:'minute'})">1 hour</span>
      <span onclick="stxx.setPeriodicity({period:1, interval:'day'})">Daily</span>
      <span onclick="stxx.setPeriodicity({period:1, interval:'week')">Weekly</span>
      <span onclick="stxx.setPeriodicity({period:1, interval:'month'})">Monthly</span>
      <span onclick="stxx.setPeriodicity({period:1, interval:'tick')">Ticks</span>

Of course the methodology will vary depending on whether you build your UI with JQuery, Angular, React or some other framework. See Implementing UI - All about menus and dialogs for examples using popular frameworks.

See {@ CIQ.ChartEngine#setPeriodicity} for API usage.

What is "masterData"?

"masterData" is what we call the Array of data that powers the chart. masterData is filled with objects in the ChartIQ OHLC format. At a minimum, each object will contain a DT field and a Close field. Usually objects will also contain Open,High,Low, and Volume. (Sometimes we say that masterData contains the "quotes" for the chart.)

Note: in some charting libraries, each series is represented by a separate array. In ChartIQ, each series is represented by a separate field in the masterData.

Typically, masterData is built automatically from data received through a quotefeed (see Tutorial quotefeed). Developers can also initialize a chart with your own array (see Tutorial static data). The chart engine will maintain the masterData object as new data is pushed into the chart (streaming), or prepended to the chart (pagination) or when new series are added (CIQ.ChartEngine#addSeries).

When you send time-series data to the chart, it must be properly sequenced. Each data point should represent the periodicity (interval + time unit) that has been specified (CIQ.ChartEngine#setPeriodicity):

  • The chart engine does not aggregate an array of data points into OHLC bars. This must be done before passing data to the chart (with the exception that you can stream data points using CIQ.ChartEngine#appendMasterData).

    Example: This chart will not automatically roll this data to 5 minute bars

    stxx.setPeriodicity({period:1, interval:5, timeUnit:"minute"}); // set the chart with a periodicity of 5 minutes
      {"Date":"2016-01-01 05:05:00", "Close":36.12},
      {"Date":"2016-01-01 05:05:12", "Close":36.11},
      {"Date":"2016-01-01 05:05:32", "Close":36.10},
      {"Date":"2016-01-01 05:05:54", "Close":36.09},
      {"Date":"2016-01-01 05:06:01", "Close":35.99},
    // Chart will display a bar for each data point!

  • Data must be ordered from oldest to newest. The chart engine assumes that array[0] is the oldest point and array[array.length-1] is the newest.

    Example: Uh oh! This array of data should be reversed!

    stxx.setPeriodicity({period:1, interval:5, timeUnit:"minute"}); // set the chart with a periodicity of 5 minutes
      {"Date":"2016-01-01 05:25:00", "Close":36.12},
      {"Date":"2016-01-01 05:20:00", "Close":36.11},
      {"Date":"2016-01-01 05:15:00", "Close":36.10},
      {"Date":"2016-01-01 05:10:00", "Close":36.09},
      {"Date":"2016-01-01 05:05:00", "Close":35.99},
    // Chart will fail!

  • It is not necessary to provide every data point. Your data can skip dates. The chart's x-axis will compress around the missing data point. If you send a data point with the Close set to null, then the chart will leave a gap around that data point:

    stxx.setPeriodicity({period:1, interval:5, timeUnit:"minute"}); // set the chart with a periodicity of 5 minutes
      {"Date":"2016-01-01 05:05:00", "Close":36.12},
      {"Date":"2016-01-01 05:10:00", "Close":36.11},
      /// We skip 05:15:00. The x-axis will compress.
      {"Date":"2016-01-01 05:20:00", "Close":36.10},
      {"Date":"2016-01-01 05:25:00", "Close":null}, // This will display as a gap, blank space on the chart
      {"Date":"2016-01-01 05:30:00", "Close":35.99},

Note: The date for a data point on a time-series chart represents the opening time for the data. For instance, if the market opens at 8am, the first data point of the session on a 5 minute chart would represent activity from 08:00:00.000 to 08:04:59.999. 08:05:00.000 would be the start of the next data point.

Automatically generated fields

If you examine stxx.masterData in the browser console you will notice that the array objects contain a few additional fields. If you provided a DT field then the chart will automatically produce a Date field. Conversely, if you provided Date then the chart will compute the DT field.

Whenever masterData is updated, the chart will perform timezone conversions to ensure that the chart's x-axis is displayed for the user's preferred timezone. The converted value is stored in displayDate which is added automatically.

Limiting the size of masterData

The masterData will generally grow indefinitely. For most charts this is not an issue, but when dealing with small time units such as seconds or milliseconds, the size of the masterData can grow very quickly. The variable stxx.maxDataSetSize can be used to limit the overall size of the chart. When the limit is reached, old bars will be dropped as new ones are added.

Methods that affect the contents of masterData

CIQ.ChartEngine#newChart - May be initialized with data, or will trigger an attached quotefeed to fetch fresh data.

CIQ.ChartEngine#setPeriodicity - Will trigger an attached quotefeed to fetch fresh data that replaces the masterData.

CIQ.ChartEngine#appendMasterData - Has a dual purpose:

  • 1-Can be used to push new OHLC bars to the chart. This will append new objects to the end of masterData.
  • 2-Can be used to push last sale data to the chart. The chart will calculate OHLC bars which will be appended to the end of masterData.

CIQ.ChartEngine#setMasterData - Developers can manually set masterData with this method. It should be used only in rare circumstances.

Next Steps: