Importing External Data

Today, all charting software is connected to the internet and thus also to a data provider. This data is called "primary data feed" in tradesignal. For intraday trading, it is important to have up-to-date and self-refreshing data.

For some trading methods or for developing trading systems and tests, however, it is important to have long histories and a high time resolution. These histories are often expensive and can only be purchased as monthly subscriptions. A less expensive and more individual solution is the inclusion of historic data of files saved on hard drives, removable storage or web servers. For such quotation data, many free and pay sources are available. This allows the users to retrieve and pay only for the histories they need for their analysis.

Offline quotations from external sources can be used for:

• Extending the price histories of the online data provider
• Inserting instruments not offered by the online data provider
• Inserting data other than quotations (business or sentiment indicators, statistics)

External data is treated the same as data from the online provider. You can apply all functions like Chart Analysis, Indicators, Strategies, the Portfolio and the Scanner.

In the following, you can find information about the import of external data and the necessary structure of the file name and the data of such quotation files.

Importing External Data in tradesignal

Editing the Data Feeds

Entry Datafeed in the tradesignal OptionsYou can open the settings for the data feed in two ways:

• Double-click the connection icon in the lower right corner of the tradesignal window.
• Alternatively, open the file menu, select the tradesignal Options entry and click on Datafeed.

The area for data connection appears in the right side of the window. At the top you can find settings for the data connection to online providers, for example "tradesignal Realtime". Below you can find the settings for the "additional data sources" that are available offline.

Primary Data Feed (Online Data)

Here you can set up the primary data feed.

• Provider - Select your online provider here. The default setting is tradesignal Realtime.
• Timezone - Here you can configure the time zone. This is useful when you have US quotations, for example. Note that the time zone setting influences the history displayed in the chart and possibly the trading system results. The time displayed in the status bar also depends on the selected time zone.
• Settings - Here you can see the settings of your connection. To edit an entry, click on the value to the right, for example "300" for the parameter "Keep Alive Interval".

Click on Reconnect to the current data provider to reconnect to the online provider. This is necessary, for example, if the connection has been interrupted (red connection signal in the lower right corner of the tradesignal window).

Additional Data Sources (Offline Data)

Here you can edit additional data sources. Depending on the settings, this data is used in addition to or as a substitution for online data.

Any available path can be used as data source. You can define folders on local hard drives, external hard drives or network drives as additional data sources. It is also possible to use web or XML servers.

The list of data sources determines the order in which tradesignal 5 searches these folders. When you use a symbol in a chart, this is important if you have more than one folder for the same symbol. In this case, the order determines which data is displayed.

• New - Create a new data source. A dialog opens, see next section.
• Edit - Edit the selected data source.
• Delete - Delete the selected data source. Only the source reference is deleted, not the data on the drives.
• Up/Down - Use these buttons to change the sequence of folders in the list.

Adding and Editing Additional Data Sources

Window view "Location of Data Source"Set up the type of the additional data source:

• In a folder on this workstation - To configure the folder, click on Browse. Select the folder in which the imported files are located. This may be any folder available on your computer or via intranet. You can also create a new folder for future import. The folder is also called "top-Up service", as it works like a data provider.

Please note that external quotation files can only be used when they are located in the configured folder and adhere to both the Naming Conventions for Files and the Data Formats.

• On a web or XML server - Select this option if the external data is to be found on a web or XML server. Enter a URL.

Select the Timezone from the list (see above).

You can enter a Text that is added to the legend name of an instrument. The default string is "top-up". You can enter any other string or delete the entry and leave the field empty. The additional label on the offline data is intended to help you distinguish them from online data. This is especially useful if both data sets have the same shortcut. Additional label won't be visble when extending the price histories of the online data.

Combining Online and Offline Data

If symbol data is available online and offline (top-up), you can configure how tradesignal should proceed:

Use top-up data to backfill the provider history before its oldest available data only (default setting) - Usually, the online data provider offers all instruments needed. However, the available history is often too short or no intraday data is available. If you want to fill the missing history by data of offline sources, the files have to be saved in the folder configured above and the data shortcut has to be the same as for the instrument of the online provider. Only if both conditions are met, tradesignal can concatenate the data. The program attempts to fill the history into the past. The additional data is inserted at the point where the history of the online data provider ends.

Use top-up data to overwrite existing provider history, using only those provider history parts that are newer than any top-up data. - This setting is only sensible if you have enough offline data for this instrument, so that most or all of the online data is unnecessary. Alternatively, you could also enter a new instrument name so that no collision or concatenation could take place. However, if you want to use a general shortcut for the data without using the online data, this is the right option.

Conditions for the Correct Display of Trading Periods

For tradesignal to recognize the time resolution (period) of your data, certain conditions have to be met for the File Names and the Data Format. In cases of display problems or certain trading periods missing in the chart although the data is available, the error can usually be tracked down to one of these conditions.

Example: Configure an Additional Data Source in tradesignal

Step 1: Opening the Connection Settings in tradesignal

1. Open the file menu to the left of the command line.
2. Click on the menu point tradesignal Options.
3. Click on the tab Datafeed.

Step 2: Define a File Folder As Data Source

1. Click on New to configure a new connection.
2. Click on Browse to open the folder list.
3. Browse to the folder "tradesignal Files", select it and click on New Folder to create a new folder.
4. Enter the name "Ascii Data" and click on Create.
5. Confirm with OK.
6. As Time Zone, select "FFR - Frankfurt".
7. Confirm with OK.
8. Apply all new settings by clicking on Apply.

Step 3: Import and Save File

Save the file Sample02-1d.txt in the folder described above (right-click the link and "save as target").

If you have read the article about File Names, you can see that the shortcut for the instrument is Sample02 and that the data has a resolution of one day. Keep this instrument name for now. You can always change it later, if desired.

Step 4: Open Chart

1. Enter the shortcut "Sample02" in the command line.
2. Select the option New Chart from the drop-down menu.

The chart with the "Sample02" data is opened.

File Names for Quotation Data

File names have an important function - ideally, they tell the user what kind of data is included and how it is to be displayed. The operating system automatically opens the program assigned to the file extension.

In the case of quotation data like in our case, the file name has another important function. It shows us and the software to which instrument the data belongs, which trading period is used and more. If you have a problem displaying data, this is probably caused by an error either in the file name or in the Data Format.

Please note where the information in a filename is used and if it is mandatory or optional.

File Names

The file name can/must include the following information:

• Shortcut (mandatory)
• Period (mandatory)
• Field Identifier (optional)

File names can be constructed in the following ways:

• Shortcut.FileExtension
• Shortcut-Period.FileExtension
• Shortcut-FID-Period.FileExtension

As a separator between the parameters, the minus sign is used. If you want to use a shortcut that already includes a minus sign, substitute it with a comma or semicolon. Valid file extensions recognized by tradesignal are .csv, .asc and .txt.

Shortcut

If you want to be able to use the file name in the Command Line of tradesignal, e.g. to open a chart with the data, the shortcut has to follow a certain convention. The shortcut is also important if you want to combine data from your online provider with offline data.

Notation: If you want to use a shortcut independent of your online provider, you may freely choose the filename. However, if you want to combine data from your online provider with offline data (for example to extend the history or to add intraday data for an instrument), the shortcut of the offline data has to be identical to the one used by the online provider. Otherwise, the external offline data will not be assigned correctly.

For shortcuts with a leading point (e.g. many indices), a trick has to be used since Windows does not allow filenames with a leading point. Just add a plus sign before the point.

Example for an instrument with leading point: +.gdaxi-5m.csv

Period

The period is optional in a file name. If no period is given, tradesignal automatically displays the data in daily intervals, even if intraday data is available in the file. The software can display the data correctly only if the period parameter is used. If you receive an error when changing the period settings of your external data in the Chart or Scanner, it is very likely that this error results from a missing or invalid period value.

Notation:

Period	Shortcode	Example
Tick Data	tick, 0, t, tick, ticks, 1t	aaa-tick.csv
1 Minute	60, 1m, m, min, minute, minutes, minuten	pumg.de-60.csv
5 Minutes	300, 5m, 5min, 5mins, 5minute, 5minuten, 5minutes	adsg.de-5m.csv
30 Minutes	1800, 30m, 30min, 30mins, 30minute, 30minuten, 30minutes	depfa-30min.csv
1 Hour	3600, 1h, h, hour, hourly, 1hour, stunde, stunden	eurusd-hour.csv
1 Day	86400, 1d, d, day, daily, tag	+.rut-1d.txt
1 Week	604800, 1w, w, week, weekly, woche	+.gdaxi-w.csv
1 Month	2592000, month, mothly, monat	dowjones-monthly.csv

Field Identifier

The Field Identifier is only mandatory if instead of the normal quotations (open, high, low and close) the bid, ask, or last prices are given in the file. These prices can only be displayed with the correct identifier. The following identifiers are valid:

• last
• bid
• ask

Example for a file with bid prices: +.gdaxi-bid-5min.csv
Example for a file with ask prices: +.gdaxi-ask-5min.csv

Data Format of External Quotations

View of an ASCII file in a text editorIn files of quotation data, the data format (i.e. the sequence of elements such as date, time, prices and volume) defines the data display and whether the program can handle it at all. Since there are many possible combinations, import problems may be caused by data in the wrong format. Often, a simple wrong separator or a non-supported date format can cause havoc with the data.

When you need to edit the data format, you can do this with a text editor. For exchanging complete columns, however, usually programs like Excel or Access are necessary.

A file with quotation data has the following components:

• Header - recommended but optional
• Data columns separated by separators (by default semicolon or comma)

tradesignal recognizes the following column types:

• Date
• Time - optional
• Open - optional
• High - optional
• Low - optional
• Close
• Volume - optional
• Open Interest - optional

For the display of intraday data, time and date stamps are necessary. Otherwise, the data is filled in beginning with the current trading day, which may not always give correct results. If you want to add information to the history of your online data provider, make sure that your file has the same data elements as the file from your provider.

Header

The first segment of the file should be the header. It contains information about the data columns, their order and the used separators. The header is optional; when no header is given, tradesignal attempts to interpret and display the segments. However, this may cause errors.

In the header, information about the columns is given in the way of identifiers. The identifiers have to be separated by the same separator as the data columns. For each column type, several identifiers are possible, as you can see in the following list:

Notation

• Date - d, date, datetime (when date and time are combined)
• time - t, time
• Open - o, op, open
• High - h, hi, high
• Low - l, lo, low
• Close - c, cl, close
• Volume - v, vo, vol, volume
• OpenInterest - oi, openint, ointerest, openinterest

Example Header: Date;Time;Open;High;Low;Close;Volume

If no header is given, tradesignal uses the following masks to analyze the column grouping:

• d,o,h,l,c
• d,o,h,l,c,v
• d,o,h,l,c,v,oi
• d,c
• d,c,v
• d,c,v,oi

Some data sources, for example Internet offers, contain data in additional columns. Since tradesignal cannot display those, the data has to be skipped. For this, the word "Skip" can be used in the header. In the following header example, the last column is skipped.

Example for a file with an additional column that is skipped:

Date,Open,High,Low,Close,Volume,Adj. Close*09.11.05,10539.24,10637.78,10466.24,10546.21,22144600,10546.21 

results in:

Date,Open,High,Low,Close,Volume,Skip09.11.05,10539.24,10637.78,10466.24,10546.21,22144600,10546.21 

Data and Time Formats

For date and time, many different formats exist.

Date Format

• European Format - Example: 01.04.2006
• English (US) Format - Example: 04/01/06
• Standard ISO Format - Example: 1998-12-22T01:02:03:987

When the European format is recognized, a comma has to be used as decimal separator. In the case of an English format, a decimal point has to be used as decimal separator.

Additional differences may exist in the day, month, year sequence, especially in the case of UK/US formats. See the example below for combinations recognized by tradesignal.

Time Stamp

No flexibility is given regarding the time stamp format. The separator has always to be a colon. If necessary, the separators of an external data file have to be exchanged to a colon in a text editor.

Components of the Time Stamp

• hour
• minute
• second
• one-tenth of a second (for tick data)

For English time zones, "AM" or "PM" can be used if the hours are given in a 12 hour interval.

Examples for Valid Date and Time Stamps in tradesignal

English Formats

• 1998/1/11 11:05:00
• 5/17/1998 11:05:00 AM
• 5/17/1998 8:05:00 PM
• 2005/10/25 12:25:11 PM
• 2005/10/25 11:05:00:124 AM
• 5-17-1998
• 1998-12-22
• 1998-12-22 01:02 PM
• 98-12-22

European Formats

• 04.12.98
• 04.12.1998
• 04.12.1998 12:23
• 04.12.1998 12:23:12
• 04.12.1998 12:23:14:122
• 04.12.1998 12:23:14.122

ISO Formats

• 1998-12-22T01:02:03
• 1998-12-22T01:02:03:987