Data Export Ticks - Saves to CSV

What is the meaning behind these two functions?
It seems like he is offsetting the volume with the price. But doesn't that result in an information loss, or what exactly is the intended effect?

        protected double vol_weighted_price(cAlgo.API.Collections.IReadonlyList<cAlgo.API.MarketDepthEntry> mkentries)
        {
            double weightdiv = 0.0;
            double tsum = 0.0;
            for (int i = 0; i < mkentries.Count; i++)
            {
                var aent = mkentries[i];
                tsum += (double)aent.Price * (double)aent.Volume;
                weightdiv += (double)aent.Volume;
            }
            if (weightdiv == 0)
            {
                return 0;
            }
            else
            {
                return tsum / weightdiv;
            }
        }


        protected double vol_weighted_cnt(cAlgo.API.Collections.IReadonlyList<cAlgo.API.MarketDepthEntry> mkentries)
        {
            double weightdiv = 0.0;
            double tsum = 0.0;
            for (int i = 0; i < mkentries.Count; i++)
            {
                var aent = mkentries[i];
                tsum += (double)aent.Volume * (double)aent.Price;
                weightdiv += (double)aent.Price;
            }
            if (weightdiv == 0)
            {
                return 0;
            }
            else
            {
                return tsum / weightdiv;
            }
        }

public static class OperationsUtlity  
     {  
        public static DataTable createDataTable()  
        {  
            DataTable table = new DataTable();     
                //columns  
            table.Columns.Add("ID", typeof(int));     
            table.Columns.Add("NAME", typeof(string));     
            table.Columns.Add("CITY", typeof(string));    
  
                //data  
            table.Rows.Add(111, "Devesh", "Ghaziabad");     
            table.Rows.Add(222, "ROLI", "KANPUR");     
            table.Rows.Add(102, "ROLI", "MAINPURI");     
            table.Rows.Add(212, "DEVESH", "KANPUR");  
            table.Rows.Add(102, "NIKHIL", "GZB");  
            table.Rows.Add(212, "HIMANSHU", "NOIDa");  
            table.Rows.Add(102, "AVINASH", "NOIDa");  
            table.Rows.Add(212, "BHUPPI", "GZB");  
              
             return table;  
          }  
         
    } 
 

Create a static class

public static class  CSVUtlity  { }
Add Extention Method 

public static void ToCSV(this DataTable dtDataTable, string strFilePath){ 
public static void ToCSV(this DataTable dtDataTable, string strFilePath) {  
    StreamWriter sw = new StreamWriter(strFilePath, false);  
    //headers    
    for (int i = 0; i < dtDataTable.Columns.Count; i++) {  
        sw.Write(dtDataTable.Columns[i]);  
        if (i < dtDataTable.Columns.Count - 1) {  
            sw.Write(",");  
        }  
    }  
Apkbeer.com

Convert To csv

    sw.Write(sw.NewLine);  
    foreach(DataRow dr in dtDataTable.Rows) {  
        for (int i = 0; i < dtDataTable.Columns.Count; i++) {  
            if (!Convert.IsDBNull(dr[i])) {  
                string value = dr[i].ToString();  
                if (value.Contains(',')) {  
                    value = String.Format("\"{0}\"", value);  
                    sw.Write(value);  
                } else {  
                    sw.Write(dr[i].ToString());  
                }  
            }  
            if (i < dtDataTable.Columns.Count - 1) {  
                sw.Write(",");  
            }  
        }  
        sw.Write(sw.NewLine);  
    }  
    sw.Close();  
} 

using System;

using System.Linq;

using System.IO;

using cAlgo.API;

using cAlgo.API.Indicators;

using cAlgo.API.Internals;

using cAlgo.Indicators;

namespace cAlgo

{

    [Robot(TimeZone = TimeZones.UTC, AccessRights = AccessRights.FileSystem)]

    public class DataExportTicks : Robot

    {

        // QUESTION:  How do I set default Source Timeframe to T1 (1 tick)

        //[Parameter("TimeFrame ")]

        //public TimeFrame TFrame { get; set; }

        // QUESTION: How do I get Backteset to default to 1 tick from server

        //  so don't have to manually reset?

        [Parameter("MA Type")]

        public MovingAverageType MAType { get; set; }

        [Parameter("Data Dir", DefaultValue = "c:\\download\\calgo")]

        public string DataDir { get; set; }

        private string fiName;

        private System.IO.FileStream fstream;

        private System.IO.StreamWriter fwriter;

        private string csvhead = "date,ask,bid,spread,num_ask,num_bid,vol_adj_ask,vol_adj_bid,vol_adj_spread,vac_ask,vac_bid,vac_bear_vs_bull\n";

        protected override void OnStart()

        {

            var ticktype = MarketSeries.TimeFrame.ToString();

            fiName = DataDir + "\\" + "exp-" + Symbol.Code + "-ticks.csv";

            Print("fiName=" + fiName);  // Best Logo Design Australia

            if (System.IO.File.Exists(fiName) == false)

            {

                // generate new file with CSV header only if

                // one does not already exist.

                System.IO.File.WriteAllText(fiName, csvhead);

            }

            // had to open file this way to prevent .net from locking it and preventing

            // access by other processes when using to download live ticks.

            fstream = File.Open(fiName, FileMode.Open, FileAccess.Write, FileShare.ReadWrite);

            // setup to append to end of file

            Print("File is Open");

            fstream.Seek(0, SeekOrigin.End);

            // write stream has to be created after seek due to .net wierdness

            // creating with 0 prevents buffering since we want tick data

            // to be available to consumers right away.

            fwriter = new System.IO.StreamWriter(fstream, System.Text.Encoding.UTF8, 1);

            // QUESTION:  How to tell when in Backtest mode so we

            //  can create the stream with a large buffer and turn off

            // auto flush to improve IO performance.

            Print("Fwriter is created");

            fwriter.AutoFlush = true;

            // with autoflush true will autocleanup

            // since we can not close since we may run forever

            Print("done onStart()");

        }

I'm facing an error in the above code. 

1. public static class OperationsUtlity  
2.      {  
3.         public static DataTable createDataTable()  
4.         {  
5.             DataTable table = new DataTable();     
6.                 //columns  
7.             table.Columns.Add("ID", typeof(int));     
8.             table.Columns.Add("NAME", typeof(string));     
9.             table.Columns.Add("CITY", typeof(string));    
10.   
11.                 //data  
12.             table.Rows.Add(111, "Devesh", "Ghaziabad");     
13.             table.Rows.Add(222, "ROLI", "KANPUR");     
14.             table.Rows.Add(102, "ROLI", "MAINPURI");     
15.             table.Rows.Add(212, "DEVESH", "KANPUR");  
16.             table.Rows.Add(102, "NIKHIL", "GZB");  
17.             table.Rows.Add(212, "HIMANSHU", "NOIDa");  
18.             table.Rows.Add(102, "AVINASH", "NOIDa");  
19.             table.Rows.Add(212, "BHUPPI", "GZB");  
20.               
21.              return table;  
22.           }  
23.          
24.     } 

Create a static class

1. public static class  CSVUtlity  { }

Add Extention Method 

1. public static void ToCSV(this DataTable dtDataTable, string strFilePath){ 

1. public static void ToCSV(this DataTable dtDataTable, string strFilePath) {  
2.     StreamWriter sw = new StreamWriter(strFilePath, false);  
3.     //headers    
4.     for (int i = 0; i < dtDataTable.Columns.Count; i++) {  
5.         sw.Write(dtDataTable.Columns[i]);  
6.         if (i < dtDataTable.Columns.Count - 1) {  
7.             sw.Write(",");  
8.         }  
9.     }  

apkact.com

Convert To csv

1.     sw.Write(sw.NewLine);  
2.     foreach(DataRow dr in dtDataTable.Rows) {  
3.         for (int i = 0; i < dtDataTable.Columns.Count; i++) {  
4.             if (!Convert.IsDBNull(dr[i])) {  
5.                 string value = dr[i].ToString();  
6.                 if (value.Contains(',')) {  
7.                     value = String.Format("\"{0}\"", value);  
8.                     sw.Write(value);  
9.                 } else {  
10.                     sw.Write(dr[i].ToString());  
11.                 }  
12.             }  
13.             if (i < dtDataTable.Columns.Count - 1) {  
14.                 sw.Write(",");  
15.             }  
16.         }  
17.         sw.Write(sw.NewLine);  
18.     }  
19.     sw.Close();  
20. } 

Export to CSV on button click

1. private void btnCSV_Click(object sender, EventArgs e) {  
2.     DataTable dt = OperationsUtlity.createDataTable();  
3.     string filename = OpenSavefileDialog();  
4.     dt.ToCSV(filename);  
5. } 

using System;

using System.Linq;

using System.IO;

using cAlgo.API;

using cAlgo.API.Indicators;

using cAlgo.API.Internals;

using cAlgo.Indicators;

namespace cAlgo

{

    [Robot(TimeZone = TimeZones.UTC, AccessRights = AccessRights.FileSystem)]

    public class DataExportTicks : Robot

    {

        // QUESTION:  How do I set default Source Timeframe to T1 (1 tick)

        //[Parameter("TimeFrame ")]

        //public TimeFrame TFrame { get; set; }

        // QUESTION: How do I get Backteset to default to 1 tick from server

        //  so don't have to manually reset?

        [Parameter("MA Type")]

        public MovingAverageType MAType { get; set; }

        [Parameter("Data Dir", DefaultValue = "c:\\download\\calgo")]

        public string DataDir { get; set; }

        private string fiName;

        private System.IO.FileStream fstream;

        private System.IO.StreamWriter fwriter;

        private string csvhead = "date,ask,bid,spread,num_ask,num_bid,vol_adj_ask,vol_adj_bid,vol_adj_spread,vac_ask,vac_bid,vac_bear_vs_bull\n";

        protected override void OnStart()

        {

            var ticktype = MarketSeries.TimeFrame.ToString();

            fiName = DataDir + "\\" + "exp-" + Symbol.Code + "-ticks.csv";

            Print("fiName=" + fiName);  // Shareit for pc

            if (System.IO.File.Exists(fiName) == false)

            {

                // generate new file with CSV header only if

                // one does not already exist.

                System.IO.File.WriteAllText(fiName, csvhead);

            }

            // had to open file this way to prevent .net from locking it and preventing

            // access by other processes when using to download live ticks.

            fstream = File.Open(fiName, FileMode.Open, FileAccess.Write, FileShare.ReadWrite);

            // setup to append to end of file

            Print("File is Open");

            fstream.Seek(0, SeekOrigin.End);

            // write stream has to be created after seek due to .net wierdness

            // creating with 0 prevents buffering since we want tick data

            // to be available to consumers right away.

            fwriter = new System.IO.StreamWriter(fstream, System.Text.Encoding.UTF8, 1);

            // QUESTION:  How to tell when in Backtest mode so we

            //  can create the stream with a large buffer and turn off

            // auto flush to improve IO performance.

            Print("Fwriter is created");

            fwriter.AutoFlush = true;

            // with autoflush true will autocleanup

            // since we can not close since we may run forever

            Print("done onStart()");

        }

I'm facing an error in the above code. 

Excume, I have typed the code above  and try to run it on Calgo but I got error messenge like this:" Error CS0120: An object reference is required for the non-static field, method, or property 'cAlgo.API.Internals.Algo.TimeZone.get' "

what can I do to fix this issue.

using System;
using System.Linq;
using System.IO;
using cAlgo.API;
using cAlgo.API.Indicators;
using cAlgo.API.Internals;
using cAlgo.Indicators;

namespace cAlgo
{
    [Robot(TimeZone = TimeZone.UTC, AccessRights = AccessRights.FileSystem)]
    public class DataExportTicks : Robot

Need help using the bot. 

The log message before crash is as follows: "29/07/2015 21:24:41.915 | Crashed in OnTick with IOException: The process cannot access the file 'c:\exp-EURUSD-ticks.csv' because it is being used by another process."

Pops up right after the start of the bot then crashes. The csv file contains only headers (date, open, close etc). 

Thanks

I guess, in backtesting we have to overwrite file. Currently it will be appended.

Sorry, Excel formatting was wrong.

To simplify excel formatting I separated Date and Time into two values.

What would be a way to get 24 hour format instead of AM/PM?

Thanks

Hi Joe,

your cBot is what I was looking to write on my own, but why have you made it cBot, instead of indicator?

Thanks

Hey Deansi,   I took the time to add the order book values you mentioned.   Let me know if it was what you intended. 

I found a free data source with older data http://ratedata.gaincapital.com  it takes a little work to coerce the file formats to match.  I have found that working with tick data it doesn't seem to do much good to feed more than 200 days into our Machine Learning algorithms but the longer time frames are helpful when back-testing to see how the engine accommodates changing market conditions. 

Hey King,   Good point on the localized numbers.    I think it would be better to use \t as the delimiter so excel can still load it as delimited file than ";"  

Densi,  Thanks for the tip on the MarketDepth.  I will change that as it makes more sense to feed into external engines.       The isBacktesting flag is much appreciated.

I updated my bar downloader to attempt to backfill correctly but don't have time to do the same with the Tickdata. Perhaps in Jan. 

Just to add to my post above, the code below will tell you if youre currently backtesting:

            if (IsBacktesting == true)

Also just a note that the MarketDepthEntry.Volume i mention in the previous post only works in live mode I believe since the server doesnt store this data at this point.

Enjoy

Joe,

Great work, I can answer a couple of your questions you had in the code comments.

The reason the volumes are always the same is the VolumeMin and VolumeMax method youre using is actually getting the min/max volume you are able to open a position in dollars etc.

You may want to try something like MarketDepth and MarketDepthEntry.Volume  

and AskEntries and BidEntries,these will give you the market pending volume available, and seems to be updated on every tick.

Ive looked into the other question about timeframe in cbots tab defaulting before, it seemed there is no way to set the option in a cbot, but you can use this code below instead so the cbots tab setting has no effect and you will get ticks output regardless:           

            // var barTime = MarketSeries.OpenTime.LastValue; //this was getting the bar start time
            var barTime = Server.Time ;  //this gets the current server time, works live and backtesting

Hi Joe,

thank you for sharing your code!

"How to get data further back than Jan?"

- I think that is the maximum that Spotware offers. I guess the way to get older tick data is by downloading it from some other places.

I would suggest to change these lines:

var sout = string.Join(",", sa);

to -> var sout = string.Join(";", sa);

and

private string csvhead = "datetime;ask;bid;spread;movavgX1X;volmin;volmax;pipsize;pipval\n";

-> , to ; and I deleted some spaces in this line

"," is a problem for people in countries where you use , as number seperator. Otherwise both the variables and numbers are separeted with the same symbol.

Kind regards