The Filter function use during Backtesting

This topic contains 4 replies, has 2 voices, and was last updated by Seer Seer 5 years, 4 months ago.

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • #614
    Seer
    Seer
    Keymaster

    The filter has two forms of syntax:

    my $set = Filter { [code] };
    my $set = Filter { [code] } $set2;

    In the first example, the code within the braces ({}) is executed against all instruments in the current systems portfolio. In the second example, the code within the braces ({}) is executed against all the instruments in the passed in set ($set2).

    When the Filter runs a block of code against a specific instrument it is ran in the context of that instrument. Consider a system with 3 instruments in the portfolio:

     

    my $set = Filter { ClosePrice };

    When the Filter executes, the code contained in the filter block will be executed for each instrument as if that instrument was running the block of code. In the above case, the ClosePrice funtion will get called three times, each with a different instrument as the running context.

    When the block of code is executed for each instrument a value needs to be returned as this value will be used in the creation of the set. The returned value can be explicitly returned via the return function, or implicitly returned by the last function executed. If a column is returned (such as from an Indicator) the value from the current bar is extracted automatically.

     

    my $set = Filter { RSI(14) };
    my $set = Filter { return RSI(14) };

    The above Filters do the same thing.

    Consider the following example:

    A systems portfolio contains the following equity instruments, MSFT, INTC, GE and KO. In the bar event of the system is the following code:

     

    my $set = Filter { RSI(14) };

    The filter will execute:

     

    RSI(14)

    For the instruments MSFT, INTC, GE and KO (in that order). The RSI indicator returns a column and the value for the current bar will be extracted to created the Set. The resulting set will look like:

    MSFT 62.08
    INTC 57.03
    GE 69.56
    KO 60.19

    When running Filters you'll often want to exclude instruments based upon various logical conditions. Imagine for example you only want to create a set where the current RSI is above 60:

     

    my $set = Filter {
      if (Now(RSI(14))>60) {
        return 1;
       }
      else {
        return undef;
      }
    };

    When the above code is executed, the resulting set will look like:

     

    MSFT 62.08
    GE 69.56
    KO 60.19

    INTC has been excluded as it's RSI is not above 60. It has been excluded by returning undef for that instrument. undef is a special function, it's interpritation can be defined as undefined or unknown. When returned, that instrument is excluded from the resulting set. Consider the following filter:

     

    my $set = Filter { Position };

    As the function Position returns 1 (true) when we hold a position, and undef when we don't hold a position, the above Set produced from the Filter contains all the instruments that we hold a position for.

     

    Set Theory and Operations on Sets

    When operating on Sets Seer supports three basic operations, Union, Intersection and Difference (or complement). In all these case Seer creates a new Set.

    Union

    A union of two sets simply adding two sets together. The operator to do this is the + (addition symbol).

    A union operation between two sets, each containing a portolio of stocks futures or forex instruments

    Intersection

    The intersection of two sets is the instruments that are common in both sets. The operator to do this is the * (multiplication symbol).

    A intersection operation between two sets, with each set containing a portfolio of stocks, futures or forex instruments

    Difference (complement)

    The difference between two sets is the instruments that exist in one set, but not in the other. The operator to do this is the - (minus symbol)

    A difference operation between two sets, with each set containing a portfolio of stocks, futures or forex instruments

    Ranking and Sorting

    Two forms of ranking are supported for Sets, positional and percentile. For sorting there are two functions, SortSet and SortSetAsc.

    Positional Ranking

    To get the top or bottom positions from a set you need to use the positional ranking operators > (greater than symbol) and < (less than symbol) with an ordinal position. For example, the two following statements will create a new set with the top ranked and the bottom ranked instruments from an existing set:

    my $ranked = $set > 1;  #creates a 1 member set with the instrument with the top value
    my $ranked = $set < 1;  #creates a 1 member set with the instrument with the bottom value

    While the following 2 statements will return the top 5 ranked instruments and the bottom 5 ranked instruments:

     

    my $ranked = $set > 5;  #creates a 5 member set with the top 5 ranked instruments
    my $ranked = $set < 1;  #creates a 5 member set with the bottom 5 ranked instruments

    Imagine you wanted to go long on the 3 instruments with the largest percentage gain on a specific bar:

     

    #produce a set with the percentage changes based on the closing price
    my $set = Filter { Percent(Close) };
    #produce a set containing the 3 largest gainers
    my $buyset = $set > 3;
    #go long on the 3 largest gainers
    Filter {BuyOpen } $buyset;

    Or more succinctly:

     

    Filter {BuyOpen } Filter {Percent(Close)} > 3;

    Percentile Ranking

    Percentile ranking works in the same way as positional ranking, but rather than using the > and < operators percentile ranking uses the >> and << operators. For example, the two following statements will create a new set with the top 1 percentile and the bottom 1 percentile instruments from an existing set:

    my $percentile = $set >> 1;  #creates an x member set with the top 1 percentile
    my $percentile = $set << 1;  #creates an x member set with the bottom 1 percentile

    Sorting

    There are two functions that allow you to sort a set, SortSet and SortSetAsc. The Sort function sorts from the highest value to the smallest (descending sort), while the SortSetAsc functions sorts from the smallest to the highest (ascending sort).

    When a set is created as a result of a rank function it has been implicitly sorted and it doesn't need to be sorted again.

    One of the main uses of Sorted set to to change the order of execution of a Filter. For example, consider this filter:

     

    Filter {BuyOpen;1; }

    This filter will call the BuyOpen money management for each instrument in the system portfolio in the order that it was added to the portfolio. Now consider this Filter:

     

    my $set = Filter { RSI(14) };
    Filter {BuyOpen;1; } SortSet ($set);

    We are now passing a sorted set to the filter so rather than execute the filter in the order that the instruments where added to the portfolio they are now executed in the sorted order in the set.

    Using the FirstNewBar

    In many cases you'll only want to run the filter once per bar rather than per symbol per bar. To do this you can use the FirstNewBar function. This function returns true if the current symbol/bar combination is the first new bar. Typical usage:

    if (FirstNewBar) {
      my $set = Filter {
        ADX(14);
      }
    }

    Using Variables within Filters

    The code block that is contained within a Filter can be complex and it can use variables that are scoped out side that block. This can include my scoped variables as well as other variables declared in symbol or system scope. The following example is valid:

     

    my $price = ClosePrice;
    my $set = Filter {
       return 1 if $price == ClosePrice;
       return undef;
    };

     

    Putting it all together

    Have a look at the example OnlyTheBest. This system is goes long for the 5 "best" symbols in the portfolio. "Best" is defined by top 5 trending symbols using the ADX indicator. When symbols drop out of the top 5 they are sold.

    Advanced Features

    For most typical uses you will not need to use the advanced features.

    Creating a Set Manually

    You can create a set by using the Set function. This function can except a string, a reference to an array or an reference to an hash.

    Examples:

    my $set = Set('MSFT');
    my $set = Set(['MSFT','INTC']);
    my $set = Set({'MSFT' => 12, 'INTC' => 16});

    For strings and arrays the value associated with the instrument is zero. If you need to set the value for each instrument use the hash.

    Getting the number of Instruments in the Set

    To get the number of instruments in the set, use the following syntax:

     

    my $set = Filter { ADX(14) }; #create a set
    my $count = $$set

    Accessing the raw values of a Set

    Sometimes you'll want to Access the raw values of a set. One typical example is for debugging. There are two approaches, accessing the Set as if was an array and as if was a hash.

     

    Accessing as an Array

    When accessing a set as an array all the instruments in the set are returned. If the set has been sorted then the instruments are returned in that sorted order.

     

    my $set = Filter {ClosePrice;};
    my @symbols = @$set;
    foreach my $symbol (@symbols) {
      Output $symbol;
    }

     

    Accessing as an Hash

    When accessing set as an hash, both the value and instrument is returned.

     

    my $set = Filter {ClosePrice;};
    my %symbols = %$set;
    foreach my $ticker (keys %symbols) {
      Output "$ticker = $symbols{$ticker}";
    }

    Limitations of using Filters and Sets

    • Filters can't be used in Account begin and end events.
    • When creating manual sets or using manual data in the manipulation of sets only instruments that exists in the system portfolio can be used.
    • Filters can't be nested (a filter within a filter)
    • When there are data inconsistencies (missing bars) between instruments that are used within a Filter, the instrument with the missing bar will not be run. In most cases this is the correct thing to do, but for some others it may not be what is expected. A warning is given in the output.
    #4507

    I’m getting the error “$set must be declared on line 19″ on this sample code. What is wrong with the scope used for $set? I’m running this code in Bar Event.

    my $LONG = 1;
    my $SHORT = -1;
    
    if( FirstNewBar) {
     my $set = Filter 
     {
     my $sma = Now( SMA( Close, $MALookback) );
    
     return $LONG if( Now( Close ) < $sma );
       
     return $SHORT if( Now(Close) > $sma);
    
     return undef;
     }
    
    #all signals have been evaluated for the portfolio. Decide which ones to take based on the total signals
    Output($set);
    
    }
    
    #4509
    Seer
    Seer
    Keymaster

    You are missing a ; on one of the lines (see the arrow below):

     my $set = Filter 
     {
     my $sma = Now( SMA( Close, $MALookback) );
     return $LONG if( Now( Close ) < $sma );
     return $SHORT if( Now(Close) > $sma);
     return undef;
     }; <--------------
    #4510

    How do I set the scope of $set to live beyond one bar? I declared it as a System variable, but $set isn’t accessible when I use this code.

    Can’t use string (“1″) as a HASH ref while “strict refs” in use at line 36.

    #$set is declared as system variable
    if( FirstNewBar )
    {
    $set = Filter{ RSI(14) }
    }
    
    #this line throws exception "Can't use string ("1") as a HASH ref while "strict refs" in use"
    my %dictionary = %$set;

    But running the filter on every bar solves the problem, even though it’s terribly wasteful.

    #$set is declared as system variable
    $set = Filter{ RSI(14) }

    #runs ok
    my %dictionary = %$set;

    #4512
    Seer
    Seer
    Keymaster

    Hi,

    Your approach is correct, it just needs some tweaking:)

    When you define a variable in the variable object, it’s scope is set to symbol with the default value being 1. What this means is that when you backtest, each symbol has it’s own ‘copy’ of the variable, so if you defined a variable called max:

    $max = ClosePrice if ClosePrice> $max;

    Then each symbol in the portfolio would record it’s own maximum price in the $max variable.

    You can change the scoping of the variable to be system. When you do this all symbols in the portfolio share the same variable. Once you have changed the scoping of the variable, your system should work as expected (and be faster in backtesting).

Viewing 5 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.