FIX Trading Community

 

 Remember me

Register  |   Lost password?

FAST Protocol

Discussions > FAST Protocol > OPRA is not FAST

OPRA is not FAST

Complete message thread from old site

Jim Northey
1993 days ago,(2008/10/31)

OPRA is not a FAST compliant market data feed - it is FAST-like but is not compliant with the FAST Specification. What are other vendors doing to address the OPRA Market data feed - which should not be confused with being a FAST market data feed?
We are a bit concerned to alter or break our standard implementations to support OPRA. Seems like an alternative product or code branch might be preferred to corrupting the main code line of a FAST compliant product.

Daniel May
1993 days ago,(2008/10/31)

Jim,
This has been a long standing issue with OPRA that seems to have fallen on deaf ears. It all started when OPRA simply used the proof on concept code that SpryWare and Pantor wrote to demonstrate the concept of FAST and released it as a production feed. They never consulted the group to review their implementation, in fact they distributed decoder source coded that still contained the original proof of concept cvs version control headers.

The proof of concept code was not FAST compliant, because FAST was not even formalized at that point. I do believe they consulted Jordan and Jordan before releasing their version 2 of OPRA FAST, "OPRA FAST for Symbology", about correcting the non compliant portions of the code. For some reason they chose not to bother.

So you are left with a feed that cannot be decoded with a generic FAST decoder. The changes are minor, simply adding a few more templates rather than having messages templates dynamically changed based on message content.

/Daniel

> OPRA is not a FAST compliant market data feed - it is FAST-like but is
> not compliant with the FAST Specification. What are other vendors doing
> to address the OPRA Market data feed - which should not be confused with
> being a FAST market data feed? We are a bit concerned to alter or break
> our standard implementations to support OPRA. Seems like an alternative
> product or code branch might be preferred to corrupting the main code
> line of a FAST compliant product.

Rolf Andersson
1993 days ago,(2008/10/31)

It was mind over matter.

(They don't mind because we obviously don't matter to them)

> Jim, This has been a long standing issue with OPRA that seems to have
> fallen on deaf ears. It all started when OPRA simply used the proof on
> concept code that SpryWare and Pantor wrote to demonstrate the concept
> of FAST and released it as a production feed. They never consulted the
> group to review their implementation, in fact they distributed decoder
> source coded that still contained the original proof of concept cvs
> version control headers.
>
> The proof of concept code was not FAST compliant, because FAST was not
> even formalized at that point. I do believe they consulted Jordan and
> Jordan before releasing their version 2 of OPRA FAST, "OPRA FAST for
> Symbology", about correcting the non compliant portions of the code. For
> some reason they chose not to bother.
>
> So you are left with a feed that cannot be decoded with a generic FAST
> decoder. The changes are minor, simply adding a few more templates
> rather than having messages templates dynamically changed based on
> message content.
>
> /Daniel
>
>
> > OPRA is not a FAST compliant market data feed - it is FAST-like but is
> > not compliant with the FAST Specification. What are other vendors
> > doing to address the OPRA Market data feed - which should not be
> > confused with being a FAST market data feed? We are a bit concerned to
> > alter or break our standard implementations to support OPRA. Seems
> > like an alternative product or code branch might be preferred to
> > corrupting the main code line of a FAST compliant product.

FIX Trading Community
1993 days ago,(2008/10/31)

[ original email was from Rich Shriver - shriver@jandj.com ]
FPL and members of MDOWG provided feedback to SIAC that the OPRA FAST implementation was non-compliant with the FAST specifications. The feedback has yet to be incorporated or addressed. What has not occured is pressure from OPRA direct feed customers to make the technology compliant with the standards. As a standards organization, we have also not effectively lobbied OPRA and their technology service provider to comply with the standards as developed. I am not saying we have not lobbied, I am simply stating we have yet to be successful.

Rich Shriver

> Jim, This has been a long standing issue with OPRA that seems to have
> fallen on deaf ears. It all started when OPRA simply used the proof on
> concept code that SpryWare and Pantor wrote to demonstrate the concept
> of FAST and released it as a production feed. They never consulted the
> group to review their implementation, in fact they distributed decoder
> source coded that still contained the original proof of concept cvs
> version control headers.
>
> The proof of concept code was not FAST compliant, because FAST was not
> even formalized at that point. I do believe they consulted Jordan and
> Jordan before releasing their version 2 of OPRA FAST, "OPRA FAST for
> Symbology", about correcting the non compliant portions of the code. For
> some reason they chose not to bother.
>
> So you are left with a feed that cannot be decoded with a generic FAST
> decoder. The changes are minor, simply adding a few more templates
> rather than having messages templates dynamically changed based on
> message content.
>
> /Daniel
>
>
> > OPRA is not a FAST compliant market data feed - it is FAST-like but is
> > not compliant with the FAST Specification. What are other vendors
> > doing to address the OPRA Market data feed - which should not be
> > confused with being a FAST market data feed? We are a bit concerned to
> > alter or break our standard implementations to support OPRA. Seems
> > like an alternative product or code branch might be preferred to
> > corrupting the main code line of a FAST compliant product.

Oleg Stetsyuk
1988 days ago,(2008/11/05)

> Jim, This has been a long standing issue with OPRA that seems to have
> fallen on deaf ears. It all started when OPRA simply used the proof on
> concept code that SpryWare and Pantor wrote to demonstrate the concept
> of FAST and released it as a production feed. They never consulted the
> group to review their implementation, in fact they distributed decoder
> source coded that still contained the original proof of concept cvs
> version control headers.
>
> The proof of concept code was not FAST compliant, because FAST was not
> even formalized at that point. I do believe they consulted Jordan and
> Jordan before releasing their version 2 of OPRA FAST, "OPRA FAST for
> Symbology", about correcting the non compliant portions of the code. For
> some reason they chose not to bother.
>
> So you are left with a feed that cannot be decoded with a generic FAST
> decoder. The changes are minor, simply adding a few more templates
> rather than having messages templates dynamically changed based on
> message content.
>
> /Daniel
>
>
> > OPRA is not a FAST compliant market data feed - it is FAST-like but is
> > not compliant with the FAST Specification. What are other vendors
> > doing to address the OPRA Market data feed - which should not be
> > confused with being a FAST market data feed? We are a bit concerned to
> > alter or break our standard implementations to support OPRA. Seems
> > like an alternative product or code branch might be preferred to
> > corrupting the main code line of a FAST compliant product.

Hello,

Could you explain please where OPRA is not compliant?
Is it about the Template ID?
It would be fine to have some facts.

Thank you,
Oleg

Rolf Andersson
1988 days ago,(2008/11/05)

Please check with your colleagues. Both Dan, I, and J&J have responded to questions and provided advice on how to make the OPRA FAST-alike feed compliant with the spec.

What's your role at NYSE/SIAC?

Best,
Rolf

> Hello,
>
> Could you explain please where OPRA is not compliant? Is it about the
> Template ID? It would be fine to have some facts.
>
> Thank you, Oleg

Daniel May
1986 days ago,(2008/11/07)

Oleg,
As far as I can recall, there are several points where the feed is not FAST compliant.

1. It includes a proprietary header and I believe a footer on each message. This would not be the end of the world if the payload was actually compliant.

2. The biggest issue is the Quote template. The quote template definition is not constant, it is defined after you process the BBO appendage fid, and based on the value, you can then determine the number of fields that follow. This could be solved by having a template ID for each possible BBO appendage.

This one was my own creation in the early FAST proof of concept days, that is how I coded it to show haw FAST could work on Opra data. It made sense at the time, but that was before FAST was formalized.

Look at the supplied fast_decode.c from Opra, and search for:

//decode bbo indicator
msg->bboIndicator = fast->decode_u32(fast, BBO_INDICATOR);

switch(msg->bboIndicator)
{

This explains it all (if you read 'C' code). The value of BBO_INDICATOR determines the number and type of fields that follow, this is not FAST compliant.

I do believe Rolf and some other members of the group are performing a more complete survey of the Opra implementation.

/Daniel

Shwetang Shah
1986 days ago,(2008/11/07)

> Oleg, As far as I can recall, there are several points where the feed is
> not FAST compliant.
>
> 1. It includes a proprietary header and I believe a footer on each
> message. This would not be the end of the world if the payload was
> actually compliant.
>
> 2. The biggest issue is the Quote template. The quote template
> definition is not constant, it is defined after you process the BBO
> appendage fid, and based on the value, you can then determine the
> number of fields that follow. This could be solved by having a
> template ID for each possible BBO appendage.
>
> This one was my own creation in the early FAST proof of concept days,
> that is how I coded it to show haw FAST could work on Opra data. It made
> sense at the time, but that was before FAST was formalized.
>
>
> Look at the supplied fast_decode.c from Opra, and search for:
>
> //decode bbo indicator msg->bboIndicator = fast->decode_u32(fast,
> BBO_INDICATOR);
>
> switch(msg->bboIndicator) {
>
> This explains it all (if you read 'C' code). The value of BBO_INDICATOR
> determines the number and type of fields that follow, this is not FAST
> compliant.
>
> I do believe Rolf and some other members of the group are performing a
> more complete survey of the Opra implementation.
>
> /Daniel

For the point # 1.
As far as I understood the specification of FAST1.1, proprietary header and trailer is completely independent of the compliance. This is application specific and does not play any role for the FAST (like UDP packet header has nothing to do with FAST). More over the greatest advantage of the packet header is for the network group who wants to check the sequence of the messages without decoding the FAST messages. This header give flexibility and information about the sequence number of the 1st message and the number of messages/packet, size of the message etc. This information will be vary valuable for the tools like Sniffer where without integrating FAST protocol to tools network engineer can analyze the gaps in the packets. Actually FAST working group should consider this idea for the future.

For the point # 2.
OPRA consist of generalize template for all the messages. FAST engine can (must) run without any business knowledge of the message and strictly based on the information received from the wire. So if the FAST 1.1 compliant decoder is placed with these messages, it should work. Later application needs to build the state engine to constitute the message based on the information received from the FAST engine(decoder/template).

Sample decoder is given as an example and to reconstitute the original message to ASCII. State machine is being used to decode the original message.

Now going back to the original comments about the compliance. OPRA uses only two different type of operators.
1. Copy : There is no change from FAST 1.0 to FAT 1.1 spec. So this operator for 1.0 is comply with 1.1
2. increment : There is no change from FAST 1.0 to FAT 1.1 spec. So this operator for 1.0 is comply with 1.1

So if the reference for adding more template IDs for bbo indicator then it is incorrect as per the spec as it will make FAST engine from generalize engine to OPRA specific.

I am still looking in the section 10.0 of FAST 1.1 where there are some ambiguity which needs to be resolve. For example, the xml statement doesn’t mandate to have the templateIdentifier as the 1st bit of the presence map.
message ::= segment
segment ::= PresenceMap TemplateIdentifier? (field | segment)*

and the other statement where TID is optional as the 1st bit.:

“A segment has a header consisting of a Presence Map followed by an optional Template Identifier. The segment has a template identifier either if it is a message segment, or if the segment appears as the result of a dynamic template reference instruction. A template identifier is encoded as if a copy operator was specified. The operator uses the global dictionary and has an internal key common to all template identifier fields. This means that a segment with a template identifier does not always contain the template identifier physically. However, the first bit in the presence map is allocated by its copy operator. “

Please let me know if I am misinterpreting the spec.

-Shwetang

Rolf Andersson
1986 days ago,(2008/11/08)

Shwetang -

SIAC's new-found interest in FAST compliance is very welcome.

Just to make sure that I'm using the right OPRA spec:

Is http://www.opradata.com/specs/fast_for_oprav2_00.pdf
the current version of the OPRA spec?

/Rolf

Daniel May
1986 days ago,(2008/11/08)

Shwetang,
While others have used a proprietary header and trailer, there are ways to accomplish what you are trying to do with a fully FAST compliant message. Greg Maynard (gmaynard@ise.com) of the ISE brought this forward a few months ago, and we came up with a "best practice" for defining a frame header with exactly the information you are trying to convey.

The idea is to use a FAST Byte Vector to carry a network byte order sequence number at the beginning of each frame. This will give a sniffer a predictable, easy to find header and sequence number in an easy to read format. The rough draft of the message looked like this:

C0 F8 = pmap + Fast reset message
F0 = pmap (TID+3 bits)
F7 = TID 119,
84 = Byte Vector size=4 (with high bit set)
00 00 00 01 = data=1 (frame seqno, uint32_t range 0 -4,294,967,295)
82 = byte vector size=2 (with high bit set)
03 E6 = data=998 (frame length, uint16_t range 0 - 65,535)
84 = byte Vector size=4 (with high bit set)
02 E2 D3 88 = byte vector data=48419720 (frame sending time, uint32_t, milliseconds since epoch)

So what you have is a standard frame header that contains a Reset Message, and using template ID 119, 3 byte vectors containing a frame sequence number, frame length, and sending time. Obviously, some of these fields are optional, but useful. You could always add things like messages per frame, etc. The fact that the header is fixed in size means you can efficiently populate it *after* frame construction, and move the values into the output buffer just prior to writing to the network. You can also discard a frame during arbitration very quickly, or skip to the next frame in a continuous stream of data.

I will contact Greg to see if we can get the exact "Block Header" that the ISE has chosen, along with the template definition so we can start effectively using it as a "best practice".

I will let Rolf follow up on item #2

/Daniel

>
>
> For the point # 1. As far as I understood the specification of FAST1.1,
> proprietary header and trailer is completely independent of the
> compliance. This is application specific and does not play any role for
> the FAST (like UDP packet header has nothing to do with FAST). More over
> the greatest advantage of the packet header is for the network group who
> wants to check the sequence of the messages without decoding the FAST
> messages. This header give flexibility and information about the
> sequence number of the 1st message and the number of messages/packet,
> size of the message etc. This information will be vary valuable for the
> tools like Sniffer where without integrating FAST protocol to tools
> network engineer can analyze the gaps in the packets. Actually FAST
> working group should consider this idea for the future.
>
> For the point # 2. OPRA consist of generalize template for all the
> messages. FAST engine can (must) run without any business knowledge of
> the message and strictly based on the information received from the
> wire. So if the FAST 1.1 compliant decoder is placed with these
> messages, it should work. Later application needs to build the state
> engine to constitute the message based on the information received from
> the FAST engine(decoder/template).
>
> Sample decoder is given as an example and to reconstitute the
> original message to ASCII. State machine is being used to decode the
> original message.
>
> Now going back to the original comments about the compliance. OPRA uses
> only two different type of operators.
> 1. Copy : There is no change from FAST 1.0 to FAT 1.1 spec. So this
> operator for 1.0 is comply with 1.1
> 2. increment : There is no change from FAST 1.0 to FAT 1.1 spec. So this
> operator for 1.0 is comply with 1.1
>
> So if the reference for adding more template IDs for bbo indicator then
> it is incorrect as per the spec as it will make FAST engine from
> generalize engine to OPRA specific.
>
>
> I am still looking in the section 10.0 of FAST 1.1 where there are some
> ambiguity which needs to be resolve. For example, the xml statement
> doesn’t mandate to have the templateIdentifier as the 1st bit of the
> presence map. message ::= segment segment ::= PresenceMap
> TemplateIdentifier? (field | segment)*
>
> and the other statement where TID is optional as the 1st bit.:
>
> “A segment has a header consisting of a Presence Map followed by an
> optional Template Identifier. The segment has a template identifier
> either if it is a message segment, or if the segment appears as the
> result of a dynamic template reference instruction. A template
> identifier is encoded as if a copy operator was specified. The operator
> uses the global dictionary and has an internal key common to all
> template identifier fields. This means that a segment with a template
> identifier does not always contain the template identifier physically.
> However, the first bit in the presence map is allocated by its copy
> operator. “
>
> Please let me know if I am misinterpreting the spec.
>
> -Shwetang

Shwetang Shah
1983 days ago,(2008/11/10)

Rolf,
Yes this is the correct URL for symbology. Please refer to page 8 for generalize template for all messages.

Individual templates are given for those vendors who were using individual state machine for separate messages.

Thanks,
Shwetang

> Shwetang -
>
> SIAC's new-found interest in FAST compliance is very welcome.
>
> Just to make sure that I'm using the right OPRA spec:
>
> Is http://www.opradata.com/specs/fast_for_oprav2_00.pdf the current
> version of the OPRA spec?
>
> /Rolf

Rolf Andersson
1983 days ago,(2008/11/10)

Thx, I'll base my comments on that doc then.
I'm not sure I understand your comment about individual templates?

Best,
Rolf

> Rolf, Yes this is the correct URL for symbology. Please refer
> to page 8 for generalize template for all messages.
>
> Individual templates are given for those vendors who were using
> individual state machine for separate messages.
>
> Thanks, Shwetang

Rolf Andersson
1977 days ago,(2008/11/17)

Shwetang,

please elaborate on the "Individual templates" that you refer to below. I wasn't aware of any vendor specific templates?

Best,
Rolf

> Rolf, Yes this is the correct URL for symbology. Please refer to page 8
> for generalize template for all messages.
>
> Individual templates are given for those vendors who were using
> individual state machine for separate messages.
>
> Thanks, Shwetang
>
> > Shwetang -
> >
> > SIAC's new-found interest in FAST compliance is very welcome.
> >
> > Just to make sure that I'm using the right OPRA spec:
> >
> > Is http://www.opradata.com/specs/fast_for_oprav2_00.pdf the current
> > version of the OPRA spec?
> >
> > /Rolf

Shwetang Shah
1973 days ago,(2008/11/20)

Rolf,
Individual templates are give to help at the application layer for stat machine not at the FAST decoding.

OPRA uses only one generic template. Point 11 On page 16 has explanation in the http://opradata.com/specs/fast_for_oprav2_00.pdf document.

Thanks,
Shwetang

> Shwetang,
>
> please elaborate on the "Individual templates" that you refer to below.
> I wasn't aware of any vendor specific templates?
>
> Best, Rolf
>
> > Rolf, Yes this is the correct URL for symbology. Please refer to page
> > 8 for generalize template for all messages.
> >
> > Individual templates are given for those vendors who were using
> > individual state machine for separate messages.
> >
> > Thanks, Shwetang
> >
> > > Shwetang -
> > >
> > > SIAC's new-found interest in FAST compliance is very welcome.
> > >
> > > Just to make sure that I'm using the right OPRA spec:
> > >
> > > Is http://www.opradata.com/specs/fast_for_oprav2_00.pdf the current
> > > version of the OPRA spec?
> > >
> > > /Rolf

Rolf Andersson
1973 days ago,(2008/11/20)

OK, so those templates are not FAST templates but some other kind of templates specific to you impl?

/Rolf

> Rolf, Individual templates are give to help at the application
> layer for stat machine not at the FAST decoding.
>
> OPRA uses only one generic template. Point 11 On page 16 has
> explanation in the http://opradata.com/specs/fast_for_oprav2_00.pdf
> document.
>
> Thanks, Shwetang

Shwetang Shah
1973 days ago,(2008/11/20)

Rolf,
yes, Specifically the state machine given in our decoder.

Thanks,
Shwetang

> OK, so those templates are not FAST templates but some other kind of
> templates specific to you impl?
>
> /Rolf
>
> > Rolf, Individual templates are give to help at the application layer
> > for stat machine not at the FAST decoding.
> >
> > OPRA uses only one generic template. Point 11 On page 16 has
> > explanation in the http://opradata.com/specs/fast_for_oprav2_00.pdf
> > document.
> >
> > Thanks, Shwetang

Rolf Andersson
1973 days ago,(2008/11/20)

Hm, I still don't get what you are referring to. I may be dense, but in terms of "individual templates", who gets what and why?

/Rolf

> Rolf, yes, Specifically the state machine given in our decoder.
>
> Thanks, Shwetang
>
> > OK, so those templates are not FAST templates but some other kind of
> > templates specific to you impl?
> >
> > /Rolf
> >
> > > Rolf, Individual templates are give to help at the application layer
> > > for stat machine not at the FAST decoding.
> > >
> > > OPRA uses only one generic template. Point 11 On page 16 has
> > > explanation in the http://opradata.com/specs/fast_for_oprav2_00.pdf
> > > document.
> > >
> > > Thanks, Shwetang

Shwetang Shah
1973 days ago,(2008/11/20)

OK. Ignore these individual templates for your evaluation. Just refer to general template on page 8.

Thanks,
Shwetang

> Hm, I still don't get what you are referring to. I may be dense, but in
> terms of "individual templates", who gets what and why?
>
> /Rolf
>
> > Rolf, yes, Specifically the state machine given in our decoder.
> >
> > Thanks, Shwetang
> >
> > > OK, so those templates are not FAST templates but some other kind of
> > > templates specific to you impl?
> > >
> > > /Rolf
> > >
> > > > Rolf, Individual templates are give to help at the application
> > > > layer for stat machine not at the FAST decoding.
> > > >
> > > > OPRA uses only one generic template. Point 11 On page 16 has
> > > > explanation in the
> > > > http://opradata.com/specs/fast_for_oprav2_00.pdf document.
> > > >
> > > > Thanks, Shwetang

Rolf Andersson
1973 days ago,(2008/11/20)

please?

> OK. Ignore these individual templates for your evaluation. Just refer to
> general template on page 8.
>
> Thanks, Shwetang
>
> > Hm, I still don't get what you are referring to. I may be dense, but
> > in terms of "individual templates", who gets what and why?
> >
> > /Rolf
> >
> > > Rolf, yes, Specifically the state machine given in our decoder.
> > >
> > > Thanks, Shwetang
> > >
> > > > OK, so those templates are not FAST templates but some other kind
> > > > of templates specific to you impl?
> > > >
> > > > /Rolf
> > > >
> > > > > Rolf, Individual templates are give to help at the application
> > > > > layer for stat machine not at the FAST decoding.
> > > > >
> > > > > OPRA uses only one generic template. Point 11 On page 16 has
> > > > > explanation in the
> > > > > http://opradata.com/specs/fast_for_oprav2_00.pdf document.
> > > > >
> > > > > Thanks, Shwetang

Rolf Andersson
1965 days ago,(2008/11/28)

I've had a look at the current spec (FAST for OPRA version 02.00 dated January 23, 2008) and I have the following observations:

The Template specification is not compliant with FAST 1.1 for the following reasons:
- Fields are not in pmap sequence
Some pmap slots are unused for each of the specified messages.

- The presence of some fields depend on the value of another field.
The presence of best bid/ask fields depend on the BBO indicator field

- A proprietary framing protocol is used.
To add insult to injury, the FAST standard specifies an alternative that provides the same function.

In my view, the use of one "generic" template creates much of the compatibility problem. Careful use of a few templates would have made it possible to efficiently use FAST-api 1.0 in a FAST 1.1 compliant manner.

I found a way to specify a single FAST template that can be used by a compliant FAST 1.1 decoder to process the individual messages after removing the proprietary framing. The resulting output however contains superfluous incorrect content and additional processing is required to remove the incorrect data generated by the decoder.

In summary: While not _impossible_ to use together with a proprietary de-framer, the generic OPRA template prevents recipients from efficiently using a standard FAST 1.1 implementation and requires a custom built decoder in order to get decent performance.

/Rolf

(on a side-note: The FAST for OPRA specification needs to be reviewed and improved. It is incomplete, ambiguous and contains a number of errors. It doesn't differentiate clearly between FAST 1.0 and 1.1. It mixes snippets of text from the FAST 1.0 and 1.1 specs. The text is partly presented out of context. Some information is not relevant to the current implementation of FAST for OPRA (eg. deltas are not used, but the spec contains a partly erroneous description of delta coding))

> OK. Ignore these individual templates for your evaluation.
> Just refer to general template on page 8.
>
> Thanks, Shwetang