Market Data와 HDF5

1.
HDF5에 익숙치 않았습니다. 우연히 @dolppi님의 글에서 HDF5를 알았습니다.

high frequency data analysis

대량데이타를 이용하여 백테스팅을 할 때 많이 사용하는 듯 합니다. 그렇지만 저같은 경우 백테스팅을 해본 경우이 없어 익숙치 않았습니다. 구굴링을 통하여 확인해보니 HDF는? 아리랑 1호 인공위성에서 사용하고 있습니다. HDF를 아주 자세하게 한글로 정리해놓고 있습니다.

HDF(Hierarchical Data Format)는 이러한 분산환경에서 과학용 데이터를 공유하기 위한 다중객체 파일형식이다.HDF는 여러 분야에서 진행되고 있는 프로젝트를 수행하는 다양한 과학자들의 요구를 만족시키기 위하여 NCSA(National Center for Supercomputing Applications)에서 개발되었다. 특히 HDF는 다음과 같은 과학용 데이터를기록하기 위한 많은 요구조건을 만족시키도록 만들어 졌다.

(*)과학자들에게 많이 사용되고 있는 데이터 형식과 메타데이터 형식을 지원한다.
(*)큰 데이터를 읽고, 쓰는데 있어서 효율적이어야 한다.
(*)컴퓨터의 기종에 관계없이 사용할 수 있어야 한다.
(*)다른 표준형식과 무리없이 사용할 수 있어야 하며, 앞으로 기능향상에 대비하여 확장할 수 있어야 한다.
아리랑 1호 인공위성 영상 이미지 파일 포맷중에서

우리말로 된 가장 완벽한(?) 자료이지 않을까 합니다. 다른 자료는 영어지만 HDF입문자료로 HDFGroup에서 만든 HDF5 Introduction가 있습니다.

만약 HDF5가 과학기술 연구에만 이용하는 데이타포맷이라고 하면 소개할 이유가 없습니다. 그런데 독일의 유명한 도이치은행에서 마켓데이타를 관리하는 표준포맷으로 HDF5를 이용하고 있다고 합니다. 이를 소개하는 자료입니다.

현재 HDF5와 관련된 기술 및 소프트웨어를 BSD방식의 라이센스에 따라 자유롭게 사용할 수 있습니다. HDF5는 아래와 같은 자료를 제공하고 있습니다.

(*)A versatile data model that can represent very complex data objects and a wide variety of metadata.
(*)A completely portable file format with no limit on the number or size of data objects in the collection.
(*)A?software library that runs on a range of computational platforms, from ?laptops to massively parallel systems, and implements a high-level API?with C, C++, Fortran 90, and Java interfaces.
(*)A rich set of integrated performance features that allow for access time and storage space optimizations.
(*)Tools and applications for managing, manipulating, viewing, and analyzing the data in the collection.

2.
이제 마켓데이타를 처리할 때 HDF5를 어떻게 이용할 수 있을지 좀더 상세하게 알아보도록 하겠습니다. Hack the market이라는 블로그의 Managing Tick Data With HDF5을 보죠. HDF5/Back-Testing기능을 가진 트레이딩시스템을 개발할 때 기본방향과 의문점을 아래와 같이 정리, HDF5 메일링리스트에 포스팅을 하였습니다.

A description of the data and its use

The data is all timestamped financial streams of “tick” data.? Each record is small (a few hundred bytes at the most), but there are many ? in a day you may see many hundred million to a few billion.? Each record is naturally partitioned by instrument (eg, “microsoft”, “ibm”, “dec crude”, etc).? There are less than 30K instruments in the universe I might care about.

I (more or less) don’t care how long it takes to construct the h5 files/structures as it will be performed offline and the only critical query I care about is something like:
“Get ticks for instruments {i1…in} from time t1 to time t2 ordered by time, instr”.

That is, I need to be able to “replay” a subset of the instruments within the data store over some period of time.? But I really care that this be as fast as possible.

Questions

0.? Am I barking up the wrong tree?? Is HDF5 an appropriate technology for the use I’ve described?

1. Given the size/volume of the data, my thought is to partition h5 files by day.? Uncompressed, the files will be on the order of ~25G.? Does this sound reasonable?? What are the key factors impacting this decision from an hdf5 perspective?

Two alternative models come immediately to mind: one big table (OBT) per day ordered by instrument and then time, or one table per instrument (OTPI) ordered by time.? My current inclination is OTPI as it seems more manageable assuming the overhead of so many tables isn’t an issue.

2a.? Are there other, better models you suggest I investigate?
2b.? With the OBT, I’d need to be able “index into” the table to identify the beginning of each instrument’s section (at least).? How would you recommend doing this?? It seems possible to do this with references or perhaps a separate table with numerical indices into the main table.? Any pros/cons/alternatives to these approaches?
2c.? With the OTPI, I’d need to have many tables (at most ~30K) per file.? Is this an issue?
2d. For both models, I’d need to be able to merge sorted sets of h5 data into one sorted set as quickly as possible.? Is there any hdf5 support for doing such a thing or external libraries created for this purpose?

3. What impact on retrieval/querying should I expect to see with varying levels of compression?

4. Any suggestions on chunksizes for this application?

이에 대해 Pytables의 개발자가 답변을 단 내용입니다

Questions

Re:0

In my experience, HDF5 is perfectly appropriate to cope with this. It is, as many other things, just a matter of properly directing it to do what you want to do ;-)? In particular, your volume of data seems that is going to be very large, so you should be very careful when choosing the different parameters for your application. Remember that experimenting is your best friend before putting code? into production.

Re:1
25GB is a completely reasonable figure for a single file. Moreover, by using compression, you can reduce this even more, so I see not problem on this.? However, how data is organised inside the file becomes very important for handling data efficiently (see below).

Re:2
Effectively, both OTPI and OBT has its advantages and drawbacks. OTPI, for me, has the disavantage of requiring a respectable amount of tables for your case (having 30000 datasets in a single file is nothing to sneeze at), as well as requiring a somewhat more complicated query code.? The OBT one is probably a simpler and better approach in that HDF5 has to deal with less metadata (the 30000 datasets are avoided), so it is probably faster, but you may need? additional logic in your programs to perform fast queries (indexing will help a lot indeed). I’d recommend doing your own experiments? here.

Regarding merging sorted datasets, you can always implement a typical merge sort for your needs.? Otherwise, you may want to use the sorting capabilities of PyTables Pro that can handle arbitrarily large tables.

Re:3
See:?http://www.pytables.org/docs/manual/ch05.html#searchOptim
for some experiments that I’ve recently done on this.? They are meant??for PyTables, but you could find interesting hints for your case too.

Re:4
See:http://www.pytables.org/docs/manual/ch05.html#chunksizeFineTune??for more experiments in that regard.

이상의 조언을 토대로 개발 및 시험한 내용은 Tick Datas And HDF5(part2)에 참조하시길 바랍니다.