Theoretical hardware recommendation for low-latency infrastructure [closed]

by

I do algorithmic trading as a hobby, and I’m about to step up my game in terms of latency.

I have two questions regarding appropriate hardware and database software.

Question 1.

Before I will purchase my hardware, I would like to know what is in general a good setup for storing many datapoints in multiple tables.

The current CPU and harddrive components I’m contemplating are roughly the following:

  • Dual CPU: Dual Intel XEON E5-2660 v2 Deca-Core 2.2GHz (3.0GHz Turbo) 25MB Cache
  • Dual Seagate 4TB SSHD
  • Kingston ECC 1600MHz (8×8 GB)

Would a double CPU + double SSHD be sufficient to be able to process many datapoints (1.000.000+) per second?

And especially, I am extremely curious to the following:

  • Is there any advantage in having more harddrives?

  • What if I split my database tables over multiple harddrives, would that increase speed?

  • Are there any other hardware (especially harddrive) setups that could increase speed when working with databases?

Question 2.

What would be the best database software I can run to store this many datapoints as fast as possible?

Also, which database software allows for simultaneous reading+writing to the same table? (so no locks)

I know of a database system called ‘MemSQL‘ which seems appropriate.
Does anybody have any experience with MemsQL? Do you know any other low-latency database software that is recommendable?

I look forward to your replies.
Thanks in advance.

Answer

In terms of disks, search for “iops calculator” in your favorite search engine to determine the configuration with the most IOPS for your risk and requirements. We have some databases on blade servers with a SAN connected with FC, and our CPU idles and RAM is used according to the requirement but disk speed is the real bottleneck. You can clearly see the databases struggling once a VM is moved to slower storage. We implement non-SSD SAS disks in RAID10 and the disks can keep up in our setup.

Some databases like to be big with few indexes while others want to be split into smaller chunks, you will have to do some research on database structures for your application.

Attribution
Source : Link , Question Author : Jean-Paul , Answer Author : Shawn Gradwell

Leave a Comment