Re: Optimizing basic operations
- Posted by James_at_Euphoria Aug 03, 2013
- 5825 views
Hi James,
I don't want to rain on your parade but I have to ask this question: What are you working on? ( If it isn't classified.)
I've been watching this thread for a while and the main question I have is where is all this data coming from? Depending on the speed of your storage device a trillion records is going to take a very long time just for the input.
Still, let's say you can input & process 1 million records per second. (A very high estimate)
1,000,000,000,000 records / 1,000,000 records / second = 1,000,000 seconds / 86400 seconds / day = 11+ days.
Unless you can break up the data and have multiple computers working at the same time on parts of the data set and then merge them at the end I don't see how it can be done.
Lonny
Thanks, Lonny, for the reality check. I'm concerned about this as well. I guess if I'm in a hurry I might have to rent a machine that can do up to 240 teraflops. Then I might be able to turn 11+ days into an hour or so. See:
I can also change the threshold (sensitivity) of the data and reduce the number of records. Additionally, I'm also looking at heuristic approaches to prune the search tree. However, these approaches risk skipping over the best solution(s).
