This is a method I use when working with slow ruby scripts that operate on huge datasets.
It caches the return value of a block of extremely slow code in a file so that subsequent runs are fast.
It's indispensable to me when doing edit-debug-edit-debug-edit cycles on giant datasets.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # This is a method I use when working with slow ruby scripts that | |
| # operate on huge datasets. | |
| # | |
| # It caches the return value of a block of extremely slow code | |
| # in a file so that subsequent runs are fast. | |
| # | |
| # It's indispensable to me when doing edit-debug-edit-debug-edit | |
| # cycles on giant datasets. | |
| # loads ruby object from a cache file or (if cache file doesn't exist) runs | |
| # provided block and writes result to cache file | |
| def cache(file) | |
| if File.exists?(file) | |
| STDERR.puts "(used cache)" | |
| Marshal.load(File.read(file)) | |
| else | |
| object = yield | |
| File.open(file, 'w') { |f| f.print Marshal.dump(object) } | |
| object | |
| end | |
| end | |
| # usage example (loading 8,000 marc-xml files) | |
| STDERR.puts "loading marc files" | |
| marc_records = cache('marc_records.cache') do | |
| marc_records = {} | |
| marc_files.each do |f| | |
| doc = Document.new(File.read(f)) | |
| title = doc.elements['//datafield[@tag="245"]/subfield[@code="a"]'] | |
| marc_records[f] = title.text | |
| end | |
| marc_records | |
| end |