Altering to use functions in util library so as to support HDFS properly. There may also be some lurking code to support parallel processing and manifests (I did copy this from parallel processing project first)