A fast implementation of MD5
Resources for CL
·
General Information
·
CL Implementations
·
PMSF Resources
·
Building CMUCL
·
MaiSQL
·
Fast MD5
·
Deflate
·
XML Parsing
·
SillyIRC
·
CLMDImporter
Prompted by discussions on the cmucl-help mailing list over an efficient implementation of MD5 message-digests for CMU CL, I tuned a silly implementation of MD5 I had done a while ago, with much help and input from other developers and users on the cmucl-help mailing list, to a level where it is competitive with the standard md5sum utility for large files. It performed within a factor of 1.15-1.5 of the md5sum utility on 40-60 MB files both on UltraSPARC and AMD K6-2-550 hardware, using current (this was before the 18d release) binaries of CMU CL.
Important
Versions of md5.lisp downloaded from this site prior to 22-03-2003 did contain a bug in the handling of padding which caused wrong message-digests being calculated for data whose length was exactly 56 + X*64 bytes. Version 1.8 of the file (see the comments at the start of the file) fixes this problem.
The result of the effort can be obtained here. Quoting from the file:
;;;; This file implements The MD5 Message-Digest Algorithm, as defined in ;;;; RFC 1321 by R. Rivest, published April 1992. ;;;; ;;;; It was written by Pierre R. Mai, with copious input from the ;;;; cmucl-help mailing-list hosted at cons.org, in November 2001 and ;;;; has been placed into the public domain. ;;;; ;;;; $Id: md5.lisp,v 1.8 2003/01/25 00:18:27 dent Exp $ ;;;; ;;;; While the implementation should work on all conforming Common ;;;; Lisp implementations, it has only been optimized for CMU CL, ;;;; where it achieved comparable performance to the standard md5sum ;;;; utility (within a factor of 1.5 or less on iA32 and UltraSparc ;;;; hardware). ;;;; ;;;; Since the implementation makes heavy use of arithmetic on ;;;; (unsigned-byte 32) numbers, acceptable performance is likely only ;;;; on CL implementations that support unboxed arithmetic on such ;;;; numbers in some form. For other CL implementations a 16bit ;;;; implementation of MD5 is probably more suitable. ;;;; ;;;; The code implements correct operation for files of unbounded size ;;;; as is, at the cost of having to do a single generic integer ;;;; addition for each call to update-md5-state. If you call ;;;; update-md5-state frequently with little data, this can pose a ;;;; performance problem. If you can live with a size restriction of ;;;; 512 MB, then you can enable fast fixnum arithmetic by putting ;;;; :md5-small-length onto *features* prior to compiling this file. ;;;; ;;;; Testing code can be compiled by including :md5-testing on ;;;; *features* prior to compilation. In that case evaluating ;;;; (md5::test-rfc1321) will run all the test-cases present in ;;;; Appendix A.5 of RFC 1321 and report on the results. ;;;; Evaluating (md5::test-other) will run further test-cases ;;;; gathered by the author to cover regressions, etc. ;;;; ;;;; This software is "as is", and has no warranty of any kind. The ;;;; authors assume no responsibility for the consequences of any use ;;;; of this software.