Date: 20180804 (Aug 4th 2018)
While playing with assembler speedups, I discovered that the issue is reproducible on the (currently) latest version: v1.2.11 ([GitHub]: madler/zlib - A massively spiffy yet delicately unobtrusive compression library (http://zlib.net)).
This error happens only (obviously, OS: Win, build toolchain: VStudio and assembly speedups enabled) for:
- CMake build (works OK for "${ZLIB_SRC_DIR}/win32/Makefile.msc")
- x64 (AMD64) architecture (works OK for x86)
Below is a "callstack" (top -> down is equivalent to outer -> inner) during decompression.
Normal case:
- inflate (inflate.c)
- inflate_fast (inffast.c)
- ...
Assembler case:
- inflate (inflate.c)
- inflate_fast (contrib/masmx64/inffast8664.c)
- inffas8664fnc (contrib/masmx64/inffasx64.asm)
- ...
Problem:
#2. is missing ("${ZLIB_SRC_DIR}/CMakeLists.txt" doesn't know anything about inffast8664.c), so the chain is broken, leading to an invalid library.
Solution:
Make CMakeLists.txt aware of that file, and that is by adding:
set(ZLIB_SRCS
${ZLIB_SRCS}
contrib/masmx64/inffas8664.c
)
at line ~#158 (enclosed by if(MSVC) and elseif (AMD64) conditionals).
Posting full changes as well.
zlib-1.2.11-msvc_x64_asm_speedups.diff:
--- CMakeLists.txt.orig 2017-01-15 08:29:40.000000000 +0200
+++ CMakeLists.txt 2018-09-03 13:41:00.314805100 +0300
@@ -79,10 +79,10 @@
endif()
set(ZLIB_PC ${CMAKE_CURRENT_BINARY_DIR}/zlib.pc)
-configure_file( ${CMAKE_CURRENT_SOURCE_DIR}/zlib.pc.cmakein
- ${ZLIB_PC} @ONLY)
-configure_file( ${CMAKE_CURRENT_SOURCE_DIR}/zconf.h.cmakein
- ${CMAKE_CURRENT_BINARY_DIR}/zconf.h @ONLY)
+configure_file(${CMAKE_CURRENT_SOURCE_DIR}/zlib.pc.cmakein
+ ${ZLIB_PC} @ONLY)
+configure_file(${CMAKE_CURRENT_SOURCE_DIR}/zconf.h.cmakein
+ ${CMAKE_CURRENT_BINARY_DIR}/zconf.h @ONLY)
include_directories(${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_SOURCE_DIR})
@@ -136,30 +136,34 @@
set(ZLIB_ASMS contrib/amd64/amd64-match.S)
endif ()
- if(ZLIB_ASMS)
- add_definitions(-DASMV)
- set_source_files_properties(${ZLIB_ASMS} PROPERTIES LANGUAGE C COMPILE_FLAGS -DNO_UNDERLINE)
- endif()
+ if(ZLIB_ASMS)
+ add_definitions(-DASMV)
+ set_source_files_properties(${ZLIB_ASMS} PROPERTIES LANGUAGE C COMPILE_FLAGS -DNO_UNDERLINE)
+ endif()
endif()
if(MSVC)
if(ASM686)
- ENABLE_LANGUAGE(ASM_MASM)
+ ENABLE_LANGUAGE(ASM_MASM)
set(ZLIB_ASMS
- contrib/masmx86/inffas32.asm
- contrib/masmx86/match686.asm
- )
+ contrib/masmx86/inffas32.asm
+ contrib/masmx86/match686.asm
+ )
elseif (AMD64)
- ENABLE_LANGUAGE(ASM_MASM)
+ ENABLE_LANGUAGE(ASM_MASM)
set(ZLIB_ASMS
- contrib/masmx64/gvmat64.asm
- contrib/masmx64/inffasx64.asm
- )
+ contrib/masmx64/gvmat64.asm
+ contrib/masmx64/inffasx64.asm
+ )
+ set(ZLIB_SRCS
+ ${ZLIB_SRCS}
+ contrib/masmx64/inffas8664.c
+ )
endif()
- if(ZLIB_ASMS)
- add_definitions(-DASMV -DASMINF)
- endif()
+ if(ZLIB_ASMS)
+ add_definitions(-DASMV -DASMINF)
+ endif()
endif()
# parse the full version number from zlib.h and include in ZLIB_FULL_VERSION
The above is a diff. See [SO]: Run/Debug a Django application's UnitTests from the mouse right click context menu in PyCharm Community Edition? (@CristiFati's answer) (Patching utrunner section) for how to apply patches on Win (basically, every line that starts with one "+" sign goes in, and every line that starts with one "-" sign goes out). I am using Cygwin, btw.
I also submitted this patch to [GitHub]: madler/zlib - Ms VisualStudio - Assembler speedups on x64, but I'm not sure what its fate is gonna be, since there are more than 100 pull requests that are pending.
Output:
e:\Work\Dev\StackOverflow\q029505121\build\x64>"c:\Install\Google\Android_SDK\cmake\3.6.4111459\bin\cmake.exe" -G "NMake Makefiles" -DAMD64=ON "e:\Work\Dev\StackOverflow\q029505121\src\zlib-1.2.11"
-- The C compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler: C:/Install/x86/Microsoft/Visual Studio Community/2015/VC/bin/amd64/cl.exe
-- Check for working C compiler: C:/Install/x86/Microsoft/Visual Studio Community/2015/VC/bin/amd64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - failed
-- Looking for fseeko
-- Looking for fseeko - not found
-- Looking for unistd.h
-- Looking for unistd.h - not found
-- Renaming
-- E:/Work/Dev/StackOverflow/q029505121/src/zlib-1.2.11/zconf.h
-- to 'zconf.h.included' because this file is included with zlib
-- but CMake generates it automatically in the build directory.
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Install/x86/Microsoft/Visual Studio Community/2015/VC/bin/amd64/ml64.exe
-- Configuring done
-- Generating done
-- Build files have been written to: E:/Work/Dev/StackOverflow/q029505121/build/x64
e:\Work\Dev\StackOverflow\q029505121\build\x64>"c:\Install\Google\Android_SDK\cmake\3.6.4111459\bin\cmake.exe" --build . --target zlibstatic
Scanning dependencies of target zlibstatic
[ 5%] Building C object CMakeFiles/zlibstatic.dir/adler32.obj
adler32.c
[ 10%] Building C object CMakeFiles/zlibstatic.dir/compress.obj
compress.c
[ 15%] Building C object CMakeFiles/zlibstatic.dir/crc32.obj
crc32.c
[ 21%] Building C object CMakeFiles/zlibstatic.dir/deflate.obj
deflate.c
Assembler code may have bugs -- use at your own risk
[ 26%] Building C object CMakeFiles/zlibstatic.dir/gzclose.obj
gzclose.c
[ 31%] Building C object CMakeFiles/zlibstatic.dir/gzlib.obj
gzlib.c
[ 36%] Building C object CMakeFiles/zlibstatic.dir/gzread.obj
gzread.c
[ 42%] Building C object CMakeFiles/zlibstatic.dir/gzwrite.obj
gzwrite.c
[ 47%] Building C object CMakeFiles/zlibstatic.dir/inflate.obj
inflate.c
[ 52%] Building C object CMakeFiles/zlibstatic.dir/infback.obj
infback.c
[ 57%] Building C object CMakeFiles/zlibstatic.dir/inftrees.obj
inftrees.c
[ 63%] Building C object CMakeFiles/zlibstatic.dir/inffast.obj
inffast.c
Assembler code may have bugs -- use at your own risk
[ 68%] Building C object CMakeFiles/zlibstatic.dir/trees.obj
trees.c
[ 73%] Building C object CMakeFiles/zlibstatic.dir/uncompr.obj
uncompr.c
[ 78%] Building C object CMakeFiles/zlibstatic.dir/zutil.obj
zutil.c
[ 84%] Building C object CMakeFiles/zlibstatic.dir/contrib/masmx64/inffas8664.obj
inffas8664.c
[ 89%] Building ASM_MASM object CMakeFiles/zlibstatic.dir/contrib/masmx64/gvmat64.obj
Microsoft (R) Macro Assembler (x64) Version 14.00.24210.0
Copyright (C) Microsoft Corporation. All rights reserved.
Assembling: E:\Work\Dev\StackOverflow\q029505121\src\zlib-1.2.11\contrib\masmx64\gvmat64.asm
[ 94%] Building ASM_MASM object CMakeFiles/zlibstatic.dir/contrib/masmx64/inffasx64.obj
Microsoft (R) Macro Assembler (x64) Version 14.00.24210.0
Copyright (C) Microsoft Corporation. All rights reserved.
Assembling: E:\Work\Dev\StackOverflow\q029505121\src\zlib-1.2.11\contrib\masmx64\inffasx64.asm
[100%] Linking C static library zlibstatic.lib
[100%] Built target zlibstatic
Notes:
- I am using VStudio 2015
- Regarding the above output:
- To keep the output as small as possible, I am only building the static version
- For the same reason (and also to keep it as just text), I'm building for "NMake Makefiles" (cmdline build)
- inffas8664.c is being built (somewhere near the end)
- You could also disable assembler speedups (by unchecking AMD64 in cmake-gui), but that would be just a workaround
- I did some rough tests (by far, I'm not claiming these results to be general), and the performance improvement of the assembler implementation over the standard one (Debug versions) was (a percent below is the ratio between times took to perform the same operation (with / without) speedups):
- Compress: ~86%
- Decompress: ~62%
Update #0
([GitHub]: madler/zlib - ASM zlib build on Windows gives erroneous results (@madler's comment)) states:
What assembly code is being used? There are a few in zlib's contrib directory. By the way, the stuff in the contrib directory is not part of zlib. It is just there as a convenience and is supported (or not) by those third-party contributors. What I will do is simply remove the offending code from the next distribution.
So does the compile warning (that everyone must have seen (and most likely ignored)):
Assembler code may have bugs -- use at your own risk
Apparently, assembler speedups and VStudio don't get along very well. More, on x86 there are several issues:
After fixing these, everything works fine, and the performance improvements are similar to x64.
